Professional Documents
Culture Documents
FUND
Learning network
evaluation
Faruk Barabhuiya
Samantha Stone
November 2017
Local Sustainability Fund
Learning Network evaluation
CONTENTS
Funded by the Office for Civil Societys Local Sustainability Fund and delivered by Big Lottery Fund
Local Sustainability Fund
Learning Network evaluation
Evaluation summary
In September 2017, 130 LSF grant holders were invited to complete a survey to ascertain the effectiveness of
the five learning network activities; online forum, learn and share events, peer visits, bulletins and a webinar.
In total 38 LSF grantholders completed the survey and the results in this report.
What was the quality of activities provided? What was good? What
would they change?
To differing extents, all five activities were seen as effective opportunities for sharing learning. Survey
respondents appreciated being able to learn about how similar organisations operate and approach mutual
challenges. Suggestions for improvements include a more personal approach, meeting more local
organisations to discuss local issues, web skills training and ensuring all organisations can attend learn and
share events.
1
Local Sustainability Fund
Learning Network evaluation
1 Introduction
NCVO Charities Evaluation Services (NCVO CES) were commissioned to evaluate the learning network
support offered by the NCVO Research team as part of the Local Sustainability Fund (LSF).
2
Local Sustainability Fund
Learning Network evaluation
Evaluation questions
This evaluation focuses on the learning network activities provided by NCVO Research Team as part of a
wider evaluation of the LSF. This report addresses the following questions:
To address these questions a short online survey was sent to all LSF grantholders, exploring the five elements
of support. This report is structured around the evaluation questions and presents the findings along usage
statistics and feedback received by the NCVO research team. The survey questions can be found in the
appendix 1.
Methodology
An online survey was distributed to 170 of the 260 total grantholders. Monitoring data was also examined
for peer visits, learn and share event and the webinar.
2 The results
Response rate
The survey was sent to 170 of the 260 grantholder contacts. Of those, 40 (24%) email addresses no longer
worked because the person had left the organisation. This means the total survey population was 130, 50%
of the total number of grantholders. The survey was open for two weeks between 25 September and 2
October 2017and with 38 responses. This is a response rate of 29%.
LSF grants were given in two rounds in year one and two. Because grants lasted for one year only and often
funded a specific role. When the grant finished the individual employed left the organisation in many cases
so could not be included in this evaluation. This also explains why 40 email address that were included no
longer worked. There may also have been an element of survey fatigue among grantholders because the
research teams last snapshot survey, sent to grantholders regularly throughout the programme, had been
sent out only a few weeks before this survey.
3
Local Sustainability Fund
Learning Network evaluation
Overall
The majority of respondents were satisfied or neutral about the learning network overall.
Respondents %
Very satisfied 7 18%
Satisfied 15 39%
Neutral 14 37%
Dissatisfied 1 3%
Very dissatisfied 1 3%
TOTAL 38 100%
4
Local Sustainability Fund
Learning Network evaluation
them to some extent. The webinar had the lowest level of engagement with only 7 respondents indicating
they engaging with it.
Respondents
engaging to some
extent
Bulletins 29
Online forum 22
Learn and share events 14
Peer visits 10
Webinar 7
Comparing the rating averages for each of the five elements (see the chart below), the two elements that
involved in-person interaction were rated the best in each of the four categories.
Raising questions with the evaluation team and Learn and share events
other grantholders
Communicating directly with other grant holders Learn and share events
Using a five point scale from, 1=very satisfied to 5=very dissatisfied, no category scored below 3.
3
2.77 2.76
2.5 2.59 2.5 2.5
2.17
2
1.69 1.6
1.54 1.54 1.54 1.5
1.3
1
0
Learn about the Raise questions Communicate Share examples of Raise questions To reflect on the
successes and with the evaluation directly with other my own good with the evaluation LSF project so far
challenges of other team and other grant holders practice/common team and other
grant holders/share grant holders challenges grant holders
learning
5
Local Sustainability Fund
Learning Network evaluation
19
15
13
10
8
7
5
0 0 1
Yes No
Bulletins
There were 13 bulletins which were posted on the online forum by the evaluation team. This was
done instead of including them within an email to try and encourage use of the online forum. An
email was sent to all contacts notifying them of new bulletins.
The majority of respondents (25 out of 33) kept up-to-date with the bulletins a little or quite a bit,
with the majority (18) responding a little showing that engagement was relatively low.
19 respondents (70%) thought they were an effective mechanism for sharing learning.
The quality, frequency and relevance of content were all rated between good and neutral.
Rating averages for the quality, relevance of content and frequency of the bulletins were each rated between
good and neutral with rating averages of between 2.32 and 2.59 (1 = very good, 2= good, 3= neutral, 4=
poor, 5= very poor) with frequency of bulletins rated the highest and relevant of the content to their
organisation rated the lowest.
6
Local Sustainability Fund
Learning Network evaluation
Rating
(n=27) average
Quality of content 2.42
Relevance of the content to their 2.59
organisation
Frequency of bulletins 2.32
Online forum
All thirteen bulletins were posted on the forum and grantholders were directed there to read them.
Some were attached directly to the email notifications.
The majority of respondents engaged with the online forum a little or not at all. Only two
respondents used it quite a bit. There was only one question asked by a grantholder on the forum.
15 respondents thought it was an effective mechanism for sharing learning. Given the low amount of
activity by grantholders on the forum it may be that they thought that it is an effective mechanism in
theory but lack of time and difficulties engaging with technology (reasons given for lack of
engagement which are explored later in this report) may have been why the online forum was not
used more.
Reflecting the lack of active engagement on the forum, rating averages for the ways the forum helped their
organisation were more towards neutral. (1= strongly agree, 5= strongly disagree). Learning about the
successes and challenges of other grantholders was rated the highest and communicating with other
grantholders the lowest.
Rating average
Learn about the successes and challenges of other grant holders 2.5
Raise questions with the evaluation team and other grant holders 2.59
7
Local Sustainability Fund
Learning Network evaluation
Attendees Orgs
Birmingham 31 30
London 42 29
Manchester 27 18
TOTAL 100 77
Yes,
No, 14
19
The thirteen respondents that answered the question indicated the learn and share events gave their
organisation an opportunity to learn about successes and challenges, raise questions, communicate with
other grantholders and share examples of good practice.
Meeting others in-person was thought to be the most valuable way of learning with five of the ten
comments mentioning it. One respondent commented:
I think it is easier for people to communicate complex ideas and issues verbally. Meeting people face-
to-face offers opportunity to ask questions.
The rating averages for this element are the highest among all five elements, along with peer visits. One
respondent noted that this was a much more effective [approach] than doing it via an online forum or by
email.
8
Local Sustainability Fund
Learning Network evaluation
Rating
Average
To learn about the successes and challenges of other 1.54
grant holders
Peer visits
30 organisations participated in 15 visits. Participation was optional and funding was provided on a
first come first served basis.
10 survey respondents took part in a peer visit and all thought they were an effective mechanism for
sharing learning and rated them highly for reflecting on the project so far, sharing learning with
another grantholders and sharing ideas on how to overcome common challenges.
Peer visits was the highest rated of the five learning network activities offered. Sharing learning with
other grantholders was rated the highest and reflecting on the LSF project so far the lowest.
Rating
average
Reflect on the LSF project so far 1.6
Share your learning with another grant holder 1.3
Share ideas on how to overcome common 1.5
challenges
9
Local Sustainability Fund
Learning Network evaluation
Rating
average
Overall experience 1.14
Application process 1.05
On the day logistics 1.10
Learning gained 1.14
Impact on practice 1.33
Reflection sheet 1.43
Webinar
22 people took part in the webinar out of 32 registrants.
A survey question at the end of the webinar asked attendees how useful they found it using a five
point scale (1= not at all useful, 5= very useful). The rating average of 2.94 indicates attendees found
the webinar moderately useful.
7 survey respondents of this evaluation took part in the webinar (n=32). 5 thought it was an effective
mechanism for sharing learning (n=6).
Respondents reflected that it allowed their organisation to learn about emergent findings of the
evaluation, learn about successes and challenges and raise questions with the evaluation team.
Rating
average
Learn about the emergent findings 2.17
of the evaluation
Learn about the successes and 2.17
challenges of other grant holders
Raise questions with the evaluation 2.5
team and other grant holders
10
Local Sustainability Fund
Learning Network evaluation
Online forum
For those that did not engage with the online forum at all, the most common reason, given by six
respondents, was lack of time to devote to exploring it. The other two most common reasons, given by three
respondents each, were not finding the forum user-friendly and not finding the information on it relevant to
their organisation.
Learn and share events
The most common reason for not attending any events, given by nine respondents was lack of time. Location
of the events was given by three respondents. Lack of staff capacity was stated by three respondents and
workload by two. Other reasons given by one respondent each were:
Distance needed to travel
Lack of funds to cover travel costs
Tried to book too late and had sold out
Did not know they were taking place
Did not feel they were of use
Peer visits
Once again the most common reason, stated by thirteen respondents was lack of time. However four
respondents indicated not knowing about them. Travel distance was stated by two. Other reasons given by
one respondent each were:
Distance needed to travel
Lack of staff capacity
Lack of funds
Webinar
Not having the time to attend was the reason given by eleven respondents for not participating. One
respondent noted,
as a small organisation with part time staff stretched to deliver change etc unfortunately dates like
this are overlooked especially if it doesn't fall on your day in [the office].
Two respondents commented that their workload at the time prevented them taking part. Not liking
webinars and preferring in-person interaction were reasons given by two respondents. Other reasons given
by one respondent each were:
Lack of staff capacity to attend
Lack of skills to enable participation
Non-compatible I.T.
It was not of any interest
Do not find webinars productive
11
Local Sustainability Fund
Learning Network evaluation
Yes
No 16
22
Eleven respondents provided examples of the ways in which they have used the learning to inform or change
their work. Learning derived from peer visits was specifically mentioned by five respondents and learning
from others more broadly was mentioned by a further three respondents. Respondents reported learning
about approaches and practices and specifically mentioned fundraising and financial management, showing
how valuable grantholders found the interaction. The following respondent gave three examples of how
learning from the programme has changed their organisations work:
Now have a strong board of trustees. Have had external marketing advice which has inspired use of
social media in a more productive way in order to advertise activities and services. Now have a
comprehensive cash-flow spreadsheet which allows monitoring the charity's finances on a daily
basis.
Learning and approaches
Three respondents reported finding it valuable to learn from other organisations supported through the
programme, As we continue to discuss our own learning we have taken the overall learning to influence our
thinking. Discovering others were dealing with similar issues was also mentioned by two respondents,
Gaining a better understanding of the issues being faced by other organisations and how these were often
parallel to ours. One respondent added that, The thorough nature of the evaluation activity was also a good
learning.
Organisational strategy
From the peer visits, three respondents mentioned learning around organisational strategy and ways of
working, such as informing/validating their approach, finding mutual solutions to shared challenges and
recognising other models of business development.
Financial management and fundraising
Four respondents gave examples of learning relating to financial management and fundraising, including the
respondent quoted at the beginning of this section.
My peer visit was very useful in discussing how the organisation had experienced a financial crisis
and learned from this. I have implemented a number of changes with my own staff team as a result.
From the peer visit I have refocused my energies on profile raising and the importance of being seen
out and about as the CEO to encourage people to donate
Awareness of new funding opportunities and new ways of doing things
12
Local Sustainability Fund
Learning Network evaluation
3 Conclusion
Despite low levels of involvement with the learning network respondents to the evaluation survey thought
the mechanisms used were effective. Individuals preferred to interact with their peers face-to-face rather
than online and learnt about challenges shared by other organisations, organisational strategy and financial
management. There were also specific examples given of changes organisations have made as a result of
their participation in the LSF programme.
Face-to-face interaction valued the most
Survey findings and attendance records show that grantholders and survey respondents preferred the two
elements that involved in-person interactions (peer visits and learn and share events) as a way of sharing
learning and interacting with other grantholders. Skills, compatible equipment and aversion to technology
were given as reasons for lack for engagement with the online forum and webinar illustrating that assessing
the digital literacy of grantholders may be appropriate before selecting online tools.
13
Local Sustainability Fund
Learning Network evaluation
14
About NCVO Charities Evaluation Services
NCVO Charities Evaluation Services (NCVO CES) is the
leading evaluation consultancy for the voluntary sector.
Over the last 26 years we have worked with tens of
thousands of voluntary organisations, their funders and
commissioners, helping them improve their effectiveness.
We do this through:
external evaluations
building organisational capacity in monitoring and evaluation
tailored training
developing resources and toolkits designed to
meet organisations needs.
ncvo.org.uk/charities-evaluation-services
NCVO
Society Building 8
All Saints Street
London N1 9RL