You are on page 1of 17

LOCAL SUSTAINABILTY

FUND

Learning network
evaluation

Faruk Barabhuiya
Samantha Stone

November 2017
Local Sustainability Fund
Learning Network evaluation

CONTENTS

Evaluation summary .................................................................................................................. 1


1 Introduction .................................................................................................................... 2
Learning network objectives..................................................................................................................... 2
What was offered? ................................................................................................................................... 2
Evaluation questions ................................................................................................................................ 3
2 The results ......................................................................................................................... 3
Response rate ........................................................................................................................................... 3
Overall ...................................................................................................................................................... 4
Engagement and quality ........................................................................................................................... 4
What did grantholders gain from the activities? .................................................................................... 12
Suggestions for improvements ............................................................................................................... 13
3 Conclusion ....................................................................................................................... 13

Funded by the Office for Civil Societys Local Sustainability Fund and delivered by Big Lottery Fund
Local Sustainability Fund
Learning Network evaluation

Evaluation summary
In September 2017, 130 LSF grant holders were invited to complete a survey to ascertain the effectiveness of
the five learning network activities; online forum, learn and share events, peer visits, bulletins and a webinar.
In total 38 LSF grantholders completed the survey and the results in this report.

What was the level of engagement of grantholders?


The survey results show grantholders engaged in all five activities to some extent, with a low level of
engagement overall. In-person interaction, such as peer visits and learn and share events, being preferred.

What factors contributed to the varied levels of engagement of


grantholders?
Not having enough time, lack of staff and workload were the most prominent reasons given for a lack of
engagement with learning activities. In-person activities were generally preferred over online interactions -
despite the potential for online activities to save time.

What was the quality of activities provided? What was good? What
would they change?
To differing extents, all five activities were seen as effective opportunities for sharing learning. Survey
respondents appreciated being able to learn about how similar organisations operate and approach mutual
challenges. Suggestions for improvements include a more personal approach, meeting more local
organisations to discuss local issues, web skills training and ensuring all organisations can attend learn and
share events.

Which of the learning network activities did grantholders engage with?


The bulletins had the highest level of engagement. The webinar had the lowest level of engagement.
Grantholders engagement with the learning network overall was light with lack of time and limited staff
capacity as the main reasons given for not engaging it more.

What, if anything, did grantholders gain from the activities?


42% of respondents, 16 respondents have used learning from the activities to inform or change elements of
their work. This included organisational strategy and governance, financial management and fundraising, and
increased awareness of other approaches and practices.

1
Local Sustainability Fund
Learning Network evaluation

1 Introduction
NCVO Charities Evaluation Services (NCVO CES) were commissioned to evaluate the learning network
support offered by the NCVO Research team as part of the Local Sustainability Fund (LSF).

Learning network objectives


The learning network aimed to provide LSF grantholders with the opportunity to:

Learn from fellow grantees


Learn from evaluation activity
Learn about new and different approaches and practices
Learn about what does not work
Have greater understanding of how voluntary, community and social enterprise organisations
can become more sustainable

What was offered?


There were five elements to the support provided.

Online forum (knowledge hub)


The purpose of the online forum was to allow grant holders to connect and share with the evaluation
team and other grant holders.
Bulletins
The research team shared bulletins with the learning network which highlighted key information and
emergent learning. The bulletins were shared via email and posted on the online forum.
Learn and share events
Three events were held in Birmingham, London and Manchester in February 2017. The events aimed to
provide grantholders with an opportunity to find out about the latest findings from the evaluation, learn
from other grantholders about successes and challenges, and provide an opportunity to raise issues
that were important to them.
Peer visits
These aimed to facilitate learning between grantholders by offering small bursaries to cover travel and
subsistence costs. The bursaries were offered on a first come first served basis. Grantholders were able
to select an organisation to visit themselves or have suggestions sent to them based on their chosen
criteria.
Webinar
One webinar was held on the 4 April 2017 to present the evaluation findings to date and offer
grantholders the opportunity to ask any questions.

2
Local Sustainability Fund
Learning Network evaluation

Evaluation questions
This evaluation focuses on the learning network activities provided by NCVO Research Team as part of a
wider evaluation of the LSF. This report addresses the following questions:

What was the level of engagement of grantholders?


What factors contributed to the varied levels of engagement of grantholders?
What was the quality of activities provided? What was good? What would they change?
Which of the learning network activities did grantholders engage with?
What, if anything, did grantees gain from the activities?

To address these questions a short online survey was sent to all LSF grantholders, exploring the five elements
of support. This report is structured around the evaluation questions and presents the findings along usage
statistics and feedback received by the NCVO research team. The survey questions can be found in the
appendix 1.

Methodology
An online survey was distributed to 170 of the 260 total grantholders. Monitoring data was also examined
for peer visits, learn and share event and the webinar.

2 The results
Response rate
The survey was sent to 170 of the 260 grantholder contacts. Of those, 40 (24%) email addresses no longer
worked because the person had left the organisation. This means the total survey population was 130, 50%
of the total number of grantholders. The survey was open for two weeks between 25 September and 2
October 2017and with 38 responses. This is a response rate of 29%.

LSF grants were given in two rounds in year one and two. Because grants lasted for one year only and often
funded a specific role. When the grant finished the individual employed left the organisation in many cases
so could not be included in this evaluation. This also explains why 40 email address that were included no
longer worked. There may also have been an element of survey fatigue among grantholders because the
research teams last snapshot survey, sent to grantholders regularly throughout the programme, had been
sent out only a few weeks before this survey.

3
Local Sustainability Fund
Learning Network evaluation

Overall
The majority of respondents were satisfied or neutral about the learning network overall.
Respondents %
Very satisfied 7 18%
Satisfied 15 39%
Neutral 14 37%
Dissatisfied 1 3%
Very dissatisfied 1 3%
TOTAL 38 100%

Accessibility, timing and content


Respondents were neutral about with content, timing and accessibly of the learning activities with
rating averages of between 2.42-2.45 (1 = very satisfied and 5 = very dissatisfied). Using a five point
scale from very satisfied to very dissatisfied, the rating averages were as follows.
Rating
average
The learning was shared in an accessible way 2.42
The timing of the learning activities was 2.53
helpful to my organisation
The content of the learning activities was 2.45
helpful to my organisation

In an open-text question a number of respondents reflected on the activities:


It could be reassuring to read that we were not seriously behind our peers, or out of sync with the
activities of many grant holders.
They were a great way of engaging with the Sustainability Fund project to learn from other
charities.

Engagement and quality


While engagement was light across all five elements, respondents who did participate felt that they were
effective opportunities to share learning.
The level of engagement was measured in two different ways:
For bulletins and the online forum respondents were asked to rate their level of engagement using a
four point scale from completely to not at all.
For learn and share events, peer visits and the webinar the survey asked whether they engaged, with
a simple yes or no answer.
The bulletins had the highest level of engagement with 29 respondents indicating they kept up to date with

4
Local Sustainability Fund
Learning Network evaluation

them to some extent. The webinar had the lowest level of engagement with only 7 respondents indicating
they engaging with it.
Respondents
engaging to some
extent
Bulletins 29
Online forum 22
Learn and share events 14
Peer visits 10
Webinar 7

Comparing the rating averages for each of the five elements (see the chart below), the two elements that
involved in-person interaction were rated the best in each of the four categories.

Sharing learning Peer visits

Raising questions with the evaluation team and Learn and share events
other grantholders

Share examples of good practice/common Peer visits


challenges

Communicating directly with other grant holders Learn and share events

Using a five point scale from, 1=very satisfied to 5=very dissatisfied, no category scored below 3.

3
2.77 2.76
2.5 2.59 2.5 2.5
2.17
2
1.69 1.6
1.54 1.54 1.54 1.5
1.3
1

0
Learn about the Raise questions Communicate Share examples of Raise questions To reflect on the
successes and with the evaluation directly with other my own good with the evaluation LSF project so far
challenges of other team and other grant holders practice/common team and other
grant holders/share grant holders challenges grant holders
learning

Online forum Learn and share events Webinar Peer visits

5
Local Sustainability Fund
Learning Network evaluation

Effectiveness for sharing learning


Survey respondents indicated that each of the five activities were effective mechanisms for sharing learning.
This is not always connected to how much they actually engaged with each of the activities, indicating that
respondents may have thought that they were effective in theory but that there were reasons why they
personally did not engage with them more, this will be explored later in this report.

19

15
13

10
8
7
5
0 0 1

Online forum Bulletins Learn and share Peer visits Webinar


events

Yes No

Bulletins
There were 13 bulletins which were posted on the online forum by the evaluation team. This was
done instead of including them within an email to try and encourage use of the online forum. An
email was sent to all contacts notifying them of new bulletins.
The majority of respondents (25 out of 33) kept up-to-date with the bulletins a little or quite a bit,
with the majority (18) responding a little showing that engagement was relatively low.
19 respondents (70%) thought they were an effective mechanism for sharing learning.
The quality, frequency and relevance of content were all rated between good and neutral.

To what extent did grantholders keep up to Was it effective? (n=27)


date with bulletins?
Respondents %
Completely 4 12%
No,
Quite a bit 7 21% 8
A little 18 55% Yes,
Not at all 4 12% 19
TOTAL 33 100%

Rating averages for the quality, relevance of content and frequency of the bulletins were each rated between
good and neutral with rating averages of between 2.32 and 2.59 (1 = very good, 2= good, 3= neutral, 4=
poor, 5= very poor) with frequency of bulletins rated the highest and relevant of the content to their
organisation rated the lowest.

6
Local Sustainability Fund
Learning Network evaluation

Rating
(n=27) average
Quality of content 2.42
Relevance of the content to their 2.59
organisation
Frequency of bulletins 2.32

Online forum
All thirteen bulletins were posted on the forum and grantholders were directed there to read them.
Some were attached directly to the email notifications.
The majority of respondents engaged with the online forum a little or not at all. Only two
respondents used it quite a bit. There was only one question asked by a grantholder on the forum.
15 respondents thought it was an effective mechanism for sharing learning. Given the low amount of
activity by grantholders on the forum it may be that they thought that it is an effective mechanism in
theory but lack of time and difficulties engaging with technology (reasons given for lack of
engagement which are explored later in this report) may have been why the online forum was not
used more.

Level of engagement Was it effective? (n=22)


Respondents %
Completely 0 0%
Quite a bit 2 6% No,
A little 20 57% 7
Yes,
Not at all 13 37%
15
TOTAL 35 100%

Reflecting the lack of active engagement on the forum, rating averages for the ways the forum helped their
organisation were more towards neutral. (1= strongly agree, 5= strongly disagree). Learning about the
successes and challenges of other grantholders was rated the highest and communicating with other
grantholders the lowest.

Rating average
Learn about the successes and challenges of other grant holders 2.5

Raise questions with the evaluation team and other grant holders 2.59

Communicate directly with other grant holders 2.77

Share examples of my own good practice 2.76

7
Local Sustainability Fund
Learning Network evaluation

Learn and share events


There were three events held in February 2017 in Birmingham, London and Manchester. Attendance
figures compiled by the research team show that 77 organisations and 100 individuals attended an
event. (30% of all grantholder organisations). This is the highest level of engagement across all five
activities.
14 survey respondents attended an event and 13 of them felt the events were an effective
mechanism for sharing learning. (n=13)

Attendees Orgs
Birmingham 31 30
London 42 29
Manchester 27 18
TOTAL 100 77

Did you attend any of the events? (n=33)

Yes,
No, 14
19

The thirteen respondents that answered the question indicated the learn and share events gave their
organisation an opportunity to learn about successes and challenges, raise questions, communicate with
other grantholders and share examples of good practice.
Meeting others in-person was thought to be the most valuable way of learning with five of the ten
comments mentioning it. One respondent commented:
I think it is easier for people to communicate complex ideas and issues verbally. Meeting people face-
to-face offers opportunity to ask questions.
The rating averages for this element are the highest among all five elements, along with peer visits. One
respondent noted that this was a much more effective [approach] than doing it via an online forum or by
email.

8
Local Sustainability Fund
Learning Network evaluation

Rating
Average
To learn about the successes and challenges of other 1.54
grant holders

To raise questions with the evaluation team and 1.54


other grant holders

To communicate directly with other grant holders 1.54

To share examples of my own good practice 1.69

(1=strongly agree, 5=strongly agree)

Peer visits
30 organisations participated in 15 visits. Participation was optional and funding was provided on a
first come first served basis.
10 survey respondents took part in a peer visit and all thought they were an effective mechanism for
sharing learning and rated them highly for reflecting on the project so far, sharing learning with
another grantholders and sharing ideas on how to overcome common challenges.
Peer visits was the highest rated of the five learning network activities offered. Sharing learning with
other grantholders was rated the highest and reflecting on the LSF project so far the lowest.
Rating
average
Reflect on the LSF project so far 1.6
Share your learning with another grant holder 1.3
Share ideas on how to overcome common 1.5
challenges

Peer visit feedback


Feedback forms were completed after each visit. Only one form was required per visit, however for some
visits both organisations submitted a form each. For this reason 21 forms were received. The NCVO Research
Team distributed a bulletin to all grantholders which collated the learning derived from the information
provided in the forms. The bulletin can be read in appendix xx of the main report.
Peer visits, while only undertaken by a small number of individuals, were the most effective mechanism for
learning and affecting changes within organisations. The feedback form asked the respondents to rate their
satisfaction with various aspects of the visits. The majority were very satisfied with the overall experience
and all elements asked about with rating averages very high, between 1.05 and 1.43 (1 = very satisfied, 5 =
very unsatisfied). This shows how useful the visits were to grantholders and how influential they were in
changing practices within the organisations involved.

9
Local Sustainability Fund
Learning Network evaluation

Rating
average
Overall experience 1.14
Application process 1.05
On the day logistics 1.10
Learning gained 1.14
Impact on practice 1.33
Reflection sheet 1.43

Webinar
22 people took part in the webinar out of 32 registrants.
A survey question at the end of the webinar asked attendees how useful they found it using a five
point scale (1= not at all useful, 5= very useful). The rating average of 2.94 indicates attendees found
the webinar moderately useful.
7 survey respondents of this evaluation took part in the webinar (n=32). 5 thought it was an effective
mechanism for sharing learning (n=6).
Respondents reflected that it allowed their organisation to learn about emergent findings of the
evaluation, learn about successes and challenges and raise questions with the evaluation team.

Rating
average
Learn about the emergent findings 2.17
of the evaluation
Learn about the successes and 2.17
challenges of other grant holders
Raise questions with the evaluation 2.5
team and other grant holders

(1=strongly agree, 5=strongly disagree)

Factors effecting levels of engagement


For each of the five elements, the survey asked for reasons for lack of engagement. Across all five elements
of support the two most prominent reasons were lack of time and workload. Grantholders were often small
organisations, which typically suffer from resourcing challenges and a lack of capacity.
Bulletins
Four respondents commented that the bulletins provided a quick and reliable source of information that I
could then look at if it was relevant. The most frequent reasons given by respondents who did not feel they
were effective or did not engage with them were lack of time (to read them or follow up on any useful
information), and the perception that the content was repetitive or irrelevant to their organisation.

10
Local Sustainability Fund
Learning Network evaluation

Online forum
For those that did not engage with the online forum at all, the most common reason, given by six
respondents, was lack of time to devote to exploring it. The other two most common reasons, given by three
respondents each, were not finding the forum user-friendly and not finding the information on it relevant to
their organisation.
Learn and share events
The most common reason for not attending any events, given by nine respondents was lack of time. Location
of the events was given by three respondents. Lack of staff capacity was stated by three respondents and
workload by two. Other reasons given by one respondent each were:
Distance needed to travel
Lack of funds to cover travel costs
Tried to book too late and had sold out
Did not know they were taking place
Did not feel they were of use
Peer visits
Once again the most common reason, stated by thirteen respondents was lack of time. However four
respondents indicated not knowing about them. Travel distance was stated by two. Other reasons given by
one respondent each were:
Distance needed to travel
Lack of staff capacity
Lack of funds
Webinar
Not having the time to attend was the reason given by eleven respondents for not participating. One
respondent noted,
as a small organisation with part time staff stretched to deliver change etc unfortunately dates like
this are overlooked especially if it doesn't fall on your day in [the office].
Two respondents commented that their workload at the time prevented them taking part. Not liking
webinars and preferring in-person interaction were reasons given by two respondents. Other reasons given
by one respondent each were:
Lack of staff capacity to attend
Lack of skills to enable participation
Non-compatible I.T.
It was not of any interest
Do not find webinars productive

11
Local Sustainability Fund
Learning Network evaluation

What did grantholders gain from the activities?


The survey asked the question, Have you used learning from the activities to inform or change any of the
elements of your work? Results show that only 16 respondents (42%) thought they had done so (n=38).

Yes
No 16
22

Eleven respondents provided examples of the ways in which they have used the learning to inform or change
their work. Learning derived from peer visits was specifically mentioned by five respondents and learning
from others more broadly was mentioned by a further three respondents. Respondents reported learning
about approaches and practices and specifically mentioned fundraising and financial management, showing
how valuable grantholders found the interaction. The following respondent gave three examples of how
learning from the programme has changed their organisations work:
Now have a strong board of trustees. Have had external marketing advice which has inspired use of
social media in a more productive way in order to advertise activities and services. Now have a
comprehensive cash-flow spreadsheet which allows monitoring the charity's finances on a daily
basis.
Learning and approaches
Three respondents reported finding it valuable to learn from other organisations supported through the
programme, As we continue to discuss our own learning we have taken the overall learning to influence our
thinking. Discovering others were dealing with similar issues was also mentioned by two respondents,
Gaining a better understanding of the issues being faced by other organisations and how these were often
parallel to ours. One respondent added that, The thorough nature of the evaluation activity was also a good
learning.
Organisational strategy
From the peer visits, three respondents mentioned learning around organisational strategy and ways of
working, such as informing/validating their approach, finding mutual solutions to shared challenges and
recognising other models of business development.
Financial management and fundraising
Four respondents gave examples of learning relating to financial management and fundraising, including the
respondent quoted at the beginning of this section.
My peer visit was very useful in discussing how the organisation had experienced a financial crisis
and learned from this. I have implemented a number of changes with my own staff team as a result.
From the peer visit I have refocused my energies on profile raising and the importance of being seen
out and about as the CEO to encourage people to donate
Awareness of new funding opportunities and new ways of doing things

12
Local Sustainability Fund
Learning Network evaluation

Suggestions for improvements


There were four suggestions given for how the evaluation team could have improved the learning activities
offered.

Get to know individual organisations more


One respondent would have liked more personal contact and another would have liked the evaluation
team to get to know individual organisations a bit more. This respondent did acknowledge that this
would be a challenge with the resources available, I realise that this would have been difficult with
there being so many.

More spaces on learn and share events


One respondent was disappointed they could not attend the learn and share events and commented it
would have been good if there were spaces for all the organisations to attend a session. Given the
limited space of venues and the number of organisations involved in the programme, practically
speaking this may not have been feasible.

Meeting local organisations


One respondent would have liked to meet up with more local organisations in order to discuss specific
sustainability issues such as difficulties engaging local authorities and Clinical Commissioning Groups.

Web skills training


One respondents would have liked training to make the most of webinars and online forums.

3 Conclusion
Despite low levels of involvement with the learning network respondents to the evaluation survey thought
the mechanisms used were effective. Individuals preferred to interact with their peers face-to-face rather
than online and learnt about challenges shared by other organisations, organisational strategy and financial
management. There were also specific examples given of changes organisations have made as a result of
their participation in the LSF programme.
Face-to-face interaction valued the most
Survey findings and attendance records show that grantholders and survey respondents preferred the two
elements that involved in-person interactions (peer visits and learn and share events) as a way of sharing
learning and interacting with other grantholders. Skills, compatible equipment and aversion to technology
were given as reasons for lack for engagement with the online forum and webinar illustrating that assessing
the digital literacy of grantholders may be appropriate before selecting online tools.

13
Local Sustainability Fund
Learning Network evaluation

Peer visits were the source of most of the learning


Survey respondents reported examples of learning derived from participation in the LSF programme.
Learning from organisations that went on peer visits was specially reported. Learning included helping
organisations to validate or inform their organisations approaches in fundraising and financial management
and improving organisation strategy and ways of working. It also helped organisations discover others who
shared the same challenges and issues.
Examples of practical changes made as a result of learning include; creation of a cash-flow spreadsheet to
improve financial monitoring, a CEO increasing time put to into raising the profile of their organisation and
changes to a staff team to better deal with a financial crisis.
Time and staffing an issue
Findings show that lack of time and staff availability were the two main reasons given for lack of engagement
with the learning activities. The twelve month duration of the grant may also have been a factor because it is
likely to have required the funded organisations to focus their attention on making sure activity took place
within the grant period. As the funded organisations were small or medium sized, many of whom would have
used their grant to create a new position to manage the funded work, it is understandable that they may not
have been able to prioritise taking part in learning activities.

14
About NCVO Charities Evaluation Services
NCVO Charities Evaluation Services (NCVO CES) is the
leading evaluation consultancy for the voluntary sector.
Over the last 26 years we have worked with tens of
thousands of voluntary organisations, their funders and
commissioners, helping them improve their effectiveness.
We do this through:
external evaluations
building organisational capacity in monitoring and evaluation
tailored training
developing resources and toolkits designed to
meet organisations needs.

ncvo.org.uk/charities-evaluation-services

NCVO
Society Building 8
All Saints Street
London N1 9RL

020 7713 6161


ncvo@ncvo.org.uk
ncvo.org.uk

Registered charity number 225922

You might also like