Creating Value, Evaluating

Impact
‘Self-evaluation is a vital part of the discipline imposed
in undertaking creative work’ (Moriarty, 2002)

What is Evaluation and why do it?

About the Project
Measuring Impact
Theories and Models
Next Steps
Kerry Traynor
November 2012
What is evaluation?
 Evaluation involves gathering evidence before, during
and after a project and using it to make judgements
about what happened.
 The evidence should prove what happened and why, and
what effect it had.
 The evidence can also help you to improve what you are
doing during the project and what you do next time.
(Woolf, 2004 cited in Arts Council, 2011)
Why evaluate?
‘Self-evaluation is hard work and time-consuming. The reward
is that it can give us the ability to do things beyond the best of
our present available knowledge’ (Moriarty, 2002)
evaluation
 helps with planning, as it makes you think about what you’re aiming to do,
how you will do it and how you will know if you’ve succeeded
on-going feedback keeps you on track and helps to avoid disasters

evaluation helps you to adapt/change as you go along

evaluation is a good way of dealing with ‘quality assurance’ – you’re keeping an eye

on things to make sure quality is maintained
evaluation helps prove the value of what you are doing

evaluation records your contribution to the field you are working in

your evaluation can help others working in the same field

information you collect can also be used for reporting back to those with an interest

in the project (eg participants, funders) and telling others about what you’ve done
the evidence you collect can support future funding applications


(Arts Council, 2011)

Evaluation helps the arts…
 Rigorous evaluation of our work enables us to :
 accumulate a collective body of evidence
 contributes to the ‘collective practice wisdom of the sector’
 builds a record of our ‘history and achievement’
(Arts Victoria, 2002)

 Cannot evaluate all projects in the same way and in the same depth
 Can be more or less formal, and more or less detailed
 Artistic judgements about process, materials, form and content
 Judgements about the results of what you did and what you have produced
 Process as well as product - quality and impact of both

(Arts Council, 2011)

Planning your evaluation:
 What kinds of information or evidence you are going to include in your
evaluation
 e.g. what people say, what they have done (process and finished work), what
you have done, how participants/audiences responded?
 What questions you are going to ask?
 How you plan to answer those questions –what sort of information you need to
answer the questions and how you will collect it.
 When you should collect the information.
 How you will collect the information?
 e.g. keeping a register, asking people in a questionnaire, asking them to video
their thoughts about a project, keeping a diary, taking photographs, etc.
 How you are going to make sense of the information you have collected?
 How you are going to present the results of the evaluation?
 Who you are going to share it with and how?

(Arts Council, 2011)

About the Project

Awards for All funding / Nov 12 – Aug 13

Aims
 Understand how impact is currently measured and evaluated in small enterprises in
the creative industries
 Understand the strengths/weaknesses/opportunities/risks of evaluating impact
 Review models of impact evaluation and identify appropriate model(s) for use in
creative communities network
 Identify needs and deliver support to evaluate impact more effectively

Outcomes
◦ Impact Evaluation Strategy for Safe Productions and the Creative Communities
Network – finding a shared approach that can measure the impact of the whole
network
◦ One-to-one support for members in implementing impact evaluation
◦ Interactive blog / web archive of useful info http://evaluatingimpact.wordpress.com/
Background
 Charity Manager in Toxteth for 6 yrs
 Director / Chair of Safe Productions for 10 yrs
 Grants Assessor for BBC CIN for 10 yrs
 Researcher & lecturer at LJMU for 8 yrs
 Manager of Community Media Enterprise
 MSc in Governance (Creative Industries)
 PhD in Impact of Local Media
What do we mean by impact?
Inputs + Outputs = Outcomes
Outcomes – What would have happened anyway = Impact
Difference achieved by a project/service
Social, environmental, financial
Exercise
Introduce yourself and your organisation
What do you believe are the main impact(s) of
your organisation?
What is your experience of impact evaluation?
What has worked well for you?
What hasn’t worked so well?
What would you like to do better?
What do you hope this could achieve?
What are your fears / concerns?
Models of Impact Evaluation
Social Return on
Investment (SROI)
European Foundation for
Quality Management
(EFQM) Excellence Model
Prove It?
Quality First
Social Enterprise Balanced
Scorecard
SIMPLE
3rd Sector Performance
Dashboard
AA1000 Assurance Standard
 The Big Picture
 Volunteering Impact
Assessment Toolkit
 ISO9001:2008
 Global Reporting Index
(GRI)
 Investors in People (IIP)
 Eco-Mapping
 EU Eco-Management and
Audit Scheme (EMAS)
 PQASSO
Why SROI?
Robust methodology developed through
academic peer review process
Flexible, universally applicable
(Relatively) straightforward
Endorsed by Cabinet Office
Becoming common practise
SROI: A simplified example
Inputs : £20,000
Outputs : 20 unemployed people complete a 6 month training
programme and get a qualification
Outcomes : 5 people get a job
But : 2 people would have got a job anyway
Impact : 3 people move into employment
Attach monetary values (returns) to that impact e.g.
◦ Reduced costs to state of paying unemployment benefits
◦ Increased income to the state from employment taxes
Calculate returns over 5 years
Divide total returns by investment e.g.
◦ £100,000 / £20,000 = 5:1
Or, for every £1 invested, £5 is created in benefit for society
 A collective approach –
find your common impacts
 Moving people towards employment /
rehousing / financial independence
 Improving the physical environment
 Improving people’s health &
wellbeing
Agree common methodology

Measure and evaluate over time

Convert into monetary values

Find your individual and collective

Social Return On Investment Rates
H&W: A simplified example
WEMWBS 7 or 14 point questionnaire
5 response codes
For participants aged 13+
Each participant completes at start/end
Points awarded for each response
H&W: A simplified example
Over 1 year, 500 people participate across
all member organisations
Results in an average increase of 5 points
per person, or
A total increase of 2,500 points across the
area
Can be analysed by postcode/age/gender
etc.
Visualising our collective impact
Visualising our collective impact
Visualising our collective impact
The Health and Wealth of Nations

The Decline: The Geography of a Recession

But does that really capture impact… ?
 Can we really put a price on social impacts?
 Will SROI let us tell the whole story?
 Or will we need more? Eg. case study, images, quotes, narrative etc.
 Creativity and the Creative Industries – arts & health
 Community development – what is valuable and sustainable? How do we build
strong and resilient communities?
 Social Capital theory – our social, economic and cultural stock (Bourdieu, 1984)
 The ties that bind us - bonding & bridging (Putnam, 2000)
 Decline in social capital = decline in moral and social standards
 Increasing social, democratic and civic engagement
 Pride, shared identity, shared vision - challenging the negative, promoting the
positive
 Alternative socio-economic models – 21 hours (nef)
 Can we use these theories and models to find a better way to demonstrate our
impact?