You are on page 1of 10

`

Owning evaluation, improving


effectiveness: Finding ways to
embed evaluative thinking in
organisational culture and process
Voluntary Sector and Volunteering
Research Conference 2014

Emma Henrion, IOD PARC


Author note: Emma Henrion is a Principal Consultant at IOD PARC, a provider of results-based
performance assessment and organisational development to public sector, third sector and
international organisations in the UK and across the world. The examples and case studies are
drawn from work undertaken by IOD PARC and elsewhere. Any errors are the responsibility of
the author.
Introduction
Most organisations have systems for reporting and evaluating their work and impact, however
the way in which these are used and their effectiveness varies widely. Evaluation and
performance reporting is often seen as an imposed external process for scrutiny and
accountability, and its value and application is not used well within the organisation. We propose
that organisational learning is also an integral part of evaluative activities and that it adds to their
value and organisational ownership. There is a need for a different paradigm of working which
sees evaluating impact as integral to learning about an organisations work, and as a positive
activity owned by the whole organisation. Rather than experiencing evaluation as an add-on
activity, evaluative thinking can be built into regular internal processes and supported through a
culture of organisational learning. An evaluative culture has been defined as one which supports
and enables an approach where evidence is sought out and seen as essential to good
management.
1
We suggest that it also includes active organisational learning.
Evaluative activities have a range of functions, but are broadly undertaken for two main functions:
to provide accountability and to learn as an organisation. Both are important and a balance
between the two is necessary in order to maximise the effectiveness and value of evaluative
activities. Without accountability to stakeholders, funders, and civil society the organisation
cannot show its value and contribution. However, without organisational learning, it will be
difficult for the organisation to set a strategy which reflects its mandate and vision, and to deliver
its aims effectively.
From reviewing IOD PARCs work over 20 years with voluntary and statutory organisations large
and small, in the UK and internationally, and from a literature of evaluation and organisational
learning , we draw our own learning about ways to build and integrate evaluative thinking and
evaluative processes to create an evaluative culture. We still see the value of external
evaluations, and of bringing in an independent perspective and challenge, but see great potential
for organisations improving their own approach to evaluation and learning.
In this paper we use the term evaluative thinking to encompass using evidence to improve
organisational performance whether this is through evaluations, the use of routinely collected
performance and monitoring data or reflections and insight from staff and stakeholders.
Evaluative activities similarly will cover both evaluation and performance and monitoring
activities.
1
Mayne, J., (2008) Building an Evaluative Culture for Effective Evaluation and Results management ILAC Working paper 8. http://www.cgiar-
ilac.org/files/publications/working_papers/ILAC_WorkingPaper_No8_EvaluativeCulture_Mayne.pdf




Evaluation for accountability
Accountability is an important function of evaluative activities, but if it is the sole rationale, it can
inhibit optimal learning. Organisations concerned only with accountability risk taking a
mechanistic approach to evaluative activities which focuses on performance compliance. If
reports are mainly seen as a way of providing accountability to funders and external stakeholders,
or inspectors, their full value is not likely to be used. Staff will see collecting and analysing data as a
time consuming burden, diverting resources from operational activity and not intrinsically useful
or valuable. If reporting is also onerous, it can be viewed negatively as a chore. In this scenario of
reporting mainly for accountability, there will be little positive engagement with collection or use
of evidence by staff and it is unlikely to promote an evaluative culture.
Reporting for accountability purposes can also have a negative impact on how evaluative
activities are used and perceived. A focus on accountability has the implicit potential for
identifying individual or organisational failure, with all the associated anxiety this can bring. In an
organisational culture which blames staff, and where underperformance is seen as failure and a
cause for blame, there can be reluctance to take part in evaluations or to fully support monitoring
from fear of identifying failure and experiencing its consequences, which may be punitive. As a
result, there can be weak or at best equivocal commitment to robust evaluations and to
providing information about activities, as they may lead to a negative personal experience.
Anxiety about publication of results also can risk people unconsciously suppressing or amending
findings so that things do not appear quite so bad, thereby losing the opportunity to find out
what really happened and learn from the evidence. Managers may be loath to report
underperformance at the risk of losing status or their job. There can also (sometimes) be
conscious massaging of findings that show weak performance. For instance in some English NHS
organisations which were under intense pressure to deliver good results, deliberate gaming
strategies were used to distort results to give the desired profile.
2
Neither of these behaviours is
helpful for organisational learning and both tend to obstruct organisational improvement
(though they may help preserve funding streams and jobs).
Accountability, however, has a vital function in identifying purpose and value and must frame
organisational learning. To demonstrate accountability an organisation needs to be able to assess
and report on performance on agreed outputs and outcomes using appropriate measures.
Clarifying outcomes and making sure there is a clear link between activities and intended effect
are critical first steps to enabling evaluative thinking. The questions organisations need to ask
themselves are:
What outcomes do we want to achieve?
What outputs will we make to help achieve those outcomes?
How can we measure how well we have achieved the outcomes?
How will we use performance information?
2
Bevan, G., and Hood, C., Have targets improved performance in the English NHS? (2006) BMJ Vol. 332, pp 419-22.



The processes which are used to develop and iteratively review these questions are important
elements in ensuring that the development of an evaluative framework that is embedded in the
culture and processes of an organisation. An approach which engages staff and stakeholders in
articulating and testing the assumptions and steps for delivery that will underpin the
organisational evaluative framework will help increase their personal commitment to using it.
Working with staff using a structured process to agree targeted outcomes, the outputs needed to
deliver these, and how performance in context will be measured (often called a theory of
change) will help embed ownership of the evaluative framework, the contribution of activities
and understanding of how evidence will be used. Evaluation for accountability purposes can in
this way become meaningful to staff and contribute to learning.
A succinct theory of change from Shelter England an
example
Shelter is a charity that works to alleviate the distress caused by homelessness and bad housing.
We do this by giving advice, advocacy to people in housing need and by campaigning for lasting
political change to end the housing crisis for good.
3

A theory of change is also a tool to help articulate the assumptions that explain the steps that lead
to achieving the organisations long-term objectives and associated outcomes, and the
connections between activities and outcomes that occur at each step of the way. Most
organisational strategies can be read as one. At a higher level the theory of change sets out what
impact you want to have, how you will achieve it and how you will understand the nature of the
contribution that the organisation is bringing to social change. This facilitates communication of
purpose to staff and external stakeholders; it also allows the organisation to build the logic model
that shows how it will be delivered and helps tracking and reflection on progress. Logic models
identify the intended outcomes and impact; however for most social change models this is not a
linear causal progression, as there will be a multiplicity of factors that are likely to influence the
progression from short-term to medium term to longer term outcomes (impact)

3
See the Shelter England website: http://england.shelter.org.uk/about_us/who_we_are




Figure 1 Simple logic model for health improvement in a deprived area (derived from an
example from NHS Scotland Evaluation)
Drivers Inputs Outputs Outcomes
Short Medium Long










Involving staff in in developing the theory of change and relevant measures in turn fosters
cultural and personal ownership, increasing the likelihood of robust reporting and review by staff.
Staff engagement will be further strengthened if there is a clear relationship between the theory
of change and the performance systems used to measure it, so that reporting and evaluation of
effectiveness and impact is built into day to day monitoring. Integrating strategic objectives with
day to day monitoring in this way also identifies the contribution of staff and strengthens their
ownership and responsibility for effective working. This is often called a golden thread
identifying the contribution of activities at all levels to the overall outcomes.
Periodic reviews of the assumptions behind the theory of change and the measures used will help
make sure they are still valid and accurate and provide a way of keeping the evaluative framework
live. As the external environment changes or your organisation develops, it may be necessary to
amend and review the original model to ensure it is still relevant and useful. This helps to ensure
that there is active use of the internal and external evidence and that learning is feeding back
systematically into revising the underlying strategy and its implementation. It is also helpful to
consider over what time frame medium term and long term outcomes will be achieved, to
ensure that ambitions are realistic.

Area of
deprivation,
with poor
health, high
take up of
welfare
benefits

Staff
Volunteers
Community
centre
Money

Use of crche
Take up of
therapy
Use of money
advice services
Training

Improved
knowledge of
factors affecting
health.
Access to
support and
advice.
Less isolated.

People
have an
increased
sense of
control
over their
lives.
Local
people are
healthier.
People are
more able to
cope.
People make
healthier
choices.
Increased
connections
with groups.
External factors
Assumptions



Setting realistic time frames case study
In a review of an international project to improve water management in developing countries,
which allocated funding in three year cycles over a ten year period, a need was identified to plan
for intermediate results, linked to the 3 year planning cycle. These allowed formative learning,
within a clear overall framework of assessing the strategic objectives and desired impact over the
10 year programme. Using shorter cycles allowed the GWI to review achievements every three
years, learn from each review and revise or refocus the program as necessary to ensure delivery
of the overall project outcomes
Developing a plan for using the information at the outset helps to ensure that the right
information is being collected for the right time periods, and can avoid over burdening systems
and staff. How will the information be used? Who is the audience? What do they need to know?
The final utilisation of findings needs to be agreed before finalising content and reporting
formats to make sure evidence is valuable and used.
A summary of the benefits of evaluation for accountability is that it provides a very clear
evaluative structure which sets out outcomes, outputs and allows measures to be set. It helps
ensure there is clarity of purpose and direction, and that underlying activities are clearly
contributing to intended outcomes, so that actions and use of resources can be prioritised. It will
help frame what is reported to whom and when; and keeping the planned use of findings in mind
will provide a second check for their relevance and utility. If staff are involved in the process of
agreeing and reviewing the evaluative structure, there is greater potential for staff ownership and
engagement in managing the evaluation and reporting process and using findings.
4

Evaluation for organisational learning
Evaluation for organisational learning purposes takes a slightly different approach in that it is
focused more on the development and improvement of the organisation and its work, but it
closely complements evaluation for accountability. An organisation which values organisational
learning helps close the gap between evaluation and planning through linking the processes
which underpin them. It can help build effectiveness as learning will draw on and encourage
active contributions from staff and better use and sharing of knowledge.
5
This approach uses a
wide range of evidence creatively to improve knowledge and understanding, including soft
organisational and contextual intelligence. It also sees learning and knowledge as drivers for
organisational development and function.
4
Carden, F., Earl, S., Infusing Evaluative Thinking as Process Use: The Case of the International Development Research Centre (IDRC)
(2007) New Directions for Evaluation, n116 p61-73
5
This section on organisational learning draws on a paper by Bruce Britton, which provides a thoughtful review of organisational learning
for NGOs. Britton, B., Organisational Learning in NGOs: creating the motive, means and opportunity. (2005) Praxis Paper no. 3 INTRAC
http://www.intrac.org/resources.php?action=resource&id=398




An organisation which actively values, supports and uses learning will better motivate staff to
learn. To make the leap from accountability to learning, staff need to be confident that their
ideas and learning will be listened to and acknowledged. There will need to be examples of staff
learning and proposals being put into practice, even if not all staff ideas are used. Such a culture
seeks staffs input to improvement, and will generally value and listen to staff. Without this
commitment and validation, staff and managers are unlikely to prioritise learning.
For effective learning to take place, it must be supported by a positive organisational culture
which seeks out what works well as well as what does not, and why. This does not mean being
uncritical nor ignoring poor performance, but using information on what does not work well as a
learning opportunity, not as a blaming opportunity. Looking for the positive, possibly drawing
on appreciative inquiry methods,
6
encourages active learning, and helps mitigate anxieties about
failure which will still persist even in a positive evaluative culture. There is a need to be
transparent and have a commitment to openness; real learning and development can only take
place if there can be genuine scrutiny.
An evaluative culture which values learning also needs to invest sufficient time to learn and to
gather evidence. Teams need time to hear the evaluation findings, discuss them and consider
how they can be best addressed through changes to priorities, resource use or processes. Time
and capacity is required for training on learning and information use and, most importantly, for
collecting and analysing information.
Organisations need to build a positive and engaged process for gathering and using data. Just as
there can be organisational resistance to monitoring and evaluation due to fear of adverse
findings, there can be reluctance to use recording and reporting tools which may be the means of
assessing performance. This can particularly be the case for organisations working in professional
sectors where robust evidence gathering is not highly valued or understood. Understanding of
what constitutes evidence may need to be broadened to include service users views, relevant
social or behavioural changes, and feedback about effective influencing as well as quantitative
outputs and inputs. The use of information needs to be well understood and shown to be valued
and promoted by senior leadership.

6
Cooperrider, D.L. & Whitney, D. (2001). A positive revolution in change. In Cooperrider, D. L. Sorenson, P., Whitney, D. & Yeager, T.
(eds.) Appreciative Inquiry: An Emerging Direction for Organization Development (929). Champaign, IL: Stipes.



Learning why measurement counts case study
A small charity providing a therapeutic service needed to understand the profile of cases, as well
as their volume, duration and outcomes to understand how well it was achieving its aims, manage
resources and report to funders. In addition the charity needed to understand and record
qualitative feedback and perceptions of clients and practitioners to assure quality and inform
areas for improvement.
The therapists practice was to take copious (confidential) case notes, and to complete paper
forms on summary case details. The forms were poorly and inconsistently completed. There were
two main challenges and an opportunity; the reporting system was inefficient and poorly
designed, so therapists did not use it, and they could not see the point of the information it
recorded and were therefore not committed to recording. However, they did value their own
case notes highly as vital information on quality.
Through facilitated conversations between managers and therapists the latter began to
understand the requirements for some of the more mundane data; took time to comment on
system redesign which led to a new form, which was eventually used on a tablet; and were also
able to include the relevant elements of qualitative information they wanted to record and which
were needed to assure and improve quality, on the standard case form. A database was set up to
take the drudgery out of collation and analysis. Regular whole organisation reviews of
performance were established to allow the practitioners to discuss workload and quality.
It is also important to be clear about how evaluative information is used. From experience of
working with a wide range of organisations it is clear that findings are sometimes not used fully
or, in some cases, at all. This undermines the value of expenditure and staff effort on evaluative
activities, and is likely to weaken organisational commitment to improving effectiveness. An
evaluative culture will use evaluation findings to learn and change and must have the capacity and
will to do so. It follows that active use and communication of findings is an indicator of the extent
to which an organisation has embedded evaluative thinking. An organisation which can reflect
well and quickly on evaluation evidence and act on it is also likely to be more able to adapt
successfully to wider contextual and organisational change.
Use will include regular reviews of activity, quality and effectiveness by teams and the wider
organisation which will be undertaken as part of operational management but also as a periodic
review of the overall strategy and associated theories of change. It will also include ad hoc
moments of discovery by teams; sharing good practice across the organisation or externally;
identifying areas where more learning is needed. All of these give greater value to learning and
help embed it in the organisation, as well as creating greater interest and support to further
develop learning.




Developing a positive approach to evaluative learning
case study
In a charitable trust which provided residential care, the directors sought a new model to improve
the assessment and quality assurance of care, following several adverse Care Quality Commission
inspections of different homes.
Within this project, there were a number of challenges to address, including: staff who were not
clear about what was expected of them, and felt little ownership for their work; lack of positive
recognition for good practice; burdensome paper based monitoring processes; anxiety about the
consequences of poor inspection results and possible job losses.
Following focus groups with staff and managers and users it was agreed that what staff and
managers both wanted (in addition to some system changes) was a learning approach that built
regular opportunities for teams to learn from each other and share good practice into the new
quality assurance and review process .
A clear future vision was developed by staff to have staff team ownership and leadership of
quality in the homes to provide excellent care and support.
A key element of ensuring good learning from evaluative activities is good communication within
the organisation and evaluations are strengthened by building in time and resource for a
communication plan at the end of the work. This provides an opportunity to share knowledge; it
also demonstrates a commitment to transparency, learning and to engagement with the wider
organisation. Organisational cascading of knowledge to teams, backed up by written documents
either emailed or printed are obvious conventional ways of getting information to people.
Electronic communication enables other methods such as webinars, which can also facilitate
listener participation where organisations are spread across different locations.
A supportive leadership is widely seen as vital to creating an organisation that values learning. This
includes leaders modelling behaviours which show a personal commitment to organisational
learning and listening to others. But it will also mean demonstrating that organisational learning is
given priority and resources within the overall strategy, and there is accountability for learning.
Leaders need to communicate the importance of learning to the organisation, and foster a
supportive culture as well as one which seeks to use robust evidence to improve.
Supportive leadership will also help contain organisational and individual anxiety, which is a
constant element in any evaluative activities. Even where a supportive, non-blaming culture exists,
the very action of scrutiny raises the possibility of a negative finding with painful consequences. It
may be perceived to be potentially harmful to the actual or psychic survival of an individual, a
team or function or the organisation as a whole. Anxiety about organisational survival can be a
real concern for groups working with marginalised groups or unpopular social issues, such as


asylum seekers, abortion rights or substance use, where the organisations work may be subject
to political, social or even physical attack. Learning in itself can engender anxiety for adults who
perceive they are unlearning existing behaviours or skills, and being required to learn or
understand something new. A safe and supportive environment is needed to manage anxiety
about individual survival in the case of evaluative findings which could lead to job loss or status
change.
7
There is therefore a need to constantly reinforce a positive stance towards the purpose
and use of evaluation, along with reiterations (and practical examples) of a commitment to staff
safety and support.
A summary of the benefits of evaluation for organisational learning is that it enables the
organisation to use evidence to understand and develop the organisation, and maximises its
application. It helps foster a culture of engaged thought, and commitment to learning from
evaluative activities to improve the quality of work and organisational function. Staff have the
opportunity and the confidence to lead changes for improvement, and feel involved in shaping
the organisations activities. Evidence is well used and the evaluative framework used is seen as
integral to the work and is part of the process of planning. Evidence is valued for its contribution
to knowledge. An organisation which achieves this approach is also likely to be more agile in
responding to internal and external changes.
Bringing together accountability and learning to
form an evaluative culture
Both evaluation for accountability and evaluation for learning are valuable. The former helps
ensure there is a clear evaluative framework, with clear outcomes and measures identified, and
strong, rigorous analysis underpinning evaluation. The latter ensures that the full value of
evidence can be used to inform new developments, improve quality and organisational
adaptation and change. To achieve the most effective use of evaluation evidence, both
approaches need to be integrated within organisational processes and values, so that there is a
balanced use of each. Without attention to accountability, there is a risk of losing rigour and
direction; without attention paid to learning, there is a risk of loss of staff engagement and
effective use of knowledge. However, bringing the approaches together can help create an
evaluative culture, which seeks out evidence and sees this as essential to good management and
organisational learning.

7
Edgar H. Schein: The Anxiety of Learning," (2002) Harvard Business Review, Vol. 80, No. 3.