Evaluating

Organizational Change:
How and Why?

Dr Kate Mackenzie Davey

Organizational Psychology
Birkbeck, University of London
k.mackenzie-davey@bbk.ac.uk
Aims

Examine the arguments for evaluating

organizational change
Consider the limitations of evaluation
Consider different methods for evaluation
Consider difficulties of evaluation in practice
Consider costs and benefits in practice

2
Arguments for evaluating
organizational change
Sound professional practice
Basis for organizational learning
Central to the development of
evidence based practice
Widespread cynicism about fads and
fashions
To influence social or governmental
policy
3
Research and evaluation

Research focuses on relations between

theory and empirical material (data)
Theory should provide a base for policy decisions
Evidence can illuminate and inform theory
Show what does not work as well as what does
Highlight areas of uncertainty and confusion
Demonstrate the complexity of cause-effect
relations
Understand predict control
4
Pragmatic Evaluation: what
matters is what works
Why it works may be unclear
Knowledge increases complexity
Reflexive monitoring of strategy links to OL
& KM
Evidence and cultural context
May be self fulfilling
Tendency to seek support for policy
Extent of sound evidence unclear
5
Why is sound evaluation
so rare?
Practice shows that evaluation is an
extremely complex, difficult and highly
political process in organizations.
Questions may be how many, not what
works

6
Evaluation models
1. Pre-evaluation
2. Goal based (Tyler, 1950)
3. Realistic evaluation (Pawson & Tilley,1997; Sanderson,
2002)
4. Experimental
5. Constructivist evaluation (Stake, 1975)
6. Contingent evaluation (Legge, 1984)
7. Action learning (Reason & Bradbury, 2001)

A study should be technically sound, administratively

convenient and politically defensible. Alec Rodger

7
1.1 Pre-evaluation (Goodman &
Dean, 1982)
The extent to which it is likely that... A
has an impact on b
Scenario planning
Evidence based practice
All current evidence thoroughly reviewed
and synthesised
Meta-analysis
Systematic literature review
Formative v summative (Scriven, 1967)

8
1.2 Pre-evaluation issues

Based on theory and past evidence:

not clear it will generalise to the
specific case
Formative: influences planning
Argument: to understand a system
you must intervene (Lewin)

9
2. 1. Goal based evaluation
Tyler (1950)

Objectives used to aid planned change

Can help clarify models
Goals from bench marking, theory or pre-
evaluation exercises
Predict changes
Measure pre and post intervention
Identify the interventions
Were objectives achieved?
10
2.2 Difficulties with Goal
based evaluation
Who sets the goals? How do you identify
the intervention?
Tendency to managerialism (unitarist)
Failure to accommodate value pluralism
Over-commitment to scientific paradigm
What is measured gets done
No recognition of unanticipated effects
Focus on single outcome, not process
11
3.1 Realistic evaluation:
Conceptual clarity (Pawson &
Tilley,1997)
Evidence needs to be based on clear ideas
about concepts
Measures may be derived from theory
Examine definitions used elsewhere
Consider specific examples
Ensure all aspects are covered

12
3.2 Realistic evaluation
Towards a theory: What are
you looking for?
Make assumptions and ideas explicit
What is your theory of cause and effect?
What are you expecting to change
(outcome)?
How are you hoping to achieve this
change (mechanism)?
What aspects of the context could be
important?
13
3.3 Realistic evaluation
Context-mechanism-
outcome
Context: What environmental
aspects may affect the outcome?
What else may influence the
outcomes?
What other effects may there be?

14
3.4 Realistic evaluation
Context-mechanism-
outcome
Mechanism: What will you do to
bring about this outcome?
How will you intervene (if at all)?
What will you observe?
How would you expect groups to
differ?
What mechanisms do you expect to
operate? 15
3.5 Realistic evaluation
Context-mechanism-
outcome
Outcome: What effect or outcome
do you aim for?
What evidence could show it
worked?
How could you measure it?

16
 4.1 Experimental
evaluation:
Explain, predict and control by identifying causal
relationships
Theory of causality makes predictions about variables
eg training increases productivity
Two randomly assigned matched groups:
experimental and control
One group experiences intervention, one does not
Measure outcome variable pre-test and post-test
(longitudinal)
Analyse for statistically significant differences
between the two groups
Outcome linked back to modify theory
The gold standard

17
4.2 Difficulties with
experimental evaluation in
organizations
Difficult to achieve in organizations
Unitarist view
Leaves out unforeseen effects
Problems with continuous change
processes
Summative not formative
Generally at best quasi-experimental
18
5.1 Constructivist or
stakeholder evaluation

Responsive evaluation (Stake, 1975) or Fourth

generation evaluation (Guba & Lincoln, 1989)
Constructivist interpretivist hermeneutic
methodology
Based on stakeholder claims concerns issues
Stakeholders: agents, beneficiaries, victims

19
5.2 Response to an IT
implementation
(Brown, 1998)

Theme The ward Laboratory IT Team

Goal Improve Improve Clinical and

quality to quality for financial
patients ward staff benefits

Outcome Waste of No Technically

time and improvemen competent
energy on a t to system -
pointless adequate misconceive
system systems d project
20
5.3 Constructivist
evaluation issues
No one right answer
Demonstrates complexity of issues
Highlights conflicts of interests
Interesting for academics
Difficult for practitioners to resolve

21
6 A Contingent approach to
evaluation
(Legge, 1984)

Do you want the proposed change

programme to be evaluated?
(Stakeholders)
What functions do you wish its
evaluation to serve? (Stakeholders)
What are the alternative approaches
to evaluation? (Researcher)
Which of the alternatives best matches
the requirements? (Discussion)
22
7. Action research

Identify good practice

(Reason & Bradbury, 2001) Action research
Responds to practical issues in
organizations
Engages in collaborative relationships
Draws on diverse evidence
Value orientation - humanist
Emergent, developmental
23
Problems with realist
models
Tendency to managerialise
Over-commitment to scientific paradigm
Context stripping,
Over-dependence on measures
Coerciveness: truth as non-negotiable
Failure to accommodate value pluralism
Every act of evaluation is a political act, not
tenable to claim it is value free
24
Problems with
Constructionist approach
Evaluation judged by who for whom
and in whose interests?
Identify different views, then what?
Who has power?
Leaves decisions open
May lead to ambiguity

25
Why not evaluate?
Expensive in time and resources
De-motivating for individuals
Contradiction between scientific evaluation
models and supportive, organization
learning models
Individual identification with activity
Difficulties in objectifying and maintaining
commitment
External evaluation off the shelf
inappropriate and unhelpful
26
Why evaluate?
(Legge, 1984)

Overt Covert
Aids decision Rally
making support/opposition
Reduce uncertainty Postpone a decision
Learn Evade responsibility
Control Fulfil grant
requirements
Surveillance

27
Conclusion
Evaluation is very expensive, demanding
and complex
Evaluation is a political process: need for
clarity about why you do it
Good evaluation always carries the risk of
exposing failure
Therefore evaluation is an emotional
process
Evaluation needs to be acceptable to the
organization

28
Conclusion 2
Plan and decide which model of evaluation
is appropriate
Identify who will carry out the evaluation
and for what purpose
Do not overload the evaluation
process:judgment or development?
Evaluation can give credibility and enhance
learning
Informal evaluation will take place whether
you plan it or not

29