Approaches to Public

Policymaking, Policy Analysis
& Evaluation Research
Kathy Luckett
University of Cape Town


CSH  Critical Realism – theory-based evaluation . poststructuralist  Critical – PAR. empowerment evaluation. pragmatic  Interpretative – constructionist.Research Paradigms  Not rigid paradigmatic incommensurability  A map for navigating choppy waters around policy analysis and evaluation methodological debates  Post-positivist – experimental.

outcomes & impact Human performance can be objectively measured tv.outputs. statistical measurement techniques Social science can contribute to improved governance or management Establish cause & effect relations bet policy/ programme objectives. rational Problem: to secure internal validity of evaluation results By 19702 disillusion set in. Efficiency & effectiveness criteria Evaluator: objective. comparison group designs) .Post-positivist: Quasi-experimental  Popper. times series. shift to quasi-experimental methods (pre. Lasswell. 1966). Campbell & Stanley (1963.& post tests. inputs & interventions --------.        Lipsey & Freeman Based on methods of the natural sciences. Rossi. neutral.

atomistic objects & events (ignores the non-observable) Causality = regularities bet variables within stat.Critique: Quasi-experimental models  Uses a model developed for closed systems for open     social systems Adopts a flat ontology – reality = regularities bet observable. sig. seldom explanation . samples Claims about causation usually unclear and unconvincing Can only provide descriptions (for a few variables on large populations).

workable results. programme accreditation  Sets up criteria and performance indicators to measure performance & accountability by institutions and individuals – a closed system . useful for decision-making  Takes policy/ programme goals as focus of evaluation  Methods: a) open-ended case study (improvement) e.g. the ‘evaluative state’ – wants practical. Patton b) closed-system sets up criteria & performance indicators to measure performance & accountability of individuals & institutions e.Post-positivist: Pragmatic (dominant model)  Developed from ‘new public management’ .g.

difficult to prove cause & effect  Ignores context & stakeholder meanings.Critique: Pragmatic Models  Assumes stable external environment  Difficult to set measurable objectives. criteria & indicators for actual performance  Difficult to control variables in open soc systems .possibility of rival explanations. ‘black box’ evaluation – seldom diagnostic  Can be prescriptive. leading to conformity .

1980s policy sociology: meaning socially constructed.Interpretive: Constructionist  1970s – 80s ‘linguistic turn’. human action culturally and discursively mediated – rejection of naturalism  Vickers (1995) policymaking as communicative activity for institutional regulation. values & interests. 4th generation evaluation: focus on subjective stakeholder meanings. evaluation useful to insiders . truth as agreement. evaluator as facilitator. a process of normsetting  Neo-institutional theory emphasises cognitive and normative factors in policy adoption and implementation  Guba & Lincoln (1989).

truth located in subjectivities of respondents  Ignores systemic asymmetries of power  Inability to rise above context  Relativist ontology .Critique: of Constructionist Models  Over-socialised. emphases subjectivity at expense of structure.

govern? How do they become institutionalised & supported legislatively. sets up subject positions that constrain ways of speaking & thinking Technologization of language for institutional ends How do certain discourses become dominant? What discourses are at work when those who govern. norms & values Policy as political artefact – as text & discourse – with unequal material & discursive effects that should be exposed Policy has a normalising & regulatory role. in dialectical relation to     practice – sets up systems of power/ knowledge. Ball (1993).Interpretive: Post-structuralist  Foucault’s ‘geneaology’. professionally & financially? . Gale (2000)  Discourse is socially constitutive.

tendency to jump from data to (preconceived) narrative  Quest to successfully link the micro and macro levels of analysis difficult to achieve  All of social life gets reduced to discourse.(materiality of the social world gets lost)  Knowledge reduced to conditions of its production and interests of its producers (epistemological relativity) .Critique: Post-structuralist Models  Weak on method. dominance of researcher as interpreter. selectivity of data.

Critical: Emancipatory  Neo-Marxist insights. value clarification – diff groups of stakeholders . Frankfurt School (Habermas empancipatory interest)  Critical policy analysis (the ‘argumentative turn’) policy discourses construct social problems & policy solutions. policymaking a form of argument to persuade & manufacture consent  Challenge: how do discourses become institutionalised & reflected in institutional practices?  Ulrich (1994) Critical systems heuristics: policy to be normatively acceptable to those affected by it.

inclusion of marginalised groups affected by the results . giving voice to the silenced.Critical: Emancipatory  Developmental evaluation (Patton)  PAR  Empowerment evaluation (Fetterman 1996)  Transformative evaluation (Mertens 2005)  Development of evaluees.

all communication already penetrated by power Why should the involved (the powerful) bother to take into account the views and concerns of the affected (the powerless)? Cannot work under conditions of coercion requires a fully functioning public sphere Needs to hold material conditions and structures as contexts for vlaues & interests Post-structuralists: consensus is neither possible nor desirable .Critique: Emancipatory Models  Utopian: the ‘better argument’ is produced through     power not rational dialogue .

actual     (events) & real (non-observable structures & causal powers) Holds tog ontological realism + epistemological relativism Both agency & structure have causal powers – attend to both (analytically separate) Openess of the social world. Sayer (1992. 2000)  Reality is stratified – empirical (experiences). plurality and contingency of causality Key to successful intervention = change of social practice . 1998).Critical Realist: Theory-based  Bhaskar (1978.

how. timeconsuming .Critical Realist: Theory-based  Pawson & Tilley (1997) Realist evaluation: what     works. but places in context of wider social explanation Evaluation can be cumulative – middle range theories Critique: Demanding to operationalise. for whom and under what conditions? (builds in context & subjectivity) Evaluator to make programme theory explicit & to check it out with stakeholders: C + M = O tests assumptions about causal relations & change Tests goal realisation.

epistemology 1st – then methodology! . stage of the policy/programme cycle & practical constraints  Think purpose (teleology). select according to values. ontology.Conclusion  Be aware of tradition & model you’re working in . purpose of evaluation.& of other possibilities  Complex nature of policy analysis & evaluation justifies methodological pluralism  But don’t use methods opportunistically.