P. 1
Synthesis in Policy Impact Evaluation - Radej (Apr2010) - EE

Synthesis in Policy Impact Evaluation - Radej (Apr2010) - EE

|Views: 43|Likes:
Published by ksenos

More info:

Published by: ksenos on Sep 15, 2010
Copyright:Attribution Non-commercial


Read on Scribd mobile: iPhone, iPad and Android.
download as DOC, PDF, TXT or read online from Scribd
See more
See less





Synthesis in Policy Impact Evaluation

Abstract: Many of those involved in the impact evaluation of large-scale and multi-scope policy proposals have had difficulties in aggregating detailed assessments of individual impacts into a summative evaluative conclusion. There is a disagreement over assumptions about the aggregation which endorse different end results. The problem arises from the fact that different policy impacts are not commensurable (i.e. in money) neither across social scales (micromeso-macro) nor across social scopes (economic-social-natural) of evaluation. A new approach to synthesis is proposed that takes incommensurability of policy impacts into account. For this reason detailed impact observations are organised into an input-output matrix of evaluation scopes from where correlation between unintended (or secondary) impacts plays a central role in synthesis. The aggregation problem is studied by the comparative evaluation of the sustainability of development program for Pomurje region with three methods: micro (no aggregation), macro (full aggregation) and meso (partial aggregation) approach. Only the latter is found unbiased. Meso approach displays interesting characteristics, which are discussed in the conclusions. Keywords: Incommensurability, scope, scale, meso, matrix, impact evaluation. JEL Code: A13, A10, F12, A19, C50, H40

1. Introduction
To evaluate policy impacts is to make a value judgment of the policy proposal by collecting evidence and systematic assessment of its “worth or merit” (Scriven, 1994) to determine if evaluation standards or criteria have been met. Despite widespread impact evaluation of policy proposals, such as programs, public budgets or legislations, governments have been experiencing systemic failures in managing complex social issues (Bar-Yam, 2003). The difficulty is that with the increasing complexity of society, it is increasingly hard to provide for substantively diverse aspirations in public sphere. Inability of a government to assess its proposals’ indiscriminately results in unsuccessful application of substantively diverse aspirations in public sphere. Evaluation of social complexity involves judgment which is based on information that is obtained from many sources, through many means, as well as the use of multiple criteria. There are essentially diverse viewpoints which value a given policy document’s “worth and merit” because that they apply different scales of evaluation (from micro, meso, or macro perspective) and also make judgement from different scopes of evaluation, such as from environmental, economic or social aspect. In the face of a complex configuration of the social reality, qualitatively different issues are equally important (incommensurability in scope), but also one and the same issue can be perceived differently, either in local specifics or in its global entirety (incommensurability in scale). This heterogeneous corpus of evaluation information is rendered sensible through the process of synthesis (Encyclopedia of Evaluation, 2004). However, as the observed policy impacts are not commensurable between each other, they can not be reduced to a common denominator (Funtowicz, Ravetz, in Martinez-Alier et al., 1998) so their synthesis is not simply putting a puzzle together to create a whole. Social incommensurability (Munda, 2004) implies that different principles of legitimacy and social primacy must be reckoned with and reconciled (Wacquant, 1997) in evaluation – and so different »numeraires« or measures need to be applied in their appraisal. This methodologically complicates currently prevailing approaches to evaluation of worth and merit of policy proposals. Social incommensurability is accompanied with value comparisons in the public domain as well as with formal systems and knowledge structures. The latter decisively differ in the propositions they take as their axioms (Kuhn, 1970; Feyerabend, 1975). Kuhn asserts that sciences organize and integrate information in different ways. Different theories weigh the appearances of the same world differently and different types of models are best suited for representing the judgment processes of a given complex social issue. This induces disagreements concerning the

Policy-makers deal with numerous sectoral policies as well as with a great number of goals – all of which may hinder. Intersectional theory examines how various socially and culturally constructed categories of discrimination which are incommensurable. such as gender. In studying such issues simple aggregation ignores essential distinctions between incommensurable categories of research and evaluation. Hill Collins. Even though social incommensurability is a determining structural characteristic of complex social systems.” What one sees is always predefined in scope and in scale of observation and so these two need to be treated separately. 2004) but poorly implemented in policy impact evaluation – in particular in matrix based methodologies. Rotmans. Ekins. so that the system as a whole can be considered at differing resolutions as well as part by part. 1994. This leads to a particular relationship of observations of parts and wholes. value conflicts are ‘un-decidable’ (Mouffe. this concept is strange to contemporary evaluation research for the reason that we often think of two aspects of incommensurability – in scale and in scope – as coupled because of the most common ways we encounter them. Idea that public policy initiatives should be evaluated from the different aspects of social incommensurability has been widely recognized (Leopold. (…We) must allow a decoupling of scope and resolution. With the incommensurable theories. 2006. 2008) i. The lack of explicit justification of the aggregation procedure is the Achilles heel of the evaluation effort (Scriven. suggesting that when observing details of the system. 2002). 1994). intersectionality emphasizes that »the complexity of processes of individual identification and social inequality cannot be captured by such arithmetical frameworks«. Weaver. As we zoom in on the image we see a smaller part of the world at a progressively greater resolution. and as a result no neutral observation of social reality is possible. honest and disinterested scientists may arrive at different problem framing and conclusions because of systematic differences in the way they summarize available information (Mumpower et al. A useful example of the aggregation problem in social research is brought forward with the theory of intersectionality in sociology (Kimberlé. 1993) which is studying patterns of gender inequality. For this purpose. and class. support or reinforce each other (Wildavsky in Carlsson. 2002. 2003. 1971. while governments intervene into social issues that are heterogeneous in scope and in scale. contributing to systematic inequality that may be hidden on the surface. no objective basis exists for rational choice between opposing facts. Otter. 1994) in impacts evaluation of large-scale and multi-scope (LS-MS) policy proposals. no clear cut judgment about “merit and worth” of policy proposals. 2002). For instance. One of immediate consequences of this finding is the need to look at the foundations of the aggregation methodology (Scriven. Standard methods are designed for the appraisal of assumingly homogeneous policy interventions with only rudimentary distinction between scope and scale of evaluation (Elbers et al. When values are irreducibly plural. Bar-Yam gives an example (2004): “Consider observing a system through a camera that has a zoom lens. Rotmans. a cumulative evaluation of such complex situations is far from trivial (Veen. 2007. Medhurst. 2000 in Trainor. Munda. Intersectionality constitutes a critical alternative to additive claims (commensurability) when studying complex issue involving multiple threats. For a fixed aperture camera. race. When confronted with multidisciplinary issues even competent.. According to Prins (2006). scale can be considered as related to the focus of a camera—a blurry image is a larger scale image—whereas scope is related to the aperture size and choice of direction of observation. Rotmans. it dismisses the additive claim that black women are twice as badly off than white women due to both sexism and racism. 2 . the use of a zoom couples scope and resolution in the image it provides. Hence. there can be ‘no rational resolution of the conflict’.e..appropriate way to sum up what we know about reality. interact on multiple areas. 1996). the whole is not being observed. 2000).

. The methodological divergence between summation approaches. 2004. There is a disagreement over assumptions about the aggregation of impacts across multiple evaluation scopes and scales. Nevertheless.e. Uusikylä. Beside. Leopold’s assessment matrix is concerned with two scopes only – economic and nature – and assesses only the possible impact or side effects of the former on the latter (second chapter).2. 2006). Linked to this. and assessment of the contribution of EU structural funds to sustainability of regional development (GHK et al. In this way they obtain four aggregated or composite indicators of policy impact – one for each evaluation scope. Crisis in policy evaluation is in part seen as a result of unresolved aggregation problem (Foster. Ekins and Medhurst (2003) recently proposed a macro evaluation approach in the complete multi-scope perspective. human.2. but with the impact scope dimension expanded from two to four scopes (social. Older methodological approaches strictly decline aggregation of the detailed assessment results. Their work is an important step towards cumulative evaluation methodology in the framework of social incommensurability (second chapter).3. … impact). territorial impact assessment (ESPON – 3. 2007). Luna Leopold et al. social. natural. Impact Assessment Guidelines (SEC(2005)791). partially. this approach is named here as the Leopold-Ekins-Medhurst impact matrix or LEM). When evaluation results are presented in a too disaggregated or too aggregated form. Majority of standard approaches are found to avoid addressing incommensurable oppositions in the evaluation of programs. (1971) were the first to address the issue in this way. within each impact scope. such as EU’s strategic impact assessment (2001/42/EC). social. 2002). i. because policy impacts are not strongly but only partially or weakly commensurable. as well as the resulting disparity 3 . Hertin et al. economic. but not between them. when it allows for the summation of all policy measures’ impacts irrespective of their diverse policy scopes (economic. the territorial impact assessment (TIA. An analogous approach to LEM is now conventionally applied in various standard impact assessment procedures. They appropriately claim that majority of policy impacts are “normal” – they conform to the system thresholds. they in particular lack relevance for the strategic considerations. there is an apparent paradigm crisis in impact evaluation (Virtanen.. 2006). ESPON . 2006). Potts. and ex-ante assessment of the contribution of the EU structural funds to regional sustainability (GHK et al. This suggests reorganizing LEM into the square input-output matrix of Leontief (1970) with equal number of rows as policy scopes and columns as evaluation scopes (third chapter). It is shown in this paper that fragmented assessment results can be aggregated in LEM only by the given source and scope of impact. 2007).… policy measures) on given impact or evaluation scope (economic. They proposed a detailed expert-based assessment method at the micro level from which synthesis of results and the macro aspect remain entirely absent. Diverse policy impacts are thus comparable to a certain degree at least until they stay within critical system thresholds. So they aggregate policy impacts internally. they further allow for aggregation of assessed impacts for all evaluation criteria within each of four evaluation scopes that are placed in columns of the LEM. Poor handling of incommensurability may be well linked to really disappointing conclusions by the Impact Assessment Board (2009). The problem is that given policy’s impacts on different evaluation scopes are not qualitatively the same – such as an impact of economic policy on economy is not of the same quality as its impact on nature – so they are not homogenous (Rotmans. such as EU’s strategic impact assessment (2001/42/EC). LEM’s summation approach is inappropriate in its second step.. So Ekins and Medhurst propose a vertically and horizontally aggregated version of the Leopold matrix.Different aggregation procedures yield different evaluative conclusions. 2002). that up to 80% of impact assessments studies currently provided to the European Commission supply the kind of information that does not inform top-level policymakers whether their global objectives can be met.

how economic policy impacts economic assessment criteria – but evaluates side effects or secondary impacts of economic policy on environmental conditions. The accession of Slovenia to the EU also imposed a more restrictive border regime between Pomurje and Croatia (EU candidate country). policy might become one of the obstacles for regional sustainability.in their end results is illustrated by a comparative evaluation of a practical example. direction (positive/negative/neutral). because not enough emphasis was placed on genuine local needs. but not sufficient. assessment of policy impacts involves only two evaluation scopes (economic policy impacts on environmental assessment criteria) and only micro scale – assessing impacts of specific economic measures on specific environmental assessment criteria. Its economic capital is weak but improving since the mid-nineties. The development program for the Slovenian region Pomurje for 2007 – 2013 (RDPP. long-term unemployment. with 6. Regional development lags have accumulated despite increased inflow of resources earmarked for less advanced regions from the national or European budget in the past two decades. Social capital is very weak and further degrading.3% of national GDP). were traditionally close. They listed the 100 most important economic policy measures horizontally and 88 areas of environmental impact vertically. bordering Croatia (south). leading to continued depopulation. Pomurje is the least advanced Slovenian region (at NUTS 2. Analysis of development indicators (1995-2006) suggests that policy choice has failed to address critical regional trends and trade-offs appropriately. In this way. which previously. (1971). 2009). In their approach. This leads to the following working hypothesis: a large deal of inconsistency in public management in general and specifically in impact evaluation is not caused by the complexity of the public domain itself but by inappropriate dealing with this complexity. next with LEM’s and finally with input-output method. 2008) is first evaluated with Leopold’. Because of systemic failures to cope with regional complexity in scale and scope. Since the beginning of the market transition in the early nineties. Leopold et al. A new approach to evaluation synthesis is proposed that is based on meso explanation which distinguishes between weakly commensurable and weakly incommensurable policy impacts.800 cells – each further divided into four sections that describe every impact by its size (large/medium/small). The distinction between commensurable and incommensurable social events is crucial. the region had been surrounded with cold war borders. Radej.e. impacts are assessed in sufficient detail to enable maximally informed decisions of elected politicians about the impacts of assessed policy proposal. the region has found itself on the main European transport corridor which increased its geo-strategic importance and exposed it to international flows of people and goods. probability (high/low) and the amount of risk (critical or not). prolonged health and social risks for vulnerable groups (the majority of the population is officially classified as vulnerable). For half a century. This created a matrix with 8. This additionally burdened social capital. The paper concludes that evaluation of complex issues requires placing synthesis concerns into the centre of methodological efforts (see Lipsey.6% of the national territory and 4. 2. This is a region with a strong cultural and ecological identity – more than a third of its territory is protected nature areas including also unique landscape along the River Mura. 4 . aim to comprehend a large picture of the complex policy issue with detailed description of its elementary ingredients. This approach is very interesting because it goes beyond plain monitoring of policy performance – i. Standard approach to evaluation The first generation methodologies for matrical impact assessment of large-scale and multi-scope policy proposals started with the suggestion by geologist Luna Leopold et al. In the finest analytical manner. brain-drain. Hungary (east) and Austria (north). in Yugoslavia.

fail to satisfy information needs at the strategic level. Prevalence of positive impacts is not sufficient evidence for a conclusion that policy is adequate 5 . Experts assessed impacts against the scale with only three possible values: positive. 2007). leaving policy-makers with full responsibility for the evaluation synthesis and for drawing its policy implications. Without explanation of how opposing views of a complex phenomenon work together. Fragmented results that are left unrelated in their interpretation. and several others (COM(2002)276. the question is how the big picture of policy overall impacts can be obtained from fragmented assessment results. Leopold explicitly rejected the summation of multifarious impacts into a cumulative impact indicator. 1994). Table 1 had been obtained in a standard way. It is logically impossible to derive strategic policy advice directly from fragmented assessment results. The rejection of summation in evaluation and shifting this task to policy-makers is problematic. They claim that detailed assessment results should be presented disaggregated. this would not be appropriate.Recognizing the incommensurability between economic and environmental scope of evaluation. Today. 2004) and offer little value to decision-makers at strategic level. neutral or negative impact. the European Strategic Environmental Assessment Directive (2001/42/EC). Group of experts convened a workshop (via internet) and applied Delphi method to assess possible impacts of proposed 47 RDPP’s measures on a selected set of assessment criteria. the whole emerges as the outcome of interactions between parts. one by one. Cumulative policy impact evaluation is far more twisted. it is impossible to substantiate evaluation findings and this leaves evaluation utterly exposed to manipulation. These three lines of reasoning would suggest evaluator to focus his/her attention on negative impacts of the proposed program. However. Behind the estimation of the matrix there is a large deal of methodological difficulties which are simply skipped in this paper because they would focus our attention on the micro level. 2005). (ii) negative impacts focus evaluator’s attention on the weakest elements of the proposal which ought to be improved or abandoned. Table 1 Results in Table 1 can be summarised following three lines of reasoning: (i) prevalence of positive impacts suggest that a majority of the program measures will positively contribute to regional development. so impacts should not be studied in isolation. Fragmented evaluation results make decisions more informed but not necessarily easier (Diamond. In complex considerations. while the paper is concerned with multi-level evaluation. For Leopold refusal of aggregation is essential as it draws demarcation line between evaluator and policy-makers to protect the former from the value judgments and political interference (Kunseler. Uusikylä. So. territorial impact assessment (ESPON 3.2. This is illustrated on the case of impact assessment of Regional Development Program for Pomurje 2007-2013 with Leopold’s approach which is modified for our specific purpose so that it is extended to three evaluation scopes. Evaluation which simply produces nonoverlapping information tends to underplay inherent system contradictions. legitimizing disregard of stakeholders’ issues in policy-making (Stake. (iii) neutral impacts (0) are not problematic provided that they are not the result of unresolved disagreements among involved experts about direction of an impact. which supports endorsement of the proposal. SEC(2004)1377). 2001). Positive (and sometimes neutral) impacts should not be always accepted as unproblematic. nor negative impact as such can be treated as disagreeable. they produce banal answers to multi-dimensional societal problems (Virtanen. It is exactly politicians’ failure as social aggregators that call for policy evaluation in the first place – see Arrow’s impossibility theorem (1951). 2006). some major procedures follow similarly fragmented approach such as the EU’s Impact Assessment Guidelines (SEC(2005)791) which is even more detailed. This argument has been accepted as an evaluation standard. Refusing to summarize is “letting the client down at exactly the moment they need you most” (Scriven.

they may well be netted-out. forcing consensus for every single assessment detail is risky because it could invoke asymmetries within the assessment team – such as different negotiating power.in the proposed form but it only indicates that it is prepared by basically competent public authority. a positive policy effect on one assessment criteria would outweigh any other policy’s negative effect judged on any other criteria. So it is important to see that prevalence of positive impacts in Leopold matrix can only inform policy-makers about their effectiveness in atomistic perspective (micro view) while it does not enable. For instance in Table 1. The Leopoldian tradition requires that positive and negative impacts should be presented separately in the evaluation report but this even further fragmentises evaluation results. is simply netting-out the opposing assessments. It would suffice to invite experts to verify arguments for their disagreements and discuss them – not to reach consensus but at least to reach “rational disagreement” (Sankey. The first relates to situations when different experts cannot reach consensus on direction of impact (positive or negative). 1995) and confirm validity of arguments that stand behind them. disagreement between them is justified and irresolvable. 2006) go further suggesting that oppositions between experts’ assessments need to be discussed with the aim of reaching consensus among them about direction and even about intensity of impact. In fact they are carefully studied as well as painfully negotiated propositions among dissimilar group interests usually much earlier then they are submitted to impact evaluation. However. Imperative of consensus exhibits a strong tendency to revert to a kind of closed. Successful realization of separate policy measures can not by itself guarantee positive socialwide impact of the policy when its goals (or assessment criteria) are in conflict. Some other approaches such as CAF (Common Assessment Framework. a trade-off between greenhouse gases and money is not adequate as a general principle. The second difficulty is related to aggregating positive and negative impacts of a given policy measure on various assessment criteria. exclusive process (Connelly. If conflicting expert assessments are well founded. and so in the pursuit of emerging wider judgment. may negative impact of entrepreneurship promotion on employment in Pomurje be outweighed with positive impacts of entrepreneurship promotion on migration? Or a more radical example: is it permitted to outweigh additional tons of greenhouse pollution with additional purchase of tradable pollution permits? If impacts were commensurable. impacts are assessed against individual criteria and thus in isolation from each other. The imperative of aggregation of assessment results ignites two methodological concerns. It is only when systematic evidence of positive impacts is obtained. So another option which is also applied in case study. one needs to reason in the multi-scale perspective. Impacts are sometimes assessed against the criteria that are selected by formally responsible implementation agencies themselves so their achievements can not be automatically seen as »neutral« from a wider social aspect. There are some additional reasons why positive impacts shall not by themselves lead evaluators to switch green light to policy proposal. and more importantly. But “systematic evidence” can only be identified at higher levels of evaluation when detailed results of assessment are the first properly aggregated. Even when this is not the case. This is exactly a situation which is normal in public sector. So. 2004) where the dominant actor prevails. Policy proposals can not be treated as if they were outcome of bad fortune. does not jet allow for a systematic conclusion about the proposal’s impact on the overall society. They are linked to two limitations of commensurability of impacts and these limitations define conditions under which negative impacts may be tradable with certain positive impacts in aggregation of detailed assessment results. Richardson. evaluator can decide about the appropriateness of the overall proposal. But we know that greenhouse emissions cause irreversible changes in the atmospheric conditions and that the economic and climate aspects of welfare are incommensurable in scope. Different expert opinions are equally valid and only partial claims when observed from a wider regional perspective. where local judgments are different from the global one. Trade-offs 6 . However.

Verheem. System thresholds thus define a social space of normality where elements of the system freely interact so their acceptance can only be resolved locally as it depends largely on specific considerations among those directly concerned. Within the safety limits an agent either does not sense the difference between two qualitatively different social conditions or refuses to declare a preference for one or the other (Luce. 1987). A policy proposal is not allowed with any of its impact to cross system thresholds because this could endanger the basic integrity of the social system. The number of rows in LEM matrix may be as large as in the case of Leopold matrix. 2006) that comply with discontinuity in individual and social values. They have proposed a highly compacted form of the Leopold matrix. system thresholds – such as ecological and social standards – have emerged (for a survey of literature see Muradian. It presents RDPP’s impacts on much wider range of values (compared to Table 1) from the most robust positive composite impact (+++) to the most negative impact (---). The concept of system thresholds is closely linked to incommensurability. 1992. 2001/42/EC). the model is reduced in this paper to a “three capitals model” because it is even in its reduced form entirely sufficient to explain the aggregation problem in multi-scope perspective. Submission to system thresholds is obligatory for authors of policy proposals so it would be unusual in evaluation to deal with basically illegal situations with high probability of incidence of negative policy impacts involving critically high system risk. 2001). but it gives no instruction on how to cumulate environmental impacts and parallel them in aggregate way to economic ones. and to the conference on sustainable development in Rio de Janeiro (UNCED. When in dilemma how to round aggregate impact from Table 1 to Table 2 a decision was taken on comparison of the 7 . as suggested in Ekins and Medhurst. two values are incommensurable if there is no general way in which A and B trade-off in the whole range of situations of choice and comparison in which they figure. environmental and human impacts. This particular four scope logic can be traced back to the Brundtland report (WCED. Within the space of system normality assessed negative impacts of different policies are by definition not critical so they may be acceptable and therefore at least conditionally tradable and thus aggregatable. For simplicity. As Wiggins (1997) explains.between income and greenhouse emissions are not accompanied by incommensurability in every single case. SEA Directive. Thresholds simply account for the fact that there are discontinuities in the measurement of value-based phenomena and value addition (Mason. but it is also reduced. One of the first macro-evaluation methods was strategic environmental assessment (Sadler. But such examples are rare in social context. 1996. 2006). each covered by smaller number of evaluation criteria (only two in Table 1. 1992). Ekins and Medhurst have proposed a macro-assessment method that acknowledges incommensurability of development in scope. with all other five intermediate possibilities. in Table 2 these are already summed up). or at least people and their communities are not willing to treat them as such. To incorporate this peculiarity in public choice. social. Social phenomena are usually incommensurable ‘only’ beyond (or below or both) their threshold values. Munasinghe. 1956 in Munda. The missing link has been contributed by Ekins and Medhurst (2003) in their novel approach to evaluation of the EU structural funds’ impact on regional sustainable development. If this danger is averted with the introduction of system thresholds then evaluation may be quite simplified as it considerably alleviates the aggregation problem. 1992. Ekins. The columns are reduced from 88 fields of possible environmental impacts to four scopes of (regional) sustainability. as in the cases of minor environmental damage that stays within ‘safe’ ecological standards. They proposed the “four capitals model” which evaluates in parallel economic. from the level of 47 program measures (Table 1) to the level of (six) main regional policies – see Table 2. Recognition of system thresholds allows methodological step forward in the cumulative impact evaluation.

This example confirms again that judgment of worth and merit of policy proposals is far from trivial in complex circumstances. 2006) but fall under jurisdiction of other policies with different sectoral scopes (Rotmans. Policy impacts can be direct or indirect. Romer) and with older Samuelson’s neo-classical synthesis. The most problematic is the negative impact of value added growth on social welfare. LEM can not say much about RDPP’s impact on integrity of the regional development. evaluator also needs to judge obtained detailed assessment results in the perspective of conflicts and synergies between its scopes. Infrastructure development will be the most welfare enhancing policy. many studies demonstrate that given policy does not influence different areas of impact in the same way (Schnellenbach. Column aggregation assumes the homogeneity of the impacts of different policies on a given evaluation scope.financial weight of the related measures. usually unpredictably. This kind of reductionist over-simplification of multi-level structure of social complexes is entirely consistent with the thesis about “micro-foundation of macro”. which is conventional in the mainstream neoKeynesian economics (Mankiw. Table 2 Table 2 presents impacts of each sectoral policy involved in the RDPP on three scopes of evaluation. Ekins and Medhurst have overlooked that the assessed impacts are vertically not fully aggregatable despite their (eventual) conformity to system thresholds. impact on areas which are not primarily targeted (Rotmans. 2005). In this way evaluation concludes that the program will have a rather acceptable overall impact. one needs to conclude that they essentially remained followers of micro-based and reductionist evaluation philosophy put forward by Luna Leopold. In a complex setting. However. This intermediate solution goes beyond micro-macro dualism and even unites both aspects of evaluation. Complex approach to evaluation If detailed assessment results are not cumulated. The program will improve regional sustainability in all three respects even tough more in the economic aspect (++) than in social and nature ones (+). the main task of policy evaluation in macro perspective is not to examine how policy makers achieve their narrowly planned goals. the assessment produces findings that are too fragmented for evaluator of system-wide impact. it does not tell how the program affects the overall development from the aspect of its internal incommensurable oppositions. At the bottom Table 2 presents aggregated impacts of program on each of three evaluation scopes. In contrast. In this section we first discuss the reasons for inconsistent aggregation in LEM and than “negotiate a compromise” between micro and macro approaches. 2006). The impact of health policy and tourism are disappointing because their measures mostly relate to preparation of plans and regional organization structures. But the overall impacts of the program which is presented in aggregate at the bottom row of Table 2 do not appear excessively problematic. the differences between scopes are relatively small. causes findings that are too amassed for the judgment about policy’s inherent structural tensions. So policy impacts 8 . They are direct when they affect the target impact area. but indirect or unintended when they. such as in LEM. such as in Leopold's method. followed by rural development policy. Both micro and macro type of assessment waste essential information for evaluation of complex social issues. However. This is not possible in LEM because it constructs macro level view mechanistically with simple accumulation of elementary information from micro level where micro and macro are seen only as quantitatively (!) different perspectives. Summary impacts of the program are positive. 3. Even though Ekins and Medhurst have made an important summative step forward in macro evaluation methodology. full aggregation.

Some authors have argued against the strong incommensurability thesis (Griffin. we can only search for weak comparability as a facilitator of collective discourse. are said to be weakly incommensurable. Additional agricultural needs for industrial goods will indirectly induce also demands for services which are needed to enable increased industrial production. because it is suitable for an explicit presentation the tension between sectors. These are evaluated against two otherwise incompatible set of judgments so their nature is hybrid. how the sectors are directly linked. 2007) point out that in situations when there is an irreducible value conflict. When weakly commensurable impacts are partially aggregated within sections. we can show in a matrix. In principle. industry and services as three sectors in an economy. (1998. Leontief initially developed his matrix to facilitate inter-sectoral studies. (iii) and between different sorts of secondary impacts – due to their weak incommensurability. interest rates in the monetary policy are mechanism which is directly more beneficial for the profit sector than for the non-profit one (Kennedy. 1972) and tax policy (Leith. This rule seems at odds with the strong version of incommensurability thesis. 2006). (ii) between primary and secondary impacts – due to weak commensurability of the latter. the matrix exists at a lower level 9 . cumulative impacts are obtained for each subsection. Absence of secondary impacts in policy considerations might help us understand why good individual policies. Impact is said to be weakly commensurable when specific limitations are imposed in summation procedure such as with the partial aggregation rule. This means that evaluator needs to distinguish between: (i) diverse primary impacts – due to their incommensurability. if we have agriculture. This rule will be referred to henceforth as a partial aggregation rule. At the same time. based on strong values and even common sense. Thadden. For example. Impacts that are weakly commensurable in two or more incommensurable scopes of the evaluation. as a partial aggregate. Square matrix exists hierarchically above the micro-level (Leopold matrix) because it is aggregated from it. in Burchell et al. Vertical summation in LEM is only permitted partially within the given source of impact (and area – as seen earlier). often lead to disappointing overall system results (Chapman. by rows and columns. 2006). 1995). in Stagl. 2007. The agriculture requires some capital goods and chemicals from the industry. 2000).. Taking into account partial aggregation rule all policy measures in the Leopold matrix (rows in Table 1) need to be first regrouped in the same way as impact areas (columns) – by their three scopes. 1986 in Morgan. direct and indirect. policy interventions should be always addressed in terms of their inadequacy due to their sectoral specialization as it can be exposed against the general interest they are supposed to serve (Donzelot. For instance.are not neutral in scope! This is confirmed even for those policies that had previously been taken as the most homogenous in impact such as monetary (Lucas. Sankey. Nola. like socioeconomic impacts which take place when a given policy measure simultaneously produce weakly commensurable impacts in economic as well as in social scope (“secondary” impact). Sectoral specialisation of public policies implies that distinction between incommensurable impact scopes must be preserved in evaluation (Ostmann. These can be presented in a square “input – output” or a Leontief's matrix (its “central quadrant”). This divides the Leopold matrix into nine sub-sections. 1991). which introduces a delicate possibility of translation between them. economic and social policy’s impacts on the environment are not commensurable so they are aggregated separately (economic impacts on nature separately from social impacts on nature). But Martinez-Alier et al. 2004). and at the same time it supplies its product output to the industrial and service sectors as their intermediate input. For instance. Yet unintended or secondary impacts routinely fade out in policy impact evaluation because it is assumed that they are too complex and thus impossible to track. They proposed to make a distinction between relations of strong and weak in/commensurability.

N∩E. S∩S. A correlation is applied when one studies connectedness (“overlaps”) between pairs of variables in causal models. To understand how qualitatively different perspective emerges in the meso-matrical view so to say “out of nothing”. etc) are located on the diagonal (top left to bottom right). in rows) and its impact or evaluation scopes (outputs. Meso is “intersectional” as it possesses hybrid characteristics (Schenk. meso is a perspective from where the modelling of complexity is the most tractable “a priori” (Easterling. The diagonal 10 . 2004) as it intersects two axes in definition of social complexity – horizontal or scope aspect (E. 2002). It not only crosses complex system divides but also intermediates between them (Knauft. intersectional and non-excludive. Square matrix is “a plane of inter-paradigmatic standards” (evaluation scopes. 2008). 1999). Intersection between input and output scopes is denoted with the intersection sign ‘∩’ from the set theory. 2002).their system oppositions and synergies. one need first to correlate ingredients of Table 3. In our case “to correlate” means to link one directional overlaps between scopes. Table 3 Table 3 is irrelevant and entirely useless for everyone who is only narrowly concerned with the assessment of performance of his/her own actions and is not able or willing to question its system-wide impact. It enables mid-level articulation of constitutive oppositions that accompany public choice and it facilitates the understanding of the basis of deep system oppositions (Mertens. As plural. while the impact of social policy on the economy is denoted as S∩E. 2007) of social complex. which means that it covers many parallel views of one closed reality containing many (pre)existing substantial contexts (scopes).than the macro (LEM). It displays “plural-relativistic” view (Geertz. Table 3 thus produces a qualitatively different policy perspective which is emergent. This approach to evaluation is therefore meso-matrical. Input-output matrix of RDPP’s impacts is presented in Table 3. 1973) which “equally tries to tread the path between difference and sameness” (Allmendinger. Table 2 does not produce qualitatively different view compared to Table 1 since they both only present effectiveness of each policy (measure) assessed against some uniform set of performance criteria. economic policy impact on the social scope of evaluation is denoted as E∩S (E intersects S). 2004) of the social complexity. micro and macro). which means that its message can not be reduced back to the fragmented pieces of performance assessment from which it is acquired. With its plural characteristics meso is situated in “the un-excluded middle” (Wallerstein. with their diagonally symmetric opposites. Evaluator would. It shows policy proposal in the perspective of overlapping between its policy scopes (inputs. The square matrix consists of two distinctive classes of relationship: (i) intra-scope or primary impacts (E∩E. The input-output matrix thus presents an intermediate or meso view of the assessed program impacts. in Renselle.. for instance. in columns). Kordig. For example. intervene from meso perspective in conflicts to help actors with different belief systems understand where their disagreements have epistemological and ethical roots and help expose the meaning systems by which these facts are being interpreted (Bovens et al. Kok. and N in input-output matrix) with vertical or scale aspect (meso. 2006) and integrates them. S. in the aim to obtain a bi-directional or reciprocal impact (NE as nature-economic overlap) that explains how two evaluation scopes work together. 2006) of its intersecting dimensions (scopes and scales). Mesomatrix is of central importance because the complex social system is built upon meso (Dopfer et al. such as E∩N. Table 3 is therefore relevant only for the highest ranking policy-makers who are concerned with overall policy consistency taking into account its wanted and unwanted impacts indiscriminately. (ii) and interscope or secondary overlaps (E∩N etc) are located below or above this diagonal. A meso-matrix refocuses evaluator’s attention from the achievements of sectoral policies on impact-driven changes in the relations among evaluation scopes .

elements provide narrow evaluation of primary effectiveness – how one particular policy scope impacts “its own” evaluation scope. similarly can be said about the feeble secondary impacts of N. This confirms the working hypothesis that a large deal of difficulties arising from inconsistent impact evaluation is not caused by the complexity of the public issues but at least sometimes by an inadequate approach to social complexity. Table 4 In Table 4 evaluative conclusion on program’s “merit and worth” is not derived solely from the aggregation of impacts but also from the co-relation between sub-aggregates of weakly commensurable impacts. S is involved in interactions that are more beneficial for E and partly for N. which is highly problematic taking into account the baseline conditions with S already deeply depressed in comparison with E and N. for the ‘socio-economic’ intersection of impacts. denoting bi-directional overlap between S∩E and E∩S. Regional economic policy would be very successful in achieving its own primary goals with the implementation of RDPP (three pluses). The program exhibits anti-social character. Broader insight into RDPP’s internal consistency is obtained “under the surface” of detailed assessment results with the correlation of non-diagonally located and weakly incommensurable sub-aggregates. Diagonal elements are strongly incommensurable between each other. but can be a consequence of inadequate strategic evaluation of options in the presence of deep value oppositions between policy stakeholders resulting from social complexity in scale and scope. So they can not be aggregated any further and are interpreted as they are. environmental and economic scopes of regional development are not treated indiscriminately in RDPP. Correlation produces three correlates in Table 4: SE. Information about anti-social character of RDPP is entirely absent from the surface view (Table 1 and 2). and inappropriate assumptions lead to misleading policy advice. Beside. It emerges solely from the summative endeavour of the evaluator. not due to different detailed expert multi-criteria assessment at the micro level. Nature conservation would be only faintly effective (one plus) either because it is not foreseeing clearly its main challenges or it pursues its primary aims inconsistently. the economic scope very poorly integrates with S and N. This information can not be extracted from Tables 1 and 2. 2 and 4 are. Conclusions 11 . NE and SN. expensive and time consuming tools for a strategic deception of decision-makers. social policy would be moderately successful (two pluses). as the socio-natural correlate. 4. of course. which remains the same in all three cases. Comparison of diagonally placed impacts indicates that social. Here we recall the previous observation that nature protection policy will not be very effective in pursuing its primary goals – so the proposed program will impose a social burden at least in relative terms for only scanty aspirations in environmental sustainability. This shows that a narrow minded economic and nature protection policy both take further advantage of already very fragile regional social capital. Paper shows that different aggregation assumptions bring about different evaluative conclusions. Policy failure to implement more cohesive policies is therefore not necessarily a result of intentional bias. This observation does not match with the previously obtained conclusion from summary row of Table 2 that suggested a broadly balanced impact of the program on the three evaluation scopes. with flawed assumptions about the nature of (in)comparability of social issues. Disagreements in conclusions derived from Tables 1. The negative SE correlate is a result of a socially narrow-minded economic growth policy. Table 4 shows that in RDPP. The conclusion is that the RDPP is not contributing to regional sustainability. In this way standard matrical evaluation methodologies serve only as sophisticated. Low correlation in SN results from a socially narrow-minded nature conservation policy. This conclusion is just the opposite to the one obtained from Table 2.

policy-makers should be increasingly aware not only of their own agency’s primary objectives narrowly defined. Ekins and Medhurst have accurately observed that aggregation of impacts within a given evaluation scope is formally correct. but also of wider implications and unwanted effects of their (in)activity. As societies grow more complex. Meso approach to evaluative synthesis is not simply a cognitive process of drawing general lessons from local projects (Geels. gender inequality. Paper sheds light on the cases when neutrality should be earned also in the synthesis of evaluation results. Refusal of aggregation is therefore inappropriate. Paper demonstrates that social incommensurability is not an irresolvable obstacle to more cohesive social research. and so there is no mechanism to install an optimal public policy. but places synthesis concerns. via neutrality considerations. which are weakly commensurable. majority of links important for the reproduction of everyday social life are weak. 2004). Today majority of evaluation studies try to earn their neutrality in the first evaluation step of scientifically objective assessment of individual impacts. or unsustainability of development. 2007). Leopold has accurately differentiated environment and economy as two incommensurable scopes of valuation. in Schnellenbach. This experiment not only confirms the need for the cumulative evaluation of complex policy proposals. but he failed to see that the assessment is actually concerned with secondary impacts. who take the view that the unintended consequences of action are the principal concern of social science and that the existence of unintended consequences is a precondition for the very possibility of the scientific understanding of a complex society (Vernon. 1976). into the centre of the efforts pertaining to improvement of policy advice by policy impact evaluation.. They provided a logically justified procedure in cumulative evaluation and offered firmer ground for making system-wide evaluative conclusions. 2007).This paper has explored how standard approaches in matrical impact evaluation comprehend social complexity in scale and in scope and how they take it into account in formation of evaluative conclusions about “worth and merit” of the policy proposal. 2005). This is a crucial precondition if we wish synthesis to become a tool for identifying. So the yardstick by which policy proposals are compared and selected must be their secondary impacts in weakly related welfare concerns. This finding justifies our research concern. But they failed to apply incommensurability consistently on the input side of policy scopes. Sectoral policies are biased but also equally important. The meso approach is proposed as an alternative against both relativization of evaluation results (as in the Leopold) as well as against their generalization beyond limited validity (as in the LEM). but only “a safety mechanism” which reminds researcher that social issues are complex in scale and in scope. avoiding or tackling systemic contradictions and systemic exclusions such as due to injustice (Pulido. Theodórsdóttir. Acknowledgements: I acknowledge valuable comments given to the previous version of the 12 . in Bondas. Even though contemporary societies are built on incommensurable oppositions which cause strong social fractures and invoke strong relationships of dis/agreement. in Hilding-Rydevik. a policy proposal that is the most secondary effective ought to be chosen (compare with Demsetz. In such a situation. The same principle is relevant to the thought of both Hayek and Popper. 2003. The case illustrated that the policy may be quite differently evaluated from the aspect of primary and secondary effectiveness. so they can be partially aggregated. At the end they remain “micro-foundationalists” when they construct macro perspective through simple aggregation of micro level observations instead of from (correlation of) meso level sub-aggregates. But strong differentiations are important for a very small number of social issues. Synthesis requires a careful framing of the research problem which calls for a formalism that theoretically remains faithful to the framework of study (Barroso et al.

The Foucault Effect: Studies in governmentality. [XII/06]. pp.do? uri=CELEX:32001L0042:EN:NOT. The Origins of Meso Economics Schumpeter's Legacy. Burchell G. M. [IV/09]. 1991.. Foster.3..jsessionid=18gjekrjr n3oh. ’t Hart. 2007. Potts. (2nd ed. Arrow K. ESPON .eu/en/campaigns/go/&tid=20. E. GHK.W. Multiscale Variety in Complex Systems. 124 pp. Feelings of Discontent and the Promise of Middle Range Theory for STS Examples from Technology Dynamics. Technical Report.int/comm/regional_policy/sources/docgener/evaluation/rado_en. 1975. p. London.com/abstract=1016945..eipa. The Mobilisation of Society. Qualitative Health Research. PSI. Max-Neef. London.. Diamond J. Social Choice and Individual Values. in Burchell et al.pdf. [VII/08]. J. Micro–meso–macro. Kuipers. Instituto de Estudios Fiscales. Presented at the 5th European Conference on Evaluation of Structural Funds. 48 pp. SEA Directive 2001/42/EC on the assessment of the effects of certain plans and programmes on the environment.com/sol3/Delivery. both from the University of Ljubljana. 317-33. 343. Bondas T. 2008. Evaluating the Contribution of the European Structural Funds to Sustainable Development. 1951.ingentaconnect. June 2627. Ekins P. Bibliography Allmendinger P.paper by professors Mojca Golobič and Srečo Dragoš. Working Paper 40(2005):159-83. http://europa. P. [X/06]. Sage.pdf. Assessing Budget Support with Statistical Impact Evaluation: a Methodological Proposal.). 2-97.ssrn.: NECSI . http://ssrn. http://www.de/evo/discussionpapers/2006-10. 1963). New Haven: Yale University Press. Volume 1-2. The Strength of Weak Ties. Chapter 15. [VI/09].mpg. Final Report. Evaluation. Discussion Paper 2007-075/2. J. Planning Theory. Geels F. 2006. 2002. 2002. 2003. System failure . [X/06]. A Four-Capital Model of Wealth Creation.. Sage. Journal of Environmental Planning and Management 47/1(January 2004):3-17. Mass.C. http://ec.europa. 9/6 (2004):15-24.com/content/routledg/nias/2002/00000003/F0020002/art00012. 303 pp. 2006.cfm/SSRN_ID721599_code114131. Bar-Yam Y. Establishing a Performance Management Framework for Government. T. 169-79. http://eur-lex. Science.espon. http://www. Potts J. Impact Assessment Guidelines. Sage. Vol. 2006..pdf [III/05] Granovetter M. Amsterdam: Tinbergen Institute.2. p. Evolutionary Economics Group.. http://www. 17/1(2007):113-21. Kok. Journal of Evolutionary Economy 14/3(2004):263–79. Hall. eds.Why governments must learn to think differently. Bar-Yam Y. School of Economics. Miller (eds). Second edition. Emergent Properties of Scale in Global Environmental Modelling – Are There Any? Integrated Assessment 3/2–3(2002):233–46. Spatial Scenarios and Orientations in Relation to the ESDP and Cohesion Policy Territorial Impact Assessment. 44 pp https://papers. London: Demos. EC. Feyerabend P. Foster J. IEEP.org/projects/yaneer/multiscalevariety.eu/LexUriServ/LexUriServ. http://ec. Dopfer K.co.pdf [IV/07]. A micro-meso-macro perspective on the methodology of evolutionary economics: integrating history. de Hoop. Jena: Max Planck Institute of Economics. Carlsson L. Against method : outline of an anarchistic theory of knowledge.europa.. Easterling W. Brussels: EIPA – European Institute of Public Administration.. 5. 2006. http://necsi. http://www. 147-55.eu. 29 pp.demos. Gordon. CE.O. The Oxford Handbook of Public Policy. American Journal of Sociology 78(1973):1360-80. & Human Values. Ekins P. New York: Routledge. The Contribution of the Structural Funds to Sustainable Development: A Synthesis Report to DG Regio. Towards a Post-Positivist Typology of Planning Theory. 610. Springer. Presupuesto y Gasto Público. Medhurst J. 24 pp.europa. 2004. Challenges in Approaching Metasynthesis Research.New England Complex Systems Institute. Gunning. Exclusion: The necessary difference between ideal and practical consensus. 1992. simulation and econometrics. J. The University of Queensland.eu/governance/impact/docs/key_docs/sec_2005_0791_en. Hertfordshire: Harvester Wheatsheaf.. [XI/04] Elbers C. 6/2(2000):201–16 Chapman J. p. in Real-Life Economics: Understanding Wealth Creation (in Ekins P. National Evaluators. SEC(2005)791. 17 pp. P..pdf?abstractid=721599&mirid=1. 2004. K. Richardson. 13 . Brussels. C.eu/regional_policy/sources/docgener/evaluation/doc/sustainable_annexes.W. http://papers. 2003. 51 pp. Discussion Paper No. Common Assessment Framework. Wiley. Technology. 1991. Donzelot J. Complexity. The Papers on Economics and Evolution No. 32/6(November 2007):627-51.. S. 2001.E.alice. Connelly S..econ. Non-Hierarchical Evaluation of Policy. Budapest.uk. Dopfer K. [IX/07]. New York : Humanities Press. Cambridge.htm. The politics of policy evaluation. Bovens M. 1/1(2002):77–99. [VI/07]. European Commission. Sage.eu/mmp/online/website/content/projects/260/716/file_2786/fr-3. [VII/08]. European Commission.2_April2007-full. A Mathematical Theory of Strong Emergence Using Multiscale Variety.pdf.. K..

. Narrative Accounts of Origins: A Blind Spot in the Intersectional Approach? European Journal of Women’s Studies 13/3(2006):277-90. EU/FP6 Project MATISSE Working Paper 6.. The Structure of Scientific Revolutions. Russel. Preprints of the Max Planck Institute for Research on Collective Goods 2007/11b. Review of Economics and Statistics. L. The Public Nature of Private Violence. 52/3(1970):262-71. The aggregate and the representation of its parts. C. Stockholm: Nordic council. Hedges.. 25 – 26 August 2003. 6/4(2006):407–30.pdf.nordregio.. Leopold L. Munasinghe M.M. Munda G. 14 . http://www. 2009. 52 pp. Environmental Economics and Sustainable Development... Hanshaw.com/abstract_id=908620. 158/3(2004):662-77. Lucas R.. NUSAP. pp. Hill Collins. 13 pp.jstor. entry in Stanford Encyclopedia of Philosophy (in Zalta N.stanford. 1994. G. Mapping the Margins: Intersectionality. D. Munda. Martinez-Alier J. [VII/08]. [VII/08]. Thinking and Reasoning. NY: Russell Sage Foundation. Okemos: Seva International. [VII/08]. Paper presented at the UN Earth Summit. Cooper. Anthropological Theory. Ostmann A. The Strength of Weak Ties: A Network Theory Revisited. 2006. Sage.S. Theodórsdóttir. Rio de Janeiro. Mumpower J. Mykitiuk (eds. 147-58. [II/08]. and Gender as Categories of Analysis and Connection. 1971. 2007. Weak comparability of values as a foundation for ecological economics. and J.nusap. 2000. Valentine (Eds). http://plato. Nola R. Kimberlé C. B. 2/2-3(1996):191-211. Monetary and fiscal policy interactions in a new Keynesian model with capital accumulation and non-Ricardian consumers. http://philosophy. 4/2(1972):103-24.: The MIT Press. 2007.net/.B.si. http://ssrn. 93-118. M.. H. O’Neill. Inderscience. Prins B.edu/about_us/meet_us/carl_mitcham/courses_taught/5110/classic_sts/structure_of _scientific_revolutions. 38 pp. 17 pp. Muradian R. 324 pp. Lipsey M. Chapter 8. http://www. 2004. Nilsson. Identifying interesting variables and analysis opportunities. Discussion: Observational Invariance. in Fineman. 1/1(2007):48-76. http://sciencepolicy. Expectations and the neutrality of money. 1970. 210 pp. Social multi-criteria evaluation: Methodological foundations and operational consequences.pdf. Blackwell. In H.matisse-project. Elsevier. 2009.edu/forster. [IX/07].H.). Value Pluralism. Toward a New Vision: Race. Morgan D. Towards a new paradigm of science in scientific policy advising.europa. Philosophy of Science. 38/1(2001):7–24. Third strategic review of Better Regulation in the European Union. Sankey. Mason E. 1/1(1993):25–45.E. Taylor and Francis. [VI/07]. Proceedings from the 5 th NEA Conference.colorado. Knauft B. Á. http://ec.pdf.umich. Identity Politics. J. Leontief W. Washington: World Bank. 42 pp. Stanford University. European Journal of Operational Research. M. W.L.edu/entries/valuepluralism/. B. Sex & Class. Environment Paper No. Journal of Mixed Methods Research. Stanford: Centre for the Study of Language and Information. 2006. Reykjavik.htm. New York: Routledge. Impact Assessment Board report for 2008. American Journal of Evaluation. [IV/07]. Paradigms Lost and Pragmatism Regained: Methodological Implications of Combining Qualitative and Quantitative Methods.com/storage/f113210112864597.net. http://www. Leith C. Elsevier.eu/governance/impact/key_en.Granovetter M. 3.pdf.A. Bonn: Max Planck Institute for Research on Collective Goods.pdf. [II/07].org/pss/186288. Washington: Geological Survey Circular 645. Stewart. L. Interest and inflation free money: creating an exchange medium that works for everybody. & J. ed. [IX/09].wisc.inderscience. After Popper. 2006. Hertin J.M. V.. Inclusive Evaluation: Implications of Transformative Theory for Evaluation. Nykvist. Environmental Repercussion and the Economic Structure: An Input-Output Approach. Chicago: The University of Chicago Press.R. Anthropology in the middle. [VII/08]. http://www. Ecological Economics. 20/1(1999):1-14.net/downloads/KunselerEssay2007.pdf. [II/09]. T.edu/people/lunaleopold/(118)%20A %20Procedure%20for%20Evaluating%20Environmental%20Impact. Planning for Sustainable Development – the practice and potential of Environmental Assessment. Cambridge Mass.). Clarke. Sage.. F.. A.se/Files/r0402. Kuhn & Feyerabend: Issues in Theories of Scientific Method. http://ssrn.B. eds. Munda G. Race. IAB.E. The Handbook of Research Synthesis and Meta-Analysis (2nd edition). International Journal of Environmental Technology and Management. and Violence against Women of Color. 2006.R. R. Australasian Studies in History and Philosophy of Science. J. Nordregio Report 2/2004. Turnpenny. Working Paper Series No 649. Kunseler E.E.. Environment and Health Department. Kordig C. 5 pp. Expert Judgement and Expert Disagreement. Ecological thresholds: a survey. 40(1973):558-69. Sage. [VII/07]. Class. Jordan. Ecological Economics 26(1998):277-86. Kluwer.W.. http://rfrost. Jr. Balsley. SEC(2009)55.berkeley. von Thadden.L. pp. Mertens D.people.. http://eps. http://www.com/abstract=1024681. The practice of policy assessment in Europe: An institutional and political analysis. Kuhn T.edu//courses/SI110/readings/In_Out_and_Beyond/Granovetter. A NAIADE based approach for sustainability benchmarking. Kennedy M. Hilding-Rydevik T. A procedure for evaluating environmental impact. P. 6/1-2(2006):65–78. R. 1995. Sociological Theory 1(1983):201-33. 1992. Journal of Economic Theory.

html [IV/10]. 2010. incommensurable values and agonistic politics. 1971. Oxford and New York: Oxford University Press. The Final Synthesis.pdf. Our Common Future. 188 pp. A Problematic Heading.sageereference. Scaling in Integrated Assessment: Problem or Challenge? Swets & Zeitlinger. Chapter 6. 2007.Pulido L. http://www. Epistemologia. Scales in Space.jstor.pdf. Cambridge.. Reading Bourdieu's «Capital». Canadian Journal of Political Science/Revue canadienne de science politique. Hague: Ministry of Housing. Swets&Zeitlinger. Tae-Hee J. Incommensurability. http://www. [IX/06]. 1997. Encyclopedia of Evaluation. Sankey H. 15 . Sage.org/esa/sustdev/documents/agenda21/index. Philadelphia: Temple University Press. Wallerstein I. 271 pp. 2000.sd-research. Otter. Scriven M.uk/wp-content/uploads/2007/04/sdrnemsvareviewfinal. 2006. http://www.edu/entries/valueincommensurable. [I/07]. Sage. Rotmans.html. SAGE Publications. Perspectivism. [III/09]. WCED. 22/3(2001):349–354. Integrated Sustainability Assessment: What? Why? How? EU/FP6 Project MATISSE.. Wiggins D.homme-moderne. The Problem of Rational Theory-Choice. S.j. Incomparability. The Integrated Assessment Journal. Sadler B. 34/8(2008): 905–24. Buffalo State College. Veen A. Weaver P.: Harvard University Press. American Journal of Evaluation. Cambridge: Polity Press.net/projectcomm/uploads/tx_article/Working_Paper_1_03. http://plato. 2000. Working Paper 1 (October). Radej B. [III/09] Virtanen P. 2004. 2007.unimelb. challenges and future directions.pdf [V/09] Trainor B. 1997. [VII/07] Renselle D.au/staff/Sankey/howard/howardpaper4. http://www. Politics as the quest for unity. P. Schenk N. Rotmans J. p.sdeval. 25 pp. http://www. 2008. Modelling energy systems: a methodological exploration of integrated resource management.html. [I/07]. The Micro-Macro Synthesis in Veblen’s Institutional Economics.philosophy.rug. Groningen: University of Groningen..si/Publikacije-za-komisijo-za-vrednotenje/Meso-MatricalSynthesis-of-the-Incommensurable. Working paper 3/2008. 15/3(1994):367-82. http://www.).quantonics. Philosophy & Social Criticism. and Practical Reason). Exploring the Missing Links between Cause and Effect A Conceptual Framework for Understanding Micro–Macro Conversions in Programme. The Dahrendorf hypothesis and its implications for (the theory of) economic policy-making.html. Vernon R. R.nl/FILES/faculties/science/2006/n. 3 Apr.matisseproject. http://www. Van Der. (in Chang R. 18(1995):299-312.htm. Agenda 21. J. [III/07] Rotmans J. Sage. Uusikylä. Annals of the American Association of Geographers 90(1):12-40. 1996. 1987. 2006. Review of Clifford Geertz' Available Light (Princeton University Press. Synthesis. Rethinking environmental racism: white privilege and urban development in southern California. 3/2–3(2002):266–79. Spatial Planning. PhD Dissertation.stanford.com/evaluation/Article_n536. Tools for Integrated Sustainability Assessment: A two-track approach. [IX/06]. Stagl S.M. 3/2–3(2002):160–6. [VII/08]. http://dissertations. Integrated Assessment. 224 pp.. http://historyofeconomics. United Nation Conference on Environment and Development. (ed. Cambridge Journal of Economics. World Commission on Environment and Development – WCED. Ljubljana: Slovenian Evaluation Society. 400 pp. Chicago: Encyclopaedia Britannica Inc. 2008. 22 pp.E. 2004. 97-115. Vancouver: University of British Columbia.un-documents. Carmel: Quantonics. Strategic environmental assessment: Status. Incommensurability: Four Proposals. Schnellenbach J.). 29/6(2005):997-1009.org/Conference08/papers/jo.J. 10/1(2004):77–91. Evaluation. http://www. UNCED. 6/4(2006):35–57. [VII/08].schenk/06_c6. The »Great Society« and the »Open Society«: Liberalism in Hayek and Popper. H.pdf. Mass. Integrated Assessment. 9/2(1976):261-76. The Uncertainties of Knowledge. 66 pp. [VI/07]. the Environment.net/wced-ocf. http://www.org. A report to the Sustainable Development Research Network.com/Review_of_Clifford_Geertz_Available_Light. American Journal of Evaluation. Wacquant L.un. 1992. http://admin. Stake R.org/societe/socio/wacquant/capital. Final Report (January). Verheem.PDF.SDRN Rapid Research and Evidence Review.org/stable/3230923.edu. Foreword to the English-language translation of The State Nobility. Webster’s Third New International Dictionary. Emerging Methods for Sustainability Valuation and Appraisal .htm.T. Meso-Matrical Synthesis of the Incommensurable.ub.

You're Reading a Free Preview

/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->