Integrating Risk Analysis Economic Analysis of Projects

in the

Handbook for

May 2002

© Asian Development Bank 2002 All rights reserved

ISBN 971-561-458-2 Publication Stock No. 060202 Published by the Asian Development Bank P Box 789, 0980 Manila, Philippines .O.

Foreword

T

his Handbook is an output of a study undertaken by the Economics and Research Department (ERD) of the Asian Development Bank (ADB). Nigel C. Rayner directed the conduct of the study, assisted by Anneli Lagman-Martin. Overall supervision of the initial stages was provided by David Edwards, then Assistant Chief Economist, while Xianbin Yao, Assistant Chief Economist, supervised the study through to its conclusion. Keith Ward (Staff Consultant) provided extensive technical inputs and advice throughout the study. Support for word processing requirements was provided by Ma. Nieva Baguisa. Preparation of the Handbook benefited from a participatory approach through consultation meetings and individual interviews. At various stages of the study, ADB staff were given the opportunity to review and comment on the draft reports. The study benefited from substantive comments from Stephen Curry, Manabu Fujimura, and Bo Lin. The Handbook was further enhanced with the inclusion of case studies, based on work undertaken by Manabu Fujimura, Vincent de Wit, Bo Lin, William Loxley, and Keith Ward. During an in-house seminar on project economic analysis, the results of the study were disseminated and the use of risk analysis software was demonstrated. This Handbook further develops the discussion of the principles on risk analysis that is contained in Chapter XI (Uncertainty: Sensitivity and Risk Analysis) of ADB's Guidelines for the Economic Analysis of Projects (1997). The Handbook contains detailed discussions of current risk analysis practice, the theory and application of various risk analysis techniques, and recommended applications for risk analysis, including a summary of typical risk circumstances on a sector-by-sector basis. To illustrate the ease of use of quantitative risk analysis software, the Handbook includes case studies based on actual ADB projects in the agriculture, education, health, and power sectors. A summary version of this Handbook is available as ERD Technical Note No. 2 Integrating Risk into ADB's Economic Analysis of Projects. The Handbook provides guidance with respect to the integration of risk analysis in the design and analysis of projects. It also provides direction on how the analysis of risk can be used to improve the focus on poverty reduction. While the Handbook is primarily targeted to ADB staff and officials of its developing member countries, it may also be of interest to others involved in development assistance.

assets quality. management quality.iv HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Abbreviations and Acronyms ADB ADF CAPM CAMEL CDF DFID EIRR ENPV EOCC FI FIRR FNPV NPV PBL PFI PIA PIR PPTA RRP SAR SERF SI SV UK VaR VOC WTP Asian Development Bank French Development Agency capital asset pricing model capital adequacy. earnings and liquidity cumulative distribution function Department for International Development (UK) economic internal rate of return economic net present value economic opportunity cost of capital financial institutions financial internal rate of return financial net present value net present value policy-based lending Private Finance Initiative (UK) policy impact analysis (matrix) poverty impact ratio project preparatory technical assistance report and recommendation of the President staff appraisal report (World Bank) shadow exchange rate factor sensitivity indicator switching value United Kingdom value at risk vehicle operating cost willingness-to-pay .

Contents v Contents Page Foreword Abbreviations and Acronyms Chapter 1: Introduction and Background Introduction Background Why Undertake Risk Analysis? Chapter 2: Technical Approaches to the Analysis of Risk in Project Economics Introduction Risk and Uncertainty: A Definition of Terms Allowing For Uncertainty Modeling Risk Quantitatively Technical Issues in Modeling Risk: Two Considerations Risk Analysis. and the 'Harberger' Critique of Recent Applied Project Economic Analysis Summary of Risk Analysis Experience Chapter 4: Poverty Reduction Objectives and Risk Analysis Introduction Planning for the Poor. Vulnerability. and Risk Aversion Poverty Reduction Objectives: Implications for Quantitative Risk Analysis iii iv 1 1 3 7 9 9 10 11 15 19 23 27 29 29 30 36 38 43 44 47 47 48 52 . Decision-Making. and Welfare Summary of Approaches to Risk Analysis Chapter 3: A Summary of Development Agency Experience in Applying Risk Analysis Introduction ADB Experience With Risk Analysis World Bank Experience With Risk Analysis Other Agency Experience With Risk Analysis Risk.

vi HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Poverty Reduction Objectives: Implications for Qualitative Risk Analysis Summary Chapter 5: Strengthening Risk Analysis in ADB Operations Introduction Summary of Risk Analysis Techniques: Applications and Limitations Strengthening Project Design: Overall Economic Benefits Estimation Promoting the Sustainability of Project Effects Supporting Poverty Reduction Objectives Risk Analysis and Policy-Based Lending Sectors and Projects: Some Typical Risk Analysis Situations Technical and Resource Considerations (including applying '@RISK') Appendixes Appendix 1: List of Reviewed ADB projects and Other ADB Project Evaluation Documents Appendix 2: Sustainable Livelihoods Framework Appendix 3: Illustration of Risk Analysis in Project Economic Analysis References 53 55 57 57 58 62 63 69 69 71 74 79 79 83 84 113 .

including a summary of the reasons why risk analysis may be undertaken. The Handbook is divided into five parts. The classic distinction between “risk” and “uncertainty” is ex- .Introduction Background and 1 Introduction The purpose of this Handbook is to support the development of a practical and operationally relevant methodological framework for the analysis of risk in project design and project economic analysis. Part II outlines the available technical approaches to modeling risk within conventional project economic analysis. Following a brief introduction.

Part IV considers the implications for risk analysis of ADB’s increasing emphasis on distribution and poverty impact analysis. and also to the UK Treasury in the application of the private finance initiative (PFI) in project design. Part V provides practical guidance on how to apply different types of risk analysis in different situations. This leads inevitably to questions of how to make choices among different risky investment possibilities (i.e. Techniques for modeling risk on the basis of probability distributions are then described. and attitudes to. different projects or alternative designs of similar projects) that may have been identified through the application of such techniques. and then some typical techniques for dealing with uncertainty are outlined. most notably the Department for International Development (DFID) of the United Kingdom (UK). Part III reviews the practical experience of ADB and other major agencies with regard to the analysis of risk in project design. Finally. it is suggested that increased lending to particular groups in society (notably the poor) now provides an imperative to extend the incorporation of risk analysis in project economic analysis. by better incorporating planners’ and project participants’ awareness and perceptions of. It will be noted that while there was quite a large academic literature on risk analysis in the 1970s (following the publication of the classic texts on project economic analysis in the 1960s and 1970s)..2 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS plained. This is expected to lead to better project selection in both cases—by both better identifying and quantifying sources of variability and by choosing project designs which meet poor peoples’ aspirations and more closely fit their circumstances. Different sorts of lending are considered in relation to real-world situations with respect to data availability. It is argued that practical applications of more rigorous risk analysis (i. as it is applied by. and a number of World Bank projects where risk analysis of different sorts has been applied (including some which are suggested to be representative of “good practice” in this regard) are reviewed. time and resource . As a guiding principle.. and that the consequences of project failure can be more extreme to those at or below poverty lines. there have been relatively few theoretical developments in recent years. Some 50 ADB projects (across all sectors and from a number of countries over recent years). This is followed by a summary of the nature and practice of sensitivity analysis in dealing with uncertain outcomes. based on existing techniques and given the emerging nature of ADB’s operations. the Asian Development Bank (ADB) and the World Bank (WB).e. This is not only because targeting the poorest may be inherently more uncertain than reaching other groups. but also because the risks of project failure are more concentrated within society. for example. risk in project design) during investment preparation can strengthen projects as well as policy-based lending. Consideration is also given to the practices of risk analysis by several other bilateral agencies.

Also. at that time) for undertaking risk analysis. Reutlinger. highlighted the statistical complexity of the techniques (particularly as regards dealing with covariance among variables). the modeling of the situation ex ante) and outcomes (i. the review of the situation ex post) had not been especially well-captured by the scope of the specific risk analyses which had been employed. Some practical guidance with applying software packages like the ‘@RISK’ is provided. The paper discussed the difficulties in obtaining reliable data in many circumstances (such that variables could be adequately described by different sorts of probability distributions). etc. including brief case studies of computer-based risk analysis applied to recent ADB projects.g. it was also striking to note that the causes of differences between expectations (i. Pouliquen—were either World Bank staff. an apparent methodological rigor in the appraisal process (as implied by a full-scale risk analysis) might actually obscure the search for radical project or policy alternatives. but noted that risk analysis in any form had only been applied in one ADB project (a port project) and two World Bank projects (a port project and a fertilizer plant project) by that date. The review also quite correctly pointed out that even where risk analysis was undertaken. among projects with net present value (NPV) > 0 at 12% discount rate] unless something was known about a society’s social welfare function . and noted the demands in staff and computer resources (mainframe.e. or were published by that institution). Of the reviewed ADB and World Bank projects in the 1985 paper.CHAPTER 1 Introduction and Background 3 implications. It may be relevant to note that this was despite the fact that major writers on the advocated techniques—e.. Various supplementary materials are contained in the Appendixes. There was also an inference that a spurious precision may appear to be attributed to results arising from risk analysis which was in reality based only on analysts’ “best guesses” for variables’ distributions rather than historical evidence.e. in itself it did not provide a basis for choice among competing acceptable projects [i..e... Background Johnson (1985) reviewed ADB’s practice (and also that of the World Bank) with respect to the application of risk analysis. The case studies are designed to highlight major features and key technical points— extending the original project materials in a demonstration context rather than necessarily a project-specific one. This paper provided a comprehensive and thorough review of literature and techniques available.

Appendix 21. to allow the construction of some sort of probability distribution) are mentioned. but not described in depth. where the economic internal rate of return (EIRR) may be just over 10-12%].” (Guidelines. or • projects are marginal [i. and recommended the application of quantitative risk analysis techniques for situations where • projects are very large (from a national point of view).e. as opposed to a single point value). no decision rules are offered in this regard “There is no fixed criterion for using such a result. page 157) as it is implicitly recognized that actual choices among differentially-risky projects will still depend upon particular levels of risk aversion being applied by different decision-makers. Risk analysis is presented in the Guidelines largely as an extension of sensitivity testing. This information should be incorporated into the decision as to whether to accept or reject the project. and it is suggested that such analysis should be conducted where sensitivity testing has shown that project returns are highly dependent upon the values which may obtain for a particular variable. ADB’s Guidelines for the Economic Analysis of Projects (1997)—hereafter the Guidelines—built upon the contents of the earlier review..e.. Practical applications of risk analysis in ADB project work did not appear to change significantly after the preparation of the review. The data requirements to undertake some sort of risk analysis (i. The Guidelines for the Financial Governance and Management of Investment Projects Financed by the Asian Development Bank (hereafter. suggesting that only certain sorts of situations were likely to be suitable for application of probability-based techniques. or in combination). The key point for analysts and planners to consider is the likelihood of a project’s ENPV falling below zero (at a 12% discount rate) or its EIRR falling below the economic opportunity cost of capital (EOCC). In conclusion.4 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS and extent of risk aversion—which in practice is still not likely to be available to analysts and planners today. or • there is considerable uncertainty over the values for key variables. Quantitative risk analysis involves consideration of a range of possible values for key variables (either singly. Financial Guidelines) . However.e. the paper recommended an extremely pragmatic approach to the use of risk analysis.. which then results in the derivation of a probability distribution of a project’s expected economic net present value (ENPV) or EIRR (i.

The Financial Guidelines therefore describes and advocates the use of risk measures such as credit at risk. One reason for the relative paucity of material on risk analysis in the Guidelines (and Financial Guidelines) is that economic theory suggests that for governments undertaking many independent projects simultaneously the consequences of risk on any one project can be ignored – risks will be “spread” across all members of society . delays in implementation of one or two years. executing agencies in transport.g. In addition to these financial measures. is in their analysis of financial institutions (FIs) which may participate in ADB investment projects.) to assess exposure to risk of FIs as a result of endogenous rather than exogenous factors.g. The analysis is seen largely in terms of sensitivity testing. etc. but provide no description of particular techniques. for economic analysis). power. the French Development Agency . etc. and contagion risk. however.. etc. Although these measures are available to project designers. some development agencies (e..e.g. ADB staff] depending upon individual country and project circumstances. +/.e. rather than for smaller projects – e..CHAPTER 1 Introduction and Background 5 also take a very similar approach to analysis of unknown financial outcomes. microcredit/credit unions. for large. Where the Financial Guidelines do differ from the Guidelines in their analysis of risk. a range of standard accounting and financial measures is used... compliance with regulations. When assessing FI performance. which includes indicators designed to assess risk. of project preparatory technical assistance (PPTA) consultants. water and sanitation projects. the Financial Guidelines mention the possibility of quantitative risk analysis and describe circumstances in which it may be appropriate (i.10% or 20% for values of critical financial cost and benefit items. etc. and sensitivity indicators (SI) and switching values (SV) calculated for each variable tested.. financial net present value (FNPV). ‘financial sector’ operations. actual practice suggests that use is not made of them as much as might be expected (they are typically considered in larger. etc. maturity risk. for agriculture banks. value at risk (VaR). marginal or very uncertain projects). These are conceptually probability-based. monitoring and evaluation systems.g.g. although the actual data upon which they are calculated may come from entirely objective sources (e. foreign exchange risk. and the advocated techniques are such that standard changes (e.)...) are measured in terms of their effects on estimates of financial internal rate of return (FIRR).) may assist with ensuring greater sustainability of project effects (as the ultimate realization of economic benefits often depends upon the operating performance of such institutions). In almost identical format to the Guidelines (i..g. historical data series) or can result from largely subjective assessments [e. It will be suggested later that their selective extension and wider application to other typical sorts of project institutions (e.AFD) use systems of indicators (covering management practices.

Some forms of risk analysis can also be undertaken within existing spreadsheet software (e. log-normal. The question remains as to how and in what circumstances such tools may be advocated as appropriate to be applied much more widely. etc.g. exponential. While the fundamental issues concerning data (and the extent to which situations can be described as “risky” as opposed to simply “uncertain”) remain and are germane to individual project situations. financial and economic analysis at any stage of project preparation..) the burden of risks becomes more concentrated within society and these general assumptions begin to break down. and an add-in to Microsoft Excel) was highly suitable for • undertaking “Monte Carlo” based simulations to derive probability distributions of outcomes (including of project EIRRs/ENPVs) • fitting distributions to data sets (using advanced algorithms). . to the extent that project lending may tend to become concentrated on specific sections of society (i. @RISK] the @RISK package (Professional Edition. A survey of commercially available risk modeling software (Mariano 2001) concluded that [among several competing packages. The implication is therefore that the tools for at least some types of risk analysis are now potentially widely available (and indeed one software package for dealing with risk in this way. i.) to data sets and generating expected values (e. etc. binomial.g. and also having easily available computer software capable of fitting probability distributions (e. Excel)—not even requiring any add-ins.e. ‘RiskEase’ (updated version of ‘RiskMaster’). for project ENPVs. considerable advances in computer software in recent years mean that (where data are available) risk analysis can now be undertaken extremely easily as an “add-in” to existing... The original conceptual basis for this argument was established in a classic article by Arrow and Lind (1970). “Risk Master” has already been applied within ADB). However. Lotus.. including ‘Crystal Ball. uniform. notably power. individual regions.e. and have been concentrated in certain sectors. As will be described in Part III of the Handbook..6 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS and “pooled” across the portfolio of all projects – and thus the government can be taken to be ‘risk neutral’ as far as individual projects are concerned.. EIRRs. One reason for this situation has undoubtedly been difficulties for staff and consultants with obtaining reliable data about key variables. beta. etc. particular groups—including the poorest in society. certain sectors. predominantly spreadsheetbased. and also for absolute values in the context of cost-effectiveness analysis) with associated measures of variance.g. normal. and • viewing graphically the distributions of variables and outcomes. actual applications of risk analysis in ADB operations have remained relatively limited since 1985.

Much of the discussion in project economic analysis texts and in academic literature concentrates on the outputs of quantitative risk analysis (i. and that the extent of any remaining risk is both quantified (i.. policy-makers. private operators) can share and manage risks appropriately. rates of adoption and usage) which are the key determinants of project outcomes • determining the likelihood of an individual project’s returns being unacceptable (i. etc.. In practice. it may be worth reviewing exactly what the purpose of applying any such techniques really is. ADB is typically not concerned with making choices among a number of mutually-exclusive. competing projects but is more usually engaged in reviewing projects one-by-one.. and • designing measures within that project environment and its sector context to mitigate the identified risks arising from the identified key factors. ‘expected’ EIRRs/ENPVs plus associated measures of variance) as being useful for making choices between different investment projects (i.. For this reason the analysis of risk through the techniques described below is practically of most use in • identifying those factors (e. In general. the principle to be .e.e... borrowing government. borrowing government.e.. known) and its existence is regarded as ‘acceptable’ (i. to ADB.e.e.) given the nature of the particular intervention proposed. EIRR<EOCC. ENPV<0) because of the effects of the identified key risk factors. however. planners. it is demonstrated that understanding a decision-maker’s (i. Following on from this.g.. quantities. society) subjective attitude towards risk—how higher expected returns are traded off against increased variability— enables consistent choices to be made across a portfolio of public sector investments.e.CHAPTER 1 Introduction and Background 7 Why Undertake Risk Analysis? Before considering in some detail the various techniques available for risk analysis.g. It is also the case that comprehensive analysis of project risks helps in designing projects so that different parties (e. projects with higher expected returns but more variability of returns may be compared to less attractive but more stable opportunities). project beneficiaries. prices. operating entity. The emphasis and presentation of any form of risk analysis in project economic analysis observed in practice is thus usually on demonstrating that risks to individual project success have already been identified and mitigated as far as possible within the proposed project design. The risk analysis techniques are thus essentially used to complement sensitivity testing in demonstrating project robustness.

building a hospital). this type of analysis has been particularly well-developed within the UK Treasury PFI methodology. It is not simply an afterthought to economic analysis. it should be noted that. appropriate consideration of all risk types enables them to be shared among the participants.. .8 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS followed is that the parties who can best manage different sorts of risks to projects (e. delays.g.. defaults. Therefore. from ADB’s perspective.g. Where project environments include a private sector participant (e. operating costs. risk analysis is fundamentally a project design tool. construction costs. in running a toll road.) should be allowed to reap the rewards or bear the costs of risk management. although risk analysis (of whatever particular form) may typically appear at the end of an economic analysis in a typical Report and Recommendation of the President (RRP). etc.

This is then followed by a summary of methods for dealing with the existence of project outcomes which can only be modeled as uncertain. This includes (but is not limited to) techniques already applied as standard practice by ADB. It then summarizes the techniques available for modeling risk on the basis of 2 . rather than risky. It begins with a conventional definition of the terms “risk” and “uncertainty”.CHAPTER 1 Introduction and Background 9 Technical Approaches to the Analysis of Risk in Project Economics Introduction This part of the Handbook provides a summary of available techniques for dealing with the outcomes of unknown events in project design and economic analysis.

risk as perceived (for example) by those investing in or affected by a proposed project. . Thus. the situation can be modeled as “risky”. Pouliquen (1970). Risk and Uncertainty: A Definition of Terms The term “risk and uncertainty” tends to be applied generically to the analysis of situations with unknown outcomes.10 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS probability distributions (this is the most widely understood area of risk analysis). This is an important aspect of risk analysis in project work because it is always the case that choice between alternative risky outcomes involves knowing something about decision-makers’ or participants’ preferences. On the other hand. ranging from perfect knowledge of the likelihood of all the possible outcomes at one end (i. risk is a quantity subject to empirical measurement. while uncertainty is of a non-quantifiable type. following Renbourg (1970) from which the quote below is taken. but merely the information available to planners and analysts which defines it as such.e. All actual project outcomes are unknown. risk) to no knowledge of the likelihood of possible outcomes at the other (i. and also Reutlinger (1970). measuring the extent of individuals’ or decision-makers’ risk aversion). the fluctuations of a variable are such that they cannot be described by a probability calculus. It is important to realize at the outset that it is not the real-world situation itself which is either “risky” or “uncertain”. This is followed by consideration of issues involved in modeling what are sometimes called “subjective” attitudes to risk—i. uncertainty). etc.e. if we do not have such data we can only describe the future in terms of “uncertainty”. If we have reliable historical or forecast data such that a probability distribution can be constructed for such variables. This document will follow the conventional distinction between risk and uncertainty made in the literature [e. each of which may take different values..]. in situations of uncertainty... Thus risk and uncertainty are best thought of as representing a spectrum of unknown situations with which an analyst may be dealing.e. in a risk situation it is possible to indicate the likelihood of the realized value of a variable falling within stated limits—typically described by the fluctuations around the average of a probability calculus. In essence.e. and in particular how they are willing to trade off increased rewards against increased risks (i.. because they occur in the future and are subject to influence by a number of variables.g.. and highlights both advantages and limitations of such approaches.

It is argued that such a distinction is in fact very useful because it helps to separate those situations which may be subject to quantitative analysis from those which are not.” Clearly such a definition blurs the distinction made previously between risk and uncertainty. If we do not have such data then possibly only “high”. and which may be described on the basis of their probabilities of occurrence. The term includes potential for gain and exposure to loss. Attempts to model the impact of uncertain outcomes and develop decision rules about what choices to make (e. It should be noted that this distinction between risk (unknown but quantified outcomes) and uncertainty (unknown and unquantified outcomes) is not usually so clearly made in typical financial analysis. Allowing For Uncertainty Project economic analysis (and the overwhelming weight of both ADB and World Bank experience) tries to allow for the existence of unknown future outcomes in the most basic sense by modeling the existence of uncertainty rather than dealing with risk per se. Similarly. population growth. depending upon whether seasonal rainfall or water flow is above or below certain levels.CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 11 In dealing with an agricultural project.. the UK Treasury Taskforce on Private Initiative (2000) quotes an accounting definition of risk as follows “A simple definition of risk as used by the accounting profession is uncertainty as to the amount of benefits. In both cases there is nothing inherently different about the circumstances of the projects themselves. etc.. only the data available to the analyst which makes modeling of risk more or less possible. if historical rainfall patterns or irrigation water supply data exist we may be able to construct a probability distribution such that crop yields can be predicted in terms of expected values with associated levels of variability. or may be modeled on the basis of a distribution of outcomes of future power demand which itself depends upon estimates of economic growth. for example. “likely” or “low” values for crop yields may be estimated.g. between different projects or alternative project designs) derive from the operations research (particularly linear programming models) . For example. analysis of an energy project may be undertaken in terms of “optimistic” or “pessimistic” assumptions about domestic and commercial power demand levels (and different returns predicted under such different scenarios).

and also in the Financial Guidelines (section 7. von Neumann. the “Laplace” criterion effectively ignores uncertainty altogether. and this is described in detail in the Guidelines (page 39. game theoretic criteria were largely abandoned as models of descriptive or prescriptive behavior. The most widely-applied technique for describing uncertainty is sensitivity testing. Morgenstern) approaches of the 1960s and 1970s. and the MINIMAX REGRET does away with normal assumptions about decision-makers’ preferences (because they are more concerned about minimizing losses ex post than about maximizing returns ex ante). both ADB and World Bank recommend practices such as: • testing for the effects of changes in aggregate project costs and benefits • testing for the effects of changes in individual underlying variables (e. consumer utilization rates . Although emphases and presentation differ. yields. projects would be chosen on the basis of various proposed criteria.11). and Appendix 21).. according to decision-makers’ preferences. For example.. without attempting to derive decision rules to guide choice under uncertainty. sensitivity testing involves changing the value of one or more selected variables which affect a project’s costs or benefits and calculating the resultant change in the project’s NPV or IRR.. Despite some historical applications for planning purposes.12 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS and game theoretic (e. In essence.g. operating costs of machinery in a roads project. In situations of different possible project alternatives and uncertain future events (“states of nature”). It can be shown that such criteria (and others that were developed around the same time) are in fact all “irrational” in different ways. areas. A full explanation of the technique and its application is also provided in Belli et al’s Economic Analysis of Investment Operations (World Bank Institute 2001). and • MINIMAX REGRET—select the project which minimizes the maximum opportunity cost of having made a wrong choice by choosing a “state of nature” which does not in fact obtain.g.g. for such reasons as just given. highest NPV) if the situation/”state of nature” turns out as badly as possible. Such proposed criteria included • “Laplace”—select the project or design alternative which yields the highest return. the MAXIMIN assumes “nature” to be as malevolent as possible (which is not the case). whatever the “state of nature” obtains • MAXIMIN—select projects or design alternatives which yield the best returns (e. prices of cement. crop prices in an agricultural project. and subsequent practice in ADB and elsewhere has largely concentrated on describing unknown outcomes alone.

Sensitivity testing leads to the calculation of switching values (SVs) and sensitivity indicators (SIs): • SV identifies the percentage change in a variable for the project NPV to become zero (i. but how likely is it that either or both will occur in practice? . publicity campaigns to promote service usage.g. The technique is extremely easy to apply.. shift the benefits stream down a year or two) testing likely combinations of variables (especially if these may in practice be linked —e.) made by the analyst. tax and tariff changes. etc.. NPV).g.). and testing for changes in economic pricing adjustments (e. and mitigating action can then be taken (if desired) to minimize the consequences of such outcomes. so as to be able to identify the ones with most impact on project NPV testing for delays in benefits or implementation (e.e.. the technique has a number of limitations: • most fundamentally.. securing long-term supply contracts (for inputs and/or outputs). standard conversion factor. or that for traffic flow may be 15. as changes to one value in a spreadsheet will reflect instantly in values for NPV. etc. However. shadow wage rate factor.500 vehicles per day the project would not be viable” • SI compares the percentage change in a variable with the percentage change in a measure of project worth (e. Likely mitigating actions include undertaking pilot projects..g. average yields would have to fall by 20%). etc. The SV for crop yields may be a fall of 20%. IRR. increasing technical assistance and training levels to support project implementation. shadow exchange rate factor. Sometimes SVs are expressed in terms of the absolute value of a variable—e. This choice of variables will usually be based on previous similar project experience and/or detailed sector knowledge as much as on the particular project in question testing variables one at a time.500 vehicles per day. “if passenger traffic volume fell to 15.CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 13 • • • • in a power or water supply project. project costs go up AND implementation delays simultaneously occur).. etc. The prime utility of sensitivity testing is that it leads to the identification of those variables to which a particular project design is most sensitive.g. for the project decision to switch between “accept” or “reject”. it does not take into account the probability of the occurrence of the events it models.g.

in determining what an appropriate “risk premium” should be). One other point may be mentioned in regard to modeling uncertainty in project economic analysis. While there is a large theoretical literature on this point. mean and modal values may be very different from one another. and misunderstanding the nature and extent of correlation between variables can lead to erroneous results. It is sometimes suggested that uncertainty can be allowed for by either applying a different discount rate in the calculation of NPV or by using a higher cutoff rate (i. etc. mean that its use has become widespread. An impression of homogeneous variability is given. Its ease of application and familiarity of concept. However. combined with analysts’ understanding of particular sectors and projects (which should lead to reasonably appropriate variable selection and extent of variation being applied) plus its usefulness in leading to development of mitigating measures and project redesign. greater than 10-12%) for investment decisions.e.. its dependence upon judgment rather than empirical evidence and its modeling of uncertainty rather than risk (plus its inability to offer any decision rules following the presentation of its results) mean that its usefulness as a technique is ultimately limited..g. • the identification of appropriate groups of variables to vary together depends on specialist knowledge. the “base case” estimate of the value of the variable is its average value) or are deviations from “most likely” (or modal) values. Overall. it is not clear whether the variations in values which are being modeled are changes from “expected” values (i. depending upon the characteristics of particular distributions (in effect the extent of skewness in the data set). and what is being captured in the base case and its variation is not clear. . which is not necessarily true.). the variability in power demand is less than in generation. the use of standard percentages for variations (changes of +/– 10% or 20% are routinely applied for example) in sensitivity testing captures quite differential extents of likely variability. and • because the distribution characteristics of different variables which determine project outcomes can differ enormously (the variability in commodity prices is less than input prices for example.e. sensitivity testing is a highly subjective technique.. in essence there is no justification for this approach— apart from any other consideration (e.14 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS • where deviations from project “base case” estimates are modeled in sensitivity testing. The discount rate is a rate of decline in the numeraire of economic value. and has nothing to do with the source of risks facing an investment. which is not warranted by reality. it assumes that risk always increases with time.

a probability distribution can be constructed for the variable(s) in question. together with the frequency (or likelihood) of each of these values occurring. or that the project IRR will fall below the opportunity cost of capital. usage of similar services elsewhere in the city for an earlier project). Appendix 21. Table 1 shows the numbers of bus passengers who may be expected on a particular route per week (and which may represent.025 .e. upper and lower limits to data values..25 60 0. values above and below the “base case”. a number of possible data points (i. page 156) While the Guidelines do not themselves provide a methodology for the application of risk techniques. various attempts have been made to properly capture the impacts of unknown outcomes through modeling risk quantitatively in project economic analysis. it is suggested that the results of sensitivity testing be used to consider which variable(s) may be appropriate to base a risk analysis upon (i.CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 15 Modeling Risk Quantitatively Because of the conceptual shortcomings of all approaches to modeling uncertainty. As the Guidelines state. The average number of passenger bus trips is 50.386 and the coefficient of variation (the ratio of the standard deviation to the mean) is 0. deriving directly from the data in the table. the purpose of quantitative risk analysis in essence is to “provide a means of estimating the probability that the project NPV will fall below zero. those that have major impacts on project outcomes).e.2 70 0.2 50 0.025 20 0.) are necessary to be specified. Having identified particular variables. From such data points and associated frequency estimates. Table 1 Number of Bus Trips and Frequency of Occurrence Number of bus trips (‘000) Probability of occurrence 10 0. the standard deviation (the average difference between all observations and the mean/average) is 27.1 40 0.05 30 0.05 90 0.. The figure below shows graphically a simple probability distribution (based on frequency of observations falling within particular class intervals) for the number of bus passengers.” (Guidelines. etc. for example.000.547.1 80 0.

for example. and a 5% chance that the number of passengers may only be 20. actual numbers of passengers on project-financed buses each week may well determine both the financial profitability of the bus company and also the size of the economic NPV of the project. . It can be seen that.000 per week—and we can now say what the probability is of this value obtaining. but there is a 10% chance that the number of passengers may only be 30.000 a week..16 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Figure 1 Passenger Bus Trips Probability Distribution From the data.000 per week. it is apparent that the expected number of bus trips is 50. The following figure (Figure 2) shows the same distribution in the form of a cumulative probability distribution function (which has been “smoothed” to allow for the effect of observations of grouped data).000 (and this may be the basis for the estimate of the project “base case” scenario).000 passengers per week. The function plots the cumulative probability that the actual outcome will be below a certain level of bus trips. the probability is relatively high (i. It may be the case. for example.5% probability that the number of bus passengers will be below 40. that sensitivity testing has indicated that the SV (switching value—the value of the variable at which the project investment decision is changed) for bus passengers is at or near 40. In this case. there is a 27.e. which may well imply that a redesign of the proposed project would be appropriate. over one chance in four). In this example.

CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 17 Figure 2 Cumulative Probability Distribution The ultimate purpose of quantitative risk analysis is to generate a similar cumulative distribution probability function for a project’s NPV. such that the probability of negative NPV being generated is explicitly identified. The following figure shows such a chart: Figure 3 Cumulative Distribution Function .

as well as ADB’s 1985 review.g. equivalent to implementing the project again and again in different circumstances—and is usually at least 1000 times. 30% in the figure). Pouliquen 1970) and also standard texts on project appraisal (e. all mention the fact that computer time and expertise is likely to be a major constraint to the use of this technique. it should be noted that identical procedures to these can be applied to projects where expected ENPV is not typically calculated in ADB practice (e. Lotus) is used to generate random values (through the use of a built-in function—@RAND in the case of Lotus) which are then repeatedly applied . In the model described in this particular article. the cost per primary school child educated would be Rupees 21. and these figures are used to calculate an estimate of the project NPV. With the discussion so far..18 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS The probability of project NPV being negative as a result of variability in underlying factors is thus able to be identified (i..000 per year. for example)..). There are even examples of risk modeling using spreadsheets alone (see Clarke and Low 1993. The only difference is that instead of expected NPVs would now be absolute measures of expected cost-effectiveness of outputs or impacts (e.g.g. the analysis of risk in this form is invariably undertaken by some kind of computer software. In recent years this constraint has largely been overcome and more than adequate computational facilities and software are now available to practitioners.. combined with other randomlygenerated values for the other variables. Risk analysis typically involves the choice of several variables to be varied simultaneously. This process is repeated a large number of times (a number which is specified by the analyst—in effect. and most literature describing such techniques. education and health projects. Little and Mirrlees 1974. as project returns are generally subject to more than one source of risk. The early literature on risk modeling (e.000 in four years out of five. Because of the mathematical complexity involved in such calculations.). annual cost per health worker employed.e. a standard spreadsheet (e.g. and provides further information as to the relative attractiveness (in terms of its riskiness) of the project. etc. and typically more than this) and an average (or “expected”) NPV is produced together with an associated probability distribution. which tend to use measures of cost-effectiveness rather than total net economic worth). Squire and Van Der Tak 1975. Excel.. Reutlinger 1970. in 75% of circumstances the cost per health worker would be less than Baht 4.. The same data could of course be used to model the financial profitability of the bus company in a similar fashion. etc.g. cost per primary school pupil educated) quoted together with distributions for those values (e.g.. This distribution can usually be viewed in the form of charts like those in Figures 1 to 3. The process which is followed (and which is usually referred to as “Monte Carlo” or simulation analysis) is that values for individual variables are generated randomly according to their respective probability distributions.

for commodity prices. It is important to note. The actual situation with data availability is likely to vary enormously both between project situations and also across different sources of variability within any one project environment. At the other extreme may only be the existence of a few data points (e. therefore. “maximum possible”) can be constructed based on “best guesses” of project . for rainfall. triangular distribution from three points (i.) which are expectations of experts/analysts involved in preparing the project. plus a range in which outcomes will lie 90% of the time (based on all variables being subject to simultaneous change). Other possibilities lying within these bounds include the forecasting / specification of power-generation theoretical capabilities adjusted for a set of likely different operating conditions. “minimum possible”..e. Software such as @RISK often has capabilities to fit probability distributions of different types to raw data sets supplied by the analyst. It is to be noted that the technique is quite simple and practicable.. etc. that very large and complete data sets from empirical sources are not always necessary for the undertaking of risk analysis. Simplifying assumptions about variable distributions can be made—as a bare minimum.g. etc. based on World Bank publications taking into account world supply and demand factors). “most likely”. despite the overcoming of computational limitations to the application of such techniques. Expected NPV and IRR for the project are calculated based on the distribution of results.g.g.. “most likely” values. and the extent to which the situation can reasonably be defined as risk (as opposed to uncertainty) through the construction of a meaningful probability distribution of outcomes. two major practical considerations (and possibly constraints) remain as regards the extent to which such techniques as Monte Carlo simulation can be used in project preparation situations. maximum possible. for traffic flows).CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 19 to variables identified within the project “base case” scenario so as to generate a distribution of outcomes (counts of observations of variables/expected NPV values are recorded in cells so as to construct a frequency distribution). At one extreme. Technical Issues in Modeling Risk: Two Considerations However. large volumes of reliable cross-sectional or time series data may be available from historical sources for the variable concerned (e. forecasts of trade flows and commodity prices (e. Such routines will fit distributions to data and also provide a measure of the goodness of the particular resulting fit. Firstly is the issue of data availability.. absolute minimum possible. and requires no more information than is already required for traditional sensitivity testing. and the base case is then presented in terms of expected NPV/IRR.

there is little difference in resulting distributions of EIRR/ENPV outcomes (i. crop yields..e. However. World Bank in a risk analysis of institutional reform in the irrigation sector in Pakistan (Dinal et al.. enrolment rates. choosing between uniform. normal. etc. minimum and maximum values. the random number generation approach necessarily produces a rectangular uniform distribution (i. expected values and variance. Well-tried empirical methods exist for developing probability distributions from such subjective sources. etc). triangular distributions for variables such as crop yields. to replace the quoted discrete distributions with relatively few values by continuous distributions based on large amounts of empirical data. Good examples of such knowledge might include rainfall and water flow. one in which all possible observations within the range defined by the analyst have equal probability of occurrence). . operating efficiency of agroprocessing machinery.e. matches or stones piled up to represent frequency of value occurrence).20 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS preparation team members. this approach may not always be appropriate for all variables. etc. Recent experience of preparation of power projects within ADB also suggests that the particular form of distributions also matters less if a large number of simulations are run... time taken to collect water or firewood. Some proponents of probability-based risk analysis (e. traffic flow. The ‘Delphic’ method of eliciting opinion from local experts is an example of this type of approach. quartiles. for example. and has been applied (in a probability-based form) by. Clarke and Low 1993) also argue that the shapes of particular distributions of individual variables (e.g. These include • visual impact techniques (e. the median. to adjust the spreadsheet model to produce (for example) a normal distribution (as opposed to a uniform one) becomes very much more complex. • structured questions to identify key points in a distribution (e. There is also often considerable expertise within the project preparation environment about the likelihood of variables or outcomes which may not be available from official sources but which can be elicited from potential project participants.g. etc. income differentials.g.g. for example.. – the “judgmental fractile” method).. In the Clarke and Low example (from an agricultural project in East Africa). although with more complex formulae being written this could be adjusted. and • the application of “smoothing” techniques in situations where a few real data points may be available. Also. and it still requires judgment on the part of the analyst about what ranges are acceptable for values to fall within.) are less important than the choice of variables themselves which are allowed to be modeled. number of family days sick per year. 1997). Even when considerable effort is made. or travel to market.

and the numbers of days and months of employment gained following nonformal training in Bangladesh. and typically simple rank correlation coefficients between pairs of variables are sufficient for most purposes (in the risk software packages. Another typical example of covariance (which would be positive in this case. as all variables are not always subject to relatively large numbers of random influences (which is what typically causes a variable to be normally distributed).e. typical risk analysis software such as @RISK can handle this within the context of specifying correlation matrices). future computational forecasts. However. the techniques applied to develop definitions and derivations of probability distributions for individual variables in most cases is likely to depend upon some subjective judgment by an appraisal team — and inevitably the extent to which these design assumptions adequately reflect the reality of the project will vary from case to case.CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 21 It is also noted in the early risk literature (e. and therefore more than one variable at a time is modeled in the Monte Carlo simulation exercise. Overall. If these underlying variables are correlated in some way (which may well be if project output is large relative to market volume. In practice. Projects are rarely subject to only one source of risk. One of the case studies using @RISK includes correlation between estimates of proportions of trainees finding employment. Where variables are in fact statistically independent of one another there is no problem. The suspicion that what appears as a full-scale risk analysis has in reality only a spurious precision can be ultimately only fully allayed if the data upon which the variables’ probability distributions are constructed (either from historical evidence. the approach to assigning particular levels of covariance between variables is quite pragmatic.. Pouliquen 1970) that there is no a priori case for the use of normal distributions (even as a default distribution). .e. therefore.. Where variables may be thought to be related in some way. project-induced water supply under an irrigation scheme improvement) may be that between area planted and average yield (i. if it is assumed to be due to improved. project revenues are typically products of both quantities sold and prices obtained. As an example. The second consideration when applying risk analysis in practice is the extent of covariance between those variables that are to be selected for risk analysis. as it is appropriate to treat them independently. project revenue) is equal to the product of the individual expected values plus the covariance between the two variables.. the extent of covariance between them needs to be taken account of when specifying the distribution of individual variables in some type of simulation (again.g. and negatively so in this case) the expected value of the product of two random variables (i. or analysts’ ‘best guesses’) are believable. with both variables as determinants of farm production volumes). statistical complexities can arise depending upon the relationships between the selected variables. however. correlation between variables—once specified—can typically be “toggled” on/off).

from objective or subjective sources. there is therefore a similar judgment to be made about the extent of disaggregation to be applied in individual circumstances. whether these distributions are triangular. etc. and this could be most appropriately captured through some item such as “construction materials” rather than by introducing additional correlation between such items (which would tend to increase unnecessarily the estimate of overall variability). etc. and quote appropriate statistical measures (e. Anderson-Darling statistics. etc. whether it is based on historical observations or forecasted projections. Chi-square. and (based on this) Explain and justify the extent of any variable disaggregation. etc. cost of floor. from an expert-based ‘Delphic’ process.. although individual construction cost items (e.e. For example. the ‘key’ variables) Fully explain the general nature of the data set which is used for modeling those variables’ values (its origin—i. the principles to be applied in practical situations to quantitative risk analysis such that the issues just described are dealt with as transparently as possible are summarized in the following table: Table 2 Principles to Apply in Data Handling for Probabilistic Risk Analysis Principles to Apply 1 2 Identify those variables for which future values are unknown and which are likely to affect project returns (i.e.e. in reality the sources of this variability all arise from one point (e.) Explain the statistical nature of those variables’ assigned probability distributions (i. from visual techniques.) Make clear the goodness of fit of the distribution to the data set (if one has been fitted using @RISK or similar software). etc.) Make explicit any correlation thought to exist between variables used in the risk analysis (i. In sum. real-world basis for the assumption. explain the method by which it was elicited (e. cost of walls) may each be thought to vary individually.g. costs of imported cement).. uniform. its extent of completeness/any missing data points.22 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS It is specifically recommended that disaggregation of individual variables be limited as much as possible so as to avoid including too much correlation in the analysis.g..)...) If the data derives from subjective sources. from subjective questioning.e. logarithmic. Kolmogorov-Smirnov.. normal. 3 4 5 6 7 .g.. its extent. cement.g.. Akin to the nature of some subjective judgment being involved in the allocation of probability distributions. the number of observations the data set contains. and the technical. exponential.

B. and also by the probability of the return falling below some unacceptable level in particular). but its expected value is lower. etc.e. If variance of returns is plotted against expected values for such project situations as A. such as variance and coefficient of variation. Projects A and C may lie on the efficient investment frontier—although at different points.CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 23 The focus of the reporting of the results of risk analysis will be the likelihood that project returns may be negative (e. and C) as shown in Figure 4. and C. Risk Analysis. the expected higher returns of project C have to be weighted against the increased degree of risk of the project. but such reporting should not obscure any basic qualifications about the analysis which need also to be included. in the sense that its variability is the same as project A. project B is clearly “inefficient”. Decision-Making. and especially about the extent of correlation which has been assumed. the following is obtained (Figure 5 in what is usually referred to as E-V space). In Figure 4. Consider the following three project alternatives (A. B. EIRR. The discussion of the ‘acceptability’ of particular levels of risk is usually presented in standard economics texts in terms of choice among competing projects.e. the expected value of their economic return (as measured by their expected ENPV.) and their degree of risk (as measured by their variability in general—captured by the distribution’s measures of dispersion. This quantitative measure of risk adds to the information available to the decision-maker. Project C has a higher expected NPV than project A but it also has a greater variability of returns (including the possibility that its return will be zero). and Welfare The result of risk analysis as just described is therefore to identify projects (or alternative designs of the same project) which now have two essential characteristics—i. How should a decision be made between the alternative projects? The traditional view taken towards public sector investment was that governments with large project portfolios could afford to ignore the riskiness of investments as long as the expected values were acceptable—i. . while project B lies within the inefficient frontier.g... “Good practice” in this regard is likely to require reporting “with” and “without” correlation being applied in the simulation as the effects of ignoring correlation can be substantial. although in itself does not necessarily provide any guide as to whether any individual project is acceptable (or even as to which project among several possible ones actually should be undertaken). In this case. “the project’s rate of return is likely to be unacceptable in about 15% of all cases”)..

24 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Figure 4 Probability Distribution: Three Projects Figure 5 Expected Returns-Variance Frontier .

one type of student. in general.CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 25 they could afford to be “risk-neutral”.. in $000) and variability of income (measured by annual income variance). What it suggests. In fact. income—against levels of variability of returns). welfare or satisfaction – and is in fact an indifference curve) from combinations of expected income (measured. Figure 6 Risk Aversion: Line of Equal Utility . in one region. is that for people to accept greater variability in their incomes they need to receive increasingly larger expected incomes.) such that the impact on those particular individuals could not be ignored. etc. say. or affected particular groups (e.. the rate at which they are prepared to trade-off levels of expected returns—or in effect. or were somehow correlated with national circumstances (such that good performance of the project in bad years for the economy as a whole was worth more in terms of a disproportionate contribution to national income).e.g. the costs of any individual project failures could be absorbed within the portfolio as a whole. Figure 6 shows how many individuals are thought to be risk averse.e. there is no answer to the question of project choice in such circumstances without reference to knowledge about the extent of decision-makers’ risk-aversion (i. Exceptions to this view were projects which were either very large. This is because. the line joins points of equal utility (i.. with a large number of investments spread across all of society.

but with analyzing risk as faced by individual projects one at a time. Anderson 1974. Again. it is necessary to combine the notions of efficient projects located in E-V space with decision-maker’s utility functions expressed in the same space. however. Probably the most well-known of these approaches is the Capital Asset Pricing Model (CAPM. ADB practice as regards project economic and financial analysis is not primarily concerned with a number of projects as they may comprise an investment portfolio (or even with one project within the context of all projects in the portfolio). academic risk literature has also concentrated on the extent to which • the consequences of particular risks are catastrophic or not.. • the risks are controllable at the micro level or not. Thus in Figure 7 below. • the consequences are reversible or not. and • the risks are insurable or not. other models have attempted to capture the characteristics of this situation within the context of financial investment portfolio analysis. To complete the above analysis and understand what decision actually should be taken in any particular circumstance. professionals and business executives were all studied).g. although the specific income levels at which this was the case and also the extent of risk aversion could vary enormously even within apparently similar circumstances (small farmers in Africa. where it was hoped that insights about preferences regarding risk would lead to making optimal project choices. . one decision-maker on the basis of an individual utility function (U1) will choose project A (lower expected NPV.) suggested that most individuals were risk averse. As well as the E-V approach to considering the risk-reward relationship.26 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS The actual nature of decision-makers’ utility functions as regards their risk aversion occupied a large literature (much of it deriving from agricultural economics and work with small farmers in developing countries) around the same time as the classic texts on project appraisal and risk analysis were written. In order to fully understand how people actually deal with risk in project and other situations in terms of the decisions they make. developed by Sharpe and Lintner in the 1980s) which measures an individual stock (or project) risk relative to the volatility of returns relative to a market (or sector) index. The results of many classic empirical studies of risk perceptions from the 1970s (e. Australia. These considerations are returned to in Part IV in relation to the attitudes of the poor. Swalm 1966. higher risk) on the basis of a utility function such as (U2). and the USA. lower risk) while another will choose project C (higher expected return. workers in large corporations. etc.

greatly increase the possibilities for application of quantitative risk analysis as a more commonplace part of project design. it is also the case that the present availability of computer software to easily process Monte Carlotype simulations from data already available to the analyst within spreadsheets.CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 27 Figure 7 Project Choice and Risk Aversion Summary of Approaches to Risk Analysis This Part of the Handbook has summarized approaches to dealing with unknown outcomes in project analysis through description of various techniques attempting to capture the content and consequences of uncertainty and risk. although it is also obvious that data and time considerations (and the difficulties in properly identifying and specifying covariance among variables) have often limited the extent of actual risk analysis practice. It is clear that the quantitative modeling of risk is in principle preferable to the simple depiction of uncertainty. plus the existence of statistical routines and computational processes to fit probability distributions to many (even quite sparse) data sets. The results of quantitative risk analysis can greatly inform the process of project design. so that mitigation measures can be put in place before projects are . However.

What level of risk is it appropriate • for ADB to accept? • for ADB to suggest that borrowing DMC governments or institutions accept? • to impose upon groups of project beneficiaries? are questions which have to be answered outside the scope of those quantitative techniques which provide the measures of project risk. however. so that their likelihood of failure is identified). They can also increase the information available to decision-makers about individual projects (i.28 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS implemented. it will be suggested (in Parts IV and V) that strengthening risk analysis in ADB operations involves both greater application of the probabilitybased techniques for modeling sources of risk in project design (located within a framework to guide such application) and also greater awareness of attitude towards risk amongst those being planned for in ADB projects. While the results of risk analysis are quite easy to understand in conceptual terms for planners and project staff.. The question is akin to the difficulty of suggesting a priori cut-off points for the Poverty Impact Ratio (PIR) as it is inherently associated with social welfare functions as perceived by decision-makers. . Following a review of agency experience with the application of the techniques just outlined (in Part III).e. these results in themselves do little to aid project investment decision-making unless and until they are combined with some consideration of planners/policymakers and/or project participants’ risk aversion.

and sector syntheses (from 1994 to 2000) 3 .CHAPTER 1 Introduction and Background 29 A Summary of Development Agency Experience in Applying Risk Analysis Introduction This Part of the Handbook reviews ADB’s and other agencies’ actual experience with risk analysis in recent years. In addition. The review of ADB experience is based on a review of some 50 recent RRPs covering projects in all sectors of lending operations to identify what risk analysis practices have been employed by ADB in practice (see list in Appendix 1). various project performance appraisal reports. impact evaluation studies (covering the late 1980s and 1990s). special studies.

and a number of projects were briefly reviewed which are propounded (e. However. It is suggested.. inter alia. although the existence of risk as affecting project . and that among the ways of dealing with such unknowns greater attention should be paid to risk analysis (through the application of simplified simulation techniques). DFID’s experience and practice is also noted for the rigor of the application of the logical framework technique and the explicit attention to identification of sources of risk within this framework. that various biases are routinely introduced into estimation of project costs and benefits. notably that of the UK’s Department for International Development (DFID 2001) and also the UK Treasury (2000). in Belli et al. 2001) as examples of “good practice” of risk analysis. an agriculture project. Actual experience with the use of risk analysis by major agencies is then considered within the context of a recent critique of project economic analysis by Harberger (1998). within this “livelihoods” framework (which is also increasingly being favored by World Bank) the need to assess vulnerability of target groups has actually increased the attention devoted to risk assessment. although not in a quantitative sense. DFID’s experience is notable for the purpose of the current study because that agency previously adopted a quite traditional approach to project economic analysis (including its technical approaches to risk analysis) but has now (along with most bilateral development agencies) shifted its analytical focus away from quantitative cost-benefit techniques and towards a more participant-centered and “pro-poor” driven “livelihoods” approach (within the context of pursuing poverty reduction as its overall objective). The experience of the World Bank is also considered in this regard. The experience of some other agencies with approaches to risk analysis is also considered. and the few power sector projects described below.30 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS were reviewed to find out to what extent those variables which were thought to have affected project impact during implementation were in fact those which had been identified as sources of risk or uncertainty at the time of project preparation and (if they had been so identified) to what extent any attempts had been made to model them as “risky”. ADB’s experience with quantitative risk analysis has been very limited.g. It will be seen that the actual practice of risk analysis falls somewhat short of what may appear to be advocated in its official publications. ADB Experience With Risk Analysis With the notable exception of a port project.

often designed to formalize and support the proposed mitigating measures which have been agreed upon • Appendix 1 of the RRP is invariably the ‘Project Framework’. etc. Sometimes such identified risks are called ‘micro’ or ‘project’ risks (i.. road and rail traffic volumes. they are risks over which the project has some control). suggest the following main points: • the standard format of an RRP invariably includes a section (usually within the ‘Financial and Economic Analysis’. A review of some recent RRPs. ‘Risks and Safeguards’. There is thus a ‘cushion’ which supports the likely acceptability of project returns • the ‘Financial and Economic Analysis’ section usually contains a fairly standardized approach to sensitivity testing. which identifies and summarizes the risks as previously discussed.. crop prices. but sometimes presented separately and called variously ‘Risks’. in which project aggregate ‘base case’ costs and benefits streams are changed by 10% or 20% each . upon which benefits have been estimated • in addition. however • the ‘Financial and Economic Analysis’ section of an RRP often argues that ‘conservative’ or ‘pessimistic’ estimates have been used for forecasts of (for example) shipping. control but which are not regarded as being significant enough to jeopardize project success) • this is often followed by a section called ‘Assurances’ which contains statements/letters to ADB from the borrower regarding operations during project implementation. yields and production. etc. and which cover all sectors of ADB project and policy lending operations. the provision of counterpart funding. and places them in the context of the project’s hierarchy of objectives. often the fact that certain benefits are identified but not quantified and valued in EIRR calculations is used to argue that the ‘base case’ EIRR is ‘conservative’ or ‘understates’ real returns.) describing qualitatively the risks which a project is expected to face • this text usually (but not always) includes descriptions of the measures which have already been incorporated to mitigate such risks. and are distinguished from ‘macro’ or ‘sector’ risks (over which the project has no. fisheries catches. Typical riskmitigation measures include the provision of technical assistance to strengthen institutions.CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 31 outcomes is indeed very well-recognized. or very limited. and also the inclusion of a number of different components being included within the project ambit. ‘Risks and Assurances’. There is no separate discussion here of risks’ likelihood and seriousness.e.

g.e. The output of the analysis is described in terms of the probability of the EIRR being more than acceptable (i. While ADB’s practice could be argued to be very strong. The analysis of this project involved the estimation of probability distributions of several types (triangular. references to ‘conservative’ traffic forecasts are combined with extensive sensitivity testing as ways of dealing with unknown future values. interestingly.. trapezoidal.32 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS and benefits are delayed by one or two years. in the ‘worst case’ scenario (when cost changes are highest. and the policy matrix will usually contain a column describing ‘actions planned’. this has been very much along the lines suggested by Harberger’s 1998 argument and in a similar fashion to the World Bank Mexican irrigation and power project examples (see below). and the effects of these on the project ‘base case’ EIRR are considered (both in isolation and in combination with one another). the Xiamen Port Project in 1997. In the case of program loans. qualitative discussions of risk are included in the text.. an agriculture project. etc. It also tends to closely link identified specific risks with those mitigating measures which are already included in project or program design. in excess of 12%) in 97% of samplings. is why such a probability-based analysis was undertaken in 1979 (when presumably access to computational resources was far more restricted than is the case in recent years) for a ports project and yet was not for later similar ports projects (e. Some studies are now more detailed. This format is remarkably similar across projects in all sectors. and. sometimes. the Belawan.e. it routinely exploits sensitivity testing to demonstrate ‘robustness’. benefits most reduced and/or delayed. i. the statement is (improbably) made that. What is perhaps interesting to ask.. The Bintulu port project was cited and extensively described in Johnson (1985). also in 1997). In the cases of two ADB power . In both these later cases.) its EIRR is still above 12%. it could not be said to incorporate quantitative risk analysis practice in any form. many of which deal with management of identified risks. uniform) for different elements of the costs and benefits streams and the construction of a probability distribution of the EIRR based on 300 samplings/ replications. Banjarmasin and Balikpapan Project. and calculate switching values and sensitivity indicators • the project is then usually described in terms of its ‘robustness’. ADB practice recognizes that risks exist at different levels of objectives achievement. given its ability to survive such adverse circumstances. ‘the project faces no risk’. The apparent exceptions to this conclusion are the Bintulu port project. Recent preparation of power projects in ADB have systematically employed risk analysis. Moreover. and several very recent power projects.

The results of the analysis presented features of a distribution of the expected values for the EIRRs of each of two project components (i. The analysis applied triangular distributions constructed around the mean for key variables. but did not present the probability of the overall EIRR being less than the opportunity cost of capital. rather than institutional. The main effort in terms of additional work lies in selecting key variables. below 10-12%) economic rates of return. The models also incorporated estimations of correlation between variables.000 times using “Risk Master” software.. The result is certainly a presentational improvement. whose upper and lower values were apparently based on subjective estimates of project designers. It is suggested by staff working in the power sector that it has been very useful during project design (typically at the fact-finding stage rather than at PPTA) to investigate concerns connected with one or two ‘key variables’ (typically elements of input or output prices). minimum and maximum values. In addition. A cumulative distribution function was also plotted in each case. .CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 33 projects1 for example.e..e. and a cumulative distribution function (CDF) was estimated for the PIR itself (see discussion in Part IV of the utility of this exercise). Each simulation was run 3. foreign exchange. irrigation). ensuring there is no correlation between them (or that its extent is fully understood). discrete probability distributions containing between three and five possible values (based on PPTA mission estimates) for five to six variables in each case (covering capital and other cost elements. the most recently-prepared of these projects (the Shen Da project) included a risk analysis following on from the calculation of the PIR. and designing a ‘best guess’ distribution for each variable. the square root of variance). copy of ‘”Risk Master” software). and usually argued to be an increase in ‘peace of mind’ regarding project robustness. together with estimates of standard deviation (i. 1 People’s Republic of China’s Windpower Development Project. indicating the probability of negative or less than acceptable (i. commissioning delays. as well as aggregate financial costs and benefits) were used to run a Monte Carlo simulation which generated expected values for project EIRR. rural roads..e. This approach to risk analysis is certainly pioneering within ADB at present (and in fact relies upon the use of a personal. willingness-to-pay (WTP) estimates. With familiarity of the sector and/or local conditions this exercise may be completed over the course of a few days. and People’s Republic of China’s Shen Da Power Transmission and Grid Rehabilitation Project. Another exception concerns the Infrastructure for Rural Productivity Enhancement Sector project in the Philippines which used the @Risk software to simulate the impact of changes in the values of key variables on the feasibility of proposed investments in rural roads and irrigation development. and running a simulation based on several thousand samplings will take only a few hours.

of impoverishment) was warranted. and technical staff involved in preparing such projects are likely to have just as much basis (deriving from both historical data and informed projections of the future) for estimation of probable outcomes as do ADB energy staff.g.. a review of ADB project performance across all sectors (based on sector synthesis reports) suggests that many events occur to project environments during implementation which are responsible for divergences between estimated economic outcomes (i. Indeed. the special evaluation study on involuntary resettlement (2000) concluded that social investigations at the PPTA stage had often been weak and/or hurried in preparation. and that greater attention to project participants’ risks (e. a 1997 special study of the macroeconomic environment and project performance (across all sectors and from 1980 to 1997) in Sri Lanka found that numerous variables such as world market prices. Power projects may be typically relatively large investments. a ‘Special Evaluation Study’ (1998) of factors affecting project performance in agriculture and social projects between 1991 and 1997 suggested that ‘greater realism’ was required at PPTA stage in estimating project costs and benefits so as not to consistently over-estimate EIRRs. the evidence suggests that there is a strong prima facie case for more rigorous risk analysis in ADB project economic analysis. but they are not necessarily more prone to risk from exogenous or endogenous sources than (for example) transport or irrigation projects.e. Adding to such findings. Again. these considerations could at least in part be tackled through greater application of risk analysis. various government controls. the nominal exchange rate. Similarly. and perhaps better project (re-) design may have resulted if risk analysis techniques had been applied. and that particular attention should be paid to dealing with aspects of local government performance in project preparation. . In addition to sector-based work. there would seem little reason as to why its use should be restricted mainly to ADB power projects. Overall. What is immediately clear is that most of the factors which have been so identified could have been subjected to some form of risk analysis. shipping costs. EIRR. What also emerges from the table is that other factors which cause differences in economic returns as estimated prior to and post-project implementation (typically depending upon how benefits—especially environmental ones or those based on willingness-to-pay – have been approached) could have been subject to greater variation quantification as well. scale of duties and tariffs.. Table 3 summarizes (by sector) what were the main technical factors causing differences between anticipated and actual outcomes. ENPV) as anticipated at appraisal and those estimated at post-evaluation and which in fact could have been subject to risk analysis at the time of project preparation.34 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Given that the estimation of risk in this way relies only upon a few estimates of likely values of several key variables and also some estimate of correlation among those variables.

health. could have been approached using sensitivity analysis. government commitment Price trends. implementation delays. WTP/resource cost savings estimates Implementation delays. by Sector OED Sector Synthesis Irrigation and rural development Rural and agricultural credit Fisheries Forestry Factors Affecting Estimates of Economic Outcomes Cost overruns. capacity and generation/ transmission losses. and/or “with or without” inclusion. traffic volumes. and land valuation could have been approached using sensitivity analysis. morbidity and mortality levels. poor water management Repayment rates. and perhaps risk analysis All conceptually subject to risk analysis Water supply and sanitation Cost overruns. cost overruns. and/ or “with or without” inclusion. cropping intensities.CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 35 Table 3 Sources of Differences Between Estimates of Economic Benefits. cost of services (participation rates and demand) Land values. debt amnesties. benefit estimation methodologies Roads and transport All conceptually subject to risk analysis Ports and shipping All conceptually subject to risk analysis . cost overruns. mills capacity and throughput Institutional performance. cost overruns.and Post-Implementation. traffic flows. and perhaps risk analysis All conceptually subject to risk analysis Many conceptually subject to risk analysis Industrial crops Health and education Urban development and housing Yields and prices. fish stock composition Benefit estimations and valuations Comments All conceptually subject to risk analysis Many (if not most) conceptually subject to risk analysis All conceptually subject to risk analysis Difficulties in estimating non-direct use values. numbers of vessels. untested technologies. rates of interest. and/ or “with or without” inclusion. Pre. benefit valuation methods Some conceptually subject to risk analysis. WTP estimation Power Cost overruns. WTP estimates could have been approached using sensitivity analysis. vehicle operating cost (VOC) savings Implementation delays. and perhaps risk analysis Some conceptually subject to risk analysis.

identifies three sources of risk (inadequate government counterpart funding. ADB 1997a). 1. In this publication. delays in surveys and studies. concluding that (risk analysis is perhaps of limited use in decision-making terms due to governments’ supposed risk-neutrality) the techniques are more likely to be of use in the processes of project design and redesign. rather than in decision-making per se. Two projects with risk analysis are given as examples of somewhat different types of “good practice”.. World Bank Experience With Risk Analysis The World Bank devotes relatively more of its basic publication on project economic analysis (Belli et al. the presentation of the material is relatively thorough. if somewhat traditional —being oriented mainly around the use of Monte Carlo techniques.1. An explicit conclusion of the study was that greater attention to analysis of these variables at project design might have improved project effectiveness. The first.e. from an irrigation project in Mexico. 0.e. These two impacts are each then divided into three scenarios (“optimistic”. World Bank accepts that in situations when projects are large with respect to a particular region or group of people the ENPV criterion alone (i.36 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS etc. had all affected project outcomes. There is also a discussion of the implications for decision-making of risk analysis. An implicit conclusion may also be that quantitative and/or qualitative risk analysis techniques could have been employed to consider such variables more carefully. without any associated measure of risk) is an inadequate measure of investment acceptability— but does not pursue the consequences of this thinking towards (for example) pro-poor project lending. . and specifically recognizes the fact that the kind of analysis and presentation of issues associated with risk which are advocated are “extremely rare in existing documents”. and unwillingness to invest/ problems with access to credit) which (collectively) have two sorts of impacts—implementation delays and low adoption rates.5. and delays of 0. and 2 years in the case of implementation) with estimated probabilities of occurrence (0. “modal”. “pessimistic” in the case of adoption rates. Overall. 2001) to the analysis of risk than does ADB in its comparable documentation (i.. a discussion of sensitivity analysis and its shortcomings is followed by a presentation of the principles of calculations based on mean-based expected values and the use of Monte Carlo simulation (including a hypothetical example). The problems of developing probability distributions for variables are discussed. and there is some description of judgmental methods applied to derive distributions in the absence of complete data sets.

which has a probability of occurrence of 0. project designers’ or participants’ ability to identify a few possible states for individual variables is likely to be plausible. three variables which affect returns (income differentials between graduates and non-graduates.org) for the same project. and appears to be the first World Bank education project for which a measure of net worth (as opposed to a cost-effectiveness approach) was used. 0. and the appropriateness of the particular techniques applied. This project was clearly innovative from the World Bank’s point of view as regards its economic analysis. which is discussed below. and could have been presented graphically as well. When considering this particular project material and the general nature of this type of approach (in contrast to the Mexico irrigation project.. for example).4) the calculated EIRR (of 12.CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 37 0. This sort of approach is very similar to that advocated by Harberger (1998). It is shown that even under the most pessimistic of assumptions (in this example this results from pessimistic adoption rates and a 2-year delay in implementation).e.7%) is still in excess of the opportunity cost of capital (assumed to be 12%). and each assigned probability distributions (log-normal. and EIRRs calculated for the resulting nine (i. 0. A full Monte Carlo simulation—using 3000 replications. In the actual Staff Appraisal Report (SAR—and available on www.1 (calculated from the multiplication of the likelihood of 0.35. employment rates of graduates. respectively). combinations of adoption rates and delays) with their associated probabilities and EIRRs.25. The result of this approach is a table of events (i. 3*3) possible different scenarios.25*0. A cumulative distribution function for the range of EIRRs is shown in tabular form. A resulting probability distribution of project benefits is shown in graphical form. and two different triangular forms. It is also the case that the risk .4. clear and can be undertaken with only very limited suppositions about the probability of certain states occurring. (2001). it should perhaps be noted that the risk analysis as described in project documentation involves considerable technical discussion of the nature of variables’ distributions.e. The other quoted example (“the most transparent and complete economic and risk analysis”) is from a technical and higher education project in Mauritius (prepared in 1995). enrolment rates) were identified.. Arithmetic calculations are trivial.4. different variables appear to have been analyzed (major cost and benefit items) using different sorts of distributions—truncated and non-truncated normal distributions and one uniform distribution. respectively). with and without correlation assumed between different variables—was ran at appraisal. and a probability distribution of net benefits was produced. 0. It is simple. and the resulting combinations of possible states is quite likely to be representative of the range of situations.worldbank. In the project’s economic analysis as presented in Belli et al.

four factors affecting project returns were modeled as having three possible states each.e. Exceptions (and . Danish Cooperation on Environment and Development (DANCED). a cumulative probability distribution of outcomes was constructed. 1995). examples of risk analysis from World Bank practice include (firstly) the $350 million Ghazi Barotha Hydropower Project in Pakistan (SAR.” Another project example is the Xiaolangdi Multipurpose Project (Stage II) in China (SAR. arguably much more typical. Other examples of World Bank practice in dealing with risk in various ways include the projects shown in the following table. The prevailing view among bilateral grant-based aid agencies. so that it could be said (for example) that “the probability of the project’s rate of return being less than the opportunity cost of capital is 8 percent. and EIRRs were calculated for each. One reason for this is obviously that such agencies are (by and large) disbursing grant or extremely soft funds. What is clearly similar to the experience within ADB is that it is projects in the power sector which are analyzed most fully in quantitative risk terms. with a focus on institutional and sustainability issues. often in relatively small amounts and do not have the same fiduciary accountability requirements as a bank such as ADB. AUSAid (Australian). (This process was undertaken for the project in the context of the power sector as a whole and for its own stand-alone operations).38 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS analysis of the Mauritius project itself apparently took four weeks of staff time (on the part of one summer intern. and FINNIDA (Finland) regarding preparation of project or sector loans reveals very little about general guidelines for economic analysis (in the sense of quantitative cost-benefit techniques). i. where probabilities of river flow volumes are discussed. as reflected in the available documented techniques for project preparation is that projects are assessed primarily in qualitative terms. Other. Other Agency Experience With Risk Analysis A scrutiny of recent literature published by bilateral agencies such as Danish International Development Assistance (DANIDA). As for the Mexican example. and then a sensitivity analysis (for delays and cost overruns) was supplemented by a risk analysis of the type used in the Mexico irrigation example.. GTZ (German). 1995)—a $430 million loan. although only appear to be modeled in terms of sensitivity analysis. or risk analysis in particular. It is also the case that operations by such agencies are increasingly in sectors where monetized costs and benefits are less obvious than historically may have been the case. where least-cost and economic net worth analysis was conducted. rather than on estimation of economic returns per se. presumably with a strong statistics background) to complete.

classified as short-term/medium and long-term/strategic. ‘sensitivity testing’ of basic assumed probabilities was also undertaken for several variables 5 ‘risk variables’ with 5–6 states each used to generate table describing expected EIRR with standard deviation. but not quantitative risk analysis. ‘risk analysis’ is really extended sensitivity testing Wide range of technical. and presented in matrix format with identified mitigating measures and parties responsible Wide range of technical. 1997) sometimes. financial. were used to generate probability distribution of EIRR. additions) to this approach are circumstances where there may be financial impacts at individual or household levels. etc. and presented in matrix format with identified mitigating measures and parties responsible Waigaoqiao Thermal Power Project (China. household. Its publication Project Appraisal of Projects In Developing Countries: A Guide for Economists (various editions 1988–1995) was a standard source text for technicians. while sensitivity testing is described at some length in this publication. the agency’s view was that . 1996) Shandong Environment Project (China. with 2 or 3 states each. and then crop (or farm. these are described qualitatively. etc. financial. classified as short-term/medium and long-term/strategic. yields and prices up or down 20%.) may be undertaken. minimum/ maximum values. 1997) Yunnan Environment Project (China. institutional and policy risks. (uses 1000 simulations in Risk Master software) Includes quantitative assessment of political-economy risk regarding implementation and impact of various reforms. 1997) Third Andhra Pradesh Irrigation Project India. etc. One agency which used to undertake comprehensive cost-benefit or (costeffectiveness) analysis of most of its projects is the UK DFID (formerly ODA). 1994) Features of Risk Analysis Four factors affecting performance.CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 39 Table 4 Examples of Risk Analysis from Recent World Bank Projects Project Title and Date Arun III Hydroelectric Project (India.g.) budgets may be modeled. However. 1997) National Drainage Program Project (Pakistan. these are described qualitatively. institutional and policy risks. In these situations it appears that some kinds of sensitivity testing (e.. probability of negative outcomes. but not measured probabilistically in terms of EIRR distribution Combinations of variables’ values used to estimate range of EIRRs. enterprise.

. and • compiling a “risk matrix” which locates each identified risk within a matrix whose dimensions are “probability of occurrence” and “seriousness of impact” (i. in recent years DFID (like ADB) has taken a very strong anti-poverty focus in its operations.. where consideration of uncertainty and risk in DFID/ODA practice was essentially seen as supplementary aspects of a quantitative cost-benefit analysis. shocks (e. This perhaps represents the last phase of thinking about quantitative cost-benefit techniques within the agency. and advocated greater use of tools such as probability/impact matrices (see below).. DFID’s approach has been far less oriented around the formal techniques of cost-benefit analysis than previously.40 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS “risk analysis (in the sense of probability-based techniques) is unlikely to be used often for economic appraisal except in the largest projects” (page 77. it certainly was a far from routinely-advocated practice. Since the publication of the 1997 “White Paper” on poverty..e. DFID is very rigorous in • identifying risks (right-hand column of the DFID format.g. natural disasters) and cultural factors (e. There are nowadays two major aspects of DFID’s approach to project preparation techniques that have direct bearing on the analysis of risk. Firstly. and more located within its “sustainable livelihoods” approach (see figure in Appendix 2). . “purpose-to-goal”) • clearly specifying what mitigating measures have been already within project design to address or minimize these risks.e. in crop prices). if the event does occur). and similar but not identical to that of ADB’s “Project Framework”) and specifying how they relate to relationships between different levels of the framework (i. One aspect of the “livelihoods” approach to promoting sustainable development is that it involves explicit assessment of households’ vulnerability to trends (e.g. as this work was abandoned in view of changed priorities..g. within the context of the application of the logical framework technique. although not in quantitative terms. compared to previous practice. An unpublished technical paper prepared within DFID in 1995 reviewed the status of risk analysis. and this has had major consequences for its approach to project appraisal. 1992 edition) and its emphasis even in such circumstances seems to have been on the capital and financial aspects of dealing with risk. and histograms to guide investment decision-making. decision trees. “output-to-purpose”. ethnicity). Secondly. While some examples of the application of the techniques of risk analysis within DFID/ODA may be found. This is effectively equivalent to raising the profile of incorporation of risk in project preparation work by putting it at the heart of strategic thinking.

technical appraisal.g.. that of the German agency GTZ) have used versions where the probability of a risk—as well as its existence—is indicated within the framework itself. Another UK agency which has been active in extending its analysis of risk (and which is probably the original source of the DFID approach) is the UK Treasury. economic and financial appraisal.CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 41 DFID is also notable for the preparation of a “risk annex” in its supporting project documentation. The following is an example of a DFID risk (or impact/probability) matrix from a recent project in China (Yunnan Environmental Development Programme. e. The identified risks comprised the following Risk 1: Chinese long term funding for poverty alleviation and environmental improvements in Yunnan is inadequate Risk 2: Inadequate state:provincial link established Risk 3: Economic development given higher political ranking than reduction of environmental degradation Risk 4: Abnormal incidence of physical shocks. If the probability of a situation occurring which would jeopardize the achievement of project objectives is thought to be sufficiently high. 2000). etc. which were identified during preparation and discussed within the project’s logical framework.. this may be regarded as a ‘killer assumption’. The risk annex and risk matrix are essentially used in a qualitative way to both promote dialogue between host governments and executing agencies and to ensure that appropriate mitigating measures are put in place. The numbers 1-10 refer to the 10 different sources of risk to the project. earthquakes Risk 5: High staff turnover leads to skills dissipation Risk 6: Sustainable development and the Agenda 21 process is not given sufficient priority by Yunnan Provincial Government Risk 7: Institutional practices limit the adoption of an integrated approach and subsequent co-ordination of activities Risk 8: Insufficient counterpart funding leads to sub-optimum choice of pilot projects Risk 9: Greater priority given to environmental benefits than to poverty reduction Risk 10: Institutional support for proposed gender improvements may not be adequate One other point with respect to risk analysis in the context of logical frameworks is that some variants of that technique (e.g. around which the project must either be scrapped or re-designed. in the context of the Private Sector Finance programs (where the private sector is . which gives it a similar status to that of social appraisal.

2. it should still be noted that none of them deal with risk proper. Based on the construction of an extensive risk matrix in this way— with its emphasis clearly on allocating and transferring risks—negotiations about contracts for the private sector to be involved in public service delivery or procurement are much clearer.e. making sure that (where possible) the ownership of risks and their consequences are transferred to the parties (public or private sector) most able to deal with them. “serious”. “marginal”. It is then the responsibility of respective parties to take action to mitigate risk. in the sense of quantitative. design and construction risks. occupancy and maintenance risks. probability- .. “critical”. through demand.42 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Figure 8 Risk Matrix: Impact and Probability Analysis Impact Probability: Low Medium High Low 4 8. 10 Medium 6 3 High 1. and “negligible”) • estimating the likelihood of risks (based on standard probability techniques). the risk-adjusted cost estimate for the project to be entirely a public sector provision and against which any private bids must be compared) is entirely apparent. and the basis for the public sector comparator (i. considerable effort is devoted to • identifying the full range of risks faced by a project (from planning. It is clearly the last of these functions which is the main purpose in the PFI context— i. such that those who can best manage the sources of risk should bear the consequences (good or bad) of such management. to legislative. 5.. Within the Treasury’s approach to PFI.e. Despite the innovativeness of these various techniques. and • allocating risks between different project participants. inflation and technology risks) • quantifying the impact of risks (based on the Construction Industry Research and Information Association classifications of “catastrophic”. 7. 9 invited to participate in what are traditionally regarded as public projects).

As a result of what Harberger calls such “ambient pressures”. and one that has definite implications for practice with regard to risk analysis. or (if they do include any probability-based constituents) they do not extend such techniques as already applied—at least on occasion—by (for example) World Bank or ADB. the relationship between what was expected at appraisal and what actually happened overall in development projects is relatively poor (the experience of the ADB projects reviewed in this Handbook largely confirms this view). for example. these lending agencies tend to take a less serious view of risk than would. a “comfortable scenario” (perhaps reinforced by organizational internal incentive structures) typically characterizes most project preparation environments. and partly because of systematic “approval culture” which is endemic to most multilateral organizations (and which tends to consistently overstate benefits and to underestimate costs of projects in a climate of “appraisal optimism”).e. The application of this technique would depend upon a few (he suggests four or five) values being chosen for the . private investors in similar circumstances.. As a result. which causes a divergence between expected values at time of appraisal and ultimate project impacts. such as in the US$ appreciation of the late 1990s). Harberger also argues that because loans made by agencies such as ADB and World Bank are in effect subject to countries’ sovereign guarantees and the banks’ capital is not effectively at risk. and some due to longer-term trends. such as unpredictable changes in petroleum prices. is that of Harberger (1998). estimation of ones more likely to represent real flows of costs and benefits) would be realized if more account were taken of risk—and in practice Harberger advocates the more widespread use of a simplified Monte Carlo simulation technique. Risk. This is partly because of the (perhaps unexpected) importance of large changes in relative prices over time (some being due to significant shocks. One of these. Harberger argues that better derivation of expected values at appraisal (i. One of Harberger’s main points is that there has been a lack of account of risk taken in project economic work in recent years. and the ‘Harberger’ Critique of Recent Applied Project Economic Analysis While reviewing the experience of ADB and other major multilateral and bilateral development agencies with respect to the analysis of risk in the context of project economic analysis. it is impossible to avoid some of the major critiques of observed technical practices which have appeared in recent years.CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 43 based techniques.

albeit in a simplified form. and the simulations re-run on the basis of only a few values for variables. This is in fact what is largely recommended to improve ADB practice. and the World Bank education . but does make a strong case for greater application of probability-based techniques. overestimation of benefits. Almost identical conclusions are drawn in relation to a review of recent World Bank practice (despite its much stronger emphasis on the utility of such techniques in its own published project economic analysis documentation). This approach can also be extended. plus fairly standardised sensitivity testing designed to demonstrate project robustness (survival in the ‘worst case’ scenario). almost the only exceptions to this situation appear to be found in the power sector. traded input price). despite the extensive academic literature describing the techniques for more quantitative approaches to risk analysis and the increasing availability of computers with which to run probability-based simulations for values of key variables.. actual examples of such practice are very rare. although the format and presentation of World Bank projects perhaps differ more across sectors (with a much larger number of projects being dealt with).g. as regards systematic underestimation of costs. Summary of Risk Analysis Experience What emerges from the review of ADB and other agency experience is that. weather. ADB experience as described in Part III appears to bear out at least some of Harberger’s criticisms (e. and an expected value for NPV being calculated based on assigned probabilities for each of these states. Within ADB. real wage growth. (This approach is therefore very similar to the World Bank’s “On-Farm and Minor Irrigation” project example mentioned earlier in Part III).e. is found across all sectors of operations. single) variable which is most critical to the project’s outcome” in most circumstances. so that even two or three other variables can be identified (which are in practice quite independent of one another – e. Within both institutions.44 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS “key (i. where simplified probability distributions for a few key variables are used to estimate expected returns plus estimated variability. A reliance on this. The cited ADB port and agriculture projects.g. etc.. the overwhelming orientation of current practice has been towards a qualitative risk description linked to mitigating actions included in the project design. Harberger offers no new techniques for risk analysis per se.)..

are ongoing themes of research work. Similarly. this can include better identification and definition of risk. and its allocation among various project participants.CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 45 project are most notable for the rarity of the application of full-scale probability techniques.e. perhaps the future challenge for ADB as regards risk analysis is therefore two-fold: • as a lending institution whose interest lies in seeing that its clients (i. and that lenders have not been rigorous enough in considering unknown future outcomes from the point of view of their borrowers (who ultimately bear the financial and economic consequences of such outcomes). Greater consideration of the expected vulnerability of poor groups in terms of the risks they face while earning their livelihoods. Although no new techniques for extending quantitative risk analysis are yet being proposed it is possible that existing techniques may be further extended in application. and institutional) of project effects and likelihood of project success. and • with the increasing emphasis on lending to reduce poverty. environmental. Critiques of some of the project economic analysis practices of major multilateral development agencies (based on reviews of project outcomes) include arguments that risk has not been sufficiently well dealt with in the past. other agencies (e. . Most bilateral agencies (such as UK DFID) are now undoubtedly moving away from quantitative cost-benefit analysis generally. borrowing governments) are not unduly exposed to economic risks. In these circumstances.. although it is also clear that certain qualitative techniques for dealing with risk and more centrally incorporating it within a pro-poor lending environment are emerging at the same time. there is scope to strengthen project design through the fuller use of risk analysis so as to ensure greater sustainability (financial.. there is a need to put the circumstances (especially as regards their vulnerability) and attitudes of those affected and targeted by projects at the heart of project economic analysis. the UK Treasury) are moving consideration of risk more towards the center of project analysis in an attempt to more properly allocate costs and benefits from risk management. and how they may best be planned for in such situations.g.

In addition. the more specific (i. the consequences of uncertain outcomes (when in fact they do turn out to be low or negative) can be disastrous for poor people. The traditional assumption about government neutrality toward risk being an appropriate basis for planning tends to break down.. concentrated on particular groups) the national project investment portfolio becomes. whose situation is often characterized by extreme vulnerability.e.Poverty Reduction Objectives and Risk Analysis Introduction This Part of the Handbook considers the relationships between poverty analysis and risk analysis. 4 .

a probability-based risk analysis of the projects produces the CDFs for the expected NPVs (or financial outcomes at household or enterprise level) of each project. argues that some knowledge of poor project participants’ attitudes is essential in making investment decisions.e. then anything which can be done to strengthen economic analysis of such projects through the incorporation of risk analysis techniques is likely to educate and inform donors. and Risk Aversion As suggested in Part II. as well as to lead to better projects. The likelihood of either of the respective projects producing negative or low outcomes can also be immediately assessed (project A does not generate negative outcomes. i. If this is true. unlike project B). therefore. Vulnerability.48 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS With ADB’s increasing emphasis on lending aimed at reducing poverty there is. Consider two projects—A and B. and considers what operational implications may follow from this for the analysis of projects’ distributional and poverty impacts in both quantitative and qualitative terms. Again. It can clearly be seen that project A is preferable to B.in that its distribution lies entirely to the right of B. vulnerable groups can be difficult to target development assistance towards (because of the effects of benefit leakage. Within the context of pro-poor lending generally. although the techniques of probability-based risk analysis may afford a good understanding of how the variability of outcomes is related to expectations of those outcomes. project A can be said to be “stochastically dominant”. In another situation (Figure 10) the considerations are more complex.. In Figure 9. this will only partially contribute to investment decision-making. in that it lies more to the right . A is said to be “stochastically dominant” (and ‘first degree’ so . at any particular level of probability its expected NPV is higher). and especially so in situations dealing with the poorest. Planning for the Poor. more rigorous investigation of the risks affecting project returns (both to individual types of participant and to the economy as a whole) from both the point of view of planners and decision-makers. a greater imperative to improve analysis of the situation of the poor with respect to uncertain outcomes from projects and policies. it is also the case that poor. for example) and therefore it may well be that pro-poor projects are inherently more risky than other projects. inter alia. This Part of the Handbook therefore summarizes the situation of the poor vis-à-vis risk. and also some consideration of the attitude of project participants themselves towards risks they may be expected to face. This involves. and thus the risk to the poorest is clearly identified.

CHAPTER 4 Poverty Reduction Objectives and Risk Analysis 49 Figure 9 CDF: Project Alternatives (1) Figure 10 CDF: Project Alternatives (2) .

.. and almost by definition. In addition.. as summarized in the DFID “livelihoods” figure in Appendix 2). A question for decision-makers would therefore be about what level of risk it might be thought proper to consider imposing on a target population (in this case upon very poor people) if the consequences of failure or negative outcomes in any one year could be potentially devastating for them. it implies that an understanding of attitudes to risk when considering . and staff training. loss of land to debt).g. and might well be characteristic of a situation in which new and more productive technology (e. equipment maintenance. and especially in the typical absence of insurance) to cope with bad (i. low producer price) crop seasons. for the poor in natural resource-based situations who have less financial reserves (if any. a proposed project) which involves the possibility of uncertain and/or negative outcomes for the poor is potentially disastrous for them—even if it would not necessarily be so for less poor populations.g. have a “focus of loss” which causes the slopes of their utility functions to become extremely steep in the wholly negative quadrant. Although estimating utility functions has proved very difficult in practice (e. and trends (e. it is now only ‘second degree’ dominant.g.50 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS of B than to the left. industrial) was being introduced by a project but its greater benefits depended upon (for example) water availability and management. property or soil loss through flooding) and/or nonreversible (e... Addressing this type of issue necessarily involves eliciting some kind of knowledge about poor people’s risk aversion within the context of a project preparation exercise. However. project A actually has greater probability of negative outcomes than project B (its CDF now crosses that of A). It is often suggested that people.g. If this is a reasonable depiction of poor populations’ attitudes to uncertain outcomes.g.. Poverty is usually associated with vulnerability to external shocks. etc. Despite its generally higher expected NPV.e.g. and • when the possibilities for losses are involved. agricultural. any situation (e. see Part II). For example. and especially poor people. empirical studies have tended to demonstrate that individuals generally become very risk averse indeed when • considering outcomes involving sums of money they are not used to dealing with. about which some implementation doubts existed. “is utility futility?”—a question which has been asked in the literature. even though it may exhibit the more normal diminishing marginal utility in the wholly positive one. the consequences of failure may actually be catastrophic (e. cultural factors... the poor are usually considered to be more risk averse than most sections of society. For these sorts of reasons. This situation is not uncommon.

• as in other investment decision situations. and also that • decisions about whether or not to accept such risks necessarily involve some knowledge about risk aversion among the target population. and can be specifically used to indicate how increased predictability may be achieved. and possible mitigating action taken (e. groups. and for whole projects) • the real-world consequences of any such possible outcomes for the lives of poor people can be considered. when considering potential investments affecting poor people. and also in relation to policy-based lending operations..g.CHAPTER 4 Poverty Reduction Objectives and Risk Analysis 51 Figure 11 ‘Focus of Loss’ Utility Function: Hypothetical Example alternative projects or project designs is essential in investment decision-making. The kind of considerations just discussed therefore implies that. safer/more robust projects designed) as a result of risk analysis. These sorts of conclusions have direct implications for how greater analysis of sources of risk to projects can be both quantitatively and qualitatively incorporated into existing ADB operations. probability-based risk analysis can usefully identify the relationships between expected project outcomes and their variability. although at particular levels of cost • such techniques can also be used to identify the likelihood of negative or very low outcomes (for individuals. especially when dealing with the poorest in society. especially in relation to distribution and poverty impact analysis. .

• calculating the proportion of total benefits going to the poor in each group (knowledge of the poor’s composition within each project group must be estimated). probability distributions applied to any variables (i.). social. and other data may be used from numerous sources to assist in this definition). the methodology for estimating poverty impact in this way involves • identifying financial and economic flows by groups of participants in a project (e. although it does require that financial and economic costs and benefits be disaggregated by participating group and that the proportion of poor in each group be identified (a wide range of income. there has necessarily been more of an analytical emphasis on measuring the distributional and povertyreducing characteristics of operations. health. “operating entity/company”. Calculation of the PIR involves no new data than that normally collected for a full financial and economic analysis of a project. • summing financial flows plus the differences between economic and financial flows (due to differences in prevailing economic prices arising through taxes. this figure is the estimated PIR). cost or benefit items) prior to a distribution/PIR calculation would lead not just to individual (i.. “consumers”. shadow wage rate. and therefore to an “expected” project PIR plus a measure of its variance. The methodology for distributional analysis and calculation of the PIR were originally contained in the Guidelines—Appendixes 25 and 26 (ADB 1997a).e. etc.. “point”) estimates of financial and economic benefits by groups but to “expected” values of benefits with associated estimates of their variance. .g. In principle of course.e. “government/rest of the economy”.52 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Poverty Reduction Objectives: Implications for Quantitative Risk Analysis With ADB’s increasing focus on lending to reduce poverty. and in particular the recommended use of distributional analysis and the calculation of the poverty impact ratio (PIR)..) accruing to each group. “farmers”. subsidies. and • summing the benefits accruing to the poor across all groups and expressing this as a proportion of total project benefits (i.. The increased use of such techniques has implications for how risk analysis can be used to design and assess ADB projects and policy interventions.e. In essence. etc. and have since been elaborated upon in Fujimura and Weiss (2000) and also in ADB (2001a). This approach has been undertaken once so far in Bank practice—in the Shen Da project described in Part III.

water authority. An example of this is given in Fujimura and Weiss (2000) where differential risk exposure between projectparticipating groups (i.g.. As yet.. the application of comprehensive probability-based risk to model the differential impact of project cost or benefit item variability between different groups as part of a full-scale distribution/PIR estimation has not been attempted in ADB practice. What the use of some type of risk analysis can probably and practically most usefully show in the context of poverty impact analysis is how likely it is that financial and/or economic returns may be very low or negative (i. however.). farmers. where the distributional analysis methodology (Adhikari and Weiss 1999) is simply a special case of general distribution analysis (only here applied to countries rather than groups within one country). etc. Part V considers how this principle can perhaps be operationalized in situations where costs and benefits are quantified and valued and ENPVs are calculated for project groups and for projects as a whole. but where the real level of project risk may be higher than for single country projects (e.e. processors.g. the case for increased use of risk analysis is therefore made stronger than hitherto. ENPV<0. government) is modeled. is that the extent of variability which might emerge from such an analysis would differ between project-participating groups to the extent to which they differentially incurred costs or received benefits whose estimation derived from those variables being subjected to risk analysis.. Again. .e. no examples of quantitative risk analysis have yet been undertaken for such projects. Quantitative risk analysis could also conceptually be employed in the case of subregional projects. farmers. because of coordination difficulties among countries and agencies. Given ADB’s increasing lending orientation towards the poor and the already described particular vulnerability of such poor populations in the face of risk exposure (as compared to national populations as a whole).CHAPTER 4 Poverty Reduction Objectives and Risk Analysis 53 What is also interesting following such a calculation.. EIRR<EOCC) for particular groups affected by the project—typically the poor (e.. exchange rate fluctuations. Poverty Reduction Objectives: Implications for Qualitative Risk Analysis In support of all projects’ textual poverty analyses (e. qualitative risk analysis routinely focuses on such issues as.g. for example. traders) who are its target population. in social assessments carried out at various stages) and also in situations where a PIR is not calculated for a particular project because its benefits cannot be reasonably estimated.

g. it should be possible for project planners to gather (in the application of quite intensive and participatory data-gathering exercises) some “subjective” information about how those expected to benefit from project or policy interventions may view choices among such options.g. etc... in the case of policy-based lending (PBL). based on socioeconomic status indicators) • general risks (e. project participants. imposition of user charges. (Policy-based lending is probably the most inherently uncertain of all lending types. such as small farmers.) and identifies the channels of their impact on the poor. and thus an implicit consideration of risk by potential project participants/beneficiaries is being undertaken. the full range of ‘what’ and ‘how’ mechanisms in a sector is likely to be much less clear). even where the scope for quantification may be limited. This kind of argument is perhaps particularly applicable in circumstances where new technologies are being introduced. as well as “objective” data which may be available from other sources upon which to estimate expected values and their variance for key variables affecting project outcomes. costs and benefits from various policy alternatives in an iterative way. the analysis concentrates. civil) which may compromise overall project success. Importantly.. Similarly. may well be generally enthusiastic about planting new high-yielding varieties of rice or wheat but some of them are unable to face the . on economic variables that change with particular policy adjustments (e. institutional. but that more could perhaps be done within such exercises to explicitly document the extent of risk aversion among such groups. and • risks of benefit leakage to non-poor groups. In other words. In these situations. the use of a poverty impact assessment (PIA) matrix is advocated to elicit the relationships and mechanisms between particular policy interventions and ultimate poverty impacts. as an input to planning for them (and which would be especially useful in situations of choice among various design alternatives). Within this matrix. inter alia. removal of subsidies. in that.g.54 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS • extent of ability of target population to cope with risk (e. the same paper suggests that use of the PIA matrix as a design tool in a participatory fashion may help lay out options. What emerges from consideration of the sorts of qualitative analysis of risk which is encapsulated in the above techniques is that project target population/ beneficiaries’ situations and views are being quite thoroughly canvassed. Recent consideration of how PBL and poverty reduction analysis can be more closely integrated (Bolt and Fujimura 2002) suggests that improvements to current procedures include greater use of statistical inference and risk analysis. while the ‘why’ of the program is likely to be well-understood.

. and it may well be the case that projects with higher expected benefits are also more risky than some with lower benefits.CHAPTER 4 Poverty Reduction Objectives and Risk Analysis 55 consequences of loss if a bad year occurs in project year 1 or 2. but its scope and intensity could be increased to include gaining an understanding of target groups’ attitude to risk. and to address poverty reduction objectives specifically.e. group and village discussions. Full-scale. and • the distribution of benefits (with poorer farmers given more opportunity to benefit from a redesigned intervention. but some may well be unwilling to accept only a 40%-50% expected increase. for example). . although it is most likely that practical applications of techniques will be limited to concentrating on describing the probability of negative returns at enterprise level. Quantitative risk analysis can make clear the implications of available project choices at household. Qualitative risk assessment based on participatory techniques (interviews. etc. farm. for example. Summary Projects have differing extents of risk attached to them. enterprise. Because poor people’s situations are typically characterized by extreme vulnerability. Understanding how farmers perceive such choices can influence • the technical (re-)design of the intervention itself (so that its expected returns are made more stable. exposure to very low or negative outcomes (in the acceptance of more variable project outcomes) can be catastrophic—and so there can be major issues of choice regarding appropriate levels of risk to expose target populations to in the introduction of (for example) new technologies and/or services. so that appropriate levels of risk can be incorporated in project and program design. depending upon uptake rates). for example–even though such an increase is clearly substantial. probability-based risk analysis techniques could conceptually be applied to strengthen existing distribution and poverty impact analyses of projects. higher. All may be happy to accept such a risk if the expected yield is double or treble current levels.) is part of existing project and policy-based lending. and project levels. or both) • the way project benefits (and costs) are calculated (i. The increased application of both sorts of risk analysis approaches should help to strengthen project design generally.

(Much of the material in this Part of the Handbook is separately presented in ERD Technical Note No. 5 . 2 on Integrating Risk into ADB’s Economic Analysis of Projects (2002).Strengthening Risk Analysis in ADB Operations Introduction This Part of the Handbook provides some practical guidance on how to strengthen the analysis of risk in ADB operations. based on the technical material and discussion contained in Parts I to IV. Its format is primarily designed to be of use to practitioners of economic and financial analysis of all types of projects financed by ADB.

risk mitigation in the broadest sense (i. both qualitative and quantitative. These are taken to be • firstly. Each project is unique. The next sections show how strengthened analysis of risk through applying such techniques could potentially contribute to various ADB operational objectives. what is fundamentally suggested in this Part of the Handbook is that a pragmatic approach to the use of risk analysis is warranted. including some points on applying the ‘@RISK’ package. contributing towards the greater achievement of poverty reduction objectives through strengthened risk analysis. . and the extent to which risk can be quantitatively dealt with will also vary. As a general rule.e. Nevertheless. contributing to a more specific focus on ensuring the sustainability of project effects—through strengthened financial.58 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS The first section briefly summarizes the key features of the various possible techniques for risk analysis. and • thirdly. and data issues will arise. the strengthened design of individual projects such that the probability of their having an EIRR<EOCC or ENPV<0 is minimized) • secondly. and similar types of risk analysis techniques will therefore be appropriate to use across a number of project types. This is followed by an outline of how risk analysis may contribute specifically (although in a fairly limited way) towards the formulation of policy/sector/program lending operations. It would not be appropriate to advocate hard and fast guidelines about application of particular risk analysis techniques to all projects. of benefit estimation. as within individual sectors and types of projects it is likely that similar technical/methodological. similar types of projects are likely to face similar analytical issues (e. The last section deals with various technical and resource implications of strengthened risk analysis being applied in practice. A consideration of some typical risk analysis situations sector-by-sector is then presented (again in tabular form).g. of sustainability).. and the sources of uncertainty and risk it faces will be similarly unique to its own individual circumstances. policy. Summary of Risk Analysis Techniques: Applications and Limitations In essence. environmental and institutional risk analyses..

• the more that risk analysis can be used to investigate the specific financial. The table classifies the techniques according to whether they are essentially qualitative or quantitative in nature. and • the more comprehensively the objective circumstances and subjective attitudes of poor project participants can be taken account of in project planning. the type and form of results that come from such analyses. These risk analysis techniques are of course likely to be applicable in different sorts of circumstances. the stronger will be overall project design (assuming mitigating measures are put in place once the scale and impacts of known risk are clear) and the likelihood of project failure (in the sense of EIRR<EOCC) will be reduced. the circumstances in which application of each of them may be likely. while the use of continuous probability distributions based on historical observations will probably remain relatively rare (for data-intensity reasons. the greater will be the likelihood of the sustainability of project effects over time. the greater chance there is of projects achieving poverty-reduction objectives. environmental and institutional aspects of project design. The suggested elaboration of risk analysis within the existing Project Framework and the construction of a risk matrix could be applied in any project situation. to subjective quantification of likelihood of event occurrence and seriousness of impact in that event. for example. It is also important to note that the use of the individual techniques are not mutually-exclusive. For example. . description of the nature of exposure to risk on the part of project participants and some estimation of their attitudes towards risk in particular circumstances. The following table (5a and 5b) summarizes the major approaches to risk analysis which have been covered in this Handbook.CHAPTER 5 Strengthening Risk Analysis in ADB Operations 59 • the greater the extent to which risk can be identified and quantified within the scope of routine project economic analysis. It outlines their main features. What is apparent from the table is that there are a number of techniques for dealing with risk in project design and analysis. if not software ones). and (finally) to probabilistic-based estimates of project returns depending on the behavior of key variables (such estimates may derive from more or less sophisticated and data-intensive techniques). They range from a spectrum of simple risk identification (and linking these risks with specific mitigating measures). the risk matrix technique can identify those risks that are thought to be the most serious and/or likely to occur so that they can then be further investigated through quantitative techniques. (It is also of course the case that such variables can be identified after sensitivity testing techniques have been applied). and also some possible constraints to their application.

and some of them try to quantify the extent of this risk.60 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS In essence. low) Allocation of risks among different project participants Poverty and Risk Vulnerability Assessment Assessment of the nature and extent of target group’s exposure to risk (catastrophic or not. it is only when some quantification has been achieved that the situation can be described as having modeled risk. reversible or not. ultimately a decision about whether to accept a project in the face of the simple known existence of a risk (or of a particular level of that risk). etc. insurable or not. It remains practically impossible to derive Table 5 Qualitative Risk Analysis Techniques Type of Risk Analysis Technique Logical framework ‘risks and assumptions’ elaboration Main Features Expansion of consideration of risks within existing ADB Project Framework Risk Matrix Construction (and ‘Risk Annex’ preparation) Construction of 3*3 (or more) cell matrix showing approximate probability of risk occurrence (high. medium. rather than simply identified a source of uncertainty). controllable at micro level or not.) Risk Aversion/’Focus of Loss’ Estimation Quantification of extent of target group’s attitude to risk (especially any risk or risk reduction implied by the proposed project) and especially towards possible losses of incomes . Whether quantified or not. all the techniques attempt to identify and describe risk. medium. (Properly of course. is a subjective decision for planners and policy-makers. low) against seriousness of impact (high.

concentrated in social and economic assessment during PPTA.. in agriculture) and negative outcomes may result . because policy-makers and planners (within ADB and borrowing governments) are typically planning for particular groups within society.e. Barest minimum of ‘risk’ analysis Any project Subjective assessment of risk exposure only.g. that most likely / most serious is not a ‘killer’) Responsibilities and rewards for managing different sorts of risks assigned to those agents best able to deal with them Part of Initial Social Assessment prior to PPTA.CHAPTER 5 Strengthening Risk Analysis in ADB Operations 61 any decision rules about the acceptability or otherwise of any particular investments. especially where it is desirable to estimate likely attitudes to uptake of new but risky technologies (e.. onstration that ‘killer’ risks have been dealt with (i. but especially Poverty Intervention ones Qualitative assessment only Supported by existing household survey data and also primary data collection Relatively demanding in terms of consultant/staff and interviewee time PBL New technology being introduced. with individual risks numbered and discussed (along with identified mitigating measures) in separate ‘Risk Annex’ to RRP Dem. Part of (modified) Poverty Impact Assessment Matrix in PBL. should show how proposed project will contribute to risk exposure reduction. Application of (for example) interview-based ELCE (equally-likely certain equivalent) technique to derive estimate of risk aversion over typical income levels of those affected by proposed project intervention Likely Applications Any project for which Project Framework is completed. limited classification of expected impacts Particularly applicable to those involving physical constructions Any project. it is useful to know something about those groups’ position with respect to exposure to risk and their attitudes towards any changes in that exposure which project interventions may imply in order to assess whether estimated levels of risk are acceptable to them or not. Minimum quantification of probability. Each identified risk described in more detail than at present and linked to at least one specific mitigation measure Risk Matrix. Type and Form of Results Textual summary of how each risk may prevent achievement of objectives at different levels of the project’s objective hierarchy. Possible Constraints / Limitations No quantitative assessment of risks’ likelihood or seriousness. However.

. WB Mexico irrigation examples) Main Features Indicates likelihood of project EIRR/ENPV being acceptable.62 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Table 5 Qualitative Risk Analysis Techniques (continued) Type of Risk Analysis Technique Simplified Probabilistic Analysis (e. ADB power. the same level of expected benefits may be found to be achievable with less risk or. if risk reduction also reduces expected benefits. Table 6 suggests some principles which can be applied to risk analysis for the purpose of overall project design. Harberger.g. Clarke and Low) Use of standard spreadsheet functions (to generate random numbers and counts of observations of key variables) to produce distribution of project outcomes ‘Monte Carlo’ Simulation with Continuous Distributions Classic risk analysis technique based on continuous distributions for key variables Strengthening Project Design: Overall Economic Benefits Estimation The primary utility of an analysis of risks faced by projects lies not in enabling choice among competing projects (as the orientation of much academic literature implies) but in the information this provides about the proposed project and its particular environment—such that consideration can be given as to how the project may be redesigned to reduce risks to an acceptable level. Ideally. . then the extent of that trade-off can be made clear to planners and/or beneficiaries. specifically in relation to the estimation of a project’s overall economic benefits.. based on consideration of key variables as determinants of project performance Spreadsheet-Based Applications (e.g.

there may also be doubt about the values which have been used in estimating the project’s economic costs and benefits (i. with measures of variance. likely to be used only in situations where risk analysis software is unavailable Estimate of expected EIRR/ENPV. availability of software (@RISK. Promoting the Sustainability of Project Effects The delivery of project effects over time depends upon sustainability being built into project design. experience may suggest that results add little to analysis over and above use of simplified distributions Estimate of expected EIRR/ENPV. such factors are included within existing sensitivity testing exercises. minimum/maximum values Likely Applications Any project where key variables can be identified and simplified. statistical complexities with co-variance. developing non-uniform distributions by writing formulae is complex Demanding in data and staff time. but there is of course no reason why they could not more routinely be subject to a simplified probabilistic analysis. RiskEase. plus CDF of project EIRR/ENPV. Sometimes.e. plus CDF of project EIRR/ENPV. plus CDF of project EIRR/ENPV.) Fairly extensive familiarity with EXCEL or LOTUS required. with measures of variance.. Following . etc. with measures of variance. minimum/maximum values No inherent advantages. and probability distributions constructed. or general conversion factors¾such as the standard conversion factor or shadow exchange rate factor).CHAPTER 5 Strengthening Risk Analysis in ADB Operations 63 Type and Form of Results Estimate of expected EIRR/ENPV. the derivation of specific conversion factors–such as the shadow wage rate factor. It is suggested here that risk analysis can specifically help with ensuring that ADB projects are made more sustainable in various ways. Possible Constraints / Limitations Results will only be as good as the distributions are realistic. minimum/maximum values Where historical/crosssectional data exist for key variables such that continuous distributions can be fitted In addition to modeling the technical variables which explain a project’s performance.

however. etc. include description of expected risks in first draft of Project Framework Construct ‘Risk Matrix’ for proposed project. etc. decide whether risk of EIRR<EOCC or ENPV<0 is ‘acceptable’ If extent of risk is regarded as ‘acceptable’ – redesign may not be necessary (but check individual distributions to see if any high values are ‘pulling up’ the expected value of the distribution..64 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Table 6 Principles in Applying Risk Analysis in Project Design Principles to Apply 1 2 3 Identify any risks facing the proposed project as soon as possible (i.. quantities. and institutional—and these can be approached in different ways through risk analysis. expert-Delphi. ‘subjective’ ‘best guesses’.g. the emphasis remains on considering and modeling sources of risk and designing mitigating actions to reduce that risk if its level is considered unacceptable. ‘objective’ historical or forecasts.) to generate CDF of expected EIRR/ENPV. output mixtures. if so. investigate any negative skewness. ranking risks according to their relative likelihood of occurrence and their expected scale of impact Identify ‘key’ variables (e. etc. perform probability-based analysis using such distributions On the basis of results from 7 and 8. Consider whether the derivation of distributions from primary/empirical sources is justified. using @RISK. output prices.e.. price and income elasticities of demand.. uptake/ adoption rates. see if positive skewness is occurring which causes average values to be substantially higher than the most likely values) If extent of risk is regarded as ‘not acceptable’ – possible redesign (in particular. and attempt re-design to truncate distribution) 4 5 6 7 8 9 10 11 the Guidelines.. RiskEase.e.) which are sources of risk and determinants of project returns Decide which of these variables may be subject to quantitative description Identify data sources for each variable (i. In all aspects. in particular. i. etc.) Construct probability distributions of key variables Perform simplified probabilistic analysis (e. environmental. the notion of sustainability has at least separate dimensions—financial. unit costs.e. check to see what can be done about any low values in distributions. pre-PPTA).g. . minimum/maximum expected values.

see Financial Guidelines. a water supply and drainage authority). a municipality. In some projects. In other situations they will be project executing agencies managing or providing technical or consumer services (e.4. Following standard international banking practice. Earnings and Liquidity’ . industrial development banks. In all cases. value at risk (VaR). credit unions. and is supposed to measure “over a 10-day period. a commercial bus company. maturity risk. a major concern of their appraisal and consideration for participation in an ADB project is their situation with respect to risk. a number of standard measures for credit risk (borrower default). contagion risk.. This is especially true for the VaR. etc. typically based on a mixture of expert judgment and some forecast data. the institutions under consideration will be financial institutions proper (e. and as summarized in the Financial Guidelines. Assets Quality. nongovernment organization-run operations. Risk analysis can be used to assist in designing projects so that there is less chance of this occurring. what is dollar amount of V such that there is only a 1% probability that a portfolio will lose more than V?” (Financial Guidelines. state-owned or commercial agriculture banks.g. They generally involve some subjective estimate being made by financial analysts/PPTA teams about the probabilities of specific outcomes. lack of financial sustainability will compromise the delivery of project effects to beneficiaries. section 6. It is the case that many of the essentially ‘static’ measures under CAMEL for assessing financial performance of whatever type of lending . It is suggested that in many cases it would be possible to extend this analysis for at least some measures of risk to be based on probability distributions. which is theoretically based on forecast values (which could be presented in probabilistic terms). Management Quality.1) to analysis of microfinance institutions—particularly relevant in the context of increased lending to the poorest.4.CHAPTER 5 Strengthening Risk Analysis in ADB Operations 65 Financial sustainability risk analysis The financial sustainability of institutions is important in most project situations.4. and is still useful even if the loan is treated as risk-free. 6.g. a state-owned plantation. ACCION International (an American nongovernment organization) has extended the CAMEL framework (‘Capital Adequacy.. housing banks). can be derived. foreign exchange risk.3.2) The calculation of VaR is essential where loans are from ADB to a nongovernment-guaranteed FI. either by causing liquidity to dry up or for service provision to be suspended and/ or curtailed. For financial institutions such as banks.

‘costs up 10%.g. and overall FIRR and FNPV estimates. it may be recalled that the ‘Harberger’ critique of project analysis practice of major multilateral agencies included the view that not only were estimates of project costs typically understated and estimates of project benefits typically overstated. ‘revenues down 10%.. although they are primarily point value or logical (i. other techniques are available to assess the performance of such institutions according to a whole range of technical and management criteria. 20%’..66 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS institution. etc.. wages) and exchange rate forecasts frequently turned out to be wildly inaccurate. cash flow.) is conducted in exactly the same way as for project EIRR/ENPV estimates. The greater use of quantitative risk analysis techniques. The French Development Agency (ADF) has experience in applying such techniques to developing country financial institutions. are presently calculated as individual ‘point’ or ‘average’ estimates. on loans to sub-borrowers). In addition to these strictly financial performance risk measures.g. services. some of the variables (e. In this respect. and • repayment rates (e. ‘Yes’ or ‘No’) indicator-based at present. 20%’.e.g. the primary concern however is typically with earnings performance. These kinds of variables are just as amenable (and probably even more so) to probabilistic-based forecasting as any more ‘technical’ variables affecting a project’s performance. and then standard sensitivity testing (i. it is quite possible to model probabilistically (for example) such financial estimates as • current and constant terms price projections for inputs and outputs (i. for at least some of these essentially financial variables. Typically.. but also that price estimates (e.e. What is quite apparent in this respect is of course that the modeling of risk to financial projections of project institutions is just as applicable as it is to estimates of economic benefits for the project as a whole. for commodities. ‘revenues delayed by 1 or 2 years’. and.. for likelihood of meeting certain targets by particular dates) could be subject to probability techniques. For project operating entities and executing agencies. probabilities of current price streams and also probabilities of particular inflation rates applying) • foreign exchange rate projections (relative appreciation and depreciation during the life of the project) • interest rates (e.. and could usefully be turned into financial forecasts if based on distributions of variables. from sub-borrowers). financial statements for such entities are prepared. would be of use in ensuring project financial sustainability..g. Instead of individual point estimates plus sensitivity testing. .e.

from soil conservation) are quoted as the basis for benefit calculation—i. etc. in a manner identical to the standard treatment of other economic benefits. Typically.. i. Biases and Uncertainties’ in such estimates is provided—along the lines of practice recommended in ADB (1997b)— which simply indicates possible biases in the estimates for particular environmental impacts and shows what the impact on the project EIRR might be if these were corrected.. investigating and estimating the economic value of environmental costs and benefits is difficult.e. this is because • biophysical relationships tend to be complicated (and thus hard to entirely capture through sampling and statistical techniques) and data are usually external to that collected during normal project preparation processes. Sometimes. and • the relevance and applicability of costs/benefits transferred from secondary sources (i. and • different techniques for economic valuation may well lead to different results. In addition. Typical ADB practice has been that where environmental benefits have been valued at all. In general. however.. it is impossible to attach probability distributions to particular outcomes.. the benefits transfer method).g. conservative or pessimistic forecasts are used for description of the base case. depending upon factors such as • how familiar the analyst(s) is with the biophysical circumstances under consideration.CHAPTER 5 Strengthening Risk Analysis in ADB Operations 67 Environmental sustainability risk analysis Another major area in which the sustainability of project effects needs to be ensured is that of impacts on the environment. In many situations. various biases (often stemming from professional perceptions and backgrounds) may arise in determining which impacts are important or will be large. it may be possible to extend this sort of analysis such that instead of simple ‘uncertainty’ being reflected. contingent valuation. the situation with respect to valuing environmental impacts is traditionally thought to be characterized more by uncertainty than risk.e. travel cost methods.. sensitivity testing is performed on such estimates where they are included in the base case EIRR calculation—i.e. Environmental sustainability of projects is ensured to the extent that environmental costs and benefits are included within the project’s economic analysis. it would be possible to model the expected benefits in terms of their risk. The scope for the application of such techniques will vary from situation to situation. ‘lower bounds’ from estimates of ‘expected values’ (e. • the quality of primary data collected from such techniques such as hedonic pricing.e.. Occasionally a table summarizing ‘Omissions. • market prices for many factors and project outputs do not exist. from carbon sequestration. .

It may also be true that as experience with primary data collection techniques for benefit valuation within Asia expands.. The Delphic approach involves asking a group of experts (in this case managers from water agencies) to assign probabilities to particular outcomes (in this case particular reform levels). This is conceptually the dimension of sustainability which is most difficult to capture quantitatively. the project institution(s) as located within a political/policy/sector context) and internal factors (i. definition of reform performance levels. some recent literature suggests that institutional performance and its risk may reasonably be quantitatively estimated. or at the very least survive in less than benign environments. The key stages included evaluation of who would win and lose from the reforms. Institutional sustainability risk analysis The last dimension of sustainability traditionally considered in project economic analysis is that of institutional sustainability. 1997) applied a multi-stage methodology to analyze quantitatively the probability of reforms (involving the shift of policy and decision-making responsibilities away from federal and state-administered agencies and towards decentralized autonomous public utilities and end-users/water groups—as such they could expect to be contentious) succeeding or not. greater knowledge about such estimates’ variability will be built-up—enabling moves towards the fuller modeling of risk rather than the simplistic description of uncertainty. and as more examples of benefits transfer are applied.. One major concern which affects many projects (as well as most policy-based loans) is to what extent project institutions will be able to implement policy changes which are critical to project success. identification of the various ways agents would seek to influence reform implementation (and the costs of such ways). Its advantage is that it provides direct assessment of risks from a collection of subjective but knowledgeable individuals and does not depend upon use of proxy measures—it .e. and few examples of quantitative benefit estimation or risk analysis from institutional performance exist. However. A recent study of the risks associated with institutional reform in Pakistan’s water sector (Dinal et al. Institutional sustainability is usually considered in terms of both external factors (i.e. and (lastly) application of a Delphic approach to estimate probabilities of level of achievement of each reform.68 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS The utility of attempting to apply quantitative risk techniques to such situations will vary according to the relative importance of environmental impacts within overall benefits (and costs) streams—for natural resource management projects (with often large but essentially unvalued benefits) the implications of such techniques may well be significant.). does the institution have sufficient resources to complete its tasks? is there enough technical assistance provided? etc.

distribution analysis (i. These involve not only considering the distribution of financial and economic outcomes at individual. given the context of their vulnerability. The following table therefore summarizes how these various techniques can be employed to support poverty reduction. ideally. Supporting Poverty Reduction Objectives In addition to more fully considering the likely distribution of overall economic benefits and also ensuring that various aspects of sustainability are built into project design. Again. the primary focus of this sort of poverty analysis is not as an add-on to eventual project description. etc. Risk Analysis and Policy-Based Lending Policy-based lending is probably the area of ADB operations least disposed to techniques of risk analysis. level—in a way which is identical to describing project economic benefits and enterprise/financial institution profitability. it can be seen that they attempt to marry project participants’ subjective circumstance and attitudes to risk with typical probability-based risk descriptions. PBL tends to be characterized by the existence of a clear economic rationale for the intervention (i.. the ‘why’ of the intervention is clear—and usually some kind of reform is envisaged within a particular .CHAPTER 5 Strengthening Risk Analysis in ADB Operations 69 could also be repeatedly performed throughout project implementation to monitor change.e.. farm. but also considering what target groups’ attitudes towards risk are. it was suggested that risk analysis could be useful in ensuring that poverty reduction objectives were better targeted. household. should emerge from a Delphic-based analysis of institutional performance is therefore the best-possible guess from knowledgeable locals about the institutional environment and the probabilities of particular outcomes expressed in a quantitative form.. analysis of benefits by groups participating in the project) can also be approached in terms of risk analysis. In addition. but should be used as early as possible in project preparation so that re-design can take place to more closely pursue ADB’s poverty reduction objectives.e. What. In contrast to project economic analysis.

in relation to the proportion of the share of the poor in total population) should be provided. 3 Estimate income/welfare impacts at individual/household/farm level: Estimate the distribution of individual.). ability for households to have any control. the PIR can be calculated and so can its distribution (as long as its estimation is directly linked within the same spreadsheet as the rest of the project economic analysis). possibilities for insurance.. and in particular investigate ‘focus of loss’ for project situations (e. with the focus on the likelihood that returns may be negative or unacceptable. (This analysis should include possibilities that ‘benefits leakage’ will occur). Poverty Impact Assessment Matrix (in PBL). justify the project design in terms of its level of risk implied for the project. but would perhaps be unacceptable to impose upon very poor communities in degraded watersheds. . 2 Estimate risk aversion/focus of loss for participants: Estimate quantitatively to the extent possible participants’ attitude to risks at different income levels.. 4 Calculate distribution of poverty impact ratio (PIR): Based on the calculation of financial and economic benefits and their distribution between groups. as part of Initial Social Assessments. This is likely to differ across project situations. household or farm incomes (based on probabilistic analysis of output quantities and prices.70 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Table 7 Application of Risk Techniques for Poverty Analysis Risk Analysis Technique 1 Describe textually nature and extent of vulnerability: This can be approached in terms of consequences of loss (catastrophic or not). etc.g.. A consideration of the likelihood that the project PIR may be below an acceptable level (i. a 25% chance of negative returns for farmers on an irrigation scheme may be acceptable if their resource base is relatively stable. 5 Justify the imposition/acceptance of any particular level of risk: On the basis of steps 1-4. These estimates can be derived through interview-based techniques offering participants choices between specific but certain outcomes on the one hand compared to higher but risky outcomes on the other.e. for example. reversibility. etc. where new technology is being introduced) where possible very low or negative outcomes may have to be considered.

.e. and therefore the analysis of risk in that project can be as unique as each individual proposed project itself. farms. However. some of the techniques already described for institutional and ‘subjective’ poverty analysis can be applied with PBL analysis. Any proposed project may be able to show.. transport. and so the kinds of approaches to risk analysis already suggested above to address such issues could equally apply to water supply. ENPV>0) depending upon values for certain key variables. and institutional sustainability. EIRR>EOCC. agriculture.. environmental. and likely attitudes towards risk by target groups • techniques for quantifying the probabilities of particular levels of institutional performance be considered (as in Dinal et al. It is also recommended that the approaches suggested previously to link any identified risk to specific mitigation measures be followed in PBL. therefore. in many situations of PBL..CHAPTER 5 Strengthening Risk Analysis in ADB Operations 71 sector) although the actual mechanisms and processes by which impacts on particular groups are delivered (i. case studies of individuals. 1997) in circumstances where major reform interventions are proposed. . households. projects. etc. it is suggested that • as the modified Poverty Impact Assessment matrix lays greater emphasis on the use of inference. or that its expected cost-effectiveness is similarly dependent on unknown but probabilistically described outcomes. are fairly typical ‘technical’ issues as they occur across different sectors and as are frequently faced by analysts. Sectors and Projects: Some Typical Risk Analysis Situations Each project design will encompass different sets of variables. how its expected EIRR/ENPV has a particular probability of being acceptable (i. for example. the ‘what’ of the intervention) are typically less clear. power. Quantitative relationships between individual variables and policy-based lending outcomes are not usually examined in PBL.e. etc. Specifically. It is therefore arguable that policy-based lending is inherently ‘uncertain’. It is also the case that many projects will share similar overall concerns to ensure financial. and statistical data for examining risks to which participants are exposed. many of whose actual outcomes will be unknown. What may be left to consider. and for this sort of reason the application of quantitative risk techniques is limited. interview-data. can be used to explain typical risk exposure and consequences.

extent of generated traffic and VOC savings Transport: Highway/Toll Roads Construction costs.g. poor maintenance of equipment.72 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Table 8 Project Types and Some Possible Risk Analysis Considerations Sector/Project Type Agriculture: Plantation/Estate Examples of Likely Analytical Concerns Realized tree crop yields and production. determinants of fishermen’s incomes Fisheries Environment and Natural Resources: Various Extent of identification. future prices as determinants of farmers’ and/or estates’ incomes Agriculture: Irrigation Scheme maintenance. pulp/wood) at that point Impact of new culture technologies from aquaculture. consumer price elasticity of demand Costs of inputs. operating costs Operating costs. realized new and existing crop yields. traffic composition mixtures. non-use and option impacts of total economic value (TEV) Transport: Rural Roads Construction costs in difficult or unknown environment.. currency depreciation for loan repayment. factory/mill throughput. adoption/uptake rates. price elasticity of demand for new road use. fish prices. and price of output (e. consumer demands for power . household and farm incomes Forestry Volume of harvestable wood in 7-20 years time. extent of maintenance. quantification and valuation of indirect. crop prices. future stocks and landings from capture. sustainability of road authority Transport: Railways and Ports/Shipping Energy: Rural Electrification Energy: Power Generation/Transmission Future passenger and/or freight volumes.

traffic forecasts modeled with several scenarios and associated probabilities. operating efficiency. adoption/uptake of new varieties Wood and by-product yields. alternative methodologies for benefit estimation Construction cost estimates. ADB/WB power) Operating/water supply costs. VOCs less well-known .. tree crop yield estimates. based on both primary and secondary techniques – consider wide range of possible values Construction costs likely to be reasonably well-known from similar projects in the same country. WTP demand estimates and foreign exchange projections can be modeled with simplified probability distributions (see material on ‘financial sustainability’) Costs based on simplified distribution estimate. more ‘subjective’ estimates for locally-consumed items. machinery operating capacity/efficiency Possible Variable and Data Characteristics Use of World Bank commodity price projections for exports. consumer demands . VOCs Contractor’s / analysts’ estimates allow for several states of costs. WTP estimates for water demand. harvest efficiency. foreign exchange projections Costs estimates. WTP estimates from interviews with target groups and extent of doubt about this can be derived at same time. current real prices plus considerable variation should be considered Data on yields from new technologies may be from research only – possibly exclude extreme values. for tuna) or ‘guesses’ for local varieties Knowledge of the extents of physical impacts may be reasonably well-known. etc. yields and prices (as above). fish stocks well modeled but sometimes highly mobile.. but estimates of impacts’ economic value can vary widely.CHAPTER 5 Strengthening Risk Analysis in ADB Operations 73 Potential Key Variables To Investigate Price projections. price estimates may be based on comprehensive data for commodities (e. distribution of WTP estimates can be derived at same time as averages All subject to simplified probability distribution analysis (e.g. as determinants of production in future periods Harvest yields and fish stocks. losses to theft. input prices.but triangular distribution as a minimum Construction costs likely to be reasonably well-known from similar projects in the same country. traffic volumes by types of vehicles. forecasts of traffic demands can be modeled continuously if necessary Costs based on simplified distribution estimate.g. machinery estimates based on design characteristics plus ‘subjective’ experience Cost estimates derived from similar schemes. passenger and freight forecasts Capital and operating costs. price elasticity of demand for road use. consumers’ demand schedules Costs of equipment. adoption rates can be modeled with triangular distribution as a minimum Considerable doubt about point estimates of volumes and prices when wood is harvested a long time into the future. yield estimates for new crops may be based only on research trials and need some adjustment. commodity price projections and local variety Quantities of particular biophysical impacts.

it is possible to draw out the following points related to technical and resource issues from the foregoing analysis: • any extended application of quantitative (i. value to consumers. following application of typical risk analysis software) are not to be generated. Again. because the application of risk analysis is likely to vary greatly in nature and extent across projects. willingness of authorities to pursue policy reforms (e. However. Health: Primary Care Education: Secondary and Post-secondary Education: Teacher Training Examples of Likely Analytical Concerns Construction costs.. it is difficult to develop firm conclusions about what may be needed to support any desired expansion of current practice.g.. probability-based) risk analysis will require expansion of most project analysts’ statistical skills if errors in the interpretation of results (i. The table is not exhaustive in its content. etc.. Technical and Resource Considerations (including applying ‘@RISK’) The last issue to consider concerns possible implications in technical resource terms for extending the analysis of risk in ADB operations. charges for service provision) Service uptake rates.e. it is meant to be indicative and general only. extent of cost recovery from rural poor. benefit estimation methodology (if applied in EIRR calculation) Nature of beneficiaries’ ultimate employment and the income differentials arising from such employment Numbers ultimately failing to find or accept work as teachers after training The following table attempts to indicate some of these typical project economic analysis technical concerns on a sector-by-sector/project type basis.74 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Table 8 Project Types and Some Possible Risk Analysis Considerations (continued) Sector / Project Type Urban: Water Supply and Sanitation/ Wastewater /Solid Waste. and to indicate how risk analysis could be applied to consideration of some key variables for such projects.e. This means that some kind of essentially technical/statistical support needs to be made available within ADB .

this results in ‘output’ cells (e. It can be applied to any existing spreadsheets. to do with expected correlations between typical determinant variables) or expected distribution characteristics for particular variables • it may also be possible to develop models for expected distributions of cost items across sectors (increasing evidence suggests that capital costs estimates for projects across a range of sectors may be log-normally distributed. DALY (disability-adjusted life year). modeling of employment through discrete or continuous distributions • in such a context. for example. on-going institutional changes. estimated WTP Employment rates.. the use of some kind of dedicated risk modeling software will be appropriate..g. probability of success of implementing institutional reforms Use of services and consumer demand/ ability to pay. discrete probability distributions or fuller continuous distributions to generate estimates of expected EIRR/ENPV and their associated variance. EIRR or ENPV estimates) having distributions generated for .g. quantitative institutional reform analysis may be considered Economic values. and primarily involves the substitution of point values in cells by user-specified distributions (in ‘input’ cells). employment rates Possible Variable and Data Characteristics Distribution of WTP estimates can be derived at same time as averages/point estimates. it may be useful to develop some ‘typical’ project-based or sector-based models of anticipated statistical issues (e.CHAPTER 5 Strengthening Risk Analysis in ADB Operations 75 Potential Key Variables To Investigate WTP estimates. While standard spreadsheets can theoretically be used for such purposes. for example) • for the construction and application of either simplified. There is no reason for analysts to try to develop their own analytical tools when ‘off-theshelf’ solutions are now widely available • dedicated risk analysis packages such as @RISK have far greater functionality and ease of use than spreadsheets • the use of @RISK is extremely simple. practitioners must be prepared to either use only uniform distributions and/or develop their own distributions through complex formulae application. may be contentious Can be extensively modeled with continuous distributions if necessary (see WB Mauritius example) Institutional aspects can potentially be investigated quantitatively. income differentials Policies such as school construction/funding programs.

it can also be used to directly generate distributions for measures of distribution and poverty impact (i. PIR). For this reason. and users can specify central tendency. the major issue involved in applying @RISK (or any such product) will be in correctly specifying distributions for existing or forecast data and determining appropriate extents of covariance among variables such that results for EIRR/ENPV distributions will be meaningful.e. if such calculations are in cells (which @RISK will designate as ‘outputs’) which depend upon variables for which distributions have been substituted for point values. the poverty impact ratio. including standard Monte Carlo random sampling and the stratified Latin Hypercube sampling the typical time taken to apply @RISK in this way is very short – typically a couple of hours for the tasks specified above.. rubbish out’ still applies @RISK will most typically be applied to demonstrate the probability that project EIRR and/or ENPV will be unacceptable.76 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS • • • • • • • • • them. once some form of spreadsheet model for EIRR/PIR estimation has been set up for most practitioners. and make the application of @RISK quite possible even for any or all variables affecting project outcomes there are a wide range of distribution types to choose from. several thousand simulations can be processed in minutes using @RISK on a fairly standard personal computer (PC) alternative sampling methods for these simulations can be used. dispersion and cut-off characteristics where appropriate. while it is easy to quickly generate attractive and precise outputs from such software. However. the general adage applied to the use of powerful computer programs of ‘rubbish in. all specified distributions can be represented graphically distributions of various forms can easily be fitted from existing historical or time-series data. it is good practice in project economic analysis to ensure that the PIR calculation (where undertaken) is seamlessly linked to the spreadsheet containing the EIRR/ENPV calculations . which can then be represented and analyzed both numerically and graphically the usual considerations of designing spreadsheets with as many individual variables specified separately therefore apply. and several alternative and complementary measures of ‘goodness of fit’ are provided users can specify estimated covariance between variables in the form of an easy to use ‘correlation matrix’ typically.

requirements for analysis of risk should be identified prior to PPTA and included in the PPTA scope of work.). in the same way that (for example) COSTAB is specified for financial cost estimation • because @RISK is so easy to use. specifically to investigate which variables are key determinants of project outcomes and about which more data describing such variables’ distributions may be collected • the undertaking of some form of risk-based analysis in the early stages of project design and the presentation of risk analysis results in a PPTA report would probably take only a few days work for an economist and/ or other staff. . it should be applied very early in project design. the analysis of risk can be expected to figure larger than in other situations. For this reason.CHAPTER 5 Strengthening Risk Analysis in ADB Operations 77 • for projects which are expected to be very large. involve several countries..g. the use of a dedicated risk analysis package such as @RISK should be specified. perhaps because they are new sorts of lending. etc. involve new technologies. marginal or particularly uncertain (e.

APPENDIX 1 List of Reviewed ADB Projects and Other ADB Project Evaluation Documents 79 APPENDIX List of Reviewed ADB Projects and Other ADB Project Evaluation Documents 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 1698 1779 1814 1404 1656 1770 1304 1515 1643 1552 1639 1781 1592 1753 1788 1524 1772 1644 1732 1809 1818 1548 1901 1715 1743 1735 1813 1554 1637 1718 THA KAZ PRC VIE PNG INO PRC VIE BAN SRI SRI VIE KAZ CAM LAO BAN PHI PRC NEP PAK PRC MON PRC PRC MON THA IND KGZ MLD VIE Agriculture Sector Program Farm Restructuring Sector Development Program (Policy) West Henan Agricultural Development Fisheries Infrastructure Improvement Fisheries Development Marine and Coastal Resources Management Yunnan-Simao Forestation and Sustainable Wood Utilization Forestry Sector Sundarbans Biodiversity Conservation Second Perennial Crops Development Tea Development Tea and Fruit Development Water Resources Management and Land Improvement Stung Chinit Irrigation and Rural Infrastructure Decentralized Irrigation Development and Management Sector Participatory Livestock Development Infrastructure for Rural Productivity Enhancement Sector Yunnan Dachaoshan Power Transmission Rural Electrification. Distribution and Transmission Capacity Enhancement in the Energy Sector Wind Power Development Ulaanbaatar Heat Efficiency Shen Da Transmission Interconnection Shanxi Environment Improvement Second Financial Sector Reform Program Restructuring of Specialized Financial Institutions Calcutta Environmental Improvement Education Sector Development Program Postsecondary Education Development Teacher Training 23-Sep-99 14-Nov-00 19-Dec-00 16-Nov-95 11-Dec-98 26-Oct-00 30-Jun-94 20-Mar-97 27-Nov-98 25-Sep-97 10-Nov-98 14-Nov-00 17-Dec-97 05-Sep-00 28-Nov-00 19-Jun-97 31-Oct-00 27-Nov-98 21-Dec-99 14-Dec-00 20-Dec-00 25-Sep-97 20-Dec-01 07-Dec-99 22-Jun-00 21-Dec-99 19-Dec-00 29-Sep-97 30-Sep-98 14-Dec-99 1 .

October 1988. . Banjarmasin. December 1998. SS0041. Report No. December 1997. SS0028. Factors Affecting Project Performance in the Agriculture and Social Sectors: A Review of Postevaluation Reports between 1991 and 1997. October 2000.80 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS 31 1447 CAM Basic Health Services 32 1675 INO Health and Nutrition Sector Development Program . SS0031. Report No. and Balikpapan Ports 41 1584 PRC Xiamen Port 42 1561 BAN Jamuna Bridge Railway Link 43 1631 UZB Railway Rehabilitation 44 1747 IND Surat-Manor Tollway 45 1774 REG Almaty-Bishkek Regional Road Rehabilitation (Kazakhstan Component) 46 1795 LAO Rural Access Roads 47 1489 THA Third Rural Telecommunications 20-Jun-96 25-Mar-99 24-Aug-00 26-Oct-99 20-Jul-00 03-Dec-98 12-Sep-00 19-Sep-00 23-Nov-79 30-Sep-97 27-Nov-97 02-Oct-97 15-Sep-98 27-Jul-00 31-Oct-00 07-Dec-00 26-Nov-96 Other ADB Project Evaluation Documents Consulted: ADB Special Studies Special Study of the Macroeconomic Environment and Project Performance in Sri Lanka. Report no. Report no. Special Study: A Review of Postevaluation Findings in Thailand. SS0005. Special Evaluation Study on the Policy Impact of Involuntary Resettlement.Policy Loan 33 1749 LAO Primary Health Care Expansion 34 1704 IND Karnataka Urban Development and Coastal Environmental Management 35 1745 PHI Pasig River Environmental Management and Rehabilitation Sector Development Program .Program Loan 36 1646 THA Samut Prakarn Wastewater Management (Supplementary) 37 1755 NEP Small Towns Water Supply and Sanitation Sector 38 1757 SRI Water Resources Management 39 0426 MAL Bintulu Deepwater Port 40 1559 INO Belawan.

APPENDIX 1 List of Reviewed ADB Projects and Other ADB Project Evaluation Documents

81

Special Study: A Review of Postevaluation Findings in Indonesia. Report no. SS0006. November 1988. Special Study: A Review of Postevaluation Findings in Malaysia. Report no. SS0007. December 1988. Special Study: A Review of Postevaluation Findings in Sri Lanka. Report no. SS0008. April 1989. Special Study: A Review of Postevaluation Findings in Nepal. Report no. SS0009. August 1989. Special Study: A Review of Postevaluation Findings in Bangladesh. Report no. SS0010. August 1989. Special Study: A Review of Postevaluation Findings in Western Samoa. Report no. SS0013. June 1990. Special Study: A Review of Postevaluation Findings in Papua New Guinea. Report no. SS0014. December 1990. Special Study: A Review of Postevaluation Findings in South Pacific Developing Member Countries. Report no. SS0019. September 1991. Special Evaluation Study on the Social and Environmental Impacts of Selected Hydropower Projects. Report no. SS036. December 1999. ADB Impact Evaluation Studies An Impact Evaluation Study of Bank Operations in the Education Sector in Indonesia. Report no. IE0022. October 1993. An Impact Evaluation Study of Bank Operations in the Water Supply and Sanitation Sector in Malaysia. Report no. IE0028. December 1994. An Impact Evaluation Study of Bank’s Assistance in the Health and Population Sector in Sri Lanka. Report no. IE0033. December 1995.

82

HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Impact Evaluation Study of the Bank’s Benefit Monitoring and Evaluation Assistance to the Agriculture. Report no. IE0035. December 1995. An Impact Evaluation Study of Bank Assistance to the Industrial Crops and Agro-Industry Sector in Sri Lanka. Report no. IE0038. July 1996.

APPENDIX 3 Illustration of Risk2 Sustainable Livelihoods Framework APPENDIX Analysis in Project Economic Analysis

83

APPENDIX

Sustainable Livelihoods Framework

2

2000 Major Technical Feature Illustrated Substitution of triangular distribution for cell point values Importance of correlations between variables Application of risk analysis to cost-effectiveness analysis. Project Title 1 2 3 Wheat Productivity Improvement Project Second Nonformal Education Project Bangladesh. The case studies have been selected to represent a number of different types of projects and to illustrate different technical features of the application of the @RISK software.e. and difference between modal/most likely values and full distributional estimation Risk analysis applied to poverty impact ratio 3 4 5 Shen Da Power Transmission and Grid Rehabilitation Project People’s Republic of China. The essence of @RISK is that it allows single point values in spreadsheet .. 2001 6 Comparability of @RISK with Risk Master software The various projects and the points which the case studies illustrate are summarised in the following table: @RISK is an Excel add-in and was applied to the original (i. 2001 Primary Healthcare Expansion Project Lao PDR. and choice among alternative distributions Construction of variables’ distributions based on secondary data.84 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS APPENDIX Illustration of Risk Analysis in Project Economic Analysis Introduction This appendix contains case studies illustrating the application of risk analysis into the project economic analysis of various ADB projects using the @RISK software. 2000 Secondary Education Modernization Project Sri Lanka. PPTA) EIRR calculations. 2001 Wind Power Development Project People’s Republic of China.

APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 85 cells (e.g. The case studies used the original Excel spreadsheets developed for the project economic analysis. revenues) to be substituted by user-specified probability distributions such that cells which include results (e. the analysis described for each project took only a few hours to prepare. including the acquisition of familiarity with the original economic analysis and also some exposure (without formal training) to a trial version of @RISK. for costs..g.. In all cases. EIRR. Such distributions (for both input data and for output results) are summarised by @RISK and can be graphed and manipulated in various ways. . ENPV) dependent upon those original sources can be described in terms of probability distributions of outcomes.

. respectively could be possible. if new technology failed to work) but also that savings of 7% and 10%.7%—based on PPTA consultants’ ‘best estimates’ of a number of key variables. @RISK) to the original Excel spreadsheets can augment the economic analysis and supply further information about. the project’s robustness. In each case. Sources of Risk Values for several variables included in the calculation of the ‘base case’ estimate of EIRR were considered to be subject to some uncertainty. it was possible that no areas at all might be affected. which might conceivably be as high as 3. it was felt that (while the minimum value was necessarily 1) there was some doubt about the maximum value. For the crop losses avoided (rust and shattering reductions)..86 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Case Study 1 Wheat Productivity Improvement Project Introduction The project is concerned with raising the productivity of wheat cultivation in an ADB developing member country.5. The economic analysis at the appraisal stage suggested that the project’s EIRR was high. For the SERF..e.g. These were • the estimated value for the shadow exchange rate factor (SERF) – little formal data upon which to estimate this parameter was available • the amount of crop losses avoided (calculated in tonnes. Similarly for land levelling. the PPTA estimates of values were single point ‘most likely’ values. deriving from estimates of proportions of current losses now saved) by the proposed project through reductions in incidence of rust on wheat • the amount of crop losses avoided (calculated in tonnes. and • the actual extent of areas (calculated in hectares) which would be subject to improved soil and water management regimes through the process of ‘land levelling’—in project years 1 to 7. but also that likely targets could be exceeded in very favourable circumstances. The application of a risk analysis software package (i. deriving from estimates of proportions of current losses now saved) by the proposed project through reductions in shattering of wheat grains. at 33. in effect modal values. it is possible that no benefits at all might be obtained (e. and confirmation of.

33% 5.APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 87 Application of @RISK Software The table below shows for each uncertain variable in the PPTA analysis. year 7) Original Point Value 2 3% 7% 18 28 .e. maximum possible) distributions were used for the present example. year 7) Type of Distribution Minimum Substituted Value Triangular Triangular Triangular Triangular Triangular 1 0 0 0 0 Substituting the distributions for the original point values has the immediate effect of replacing the original (i. years 3-6) Land levelled area (000 ha. In the absence of any historical or empirical data.67% 14. the original point value used in the ‘base case’ EIRR calculation. and the characteristics of the distributions which were substituted for these point values. In this case. the mean values for the four variables become as follows: New Distribution Mean Value 2..167 3.5 7% 10% 25 40 Variable SERF Rust losses avoided Shattering losses avoided Land levelled area (000 ha.333 22.666 Variable SERF Rust losses avoided Shattering losses avoided Land levelled area (000 ha. Original Point Value 2 3% 7% 18 28 Most Likely Value 2 3% 7% 18 28 Maximum Possible Value 3. very simple triangular (minimum possible. years 3–6) Land levelled area (000 ha. most likely. most likely) values with the mean values for the new distributions—and thus recalculates the ‘base case’ EIRR.

7%). What also is apparent from this exercise is that there is only about a 1% probability that the EIRR will be below 22%. Having specified the distributions. .e.02%. non-stratified (i. No correlation between variables was assumed to exist. The graph below summarises the results. Risk Analysis Results The main output from the foregoing analysis is a cumulative distribution function for the project’s EIRR. not Latin Hypercube) technique. the expected EIRR is now 30. Based on the data randomly sampled from the 1000 iterations.. the new estimated value for the project EIRR falls to 30.88 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Based on such mean values in the spreadsheet. a simulation was run in which possible values for the variables were randomly sampled 1000 times using a Monte Carlo.3% (from the original 33.

the importance of correlations between variables determining project outcomes is also apparent.. Two ‘pessimistic’ scenarios—one of ‘low wages’.g.e. In each case. for the proportions of trainees both completing the programme and finding work. However. 2001) Introduction The project was concerned with establishing an effective community-based continuing education program for poor and disadvantaged neo-literates (at least 50% of direct beneficiaries are expected to be women). in effect modal values. Sources of Risk Values for several variables included in the calculation of the ‘base case’ estimate of EIRR were considered to be subject to some uncertainty. actual employment could conceivably be below the ‘low employment’ scenario modelled in the original analysis. the RRP/appraisal estimates of values in the base case appear to be single point ‘most likely’ values. there may be some doubt about the extent of the possible downside (e. a combination of low wages and low employment which would cause the project to become nonviable).APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 89 Case Study 2 Second Nonformal Education Project (Bangladesh. The application of @RISK to the appraisal Excel spreadsheets can augment the economic analysis and supply further information about.. for example. as was the ‘switching scenario’ (i. In this case. the other of ‘low employment’ were also modelled in the original analysis.3% in the base case— based on PPTA consultants’ best estimates of a number of key variables. Quantified economic benefits from the project are those captured privately through higher daily earnings. if general macroeconomic conditions . the project’s robustness. at 35. These were • the proportion of participants completing the education programme • the proportion of graduate trainees finding employment • the days worked per month by trainees • the months worked per year by trainees • the incremental wage gained by trainees due to better education. The economic analysis suggested that the project’s EIRR was high. and confirmation of. and • the number of trainees at full operation of the programme.

.9 0.35 Variable Proportion completing training Proportion of trainees finding employment Number of days worked per month Number of months worked per year Daily wage increment due to training ($/day) Trainees at full operation of project Type of Distribution Minimum Substituted Value Triangular Triangular Triangular Triangular Triangular Triangular 0.600.15 1.9 20 6 0. although there are more natural upper limits likely to be in operation.e.9 0. maximum possible) distributions were used for the present example.200. although it is probably very unlikely that more than 95% of participants would both graduate and find work.6 0.25 Maximum Possible Value 0. and the characteristics of the distributions which were substituted for these point values.000 Substituting the distributions for the original point values has the immediate effect of replacing the original (i. the mean values for the four variables become as follows: . Similarly.90 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS deteriorated). Application of @RISK Software The table below shows for each uncertain variable in the RRP analysis.95 26 10 0.800.6 12 3 0.600. In this case. Original Point Value 0.25 1.9 20 6 0.000 Most Likely Value 0. minimum possible.e. the potential downside of days and months worked by trainee graduates is apparent. the original point value used in the ‘base case’ EIRR calculation.95 0.. very simple triangular (i. most likely) values with the mean values for the new distributions—and thus recalculates the ‘base case’ EIRR. most likely. In the absence of any historical or empirical data.000 1.000 1.

9 0.533.75 Days Worked Per Month 0.4% (from the original 35.75 0. a simulation was run in which possible values for the variables were randomly sampled 5000 times using a Monte Carlo.75 1.82 19.233 1.9 20 6 0.. .600. Accordingly.0 0. a correlation matrix was complied in @RISK which specified a positive 0. non-stratified (i. The graph below summarises the results.82 0.75 0.33 6.3%).0 Variable Percent employed Days worked per month Months worked per year Having specified the distributions.75 1.33 0. the new estimated value for the project EIRR falls to 28. not Latin Hypercube) technique.e.75 correlation between all three of these variables (implying that more people were likely to find work for more days and months as the general level of employment rose). It was assumed that some correlation existed among these variables.000 New Distribution Mean Value 0. In effect. the correlation matrix for the three variables looks like the following: Original Point Value 1.75 Months Worked Per Year 0.25 1.333 Based on such mean values in the spreadsheet. Risk Analysis Results The main output from the foregoing analysis is a cumulative distribution function for the project’s EIRR.APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 91 Variable Proportion completing training Proportion of trainees finding employment Number of days worked per month Number of months worked per year Daily wage increment due to training ($/day) Trainees at full operation of project Original Point Value 0. specifically between the proportions finding employment and days and months worked.0 0.

If the correlation is omitted from the simulation. or one chance in 20).6%. but the chances of project failure are now estimated to be reduced (to about 6%.6%). the results are similar—the expected EIRR is slightly lower (27. . the expected EIRR is now 28. What also is apparent from this exercise is that there is about a 13% probability that the EIRR will be below the EOCC of 12%.92 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Based on the data randomly sampled from the 5000 iterations.

‘high’). the economic and financial costs of each DALY differ . and the rural poor by providing more cost-effective and better-focused access to improved quality services. risk analysis can be used to refine this type of approach.APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 93 Case Study 3 Primary Healthcare Expansion Project (Lao PDR. rather than an economic rate of return estimate). the project’s robustness (in this case about where the expected values per cost of DALY saved lie in relation to measures of similar projects). and confirmation of. ‘moderate’.. each of which is characterised by a differential proportion of the target population (grouped by women. Sources of Risk The greatest difficulty in economic analysis of heath projects lies in the uncertainties of the reduction of disease burden. As a result of the total costs of the project being distributed over different numbers of estimated DALYs. The analysis of costeffectiveness in the RRP depended on the use of three scenarios to model doubts about the estimates of effective coverage of the project. 2000) Introduction This project was concerned with expanding primary health care in eight northern provinces in Lao PDR. the analysis using @RISK also tries to show the importance of properly understanding and defining variables’ distributions as determinants of estimates of project economic outcomes. The uncertainty arises both from doubts about changes in the incidence of disease which project interventions bring and also about the actual coverage of service delivery which the project will achieve. children. ethnic minorities. The economic analysis contained in the project’s RRP deals with this uncertainty by modelling three scenarios (‘low’. In addition. children and the total population) which is covered by the primary health care (PHC) services. It was designed to target women. in this case. The application of @RISK to the original Excel spreadsheets can thus augment the economic analysis and supply further information about.e. or saving in the otherwise lost DALYs. A major feature of the economic analysis of the project is that it is based on a monetary measure of cost-effectiveness per disability-adjusted life year (DALY) saved (i. The table below shows details of such estimated values for the early years of the project.

603 51.543 87. These distributions were • a normal distribution. the most likely (i. Both exploit data points from the original PPTA/RRP work.556 2003 16..e. In the case of the normal distribution replacement the mean is the same as the original ‘moderate’ values—and so the final cost/DALY estimate is unaffected. when distributions are substituted for point values the original cell point value is replaced by the mean of the particular distribution. Application of @RISK Software The data in the original spreadsheets were used for the distribution-based calculations. for early years of the project) were substituted by the particular normal distribution just described and then (in a second separate simulation) by the triangular distribution also described.668 37. and range from $45. .82 per DALY under the ‘low’ scenario to $21.896 2002 8. dispersion.922 15. the original point values in the cells for the moderate scenario (see below. Accordingly.205 68. are centrally fixed on the original ‘moderate’ value estimates.116 89. and thus to increase the estimated cost/ DALY.274 2005 34. and • a triangular distribution. and appear to have believable measures of. children.139 24. with the triangular distribution substitution the effect in this case is to reduce the mean estimate.005 2004 25.956 65.391 2006 44. However.09 per DALY under the ’high’ scenario (the ‘moderate’ scenario value is $28.11). with the minimum value being the ‘low’ scenario estimate. As in the other case studies. modal) value being the ‘moderate’ scenario value and the maximum value being the ‘high’ scenario value It may be argued that either or both of these distributions might reasonably model the uncertainty with respect to possible values for project coverage of the population.504 50.744 67.488 by scenario. with the mean being the value from the ‘moderate’ scenario and the standard deviation being 25% of the mean.94 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS DALYs saved under: Low Scenario Moderate Scenario High Scenario 2001 – 11. or limits to. and the total population contained in the ‘moderate’ scenario were used as the basis for the estimation of two sorts of probability distributions. The population coverage estimates for women.417 32.

355 62.09) is very close to that of the original moderate scenario. measured in terms of the cost per DALY saved. It can also be seen that there is only a 5% chance that the actual cost of a DALY would exceed $40 (well within acceptable ADB parameters).116 40.427 198.039 354.909 2003 39. The first graph shows the cumulative distribution function for the project’s cost per DALY based on the normal distribution. non-stratified (i. The graphs below summarise the results.APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 95 Moderate Scenario Population Estimates 2001 Women 15–44 Children 0–4 Total Population 12..636 29.944 2001 25.e. The mean of the distribution (at $29. not Latin Hypercube) technique. .931 52.344 Having specified the distributions. Risk Analysis Results The main output from the foregoing analysis is a cumulative distribution function for the project’s cost-effectiveness.600 9.805 19.005 2004 54. for each of the two types of distribution.662 346.343 2005 69.428 2006 70.158 128.269 51.178 270. two simulations were run in which possible values for the variables were randomly sampled 5000 times using a Monte Carlo.

. Although the particular project design in this instance may find both results within acceptable limits. this example does highlight the critical importance of specifying variables’ distributions correctly.96 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS The second graph shows the same result. but now based on the triangular distribution instead of the normal one. and the 5% upper value unlikely to be exceeded is now some $70. even two apparently reasonable-looking distributions for the same variables can produce quite different results in terms of estimates of project costs or returns. The mean value for the distribution in this case is some 33% higher than previously (at just under $40).

Sources of Risk In the case of this project.g. school-based assessment (SBA). major sources of uncertainty include • differences which will be achieved in earnings between those with upper secondary.. from labor surveys. and • examination pass rates at grades 11 and 13. It can clearly be seen that the returns from the project may be affected adversely if there is a fall in the O/L and A/L pass rates. The project sensitivity test results in this respect are summarised in the following table. the project design aimed to increase percentages qualifying to enter grade 12 from 35% to over 50% by 2005. etc. 2000) Introduction The objective of the project was to make Sri Lanka more economically productive and competitive by modernizing the secondary education system through improved curriculum instruction. and school-based management (SBM). This is taken as a proxy for the incremental social benefit to the country as a whole. increase the percentage qualifying to enter grade 12 from 15% to 20% for rural students by 2005. from similar projects in other countries. In this case study. a sensitivity analysis was conducted using different values for these variables. to implement SBA and examination reform by 2005. The project economic analysis calculated an EIRR based on incremental earnings of those receiving improved education in a competitive labor market. .) can be used to construct probability distributions which can then be substituted for the original point values. It also shows the implications for estimates of project returns based on point values as compared to those based on distributional data. the analysis using @RISK shows how data from other sources (e. ‘O’ level (O/L) and ‘A’ level (A/L) education compared to laborers with no schooling.APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 97 Case Study 4 Secondary Education Modernization Project (Sri Lanka. and implement a SBM system in all secondary schools by 2005. Because of doubts about actual examination pass rates and income differentials between non-educated and educated workers. Specifically.

822.894 140.) based on various measures of the ‘goodness of fit’ to the available data (including Chi-square.36 x 4 x 12 x 115/95 Wage of employed O/L worker 1. K-S and A-D tests). beta.. exponential..579 77.987 36% 50% 48% 60% .336. and between O/L worker and A/L worker EIRR 20% 11% 27% 18% Application of @RISK Software In the previous case studies. The software user is able to choose between different types of distribution to be fitted (e. etc. In this case study. @RISK generated distributions for variables in cells based on either one or two user-specified characteristics (i..g. The data used for the project economic analysis base case is contained in the following table: Sri Lanka: Secondary Education Modernization Project: Base Case Basic Data Wage of laborer with no schooling 1. maximum and most likely – in the case of the triangular distributions).98 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Sensitivity Test Base case Reduce O/L pass rate from 50% to 45% and A/L pass rate from 60% to 55% Increase O/L pass rate from 50% to 55% and A/L pass rate from 60% to 65% Reduce income differentials by 10% between worker with upper secondary education and O/L worker. mean and variance/ standard deviation) or a few data points (e. normal.45 x 4 x 12 x 115/95 Wage of employed A/L worker 2. minimum.40 x 4 x 12 x 115/95 Grade 11 exam pass rate Implementation year 1 (and without project) Implementation year 5 Grade 13 exam pass rate Implementation year 1 (and without project) Implementation year 5 59.025.591 105.g. hypothetical data from secondary sources covering examination pass rates and workers’ earnings are entered into the spreadsheet and distributions are then fitted to the variable(s) in question.e.36 x 4 x 12 x 115/95 Wage of laborer with upper secondary education 1.426. lognormal.

) may rank distributions differently. and the figure quoted for A/L wages (SRL Rupees 140.g. etc. suppose historical data were available (for example) for wages of A/L workers (e. The default measure is the Chi-square value. For simplicity.. The software generates distributions of different types. The distribution has an identical mean to the original spreadsheet point value. the Anderson-Darling statistic. and a (relatively small) standard deviation of 3370. the Kolmogorov-Smirnov statistic. from similar projects in other countries implemented in recent years) in the following form: Earnings of A/L workers: 142000 141200 140800 142100 141370 141350 130600 141700 139800 140980 144500 139500 146000 148000 143000 143500 142850 135500 138700 141000 136000 141500 138000 139500 141100 143100 140987 143000 Grade 11 pass rates: 50 52 45 50 53 56 49 45 48 50 52 50 Grade 13 pass rates: 60 62 55 60 63 66 59 55 58 60 62 60 @RISK allows a distribution to be fitted by users to such observations. The following chart shows a normal distribution fitted to the earnings data. from cross-sector employment surveys) and grade 11 and 13 pass rates (e. It should however be noted that different measures of goodness of fit (e.. and ranks them according to various measures of goodness of fit. the risk analysis will look more closely at variability associated with only the earnings of A/L workers and with grades 11 and 13 pass rates.987) is presumably an average (mean) value for the country as a whole.g. Considerable care should therefore be exercised in interpreting .APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 99 The earnings data come from the Household Income and Expenditure Survey (1995/96) conducted by the Department of Census and Statistics.. although it could equally be applied to any variables in the project economic analysis.g. For illustration purposes.

In this case. and the only (very slightly) better fit in terms of its Chi-square value—a beta (general) fit—has a different predicted mean (on both other major estimates of goodness of fit—the K-S and A-D statistics—the normal fit is better). The following charts show (as an example. a simulation was run in which possible values for the variables were randomly sampled 5000 times using a Monte Carlo. although in this case ‘Logistic’ and ‘Extvalue’ distributions have perhaps slightly better fits based on their K-S and A-D measures— even though all three distributions’ Chi-square values are the same. again normal distributions (for each of the grade 11 and 13 pass rates) have been chosen to replace each of the original point values because the mean of these distributions alone among the available choices is the same as the original values.e. ..100 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS the statistics and applying particular fits. non-stratified (i. the normal fitted distribution has been chosen because its mean is identical to the original point value data. the grade 11 values in the distribution of that variable are identically-distributed but 10% lower in all cases) the original grade 13 examination pass rate data grouped by class interval and the various fitted distributions of different types: Having specified the distributions for the three variables. For the examination pass data. not Latin Hypercube) technique.

given that the selected variables are modelled to be normally distributed around identical mean values to those in the original spreadsheets. depending upon how such estimates are used. The . In many project analysis circumstances. The examination pass rate data in this case study can also be used to demonstrate one other point about risk (i. Consider the following example (hypothetical – and unrelated to the actual project under consideration).APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 101 Risk Analysis Results The distribution which results for the EIRR on this basis has a mean which is very close to the base case mean— unsurprisingly. it is not clear whether point value data in spreadsheet cells represent the analyst’s ‘best guesses’ of ‘most likely’ (or somehow ‘typical’) values or whether it represents average/ mean expectations. probability-based) analysis as applied in such situations. In fact great differences in results can emerge in calculations of returns.e. Based on the (hypothetical) distribution data it appears that there is only about a 7% chance that the EIRR for the project would be below the EOCC of 12%.

. even though other (lower) values can also occur. Grade 11 pass rates 50 40 50 40 50 40 50 50 50 50 50 50 Grade 13 pass rates 60 50 60 50 60 50 60 60 60 60 60 60 If estimates of EIRR/ENPV. If we construct probability distributions around such values with the original points as their mean we get a similar expected EIRR value and a distribution as shown in the chart immediately above. The following distribution of observations in such a case may suggest that values of 50% and 60% passes are the values most likely to occur (in nine out of twelve observations). 50% for grade 11 and 60% for grade 13) may in fact represent the analyst’s best guesses of ‘most likely’ outcomes (i. 50% and 60%) we get the original base case estimate of 20% EIRR. etc.e. .e.102 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS estimates of exam pass rates in the original data (i. unadjusted for any probability of occurrence). the single value from among a range most likely to occur.e. are based on the most likely point values alone (i...

. however.APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 103 If. we use the actual data from observations as just shown in the last table (and which could have been used to derive the original estimates based on a ‘most likely’ or modal basis as opposed to a mean-based one) the mean and distribution of EIRR are as follows. but that the risk of project failure has now increased over fourfold (to 29% from about 7%). It can be seen that not only is the estimate of the expected EIRR proportionately lower.

Hence. A reduction in vulnerability is a central element in a poverty reduction program and a sharing of risk across a wide portfolio of projects may still leave the poor excessively exposed. RiskMaster was used on the original Excel spreadsheets to supply further information about the PIR of the project. an integrated regional grid that will enable restructuring of the power sector in Liaoning Province will be developed. for important. Five variables for the PIR analysis have been considered to be subject to some uncertainty. For this case study illustration. These were: • net financial benefits • net economic benefits • the share of consumer surplus to the poor • the share of the Government net benefits to the poor • the share of labor surplus to the poor. In each case. 2001) Introduction The project aims to increase the capacity and efficiency of electricity transmission from the northeastern part of the PRC to the southern part of Liaoning Province. Furthermore. it is desirable that both the probability of overall project failure and the probability of a negative outcome for the poor be assessed. the RRP estimates of values were single point ‘ most likely’ values. thereby increasing the availability of electricity and reliability of the electricity transmission and distribution systems in Liaoning Province. @Risk was used.104 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Case Study 5 Shen-Da Power Transmission and Grid Rehabilitation Project (People’s Republic of China. . The major economic benefits for the project are efficiency improvement (reduced losses) and increased supply. Sources of Risk The risk analysis was carried out for the PIR of the Project to ensure that there would be no negative outcome for the poor. poverty-focused projects for which monetary estimates of costs and benefits to the poor are feasible. The dimension of risk in project analysis is of significance in discussions of poverty since the poor are the most vulnerable to unexpected unfavorable outcomes.

APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 105 Risk Variables Report .

The table below shows the five selected variables including the original point value used in the PIR calculation. and the assigned probability distributions. no correlation between variables was assumed to exist.106 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Risk Variables Report (continued) Application of @RISK Software The probability distributions attached to the selected variables and graphed below take into account past ADB experience in power projects in general and the project/country circumstances in particular. The results were based on 5000 iterations. . The risk analysis was carried out using the Monte Carlo simulation technique. the assumed value ranges.

The probability of the PIR falling below the value for the poor as a proportion of the population as a whole.9%. based on the weighted average of all simulated combinations is 19.8% (with a standard deviation of 3. PIRs range from a minimum of 6.6% to a maximum of 26.e. .. without the consideration of risk).APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 107 Risk Analysis Results The results of risk analysis have shown that the expected PIR. slightly lower than the base case value (i. and thus not having a pro-poor impact.6%). is estimated to be about 12%.

TSP and CO2 associated with conventional thermal power generation. Sources of Risk Project parameters that were selected as risk variables based on the sensitivity analysis include: • capital cost • generation • foreign exchange rate • commission date • avoided cost. the RRP estimates of values were single point ‘ most likely’ (i. In this case. In each case. Nontradable commodities were valued at shadow prices using a mix of standard and specific conversion factors. and environmental benefits. the analysis has utilized @RISK to test its comparability to Risk Master. . ‘modal’) values. and a discount rate of 12%. Risk analysis on the original Excel spreadsheets was originally carried out using Risk Master.108 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Case Study 6 Wind Power Development Project (People’s Republic of China. NOX. 2000) Introduction This project will produce electricity in an environment-friendly manner and increase the share of wind-based electricity in overall power generation through the establishment of three grid-connected wind farms in the Xinjiang Uygur Autonomous Region and Heilongjiang and Liaoning provinces. The main benefits are the incremental supply of electricity.e.. The calculation of the EIRR covered 1999-2022. valued through the avoided costs of supplying an equal amount of electricity as the wind-based electricity to be produced under the Project. and used constant 1999 prices. SO2. It will avoid emissions of . Tradable commodities were valued at border prices at the prevailing exchange rate.

Risk Variables Report .APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 109 Application of @RISK Software The probability distributions attached to the selected variables take into account past ADB experience in power projects in general and the project/country circumstances in particular. The table below shows the five selected variables including the assumed value ranges and the assigned probability distributions. as well as extensive discussions with the executing agencies and other relevant agencies. A Monte Carlo simulation was used to model the likely distribution of these risk variables and the associated EIRRs.

110 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Risk Variables Report (continued) .

APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 111

The results were based on 3000 iterations; no correlation between variables was assumed to exist.

Risk Analysis Results
The results of risk analysis using @RISK shows that the expected EIRR, based on the weighted average of all simulated combinations is 12.6% (with a standard deviation of 2.5%)—about 0.3% lower than the base case EIRR of 12.9% (without consideration of risks). The EIRRs associated with assumed probabilities range from a minimum of 4.8% to a maximum of 22.6%. For this project, the probability of the EIRR to be below the considered discount rate of 10% is 15.4%. The expected EIRR using Risk Master produced a weighted average of 12.7% with a standard deviation of 2.4%—involving only a slight difference of 0.1% between the two software programs.
EIRR (Local Env. Only) Expected value Standard deviation Minimum Maximum Probability of negative outcome 12.6% 2.5% 4.8% 22.6% 0.0%

112 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

References
Adhikari, R. and J. Weiss. 1999. Economic Analysis of Subregional Projects. EDRC Methodology Series No. 1. March 1999. Manila: ADB. Anderson, J.R. 1974. “Sparse Data, Estimational Reliability and Risk-Efficient Decisions.” American Journal of Agricultural Economics. Arrow, K. and N. Lind. 1970. “Uncertainty and the Evaluation of Public Investment Decisions.” American Economic Review. Volume 60 No. 3. ADB. 1979. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to Malaysia for the Bintulu Deepwater Port Project. Manila. ADB. 1997a. Guidelines for the Economic Analysis of Projects. Manila: ADB. ADB. 1997b. Workbook on the Economic Valuation of Environmental Impacts. Manila: ADB. ADB. 2000a. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to Sri Lanka for the Secondary Education Modernization Project. Manila. ADB. 2000b. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to the Lao People’s Democratic Republic for the Primary Health Care Expansion Project. Manila. ADB. 2000c. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to the Philippines for the Infrastructure for Rural Productivity Enhancement Sector Project. Manila. ADB. 2000d. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to the People’s Republic of China for the Wind Power Development Project. Manila. ADB. 2001a. Handbook for Integrating Poverty Impact Assessment in the Economic Analysis of Projects. Manila: ADB.

APPENDIX 1 List of Reviewed ADB Projects and Other ADB Project Evaluation REFERENCES 113 Documents

ADB. 2001b. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to the People’s Republic of China for the Shen Da Power Transmission and Grid Rehabilitation Project. Manila. ADB. 2001c. Guidelines for the Financial Governance and Management of Investment Projects Financed by the Asian Development Bank. Manila. ADB. 2001d. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to Bangladesh for the Second Nonformal Education Project. Manila. Belli, P J. Anderson, H. Barnum, J. Dixon, and J. Tan. 2001. Economic Analysis of ., Investment Operations: Analytical Tools and Practical Applications. Washington D.C.: World Bank Institute. Bolt, R. and M. Fujimura. 2002. Policy-based Lending and Poverty Reduction: An Introduction to Processes, Assessment and Options. ERD Working Paper Series No. 2. Manila: ADB. Clarke, R. and A. Low. 1993. “Risk Analysis in Project Planning: A Simple Spreadsheet Application using Monte Carlo Techniques.” Project Appraisal. Volume 8 Number 3: 141-6. September 1993. Dinal, A., T.K. Balakrishnan and J. Wambia. 1997. “Political Economy and Political Risks of Institutional Reforms in the Water Sector.” World Bank Working Paper (draft), Washington DC: World Bank. European Commission. 1997. Financial and Economic Evaluation of Development Projects. Fujimura, M. and J. Weiss. 2000. Integration of Poverty Impact In Project Economic Analysis: Issues in Theory and Practice. EDRC Methodology Series No. 2. October 2000. Manila: ADB. Harberger, A. 1998. “Economic Project Evaluation, Part I: Some Lessons from the 1990s.” Canadian Journal of Program Evaluation. Special Issue: 5–46. Johnson, K. 1985. Risk Analysis and Project Selection: A Review of Practical Issues. ADB Economics Staff Paper No. 28. Manila: ADB.

Yunnan Environmental Development Project (Project Memorandum). UK Treasury Taskforce – Private Finance. Uncertainty and Profit. ERD Technical Note No. UK Overseas Development Agency. 1974. 2002. Staff Appraisal Reports (www. various.D. Mariano. Technical Note Number 5.” In Project Appraisal and Planning for Developing Countries. 1992 (various editions 1998-1995). 10. 1966.org). A. White Paper.worldbank.M. Reutlinger. Swalm. 2000. 11. 2. Rural Development Department. “Utility Theory – Insights into Decision-Taking. 1970. C. Baltimore: The Johns Hopkins University Press. World Bank. 2001.. R. I.114 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS Little. Baltimore: The Johns Hopkins University Press. Manila: ADB. World Bank. 2001. Risk Analysis in Project Appraisal. Political Economy and Political Risks of Institutional Reforms in the Water Sector (draft). Ward.. “Risk. Unpublished report under RETA 5932. 1975.” Harvard Business Review. N. Integrating Risk into ADB’s Economic Analysis of Projects. K. Rayner. Risk Analysis Software Packages Evaluation Report. Project Appraisal in Developing Countries: A Guide for Economists. and J. UK-DFID. Manila: ADB. A. How to Construct a Public Sector Comparator. and H. Techniques for Project Appraisal under Uncertainty. L. 2000. L. World Bank Staff Occasional Paper No. May 2002. UK-DFID. Pouliquen. Lagman-Martin. 1970. van der Tak. S. London. Mirrlees. 1997. Squire. London: Heinemann. . Baltimore: The Johns Hopkins University Press. Economic Analysis of Projects. Eliminating World Poverty: A Challenge for the 21st Century. World Bank Staff Occasional Paper No.

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer: Get 4 months of Scribd and The New York Times for just $1.87 per week!

Master Your Semester with a Special Offer from Scribd & The New York Times