You are on page 1of 117

Handbook for

Integrating Risk Analysis


in the
Economic Analysis
of Projects

May 2002
© Asian Development Bank 2002

All rights reserved

ISBN 971-561-458-2
Publication Stock No. 060202

Published by the Asian Development Bank


P.O. Box 789, 0980 Manila, Philippines
Foreword

T
his Handbook is an output of a study undertaken by the Economics and
Research Department (ERD) of the Asian Development Bank (ADB). Nigel
C. Rayner directed the conduct of the study, assisted by Anneli Lagman-Martin.
Overall supervision of the initial stages was provided by David Edwards, then Assistant
Chief Economist, while Xianbin Yao, Assistant Chief Economist, supervised the study
through to its conclusion. Keith Ward (Staff Consultant) provided extensive technical
inputs and advice throughout the study. Support for word processing requirements
was provided by Ma. Nieva Baguisa.
Preparation of the Handbook benefited from a participatory approach through
consultation meetings and individual interviews. At various stages of the study, ADB
staff were given the opportunity to review and comment on the draft reports. The
study benefited from substantive comments from Stephen Curry, Manabu Fujimura,
and Bo Lin. The Handbook was further enhanced with the inclusion of case studies,
based on work undertaken by Manabu Fujimura, Vincent de Wit, Bo Lin, William Loxley,
and Keith Ward. During an in-house seminar on project economic analysis, the results
of the study were disseminated and the use of risk analysis software was demonstrated.
This Handbook further develops the discussion of the principles on risk analysis
that is contained in Chapter XI (Uncertainty: Sensitivity and Risk Analysis) of ADB's
Guidelines for the Economic Analysis of Projects (1997). The Handbook contains detailed
discussions of current risk analysis practice, the theory and application of various
risk analysis techniques, and recommended applications for risk analysis, including
a summary of typical risk circumstances on a sector-by-sector basis. To illustrate the
ease of use of quantitative risk analysis software, the Handbook includes case studies
based on actual ADB projects in the agriculture, education, health, and power sectors.
A summary version of this Handbook is available as ERD Technical Note No. 2
Integrating Risk into ADB's Economic Analysis of Projects.
The Handbook provides guidance with respect to the integration of risk
analysis in the design and analysis of projects. It also provides direction on how the
analysis of risk can be used to improve the focus on poverty reduction. While the
Handbook is primarily targeted to ADB staff and officials of its developing member
countries, it may also be of interest to others involved in development assistance.
iv HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Abbreviations and Acronyms


ADB Asian Development Bank
ADF French Development Agency
CAPM capital asset pricing model
CAMEL capital adequacy, assets quality, management quality,
earnings and liquidity
CDF cumulative distribution function
DFID Department for International Development (UK)
EIRR economic internal rate of return
ENPV economic net present value
EOCC economic opportunity cost of capital
FI financial institutions
FIRR financial internal rate of return
FNPV financial net present value
NPV net present value
PBL policy-based lending
PFI Private Finance Initiative (UK)
PIA policy impact analysis (matrix)
PIR poverty impact ratio
PPTA project preparatory technical assistance
RRP report and recommendation of the President
SAR staff appraisal report (World Bank)
SERF shadow exchange rate factor
SI sensitivity indicator
SV switching value
UK United Kingdom
VaR value at risk
VOC vehicle operating cost
WTP willingness-to-pay
Contents v

Contents
Page
Foreword iii
Abbreviations and Acronyms iv

Chapter 1: Introduction and Background 1


Introduction 1
Background 3
Why Undertake Risk Analysis? 7
Chapter 2: Technical Approaches to the Analysis of Risk in Project
Economics 9
Introduction 9
Risk and Uncertainty: A Definition of Terms 10
Allowing For Uncertainty 11
Modeling Risk Quantitatively 15
Technical Issues in Modeling Risk: Two Considerations 19
Risk Analysis, Decision-Making, and Welfare 23
Summary of Approaches to Risk Analysis 27
Chapter 3: A Summary of Development Agency Experience in
Applying Risk Analysis 29
Introduction 29
ADB Experience With Risk Analysis 30
World Bank Experience With Risk Analysis 36
Other Agency Experience With Risk Analysis 38
Risk, and the 'Harberger' Critique of Recent Applied Project
Economic Analysis 43
Summary of Risk Analysis Experience 44
Chapter 4: Poverty Reduction Objectives and Risk Analysis 47
Introduction 47
Planning for the Poor, Vulnerability, and Risk Aversion 48
Poverty Reduction Objectives: Implications for Quantitative
Risk Analysis 52
vi HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Poverty Reduction Objectives: Implications for Qualitative


Risk Analysis 53
Summary 55
Chapter 5: Strengthening Risk Analysis in ADB Operations 57
Introduction 57
Summary of Risk Analysis Techniques: Applications
and Limitations 58
Strengthening Project Design: Overall Economic Benefits
Estimation 62
Promoting the Sustainability of Project Effects 63
Supporting Poverty Reduction Objectives 69
Risk Analysis and Policy-Based Lending 69
Sectors and Projects: Some Typical Risk Analysis Situations 71
Technical and Resource Considerations
(including applying '@RISK') 74

Appendixes 79
Appendix 1: List of Reviewed ADB projects and Other ADB
Project Evaluation Documents 79
Appendix 2: Sustainable Livelihoods Framework 83
Appendix 3: Illustration of Risk Analysis in Project Economic Analysis 84

References 113
Introduction
1
Introduction
Background
and

The purpose of this Handbook is to support the development of a practical


and operationally relevant methodological framework for the analysis of risk in project
design and project economic analysis.
The Handbook is divided into five parts. Following a brief introduction,
including a summary of the reasons why risk analysis may be undertaken, Part II
outlines the available technical approaches to modeling risk within conventional project
economic analysis. The classic distinction between “risk” and “uncertainty” is ex-
2 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

plained, and then some typical techniques for dealing with uncertainty are outlined.
This is followed by a summary of the nature and practice of sensitivity analysis in
dealing with uncertain outcomes, as it is applied by, for example, the Asian Devel-
opment Bank (ADB) and the World Bank (WB). Techniques for modeling risk on the
basis of probability distributions are then described. This leads inevitably to questions
of how to make choices among different risky investment possibilities (i.e., different
projects or alternative designs of similar projects) that may have been identified
through the application of such techniques. It will be noted that while there was
quite a large academic literature on risk analysis in the 1970s (following the publi-
cation of the classic texts on project economic analysis in the 1960s and 1970s), there
have been relatively few theoretical developments in recent years.
Part III reviews the practical experience of ADB and other major agencies
with regard to the analysis of risk in project design. Some 50 ADB projects (across
all sectors and from a number of countries over recent years), and a number of World
Bank projects where risk analysis of different sorts has been applied (including some
which are suggested to be representative of “good practice” in this regard) are
reviewed. Consideration is also given to the practices of risk analysis by several
other bilateral agencies, most notably the Department for International Development
(DFID) of the United Kingdom (UK), and also to the UK Treasury in the application
of the private finance initiative (PFI) in project design.
Part IV considers the implications for risk analysis of ADB’s increasing emphasis
on distribution and poverty impact analysis. As a guiding principle, it is suggested
that increased lending to particular groups in society (notably the poor) now provides
an imperative to extend the incorporation of risk analysis in project economic analysis.
This is not only because targeting the poorest may be inherently more uncertain
than reaching other groups, but also because the risks of project failure are more
concentrated within society, and that the consequences of project failure can be more
extreme to those at or below poverty lines. It is argued that practical applications
of more rigorous risk analysis (i.e., by better incorporating planners’ and project
participants’ awareness and perceptions of, and attitudes to, risk in project design)
during investment preparation can strengthen projects as well as policy-based
lending. This is expected to lead to better project selection in both cases—by
both better identifying and quantifying sources of variability and by choosing
project designs which meet poor peoples’ aspirations and more closely fit their
circumstances.
Finally, Part V provides practical guidance on how to apply different types of
risk analysis in different situations, based on existing techniques and given the
emerging nature of ADB’s operations. Different sorts of lending are considered in
relation to real-world situations with respect to data availability, time and resource
CHAPTER 1 Introduction and Background 3

implications, etc. Some practical guidance with applying software packages like the
‘@RISK’ is provided.
Various supplementary materials are contained in the Appendixes, including
brief case studies of computer-based risk analysis applied to recent ADB projects.
The case studies are designed to highlight major features and key technical points—
extending the original project materials in a demonstration context rather than
necessarily a project-specific one.

Background

Johnson (1985) reviewed ADB’s practice (and also that of the World Bank)
with respect to the application of risk analysis. This paper provided a comprehensive
and thorough review of literature and techniques available, but noted that risk analysis
in any form had only been applied in one ADB project (a port project) and two World
Bank projects (a port project and a fertilizer plant project) by that date. It may be
relevant to note that this was despite the fact that major writers on the advocated
techniques—e.g., Reutlinger, Pouliquen—were either World Bank staff, or were
published by that institution).
The paper discussed the difficulties in obtaining reliable data in many
circumstances (such that variables could be adequately described by different sorts
of probability distributions), highlighted the statistical complexity of the techniques
(particularly as regards dealing with covariance among variables), and noted the
demands in staff and computer resources (mainframe, at that time) for undertaking
risk analysis. There was also an inference that a spurious precision may appear to
be attributed to results arising from risk analysis which was in reality based only on
analysts’ “best guesses” for variables’ distributions rather than historical evidence.
Also, an apparent methodological rigor in the appraisal process (as implied by a
full-scale risk analysis) might actually obscure the search for radical project or policy
alternatives.
Of the reviewed ADB and World Bank projects in the 1985 paper, it was also
striking to note that the causes of differences between expectations (i.e., the modeling
of the situation ex ante) and outcomes (i.e., the review of the situation ex post) had
not been especially well-captured by the scope of the specific risk analyses which
had been employed. The review also quite correctly pointed out that even where risk
analysis was undertaken, in itself it did not provide a basis for choice among competing
acceptable projects [i.e., among projects with net present value (NPV) > 0 at 12%
discount rate] unless something was known about a society’s social welfare function
4 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

and extent of risk aversion—which in practice is still not likely to be available to


analysts and planners today.
In conclusion, the paper recommended an extremely pragmatic approach to
the use of risk analysis, suggesting that only certain sorts of situations were likely
to be suitable for application of probability-based techniques. Practical applications
of risk analysis in ADB project work did not appear to change significantly after the
preparation of the review.
ADB’s Guidelines for the Economic Analysis of Projects (1997)—hereafter the
Guidelines—built upon the contents of the earlier review, and recommended the
application of quantitative risk analysis techniques for situations where
• projects are very large (from a national point of view), or
• projects are marginal [i.e., where the economic internal rate of return (EIRR)
may be just over 10-12%], or
• there is considerable uncertainty over the values for key variables.

Quantitative risk analysis involves consideration of a range of possible val-


ues for key variables (either singly, or in combination), which then results in the
derivation of a probability distribution of a project’s expected economic net present
value (ENPV) or EIRR (i.e., as opposed to a single point value). The key point for
analysts and planners to consider is the likelihood of a project’s ENPV falling
below zero (at a 12% discount rate) or its EIRR falling below the economic oppor-
tunity cost of capital (EOCC). This information should be incorporated into the
decision as to whether to accept or reject the project. However, no decision rules
are offered in this regard

“There is no fixed criterion for using such a result.” (Guidelines,


Appendix 21, page 157)

as it is implicitly recognized that actual choices among differentially-risky projects


will still depend upon particular levels of risk aversion being applied by different
decision-makers.
Risk analysis is presented in the Guidelines largely as an extension of sensitivity
testing, and it is suggested that such analysis should be conducted where sensitivity
testing has shown that project returns are highly dependent upon the values which
may obtain for a particular variable. The data requirements to undertake some sort
of risk analysis (i.e., to allow the construction of some sort of probability distribution)
are mentioned, but not described in depth.
The Guidelines for the Financial Governance and Management of Investment
Projects Financed by the Asian Development Bank (hereafter, Financial Guidelines)
CHAPTER 1 Introduction and Background 5

also take a very similar approach to analysis of unknown financial outcomes. The
analysis is seen largely in terms of sensitivity testing, and the advocated techniques
are such that standard changes (e.g., +/- 10% or 20% for values of critical financial
cost and benefit items, delays in implementation of one or two years, etc.) are measured
in terms of their effects on estimates of financial internal rate of return (FIRR), financial
net present value (FNPV), etc., and sensitivity indicators (SI) and switching values
(SV) calculated for each variable tested. In almost identical format to the Guidelines
(i.e., for economic analysis), the Financial Guidelines mention the possibility of
quantitative risk analysis and describe circumstances in which it may be appropriate
(i.e., for large, marginal or very uncertain projects), but provide no description of
particular techniques.
Where the Financial Guidelines do differ from the Guidelines in their analysis
of risk, however, is in their analysis of financial institutions (FIs) which may participate
in ADB investment projects. When assessing FI performance, a range of standard
accounting and financial measures is used, which includes indicators designed to
assess risk. The Financial Guidelines therefore describes and advocates the use
of risk measures such as credit at risk, value at risk (VaR), foreign exchange risk,
maturity risk, and contagion risk. These are conceptually probability-based,
although the actual data upon which they are calculated may come from entirely
objective sources (e.g., historical data series) or can result from largely subjective
assessments [e.g., of project preparatory technical assistance (PPTA) consultants,
ADB staff] depending upon individual country and project circumstances. In addition
to these financial measures, some development agencies (e.g., the French Development
Agency - AFD) use systems of indicators (covering management practices, monitoring
and evaluation systems, compliance with regulations, etc.) to assess exposure to
risk of FIs as a result of endogenous rather than exogenous factors. Although these
measures are available to project designers, actual practice suggests that use is not
made of them as much as might be expected (they are typically considered in larger,
‘financial sector’ operations, rather than for smaller projects – e.g., for agriculture
banks, microcredit/credit unions, etc.). It will be suggested later that their selective
extension and wider application to other typical sorts of project institutions (e.g.,
executing agencies in transport, power, water and sanitation projects, etc.) may
assist with ensuring greater sustainability of project effects (as the ultimate
realization of economic benefits often depends upon the operating performance
of such institutions).
One reason for the relative paucity of material on risk analysis in the Guidelines
(and Financial Guidelines) is that economic theory suggests that for governments
undertaking many independent projects simultaneously the consequences of risk on
any one project can be ignored – risks will be “spread” across all members of society
6 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

and “pooled” across the portfolio of all projects – and thus the government can be
taken to be ‘risk neutral’ as far as individual projects are concerned. The original
conceptual basis for this argument was established in a classic article by Arrow
and Lind (1970). However, to the extent that project lending may tend to become
concentrated on specific sections of society (i.e., particular groups—including the
poorest in society, individual regions, certain sectors, etc.) the burden of risks
becomes more concentrated within society and these general assumptions be-
gin to break down.
As will be described in Part III of the Handbook, actual applications of risk
analysis in ADB operations have remained relatively limited since 1985, and have
been concentrated in certain sectors, notably power. One reason for this situation
has undoubtedly been difficulties for staff and consultants with obtaining reliable
data about key variables, and also having easily available computer software capable
of fitting probability distributions (e.g., normal, uniform, log-normal, binomial, beta,
exponential, etc.) to data sets and generating expected values (e.g., for project ENPVs,
EIRRs, etc., and also for absolute values in the context of cost-effectiveness analysis)
with associated measures of variance.
While the fundamental issues concerning data (and the extent to which
situations can be described as “risky” as opposed to simply “uncertain”) remain and
are germane to individual project situations, considerable advances in computer
software in recent years mean that (where data are available) risk analysis can now
be undertaken extremely easily as an “add-in” to existing, predominantly spreadsheet-
based, financial and economic analysis at any stage of project preparation. Some
forms of risk analysis can also be undertaken within existing spreadsheet software
(e.g., Lotus, Excel)—not even requiring any add-ins.
A survey of commercially available risk modeling software (Mariano 2001)
concluded that [among several competing packages, including ‘Crystal Ball, ‘RiskEase’
(updated version of ‘RiskMaster’), @RISK] the @RISK package (Professional Edition,
and an add-in to Microsoft Excel) was highly suitable for
• undertaking “Monte Carlo” based simulations to derive probability
distributions of outcomes (including of project EIRRs/ENPVs)
• fitting distributions to data sets (using advanced algorithms), and
• viewing graphically the distributions of variables and outcomes.

The implication is therefore that the tools for at least some types of risk analysis
are now potentially widely available (and indeed one software package for dealing
with risk in this way, i.e., “Risk Master” has already been applied within ADB). The
question remains as to how and in what circumstances such tools may be advocated
as appropriate to be applied much more widely.
CHAPTER 1 Introduction and Background 7

Why Undertake Risk Analysis?

Before considering in some detail the various techniques available for risk
analysis, it may be worth reviewing exactly what the purpose of applying any such
techniques really is.
Much of the discussion in project economic analysis texts and in academic
literature concentrates on the outputs of quantitative risk analysis (i.e., ‘expected’
EIRRs/ENPVs plus associated measures of variance) as being useful for making choices
between different investment projects (i.e., projects with higher expected returns
but more variability of returns may be compared to less attractive but more stable
opportunities). Following on from this, it is demonstrated that understanding a
decision-maker’s (i.e., planners, policy-makers, society) subjective attitude towards
risk—how higher expected returns are traded off against increased variability—
enables consistent choices to be made across a portfolio of public sector investments.
In practice, however, ADB is typically not concerned with making choices
among a number of mutually-exclusive, competing projects but is more usually en-
gaged in reviewing projects one-by-one. For this reason the analysis of risk through
the techniques described below is practically of most use in
• identifying those factors (e.g., quantities, prices, rates of adoption and usage)
which are the key determinants of project outcomes
• determining the likelihood of an individual project’s returns being
unacceptable (i.e., EIRR<EOCC, ENPV<0) because of the effects of the
identified key risk factors, and
• designing measures within that project environment and its sector context
to mitigate the identified risks arising from the identified key factors.

The emphasis and presentation of any form of risk analysis in project economic
analysis observed in practice is thus usually on demonstrating that risks to individual
project success have already been identified and mitigated as far as possible within
the proposed project design, and that the extent of any remaining risk is both quantified
(i.e., known) and its existence is regarded as ‘acceptable’ (i.e., to ADB, borrowing
government, project beneficiaries, etc.) given the nature of the particular intervention
proposed. The risk analysis techniques are thus essentially used to complement
sensitivity testing in demonstrating project robustness.
It is also the case that comprehensive analysis of project risks helps in designing
projects so that different parties (e.g., borrowing government, operating entity, private
operators) can share and manage risks appropriately. In general, the principle to be
8 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

followed is that the parties who can best manage different sorts of risks to projects
(e.g., construction costs, operating costs, defaults, delays, etc.) should be allowed
to reap the rewards or bear the costs of risk management. Where project environments
include a private sector participant (e.g., in running a toll road, building a hospital),
appropriate consideration of all risk types enables them to be shared among the
participants; this type of analysis has been particularly well-developed within the
UK Treasury PFI methodology.
Therefore, although risk analysis (of whatever particular form) may typically
appear at the end of an economic analysis in a typical Report and Recommendation
of the President (RRP), it should be noted that, from ADB’s perspective, risk analysis
is fundamentally a project design tool. It is not simply an afterthought to economic
analysis.
CHAPTER 1 Introduction and Background 9

Technical Approaches
to the Analysis of Risk
in Project Economics

Introduction
2
This part of the Handbook provides a summary of available techniques for
dealing with the outcomes of unknown events in project design and economic analysis.
It begins with a conventional definition of the terms “risk” and “uncertainty”.
This is then followed by a summary of methods for dealing with the existence of
project outcomes which can only be modeled as uncertain, rather than risky. This
includes (but is not limited to) techniques already applied as standard practice by
ADB. It then summarizes the techniques available for modeling risk on the basis of
10 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

probability distributions (this is the most widely understood area of risk analysis),
and highlights both advantages and limitations of such approaches.
This is followed by consideration of issues involved in modeling what are
sometimes called “subjective” attitudes to risk—i.e., risk as perceived (for example)
by those investing in or affected by a proposed project. This is an important aspect
of risk analysis in project work because it is always the case that choice between
alternative risky outcomes involves knowing something about decision-makers’ or
participants’ preferences, and in particular how they are willing to trade off increased
rewards against increased risks (i.e., measuring the extent of individuals’ or
decision-makers’ risk aversion).

Risk and Uncertainty: A Definition of Terms

The term “risk and uncertainty” tends to be applied generically to the analysis
of situations with unknown outcomes. This document will follow the conventional
distinction between risk and uncertainty made in the literature [e.g., following
Renbourg (1970) from which the quote below is taken, and also Reutlinger (1970),
Pouliquen (1970), etc.].
In essence, risk is a quantity subject to empirical measurement, while
uncertainty is of a non-quantifiable type. Thus, in a risk situation it is possible
to indicate the likelihood of the realized value of a variable falling within stated
limits—typically described by the fluctuations around the average of a probability
calculus.
On the other hand, in situations of uncertainty, the fluctuations of a variable
are such that they cannot be described by a probability calculus.
Thus risk and uncertainty are best thought of as representing a spectrum of
unknown situations with which an analyst may be dealing, ranging from perfect
knowledge of the likelihood of all the possible outcomes at one end (i.e., risk) to no
knowledge of the likelihood of possible outcomes at the other (i.e., uncertainty).
It is important to realize at the outset that it is not the real-world situation
itself which is either “risky” or “uncertain”, but merely the information available to
planners and analysts which defines it as such. All actual project outcomes are
unknown, because they occur in the future and are subject to influence by a number
of variables, each of which may take different values. If we have reliable historical
or forecast data such that a probability distribution can be constructed for such
variables, the situation can be modeled as “risky”, if we do not have such data we
can only describe the future in terms of “uncertainty”.
CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 11

In dealing with an agricultural project, for example, if historical rainfall patterns


or irrigation water supply data exist we may be able to construct a probability dis-
tribution such that crop yields can be predicted in terms of expected values with
associated levels of variability. If we do not have such data then possibly only “high”,
“likely” or “low” values for crop yields may be estimated, depending upon whether
seasonal rainfall or water flow is above or below certain levels. Similarly, analysis
of an energy project may be undertaken in terms of “optimistic” or “pessimistic”
assumptions about domestic and commercial power demand levels (and different
returns predicted under such different scenarios), or may be modeled on the basis
of a distribution of outcomes of future power demand which itself depends upon
estimates of economic growth, population growth, etc., and which may be described
on the basis of their probabilities of occurrence. In both cases there is nothing inherently
different about the circumstances of the projects themselves, only the data available
to the analyst which makes modeling of risk more or less possible.
It should be noted that this distinction between risk (unknown but quantified
outcomes) and uncertainty (unknown and unquantified outcomes) is not usually so
clearly made in typical financial analysis. For example, the UK Treasury Taskforce on
Private Initiative (2000) quotes an accounting definition of risk as follows

“A simple definition of risk as used by the accounting profession is


uncertainty as to the amount of benefits. The term includes potential
for gain and exposure to loss.”

Clearly such a definition blurs the distinction made previously between risk and
uncertainty. It is argued that such a distinction is in fact very useful because it helps
to separate those situations which may be subject to quantitative analysis from those
which are not.

Allowing For Uncertainty

Project economic analysis (and the overwhelming weight of both ADB and
World Bank experience) tries to allow for the existence of unknown future outcomes
in the most basic sense by modeling the existence of uncertainty rather than dealing
with risk per se.
Attempts to model the impact of uncertain outcomes and develop decision
rules about what choices to make (e.g., between different projects or alternative project
designs) derive from the operations research (particularly linear programming models)
12 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

and game theoretic (e.g., von Neumann, Morgenstern) approaches of the 1960s
and 1970s. In situations of different possible project alternatives and uncertain
future events (“states of nature”), projects would be chosen on the basis of various
proposed criteria, according to decision-makers’ preferences. Such proposed
criteria included
• “Laplace”—select the project or design alternative which yields the highest
return, whatever the “state of nature” obtains
• MAXIMIN—select projects or design alternatives which yield the best
returns (e.g., highest NPV) if the situation/”state of nature” turns out as
badly as possible, and
• MINIMAX REGRET—select the project which minimizes the maximum
opportunity cost of having made a wrong choice by choosing a “state of
nature” which does not in fact obtain.

It can be shown that such criteria (and others that were developed around
the same time) are in fact all “irrational” in different ways. For example, the “Laplace”
criterion effectively ignores uncertainty altogether, the MAXIMIN assumes “nature”
to be as malevolent as possible (which is not the case), and the MINIMAX REGRET
does away with normal assumptions about decision-makers’ preferences (because
they are more concerned about minimizing losses ex post than about maximizing
returns ex ante).
Despite some historical applications for planning purposes, for such reasons
as just given, game theoretic criteria were largely abandoned as models of descriptive
or prescriptive behavior, and subsequent practice in ADB and elsewhere has largely
concentrated on describing unknown outcomes alone, without attempting to derive
decision rules to guide choice under uncertainty.
The most widely-applied technique for describing uncertainty is sensitivity
testing, and this is described in detail in the Guidelines (page 39, and Appendix 21),
and also in the Financial Guidelines (section 7.11). A full explanation of the technique
and its application is also provided in Belli et al’s Economic Analysis of Investment
Operations (World Bank Institute 2001).
In essence, sensitivity testing involves changing the value of one or more
selected variables which affect a project’s costs or benefits and calculating the resultant
change in the project’s NPV or IRR. Although emphases and presentation differ, both
ADB and World Bank recommend practices such as:
• testing for the effects of changes in aggregate project costs and benefits
• testing for the effects of changes in individual underlying variables (e.g.,
areas, yields, crop prices in an agricultural project; prices of cement,
operating costs of machinery in a roads project; consumer utilization rates
CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 13

in a power or water supply project, etc.). This choice of variables will usually
be based on previous similar project experience and/or detailed sector
knowledge as much as on the particular project in question
• testing variables one at a time, so as to be able to identify the ones with
most impact on project NPV
• testing for delays in benefits or implementation (e.g., shift the benefits
stream down a year or two)
• testing likely combinations of variables (especially if these may in practice
be linked —e.g., project costs go up AND implementation delays
simultaneously occur), and
• testing for changes in economic pricing adjustments (e.g., shadow wage
rate factor, shadow exchange rate factor, standard conversion factor, etc.)
made by the analyst.

Sensitivity testing leads to the calculation of switching values (SVs) and


sensitivity indicators (SIs):
• SV identifies the percentage change in a variable for the project NPV to
become zero (i.e., for the project decision to switch between “accept” or
“reject”, average yields would have to fall by 20%). Sometimes SVs are
expressed in terms of the absolute value of a variable—e.g., “if passenger
traffic volume fell to 15,500 vehicles per day the project would not be
viable”
• SI compares the percentage change in a variable with the percentage change
in a measure of project worth (e.g., NPV).

The prime utility of sensitivity testing is that it leads to the identification of


those variables to which a particular project design is most sensitive, and mitigating
action can then be taken (if desired) to minimize the consequences of such outcomes.
Likely mitigating actions include undertaking pilot projects, securing long-term
supply contracts (for inputs and/or outputs), increasing technical assistance and
training levels to support project implementation, publicity campaigns to promote
service usage, tax and tariff changes, etc. The technique is extremely easy to
apply, as changes to one value in a spreadsheet will reflect instantly in values
for NPV, IRR, etc.
However, the technique has a number of limitations:
• most fundamentally, it does not take into account the probability of the
occurrence of the events it models. The SV for crop yields may be a fall of
20%, or that for traffic flow may be 15,500 vehicles per day, but how likely
is it that either or both will occur in practice?
14 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

• where deviations from project “base case” estimates are modeled in


sensitivity testing, it is not clear whether the variations in values which
are being modeled are changes from “expected” values (i.e., the “base
case” estimate of the value of the variable is its average value) or are
deviations from “most likely” (or modal) values; depending upon the
characteristics of particular distributions (in effect the extent of skewness
in the data set), mean and modal values may be very different from one
another, and what is being captured in the base case and its variation is
not clear;
• the identification of appropriate groups of variables to vary together depends
on specialist knowledge, and misunderstanding the nature and extent of
correlation between variables can lead to erroneous results; and
• because the distribution characteristics of different variables which
determine project outcomes can differ enormously (the variability in
commodity prices is less than input prices for example, the variability in
power demand is less than in generation, etc.), the use of standard
percentages for variations (changes of +/– 10% or 20% are routinely applied
for example) in sensitivity testing captures quite differential extents of likely
variability. An impression of homogeneous variability is given, which is
not warranted by reality.

Overall, sensitivity testing is a highly subjective technique. Its ease of


application and familiarity of concept, combined with analysts’ understanding of
particular sectors and projects (which should lead to reasonably appropriate variable
selection and extent of variation being applied) plus its usefulness in leading to
development of mitigating measures and project redesign, mean that its use has
become widespread. However, its dependence upon judgment rather than empirical
evidence and its modeling of uncertainty rather than risk (plus its inability to offer
any decision rules following the presentation of its results) mean that its usefulness
as a technique is ultimately limited.
One other point may be mentioned in regard to modeling uncertainty in project
economic analysis. It is sometimes suggested that uncertainty can be allowed for by
either applying a different discount rate in the calculation of NPV or by using a higher
cutoff rate (i.e., greater than 10-12%) for investment decisions. While there is a large
theoretical literature on this point, in essence there is no justification for this approach—
apart from any other consideration (e.g., in determining what an appropriate “risk
premium” should be), it assumes that risk always increases with time, which is not
necessarily true. The discount rate is a rate of decline in the numeraire of economic
value, and has nothing to do with the source of risks facing an investment.
CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 15

Modeling Risk Quantitatively

Because of the conceptual shortcomings of all approaches to modeling


uncertainty, various attempts have been made to properly capture the impacts of
unknown outcomes through modeling risk quantitatively in project economic analysis.
As the Guidelines state, the purpose of quantitative risk analysis in essence
is to

“provide a means of estimating the probability that the project NPV


will fall below zero, or that the project IRR will fall below the opportunity
cost of capital.” (Guidelines, Appendix 21, page 156)

While the Guidelines do not themselves provide a methodology for the


application of risk techniques, it is suggested that the results of sensitivity testing
be used to consider which variable(s) may be appropriate to base a risk analysis
upon (i.e., those that have major impacts on project outcomes). Having identified
particular variables, a number of possible data points (i.e., values above and below
the “base case”, upper and lower limits to data values, etc.) are necessary to be
specified, together with the frequency (or likelihood) of each of these values occurring.
From such data points and associated frequency estimates, a probability distribution
can be constructed for the variable(s) in question.
Table 1 shows the numbers of bus passengers who may be expected on a
particular route per week (and which may represent, for example, usage of similar
services elsewhere in the city for an earlier project). The average number of passen-
ger bus trips is 50,000, the standard deviation (the average difference between all
observations and the mean/average) is 27,386 and the coefficient of variation (the
ratio of the standard deviation to the mean) is 0.547.
The figure below shows graphically a simple probability distribution (based
on frequency of observations falling within particular class intervals) for the number
of bus passengers, deriving directly from the data in the table.

Table 1
Number of Bus Trips and Frequency of Occurrence

Number of bus trips (‘000) 10 20 30 40 50 60 70 80 90

Probability of occurrence 0.025 0.05 0.1 0.2 0.25 0.2 0.1 0.05 0.025
16 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Figure 1
Passenger Bus Trips Probability Distribution

From the data, it is apparent that the expected number of bus trips is 50,000
(and this may be the basis for the estimate of the project “base case” scenario), but
there is a 10% chance that the number of passengers may only be 30,000 per week,
and a 5% chance that the number of passengers may only be 20,000 passengers
per week.
The following figure (Figure 2) shows the same distribution in the form of a
cumulative probability distribution function (which has been “smoothed” to allow
for the effect of observations of grouped data). The function plots the cumulative
probability that the actual outcome will be below a certain level of bus trips. It can
be seen that, for example, there is a 27.5% probability that the number of bus
passengers will be below 40,000 a week. In this example, actual numbers of passen-
gers on project-financed buses each week may well determine both the financial
profitability of the bus company and also the size of the economic NPV of the project.
It may be the case, for example, that sensitivity testing has indicated that the SV
(switching value—the value of the variable at which the project investment decision
is changed) for bus passengers is at or near 40,000 per week—and we can now say
what the probability is of this value obtaining. In this case, the probability is rela-
tively high (i.e., over one chance in four), which may well imply that a redesign of
the proposed project would be appropriate.
CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 17

Figure 2
Cumulative Probability Distribution

The ultimate purpose of quantitative risk analysis is to generate a similar


cumulative distribution probability function for a project’s NPV, such that the
probability of negative NPV being generated is explicitly identified. The following
figure shows such a chart:
Figure 3
Cumulative Distribution Function
18 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

The probability of project NPV being negative as a result of variability in


underlying factors is thus able to be identified (i.e., 30% in the figure), and provides
further information as to the relative attractiveness (in terms of its riskiness) of the
project. The same data could of course be used to model the financial profitability
of the bus company in a similar fashion.
With the discussion so far, and most literature describing such techniques,
it should be noted that identical procedures to these can be applied to projects where
expected ENPV is not typically calculated in ADB practice (e.g., education and health
projects, which tend to use measures of cost-effectiveness rather than total net
economic worth). The only difference is that instead of expected NPVs would now
be absolute measures of expected cost-effectiveness of outputs or impacts (e.g., annual
cost per health worker employed, cost per primary school pupil educated) quoted
together with distributions for those values (e.g., in 75% of circumstances the cost
per health worker would be less than Baht 4,000 per year; the cost per primary school
child educated would be Rupees 21,000 in four years out of five, etc.).
Risk analysis typically involves the choice of several variables to be varied
simultaneously, as project returns are generally subject to more than one source of
risk. Because of the mathematical complexity involved in such calculations, the analysis
of risk in this form is invariably undertaken by some kind of computer software. The
process which is followed (and which is usually referred to as “Monte Carlo” or
simulation analysis) is that values for individual variables are generated randomly
according to their respective probability distributions, combined with other randomly-
generated values for the other variables, and these figures are used to calculate an
estimate of the project NPV. This process is repeated a large number of times (a
number which is specified by the analyst—in effect, equivalent to implementing the
project again and again in different circumstances—and is usually at least 1000 times,
and typically more than this) and an average (or “expected”) NPV is produced together
with an associated probability distribution. This distribution can usually be viewed
in the form of charts like those in Figures 1 to 3.
The early literature on risk modeling (e.g., Reutlinger 1970, Pouliquen 1970)
and also standard texts on project appraisal (e.g., Little and Mirrlees 1974, Squire
and Van Der Tak 1975, etc.), as well as ADB’s 1985 review, all mention the fact that
computer time and expertise is likely to be a major constraint to the use of this
technique. In recent years this constraint has largely been overcome and more than
adequate computational facilities and software are now available to practitioners.
There are even examples of risk modeling using spreadsheets alone (see Clarke
and Low 1993, for example). In the model described in this particular article, a standard
spreadsheet (e.g., Excel, Lotus) is used to generate random values (through the use
of a built-in function—@RAND in the case of Lotus) which are then repeatedly applied
CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 19

to variables identified within the project “base case” scenario so as to generate a


distribution of outcomes (counts of observations of variables/expected NPV values
are recorded in cells so as to construct a frequency distribution). Expected NPV and
IRR for the project are calculated based on the distribution of results, and the base
case is then presented in terms of expected NPV/IRR, plus a range in which outcomes
will lie 90% of the time (based on all variables being subject to simultaneous change).
It is to be noted that the technique is quite simple and practicable, and requires no
more information than is already required for traditional sensitivity testing.

Technical Issues in Modeling Risk: Two Considerations

However, despite the overcoming of computational limitations to the appli-


cation of such techniques, two major practical considerations (and possibly constraints)
remain as regards the extent to which such techniques as Monte Carlo simulation
can be used in project preparation situations.
Firstly is the issue of data availability, and the extent to which the situation
can reasonably be defined as risk (as opposed to uncertainty) through the construction
of a meaningful probability distribution of outcomes. The actual situation with data
availability is likely to vary enormously both between project situations and also
across different sources of variability within any one project environment. At one
extreme, large volumes of reliable cross-sectional or time series data may be available
from historical sources for the variable concerned (e.g., for rainfall, for commodity
prices, for traffic flows). At the other extreme may only be the existence of a few data
points (e.g., “most likely” values, absolute minimum possible, maximum possible,
etc.) which are expectations of experts/analysts involved in preparing the project.
Other possibilities lying within these bounds include the forecasting / specification
of power-generation theoretical capabilities adjusted for a set of likely different
operating conditions, forecasts of trade flows and commodity prices (e.g., based on
World Bank publications taking into account world supply and demand factors), etc.
Software such as @RISK often has capabilities to fit probability distributions of different
types to raw data sets supplied by the analyst. Such routines will fit distributions
to data and also provide a measure of the goodness of the particular resulting fit.
It is important to note, therefore, that very large and complete data sets from
empirical sources are not always necessary for the undertaking of risk analysis.
Simplifying assumptions about variable distributions can be made—as a bare
minimum, triangular distribution from three points (i.e., “most likely”, “minimum
possible”, “maximum possible”) can be constructed based on “best guesses” of project
20 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

preparation team members. There is also often considerable expertise within the
project preparation environment about the likelihood of variables or outcomes which
may not be available from official sources but which can be elicited from potential
project participants. Good examples of such knowledge might include rainfall and
water flow, crop yields, time taken to collect water or firewood, or travel to market,
operating efficiency of agroprocessing machinery, number of family days sick per
year, etc. The ‘Delphic’ method of eliciting opinion from local experts is an example
of this type of approach, and has been applied (in a probability-based form) by, for
example, World Bank in a risk analysis of institutional reform in the irrigation sector
in Pakistan (Dinal et al. 1997).
Well-tried empirical methods exist for developing probability distributions
from such subjective sources. These include
• visual impact techniques (e.g., matches or stones piled up to represent
frequency of value occurrence),
• structured questions to identify key points in a distribution (e.g., the median,
quartiles, etc. – the “judgmental fractile” method), and
• the application of “smoothing” techniques in situations where a few real
data points may be available.

Some proponents of probability-based risk analysis (e.g., Clarke and Low 1993)
also argue that the shapes of particular distributions of individual variables (e.g.,
choosing between uniform, normal, triangular distributions for variables such as crop
yields, traffic flow, enrolment rates, income differentials, etc.) are less important than
the choice of variables themselves which are allowed to be modeled. In the Clarke
and Low example (from an agricultural project in East Africa), the random number
generation approach necessarily produces a rectangular uniform distribution (i.e.,
one in which all possible observations within the range defined by the analyst have
equal probability of occurrence), although with more complex formulae being written
this could be adjusted. Recent experience of preparation of power projects within
ADB also suggests that the particular form of distributions also matters less if a
large number of simulations are run. Even when considerable effort is made, for
example, to replace the quoted discrete distributions with relatively few values by
continuous distributions based on large amounts of empirical data, there is little
difference in resulting distributions of EIRR/ENPV outcomes (i.e., expected values
and variance, minimum and maximum values, etc).
However, this approach may not always be appropriate for all variables, and
it still requires judgment on the part of the analyst about what ranges are acceptable
for values to fall within. Also, to adjust the spreadsheet model to produce (for example)
a normal distribution (as opposed to a uniform one) becomes very much more complex.
CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 21

It is also noted in the early risk literature (e.g., Pouliquen 1970) that there is no a
priori case for the use of normal distributions (even as a default distribution), as all
variables are not always subject to relatively large numbers of random influences
(which is what typically causes a variable to be normally distributed).
Overall, therefore, the techniques applied to develop definitions and derivations
of probability distributions for individual variables in most cases is likely to depend
upon some subjective judgment by an appraisal team — and inevitably the extent
to which these design assumptions adequately reflect the reality of the project will
vary from case to case. The suspicion that what appears as a full-scale risk analysis
has in reality only a spurious precision can be ultimately only fully allayed if the data
upon which the variables’ probability distributions are constructed (either from historical
evidence, future computational forecasts, or analysts’ ‘best guesses’) are believable.
The second consideration when applying risk analysis in practice is the extent
of covariance between those variables that are to be selected for risk analysis. Projects
are rarely subject to only one source of risk, and therefore more than one variable at
a time is modeled in the Monte Carlo simulation exercise. However, statistical
complexities can arise depending upon the relationships between the selected
variables. Where variables are in fact statistically independent of one another there
is no problem, as it is appropriate to treat them independently. Where variables may
be thought to be related in some way, however, the extent of covariance between
them needs to be taken account of when specifying the distribution of individual
variables in some type of simulation (again, typical risk analysis software such as
@RISK can handle this within the context of specifying correlation matrices).
As an example, project revenues are typically products of both quantities sold
and prices obtained. If these underlying variables are correlated in some way (which
may well be if project output is large relative to market volume, and negatively so
in this case) the expected value of the product of two random variables (i.e., project
revenue) is equal to the product of the individual expected values plus the covariance
between the two variables. Another typical example of covariance (which would be
positive in this case, if it is assumed to be due to improved, project-induced water
supply under an irrigation scheme improvement) may be that between area planted
and average yield (i.e., with both variables as determinants of farm production
volumes). One of the case studies using @RISK includes correlation between estimates
of proportions of trainees finding employment, and the numbers of days and months
of employment gained following nonformal training in Bangladesh. In practice, the
approach to assigning particular levels of covariance between variables is quite
pragmatic, and typically simple rank correlation coefficients between pairs of variables
are sufficient for most purposes (in the risk software packages, correlation between
variables—once specified—can typically be “toggled” on/off).
22 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

It is specifically recommended that disaggregation of individual variables be


limited as much as possible so as to avoid including too much correlation in the
analysis. For example, although individual construction cost items (e.g., cement, cost
of floor, cost of walls) may each be thought to vary individually, in reality the sources
of this variability all arise from one point (e.g., costs of imported cement), and this
could be most appropriately captured through some item such as “construction
materials” rather than by introducing additional correlation between such items (which
would tend to increase unnecessarily the estimate of overall variability). Akin to the
nature of some subjective judgment being involved in the allocation of probability
distributions, there is therefore a similar judgment to be made about the extent of
disaggregation to be applied in individual circumstances.
In sum, the principles to be applied in practical situations to quantitative risk
analysis such that the issues just described are dealt with as transparently as possible
are summarized in the following table:

Table 2
Principles to Apply in Data Handling for Probabilistic Risk Analysis

Principles to Apply
1 Identify those variables for which future values are unknown and which are likely to
affect project returns (i.e., the ‘key’ variables)
2 Fully explain the general nature of the data set which is used for modeling those variables’
values (its origin—i.e., from objective or subjective sources, whether it is based on historical
observations or forecasted projections, the number of observations the data set contains,
its extent of completeness/any missing data points, etc.)
3 If the data derives from subjective sources, explain the method by which it was elicited
(e.g., from visual techniques, from subjective questioning, from an expert-based ‘Delphic’
process, etc.)
4 Explain the statistical nature of those variables’ assigned probability distributions (i.e.,
whether these distributions are triangular, uniform, normal, logarithmic, exponential, etc.)
5 Make clear the goodness of fit of the distribution to the data set (if one has been fitted
using @RISK or similar software), and quote appropriate statistical measures (e.g.,
Chi-square, Kolmogorov-Smirnov, Anderson-Darling statistics, etc.)
6 Make explicit any correlation thought to exist between variables used in the risk analysis (i.e.,
its extent, and the technical, real-world basis for the assumption, etc.), and (based on this)
7 Explain and justify the extent of any variable disaggregation.
CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 23

The focus of the reporting of the results of risk analysis will be the likelihood
that project returns may be negative (e.g., “the project’s rate of return is likely to be
unacceptable in about 15% of all cases”), but such reporting should not obscure any
basic qualifications about the analysis which need also to be included, and especially
about the extent of correlation which has been assumed. “Good practice” in this
regard is likely to require reporting “with” and “without” correlation being applied
in the simulation as the effects of ignoring correlation can be substantial.

Risk Analysis, Decision-Making, and Welfare

The result of risk analysis as just described is therefore to identify projects


(or alternative designs of the same project) which now have two essential
characteristics—i.e., the expected value of their economic return (as measured by
their expected ENPV, EIRR, etc.) and their degree of risk (as measured by their vari-
ability in general—captured by the distribution’s measures of dispersion, such as
variance and coefficient of variation, and also by the probability of the return falling
below some unacceptable level in particular). This quantitative measure of risk adds
to the information available to the decision-maker, although in itself does not
necessarily provide any guide as to whether any individual project is acceptable
(or even as to which project among several possible ones actually should be
undertaken).
The discussion of the ‘acceptability’ of particular levels of risk is usually
presented in standard economics texts in terms of choice among competing projects.
Consider the following three project alternatives (A, B, and C) as shown in Figure 4.
In Figure 4, project B is clearly “inefficient”, in the sense that its variability
is the same as project A, but its expected value is lower. Project C has a higher expected
NPV than project A but it also has a greater variability of returns (including the
possibility that its return will be zero). If variance of returns is plotted against expected
values for such project situations as A, B, and C, the following is obtained (Figure 5 -
in what is usually referred to as E-V space). Projects A and C may lie on the efficient
investment frontier—although at different points, while project B lies within the
inefficient frontier.
In this case, the expected higher returns of project C have to be weighted
against the increased degree of risk of the project. How should a decision be made
between the alternative projects? The traditional view taken towards public sector
investment was that governments with large project portfolios could afford to ignore
the riskiness of investments as long as the expected values were acceptable—i.e.,
24 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Figure 4
Probability Distribution: Three Projects

Figure 5
Expected Returns-Variance Frontier
CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 25

they could afford to be “risk-neutral”. This is because, with a large number of


investments spread across all of society, the costs of any individual project failures
could be absorbed within the portfolio as a whole. Exceptions to this view were
projects which were either very large, or were somehow correlated with national
circumstances (such that good performance of the project in bad years for the economy
as a whole was worth more in terms of a disproportionate contribution to national
income), or affected particular groups (e.g., in one region, one type of student, etc.)
such that the impact on those particular individuals could not be ignored.
In fact, there is no answer to the question of project choice in such circumstances
without reference to knowledge about the extent of decision-makers’ risk-aversion
(i.e., the rate at which they are prepared to trade-off levels of expected returns—or
in effect, income—against levels of variability of returns). Figure 6 shows how many
individuals are thought to be risk averse; the line joins points of equal utility (i.e.,
welfare or satisfaction – and is in fact an indifference curve) from combinations
of expected income (measured, say, in $000) and variability of income (measured
by annual income variance). What it suggests, in general, is that for people to accept
greater variability in their incomes they need to receive increasingly larger expected
incomes.

Figure 6
Risk Aversion: Line of Equal Utility
26 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

The actual nature of decision-makers’ utility functions as regards their risk


aversion occupied a large literature (much of it deriving from agricultural economics
and work with small farmers in developing countries) around the same time as the
classic texts on project appraisal and risk analysis were written, where it was hoped
that insights about preferences regarding risk would lead to making optimal project
choices. The results of many classic empirical studies of risk perceptions from the
1970s (e.g., Anderson 1974, Swalm 1966, etc.) suggested that most individuals were
risk averse, although the specific income levels at which this was the case and also
the extent of risk aversion could vary enormously even within apparently similar
circumstances (small farmers in Africa, Australia, and the USA, workers in large
corporations, professionals and business executives were all studied).
In order to fully understand how people actually deal with risk in project and
other situations in terms of the decisions they make, academic risk literature has
also concentrated on the extent to which
• the consequences of particular risks are catastrophic or not,
• the risks are controllable at the micro level or not,
• the consequences are reversible or not, and
• the risks are insurable or not.

These considerations are returned to in Part IV in relation to the attitudes of


the poor.
To complete the above analysis and understand what decision actually should
be taken in any particular circumstance, it is necessary to combine the notions of
efficient projects located in E-V space with decision-maker’s utility functions expressed
in the same space. Thus in Figure 7 below, one decision-maker on the basis of an
individual utility function (U1) will choose project A (lower expected NPV, lower risk)
while another will choose project C (higher expected return, higher risk) on the basis
of a utility function such as (U2).
As well as the E-V approach to considering the risk-reward relationship, other
models have attempted to capture the characteristics of this situation within the
context of financial investment portfolio analysis. Probably the most well-known of
these approaches is the Capital Asset Pricing Model (CAPM, developed by Sharpe
and Lintner in the 1980s) which measures an individual stock (or project) risk relative
to the volatility of returns relative to a market (or sector) index. Again, however, ADB
practice as regards project economic and financial analysis is not primarily concerned
with a number of projects as they may comprise an investment portfolio (or even
with one project within the context of all projects in the portfolio), but with analyzing
risk as faced by individual projects one at a time.
CHAPTER 2 Technical Approaches to the Analysis of Risk in Project Economics 27

Figure 7
Project Choice and Risk Aversion

Summary of Approaches to Risk Analysis

This Part of the Handbook has summarized approaches to dealing with


unknown outcomes in project analysis through description of various techniques
attempting to capture the content and consequences of uncertainty and risk. It is
clear that the quantitative modeling of risk is in principle preferable to the simple
depiction of uncertainty, although it is also obvious that data and time considerations
(and the difficulties in properly identifying and specifying covariance among variables)
have often limited the extent of actual risk analysis practice. However, it is also the
case that the present availability of computer software to easily process Monte Carlo-
type simulations from data already available to the analyst within spreadsheets, plus the
existence of statistical routines and computational processes to fit probability dis-
tributions to many (even quite sparse) data sets, greatly increase the possibilities for
application of quantitative risk analysis as a more commonplace part of project design.
The results of quantitative risk analysis can greatly inform the process of project
design, so that mitigation measures can be put in place before projects are
28 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

implemented. They can also increase the information available to decision-makers


about individual projects (i.e., so that their likelihood of failure is identified). While
the results of risk analysis are quite easy to understand in conceptual terms for planners
and project staff, however, these results in themselves do little to aid project investment
decision-making unless and until they are combined with some consideration of
planners/policymakers and/or project participants’ risk aversion. What level of risk
is it appropriate
• for ADB to accept?
• for ADB to suggest that borrowing DMC governments or institutions accept?
• to impose upon groups of project beneficiaries?

are questions which have to be answered outside the scope of those quantitative
techniques which provide the measures of project risk. The question is akin to the
difficulty of suggesting a priori cut-off points for the Poverty Impact Ratio (PIR) as it
is inherently associated with social welfare functions as perceived by decision-makers.
Following a review of agency experience with the application of the techniques
just outlined (in Part III), it will be suggested (in Parts IV and V) that strengthening
risk analysis in ADB operations involves both greater application of the probability-
based techniques for modeling sources of risk in project design (located within a
framework to guide such application) and also greater awareness of attitude towards
risk amongst those being planned for in ADB projects.
CHAPTER 1 Introduction and Background 29

A Summary of
Development Agency
Experience in
Applying Risk Analysis

Introduction
3
This Part of the Handbook reviews ADB’s and other agencies’ actual experience
with risk analysis in recent years.
The review of ADB experience is based on a review of some 50 recent RRPs
covering projects in all sectors of lending operations to identify what risk analysis
practices have been employed by ADB in practice (see list in Appendix 1). In addition,
various project performance appraisal reports, impact evaluation studies (covering
the late 1980s and 1990s), special studies, and sector syntheses (from 1994 to 2000)
30 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

were reviewed to find out to what extent those variables which were thought to
have affected project impact during implementation were in fact those which had
been identified as sources of risk or uncertainty at the time of project preparation
and (if they had been so identified) to what extent any attempts had been made to
model them as “risky”.
The experience of the World Bank is also considered in this regard, and a
number of projects were briefly reviewed which are propounded (e.g., in Belli et al.
2001) as examples of “good practice” of risk analysis. It will be seen that the actual
practice of risk analysis falls somewhat short of what may appear to be advocated
in its official publications.
The experience of some other agencies with approaches to risk analysis is
also considered, notably that of the UK’s Department for International Development
(DFID 2001) and also the UK Treasury (2000). DFID’s experience is notable for the
purpose of the current study because that agency previously adopted a quite traditional
approach to project economic analysis (including its technical approaches to risk
analysis) but has now (along with most bilateral development agencies) shifted its
analytical focus away from quantitative cost-benefit techniques and towards a more
participant-centered and “pro-poor” driven “livelihoods” approach (within the context
of pursuing poverty reduction as its overall objective). However, within this
“livelihoods” framework (which is also increasingly being favored by World Bank)
the need to assess vulnerability of target groups has actually increased the attention
devoted to risk assessment, although not in a quantitative sense. DFID’s experience
and practice is also noted for the rigor of the application of the logical framework
technique and the explicit attention to identification of sources of risk within this
framework.
Actual experience with the use of risk analysis by major agencies is then
considered within the context of a recent critique of project economic analysis by
Harberger (1998). It is suggested, inter alia, that various biases are routinely introduced
into estimation of project costs and benefits, and that among the ways of dealing
with such unknowns greater attention should be paid to risk analysis (through the
application of simplified simulation techniques).

ADB Experience With Risk Analysis

With the notable exception of a port project, an agriculture project, and the
few power sector projects described below, ADB’s experience with quantitative risk
analysis has been very limited, although the existence of risk as affecting project
CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 31

outcomes is indeed very well-recognized. A review of some recent RRPs, and which
cover all sectors of ADB project and policy lending operations, suggest the following
main points:
• the standard format of an RRP invariably includes a section (usually
within the ‘Financial and Economic Analysis’, but sometimes presented
separately and called variously ‘Risks’, ‘Risks and Safeguards’, ‘Risks
and Assurances’, etc.) describing qualitatively the risks which a project
is expected to face
• this text usually (but not always) includes descriptions of the measures
which have already been incorporated to mitigate such risks. Typical risk-
mitigation measures include the provision of technical assistance to
strengthen institutions, the provision of counterpart funding, and also the
inclusion of a number of different components being included within the
project ambit. Sometimes such identified risks are called ‘micro’ or ‘project’
risks (i.e., they are risks over which the project has some control), and are
distinguished from ‘macro’ or ‘sector’ risks (over which the project has no,
or very limited, control but which are not regarded as being significant
enough to jeopardize project success)
• this is often followed by a section called ‘Assurances’ which contains state-
ments/letters to ADB from the borrower regarding operations during project
implementation, often designed to formalize and support the proposed
mitigating measures which have been agreed upon
• Appendix 1 of the RRP is invariably the ‘Project Framework’, which identifies
and summarizes the risks as previously discussed, and places them in the
context of the project’s hierarchy of objectives. There is no separate
discussion here of risks’ likelihood and seriousness, however
• the ‘Financial and Economic Analysis’ section of an RRP often argues that
‘conservative’ or ‘pessimistic’ estimates have been used for forecasts of
(for example) shipping, road and rail traffic volumes, crop prices, yields
and production, fisheries catches, etc., upon which benefits have been
estimated
• in addition, often the fact that certain benefits are identified but not
quantified and valued in EIRR calculations is used to argue that the
‘base case’ EIRR is ‘conservative’ or ‘understates’ real returns. There is
thus a ‘cushion’ which supports the likely acceptability of project
returns
• the ‘Financial and Economic Analysis’ section usually contains a fairly
standardized approach to sensitivity testing, in which project aggregate
‘base case’ costs and benefits streams are changed by 10% or 20% each
32 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

and benefits are delayed by one or two years, and the effects of these on
the project ‘base case’ EIRR are considered (both in isolation and in
combination with one another). Some studies are now more detailed, and
calculate switching values and sensitivity indicators
• the project is then usually described in terms of its ‘robustness’, i.e., in the
‘worst case’ scenario (when cost changes are highest, benefits most reduced
and/or delayed, etc.) its EIRR is still above 12%; sometimes, the statement
is (improbably) made that, given its ability to survive such adverse
circumstances, ‘the project faces no risk’.

This format is remarkably similar across projects in all sectors. In the case of
program loans, qualitative discussions of risk are included in the text, and the policy
matrix will usually contain a column describing ‘actions planned’, many of which
deal with management of identified risks.
ADB practice recognizes that risks exist at different levels of objectives
achievement. It also tends to closely link identified specific risks with those mitigating
measures which are already included in project or program design. Moreover, it
routinely exploits sensitivity testing to demonstrate ‘robustness’. While ADB’s practice
could be argued to be very strong, it could not be said to incorporate quantitative
risk analysis practice in any form.
The apparent exceptions to this conclusion are the Bintulu port project, an
agriculture project, and several very recent power projects. The Bintulu port project
was cited and extensively described in Johnson (1985). The analysis of this project
involved the estimation of probability distributions of several types (triangular,
trapezoidal, uniform) for different elements of the costs and benefits streams and the
construction of a probability distribution of the EIRR based on 300 samplings/
replications. The output of the analysis is described in terms of the probability of the
EIRR being more than acceptable (i.e., in excess of 12%) in 97% of samplings. What
is perhaps interesting to ask, is why such a probability-based analysis was undertaken
in 1979 (when presumably access to computational resources was far more restricted
than is the case in recent years) for a ports project and yet was not for later similar
ports projects (e.g., the Xiamen Port Project in 1997, the Belawan, Banjarmasin and
Balikpapan Project, also in 1997). In both these later cases, references to ‘conservative’
traffic forecasts are combined with extensive sensitivity testing as ways of dealing
with unknown future values.
Recent preparation of power projects in ADB have systematically employed
risk analysis, and, interestingly, this has been very much along the lines suggested
by Harberger’s 1998 argument and in a similar fashion to the World Bank Mexican
irrigation and power project examples (see below). In the cases of two ADB power
CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 33

projects1 for example, discrete probability distributions containing between three


and five possible values (based on PPTA mission estimates) for five to six variables
in each case (covering capital and other cost elements, commissioning delays,
willingness-to-pay (WTP) estimates, foreign exchange, as well as aggregate financial
costs and benefits) were used to run a Monte Carlo simulation which generated
expected values for project EIRR, together with estimates of standard deviation (i.e.,
the square root of variance), minimum and maximum values. A cumulative distribution
function was also plotted in each case, indicating the probability of negative or less
than acceptable (i.e., below 10-12%) economic rates of return. Each simulation was
run 3,000 times using “Risk Master” software. The models also incorporated
estimations of correlation between variables. In addition, the most recently-prepared
of these projects (the Shen Da project) included a risk analysis following on from the
calculation of the PIR, and a cumulative distribution function (CDF) was estimated
for the PIR itself (see discussion in Part IV of the utility of this exercise).
This approach to risk analysis is certainly pioneering within ADB at present
(and in fact relies upon the use of a personal, rather than institutional, copy of ‘”Risk
Master” software). It is suggested by staff working in the power sector that it has
been very useful during project design (typically at the fact-finding stage rather than
at PPTA) to investigate concerns connected with one or two ‘key variables’ (typically
elements of input or output prices). The main effort in terms of additional work lies in
selecting key variables, ensuring there is no correlation between them (or that its extent
is fully understood), and designing a ‘best guess’ distribution for each variable. With
familiarity of the sector and/or local conditions this exercise may be completed over
the course of a few days, and running a simulation based on several thousand samplings
will take only a few hours. The result is certainly a presentational improvement, and
usually argued to be an increase in ‘peace of mind’ regarding project robustness.
Another exception concerns the Infrastructure for Rural Productivity
Enhancement Sector project in the Philippines which used the @Risk software to
simulate the impact of changes in the values of key variables on the feasibility of
proposed investments in rural roads and irrigation development. The analysis applied
triangular distributions constructed around the mean for key variables, whose upper
and lower values were apparently based on subjective estimates of project designers.
The results of the analysis presented features of a distribution of the expected val-
ues for the EIRRs of each of two project components (i.e., rural roads, irrigation), but
did not present the probability of the overall EIRR being less than the opportunity
cost of capital.

1 People’s Republic of China’s Windpower Development Project, and People’s Republic of China’s
Shen Da Power Transmission and Grid Rehabilitation Project.
34 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Given that the estimation of risk in this way relies only upon a few estimates
of likely values of several key variables and also some estimate of correlation among
those variables, there would seem little reason as to why its use should be restricted
mainly to ADB power projects. Power projects may be typically relatively large
investments, but they are not necessarily more prone to risk from exogenous or
endogenous sources than (for example) transport or irrigation projects, and technical
staff involved in preparing such projects are likely to have just as much basis (deriving
from both historical data and informed projections of the future) for estimation of
probable outcomes as do ADB energy staff.
Indeed, a review of ADB project performance across all sectors (based on
sector synthesis reports) suggests that many events occur to project environments
during implementation which are responsible for divergences between estimated
economic outcomes (i.e., EIRR, ENPV) as anticipated at appraisal and those estimated
at post-evaluation and which in fact could have been subject to risk analysis at the
time of project preparation. Table 3 summarizes (by sector) what were the main
technical factors causing differences between anticipated and actual outcomes. What
is immediately clear is that most of the factors which have been so identified could
have been subjected to some form of risk analysis, and perhaps better project (re-)
design may have resulted if risk analysis techniques had been applied.
What also emerges from the table is that other factors which cause differences
in economic returns as estimated prior to and post-project implementation (typically
depending upon how benefits—especially environmental ones or those based on
willingness-to-pay – have been approached) could have been subject to greater
variation quantification as well. Overall, the evidence suggests that there is a strong
prima facie case for more rigorous risk analysis in ADB project economic analysis.
Adding to such findings, a ‘Special Evaluation Study’ (1998) of factors affecting
project performance in agriculture and social projects between 1991 and 1997
suggested that ‘greater realism’ was required at PPTA stage in estimating project
costs and benefits so as not to consistently over-estimate EIRRs, and that particular
attention should be paid to dealing with aspects of local government performance
in project preparation. Again, these considerations could at least in part be tackled
through greater application of risk analysis. Similarly, the special evaluation study
on involuntary resettlement (2000) concluded that social investigations at the PPTA
stage had often been weak and/or hurried in preparation, and that greater attention
to project participants’ risks (e.g., of impoverishment) was warranted.
In addition to sector-based work, a 1997 special study of the macroeconomic
environment and project performance (across all sectors and from 1980 to 1997) in
Sri Lanka found that numerous variables such as world market prices, the nominal
exchange rate, scale of duties and tariffs, shipping costs, various government controls,
CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 35

Table 3
Sources of Differences Between Estimates of Economic Benefits,
Pre- and Post-Implementation, by Sector

OED Sector Factors Affecting Estimates of


Synthesis Economic Outcomes Comments
Irrigation and Cost overruns; implementation delays; All conceptually subject to risk analysis
rural development untested technologies; cropping
intensities; poor water management
Rural and Repayment rates; debt amnesties; rates Many (if not most) conceptually subject
agricultural credit of interest; government commitment to risk analysis
Fisheries Price trends; numbers of vessels; fish All conceptually subject to risk analysis
stock composition
Forestry Benefit estimations and valuations Difficulties in estimating non-direct use
values; could have been approached using
sensitivity analysis, and/or “with or with-
out” inclusion, and perhaps risk analysis
Industrial crops Yields and prices; cost overruns; mills All conceptually subject to risk analysis
capacity and throughput
Health and Institutional performance; health, mor- Many conceptually subject to risk analysis
education bidity and mortality levels; cost of ser-
vices (participation rates and demand)
Urban development Land values; benefit valuation methods Some conceptually subject to risk analy-
and housing sis, and land valuation could have been
approached using sensitivity analysis, and/
or “with or without” inclusion, and perhaps
risk analysis
Water supply Cost overruns; WTP estimation Some conceptually subject to risk analy-
and sanitation sis; WTP estimates could have been
approached using sensitivity analysis, and/
or “with or without” inclusion, and perhaps
risk analysis
Power Cost overruns; capacity and generation/ All conceptually subject to risk analysis
transmission losses; WTP/resource cost
savings estimates
Roads and transport Implementation delays; cost overruns; All conceptually subject to risk analysis
traffic flows; vehicle operating cost
(VOC) savings
Ports and shipping Implementation delays; cost overruns; All conceptually subject to risk analysis
traffic volumes; benefit estimation
methodologies
36 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

etc. had all affected project outcomes. An explicit conclusion of the study was that
greater attention to analysis of these variables at project design might have improved
project effectiveness. An implicit conclusion may also be that quantitative and/or
qualitative risk analysis techniques could have been employed to consider such
variables more carefully.

World Bank Experience With Risk Analysis

The World Bank devotes relatively more of its basic publication on project
economic analysis (Belli et al. 2001) to the analysis of risk than does ADB in its
comparable documentation (i.e., ADB 1997a). In this publication, a discussion of
sensitivity analysis and its shortcomings is followed by a presentation of the principles
of calculations based on mean-based expected values and the use of Monte Carlo
simulation (including a hypothetical example). The problems of developing probability
distributions for variables are discussed, and there is some description of judgmental
methods applied to derive distributions in the absence of complete data sets. There
is also a discussion of the implications for decision-making of risk analysis, concluding
that (risk analysis is perhaps of limited use in decision-making terms due to
governments’ supposed risk-neutrality) the techniques are more likely to be of use
in the processes of project design and redesign, rather than in decision-making per se.
Overall, the presentation of the material is relatively thorough, if somewhat
traditional —being oriented mainly around the use of Monte Carlo techniques, and
specifically recognizes the fact that the kind of analysis and presentation of issues
associated with risk which are advocated are “extremely rare in existing documents”.
World Bank accepts that in situations when projects are large with respect to a
particular region or group of people the ENPV criterion alone (i.e., without any
associated measure of risk) is an inadequate measure of investment acceptability—
but does not pursue the consequences of this thinking towards (for example) pro-poor
project lending.
Two projects with risk analysis are given as examples of somewhat different
types of “good practice”. The first, from an irrigation project in Mexico, identifies
three sources of risk (inadequate government counterpart funding, delays in sur-
veys and studies, and unwillingness to invest/ problems with access to credit) which
(collectively) have two sorts of impacts—implementation delays and low adoption
rates. These two impacts are each then divided into three scenarios (“optimistic”,
“modal”, “pessimistic” in the case of adoption rates, and delays of 0, 1, and 2 years
in the case of implementation) with estimated probabilities of occurrence (0.1, 0.5,
CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 37

0.4, 0.35, 0.4, 0.25, respectively), and EIRRs calculated for the resulting nine (i.e.,
3*3) possible different scenarios. The result of this approach is a table of events (i.e.,
combinations of adoption rates and delays) with their associated probabilities and
EIRRs. It is shown that even under the most pessimistic of assumptions (in this example
this results from pessimistic adoption rates and a 2-year delay in implementation),
which has a probability of occurrence of 0.1 (calculated from the multiplication of the
likelihood of 0.25*0.4) the calculated EIRR (of 12.7%) is still in excess of the opportunity
cost of capital (assumed to be 12%). A cumulative distribution function for the range
of EIRRs is shown in tabular form, and could have been presented graphically as well.
This sort of approach is very similar to that advocated by Harberger (1998),
which is discussed below. It is simple, clear and can be undertaken with only very
limited suppositions about the probability of certain states occurring. Arithmetic
calculations are trivial, project designers’ or participants’ ability to identify a few
possible states for individual variables is likely to be plausible, and the resulting
combinations of possible states is quite likely to be representative of the range of
situations.
The other quoted example (“the most transparent and complete economic
and risk analysis”) is from a technical and higher education project in Mauritius
(prepared in 1995). This project was clearly innovative from the World Bank’s point
of view as regards its economic analysis, and appears to be the first World Bank
education project for which a measure of net worth (as opposed to a cost-effectiveness
approach) was used.
In the project’s economic analysis as presented in Belli et al. (2001), three
variables which affect returns (income differentials between graduates and
non-graduates, employment rates of graduates, enrolment rates) were identified,
and each assigned probability distributions (log-normal, and two different triangular
forms, respectively). A resulting probability distribution of project benefits is shown
in graphical form. In the actual Staff Appraisal Report (SAR—and available on
www.worldbank.org) for the same project, different variables appear to have been
analyzed (major cost and benefit items) using different sorts of distributions—truncated
and non-truncated normal distributions and one uniform distribution. A full Monte
Carlo simulation—using 3000 replications, with and without correlation assumed
between different variables—was ran at appraisal, and a probability distribution of
net benefits was produced.
When considering this particular project material and the general nature of
this type of approach (in contrast to the Mexico irrigation project, for example), it
should perhaps be noted that the risk analysis as described in project documentation
involves considerable technical discussion of the nature of variables’ distributions, and
the appropriateness of the particular techniques applied. It is also the case that the risk
38 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

analysis of the Mauritius project itself apparently took four weeks of staff time (on the
part of one summer intern, presumably with a strong statistics background) to complete.
Other, arguably much more typical, examples of risk analysis from World Bank
practice include (firstly) the $350 million Ghazi Barotha Hydropower Project in Pakistan
(SAR, 1995), where least-cost and economic net worth analysis was conducted, and
then a sensitivity analysis (for delays and cost overruns) was supplemented by a
risk analysis of the type used in the Mexico irrigation example, i.e., four factors affecting
project returns were modeled as having three possible states each, and EIRRs were
calculated for each. (This process was undertaken for the project in the context of the
power sector as a whole and for its own stand-alone operations). As for the Mexican
example, a cumulative probability distribution of outcomes was constructed, so that
it could be said (for example) that “the probability of the project’s rate of return being
less than the opportunity cost of capital is 8 percent.” Another project example is the
Xiaolangdi Multipurpose Project (Stage II) in China (SAR, 1995)—a $430 million loan,
where probabilities of river flow volumes are discussed, although only appear to be
modeled in terms of sensitivity analysis.
Other examples of World Bank practice in dealing with risk in various ways
include the projects shown in the following table. What is clearly similar to the
experience within ADB is that it is projects in the power sector which are analyzed
most fully in quantitative risk terms.

Other Agency Experience With Risk Analysis

A scrutiny of recent literature published by bilateral agencies such as Danish


International Development Assistance (DANIDA), Danish Cooperation on Environment
and Development (DANCED), AUSAid (Australian), GTZ (German), and FINNIDA (Fin-
land) regarding preparation of project or sector loans reveals very little about general
guidelines for economic analysis (in the sense of quantitative cost-benefit techniques),
or risk analysis in particular. One reason for this is obviously that such agencies are (by
and large) disbursing grant or extremely soft funds, often in relatively small amounts
and do not have the same fiduciary accountability requirements as a bank such as ADB.
It is also the case that operations by such agencies are increasingly in sectors where
monetized costs and benefits are less obvious than historically may have been the case.
The prevailing view among bilateral grant-based aid agencies, as reflected
in the available documented techniques for project preparation is that projects are
assessed primarily in qualitative terms, with a focus on institutional and sustainability
issues, rather than on estimation of economic returns per se. Exceptions (and
CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 39

Table 4
Examples of Risk Analysis from Recent World Bank Projects

Project Title and Date Features of Risk Analysis

Arun III Hydroelectric Project Four factors affecting performance, with 2 or 3 states each,
(India, 1994) were used to generate probability distribution of EIRR; ‘sensitivity
testing’ of basic assumed probabilities was also undertaken
for several variables
Waigaoqiao Thermal Power 5 ‘risk variables’ with 5–6 states each used to generate table
Project (China, 1997) describing expected EIRR with standard deviation, minimum/
maximum values, probability of negative outcomes, etc.
(uses 1000 simulations in Risk Master software)
National Drainage Program Includes quantitative assessment of political-economy risk
Project (Pakistan, 1997) regarding implementation and impact of various reforms, but
not measured probabilistically in terms of EIRR distribution
Third Andhra Pradesh Irriga- Combinations of variables’ values used to estimate range of
tion Project India, 1997) EIRRs; ‘risk analysis’ is really extended sensitivity testing
Yunnan Environment Project Wide range of technical, financial, institutional and policy risks,
(China, 1996) classified as short-term/medium and long-term/strategic; these
are described qualitatively, and presented in matrix format
with identified mitigating measures and parties responsible
Shandong Environment Wide range of technical, financial, institutional and policy
Project (China, 1997) risks, classified as short-term/medium and long-term/strategic;
these are described qualitatively, and presented in matrix format
with identified mitigating measures and parties responsible

sometimes, additions) to this approach are circumstances where there may be financial
impacts at individual or household levels, and then crop (or farm, household, enterprise,
etc.) budgets may be modeled. In these situations it appears that some kinds of
sensitivity testing (e.g., yields and prices up or down 20%, etc.) may be undertaken,
but not quantitative risk analysis.
One agency which used to undertake comprehensive cost-benefit or (cost-
effectiveness) analysis of most of its projects is the UK DFID (formerly ODA). Its
publication Project Appraisal of Projects In Developing Countries: A Guide for
Economists (various editions 1988–1995) was a standard source text for technicians.
However, while sensitivity testing is described at some length in this publication,
the agency’s view was that
40 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

“risk analysis (in the sense of probability-based techniques) is unlikely


to be used often for economic appraisal except in the largest projects”
(page 77, 1992 edition)

and its emphasis even in such circumstances seems to have been on the capital and
financial aspects of dealing with risk. While some examples of the application of the
techniques of risk analysis within DFID/ODA may be found, it certainly was a far
from routinely-advocated practice. An unpublished technical paper prepared within
DFID in 1995 reviewed the status of risk analysis, and advocated greater use of tools
such as probability/impact matrices (see below), decision trees, and histograms to
guide investment decision-making. This perhaps represents the last phase of thinking
about quantitative cost-benefit techniques within the agency, as this work was
abandoned in view of changed priorities.
There are nowadays two major aspects of DFID’s approach to project
preparation techniques that have direct bearing on the analysis of risk, although not
in quantitative terms. Firstly, in recent years DFID (like ADB) has taken a very strong
anti-poverty focus in its operations, and this has had major consequences for its
approach to project appraisal. Since the publication of the 1997 “White Paper” on
poverty, DFID’s approach has been far less oriented around the formal techniques
of cost-benefit analysis than previously, and more located within its “sustainable
livelihoods” approach (see figure in Appendix 2). One aspect of the “livelihoods”
approach to promoting sustainable development is that it involves explicit assessment
of households’ vulnerability to trends (e.g., in crop prices), shocks (e.g., natural
disasters) and cultural factors (e.g., ethnicity). This is effectively equivalent to raising
the profile of incorporation of risk in project preparation work by putting it at the
heart of strategic thinking, compared to previous practice, where consideration of
uncertainty and risk in DFID/ODA practice was essentially seen as supplementary
aspects of a quantitative cost-benefit analysis.
Secondly, within the context of the application of the logical framework
technique, DFID is very rigorous in
• identifying risks (right-hand column of the DFID format, and similar but
not identical to that of ADB’s “Project Framework”) and specifying how
they relate to relationships between different levels of the framework (i.e.,
“output-to-purpose”, “purpose-to-goal”)
• clearly specifying what mitigating measures have been already within
project design to address or minimize these risks, and
• compiling a “risk matrix” which locates each identified risk within a matrix
whose dimensions are “probability of occurrence” and “seriousness of
impact” (i.e., if the event does occur).
CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 41

DFID is also notable for the preparation of a “risk annex” in its supporting
project documentation, which gives it a similar status to that of social appraisal,
technical appraisal, economic and financial appraisal, etc. The risk annex and risk
matrix are essentially used in a qualitative way to both promote dialogue between
host governments and executing agencies and to ensure that appropriate mitigating
measures are put in place.
The following is an example of a DFID risk (or impact/probability) matrix from
a recent project in China (Yunnan Environmental Development Programme, 2000).
The numbers 1-10 refer to the 10 different sources of risk to the project, which were
identified during preparation and discussed within the project’s logical framework.
The identified risks comprised the following
Risk 1: Chinese long term funding for poverty alleviation and environmental
improvements in Yunnan is inadequate
Risk 2: Inadequate state:provincial link established
Risk 3: Economic development given higher political ranking than reduction
of environmental degradation
Risk 4: Abnormal incidence of physical shocks, e.g., earthquakes
Risk 5: High staff turnover leads to skills dissipation
Risk 6: Sustainable development and the Agenda 21 process is not given
sufficient priority by Yunnan Provincial Government
Risk 7: Institutional practices limit the adoption of an integrated approach
and subsequent co-ordination of activities
Risk 8: Insufficient counterpart funding leads to sub-optimum choice of pilot
projects
Risk 9: Greater priority given to environmental benefits than to poverty
reduction
Risk 10: Institutional support for proposed gender improvements may not
be adequate

One other point with respect to risk analysis in the context of logical frameworks
is that some variants of that technique (e.g., that of the German agency GTZ) have
used versions where the probability of a risk—as well as its existence—is indicated
within the framework itself. If the probability of a situation occurring which would
jeopardize the achievement of project objectives is thought to be sufficiently high,
this may be regarded as a ‘killer assumption’, around which the project must either
be scrapped or re-designed.
Another UK agency which has been active in extending its analysis of risk
(and which is probably the original source of the DFID approach) is the UK Treasury,
in the context of the Private Sector Finance programs (where the private sector is
42 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Figure 8
Risk Matrix: Impact and Probability Analysis

Impact

Probability: Low Medium High

Low 4 6 1, 2, 5, 7, 9

Medium 8, 10 3

High

invited to participate in what are traditionally regarded as public projects). Within


the Treasury’s approach to PFI, considerable effort is devoted to
• identifying the full range of risks faced by a project (from planning, design
and construction risks, through demand, occupancy and maintenance risks,
to legislative, inflation and technology risks)
• quantifying the impact of risks (based on the Construction Industry Research
and Information Association classifications of “catastrophic”, “critical”,
“serious”, “marginal”, and “negligible”)
• estimating the likelihood of risks (based on standard probability techniques),
and
• allocating risks between different project participants, such that those who
can best manage the sources of risk should bear the consequences (good
or bad) of such management.

It is clearly the last of these functions which is the main purpose in the PFI
context— i.e., making sure that (where possible) the ownership of risks and their
consequences are transferred to the parties (public or private sector) most able to
deal with them. It is then the responsibility of respective parties to take action to
mitigate risk. Based on the construction of an extensive risk matrix in this way—
with its emphasis clearly on allocating and transferring risks—negotiations about
contracts for the private sector to be involved in public service delivery or procurement
are much clearer, and the basis for the public sector comparator (i.e., the risk-adjusted
cost estimate for the project to be entirely a public sector provision and against which
any private bids must be compared) is entirely apparent.
Despite the innovativeness of these various techniques, it should still be noted
that none of them deal with risk proper, in the sense of quantitative, probability-
CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 43

based techniques, or (if they do include any probability-based constituents) they do


not extend such techniques as already applied—at least on occasion—by (for example)
World Bank or ADB.

Risk, and the ‘Harberger’ Critique of Recent Applied


Project Economic Analysis

While reviewing the experience of ADB and other major multilateral and
bilateral development agencies with respect to the analysis of risk in the context of
project economic analysis, it is impossible to avoid some of the major critiques of
observed technical practices which have appeared in recent years. One of these, and
one that has definite implications for practice with regard to risk analysis, is that of
Harberger (1998).
One of Harberger’s main points is that there has been a lack of account of risk
taken in project economic work in recent years. This is partly because of the (perhaps
unexpected) importance of large changes in relative prices over time (some being
due to significant shocks, such as unpredictable changes in petroleum prices, and
some due to longer-term trends, such as in the US$ appreciation of the late 1990s),
and partly because of systematic “approval culture” which is endemic to most
multilateral organizations (and which tends to consistently overstate benefits and to
underestimate costs of projects in a climate of “appraisal optimism”). As a result, the
relationship between what was expected at appraisal and what actually happened
overall in development projects is relatively poor (the experience of the ADB projects
reviewed in this Handbook largely confirms this view). Harberger also argues that
because loans made by agencies such as ADB and World Bank are in effect subject
to countries’ sovereign guarantees and the banks’ capital is not effectively at risk,
these lending agencies tend to take a less serious view of risk than would, for example,
private investors in similar circumstances.
As a result of what Harberger calls such “ambient pressures”, a “comfortable
scenario” (perhaps reinforced by organizational internal incentive structures) typically
characterizes most project preparation environments, which causes a divergence
between expected values at time of appraisal and ultimate project impacts. Harberger
argues that better derivation of expected values at appraisal (i.e., estimation of ones
more likely to represent real flows of costs and benefits) would be realized if more
account were taken of risk—and in practice Harberger advocates the more widespread
use of a simplified Monte Carlo simulation technique. The application of this technique
would depend upon a few (he suggests four or five) values being chosen for the
44 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

“key (i.e., single) variable which is most critical to the project’s outcome”

in most circumstances, and an expected value for NPV being calculated based on
assigned probabilities for each of these states. This approach can also be extended,
so that even two or three other variables can be identified (which are in practice
quite independent of one another – e.g., weather, real wage growth, traded input
price), and the simulations re-run on the basis of only a few values for variables.
(This approach is therefore very similar to the World Bank’s “On-Farm and Minor
Irrigation” project example mentioned earlier in Part III).
Harberger offers no new techniques for risk analysis per se, but does make
a strong case for greater application of probability-based techniques, albeit in a
simplified form. This is in fact what is largely recommended to improve ADB practice.
ADB experience as described in Part III appears to bear out at least some of Harberger’s
criticisms (e.g., as regards systematic underestimation of costs, overestimation of
benefits, etc.).

Summary of Risk Analysis Experience

What emerges from the review of ADB and other agency experience is that,
despite the extensive academic literature describing the techniques for more
quantitative approaches to risk analysis and the increasing availability of computers
with which to run probability-based simulations for values of key variables, actual
examples of such practice are very rare.
Within ADB, the overwhelming orientation of current practice has been towards
a qualitative risk description linked to mitigating actions included in the project design.
A reliance on this, plus fairly standardised sensitivity testing designed to demonstrate
project robustness (survival in the ‘worst case’ scenario), is found across all sectors
of operations.
Almost identical conclusions are drawn in relation to a review of recent World
Bank practice (despite its much stronger emphasis on the utility of such techniques
in its own published project economic analysis documentation), although the format
and presentation of World Bank projects perhaps differ more across sectors (with a
much larger number of projects being dealt with).
Within both institutions, almost the only exceptions to this situation appear
to be found in the power sector, where simplified probability distributions for a
few key variables are used to estimate expected returns plus estimated variabil-
ity. The cited ADB port and agriculture projects, and the World Bank education
CHAPTER 3 A Summary of Development Agency Experience in Applying Risk Analysis 45

project are most notable for the rarity of the application of full-scale probability
techniques.
Most bilateral agencies (such as UK DFID) are now undoubtedly moving away
from quantitative cost-benefit analysis generally, although it is also clear that certain
qualitative techniques for dealing with risk and more centrally incorporating it within
a pro-poor lending environment are emerging at the same time. Greater consideration
of the expected vulnerability of poor groups in terms of the risks they face while
earning their livelihoods, and how they may best be planned for in such situations,
are ongoing themes of research work. Similarly, other agencies (e.g., the UK Treasury)
are moving consideration of risk more towards the center of project analysis in an
attempt to more properly allocate costs and benefits from risk management. Although
no new techniques for extending quantitative risk analysis are yet being proposed
it is possible that existing techniques may be further extended in application.
Critiques of some of the project economic analysis practices of major multilateral
development agencies (based on reviews of project outcomes) include arguments
that risk has not been sufficiently well dealt with in the past, and that lenders have
not been rigorous enough in considering unknown future outcomes from the point
of view of their borrowers (who ultimately bear the financial and economic conse-
quences of such outcomes).
In these circumstances, perhaps the future challenge for ADB as regards risk
analysis is therefore two-fold:
• as a lending institution whose interest lies in seeing that its clients (i.e.,
borrowing governments) are not unduly exposed to economic risks, there
is scope to strengthen project design through the fuller use of risk analysis
so as to ensure greater sustainability (financial, environmental, and
institutional) of project effects and likelihood of project success; this can
include better identification and definition of risk, and its allocation among
various project participants, and
• with the increasing emphasis on lending to reduce poverty, there is a need
to put the circumstances (especially as regards their vulnerability) and
attitudes of those affected and targeted by projects at the heart of project
economic analysis.
Introduction
Poverty Reduction
Objectives and
Risk Analysis
4
This Part of the Handbook considers the relationships between poverty analysis
and risk analysis.
The traditional assumption about government neutrality toward risk being
an appropriate basis for planning tends to break down, the more specific (i.e.,
concentrated on particular groups) the national project investment portfolio becomes.
In addition, the consequences of uncertain outcomes (when in fact they do turn out
to be low or negative) can be disastrous for poor people, whose situation is often
characterized by extreme vulnerability.
48 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

With ADB’s increasing emphasis on lending aimed at reducing poverty there


is, therefore, a greater imperative to improve analysis of the situation of the poor
with respect to uncertain outcomes from projects and policies. This involves, inter
alia, more rigorous investigation of the risks affecting project returns (both to individual
types of participant and to the economy as a whole) from both the point of view of
planners and decision-makers, and also some consideration of the attitude of project
participants themselves towards risks they may be expected to face.
This Part of the Handbook therefore summarizes the situation of the poor
vis-à-vis risk, argues that some knowledge of poor project participants’ attitudes is
essential in making investment decisions, and considers what operational implications
may follow from this for the analysis of projects’ distributional and poverty impacts
in both quantitative and qualitative terms.
Within the context of pro-poor lending generally, it is also the case that poor,
vulnerable groups can be difficult to target development assistance towards (because
of the effects of benefit leakage, for example) and therefore it may well be that pro-poor
projects are inherently more risky than other projects. If this is true, then anything
which can be done to strengthen economic analysis of such projects through the
incorporation of risk analysis techniques is likely to educate and inform donors, as
well as to lead to better projects.

Planning for the Poor, Vulnerability, and Risk Aversion

As suggested in Part II, although the techniques of probability-based risk


analysis may afford a good understanding of how the variability of outcomes is related
to expectations of those outcomes, this will only partially contribute to investment
decision-making, and especially so in situations dealing with the poorest.
Consider two projects—A and B. In Figure 9, a probability-based risk analy-
sis of the projects produces the CDFs for the expected NPVs (or financial outcomes
at household or enterprise level) of each project. It can clearly be seen that project
A is preferable to B. A is said to be “stochastically dominant” (and ‘first degree’ so
- in that its distribution lies entirely to the right of B, i.e., at any particular level of
probability its expected NPV is higher). The likelihood of either of the respective
projects producing negative or low outcomes can also be immediately assessed (project
A does not generate negative outcomes, unlike project B), and thus the risk to the
poorest is clearly identified.
In another situation (Figure 10) the considerations are more complex. Again,
project A can be said to be “stochastically dominant”, in that it lies more to the right
CHAPTER 4 Poverty Reduction Objectives and Risk Analysis 49

Figure 9
CDF: Project Alternatives (1)

Figure 10
CDF: Project Alternatives (2)
50 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

of B than to the left. However, it is now only ‘second degree’ dominant. Despite its
generally higher expected NPV, project A actually has greater probability of nega-
tive outcomes than project B (its CDF now crosses that of A).
This situation is not uncommon, and might well be characteristic of a situation
in which new and more productive technology (e.g., agricultural, industrial) was
being introduced by a project but its greater benefits depended upon (for example)
water availability and management, equipment maintenance, and staff training, etc.,
about which some implementation doubts existed.
A question for decision-makers would therefore be about what level of risk
it might be thought proper to consider imposing on a target population (in this case
upon very poor people) if the consequences of failure or negative outcomes in any
one year could be potentially devastating for them. Addressing this type of issue
necessarily involves eliciting some kind of knowledge about poor people’s risk aversion
within the context of a project preparation exercise.
Poverty is usually associated with vulnerability to external shocks, cultural
factors, and trends (e.g., as summarized in the DFID “livelihoods” figure in Appen-
dix 2). In addition, and almost by definition, any situation (e.g., a proposed project)
which involves the possibility of uncertain and/or negative outcomes for the poor is
potentially disastrous for them—even if it would not necessarily be so for less poor
populations. For example, for the poor in natural resource-based situations who have
less financial reserves (if any, and especially in the typical absence of insurance) to
cope with bad (i.e., low producer price) crop seasons, the consequences of failure
may actually be catastrophic (e.g., property or soil loss through flooding) and/or non-
reversible (e.g., loss of land to debt).
For these sorts of reasons, the poor are usually considered to be more risk
averse than most sections of society. Although estimating utility functions has proved
very difficult in practice (e.g., “is utility futility?”—a question which has been asked
in the literature, see Part II), empirical studies have tended to demonstrate that
individuals generally become very risk averse indeed when
• considering outcomes involving sums of money they are not used to dealing
with, and
• when the possibilities for losses are involved.

It is often suggested that people, and especially poor people, have a “focus
of loss” which causes the slopes of their utility functions to become extremely steep
in the wholly negative quadrant, even though it may exhibit the more normal
diminishing marginal utility in the wholly positive one.
If this is a reasonable depiction of poor populations’ attitudes to uncertain
outcomes, it implies that an understanding of attitudes to risk when considering
CHAPTER 4 Poverty Reduction Objectives and Risk Analysis 51

Figure 11
‘Focus of Loss’ Utility Function: Hypothetical Example

alternative projects or project designs is essential in investment decision-making,


especially when dealing with the poorest in society.
The kind of considerations just discussed therefore implies that, when
considering potential investments affecting poor people,
• as in other investment decision situations, probability-based risk analysis
can usefully identify the relationships between expected project outcomes
and their variability, and can be specifically used to indicate how increased
predictability may be achieved, although at particular levels of cost
• such techniques can also be used to identify the likelihood of negative or
very low outcomes (for individuals, groups, and for whole projects)
• the real-world consequences of any such possible outcomes for the lives of
poor people can be considered, and possible mitigating action taken (e.g.,
safer/more robust projects designed) as a result of risk analysis, and also that
• decisions about whether or not to accept such risks necessarily involve
some knowledge about risk aversion among the target population.

These sorts of conclusions have direct implications for how greater analysis
of sources of risk to projects can be both quantitatively and qualitatively incorporated
into existing ADB operations, especially in relation to distribution and poverty impact
analysis, and also in relation to policy-based lending operations.
52 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Poverty Reduction Objectives:


Implications for Quantitative Risk Analysis

With ADB’s increasing focus on lending to reduce poverty, there has necessarily
been more of an analytical emphasis on measuring the distributional and poverty-
reducing characteristics of operations, and in particular the recommended use of
distributional analysis and the calculation of the poverty impact ratio (PIR). The
increased use of such techniques has implications for how risk analysis can be used
to design and assess ADB projects and policy interventions.
The methodology for distributional analysis and calculation of the PIR were
originally contained in the Guidelines—Appendixes 25 and 26 (ADB 1997a), and have
since been elaborated upon in Fujimura and Weiss (2000) and also in ADB (2001a).
In essence, the methodology for estimating poverty impact in this way involves
• identifying financial and economic flows by groups of participants in a
project (e.g., “consumers”, “farmers”, “operating entity/company”,
“government/rest of the economy”, etc.),
• summing financial flows plus the differences between economic and
financial flows (due to differences in prevailing economic prices arising
through taxes, subsidies, shadow wage rate, etc.) accruing to each group,
• calculating the proportion of total benefits going to the poor in each group
(knowledge of the poor’s composition within each project group must be
estimated), and
• summing the benefits accruing to the poor across all groups and expressing
this as a proportion of total project benefits (i.e., this figure is the estimated
PIR).

Calculation of the PIR involves no new data than that normally collected for
a full financial and economic analysis of a project, although it does require that financial
and economic costs and benefits be disaggregated by participating group and that
the proportion of poor in each group be identified (a wide range of income, social,
health, and other data may be used from numerous sources to assist in this definition).
In principle of course, probability distributions applied to any variables (i.e.,
cost or benefit items) prior to a distribution/PIR calculation would lead not just to
individual (i.e., “point”) estimates of financial and economic benefits by groups but to
“expected” values of benefits with associated estimates of their variance, and therefore
to an “expected” project PIR plus a measure of its variance. This approach has been
undertaken once so far in Bank practice—in the Shen Da project described in Part III.
CHAPTER 4 Poverty Reduction Objectives and Risk Analysis 53

What is also interesting following such a calculation, however, is that the


extent of variability which might emerge from such an analysis would differ between
project-participating groups to the extent to which they differentially incurred costs
or received benefits whose estimation derived from those variables being subjected
to risk analysis. As yet, the application of comprehensive probability-based risk to
model the differential impact of project cost or benefit item variability between different
groups as part of a full-scale distribution/PIR estimation has not been attempted in
ADB practice.
What the use of some type of risk analysis can probably and practically most
usefully show in the context of poverty impact analysis is how likely it is that financial
and/or economic returns may be very low or negative (i.e., ENPV<0, EIRR<EOCC)
for particular groups affected by the project—typically the poor (e.g., farmers,
processors, traders) who are its target population. An example of this is given in
Fujimura and Weiss (2000) where differential risk exposure between project-
participating groups (i.e., water authority, farmers, government) is modeled. Given
ADB’s increasing lending orientation towards the poor and the already described
particular vulnerability of such poor populations in the face of risk exposure (as
compared to national populations as a whole), the case for increased use of risk analysis
is therefore made stronger than hitherto. Part V considers how this principle can
perhaps be operationalized in situations where costs and benefits are quantified
and valued and ENPVs are calculated for project groups and for projects as a whole.
Quantitative risk analysis could also conceptually be employed in the case of
subregional projects, where the distributional analysis methodology (Adhikari and
Weiss 1999) is simply a special case of general distribution analysis (only here applied
to countries rather than groups within one country), but where the real level of
project risk may be higher than for single country projects (e.g., because of
coordination difficulties among countries and agencies, exchange rate fluctuations,
etc.). Again, no examples of quantitative risk analysis have yet been undertaken
for such projects.

Poverty Reduction Objectives:


Implications for Qualitative Risk Analysis

In support of all projects’ textual poverty analyses (e.g., in social assessments


carried out at various stages) and also in situations where a PIR is not calculated for
a particular project because its benefits cannot be reasonably estimated, qualitative
risk analysis routinely focuses on such issues as, for example,
54 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

• extent of ability of target population to cope with risk (e.g., based on


socioeconomic status indicators)
• general risks (e.g., institutional, civil) which may compromise overall project
success, and
• risks of benefit leakage to non-poor groups.

Similarly, in the case of policy-based lending (PBL), the use of a poverty impact
assessment (PIA) matrix is advocated to elicit the relationships and mechanisms
between particular policy interventions and ultimate poverty impacts. (Policy-based
lending is probably the most inherently uncertain of all lending types, in that, while
the ‘why’ of the program is likely to be well-understood, the full range of ‘what’ and
‘how’ mechanisms in a sector is likely to be much less clear). Within this matrix, the
analysis concentrates, inter alia, on economic variables that change with particular
policy adjustments (e.g., removal of subsidies, imposition of user charges, etc.) and
identifies the channels of their impact on the poor.
Recent consideration of how PBL and poverty reduction analysis can be more
closely integrated (Bolt and Fujimura 2002) suggests that improvements to current
procedures include greater use of statistical inference and risk analysis, even where
the scope for quantification may be limited. Importantly, the same paper suggests
that use of the PIA matrix as a design tool in a participatory fashion may help lay
out options, costs and benefits from various policy alternatives in an iterative way,
and thus an implicit consideration of risk by potential project participants/beneficiaries
is being undertaken.
What emerges from consideration of the sorts of qualitative analysis of risk
which is encapsulated in the above techniques is that project target population/
beneficiaries’ situations and views are being quite thoroughly canvassed, but that
more could perhaps be done within such exercises to explicitly document the extent
of risk aversion among such groups, as an input to planning for them (and which
would be especially useful in situations of choice among various design alterna-
tives). In other words, as well as “objective” data which may be available from other
sources upon which to estimate expected values and their variance for key variables
affecting project outcomes, it should be possible for project planners to gather (in
the application of quite intensive and participatory data-gathering exercises) some
“subjective” information about how those expected to benefit from project or policy
interventions may view choices among such options.
This kind of argument is perhaps particularly applicable in circumstances
where new technologies are being introduced. In these situations, project partici-
pants, such as small farmers, may well be generally enthusiastic about planting new
high-yielding varieties of rice or wheat but some of them are unable to face the
CHAPTER 4 Poverty Reduction Objectives and Risk Analysis 55

consequences of loss if a bad year occurs in project year 1 or 2, for example. All may
be happy to accept such a risk if the expected yield is double or treble current levels,
but some may well be unwilling to accept only a 40%-50% expected increase, for
example–even though such an increase is clearly substantial. Understanding how
farmers perceive such choices can influence
• the technical (re-)design of the intervention itself (so that its expected returns
are made more stable, higher, or both)
• the way project benefits (and costs) are calculated (i.e., depending upon
uptake rates), and
• the distribution of benefits (with poorer farmers given more opportunity
to benefit from a redesigned intervention, for example).

Summary

Projects have differing extents of risk attached to them, and it may well be
the case that projects with higher expected benefits are also more risky than some
with lower benefits. Because poor people’s situations are typically characterized by
extreme vulnerability, exposure to very low or negative outcomes (in the acceptance
of more variable project outcomes) can be catastrophic—and so there can be major
issues of choice regarding appropriate levels of risk to expose target populations to
in the introduction of (for example) new technologies and/or services.
Quantitative risk analysis can make clear the implications of available project
choices at household, farm, enterprise, and project levels. Full-scale, probability-based
risk analysis techniques could conceptually be applied to strengthen existing
distribution and poverty impact analyses of projects, although it is most likely that
practical applications of techniques will be limited to concentrating on describing
the probability of negative returns at enterprise level.
Qualitative risk assessment based on participatory techniques (interviews,
group and village discussions, etc.) is part of existing project and policy-based lending,
but its scope and intensity could be increased to include gaining an understanding
of target groups’ attitude to risk, so that appropriate levels of risk can be incorporated
in project and program design.
The increased application of both sorts of risk analysis approaches should
help to strengthen project design generally, and to address poverty reduction
objectives specifically.
Introduction
Strengthening
Risk Analysis in
ADB Operations
5
This Part of the Handbook provides some practical guidance on how to
strengthen the analysis of risk in ADB operations, based on the technical material
and discussion contained in Parts I to IV. Its format is primarily designed to be
of use to practitioners of economic and financial analysis of all types of projects
financed by ADB. (Much of the material in this Part of the Handbook is separately
presented in ERD Technical Note No. 2 on Integrating Risk into ADB’s Economic
Analysis of Projects (2002).
58 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

The first section briefly summarizes the key features of the various possible
techniques for risk analysis, both qualitative and quantitative.
The next sections show how strengthened analysis of risk through applying
such techniques could potentially contribute to various ADB operational objectives.
These are taken to be
• firstly, risk mitigation in the broadest sense (i.e., the strengthened design
of individual projects such that the probability of their having an
EIRR<EOCC or ENPV<0 is minimized)
• secondly, contributing to a more specific focus on ensuring the sustainability
of project effects—through strengthened financial, environmental and
institutional risk analyses, and
• thirdly, contributing towards the greater achievement of poverty reduction
objectives through strengthened risk analysis.

This is followed by an outline of how risk analysis may contribute specifically


(although in a fairly limited way) towards the formulation of policy/sector/program
lending operations.
A consideration of some typical risk analysis situations sector-by-sector is
then presented (again in tabular form), as within individual sectors and types of
projects it is likely that similar technical/methodological, policy, and data issues will
arise. The last section deals with various technical and resource implications of
strengthened risk analysis being applied in practice, including some points on applying
the ‘@RISK’ package.

Summary of Risk Analysis Techniques:


Applications and Limitations

In essence, what is fundamentally suggested in this Part of the Handbook is


that a pragmatic approach to the use of risk analysis is warranted. Each project is
unique, and the sources of uncertainty and risk it faces will be similarly unique to
its own individual circumstances, and the extent to which risk can be quantitatively
dealt with will also vary. It would not be appropriate to advocate hard and fast
guidelines about application of particular risk analysis techniques to all projects.
Nevertheless, similar types of projects are likely to face similar analytical issues
(e.g., of benefit estimation, of sustainability), and similar types of risk analysis
techniques will therefore be appropriate to use across a number of project types. As
a general rule,
CHAPTER 5 Strengthening Risk Analysis in ADB Operations 59

• the greater the extent to which risk can be identified and quantified within
the scope of routine project economic analysis, the stronger will be overall
project design (assuming mitigating measures are put in place once the
scale and impacts of known risk are clear) and the likelihood of project
failure (in the sense of EIRR<EOCC) will be reduced,
• the more that risk analysis can be used to investigate the specific financial,
environmental and institutional aspects of project design, the greater will
be the likelihood of the sustainability of project effects over time, and
• the more comprehensively the objective circumstances and subjective
attitudes of poor project participants can be taken account of in project
planning, the greater chance there is of projects achieving poverty-reduction
objectives.

The following table (5a and 5b) summarizes the major approaches to risk
analysis which have been covered in this Handbook. It outlines their main features,
the type and form of results that come from such analyses, the circumstances in which
application of each of them may be likely, and also some possible constraints to their
application. The table classifies the techniques according to whether they are es-
sentially qualitative or quantitative in nature. What is apparent from the table is that
there are a number of techniques for dealing with risk in project design and analysis.
They range from a spectrum of simple risk identification (and linking these risks
with specific mitigating measures), to subjective quantification of likelihood of event
occurrence and seriousness of impact in that event, description of the nature of
exposure to risk on the part of project participants and some estimation of their
attitudes towards risk in particular circumstances, and (finally) to probabilistic-based
estimates of project returns depending on the behavior of key variables (such estimates
may derive from more or less sophisticated and data-intensive techniques).
These risk analysis techniques are of course likely to be applicable in different
sorts of circumstances. The suggested elaboration of risk analysis within the existing
Project Framework and the construction of a risk matrix could be applied in any project
situation, while the use of continuous probability distributions based on historical
observations will probably remain relatively rare (for data-intensity reasons, if not
software ones), for example.
It is also important to note that the use of the individual techniques are not
mutually-exclusive. For example, the risk matrix technique can identify those risks
that are thought to be the most serious and/or likely to occur so that they can then
be further investigated through quantitative techniques. (It is also of course the case
that such variables can be identified after sensitivity testing techniques have
been applied).
60 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

In essence, all the techniques attempt to identify and describe risk, and some
of them try to quantify the extent of this risk. (Properly of course, it is only when
some quantification has been achieved that the situation can be described as having
modeled risk, rather than simply identified a source of uncertainty). Whether quantified
or not, ultimately a decision about whether to accept a project in the face of the
simple known existence of a risk (or of a particular level of that risk), is a subjective
decision for planners and policy-makers. It remains practically impossible to derive

Table 5
Qualitative Risk Analysis Techniques

Type of Risk Analysis Technique Main Features

Logical framework ‘risks and Expansion of consideration of risks within existing ADB Project
assumptions’ elaboration Framework

Risk Matrix Construction Construction of 3*3 (or more) cell matrix showing approximate
(and ‘Risk Annex’ preparation) probability of risk occurrence (high, medium, low) against seriousness
of impact (high, medium, low)

Allocation of risks among different project participants

Poverty and Risk Vulnerability Assessment of the nature and extent of target group’s exposure to
Assessment risk (catastrophic or not, controllable at micro level or not, reversible
or not, insurable or not, etc.)

Risk Aversion/’Focus of Loss’ Quantification of extent of target group’s attitude to risk (especially
Estimation any risk or risk reduction implied by the proposed project) and
especially towards possible losses of incomes
CHAPTER 5 Strengthening Risk Analysis in ADB Operations 61

any decision rules about the acceptability or otherwise of any particular investments.
However, because policy-makers and planners (within ADB and borrowing
governments) are typically planning for particular groups within society, it is useful
to know something about those groups’ position with respect to exposure to risk and
their attitudes towards any changes in that exposure which project interventions
may imply in order to assess whether estimated levels of risk are acceptable to
them or not.

Possible Constraints /
Type and Form of Results Likely Applications
Limitations
Textual summary of how each risk may pre- Any project for which Project No quantitative assess-
vent achievement of objectives at different levels Framework is completed. ment of risks’ likelihood
of the project’s objective hierarchy. Each or seriousness.
identified risk described in more detail than at
present and linked to at least one specific Barest minimum of ‘risk’
mitigation measure analysis
Risk Matrix, with individual risks numbered and Any project Subjective assessment of
discussed (along with identified mitigating risk exposure only.
measures) in separate ‘Risk Annex’ to RRP. Dem- Minimum quantification
onstration that ‘killer’ risks have been dealt with of probability; limited
(i.e., that most likely / most serious is not a ‘killer’) classification of expected
impacts
Responsibilities and rewards for managing Particularly applicable to
different sorts of risks assigned to those agents those involving physical
best able to deal with them constructions
Part of Initial Social Assessment prior to PPTA, Any project, but especially Qualitative assessment
concentrated in social and economic assessment Poverty Intervention ones only
during PPTA; should show how proposed project
will contribute to risk exposure reduction. Supported by existing
household survey data
Part of (modified) Poverty Impact Assessment PBL and also primary data
Matrix in PBL. collection
Application of (for example) interview-based New technology being intro- Relatively demanding in
ELCE (equally-likely certain equivalent) tech- duced, especially where it is terms of consultant/staff
nique to derive estimate of risk aversion over desirable to estimate likely and interviewee time
typical income levels of those affected by pro- attitudes to uptake of new
posed project intervention but risky technologies (e.g.,
in agriculture) and negative
outcomes may result
62 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Table 5
Qualitative Risk Analysis Techniques (continued)

Type of Risk Analysis Technique Main Features

Simplified Probabilistic Analysis Indicates likelihood of project EIRR/ENPV being acceptable, based
(e.g., Harberger, ADB power, WB on consideration of key variables as determinants of project
Mexico irrigation examples) performance

Spreadsheet-Based Applications Use of standard spreadsheet functions (to generate random numbers
(e.g., Clarke and Low) and counts of observations of key variables) to produce distribution
of project outcomes

‘Monte Carlo’ Simulation with Classic risk analysis technique based on continuous distributions
Continuous Distributions for key variables

Strengthening Project Design:


Overall Economic Benefits Estimation

The primary utility of an analysis of risks faced by projects lies not in enabling
choice among competing projects (as the orientation of much academic literature implies)
but in the information this provides about the proposed project and its particular
environment—such that consideration can be given as to how the project may be re-
designed to reduce risks to an acceptable level. Ideally, the same level of expected benefits
may be found to be achievable with less risk or, if risk reduction also reduces expected
benefits, then the extent of that trade-off can be made clear to planners and/or beneficiaries.
Table 6 suggests some principles which can be applied to risk analysis for
the purpose of overall project design, specifically in relation to the estimation of a
project’s overall economic benefits.
CHAPTER 5 Strengthening Risk Analysis in ADB Operations 63

Possible Constraints /
Type and Form of Results Likely Applications
Limitations
Estimate of expected EIRR/ENPV, plus CDF of Any project where key vari- Results will only be as
project EIRR/ENPV, with measures of variance, ables can be identified and good as the distributions
minimum/maximum values simplified, and probability are realistic; statistical
distributions constructed. complexities with co-vari-
ance; availability of soft-
ware (@RISK, RiskEase,
etc.)
Estimate of expected EIRR/ENPV, plus CDF of No inherent advantages; Fairly extensive familiarity
project EIRR/ENPV, with measures of variance, likely to be used only in with EXCEL or LOTUS
minimum/maximum values situations where risk analysis required; developing
software is unavailable non-uniform distributions
by writing formulae is
complex
Estimate of expected EIRR/ENPV, plus CDF of Where historical/cross- Demanding in data and
project EIRR/ENPV, with measures of variance, sectional data exist for key staff time; experience
minimum/maximum values variables such that continu- may suggest that results
ous distributions can be add little to analysis over
fitted and above use of sim-
plified distributions

In addition to modeling the technical variables which explain a project’s


performance, there may also be doubt about the values which have been used in
estimating the project’s economic costs and benefits (i.e., the derivation of specific
conversion factors–such as the shadow wage rate factor, or general conversion
factors¾such as the standard conversion factor or shadow exchange rate factor).
Sometimes, such factors are included within existing sensitivity testing exercises,
but there is of course no reason why they could not more routinely be subject to a
simplified probabilistic analysis.

Promoting the Sustainability of Project Effects

The delivery of project effects over time depends upon sustainability being
built into project design. It is suggested here that risk analysis can specifically help
with ensuring that ADB projects are made more sustainable in various ways. Following
64 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Table 6
Principles in Applying Risk Analysis in Project Design

Principles to Apply
1 Identify any risks facing the proposed project as soon as possible (i.e., pre-PPTA); include
description of expected risks in first draft of Project Framework
2 Construct ‘Risk Matrix’ for proposed project, ranking risks according to their relative
likelihood of occurrence and their expected scale of impact
3 Identify ‘key’ variables (e.g., quantities, unit costs, output mixtures, output prices, uptake/
adoption rates, price and income elasticities of demand, etc.) which are sources of risk
and determinants of project returns
4 Decide which of these variables may be subject to quantitative description
5 Identify data sources for each variable (i.e., ‘objective’ historical or forecasts, ‘subjective’
‘best guesses’, expert-Delphi, etc.)
6 Construct probability distributions of key variables
7 Perform simplified probabilistic analysis (e.g., using @RISK, RiskEase, etc.) to generate
CDF of expected EIRR/ENPV, minimum/maximum expected values, etc.
8 Consider whether the derivation of distributions from primary/empirical sources is justified;
if so, perform probability-based analysis using such distributions
9 On the basis of results from 7 and 8, decide whether risk of EIRR<EOCC or ENPV<0
is ‘acceptable’
10 If extent of risk is regarded as ‘acceptable’ – redesign may not be necessary (but check
individual distributions to see if any high values are ‘pulling up’ the expected value of
the distribution, i.e., see if positive skewness is occurring which causes average values
to be substantially higher than the most likely values)
11 If extent of risk is regarded as ‘not acceptable’ – possible redesign (in particular, check
to see what can be done about any low values in distributions; in particular, investigate
any negative skewness, and attempt re-design to truncate distribution)

the Guidelines, the notion of sustainability has at least separate dimensions—financial,


environmental, and institutional—and these can be approached in different ways
through risk analysis. In all aspects, however, the emphasis remains on considering
and modeling sources of risk and designing mitigating actions to reduce that risk if
its level is considered unacceptable.
CHAPTER 5 Strengthening Risk Analysis in ADB Operations 65

Financial sustainability risk analysis


The financial sustainability of institutions is important in most project situations.
In some projects, the institutions under consideration will be financial institutions
proper (e.g., state-owned or commercial agriculture banks, industrial development
banks, credit unions, nongovernment organization-run operations, housing banks).
In other situations they will be project executing agencies managing or providing
technical or consumer services (e.g., a municipality, a commercial bus company, a
state-owned plantation, a water supply and drainage authority). In all cases, lack of
financial sustainability will compromise the delivery of project effects to beneficiaries,
either by causing liquidity to dry up or for service provision to be suspended and/
or curtailed. Risk analysis can be used to assist in designing projects so that there
is less chance of this occurring.
For financial institutions such as banks, a major concern of their appraisal
and consideration for participation in an ADB project is their situation with respect
to risk. Following standard international banking practice, and as summarized in the
Financial Guidelines, a number of standard measures for credit risk (borrower default),
value at risk (VaR), foreign exchange risk, maturity risk, contagion risk, etc. can be
derived. They generally involve some subjective estimate being made by financial
analysts/PPTA teams about the probabilities of specific outcomes, typically based
on a mixture of expert judgment and some forecast data. It is suggested that in many
cases it would be possible to extend this analysis for at least some measures of risk
to be based on probability distributions. This is especially true for the VaR, which
is theoretically based on forecast values (which could be presented in probabilistic
terms), and is supposed to measure

“over a 10-day period, what is dollar amount of V such that there is


only a 1% probability that a portfolio will lose more than V?” (Financial
Guidelines, section 6.4.4.2)

The calculation of VaR is essential where loans are from ADB to a


nongovernment-guaranteed FI, and is still useful even if the loan is treated as
risk-free.
ACCION International (an American nongovernment organization) has
extended the CAMEL framework (‘Capital Adequacy, Assets Quality, Manage-
ment Quality, Earnings and Liquidity’ - see Financial Guidelines, 6.4.3.1) to analysis
of microfinance institutions—particularly relevant in the context of increased
lending to the poorest. It is the case that many of the essentially ‘static’ measures
under CAMEL for assessing financial performance of whatever type of lending
66 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

institution, are presently calculated as individual ‘point’ or ‘average’ estimates,


and could usefully be turned into financial forecasts if based on distributions of
variables.
In addition to these strictly financial performance risk measures, other
techniques are available to assess the performance of such institutions according to
a whole range of technical and management criteria. The French Development
Agency (ADF) has experience in applying such techniques to developing coun-
try financial institutions, and, although they are primarily point value or logical
(i.e., ‘Yes’ or ‘No’) indicator-based at present, some of the variables (e.g., for like-
lihood of meeting certain targets by particular dates) could be subject to prob-
ability techniques.
For project operating entities and executing agencies, the primary concern
however is typically with earnings performance, cash flow, and overall FIRR and
FNPV estimates. Typically, financial statements for such entities are prepared, and
then standard sensitivity testing (i.e., ‘costs up 10%, 20%’; ‘revenues down 10%, 20%’;
‘revenues delayed by 1 or 2 years’, etc.) is conducted in exactly the same way as for
project EIRR/ENPV estimates. What is quite apparent in this respect is of course that
the modeling of risk to financial projections of project institutions is just as applicable
as it is to estimates of economic benefits for the project as a whole. Instead of individual
point estimates plus sensitivity testing, it is quite possible to model probabilistically
(for example) such financial estimates as
• current and constant terms price projections for inputs and outputs (i.e.,
probabilities of current price streams and also probabilities of particular
inflation rates applying)
• foreign exchange rate projections (relative appreciation and depreciation
during the life of the project)
• interest rates (e.g., on loans to sub-borrowers), and
• repayment rates (e.g., from sub-borrowers).

These kinds of variables are just as amenable (and probably even more so)
to probabilistic-based forecasting as any more ‘technical’ variables affecting a project’s
performance. In this respect, it may be recalled that the ‘Harberger’ critique of project
analysis practice of major multilateral agencies included the view that not only were
estimates of project costs typically understated and estimates of project benefits
typically overstated, but also that price estimates (e.g., for commodities, services,
wages) and exchange rate forecasts frequently turned out to be wildly inaccurate.
The greater use of quantitative risk analysis techniques, for at least some of these
essentially financial variables, would be of use in ensuring project financial
sustainability.
CHAPTER 5 Strengthening Risk Analysis in ADB Operations 67

Environmental sustainability risk analysis


Another major area in which the sustainability of project effects needs to be
ensured is that of impacts on the environment. Environmental sustainability of projects
is ensured to the extent that environmental costs and benefits are included within
the project’s economic analysis. Typically, investigating and estimating the economic
value of environmental costs and benefits is difficult; this is because
• biophysical relationships tend to be complicated (and thus hard to entirely
capture through sampling and statistical techniques) and data are usually
external to that collected during normal project preparation processes,
• market prices for many factors and project outputs do not exist, and
• different techniques for economic valuation may well lead to different results.

In addition, various biases (often stemming from professional perceptions and


backgrounds) may arise in determining which impacts are important or will be large.
In general, the situation with respect to valuing environmental impacts is
traditionally thought to be characterized more by uncertainty than risk, i.e., it is
impossible to attach probability distributions to particular outcomes. Typical ADB
practice has been that where environmental benefits have been valued at all, ‘lower
bounds’ from estimates of ‘expected values’ (e.g., from carbon sequestration, from
soil conservation) are quoted as the basis for benefit calculation—i.e., conservative
or pessimistic forecasts are used for description of the base case. Sometimes, sensitivity
testing is performed on such estimates where they are included in the base case
EIRR calculation—i.e., in a manner identical to the standard treatment of other economic
benefits. Occasionally a table summarizing ‘Omissions, Biases and Uncertainties’ in
such estimates is provided—along the lines of practice recommended in ADB (1997b)—
which simply indicates possible biases in the estimates for particular environmental
impacts and shows what the impact on the project EIRR might be if these were
corrected.
In many situations, however, it may be possible to extend this sort of analysis
such that instead of simple ‘uncertainty’ being reflected, it would be possible to
model the expected benefits in terms of their risk. The scope for the application of
such techniques will vary from situation to situation, depending upon factors such as
• how familiar the analyst(s) is with the biophysical circumstances under
consideration,
• the quality of primary data collected from such techniques such as hedonic
pricing, contingent valuation, travel cost methods, etc., and
• the relevance and applicability of costs/benefits transferred from secondary
sources (i.e., the benefits transfer method).
68 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

The utility of attempting to apply quantitative risk techniques to such situations


will vary according to the relative importance of environmental impacts within overall
benefits (and costs) streams—for natural resource management projects (with often
large but essentially unvalued benefits) the implications of such techniques may
well be significant. It may also be true that as experience with primary data collection
techniques for benefit valuation within Asia expands, and as more examples of benefits
transfer are applied, greater knowledge about such estimates’ variability will be
built-up—enabling moves towards the fuller modeling of risk rather than the sim-
plistic description of uncertainty.

Institutional sustainability risk analysis


The last dimension of sustainability traditionally considered in project economic
analysis is that of institutional sustainability. This is conceptually the dimension of
sustainability which is most difficult to capture quantitatively, and few examples of
quantitative benefit estimation or risk analysis from institutional performance exist.
However, some recent literature suggests that institutional performance and its risk
may reasonably be quantitatively estimated.
Institutional sustainability is usually considered in terms of both external factors
(i.e., the project institution(s) as located within a political/policy/sector context) and
internal factors (i.e., does the institution have sufficient resources to complete its
tasks? is there enough technical assistance provided? etc.). One major concern which
affects many projects (as well as most policy-based loans) is to what extent project
institutions will be able to implement policy changes which are critical to project
success, or at the very least survive in less than benign environments. A recent study
of the risks associated with institutional reform in Pakistan’s water sector (Dinal et
al. 1997) applied a multi-stage methodology to analyze quantitatively the probabil-
ity of reforms (involving the shift of policy and decision-making responsibilities away
from federal and state-administered agencies and towards decentralized autono-
mous public utilities and end-users/water groups—as such they could expect to be
contentious) succeeding or not. The key stages included evaluation of who would
win and lose from the reforms, definition of reform performance levels, identification
of the various ways agents would seek to influence reform implementation (and the
costs of such ways), and (lastly) application of a Delphic approach to estimate
probabilities of level of achievement of each reform. The Delphic approach involves
asking a group of experts (in this case managers from water agencies) to assign
probabilities to particular outcomes (in this case particular reform levels). Its advan-
tage is that it provides direct assessment of risks from a collection of subjective but
knowledgeable individuals and does not depend upon use of proxy measures—it
CHAPTER 5 Strengthening Risk Analysis in ADB Operations 69

could also be repeatedly performed throughout project implementation to monitor


change.
What, ideally, should emerge from a Delphic-based analysis of institutional
performance is therefore the best-possible guess from knowledgeable locals about
the institutional environment and the probabilities of particular outcomes expressed
in a quantitative form.

Supporting Poverty Reduction Objectives

In addition to more fully considering the likely distribution of overall economic


benefits and also ensuring that various aspects of sustainability are built into project
design, it was suggested that risk analysis could be useful in ensuring that poverty
reduction objectives were better targeted. These involve not only considering the
distribution of financial and economic outcomes at individual, household, farm, etc.,
level—in a way which is identical to describing project economic benefits and
enterprise/financial institution profitability, but also considering what target
groups’ attitudes towards risk are, given the context of their vulnerability. The
following table therefore summarizes how these various techniques can be
employed to support poverty reduction; it can be seen that they attempt to marry
project participants’ subjective circumstance and attitudes to risk with typical
probability-based risk descriptions. In addition, distribution analysis (i.e., analysis
of benefits by groups participating in the project) can also be approached in terms
of risk analysis.
Again, the primary focus of this sort of poverty analysis is not as an add-on
to eventual project description, but should be used as early as possible in project
preparation so that re-design can take place to more closely pursue ADB’s poverty
reduction objectives.

Risk Analysis and Policy-Based Lending

Policy-based lending is probably the area of ADB operations least disposed


to techniques of risk analysis.
In contrast to project economic analysis, PBL tends to be characterized by the
existence of a clear economic rationale for the intervention (i.e., the ‘why’ of the
intervention is clear—and usually some kind of reform is envisaged within a particular
70 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Table 7
Application of Risk Techniques for Poverty Analysis

Risk Analysis Technique


1 Describe textually nature and extent of vulnerability:
This can be approached in terms of consequences of loss (catastrophic or not), reversibility,
ability for households to have any control, possibilities for insurance, etc., as part of
Initial Social Assessments, Poverty Impact Assessment Matrix (in PBL).
2 Estimate risk aversion/focus of loss for participants:
Estimate quantitatively to the extent possible participants’ attitude to risks at different
income levels, and in particular investigate ‘focus of loss’ for project situations (e.g.,
where new technology is being introduced) where possible very low or negative outcomes
may have to be considered. These estimates can be derived through interview-based
techniques offering participants choices between specific but certain outcomes on the
one hand compared to higher but risky outcomes on the other.
3 Estimate income/welfare impacts at individual/household/farm level:
Estimate the distribution of individual, household or farm incomes (based on probabilistic
analysis of output quantities and prices, etc.), with the focus on the likelihood that returns
may be negative or unacceptable. (This analysis should include possibilities that ‘benefits
leakage’ will occur).
4 Calculate distribution of poverty impact ratio (PIR):
Based on the calculation of financial and economic benefits and their distribution between
groups, the PIR can be calculated and so can its distribution (as long as its estimation
is directly linked within the same spreadsheet as the rest of the project economic analysis).
A consideration of the likelihood that the project PIR may be below an acceptable level
(i.e., in relation to the proportion of the share of the poor in total population) should
be provided.
5 Justify the imposition/acceptance of any particular level of risk:
On the basis of steps 1-4, justify the project design in terms of its level of risk implied
for the project. This is likely to differ across project situations; for example, a 25%
chance of negative returns for farmers on an irrigation scheme may be acceptable if
their resource base is relatively stable, but would perhaps be unacceptable to impose
upon very poor communities in degraded watersheds.
CHAPTER 5 Strengthening Risk Analysis in ADB Operations 71

sector) although the actual mechanisms and processes by which impacts on particu-
lar groups are delivered (i.e., the ‘what’ of the intervention) are typically less clear.
It is therefore arguable that policy-based lending is inherently ‘uncertain’. Quanti-
tative relationships between individual variables and policy-based lending outcomes
are not usually examined in PBL, and for this sort of reason the application of quan-
titative risk techniques is limited.
However, some of the techniques already described for institutional and
‘subjective’ poverty analysis can be applied with PBL analysis. Specifically, it is
suggested that
• as the modified Poverty Impact Assessment matrix lays greater emphasis
on the use of inference, interview-data, and statistical data for examining
risks to which participants are exposed, in many situations of PBL, case
studies of individuals, households, farms, etc., can be used to explain typical
risk exposure and consequences, and likely attitudes towards risk by
target groups
• techniques for quantifying the probabilities of particular levels of
institutional performance be considered (as in Dinal et al. 1997) in
circumstances where major reform interventions are proposed.

It is also recommended that the approaches suggested previously to link any


identified risk to specific mitigation measures be followed in PBL.

Sectors and Projects: Some Typical Risk Analysis Situations

Each project design will encompass different sets of variables, many of whose
actual outcomes will be unknown, and therefore the analysis of risk in that project
can be as unique as each individual proposed project itself. Any proposed project
may be able to show, for example, how its expected EIRR/ENPV has a particular
probability of being acceptable (i.e., EIRR>EOCC, ENPV>0) depending upon values
for certain key variables, or that its expected cost-effectiveness is similarly dependent
on unknown but probabilistically described outcomes. It is also the case that many
projects will share similar overall concerns to ensure financial, environmental, and
institutional sustainability, and so the kinds of approaches to risk analysis already
suggested above to address such issues could equally apply to water supply, transport,
power, agriculture, etc., projects. What may be left to consider, therefore, are fairly
typical ‘technical’ issues as they occur across different sectors and as are frequently
faced by analysts.
72 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Table 8
Project Types and Some Possible Risk Analysis Considerations

Sector/Project Type Examples of Likely Analytical Concerns

Agriculture: Realized tree crop yields and production; factory/mill throughput;


Plantation/Estate future prices as determinants of farmers’ and/or estates’ incomes

Agriculture: Scheme maintenance; realized new and existing crop yields; crop
Irrigation prices; adoption/uptake rates; household and farm incomes

Forestry Volume of harvestable wood in 7-20 years time, and price of output
(e.g., pulp/wood) at that point

Fisheries Impact of new culture technologies from aquaculture; future stocks


and landings from capture; fish prices; determinants of fishermen’s
incomes

Environment and Natural Extent of identification, quantification and valuation of indirect,


Resources: Various non-use and option impacts of total economic value (TEV)

Transport: Construction costs in difficult or unknown environment; traffic


Rural Roads composition mixtures; extent of generated traffic and VOC savings

Transport: Construction costs, price elasticity of demand for new road use;
Highway/Toll Roads currency depreciation for loan repayment; sustainability of road
authority

Transport: Future passenger and/or freight volumes; extent of maintenance,


Railways and Ports/Shipping operating costs
Energy: Operating costs, consumer price elasticity of demand
Rural Electrification
Energy: Costs of inputs; poor maintenance of equipment; consumer demands
Power Generation/Transmission for power
CHAPTER 5 Strengthening Risk Analysis in ADB Operations 73

Potential Key Variables


Possible Variable and Data Characteristics
To Investigate
Price projections; Use of World Bank commodity price projections for exports;
tree crop yield estimates; machinery more ‘subjective’ estimates for locally-consumed items; yield
operating capacity/efficiency estimates for new crops may be based only on research trials
and need some adjustment; machinery estimates based on
design characteristics plus ‘subjective’ experience
Operating/water supply costs; yields and Cost estimates derived from similar schemes; WTP estimates
prices (as above); WTP estimates for water from interviews with target groups and extent of doubt about
demand; adoption/uptake of new varieties this can be derived at same time; adoption rates can be modeled
with triangular distribution as a minimum
Wood and by-product yields, losses to Considerable doubt about point estimates of volumes and prices
theft, harvest efficiency, etc. as determi- when wood is harvested a long time into the future; current real
nants of production in future periods prices plus considerable variation should be considered
Harvest yields and fish stocks; commodity Data on yields from new technologies may be from research
price projections and local variety only – possibly exclude extreme values; fish stocks well mod-
eled but sometimes highly mobile; price estimates may be
based on comprehensive data for commodities (e.g., for tuna)
or ‘guesses’ for local varieties
Quantities of particular biophysical Knowledge of the extents of physical impacts may be reasonably
impacts; alternative methodologies for well-known, but estimates of impacts’ economic value can
benefit estimation vary widely, based on both primary and secondary techniques
– consider wide range of possible values
Construction cost estimates; traffic volumes Construction costs likely to be reasonably well-known from
by types of vehicles; VOCs similar projects in the same country; traffic forecasts modeled
with several scenarios and associated probabilities; VOCs less
well-known - but triangular distribution as a minimum
Contractor’s / analysts’ estimates allow Construction costs likely to be reasonably well-known from
for several states of costs; price elasticity similar projects in the same country; WTP demand estimates
of demand for road use; foreign exchange and foreign exchange projections can be modeled with simplified
projections probability distributions (see material on ‘financial
sustainability’)
Costs estimates, passenger and freight Costs based on simplified distribution estimate; forecasts of
forecasts traffic demands can be modeled continuously if necessary
Capital and operating costs; consumers’ Costs based on simplified distribution estimate; distribution
demand schedules of WTP estimates can be derived at same time as averages
Costs of equipment; input prices; All subject to simplified probability distribution analysis (e.g.,
operating efficiency; consumer demands ADB/WB power)
74 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Table 8
Project Types and Some Possible Risk Analysis Considerations (continued)

Sector / Project Type Examples of Likely Analytical Concerns

Urban: Construction costs; value to consumers; willingness of authorities


Water Supply and Sanitation/ to pursue policy reforms (e.g., charges for service provision)
Wastewater /Solid Waste, etc.
Health: Service uptake rates; extent of cost recovery from rural poor; benefit
Primary Care estimation methodology (if applied in EIRR calculation)
Education: Nature of beneficiaries’ ultimate employment and the income
Secondary and Post-secondary differentials arising from such employment
Education: Numbers ultimately failing to find or accept work as teachers after
Teacher Training training

The following table attempts to indicate some of these typical project economic
analysis technical concerns on a sector-by-sector/project type basis, and to indicate
how risk analysis could be applied to consideration of some key variables for such
projects. The table is not exhaustive in its content; it is meant to be indicative and
general only.

Technical and Resource Considerations


(including applying ‘@RISK’)

The last issue to consider concerns possible implications in technical resource


terms for extending the analysis of risk in ADB operations.
Again, because the application of risk analysis is likely to vary greatly in nature
and extent across projects, it is difficult to develop firm conclusions about what may
be needed to support any desired expansion of current practice. However, it is possible
to draw out the following points related to technical and resource issues from the
foregoing analysis:
• any extended application of quantitative (i.e., probability-based) risk analysis
will require expansion of most project analysts’ statistical skills if errors in the
interpretation of results (i.e., following application of typical risk analysis
software) are not to be generated. This means that some kind of essentially
technical/statistical support needs to be made available within ADB
CHAPTER 5 Strengthening Risk Analysis in ADB Operations 75

Potential Key Variables


Possible Variable and Data Characteristics
To Investigate
WTP estimates; probability of success of Distribution of WTP estimates can be derived at same time as
implementing institutional reforms averages/point estimates; quantitative institutional reform
analysis may be considered
Use of services and consumer demand/ Economic values, for example, DALY (disability-adjusted life
ability to pay; estimated WTP year), may be contentious
Employment rates; income differentials Can be extensively modeled with continuous distributions if
necessary (see WB Mauritius example)
Policies such as school construction/funding Institutional aspects can potentially be investigated
programs; on-going institutional changes; quantitatively; modeling of employment through discrete or
employment rates continuous distributions

• in such a context, it may be useful to develop some ‘typical’ project-based


or sector-based models of anticipated statistical issues (e.g., to do with
expected correlations between typical determinant variables) or expected
distribution characteristics for particular variables
• it may also be possible to develop models for expected distributions of
cost items across sectors (increasing evidence suggests that capital costs
estimates for projects across a range of sectors may be log-normally
distributed, for example)
• for the construction and application of either simplified, discrete probability
distributions or fuller continuous distributions to generate estimates of
expected EIRR/ENPV and their associated variance, the use of some kind
of dedicated risk modeling software will be appropriate. While standard
spreadsheets can theoretically be used for such purposes, practitioners
must be prepared to either use only uniform distributions and/or develop
their own distributions through complex formulae application. There is no
reason for analysts to try to develop their own analytical tools when ‘off-the-
shelf’ solutions are now widely available
• dedicated risk analysis packages such as @RISK have far greater
functionality and ease of use than spreadsheets
• the use of @RISK is extremely simple. It can be applied to any existing
spreadsheets, and primarily involves the substitution of point values in
cells by user-specified distributions (in ‘input’ cells); this results in ‘output’
cells (e.g., EIRR or ENPV estimates) having distributions generated for
76 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

them, which can then be represented and analyzed both numerically and
graphically
• the usual considerations of designing spreadsheets with as many individual
variables specified separately therefore apply, and make the application
of @RISK quite possible even for any or all variables affecting project
outcomes
• there are a wide range of distribution types to choose from, and users can
specify central tendency, dispersion and cut-off characteristics where
appropriate; all specified distributions can be represented graphically
• distributions of various forms can easily be fitted from existing historical
or time-series data, and several alternative and complementary measures
of ‘goodness of fit’ are provided
• users can specify estimated covariance between variables in the form of
an easy to use ‘correlation matrix’
• typically, several thousand simulations can be processed in minutes using
@RISK on a fairly standard personal computer (PC)
• alternative sampling methods for these simulations can be used, including
standard Monte Carlo random sampling and the stratified Latin Hypercube
sampling
• the typical time taken to apply @RISK in this way is very short – typically
a couple of hours for the tasks specified above, once some form of
spreadsheet model for EIRR/PIR estimation has been set up
• for most practitioners, the major issue involved in applying @RISK (or any
such product) will be in correctly specifying distributions for existing or
forecast data and determining appropriate extents of covariance among
variables such that results for EIRR/ENPV distributions will be meaningful;
while it is easy to quickly generate attractive and precise outputs from
such software, the general adage applied to the use of powerful computer
programs of ‘rubbish in, rubbish out’ still applies
• @RISK will most typically be applied to demonstrate the probability that
project EIRR and/or ENPV will be unacceptable. However, it can also be
used to directly generate distributions for measures of distribution and
poverty impact (i.e., the poverty impact ratio, PIR), if such calculations are
in cells (which @RISK will designate as ‘outputs’) which depend upon
variables for which distributions have been substituted for point values.
For this reason, it is good practice in project economic analysis to ensure
that the PIR calculation (where undertaken) is seamlessly linked to the
spreadsheet containing the EIRR/ENPV calculations
CHAPTER 5 Strengthening Risk Analysis in ADB Operations 77

• for projects which are expected to be very large, marginal or particularly


uncertain (e.g., perhaps because they are new sorts of lending, involve
several countries, involve new technologies, etc.), the analysis of risk can
be expected to figure larger than in other situations. For this reason,
requirements for analysis of risk should be identified prior to PPTA and
included in the PPTA scope of work; the use of a dedicated risk analysis
package such as @RISK should be specified, in the same way that (for
example) COSTAB is specified for financial cost estimation
• because @RISK is so easy to use, it should be applied very early in project
design, specifically to investigate which variables are key determinants of
project outcomes and about which more data describing such variables’
distributions may be collected
• the undertaking of some form of risk-based analysis in the early stages
of project design and the presentation of risk analysis results in a PPTA
report would probably take only a few days work for an economist and/
or other staff.
APPENDIX 1 List of Reviewed ADB Projects and Other ADB Project Evaluation Documents 79

APPENDIX 1
List of Reviewed ADB Projects and Other ADB Project
Evaluation Documents

1 1698 THA Agriculture Sector Program 23-Sep-99


2 1779 KAZ Farm Restructuring Sector Development Program (Policy) 14-Nov-00
3 1814 PRC West Henan Agricultural Development 19-Dec-00
4 1404 VIE Fisheries Infrastructure Improvement 16-Nov-95
5 1656 PNG Fisheries Development 11-Dec-98
6 1770 INO Marine and Coastal Resources Management 26-Oct-00
7 1304 PRC Yunnan-Simao Forestation and Sustainable Wood Utilization 30-Jun-94
8 1515 VIE Forestry Sector 20-Mar-97
9 1643 BAN Sundarbans Biodiversity Conservation 27-Nov-98
10 1552 SRI Second Perennial Crops Development 25-Sep-97
11 1639 SRI Tea Development 10-Nov-98
12 1781 VIE Tea and Fruit Development 14-Nov-00
13 1592 KAZ Water Resources Management and Land Improvement 17-Dec-97
14 1753 CAM Stung Chinit Irrigation and Rural Infrastructure 05-Sep-00
15 1788 LAO Decentralized Irrigation Development and Management 28-Nov-00
Sector
16 1524 BAN Participatory Livestock Development 19-Jun-97
17 1772 PHI Infrastructure for Rural Productivity Enhancement Sector 31-Oct-00
18 1644 PRC Yunnan Dachaoshan Power Transmission 27-Nov-98
19 1732 NEP Rural Electrification, Distribution and Transmission 21-Dec-99
20 1809 PAK Capacity Enhancement in the Energy Sector 14-Dec-00
21 1818 PRC Wind Power Development 20-Dec-00
22 1548 MON Ulaanbaatar Heat Efficiency 25-Sep-97
23 1901 PRC Shen Da Transmission Interconnection 20-Dec-01
24 1715 PRC Shanxi Environment Improvement 07-Dec-99
25 1743 MON Second Financial Sector Reform Program 22-Jun-00
26 1735 THA Restructuring of Specialized Financial Institutions 21-Dec-99
27 1813 IND Calcutta Environmental Improvement 19-Dec-00
28 1554 KGZ Education Sector Development Program 29-Sep-97
29 1637 MLD Postsecondary Education Development 30-Sep-98
30 1718 VIE Teacher Training 14-Dec-99
80 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

31 1447 CAM Basic Health Services 20-Jun-96


32 1675 INO Health and Nutrition Sector Development Program 25-Mar-99
- Policy Loan
33 1749 LAO Primary Health Care Expansion 24-Aug-00
34 1704 IND Karnataka Urban Development and Coastal 26-Oct-99
Environmental Management
35 1745 PHI Pasig River Environmental Management and Rehabilitation 20-Jul-00
Sector Development Program - Program Loan
36 1646 THA Samut Prakarn Wastewater Management (Supplementary) 03-Dec-98
37 1755 NEP Small Towns Water Supply and Sanitation Sector 12-Sep-00
38 1757 SRI Water Resources Management 19-Sep-00
39 0426 MAL Bintulu Deepwater Port 23-Nov-79
40 1559 INO Belawan, Banjarmasin, and Balikpapan Ports 30-Sep-97
41 1584 PRC Xiamen Port 27-Nov-97
42 1561 BAN Jamuna Bridge Railway Link 02-Oct-97
43 1631 UZB Railway Rehabilitation 15-Sep-98
44 1747 IND Surat-Manor Tollway 27-Jul-00
45 1774 REG Almaty-Bishkek Regional Road Rehabilitation 31-Oct-00
(Kazakhstan Component)
46 1795 LAO Rural Access Roads 07-Dec-00
47 1489 THA Third Rural Telecommunications 26-Nov-96

Other ADB Project Evaluation Documents Consulted:

ADB Special Studies

Special Study of the Macroeconomic Environment and Project Performance in Sri Lanka.
Report no. SS0028. December 1997.

Factors Affecting Project Performance in the Agriculture and Social Sectors: A Review
of Postevaluation Reports between 1991 and 1997. Report No. SS0031. December 1998.

Special Evaluation Study on the Policy Impact of Involuntary Resettlement.


Report No. SS0041. October 2000.

Special Study: A Review of Postevaluation Findings in Thailand.


Report no. SS0005. October 1988.
APPENDIX 1 List of Reviewed ADB Projects and Other ADB Project Evaluation Documents 81

Special Study: A Review of Postevaluation Findings in Indonesia.


Report no. SS0006. November 1988.

Special Study: A Review of Postevaluation Findings in Malaysia.


Report no. SS0007. December 1988.

Special Study: A Review of Postevaluation Findings in Sri Lanka.


Report no. SS0008. April 1989.

Special Study: A Review of Postevaluation Findings in Nepal.


Report no. SS0009. August 1989.

Special Study: A Review of Postevaluation Findings in Bangladesh.


Report no. SS0010. August 1989.

Special Study: A Review of Postevaluation Findings in Western Samoa.


Report no. SS0013. June 1990.

Special Study: A Review of Postevaluation Findings in Papua New Guinea.


Report no. SS0014. December 1990.

Special Study: A Review of Postevaluation Findings in South Pacific Developing Member


Countries. Report no. SS0019. September 1991.

Special Evaluation Study on the Social and Environmental Impacts of Selected Hydropower
Projects. Report no. SS036. December 1999.

ADB Impact Evaluation Studies

An Impact Evaluation Study of Bank Operations in the Education Sector in Indonesia.


Report no. IE0022. October 1993.

An Impact Evaluation Study of Bank Operations in the Water Supply and Sanitation
Sector in Malaysia. Report no. IE0028. December 1994.

An Impact Evaluation Study of Bank’s Assistance in the Health and Population Sector
in Sri Lanka. Report no. IE0033. December 1995.
82 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Impact Evaluation Study of the Bank’s Benefit Monitoring and Evaluation Assistance to
the Agriculture. Report no. IE0035. December 1995.

An Impact Evaluation Study of Bank Assistance to the Industrial Crops and Agro-Industry
Sector in Sri Lanka. Report no. IE0038. July 1996.
APPENDIX 3 Illustration of Risk2Analysis
APPENDIX Sustainable
in Project
Livelihoods
Economic
Framework
Analysis 83

APPENDIX 2
Sustainable Livelihoods Framework
84 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

APPENDIX 3
Illustration of Risk Analysis in
Project Economic Analysis

Introduction

This appendix contains case studies illustrating the application of risk analysis
into the project economic analysis of various ADB projects using the @RISK software.
The case studies have been selected to represent a number of different types of projects
and to illustrate different technical features of the application of the @RISK software.

Project Title Major Technical Feature Illustrated


1 Wheat Productivity Improvement Project Substitution of triangular distribution for cell
point values
2 Second Nonformal Education Project Importance of correlations between variables
Bangladesh, 2001
3 Primary Healthcare Expansion Project Application of risk analysis to cost-effective-
Lao PDR, 2000 ness analysis, and choice among alternative
distributions
4 Secondary Education Modernization Project Construction of variables’ distributions based
Sri Lanka, 2000 on secondary data, and difference between
modal/most likely values and full distributional
estimation
5 Shen Da Power Transmission and Grid Risk analysis applied to poverty impact ratio
Rehabilitation Project
People’s Republic of China, 2001
6 Wind Power Development Project Comparability of @RISK with Risk Master
People’s Republic of China, 2001 software

The various projects and the points which the case studies illustrate are
summarised in the following table:
@RISK is an Excel add-in and was applied to the original (i.e., PPTA) EIRR
calculations. The essence of @RISK is that it allows single point values in spreadsheet
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 85

cells (e.g., for costs, revenues) to be substituted by user-specified probability


distributions such that cells which include results (e.g., EIRR, ENPV) dependent upon
those original sources can be described in terms of probability distributions of
outcomes. Such distributions (for both input data and for output results) are
summarised by @RISK and can be graphed and manipulated in various ways.
The case studies used the original Excel spreadsheets developed for the project
economic analysis. In all cases, the analysis described for each project took only a
few hours to prepare, including the acquisition of familiarity with the original economic
analysis and also some exposure (without formal training) to a trial version of @RISK.
86 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Case Study 1
Wheat Productivity Improvement Project

Introduction

The project is concerned with raising the productivity of wheat cultivation in


an ADB developing member country. The economic analysis at the appraisal stage
suggested that the project’s EIRR was high, at 33.7%—based on PPTA consultants’
‘best estimates’ of a number of key variables. The application of a risk analysis software
package (i.e., @RISK) to the original Excel spreadsheets can augment the economic
analysis and supply further information about, and confirmation of, the project’s
robustness.

Sources of Risk

Values for several variables included in the calculation of the ‘base case’
estimate of EIRR were considered to be subject to some uncertainty. These were
• the estimated value for the shadow exchange rate factor (SERF) – little
formal data upon which to estimate this parameter was available
• the amount of crop losses avoided (calculated in tonnes, deriving from
estimates of proportions of current losses now saved) by the proposed
project through reductions in incidence of rust on wheat
• the amount of crop losses avoided (calculated in tonnes, deriving from
estimates of proportions of current losses now saved) by the proposed
project through reductions in shattering of wheat grains, and
• the actual extent of areas (calculated in hectares) which would be subject
to improved soil and water management regimes through the process of
‘land levelling’—in project years 1 to 7.

In each case, the PPTA estimates of values were single point ‘most likely’
values, in effect modal values.
For the SERF, it was felt that (while the minimum value was necessarily 1)
there was some doubt about the maximum value, which might conceivably be as
high as 3.5. For the crop losses avoided (rust and shattering reductions), it is pos-
sible that no benefits at all might be obtained (e.g., if new technology failed to work)
but also that savings of 7% and 10%, respectively could be possible. Similarly for land
levelling, it was possible that no areas at all might be affected, but also that likely
targets could be exceeded in very favourable circumstances.
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 87

Application of @RISK Software

The table below shows for each uncertain variable in the PPTA analysis, the
original point value used in the ‘base case’ EIRR calculation, and the characteristics
of the distributions which were substituted for these point values. In the absence of
any historical or empirical data, very simple triangular (minimum possible, most likely,
maximum possible) distributions were used for the present example.

Original Most Maximum


Point Type of Distribution Minimum Likely Possible
Variable Value Substituted Value Value Value

SERF 2 Triangular 1 2 3.5


Rust losses avoided 3% Triangular 0 3% 7%
Shattering losses avoided 7% Triangular 0 7% 10%
Land levelled area
(000 ha; years 3–6) 18 Triangular 0 18 25
Land levelled area
(000 ha; year 7) 28 Triangular 0 28 40

Substituting the distributions for the original point values has the immediate
effect of replacing the original (i.e., most likely) values with the mean values for the
new distributions—and thus recalculates the ‘base case’ EIRR. In this case, the mean
values for the four variables become as follows:

New Distribution
Variable Original Point Value Mean Value

SERF 2 2.167
Rust losses avoided 3% 3.33%
Shattering losses avoided 7% 5.67%
Land levelled area
(000 ha; years 3-6) 18 14.333
Land levelled area
(000 ha; year 7) 28 22.666
88 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Based on such mean values in the spreadsheet, the new estimated value for
the project EIRR falls to 30.3% (from the original 33.7%).
Having specified the distributions, a simulation was run in which possible
values for the variables were randomly sampled 1000 times using a Monte Carlo,
non-stratified (i.e., not Latin Hypercube) technique. No correlation between variables
was assumed to exist.

Risk Analysis Results

The main output from the foregoing analysis is a cumulative distribution


function for the project’s EIRR. The graph below summarises the results.

Based on the data randomly sampled from the 1000 iterations, the expected
EIRR is now 30.02%. What also is apparent from this exercise is that there is only
about a 1% probability that the EIRR will be below 22%.
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 89

Case Study 2
Second Nonformal Education Project
(Bangladesh, 2001)

Introduction

The project was concerned with establishing an effective community-based


continuing education program for poor and disadvantaged neo-literates (at least 50%
of direct beneficiaries are expected to be women). Quantified economic benefits from
the project are those captured privately through higher daily earnings. The economic
analysis suggested that the project’s EIRR was high, at 35.3% in the base case—
based on PPTA consultants’ best estimates of a number of key variables. Two ‘pes-
simistic’ scenarios—one of ‘low wages’, the other of ‘low employment’ were also
modelled in the original analysis, as was the ‘switching scenario’ (i.e., a combination
of low wages and low employment which would cause the project to become non-
viable).
The application of @RISK to the appraisal Excel spreadsheets can augment
the economic analysis and supply further information about, and confirmation of, the
project’s robustness. In this case, the importance of correlations between variables
determining project outcomes is also apparent.

Sources of Risk

Values for several variables included in the calculation of the ‘base case’
estimate of EIRR were considered to be subject to some uncertainty. These were
• the proportion of participants completing the education programme
• the proportion of graduate trainees finding employment
• the days worked per month by trainees
• the months worked per year by trainees
• the incremental wage gained by trainees due to better education, and
• the number of trainees at full operation of the programme.

In each case, the RRP/appraisal estimates of values in the base case appear
to be single point ‘most likely’ values, in effect modal values.
However, for the proportions of trainees both completing the programme and
finding work, there may be some doubt about the extent of the possible downside
(e.g., actual employment could conceivably be below the ‘low employment’ scenario
modelled in the original analysis, for example, if general macroeconomic conditions
90 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

deteriorated), although it is probably very unlikely that more than 95% of partici-
pants would both graduate and find work. Similarly, the potential downside of days
and months worked by trainee graduates is apparent, although there are more natu-
ral upper limits likely to be in operation.

Application of @RISK Software

The table below shows for each uncertain variable in the RRP analysis, the
original point value used in the ‘base case’ EIRR calculation, and the characteristics
of the distributions which were substituted for these point values. In the absence of
any historical or empirical data, very simple triangular (i.e., minimum possible, most
likely, maximum possible) distributions were used for the present example.

Original Most Maximum


Point Type of Distribution Minimum Likely Possible
Variable Value Substituted Value Value Value

Proportion completing
training 0.9 Triangular 0.6 0.9 0.95
Proportion of trainees
finding employment 0.9 Triangular 0.6 0.9 0.95
Number of days worked
per month 20 Triangular 12 20 26
Number of months
worked per year 6 Triangular 3 6 10
Daily wage increment
due to training ($/day) 0.25 Triangular 0.15 0.25 0.35
Trainees at full operation
of project 1,600,000 Triangular 1,200,000 1,600,000 1,800,000

Substituting the distributions for the original point values has the immediate
effect of replacing the original (i.e., most likely) values with the mean values for the
new distributions—and thus recalculates the ‘base case’ EIRR. In this case, the mean
values for the four variables become as follows:
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 91

New Distribution
Variable Original Point Value Mean Value

Proportion completing training 0.9 0.82


Proportion of trainees finding
employment 0.9 0.82
Number of days worked per month 20 19.33
Number of months worked per year 6 6.33
Daily wage increment due to training
($/day) 0.25 0.233
Trainees at full operation of project 1,600,000 1,533,333

Based on such mean values in the spreadsheet, the new estimated value for
the project EIRR falls to 28.4% (from the original 35.3%).
It was assumed that some correlation existed among these variables,
specifically between the proportions finding employment and days and months
worked. Accordingly, a correlation matrix was complied in @RISK which specified
a positive 0.75 correlation between all three of these variables (implying that more
people were likely to find work for more days and months as the general level of
employment rose). In effect, the correlation matrix for the three variables looks like
the following:

Original Point Days Worked Months Worked


Variable Value Per Month Per Year

Percent employed 1.0 0.75 0.75


Days worked per month 0.75 1.0 0.75
Months worked per year 0.75 0.75 1.0

Having specified the distributions, a simulation was run in which possible


values for the variables were randomly sampled 5000 times using a Monte Carlo,
non-stratified (i.e., not Latin Hypercube) technique.

Risk Analysis Results

The main output from the foregoing analysis is a cumulative distribution


function for the project’s EIRR. The graph below summarises the results.
92 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Based on the data randomly sampled from the 5000 iterations, the expected
EIRR is now 28.6%. What also is apparent from this exercise is that there is about a
13% probability that the EIRR will be below the EOCC of 12%.
If the correlation is omitted from the simulation, the results are similar—the
expected EIRR is slightly lower (27.6%), but the chances of project failure are now
estimated to be reduced (to about 6%, or one chance in 20).
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 93

Case Study 3
Primary Healthcare Expansion Project
(Lao PDR, 2000)

Introduction

This project was concerned with expanding primary health care in eight
northern provinces in Lao PDR. It was designed to target women, children, ethnic
minorities, and the rural poor by providing more cost-effective and better-focused
access to improved quality services.
A major feature of the economic analysis of the project is that it is based on
a monetary measure of cost-effectiveness per disability-adjusted life year (DALY)
saved (i.e., rather than an economic rate of return estimate). The analysis of cost-
effectiveness in the RRP depended on the use of three scenarios to model doubts
about the estimates of effective coverage of the project; risk analysis can be used to
refine this type of approach. The application of @RISK to the original Excel
spreadsheets can thus augment the economic analysis and supply further informa-
tion about, and confirmation of, the project’s robustness (in this case about where
the expected values per cost of DALY saved lie in relation to measures of similar
projects). In addition, in this case, the analysis using @RISK also tries to show the
importance of properly understanding and defining variables’ distributions as de-
terminants of estimates of project economic outcomes.

Sources of Risk

The greatest difficulty in economic analysis of heath projects lies in the


uncertainties of the reduction of disease burden, or saving in the otherwise lost DALYs.
The uncertainty arises both from doubts about changes in the incidence of disease
which project interventions bring and also about the actual coverage of service delivery
which the project will achieve.
The economic analysis contained in the project’s RRP deals with this uncertainty
by modelling three scenarios (‘low’, ‘moderate’, ‘high’), each of which is characterised
by a differential proportion of the target population (grouped by women, children
and the total population) which is covered by the primary health care (PHC) ser-
vices. The table below shows details of such estimated values for the early years of
the project.
As a result of the total costs of the project being distributed over different
numbers of estimated DALYs, the economic and financial costs of each DALY differ
94 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

DALYs saved under: 2001 2002 2003 2004 2005 2006


Low Scenario – 8,139 16,668 25,603 34,956 44,744
Moderate Scenario 11,922 24,417 37,504 51,205 65,543 67,116
High Scenario 15,896 32,556 50,005 68,274 87,391 89,488

by scenario, and range from $45.82 per DALY under the ‘low’ scenario to $21.09 per
DALY under the ’high’ scenario (the ‘moderate’ scenario value is $28.11).

Application of @RISK Software

The data in the original spreadsheets were used for the distribution-based
calculations. The population coverage estimates for women, children, and the total
population contained in the ‘moderate’ scenario were used as the basis for the
estimation of two sorts of probability distributions. These distributions were
• a normal distribution, with the mean being the value from the ‘moderate’
scenario and the standard deviation being 25% of the mean, and
• a triangular distribution, with the minimum value being the ‘low’ scenario
estimate, the most likely (i.e., modal) value being the ‘moderate’ scenario
value and the maximum value being the ‘high’ scenario value

It may be argued that either or both of these distributions might reasonably


model the uncertainty with respect to possible values for project coverage of the
population. Both exploit data points from the original PPTA/RRP work, are centrally
fixed on the original ‘moderate’ value estimates, and appear to have believable
measures of, or limits to, dispersion.
Accordingly, the original point values in the cells for the moderate scenario
(see below, for early years of the project) were substituted by the particular normal
distribution just described and then (in a second separate simulation) by the triangular
distribution also described.
As in the other case studies, when distributions are substituted for point
values the original cell point value is replaced by the mean of the particular
distribution. In the case of the normal distribution replacement the mean is the
same as the original ‘moderate’ values—and so the final cost/DALY estimate is
unaffected. However, with the triangular distribution substitution the effect in
this case is to reduce the mean estimate, and thus to increase the estimated cost/
DALY.
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 95

Moderate Scenario Population Estimates

2001 2001 2003 2004 2005 2006


Women 15–44 12,600 25,805 39,636 54,116 69,269 70,931
Children 0–4 9,355 19,158 29,427 40,178 51,428 52,662
Total Population 62,944 128,909 198,005 270,343 346,039 354,344

Having specified the distributions, two simulations were run in which possible
values for the variables were randomly sampled 5000 times using a Monte Carlo,
non-stratified (i.e., not Latin Hypercube) technique.

Risk Analysis Results

The main output from the foregoing analysis is a cumulative distribution


function for the project’s cost-effectiveness, measured in terms of the cost per DALY
saved, for each of the two types of distribution. The graphs below summarise the results.
The first graph shows the cumulative distribution function for the project’s
cost per DALY based on the normal distribution. The mean of the distribution (at
$29.09) is very close to that of the original moderate scenario. It can also be seen that
there is only a 5% chance that the actual cost of a DALY would exceed $40 (well
within acceptable ADB parameters).
96 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

The second graph shows the same result, but now based on the triangular
distribution instead of the normal one. The mean value for the distribution in this
case is some 33% higher than previously (at just under $40), and the 5% upper value
unlikely to be exceeded is now some $70.
Although the particular project design in this instance may find both results
within acceptable limits, this example does highlight the critical importance of
specifying variables’ distributions correctly; even two apparently reasonable-look-
ing distributions for the same variables can produce quite different results in terms
of estimates of project costs or returns.
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 97

Case Study 4
Secondary Education Modernization Project
(Sri Lanka, 2000)

Introduction

The objective of the project was to make Sri Lanka more economically productive
and competitive by modernizing the secondary education system through improved
curriculum instruction, school-based assessment (SBA), and school-based
management (SBM). Specifically, the project design aimed to increase percentages
qualifying to enter grade 12 from 35% to over 50% by 2005, increase the percentage
qualifying to enter grade 12 from 15% to 20% for rural students by 2005, to implement
SBA and examination reform by 2005, and implement a SBM system in all secondary
schools by 2005.
The project economic analysis calculated an EIRR based on incremental
earnings of those receiving improved education in a competitive labor market. This
is taken as a proxy for the incremental social benefit to the country as a whole. Because
of doubts about actual examination pass rates and income differentials between
non-educated and educated workers, a sensitivity analysis was conducted using
different values for these variables.
In this case study, the analysis using @RISK shows how data from other sources
(e.g., from similar projects in other countries, from labor surveys, etc.) can be used
to construct probability distributions which can then be substituted for the original
point values. It also shows the implications for estimates of project returns based on
point values as compared to those based on distributional data.

Sources of Risk

In the case of this project, major sources of uncertainty include


• differences which will be achieved in earnings between those with upper
secondary, ‘O’ level (O/L) and ‘A’ level (A/L) education compared to laborers
with no schooling, and
• examination pass rates at grades 11 and 13.

The project sensitivity test results in this respect are summarised in the
following table. It can clearly be seen that the returns from the project may be affected
adversely if there is a fall in the O/L and A/L pass rates.
98 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Sensitivity Test EIRR


Base case 20%
Reduce O/L pass rate from 50% to 45% and A/L pass rate from 60% to 55% 11%
Increase O/L pass rate from 50% to 55% and A/L pass rate from 60% to 65% 27%
Reduce income differentials by 10% between worker with upper secondary
education and O/L worker, and between O/L worker and A/L worker 18%

Application of @RISK Software

In the previous case studies, @RISK generated distributions for variables in


cells based on either one or two user-specified characteristics (i.e., mean and variance/
standard deviation) or a few data points (e.g., minimum, maximum and most likely
– in the case of the triangular distributions). In this case study, hypothetical data
from secondary sources covering examination pass rates and workers’ earnings are
entered into the spreadsheet and distributions are then fitted to the variable(s) in
question. The software user is able to choose between different types of distribution to
be fitted (e.g., normal, lognormal, beta, exponential, etc.) based on various measures of
the ‘goodness of fit’ to the available data (including Chi-square, K-S and A-D tests).
The data used for the project economic analysis base case is contained in the
following table:

Sri Lanka: Secondary Education Modernization Project:


Base Case Basic Data
Wage of laborer with no schooling
1,025.36 x 4 x 12 x 115/95 59,579
Wage of laborer with upper secondary education
1,336.36 x 4 x 12 x 115/95 77,591
Wage of employed O/L worker
1,822.45 x 4 x 12 x 115/95 105,894
Wage of employed A/L worker
2,426.40 x 4 x 12 x 115/95 140,987
Grade 11 exam pass rate
Implementation year 1 (and without project) 36%
Implementation year 5 50%
Grade 13 exam pass rate
Implementation year 1 (and without project) 48%
Implementation year 5 60%
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 99

The earnings data come from the Household Income and Expenditure Survey
(1995/96) conducted by the Department of Census and Statistics, and the figure quoted
for A/L wages (SRL Rupees 140,987) is presumably an average (mean) value for the
country as a whole. For simplicity, the risk analysis will look more closely at variability
associated with only the earnings of A/L workers and with grades 11 and 13 pass
rates, although it could equally be applied to any variables in the project economic
analysis.
For illustration purposes, suppose historical data were available (for example)
for wages of A/L workers (e.g., from cross-sector employment surveys) and grade
11 and 13 pass rates (e.g., from similar projects in other countries implemented in
recent years) in the following form:

Earnings of A/L workers:


142000 141370 139800 146000 142850 136000 141100
141200 141350 140980 148000 135500 141500 143100
140800 130600 144500 143000 138700 138000 140987
142100 141700 139500 143500 141000 139500 143000

Grade 11 pass rates:


50 45 53 49 48 52
52 50 56 45 50 50

Grade 13 pass rates:


60 55 63 59 58 62
62 60 66 55 60 60

@RISK allows a distribution to be fitted by users to such observations. The


following chart shows a normal distribution fitted to the earnings data. The distribution
has an identical mean to the original spreadsheet point value, and a (relatively small)
standard deviation of 3370. The software generates distributions of different types,
and ranks them according to various measures of goodness of fit. The default measure
is the Chi-square value.
It should however be noted that different measures of goodness of fit (e.g.,
the Kolmogorov-Smirnov statistic, the Anderson-Darling statistic, etc.) may rank dis-
tributions differently. Considerable care should therefore be exercised in interpreting
100 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

the statistics and applying particular fits. In this case, the normal fitted distribution
has been chosen because its mean is identical to the original point value data, and
the only (very slightly) better fit in terms of its Chi-square value—a beta (general)
fit—has a different predicted mean (on both other major estimates of goodness of
fit—the K-S and A-D statistics—the normal fit is better).

For the examination pass data, again normal distributions (for each of the
grade 11 and 13 pass rates) have been chosen to replace each of the original point
values because the mean of these distributions alone among the available choices
is the same as the original values, although in this case ‘Logistic’ and ‘Extvalue’
distributions have perhaps slightly better fits based on their K-S and A-D measures—
even though all three distributions’ Chi-square values are the same.
The following charts show (as an example, the grade 11 values in the
distribution of that variable are identically-distributed but 10% lower in all cases)
the original grade 13 examination pass rate data grouped by class interval and the
various fitted distributions of different types:
Having specified the distributions for the three variables, a simulation was
run in which possible values for the variables were randomly sampled 5000 times
using a Monte Carlo, non-stratified (i.e., not Latin Hypercube) technique.
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 101

Risk Analysis Results

The distribution which


results for the EIRR on this basis
has a mean which is very close
to the base case mean—
unsurprisingly, given that the
selected variables are modelled
to be normally distributed
around identical mean values to
those in the original spread-
sheets. Based on the (hypo-
thetical) distribution data it
appears that there is only about
a 7% chance that the EIRR for
the project would be below the
EOCC of 12%.
The examination pass
rate data in this case study can
also be used to demonstrate
one other point about risk (i.e.
probability-based) analysis as
applied in such situations. In
many project analysis circum-
stances, it is not clear whether
point value data in spreadsheet
cells represent the analyst’s
‘best guesses’ of ‘most likely’
(or somehow ‘typical’) values or
whether it represents average/
mean expectations. In fact great
differences in results can
emerge in calculations of
returns, depending upon how
such estimates are used.
Consider the following
example (hypothetical – and
unrelated to the actual project
under consideration). The
102 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

estimates of exam pass rates in the original data (i.e., 50% for grade 11 and 60% for
grade 13) may in fact represent the analyst’s best guesses of ‘most likely’ outcomes
(i.e., the single value from among a range most likely to occur, unadjusted for any
probability of occurrence). The following distribution of observations in such a case
may suggest that values of 50% and 60% passes are the values most likely to occur
(in nine out of twelve observations), even though other (lower) values can also occur.

Grade 11 pass rates


50 50 50 50 50 50
40 40 40 50 50 50

Grade 13 pass rates


60 60 60 60 60 60
50 50 50 60 60 60

If estimates of EIRR/ENPV, etc. are based on the most likely point values alone
(i.e., 50% and 60%) we get the original base case estimate of 20% EIRR. If we construct
probability distributions around such values with the original points as their mean
we get a similar expected EIRR value and a distribution as shown in the chart
immediately above.
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 103

If, however, we use the actual data from observations as just shown in the
last table (and which could have been used to derive the original estimates based
on a ‘most likely’ or modal basis as opposed to a mean-based one) the mean and
distribution of EIRR are as follows. It can be seen that not only is the estimate of the
expected EIRR proportionately lower, but that the risk of project failure has now
increased over fourfold (to 29% from about 7%).
104 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Case Study 5
Shen-Da Power Transmission and Grid Rehabilitation Project
(People’s Republic of China, 2001)

Introduction

The project aims to increase the capacity and efficiency of electricity


transmission from the northeastern part of the PRC to the southern part of Liaoning
Province, thereby increasing the availability of electricity and reliability of the electricity
transmission and distribution systems in Liaoning Province. Furthermore, an integrated
regional grid that will enable restructuring of the power sector in Liaoning Province
will be developed. The major economic benefits for the project are efficiency
improvement (reduced losses) and increased supply.
RiskMaster was used on the original Excel spreadsheets to supply further
information about the PIR of the project. For this case study illustration, @Risk was
used. The dimension of risk in project analysis is of significance in discussions of
poverty since the poor are the most vulnerable to unexpected unfavorable outcomes.
A reduction in vulnerability is a central element in a poverty reduction program and
a sharing of risk across a wide portfolio of projects may still leave the poor excessively
exposed. Hence, for important, poverty-focused projects for which monetary estimates
of costs and benefits to the poor are feasible, it is desirable that both the probability
of overall project failure and the probability of a negative outcome for the poor be
assessed.

Sources of Risk

The risk analysis was carried out for the PIR of the Project to ensure that there
would be no negative outcome for the poor. Five variables for the PIR analysis have
been considered to be subject to some uncertainty. These were:
• net financial benefits
• net economic benefits
• the share of consumer surplus to the poor
• the share of the Government net benefits to the poor
• the share of labor surplus to the poor.

In each case, the RRP estimates of values were single point ‘ most likely’ values.
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 105

Risk Variables Report


106 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Risk Variables Report (continued)

Application of @RISK Software

The probability distributions attached to the selected variables and graphed


below take into account past ADB experience in power projects in general and the
project/country circumstances in particular. The table below shows the five selected
variables including the original point value used in the PIR calculation, the assumed
value ranges, and the assigned probability distributions. The risk analysis was carried
out using the Monte Carlo simulation technique. The results were based on 5000
iterations; no correlation between variables was assumed to exist.
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 107

Risk Analysis Results

The results of risk analysis have shown that the expected PIR, based on the
weighted average of all simulated combinations is 19.8% (with a standard deviation
of 3.6%), slightly lower than the base case value (i.e., without the consideration of
risk). PIRs range from a minimum of 6.6% to a maximum of 26.9%.
The probability of the PIR falling below the value for the poor as a proportion
of the population as a whole, and thus not having a pro-poor impact, is estimated to
be about 12%.
108 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Case Study 6
Wind Power Development Project
(People’s Republic of China, 2000)

Introduction

This project will produce electricity in an environment-friendly manner and


increase the share of wind-based electricity in overall power generation through the
establishment of three grid-connected wind farms in the Xinjiang Uygur Autono-
mous Region and Heilongjiang and Liaoning provinces. It will avoid emissions of
SO2, NOX, TSP, and CO2 associated with conventional thermal power generation.
The calculation of the EIRR covered 1999-2022, and used constant 1999 prices,
and a discount rate of 12%. Tradable commodities were valued at border prices at the
prevailing exchange rate. Nontradable commodities were valued at shadow prices
using a mix of standard and specific conversion factors. The main benefits are the
incremental supply of electricity, valued through the avoided costs of supplying an
equal amount of electricity as the wind-based electricity to be produced under the
Project, and environmental benefits.
Risk analysis on the original Excel spreadsheets was originally carried out
using Risk Master. In this case, the analysis has utilized @RISK to test its comparability
to Risk Master.

Sources of Risk

Project parameters that were selected as risk variables based on the sensitivity
analysis include:
• capital cost
• generation
• foreign exchange rate
• commission date
• avoided cost.

In each case, the RRP estimates of values were single point ‘ most likely’ (i.e.,
‘modal’) values.
APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 109

Application of @RISK Software

The probability distributions attached to the selected variables take into


account past ADB experience in power projects in general and the project/country
circumstances in particular, as well as extensive discussions with the executing
agencies and other relevant agencies. A Monte Carlo simulation was used to model
the likely distribution of these risk variables and the associated EIRRs. The table
below shows the five selected variables including the assumed value ranges and
the assigned probability distributions.

Risk Variables Report


110 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Risk Variables Report (continued)


APPENDIX 3 Illustration of Risk Analysis in Project Economic Analysis 111

The results were based on 3000 iterations; no correlation between variables


was assumed to exist.

Risk Analysis Results

The results of risk analysis using @RISK shows that the expected EIRR, based
on the weighted average of all simulated combinations is 12.6% (with a standard
deviation of 2.5%)—about 0.3% lower than the base case EIRR of 12.9% (without
consideration of risks). The EIRRs associated with assumed probabilities range from
a minimum of 4.8% to a maximum of 22.6%. For this project, the probability of the
EIRR to be below the considered discount rate of 10% is 15.4%.
The expected EIRR using Risk Master produced a weighted average of 12.7%
with a standard deviation of 2.4%—involving only a slight difference of 0.1% between
the two software programs.

EIRR (Local Env. Only)


Expected value 12.6%
Standard deviation 2.5%
Minimum 4.8%
Maximum 22.6%
Probability of negative outcome 0.0%
112 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

References
Adhikari, R. and J. Weiss. 1999. Economic Analysis of Subregional Projects. EDRC
Methodology Series No. 1. March 1999. Manila: ADB.

Anderson, J.R. 1974. “Sparse Data, Estimational Reliability and Risk-Efficient


Decisions.” American Journal of Agricultural Economics.

Arrow, K. and N. Lind. 1970. “Uncertainty and the Evaluation of Public Investment
Decisions.” American Economic Review. Volume 60 No. 3.

ADB. 1979. Report and Recommendation of the President to the Board of Directors
on a Proposed Loan to Malaysia for the Bintulu Deepwater Port Project. Manila.

ADB. 1997a. Guidelines for the Economic Analysis of Projects. Manila: ADB.

ADB. 1997b. Workbook on the Economic Valuation of Environmental Impacts. Manila:


ADB.

ADB. 2000a. Report and Recommendation of the President to the Board of Directors
on a Proposed Loan to Sri Lanka for the Secondary Education Modernization Project.
Manila.

ADB. 2000b. Report and Recommendation of the President to the Board of Directors
on a Proposed Loan to the Lao People’s Democratic Republic for the Primary Health
Care Expansion Project. Manila.

ADB. 2000c. Report and Recommendation of the President to the Board of Directors
on a Proposed Loan to the Philippines for the Infrastructure for Rural Productivity
Enhancement Sector Project. Manila.

ADB. 2000d. Report and Recommendation of the President to the Board of Directors
on a Proposed Loan to the People’s Republic of China for the Wind Power Development
Project. Manila.

ADB. 2001a. Handbook for Integrating Poverty Impact Assessment in the Economic
Analysis of Projects. Manila: ADB.
APPENDIX 1 List of Reviewed ADB Projects and Other ADB Project Evaluation RDocuments
EFERENCES 113

ADB. 2001b. Report and Recommendation of the President to the Board of Directors
on a Proposed Loan to the People’s Republic of China for the Shen Da Power
Transmission and Grid Rehabilitation Project. Manila.

ADB. 2001c. Guidelines for the Financial Governance and Management of Investment
Projects Financed by the Asian Development Bank. Manila.

ADB. 2001d. Report and Recommendation of the President to the Board of Directors
on a Proposed Loan to Bangladesh for the Second Nonformal Education Project. Manila.

Belli, P., J. Anderson, H. Barnum, J. Dixon, and J. Tan. 2001. Economic Analysis of
Investment Operations: Analytical Tools and Practical Applications. Washington D.C.:
World Bank Institute.

Bolt, R. and M. Fujimura. 2002. Policy-based Lending and Poverty Reduction: An


Introduction to Processes, Assessment and Options. ERD Working Paper Series No. 2.
Manila: ADB.

Clarke, R. and A. Low. 1993. “Risk Analysis in Project Planning: A Simple Spread-
sheet Application using Monte Carlo Techniques.” Project Appraisal. Volume 8
Number 3: 141-6. September 1993.

Dinal, A., T.K. Balakrishnan and J. Wambia. 1997. “Political Economy and Political
Risks of Institutional Reforms in the Water Sector.” World Bank Working Paper (draft),
Washington DC: World Bank.

European Commission. 1997. Financial and Economic Evaluation of Development


Projects.

Fujimura, M. and J. Weiss. 2000. Integration of Poverty Impact In Project Economic


Analysis: Issues in Theory and Practice. EDRC Methodology Series No. 2. October
2000. Manila: ADB.

Harberger, A. 1998. “Economic Project Evaluation, Part I: Some Lessons from the 1990s.”
Canadian Journal of Program Evaluation. Special Issue: 5–46.

Johnson, K. 1985. Risk Analysis and Project Selection: A Review of Practical Issues.
ADB Economics Staff Paper No. 28. Manila: ADB.
114 HANDBOOK FOR INTEGRATING RISK ANALYSIS IN THE ECONOMIC ANALYSIS OF PROJECTS

Little, I.M.D., and J. A. Mirrlees. 1974. “Risk, Uncertainty and Profit.” In Project Appraisal
and Planning for Developing Countries. London: Heinemann.

Mariano, C. 2001. Risk Analysis Software Packages Evaluation Report. Unpublished


report under RETA 5932. Manila: ADB.

Pouliquen, L. 1970. Risk Analysis in Project Appraisal. World Bank Staff Occasional
Paper No. 11. Baltimore: The Johns Hopkins University Press.

Rayner, N., A. Lagman-Martin, K. Ward. 2002. Integrating Risk into ADB’s Economic
Analysis of Projects. ERD Technical Note No. 2. May 2002. Manila: ADB.

Reutlinger, S. 1970. Techniques for Project Appraisal under Uncertainty. World Bank
Staff Occasional Paper No. 10. Baltimore: The Johns Hopkins University Press.

Squire, L. and H. van der Tak. 1975. Economic Analysis of Projects. Baltimore: The
Johns Hopkins University Press.

Swalm, R. 1966. “Utility Theory – Insights into Decision-Taking.” Harvard Business


Review.

UK-DFID. 1997. Eliminating World Poverty: A Challenge for the 21st Century. White
Paper.

UK-DFID. 2000. Yunnan Environmental Development Project (Project Memorandum).


London.

UK Overseas Development Agency. 1992 (various editions 1998-1995). Project Appraisal


in Developing Countries: A Guide for Economists.

UK Treasury Taskforce – Private Finance. 2000. How to Construct a Public Sector


Comparator. Technical Note Number 5.

World Bank. 2001. Political Economy and Political Risks of Institutional Reforms in the
Water Sector (draft). Rural Development Department.

World Bank. various. Staff Appraisal Reports (www.worldbank.org).