Program Evaluation

- Undergone changes in conceptual and functional level

- Designed to assess efficient allocation of resources
- Grown into a discipline concerned with all-around development of a program
Cost-Benefit Analysis
- Earliest methods of program evaluation
- Analyzes economic viability of programs by comparing total benefits and total costs
- Can be used for prioritization of programs based on net worth
- Estimates net present value of a program by comparing benefits with associated costs
o Notion of benefits and costs depend on evaluation perspective
o All costs and benefits for the community/society should be taken into account
 Sponsor Perspective
 Difficult to calculate
 Public Sector Projects
 Conducted from a social perspective
- All costs and benefits for the community/society should be taken into account
o Benefits and costs can be direct or indirect
 Example: Irrigation Project – direct costs: capital & operational costs of the project &
compensation costs for loss of land due to construction/inundation, direct benefits:
regeneration of degraded lands, enhanced crop yields, etc, indirect benefits: employment
generation for irrigation project & improvement of quality life
- Requires determination of length of program (e.g. number of years which benefits and costs are to be
evaluated)
- Monetary values are assigned to all benefits and costs
- Program is feasible when net present value is positive
o Used as criterion for allocating funds among competing programs
- Benefit-cost ratios as criterion for allocating funds among competing programs
o Ratio of present value of benefits to present value of costs
- For mutually exclusive programs, implementation of one program precludes the possibility of others =
comparison of programs based on benefit-cost ratios can be misleading
o Net present value should be used
- Social cost-benefit analysis
o Important aspect: choice of appropriate discount rate
 Reflects opportunity cost of capital
- For public programs, interest rate on government borrowing can be used for discounting future benefits and
costs
o If public investment displaces private investment, interest rate will not reflect
o Appropriate discount rate = social opportunity cost rate
 the return on private sector investment that has been displaced by the public project
o Alternative = social time preference rate
 represents society’s preference between current and future consumption
 required additional future consumption that is necessary to compensate for the loss of one
unit of present consumption
- There is no authoritative way of choosing discount rate but outcome depends on choice of discount rate
 o Lower discount rate = future costs and benefits
o Higher discount rate = short-term benefits and costs
o Discount rate increases = net present value of project decreases
- Sensitivity analysis is done to check results
o repeating the same cost-benefit analysis for different discount rates
o Alternately, Internal Rate of Return (IRR) can be reported
 discount rate at which the net present value of a program is zero
 it gives the decision makers a measure of risk associated with the project
 not a criterion for program selection
- Important aspect: estimation and valuation of social benefits and costs
o involves the estimation of incremental benefits and incremental costs that can be assigned solely
to the project
o construct the counterfactual = what would happen or would have happened without the project
o Incremental values can then be calculated by comparing the benefits and costs with and without
the project
 In ex post analysis, the problem is constructing the counterfactual
 For ex ante analysis, the problem is not only constructing the counterfactual but also
forecasting the expected benefits and costs of the project
- Can be useful for evaluation when it is relatively easy to assign monetary values to benefits and costs
- Problems arise when cost-benefit analysis is extended to evaluation of public sector programs
o E.g. Construction of dams for irrigation purposes
 Has negative spillover effects
 CBA fails to take into account spillover effects
o E.g. Programs involved are health campaigns and family planning
 assigning monetary values to benefits and costs of some programs can be very
complicated and controversial
- Possible Remedy: separate CBA for different subgroups of the population
o facilitate a better understanding of the distributional effects of the program

Cost-Effectiveness Analysis
- alternative of CBA
- in programs where it is difficult to assign monetary value cost-effective analysis (CEA) can help the decision
makers in making judicious allocation of resources
- has been used extensively for evaluations of health, educational, and environmental programs
- basic purpose: to identify programs that can attain specific objectives at the lowest cost
- involves identification of the alternatives, an estimation of their costs, and an assessment of their
effectiveness
- Cost Estimation
o takes into account both the direct and indirect costs of the intervention
o Direct Costs
 costs of inputs, like personnel, equipment, materials, utilities
o Indirect Costs
 externalities or ‘spillover’ effects associated with the intervention and should be included
in the cost calculation
o Volunteers and Donated Resources
 omitted in normal cost accounting
  use opportunity cost = market prices reflect opportunity cost
 If not available, use proxies
- Cost-effectiveness is measured using
o Cost-effectiveness Ratio
 ratio of the cost of a program to the effects produced
 For independent programs, ratio is calculated for each alternative and ranked in
ascending order = project with the lowest cost-effectiveness ratio is the most cost efficient
and gets the highest priority
o Incremental Cost-effectiveness Ratio (ICER)
 ratio of the difference in costs to the difference in effects between two alternative
programs
 For mutually exclusive programs, programs are ranked based on their effectiveness and
principle of strong dominance is applied, wherein program that is less expensive and more
effective is considered for implementation (based on the availability of funds or the
society’s willingness to pay)
- CEA a very useful tool for program appraisal, planning, and implementation
- Problems of CEA
o Similar to CBA
o There should be a discount rate because costs and effects can change through the years
o Choice of discount rate that may affect the outcome of the analysis
o Qualitative differences of the program that will be added as costs to deficient program
o Uncertainty about the future
- Cost-effectiveness acceptability curve (CEAC) is one such measure that indicates the probability that a
program is cost-effective in comparison to the alternative for a given value of the maximum socially
acceptable cost-effectiveness ratio
o Important to carry out sensitivity analysis

Social Marginal Cost of Funds Analysis

- to evaluate public expenditure programs, the social benefits from the program should be compared with the
social costs of the program and at the margin, marginal social benefits (MSB) should equal marginal social
costs (MSC)
- Marginal Cost (MSC) = Marginal Cost of Funds (MCF)
o Lost of consumer and producer surplus per dollar of additional tax revenue raised to finance public
goods
- Depending on tax instrument, additional financing = implications on government revenues
- Special considerations in measuring MCF: nontax distortions
o Monopoly
 Degree of tax shifting
o Externalities
 Taxing goods with positive externalities
 Taxing negative-externality-producing goods instead of non-externality-producing goods
Data Envelopment Analysis
- non-parametric, linear programming-based approach to performance evaluation within organizational units
- a technique to determine relative efficiency of homogeneous units (schools, hospitals, banks, etc.)
- originally a tool for better evaluation and management of public sector activities
 - organization unit is efficient if it is not possible to increase any output without increasing any input and
without decreasing other output
o efficient units = pareto optimal
- efficient organizational units that are the best-practice units and they form the extreme points or the
efficiency frontier
- Efficient Composite Units a.k.a. virtual producers and best-practice units
o constitute the entire efficient frontier and receive a relative efficiency score of one
- Central problem: to identify the best-practicing unit with which it is to be compared
o Can be formulated as linear programming problem
o Solution: on the weights chosen, evaluate unit that gives maximum relative efficiency score
- Can identify sources and level of inefficiency for each inputs and outputs
- Can assign targets to maximize output
- Can be used for identification of improvement priorities
- Main advantage: non-parametric nature of analysis and do not require specification of any functional form
o Focuses on each evaluation unit and how it compares to the reference unit rather than entire
population average
o Avoids usual valuation problems associated with CBA and CEA
o Can handle multiple inputs and outputs
o Do not require inputs and outputs to have the same unit of measurement
o Can incorporate exogenous factors in evaluation
 Done by treating variables as inputs or outputs
o Can handle categorical variables
- can be used for identification of inefficient units, reallocation of resources for efficient utilization, and setting
targets for inefficient units to improve performance
- Main problem: relies on authenticity of the data, uses extreme point method to estimate efficiency frontier
o Error in data creates significant problems

Multiple-Criteria Evaluation (MCE) / Multiple-Objectives Evaluation (MOE)

- an all-round assessment of a program in terms of its relevance, efficacy, efficiency, sustainability, and its
contribution towards development of institutions necessary for design, delivery, and implementation of
successful programs in the future
- used by World Bank
- Stages of a program: conceptualization, design, implementation, and outcome
o MOE can be used a any stages
 Can be categorized as formative or summative
- Formative evaluation: plays a critical role in overall development, assists in the development of a program
o Has two components:
 Diagnostic: caried out at planning and design stage
 Process: concerned with monitoring of program activities and operations
- Summative evaluation: includes both outcome evaluation and impact assessment
o Outcome evaluation: studies the usefulness of the program in delivering the objectives
o Impact assessment: evaluates program on its objectives and indirect and unintended effects of the
program
- Diagnostic Evaluation
o A program conceived at a political, community, or administrative level contains in it a broad
definition of the problem and a list of goals that are equally broad in nature.
 o Main purpose: to identify and define the problem
- Process Evaluation
o Main objective: to provide a systematic assessment of program performance to facilitate progress
towards achievement of program objectives
o an evaluation strategy that tries to ensure that the activities and outputs contribute to the
achievement of program objectives.
o an integral part of a result-based monitoring system.
o provides crucial information on program coverage, quality, and propriety of interventions and
effectiveness of the interventions in producing the desired effects.
o constitutes a feedback mechanism that promotes informed decision making, better decision
making, improved performance, and greater accountability.
- Outcome Evaluation
o study the effectiveness of programs in delivering the program objectives, which can be both short
term and long term.
o Can be used for both formative and summative evaluations
o Basic idea: to study the net effect of an intervention on participants or beneficiaries in terms of the
outcome variables
 refers to the changes in the outcome measures that can be attributed solely to the
intervention.

References
Shah, A. (2020). Policy, Program and Project Evaluatiom. Cham Switzerland: Springer Nature Switzerland
AG.