Operations Evaluation

Country Assistance
Evaluation at OED:
Methodology and Challenges

World Bank Operations Evaluation Department

Presentation at 2nd Meeting of DAC Network on Development Evaluation
Paris, November 9-10, 2004
 Operations Roadmap
Operations Evaluation
Outline of Presentation

1. Background
2. Country Assistance Evaluation
Methodology
3. Rethinking the Approach
4. Relevant Developments in Country
Evaluations at the World Bank

2
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation
Background

Since FY95, OED has undertaken 70

Country Assistance Evaluations, covering
nearly three quarters of the Bank’s
operational program.
Roughly two-thirds of Bank country
assistance programs have been rated
satisfactory.
The usefulness of country evaluations
illustrated by disconnect with aggregate
portfolio performance.

3
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation Satisfactory Project Outcomes do not
automatically translate into Satisfactory
Assistance Outcomes…

CAE Outcome Ratings and Project Outcome Ratings

Country Portfolio Outcomes of Bank Assistance Programs

Outcomes
Satisfactory Unsatisfactory

Satisfactory 53% 33%

Unsatisfactory 7% 7%

4
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation

OED Methodology

5
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation The first step is to clarify the
object of evaluation…

 Country development performance;

 Assistance program performance; and

 Donor (World Bank) performance

6
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation
Methodology

Country Assistance Programs are assessed

across three dimensions:

 Development Impact

 Products and Services

 Attribution

7
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation Dimension I: Products and Services
(Bottom-up)

 Loans, Credits, and Grants

 Analytical and Advisory Services

 Aid Coordination and Resource Mobilization

8
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation Dimension II: Development Impact
(Top-Down)
 Main Country-level Program Objectives are derived
from:
 The country assistance strategy agreed with
the Client.
 The evaluator’s ex-post assessment of fundamental
development constraints.
 Major Ratings Same as in Project Evaluations
 Relevance
 Efficacy
 Efficiency
 Outcome
 Institutional Development
 Sustainability
9
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation
Dimension III: Attribution

 Bank Performance

 Borrower Performance

 Partner Performance

 Exogenous Factors

10
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation Attributing Program Results Is a Key
Challenge

 Individual donor attribution not possible

using our current counterfactuals.

 Bank performance currently measured

against our best practice corporate
standards.

11
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation
Bank Performance

 Professional Services
 Strategy consistent with key development
constraints, and corporate priorities
 Quality at entry
 Implementation and follow-up

 Selectivity
 Participation and partnership
 Creativity, initiative, and efficiency

12
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation
Client performance

 Ownership of assistance program

 Support for national and international

development priorities (MDGs, national
development plan, etc.)

 Respect for safeguards

13
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation
Aid Partner Performance

 Impact on design of assistance program

 Impact on implementation of assistance

program

14
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation
Exogenous Factors

 World economic shocks

 Events of nature

 War/civil disturbances

 Other

15
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation Ongoing Review of Methodology…Staff
and Board Views

Usage Surveys of Board and Bank Staff revealed:

High awareness of CAEs and CAE findings
Incorporation of recommendations
Frequent mention in strategies and Board
discussions
But,
 Staff views negative overall…timing,
accountability, consistency
 Board views more positive, but need to focus
more on results and less on compliance

16
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation
….External Evaluator’s view

Methodology reviewed by independent,

external reviewer:
The approach may place too much
emphasis on instruments rather than impact
Different sections of CAE appear to lead to
different conclusions, ie project outcomes
Focus shifts from country to bank to country
—may be confusing
Need to ensure consistency of terminology

17
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation
…Other Agencies Methods

OED made a brief examination of methods

employed by other agencies:
OED has completed the most country
evaluations by far
Only about half the agencies have formal,
codified methodologies
OED is only agency to rate; very few
attempt to construct counterfactuals or
attribute

18
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation Contemplated Changes to the
Methodology…
Based on our ongoing self-assessment, we are
considering:
Making the CAEs more-results based, by organizing
by objectives
 Establishing what the program attempted to
accomplish by objective
 Analyzing the product dimension by objective
rather than overall
 Preparing sub-ratings by objectives,
disaggregating the overall rating
Changing attribution to the notion of “plausible
association” or Bank contribution.

19
Operations Evaluation Department
 Operations Roadmap
Operations Evaluation Other Relevant Developments in
the Bank…
More institution-wide attention to evaluating the
impact of country programs
Mainstreaming of Results-based CASs, which will:
 Contain a CAS Completion Report (self-evaluation), to be
validated by OED
 More clearly set out what the country program intended to
achieve
The Quality Assurance Group (internal
assessments) is also moving to higher-level
assessments, such as country-wide ESW or lending
reviews.

20
Operations Evaluation Department