Procurement Vol2 Appendix

The World Bank and Public Procurement—An
Independent Evaluation
Appendixes to Volume II: Achieving Development

Effectiveness through Bank Procurement
Contents
ABBREVIATIONS ..................................................................................................................................................................... I
APPENDIX A. BANK GUIDELINES AND PROCESSES ........................................................................................................ 3
APPENDIX B. CDD PROCUREMENT ................................................................................................................................... 31
PPP PROCUREMENT: COMPARISON OF THE BANK AND OTHER ORGANIZATIONS.................................................. 55
APPENDIX C. PROCUREMENT RISK CONCEPTS AND THRESHOLDS ........................................................................... 61
APPENDIX D. PROCUREMENT TRACKING SYSTEMS, TIME ANALYSIS, AND AFRICA REGION REVIEW ................. 89
BIBLIOGRAPHY .................................................................................................................................................................. 110
ENDNOTES .......................................................................................................................................................................... 119
ii
Abbreviations
ADB Asian Development Bank ISR Implementation status report
AfDB African Development Bank MAPS Methodology for Assessing
APEC Asian Pacific Economic Cooperation Procurement Systems
CDD Community-driven development MDB Multilateral development bank
CPAR Country procurement assessment MEAT Most economically advantageous
report tender
CPIA Country policy and institutional MRRD Ministry of Rural Rehabilitation and
assessment Development
DfID Department for International NCB National competitive bidding
Development NEPE State Technical Unit
FMIS Financial management information OPRC Operational Procurement Review
system Committee
HDI United Nations Human Development OPCS Operations Policy and Country
Index Services
IAD Internal Audit Vice Presidency PAD Project appraisal document
IAEA The International Atomic Energy PPP Public-private partnership
Agency P-RAMS Procurement risk assessment
ICB International competitive bidding management system.
process PROCYS Procurement Cycle Tracking System—
ICR Implementation completion report Africa Region
ICRR Implementation completion and RPM Regional procurement managers
results reports SEPA Procurement plan execution system
ICT Information technology SME Small and medium-sized enterprises
IDA International Development Agency TOR Terms of reference
IDB Inter-American Development Bank UNCITRAL United Nations Commission on
IEG Independent Evaluation Group International Trade Law
IFC International Finance Corporation WBI World Bank Institute
INT Integrity Vice-Presidency WTO World Trade Organization
ISDB Islamic Development Bank
Appendix A. Bank Guidelines and Processes
Overall Perceptions
Table A.1. Bank Procurement Methods—Adequacy of Competition

Bank
Country proc Bank Country Private Civil
Question Avg mgmt staff TTLs clients sector society
Do the Bank's Procurement procedures ICB,
NCB, and Consultants Selection Procedures
in particular) lead to adequate competition in 2.9 2.9 3.2 2.8 3.3 2.9 2.5
terms of numbers of bidders, openness and
fairness, geographical reach, etc.?
Source: IEG questionnaire.
Note: 4 = more than adequate; 3 = adequate; 2 = only somewhat adequate; 1 = inadequate; TTL = task team leader.
Table A.2. Bank Procurement—Support to Integrity, Transparency, Cost, Quality, and Timelines
Bank
To what extent do the Bank's procurement
requirements positively support the integrity,
transparency, cost, quality, and timeliness of 3.1 2.6 3.6 3.0 3.5 3.0 2.8
delivering public sector projects?

Table A.3. Bank Procurement—Capturing Bidder Interest

Bank
Are bidders s more likely to show interest if
the project is under Bank procurement policy
2.8 3.6 3.0 3.2 3.2 3.2
and procedures, as opposed to a project 3.2
under national guidelines and processes?
APPENDIX A
BANK GUIDELINES AND PROCESSES
Table A.4. Bank Procurement—Emphasis on Price Factors
Question Bank
proc Bank Country Private
Avg staff TTLs clients sector
To what extent does the Bank's oversight (no-objection) of
prequalification/ short-listing, bidding documents/RFP, 2.9 2.9 2.7 2.9 3.0
bid/proposal evaluation, and contract award emphasize price?
Table A.5. Bank Procurement—Flexibility in Response

Bank
proc Bank Country
Question Avg staff TTLs clients
Is the Bank able to respond quickly to special circumstances such as those
requiring deviations from policies and procedures, or changes to
2.3 2.7 2.1 2.2
procurement provisions of the financing agreement, or changes to previously
agreed arrangements?
Table A.6. Bank Procurement—Incidence of Delay in Contract Award

Bank
Ctry Proc Bank Ctry Private Civil
Question Avg mgmt Staff TTLs clients sector society
How frequent is the incidence of delay in the award
2.9 3.2 2.5 2.9 2.7 2.9 3.2
of contracts?
Methodology for Assessing Procurement Systems Indicators and Scores
Table A.7. MAPS Indicators and Passing Scores Under the Piloting Program
UCS required
score
Pillar I: Legislative and Regulatory Framework
Indicator 1: Public procurement legislative and regulatory framework achieves the agreed standards
and complies with applicable obligations.
Sub-indicator 1(a): Scope of application and coverage of the legislative and regulatory framework. 3
Sub-indicator 1(b): Procurement Methods 2+
Sub-indicator 1(c): Advertising rules and time limits 3
Sub-indicator 1(d): Rules on Participation 3
Sub-indicator 1(e): Tender documentation and technical specifications 3
Sub-indicator 1(f): Tender evaluation and award criteria 3
Sub-indicator 1(g): Submission, receipt and opening of tenders 3
4
UCS required
score
Sub-indicator 1(h): Complaints 3
Indicator 2: Existence of Implementing Regulations and Documentation.
Sub-indicator 2(a): Implementing regulation (sic) that provide defined processes and procedures not included 2
in higher-level legislation
Sub-indicator 2(b): Model tender documents for goods, works, and services 2
Sub-indicator 2(c): Procedures for pre-qualification 2+
Sub-indicator 2(d): Procedures suitable for contracting for services or other requirements in which technical 2+
capacity is a key criterion.
Sub-indicator 2(e): User's guide or manual for contracting entities 2
Sub-indicator 2(f): General Conditions of Contracts (GCC) for public sector contracts covering goods, works 3
and services consistent with national requirements and, when applicable, international requirements
Pillar II: Institutional Framework and Management Capacity
Indicator 3: The public procurement system is mainstreamed and well integrated into the public
sector governance system.
Sub-indicator 3(a): Procurement planning and associated expenditures are part of the budget formulation 2
process and contribute to multiyear planning.
Sub-indicator 3(b): Budget law and financial procedures support timely procurement, contract execution, and 2
payment.
Sub-indicator 3(c): No initiation of procurement actions without existing budget appropriations. 2
Sub-indicator 3(d): Systematic completion reports are prepared for certification of budget execution and for 2
reconciliation of delivery with budget programming.
Indicator 4: The country has a functional normative/regulatory body.
Sub-indicator 4(a): The status and basis for the normative/regulatory body is covered in the legislative and 3
regulatory framework.
Sub-indicator 4(b): The body has a defined set of responsibilities including the following… 2
Sub-indicator 4(c): The body's organization, funding, staffing, and level of independence and authority (formal 2
power) to exercise its duties should be sufficient and consistent with the responsibilities.
Sub-indicator 4(d): The responsibilities should also provide for separation and clarity so as to avoid conflict of 3
interest and direct involvement in the execution of procurement transactions.
Indicator 5: Existence of institutional development capacity.
Sub-indicator 5(a): The country has a system for collection and disseminating procurement information, 2
including tender invitations, requests for proposals, and contract award information.
Sub-indicator 5(b): The country has systems and procedures for collecting and monitoring national 2
procurement statistics.
Sub-indicator 5(c): A sustainable strategy and training capacity exists to provide training advice and 2
assistance to develop the capacity of government and private sector participation to understand the rules and
regulations and how they should be implemented.
Sub-indicator 5(d): Quality control standards are disseminated and used to evaluate staff performance and 2
address capacity development issues.
5
APPENDIX A
UCS required
score
Pillar III: Procurement Operations and Market Practices
Indicator 6: The country's procurement operations and practices are efficient.
Sub-indicator 6(a): The level of procurement competence among government officials within the entity is 2
consistent with their procurement responsibilities.
Sub-indicator 6(b): The procurement training and information programs for government officials and for 2
private sector participants are consistent with demand.
Sub-indicator 6(c): There are established norms for the safekeeping of records and documents related to 2+
transactions and contract management.
Sub-indicator 6(d): There are provisions for delegating authority to others who have the capacity to exercise 2
responsibilities.
Indicator 7: Functionality of the public procurement market
Subindicator 7(a): There are effective mechanisms for partnerships between the public and private sector. 2
Sub-indicator 7(b): Private sector institutions are well organized and able to facilitate access to the market. 2
Sub-indicator 7(c): There are no major systemic constraints (e.g. inadequate access to credit, contracting 2+
practices, etc.) inhibiting the private sector's capacity to access the procurement market.
Sub-indicator 7(d) – Clarity and transparency of rules for determining whether to engage international or 2
national markets. *Only analyzed for 3 countries.
Indicator 8: Existence of contract administration and dispute resolution provisions
Sub-indicator 8(a): Procedures are clearly defined for undertaking contract administration responsibilities that 2
include inspection and acceptance procedures, quality control procedures, and methods to review and issue
contract amendments in a timely manner.
Sub-indicator 8(b): Contracts include dispute resolution procedures that provide for an efficient and fair 3
process to resolve disputes arising during the performance of the contract.
Sub-indicator 8(c): Procedures exist to enforce the outcome of the dispute resolution process. 3
Pillar IV: Integrity and Transparency of the Public Procurement System
Indicator 9: The country has effective control and audit systems.
Sub-indicator 9(a): A legal framework, organization, policy, and procedures for internal and external control 2+
and audit of public procurement operations are in place to provide a functioning control framework.
Sub-indicator 9(b): Enforcement and follow-up on findings and recommendations of the control framework 2+
provide an environment that fosters compliance.
Sub-indicator 9(c): The internal control system provides timely information on compliance to enable 2+
management action.
Sub-indicator 9(d): The internal control systems are sufficiently defined to allow performance audits to be 2
conducted.
Sub-indicator 9(e): Auditors are sufficiently informed about procurement requirements and control systems to 2
conduct quality audits that contribute to compliance.
Indicator 10: Efficiency of appeals mechanism.
Sub-indicator 10(a): Decisions are deliberated on the basis of available information, and the final decision can 3
be reviewed and rules upon by a body (or authority) with enforcement capacity under the law.
Sub-indicator 10(b): The complaint review system has the capacity to handle complaints efficiently and a 3
means to enforce the remedy imposed.
Sub-indicator 10(c): The system operates in a fair manner, with outcomes of decisions balanced and justified 3
on the basis of available information.
Sub-indicator 10(d): Decisions are published and made available to all interested parties and to the public. 2
Sub-indicator 10(e): The system ensures that the complaint review body has full authority and independence 3
for resolution of complaints.
Indicator 11: Degree of access to information
Sub-indicator 11(a): Information is published and distributed through available media with support from 2+
information technology when feasible.
Indicator 12: The country has ethics and anticorruption measures in place.
Sub-indicator 12(a): The legal and regulatory framework for procurement, including tender and contract 3
documents, includes provisions addressing corruption, fraud, conflict of interest, and unethical behavior and
sets out (either directly or by reference to other laws) the actions that can be taken with regard to such
behavior.
Sub-indicator 12(b): The legal system defines responsibilities, accountabilities, and penalties for individuals 3
and firms found to have engaged in fraudulent or corrupt practices.
Sub-indicator 12(c): Evidence of enforcement of rulings and penalties exists. 2+
6
UCS required
score
Sub-indicator 12(d): Special measures exist to prevent and detect fraud and corruption in public procurement. 3
Sub-indicator 12(e): Stakeholders (private sector, civil society, and ultimate beneficiaries of procurement/end- 2
users) support the creation of a procurement market known for its integrity and ethical behaviors.
Sub-indicator 12(f): The country should have in place a secure mechanism for reporting fraudulent, corrupt, or 3
unethical behavior.
Sub-indicator 12(g): Existence of Codes of Conduct/Codes of Ethics for participants that are involved in 2
aspects of the public financial management systems that also provide for disclosure for those in decision
making positions.
7
National Competitive Bidding Requirements and Compliance
Table A.8. Bank NCB Requirements and Country Systems

AZ BD ET ID MA PE PH SN
Reports Countries Field Visit Field Visit Field Visit Field Visit/UCS Field Field Field Visit/UCS Field
Reviewed Reviewed Visit Visit Visit/UCS
/UCS
2012
2002 2005 2008 2005 2010 2008 2001 2004 2005 2012 2011 2004 2009 2012
35 26 CPAR 2009 LA
CPAR LA LA LA LA LA CPAR LA LA LA LA LA LA CPAR
Draft
Bank Requirements for NCB
There may be no regional / domestic preferences regarding the
18 15 1 1 1 1 1 1 1 1 1 1
sources of labor and material.1
Prospective bidders must be allowed at least 30 days for bid
19 14 1 1 1 1 1 1 1 1 1
preparation.2
Appropriate standard bidding and prequalification documents
15 14 1 1 1 1 1
must be used.
Eligibility cannot be restricted based on nationality of bidder
17 13 1 1 1 1 1 1 1 1
and/or origin of goods.3
Award must be made to the lowest evaluated qualified and
15 13 1 1 1 1 1 1
responsive bidder.
Minimum requirements must be explicitly stated in the
12 12 1 1 1 1
documents.
Bidding opportunities must be advertised in the local press. 11 11 1 1 1 1
Bidders are not generally required to register with a local or
13 11 1 1 1 1 1 1 1
federal authority except under certain conditions.4
The procurement process cannot be cancelled, all bids rejected,
13 11 1 1 1 1 1 1 1 1
and/or rebidding conducted without approval.
Public bid opening is required. 11 9 1 1 1 1 1 1
Bid evaluation criteria other than price may be allowed only if
8 8 1
quantified in monetary terms.
Price negotiations may not be conducted with “winning” bidders
10 8 1 1 1 1
prior to contract signature.5
Parastatals may only be allowed to bid under certain conditions.6 9 7 1 1 1
Bids may not be rejected based only on a comparison with the
procuring entity's estimate; invitations to bid shall not establish 11 7 1 1 1 1 1 1 1
minimums and maximums.7
Audits and inspection of records related to bid submission and
8 7 1
performance of the supplier are permitted.
APPENDIX A
/UCS
2012
2002 2005 2008 2005 2010 2008 2001 2004 2005 2012 2011 2004 2009 2012
35 26 CPAR 2009 LA
Draft
“Two envelope” bid opening procedure is permitted for
procurement of goods or works under specific conditions, 6 6 1 1 1
notably if domestic law precludes use of one envelope.8
Foreign firms' eligibility cannot be conditioned on joint ventures
7 5 1 1 1 1
with local firms.
Qualification criteria shall be applied on a pass/fail basis. 5 5 1
Extension of the time period to prepare bids may only be
5 5
allowed under exceptional circumstances.
An inflation clause is recommended for contracts over a year.9 7 5 1 1 1
Bidding documents are freely available.10 7 4 1 1 1 1 1 1
Bid security shall be in the form of a letter of credit or bank
5 4 1 1 1 1
guarantee from a reputable bank.11
Joint venture partners must be jointly and severally liable. 6 3 1 1 1 1 1
There may be no restrictions on the means of delivery of bids.12 3 3
Pre-qualification should be used only for large works projects. 5 3 1 1 1 1
Award must be published. 4 3 1 1
Bidding documents and contract shall include provisions on
3 3
sanctions for fraud and corruption.
No preference may be given to suppliers or contractors based
on region or locality of registration, small size, ethnic ownership, 4 2 1 1 1 1
etc.
For contracts subject to prior review under the Loan Agreement,
scope/conditions may not be modified during implementation 6 2 1 1 1 1 1 1
without Bank approval.
Bidders shall not be eliminated on the basis of minor deviations. 4 2 1 1 1
Contractors/suppliers must be prequalified for large or
2 1 1 1
specialized contracts.
An invitation to prequalify must be advertised for each
1 1 1
procurement involving large or complex potential contracts.
There must be no set limitations to the number of firms who can
1 1
bid for a contract.
Requests for clarification of bidding documents must be made in
1 1
writing, and response should be sent to all prospective bidders.
9
APPENDIX A
/UCS
2012
2002 2005 2008 2005 2010 2008 2001 2004 2005 2012 2011 2004 2009 2012
35 26 CPAR 2009 LA
Draft
The experience requirement for works contracts shall be (i) at
least one previous contract at 80% of cost of contract being
2 1 1 1
procured and (b) an average annual turnover of 100% of
contract being procured.
Bidders shall be given at least twenty-eight (28) days from the
receipt of notification of 1 1
award to submit performance securities.
Bids shall not be disclosed excepted as needed for evaluation
without bidder's written authorization until the award of the 1 1
contract.
Automatic rebidding may be required if too few bids are
4 1 1 1 1
received.13
10
APPENDIX A
TZ TR BT BR BF MK MU PA PL GH CO LA VN MZ SL AL BA EG
Reports Countries Field Field UCS UCS UCS UCS UCS UCS UCS UCS Desk Desk Desk Desk Desk Desk Other Other
Reviewe Reviewed Visit Visit/ Sampl Sample Sample Sample Sample Sample
d UCS e
2003
2009 2008 2012 2010 2011 2012 2009 2007 2007 2009 2011 2010 2002 2010 2012 2012 2002 2002
35 26 CPAR
LA LA LA LA LA LA LA LA LA LA LA LA CPAR LA CPAR LA CPAR CPAR
excerpt
Bank Requirements for NCB
There may be no regional / domestic preferences regarding the sources
18 15 1 1 1 1 1 1 1 1
of labor and material.1
Prospective bidders must be allowed at least 30 days for bid
19 14 1 1 1 1 1 1 1 1 1 1
preparation.2
Appropriate standard bidding and prequalification documents must be
15 14 1 1 1 1 1 1 1 1 1 1
used.
Eligibility cannot be restricted based on nationality of bidder and/or origin
17 13 1 1 1 1 1 1 1 1 1
of goods.3
Award must be made to the lowest evaluated qualified and responsive
15 13 1 1 1 1 1 1 1 1 1
bidder.
Minimum requirements must be explicitly stated in the documents. 12 12 1 1 1 1 1 1 1 1
Bidding opportunities must be advertised in the local press. 11 11 1 1 1 1 1 1 1
Bidders are not generally required to register with a local or federal
13 11 1 1 1 1 1 1
authority except under certain conditions.4
The procurement process cannot be cancelled, all bids rejected, and/or
13 11 1 1 1 1 1
rebidding conducted without approval.
Public bid opening is required. 11 9 1 1 1 1 1
Bid evaluation criteria other than price may be allowed only if quantified
8 8 1 1 1 1 1 1 1
in monetary terms.
Price negotiations may not be conducted with “winning” bidders prior to
10 8 1 1 1 1 1 1
contract signature.5
Parastatals may only be allowed to bid under certain conditions.6 9 7 1 1 1 1 1 1
Bids may not be rejected based only on a comparison with the procuring
entity's estimate; invitations to bid shall not establish minimums and 11 7 1 1 1 1
maximums.7
Audits and inspection of records related to bid submission and
8 7 1 1 1 1 1 1 1
performance of the supplier are permitted.
“Two envelope” bid opening procedure is permitted for procurement of
goods or works under specific conditions, notably if domestic law 6 6 1 1 1
precludes use of one envelope.8
Foreign firms' eligibility cannot be conditioned on join ventures with local
7 5 1 1 1
firms.
Qualification criteria shall be applied on a pass/fail basis. 5 5 1 1 1 1
Extension of the time period to prepare bids may only be allowed under
5 5 1 1 1 1 1
exceptional circumstances.
An inflation clause is recommended for contracts over a year.9 7 5 1 1 1 1
Bidding documents are freely available.10 7 4 1
11
APPENDIX A
TZ TR BT BR BF MK MU PA PL GH CO LA VN MZ SL AL BA EG
Reports Countries Field Field UCS UCS UCS UCS UCS UCS UCS UCS Desk Desk Desk Desk Desk Desk Other Other
Reviewe Reviewed Visit Visit/ Sampl Sample Sample Sample Sample Sample
d UCS e
2003
2009 2008 2012 2010 2011 2012 2009 2007 2007 2009 2011 2010 2002 2010 2012 2012 2002 2002
35 26 CPAR
LA LA LA LA LA LA LA LA LA LA LA LA CPAR LA CPAR LA CPAR CPAR
excerpt
Bid security shall be in the form of a letter of credit or bank guarantee
5 4 1
from a reputable bank.11
Joint venture partners must be jointly and severally liable. 6 3 1
There may be no restrictions on the means of delivery of bids.12 3 3 1 1 1
Pre-qualification should be used only for large works projects. 5 3 1
Award must be published. 4 3 1 1
Bidding documents and contract shall include provisions on sanctions for
3 3 1 1 1
fraud and corruption.
No preference may be given to suppliers or contractors based on region
4 2
or locality of registration, small size, ethnic ownership, etc.
For contracts subject to prior review under the Loan Agreement,
scope/conditions may not be modified during implementation without 6 2
Bank approval.
Bidders shall not be eliminated on the basis of minor deviations. 4 2 1
Contractors/suppliers must be prequalified for large or specialized
2 1
contracts.
An invitation to prequalify must be advertised for each procurement
1 1
involving large or complex potential contracts.
There must be no set limitations to the number of firms who can bid for a
1 1 1
contract.
Requests for clarification of bidding documents must be made in writing,
1 1 1
and response should be sent to all prospective bidders.
The experience requirement for works contracts shall be (i) at least one
previous contract at 80% of cost of contract being procured and (b) an 2 1
average annual turnover of 100% of contract being procured.
Bidders shall be given at least twenty-eight (28) days from the receipt of
notification of 1 1 1
award to submit performance securities.
Bids shall not be disclosed excepted as needed for evaluation without
1 1 1
bidder's written authorization until the award of the contract.
Automatic rebidding may be required if too few bids are received.13 4 1 1
Source: World Bank; data compiled from CPARs and Loan Agreements.
1 With the exception of unskilled labor, if available locally.
2 Except for commodities/small goods contracts.
3 Primary boycotts, in which the government maintains a policy of not purchasing from a particular country, are permitted, however.
4 Registration may be required if (i) registration criteria, process and cost are reasonable/efficient and (ii) qualified foreign firms are not precluded from competing. Notwithstanding, the Bank stated
that a registration requirement should be discouraged.
5 There is one exception: where the bid price is substantially above market or budget levels and then only if negotiations are carried out to try to reach a satisfactory contract through reduction in
scope and/or reallocation of risk and responsibility which can be reflected in a reduction in Contract Price. (See Guidelines para 2.60)
12
APPENDIX A
6 Participation by parastatals is permitted providing they (i) are financially autonomous, (ii) operate under commercial law, and (iii) are independent from borrower and its purchasing/contracting
authority.
7 Except where the bid prices exceed the available budget. The practices or rejection of bids outside a range or “bracket” of bid values is also known as "price bracketing."
8 Two envelope procedure is also permitted where there are adequate safeguards against retaining second envelope unopened are incorporated in the two envelope procedures and effective bid
protest mechanisms are already in place for the due processing of bid complaints. (All technical envelopes are opened first and, after review, price envelopes of all or only qualified/responsive bids
are opened in the second round.)
9 Recommended for works contracts of one year or more in duration when domestic inflation rate is high.
10 In some cases the Bank has specified there shall be no fee; in other countries fees have been permitted.
11 It may sometimes also be a certified check. In Macedonia, bid security was to "follow the generally acceptable practice used in the local market."
12 A method of delivery may be specified in the case where bidders have to submit physical sample, however.
13 Provided that (1) all responsible bidders are allowed to bid, (2) the process is efficient and (3) no serious delays result.
13
Stage II Equivalence Analysis
Table A.9. Stage II Equivalence Analysis—Average Scores

Average score
World Bank Guidelines Para. Policy Requirements Piloting Program Requirements (out of 3)
Eligibility (paragraphs 1.6-1.8) Conditions for participation to be limited Those provisions of the Guidelines 1.7
to those that are essential to fulfil the would be made applicable through
contract requirements. the Legal Agreement
Exceptions indicated in para 1.8. (LA).
Two-stage Bidding (paragraph Applies to large and complex, or Bank review of complex, high value, 2.4
2.6) turnkey, contracts that potentially attract and non-standardized bidding
the participation of foreign bidders. processes as part of its assessment
Turnkey contract of the country systems.
(paragraph 2.22) Procedure may follow two stages: 1)
non-priced technical proposals; 2) Possibility to use Bank SBDs or
issuance of BDs to bidders, and harmonized MPDs for specific bids /
submission of technical priced contracts to be identified in the
proposals. Procurement Plan for the pilot pro-
ject.
Modifications as may be needed by

the country institutional and legal
framework to allow for the country
policies and procedures to apply.
Notification and Advertisement General Procurement Notice and The executing agency of each pilot 2.4
(paragraphs 2.7-2.8) Specific Procurement Notices for ICB project will be required to publish in
contracts to be published on UN UN Development Business, and on
Development Business online and Dg an electronic free access portal, all
market. contracts identified as likely to attract
foreign competition. Such contracts
to be flagged in the Procurement
Plan. Publication of all mandatory
documents at the pilot project level
on a fully functional and freely
accessible website of the executing
agency.
Examination, evaluation and Bids to be evaluated and awarded on To be part of the bidding documents 2.5
comparison of bids; Post- the basis of the criteria stated in the accepted for use under pilot projects
qualification (paragraphs 2.48- bidding documents and quantified using when international competition is
2.54 and 2.58) cost to the maximum extent. Post- anticipated (see checklist below).
qualification applies when there is no
pre-qualification.
Alternative Conditions of Contract documents must clearly define General Conditions of Contracts 2.6
Contract (paragraphs 2.37 and the procedure to submit alternative bids, (GCC) for contracts covering goods,
2.38) and the scope of work, rights and works and services to be consistent
obligations of the parties, and provide with international requirements when
for fair and clear allocation of risk. international competition is sought.
To be part of the bidding documents
accepted for use under pilot projects
when international competition is
anticipated (see checklist below).
Domestic preference for goods Defines the Bank's procedures for use Use of discriminatory domestic 2.6
(paragraphs 2.55-2.56) of domestic preference. preferences (e.g. by nationality) will
not be allowed in the bidding
documents accepted for use under
pilot projects when international
APPENDIX A
competition is anticipated (see
checklist below).
Bid validity, Award of Contract Award to the lowest evaluated bidder Assessment of legal framework to 2.6
which meets the requirements of the ensure that the national law covers
bidding documents including technical bid validity, contract award, and
capacity and financial resources. rejection of bids requirements, and
evaluation of compliance at the pilot
project executing agency level, and
monitoring during pilot project
implementation.
Bidding Documents Bidding documents must provide Existence of national sample bidding 2.7
(paragraphs 2.11-2.12, and information sufficient for a prospective documents not required, but
2.16-2.18) bidder to prepare a bid. Use of Bank mandatory use of standard bidding
standard bidding documents is documents acceptable to the Bank
mandatory for ICB - this requirement will when international competition is
have to be modified for the pilots. sought under pilot projects. Specific
provisions to be validated vis-à-vis
the checklist below, and to be stated
in the LA.
Publication (paragraphs 2.57, Publication of awards is required within The executing agency for each pilot 2.7
2.59-2.60, and 2.61-2.64) two weeks. Rejection of all bids to be project would be required to have (a)
justified. a functioning electronic system for
retaining information on its
procurement processes and
managing invoices and certificates
payments; and (b) a fully functional
website that is freely accessible to all
stakeholders, including bidders and
civil society, for posting information
regarding its organization,
procurement regulations, SBDs,
procurement opportunities,
information related to the award of
contracts, and procurement plans.
Pricing (paragraphs 2.21 and Bid must be invited on the basis of Mandatory use of INCOTERMS for 2.7
2.23) specified INCOTERMS comparison purposes, or fully
equivalent provisions.
Prequalification(paragraphs Use of prequalification for large complex The Procurement Plan will be used to 2.8
2.9-2.10) requirements limited to capability and identify bidding processes that would
resources to perform the particular benefit from prequalification.
contract with regard to experience, past Advertisement as per requirements
performance, personnel, facilities and above. Prequalification requirements
financial position. to be assessed by comparison with
the harmonized prequalification MPD.
Joint ventures(paragraph 1.10) Mandatory association between firms is Those provisions of the Guidelines 2.8
not acceptable. would be made applicable through
the LA.
Validity of bids and bid security Bidding documents must state bid Review of clauses in the bidding 2.8
(paragraphs 2.13-2.14) validity period. Bid security not documents accepted for use under
mandatory pilot projects when international
competition is anticipated.
Language (paragraph 2.15) English, French, or Spanish, in addition National language can be used 2.8
to the national language of the country. provided that it is used internationally
and that a translation in English,
French or Spanish is made available
to foreign bidders for contracts that
have been identified for international
bidding in the Procurement Plan.
Advertisement and bidding
15
APPENDIX A
documents should be issued in an
international language (English,
French or Spanish) in addition to the
national language, and contracts with
foreign entities should be signed in
the international language the
bidders used for their bids.
Price Adjustment (paragraphs Bidding documents must specify if price Requirement for price adjustment as 2.8
2.24-2.25) adjustment applies and price adjustment a contractual term to be included in
must be based on a prescribed formula. the bidding documents accepted for
use under pilot projects when
international competition is
anticipated.
Currency and Payments Bids documents must state currency National currency acceptable 2.8
(paragraphs 2.28-2.36) provisions and the conversion provided it is freely and fully
mechanism. Payment provisions must convertible and country does not
also be stated in the bidding documents. apply restriction or control on foreign
exchange. Contract terms to allow
payment in foreign currency directly
and/or as percentage of contract
price at predefined exchange rates
(in no more than 3 currencies)
expressed in the bidding documents
accepted for use under pilot projects
when international competition is
anticipated.
Performance Security, Format and provisions to be included in (to be assessed as part of the 2.8
Liquidated Damages Force the bidding documents. assessment of the country legal and
Majeure (paragraphs institutional framework).
2.39/2.40, 2.41 and 2.42)
Applicable law and settlement The conditions of contract shall provide National law to apply. 2.8
of disputes (paragraph 2.43) for settlement of disputes and state the
applicable law.
Bid opening procedures Bids to be opened in public at the time Assessment of legal framework to 2.8
(paragraph 2.45) and place designated in the bidding ensure that the national law covers
documents. Rejection of late bids. this requirement, and evaluation of
compliance at the pilot project
executing agency level, and
monitoring during pilot project
implementation.
Rejection of bids 2.8
Time for bid preparation Time allowed for the preparation and Requirement to be stipulated for each 2.9
(paragraph 2.44) submission of bids shall be determined executing agency implementing pilot
on the basis of the magnitude and projects. To be closely monitored as
complexity of the requirement. one key compliance indicator (see
Stage III).
Confidentiality (paragraph After bid opening, evaluation and National law to apply. 2.9
2.47) recommendations and other information
shall not be disclosed until after
publication of the award.
Source: IEG analysis of UCS pilot data.
Note: On a scale of 0–3.
16
Applying the Concept of Value for Money
EXAMPLES FROM SIX PUBLIC PROCUREMENT JURISDICTIONS
The United Kingdom, the European Union, and Australia have taken a long-term
and systematic approach to integrate value for money concepts into their public
procurement practices. 1 Additionally, the Asia Pacific Economic Cooperation Forum
(APEC) has endorsed value for money concepts since 1999. The International
Atomic Energy Agency is one United Nations agency that has adopted value for
money principles.
The United Kingdom
Guidance toward the approach taken by the United Kingdom has been provided for
the most part by the U.K. Office of Government Commerce and the National Audit
Office. 2 Their documents emphasize the following:
Focusing on cost reduction, notably both the transaction overhead costs of

conducting the procurement, and the cost of the goods, works, or services
acquired. Getting better prices suggests negotiating better deals (price/quality),
and when possible aggregating demand to capture scale economies.
Improving management of the project, the contract, and the acquired assets.
Balancing competition with risk. Competition is encouraged to obtain value, but
risks need to be taken into account and where competition is weak
benchmarking and theoretical cost models are suggested to verify that the
purchasing agent is obtaining favorable offers.
Taking the whole length of the contract into account (including
decommissioning).
Accounting for wider societal and environmental costs of the contract.
Developing bid evaluation criteria that are relevant to the project—published by
the purchaser and widely available.
Recommending the use, by purchasing agents, of electronic commerce
technology and new procurement practices.
The U.K. Department for International Development (DfID) has also adopted a
value for money approach (DfID 2011). It concurs with much of the above,
emphasizing the importance of having a sound procurement strategy, investing up
front in well-defined bid criteria and good terms of reference, and negotiating
favorable contract terms. It adds that purchasing agents should develop
relationships with suppliers. This last point is a feature of discussions in the United
Kingdom over the need for long-term collaboration with suppliers, to save costs and
17
ensure continuity of supply and consistency of quality (albeit sometimes at the
expense of small and medium-sized enterprises (SMEs) and of competition more
generally).
The United States—“Best Value” Procurement
Traditionally, best value has been used in the United States to describe the trade-off
process, and since 1997 the term has come to encompass a variety of techniques that
can be used in conducting competitive procurement. 3 Today, the Federal Acquisition
Regulation describes the best value continuum as any one or a combination of
selection approaches that are used to obtain best value or value for money. There are
two types of procurement techniques for obtaining best value:
The tradeoff process is a competitive negotiation process where the purchaser

conducts a tradeoff analysis of both price and non-price factors, and awards the
contract to the bidder proposing the combination of factors that offers the best
overall value. The contract may be award to other than the lowest priced, or the
highest technically rated bid.
The lowest price technically acceptable process describes a distinct variation on the
competitive negotiation process, where the non-price factors are stated criteria of
bid responsiveness. The contract is awarded to the bidder offering the technically
acceptable proposal with the lowest price, with “technically acceptable” defined
as complying with the technical criteria, including any non-price factors.
The lowest price technically acceptable process is more closely aligned with standard
bidding techniques, where value criteria are stated in advance, as is typical of Bank
procedures. Best value is defined when planning the procurement—the non-price
factors are evaluated for conformance with the technical criteria expressed in the
solicitation, typically without gradations of scores for higher levels of achievement.
In contrast, under the tradeoff process, best value is decided during evaluation,
through the trade-off analysis of the different bids.
Overall, best value procedures differ fundamentally from traditional sealed bidding
procedures in which bidders offer the same or similar service or product in response
to detailed solicitations, and bids differ only on price. In best value procurement,
bids differ in elements beyond price. The purchaser may issue an award to a higher
priced but technically superior offer or to a lower priced but less technically
qualified proposal, depending on its view of which proposal presents the best
overall value. Best value procurement has its critics, whose primary concern is the
perception of fairness and the role of a procurement system in encouraging
inclusiveness and transparency in government contracting. Concerns center on three
issues:
18
The extent and consequence of discretion in ascribing and evaluating best value
criteria, and in communicating with competitors in the conduct of negotiated
procurement.
Past performance is arguably the most significant noncost/nontechnical criteria
in American best value procurement. Concerns are that it prejudices unjustly
firms that do not have a past history of government contracting and can
ultimately become little more than a tool for incumbency.
There is a general concern that non-stated evaluative criteria appear to be
subjective, and may preclude open and fair competition. The Federal Acquisition
Regulation note that the procuring agency can obtain best value by various
means, but explicitly provide only a few alternate considerations to price. If a
requirement is not well defined and there is a performance risk to consider, the
agency may properly rely more heavily on technical or past performance
considerations. However, there are open questions as to whether the procuring
agency might adopt evaluative factors for administrative ease.
The flexibilities and innovations of best value procurement continue to attract
advocates and detractors alike. Debate continues over the loosening of regulations
regarding communication, the incorporation of less-traditional considerations, and
the devolution of discretion entailed. The choice to place best value criteria above
traditional price/responsiveness metrics requires a developed and supporting legal
framework related to ethics, fraud and corruption, influence peddling, and similar
issues. Where problems arise, mechanisms must be in place to address them.
The European Union
Value for money principles are incorporated in the European Union’s Most
Economically Advantageous Tender (MEAT) system, which enables the contracting
authority to use criteria that reflect qualitative, technical and sustainable aspects of a
tender, in addition to price, when reaching an award decision. 4
Application of MEAT is a choice of the procuring agency—according to the

directive, public contracts in the European Union must be awarded either solely on
the basis of the lowest price, or under MEAT criteria. 5 But where MEAT is used, it
may involve the use of various criteria such as quality, technical merit, aesthetic and
functional characteristics, environmental characteristics, running costs, cost-
effectiveness, after-sales service and technical assistance, delivery date and delivery
period, in addition to price. Additional features of MEAT include:
The contract notice must indicate whether MEAT criteria will be used to award
the contract and what the criteria are.
19
The bid assessment and award criteria should not confer unrestricted freedom of
choice on the contracting authority and should be made public.
Bidding and contract award criteria should ensure effective competition.
Whereas MEAT procedures offer procuring agents considerable latitude to define
requirements, the directive maintains principles of open and competitive tendering,
and fairness through clear specifications and objective assessment. The directive
states:
….In order to guarantee equal treatment [of bidders], the criteria for the award of the
contract should enable tenders to be compared and assessed objectively. If these
conditions are fulfilled, economic and qualitative criteria for the award of the
contract, such as meeting environmental requirements, may enable the contracting
authority to meet the needs of the public concerned, as expressed in the
specifications of the contract. Under the same conditions, a contracting authority
may use criteria aiming to meet social requirements, in response in particular to the
needs—defined in the specifications of the contract—of particularly disadvantaged
groups of people to which those receiving/using the works, supplies or services
which are the object of the contract belong.
As of 2012, thresholds for application of the European Union directive for central
government and subcentral contracting authorities were €5,000,000 for Works
(including concession contracts); €130,000 or €200,000 for services, depending on the
type of service; and €130,000 (€200,000, for certain defense contracts) for supply.
Australia
Australia has made value for money the core principle underpinning Australian
government procurement. 6 Its procurement guidelines elaborate value for money
concepts as:
• Life-cycle analysis of all relevant (direct and indirect) costs and benefits on
a fair and common basis.
• Nondiscriminatory and competitive procurement processes based on
bidding requirements and evaluation criteria that are logical, clear,
comprehensive, and relevant.
• Efficient, effective, and ethical use of resources.
• Accountable and transparent decision making.
The guidelines clearly state that cost is not the only determining factor in assessing
value for money. In addition to life-cycle analysis, value for money assessment
could include fitness for purpose, supplier performance history, and relative risk of
competing proposals.
20
The Australian guidelines specifically address business development of SMEs
through public procurement, saying that that government is committed to sourcing
at least 10 percent of procurement to Australian and New Zealand firms with fewer
than 200 employees. This is accomplished, first, by ensuring fair treatment
(nondiscrimination) of SMEs, and second, by considering in the procurement
process the value for money benefits of doing business with SMEs. This falls short of
requiring an explicit preference for SMEs, but implies that requirements may be
tailored to fit SME capabilities, and that evaluations may consider benefits of
awarding contracts to SMEs.
Asia-Pacific Economic Cooperation Forum
In 1999, APEC’s Government Procurement Experts Group identified the principles

of public procurement as: “transparency, value for money, open and effective
competition, fair dealing, accountability and due process, and non-discrimination.” 7
Value for money is integral to the APEC procurement principles, which say:
…The test of the best available value for money is a comparison of relevant benefits
and costs on a whole of life basis…The lowest priced compliant offer does not
necessarily represent best value for money.
Other aspects of APEC’s value for money approach include:
• The procurement function can deliver value for money, in taxpayer terms,
through improvements in procurement processes and management.
• Objective and functional requirements, which focus on what is to be
achieved rather than how it is to be done, can encourage innovative
solutions that may improve the value for money outcome.
• Costs and benefits are to be assessed on a life-cycle basis, possibly
including sensitivity analysis and calculations of discounted cash flows.
• Besides price and fitness-for-purpose, other factors that may be taken into
account include performance, quality, reliability, delivery, inventory costs,
running costs, warranties and after-sale support, and disposal.
• Competition is encouraged. Procurement methods should be chosen
based on the best value for money outcome, especially in situations of
limited competition.
• Purchasing agents should be knowledgeable about the market in which
they operate, assessing risks and avoiding unnecessary costs.
• It is important to establish competence, viability, and capability of
prospective suppliers.
21
• Where allowed, negotiation should be used to improve value for money.
With respect to value for money, APEC covers many of the same areas as the United
Kingdom and European Union. APEC does not make specific reference to
environmental and social factors, but these are inferred as justifiable considerations.
Its principles address ideas of negotiation, and alternative methods to address non-
competitive markets, but are vague on details and on how the principles should be
operationalized.
The International Atomic Energy Agency
The International Atomic Energy Agency (IAEA), a United Nations agency, offers a
different perspective from the above organizations. While the approaches described
above, like those of the Bank, are meant to provide guidance to others, IAEA
principles and procedures are developed to guide the Agency in its own
procurement activities. Accordingly, it is able to prescribe a system that fits with its
internal technical and management capacity.
Value for money is central to IAEA principles and processes, as stated in its
regulations (IAEA 2011). The introduction of these concepts is the result of a
concerted effort to change the Agency’s procurement from a legalistic or rules based
approach, to an outcomes or strategic management approach (Tonkin 2012). IAEA
emphasizes several areas where value for money can be achieved, through:
• Integration of procurement strategies and planning with program

delivery.
• Identification of risks (market risks, or continuity of supply risks, for
example) and development of mitigation strategies within the
procurement strategy and plans.
• Exploiting innovations that improve program delivery, through
strengthened agency expertise, and by tapping supplier expertise.
• Optimizing cost over the life of the acquisition, trading off quality and
cost factors where appropriate.
• Managing implementation to ensure that what was contracted is
delivered, and to exploit opportunities to enhance value during
implementation, especially in longer-term service contracts (by taking
advantage of new technologies, for example).
An interesting feature of IAEA’s practices is to separate high value, complex
procurement from low-value straightforward purchasing. The former is assigned to
procurement specialist teams, whereas the latter is considered low risk and is
assigned to contracting officers. This lowers the administrative overhead cost of
22
managing the procurement process. The IAEA also provides more latitude to
procuring agents than other organizations. Although, like others, it sees open and
competitive tendering as a value for money principle, it provides a range of selection
methods other than lowest price (or lowest life cycle cost). These include best
technical proposal (similar to the Bank’s fixed budget selection method for
consulting services), and competitive negotiation options (similar to UNCITRAL
procedure).
BROADER CONCEPTS OF VALUE FOR MONEY
Non-Price Factors and Preferences
The Australian provisions to use public procurement to strategically develop its

SME sector are one example. The Independent Evaluation Group (IEG) found
support for preferences in its 11-country survey, and notes that it is a common
feature of many national procurement systems. Apart from the Bank’s domestic
preference policy, seen by IEG as ineffective, the Bank does not allow preferences as
it contradicts the principles of fairness and competition. To the extent to which the
Bank wishes to retain the principle and practice of giving domestic preferences,
discussed earlier in the present chapter, IEG recommends that the Bank explore
other mechanisms to achieve this, in view of its present limited framework.
Investing in procurement planning and developing clear and appropriate evaluation criteria
The importance of strong front-end work is emphasized by the United Kingdom and
IAEA. The objectives appear to be twofold: (i) to assure transparency, and (ii) to best
align the stated requirements of the procurement with the true needs of the project
(fit for purpose). The Bank Guidelines mandate procurement planning, seen to be an
integral part of project design. IEG notes that although the Bank has traditionally
supported up-front procurement planning, such support has been found to be below
optimal, especially in complex projects. Management proposes to make stronger use
of procurement planning as a tool in the future. 8
Reducing Cost through Competition and Negotiation
Competition is generally accepted, within value for money principles, as an effective

way to lower cost of acquisition. However, value for money also envisages instances
of poor competition, and suggests that good negotiation can contribute to better
outcomes in terms of cost and quality. 9 Bank policy and practice is very much
oriented toward competition. Negotiation to improve outcomes, post contract
award, is limited to acquiring professional services, following consultant
recruitment procedures. Yet it is unlikely that the Bank could recommend
23
negotiation in all but restricted circumstances, as it would run counter to other
principles of transparency and fairness.
The Bank offers only limited solutions to situations of poor competition and other
aspects of addressing market risk. Direct negotiation is an option, but it is an
exception and little guidance is offered on how best to conduct the process or to
determine if good value is achieved. 10 Limited competition, such as is found in
information and communications technology (ICT) procurement, or sole sourcing
(to incumbent public-private partnership [PPP] concessions, for example) are also
not well accommodated in Bank practice. It is suggested that the Bank explore
greater use of less-competitive methods, such as competitive negotiation as
provided for by UNCITRAL and European Union, and as reported here is practiced
in IAEA. 11
Strengthening Relationships with Suppliers
Building “appropriate” relationships with suppliers is a feature of the U.K. value for
money principles, although admitting that it limits competition and could be unfair
to some classes of firms. In a similar vein, IAEA principles recommend that
purchasing agents understand supplier capabilities and practices, and use
prequalification and staged bidding to better assess bidder capability. Bank policies
and practice encourage prequalification and multistage bidding, and would not
discourage procuring agents from knowing their markets. However, Bank practice
suggests assurance of an “arms-length” relationship between purchaser and bidder.
Many Bank procedures are written to ensure limited interaction (detailed
procedures of communications during bidding, for example), partly to avoid the
possibility of collusion and influence. Maintaining its fraud and corruption
mitigation agenda could be difficult if the Bank were to go in this direction.
There is a concern that the traditional focus of public procurement on regulations

and procedures discourages the exploitation of value-adding procurement methods
thereby reducing the potential for innovation. Proposed revisions to the European
Union Procurement Directive recommends introduction of the “innovative
partnership” procedure (see the next session). This new method of procurement
would allow procuring agents to enter into partnership arrangements with suppliers
to develop, and subsequently supply new and innovative products, works and
services according to specified requirements that are performance oriented. The
developmental benefits of such supplier relationships are obvious, and mechanisms
to support such arrangements within the Bank’s principles of fair and open
competition, are recommended.
24
Reducing Transaction Costs
The United Kingdom and IAEA both note that reducing the administrative
overhead cost of procurement offers value for money benefits. IAEA has established
lower cost processes for low-value, low-risk procurement. The Bank’s shopping
procedure, might be seen as a similar approach. Besides, as noted in previous
chapters, Bank practices in terms of more “proportionality” of resource use, relative
to the objective of procurement, could be further improved through a series of
measures; improved guidelines and processes for complex procurement (Chapter 2);
more tolerance of risk, and the use of risk-based rather than value thresholds, frugal
use of higher level clearance (Chapter 3); and more attention to elapsed time in
contract processing (Chapter 4). All these measures would strengthen the
application of value for money principles.
Supporting Contract Implementation
IAEA and the United Kingdom cite attention to managing contracts, as an important
value for money factor—ensuring that what was contracted is delivered, is as
important as the selection process. Furthermore, IAEA notes opportunities to adjust
contracts during implementation, to reap gains of new technology for example,
especially important for long-term service contracts. Regarding the capturing of
value for money benefits through adjustment to contracts during implementation,
this is to some extent covered in the Bank’s works contracts (closely modeled on the
International Federation of Consulting Engineers), which give authority to the
engineer to improve the implementation of works through variations. IEG has also
noted the need to strengthen long-term service contracts, as these especially would
almost certainly require ongoing renegotiation to obtain optimum outputs.
IEG discusses the broader issue of contract implementation earlier in this chapter,
and has pointed towards client requests for support. Greater attention in Bank
practices to oversee and assist contract implementation is needed, though balance
will have to be achieved between the role of different players within the Bank
(especially task team leaders and procurement staff), as well as outside the Bank, for
example, through national audit offices, country based citizens’ groups and the like.
To summarize, the principle of value for money in procurement is being

increasingly incorporated into the procurement approach of a number of public
jurisdictions, with varying degrees of operational instruction and varying degrees of
emphasis in terms of the aspects of value for money that they each seek to
emphasize. Many aspects of these principles are already implicit in Bank
procurement, and some are frequently used. Stronger direction could be given to
staff and borrowers to fortify the adoption of such practices. A number of the
25
efficiency increasing (and hence transaction cost reducing) measures proposed in the
present report would expand value for money, as would better management of risk,
and the embracing of new procurement practices in frontier areas of Bank business.
In such areas, more attention to specific procurement modalities would be required
to achieve value for money. However, areas remain, where for reasons of open
access, or transparency, the Bank may prefer to maintain its present systems and
practices.
Bank requirements do not accommodate the inclusion of social objectives through

procurement (using procurement to create employment, perhaps targeting youth or
minorities, for example). The difficulty for the Bank is to balance objectives of
domestic laws and policies that target social and domestic industrial strategies, with
its obligations to ensure fair treatment of international bidders.
26
Proposal to Update the European Union Public Procurement Directive
A proposal to update the current European Union procurement directive
(2004/18/EC) was issued on 20 December 2011; the “Proposal for a Directive of the
European Parliament and of the Council on Public Procurement.” 12 The extent to
which the proposal will be implemented, or the timing is not known, although there
is some expectation that reforms maybe introduced later this year. 13
CAUTIOUS APPROACH
The proposal is cautious in its ambitions. It says that it endeavors to retain

continuity with the current directive, and not to depart from concepts that have been
developed over the years, have evolved through case law, and are well known to
practitioners. This is in line with the Bank approach, which will retain the
Guidelines and keep much of the existing system intact.
The proposal notes obligations of member states to comply with World Trade
Organization (WTO) government procurement agreement requirements, although
done so with respect to WTO’s proposed lighter regime for contracting authorities
below the central government level. Thus it offers simplified procedures, yet invokes
alignment with international best practice. This is something for the Bank to
consider in light of its move towards country systems—use of country systems can
simplify things for borrowers, but continued compliance with international norms is
important.
TOOL BOX
A “toolbox” approach is proposed, but in fact does not change much. The options of
open and restrictive tendering are retained, as are the negotiation procedures
(competitive negotiation and competitive dialog—although with simplified
procedures and broadened application beyond “complex” procurement). The one
new procedure that is introduced is the “innovative partnership” procedure (e-
procurement procedures are also emphasized as noted below). This allows
procuring agents to enter into partnership arrangements with suppliers to develop,
and subsequently supply, new and innovative products, works, and services
according to specified requirements that are performance oriented.
The toolbox approach is similar to the direction the Bank is suggesting – to

transform guidelines and standard bidding documents from mandatory, to optional
guidance documents, for example. The Bank does not use the negotiation
procedures of the European Union (and others), although IEG recommends this. If
adopted, the new “innovative partnership” is perhaps something worth exploring
27
by the Bank, especially for borrowers who would follow European Union directives.
The objective is to use public procurement strategically to improve innovation (and
competitiveness). If properly managed, for example by using open and competitive
bidding, it offers developmental benefits and could be done without violating Bank
principles.
EMPHASIS ON E-PROCUREMENT
Several measures are proposed to promote and mandate e-procurement:
• Mandatory electronic notices

• Mandatory electronic documents
• Switch to electronic communications, including electronic bid
submissions, within two years
• Acceptance of dynamic purchasing systems, electronic auctions, electronic
catalogues.
This goes beyond the Bank’s current position on e-procurement, or what the Bank
proposes in terms of promoting greater use. The Bank allows all of these features,
but does not mandate them--given the Bank’s diverse client base, it is unlikely that it
could (with some exceptions, such as the current mandatory publication of notices
on the United Nations Development Business website, for example).
MODERNIZATION/SIMPLIFICATION
In addition to e-procurement, further suggestions to modernize, streamline,

simplify, include:
• Shortened time limits for bidding

• Exclusion of firms from bidding for nonperformance
• Provisions to modify contracts during implementation.
These are relatively minor items, but the Bank might consider similar measures. The
Bank’s mandatory bidding periods were criticized by clients in IEG’s country case
studies, especially for international competitive bidding (ICB). With modern
communications and more of the bidding process going on-line, shortened periods
could be considered where it can be demonstrated not to compromise competition.
Modification of contracts, especially long-term service contracts, has been noted
earlier in this chapter.
28
VALUE FOR MONEY
As reported earlier in this appendix, value for money is a feature of the existing
European Union directive. It is proposed to expand this in several respects:
• More explicit instruction to apply lif- cycle costing, noting that external
costs are to be considered (such as environmental costs) where they can be
monetized and verified. A common European Union methodology for
life-cycle costing is to be used, when developed.
• Factors directly linked to production processes may be included in
technical specifications and technical award criteria.
• Labels may be used in technical specification related to environmental
factors (eco label) and other social factors (child labor, for example).
• Firms may be sanctioned for violating social, labor and environmental
laws.
Value for money is a key element of the Bank’s proposed procurement reform, and
as commented, the Bank’s system already incorporates many of the measures noted
above (life-cycle costing, environmental specifications, and so forth). But the Bank
should also consider developing methodologies around these approaches.
SUPPORT TO SMES
To address its objective of supporting local industry, the Bank could also consider
the European Union suggestions to support SMEs. Several suggestions are made to
promote SMEs: removing barriers to SMEs (reducing turn-over requirements, for
example), simplifying bidding requirements (for example, allowing self-declarations
as prima-facie evidence), “packaging” procurement to make packages more
attractive to SMEs, and direct payment to subcontractors.
KNOWLEDGE CENTERS
The European Union proposal observes that there is a lack of capacity in many
contracting authorities, and that this limits effective procurement. The proposal
obliges member states to provide support structures offering legal and economic
advice, guidance, training and assistance in preparing and conducting procurement
procedures. The Bank has similarly identified borrower capacity building as
important. It might similarly consider the establishment of a knowledge center that
borrowers can tap into on an as needs basis.
29
Appendix B. CDD Procurement
Summary of Previous Reports

Commensurate with the increasing importance of community-driven development
(CDD) projects in the Bank’s portfolio, procurement under CDD projects has been
the subject of many reviews by the Networks and by IEG, and they yield mixed
evidence. An early review, with positive findings, was undertaken by the central
Social Protection Team unit. 14 It found that contracting by community groups
through local shopping was “more efficient and transparent” and got lower prices
than central government bidding. The review also noted several “good practice”
procedures, including project designs that reflect skills and needs of communities,
use of very simple contracting procedures, building-in incentives for efficient
procurement, provision of active support to communities on prices and quality to
help them in negotiating with suppliers, and ensuring transparency of procurement
through proactive information sharing and disclosures. 15
A later field-based review by the South Asia Region of 84 CDD subprojects in India
also concluded, positively, that better value for money was achieved in works
implemented by communities without involving contractors, with savings in costs
and time in most of the analyzed cases. 16 More interestingly from the procurement
perspective, the review found that “compared to the conventional procurement
cycle, the subprojects rated very highly in terms of procurement planning and
scheduling, developing specifications, awarding decisions, and disclosure and
contract management while scoring poorly in terms of market search for bidders,
tendering, tender opening and evaluation.” The review concluded that success could
be ascribed to the practice of “relationship-based procurement” wherein the
communities established a mutually beneficial and accountable commercial
relationship between the purchaser and the supplier. The report concluded that
there was scope for further adaptation of the CDD procurement processes with
focus on delivery of outcomes and results rather than on oversight and control of
contracting for inputs.
Yet other reviews by IEG, the Internal Audit Vice Presidency (IAD), and the
Integrity Vice Presidency (INT) have flagged persistent concerns about the adequacy
of fiduciary oversight of CDD projects. A comprehensive review of the CDD
portfolio by IEG in 2005 noted that almost a quarter of surveyed staff expressed
concerns about their ability to adequately monitor compliance with agreed fiduciary
31
APPENDIX B
CDD PROCUREMENT
processes in CDD projects (IEG 2005). In a similar vein, a 2008 IAD audit reported
that the Bank did not conduct required procurement post-reviews for five out of 11
projects that required such reviews--impeding the Bank’s ability to detect contracts
that did not comply with the agreed procurement procedures. And an INT
investigation of two CDD projects in Kenya in 2011 was significantly critical in its
findings: it pointed towards large scale misappropriations and leakages caused by
failure to comply with agreed procedures during implementation (Al-Azar 2011;
World Bank 2011a).
Responding to INT’s concerns, Operations Policy and Country Services (OPCS)

undertook a major review in 2012 of governance issues related to CDD projects
(World Bank 2011a). Its review concluded that a well-designed CDD project can
strengthen governance at the local level, since it incorporates the three key
principles of good governance: transparency, accountability and participation.
However, complementary and interdependent pieces have to come together to make
CDD projects work as intended. Fraud and corruption in the procurement process
(during contract implementation), for example, short-changing of project quality
because of collusive practices, were seen as important risks. The review
recommended special attention to risk assessment, especially the risks of “political
capture” during project design. It also argued for field-based, intensive,
collaborative supervision by the Bank, front-loaded during the first two years. The
review concluded that with adequate care in design and supervision, CDD projects
can be successful even in cases where governance is poor and corruption is rampant.
32
Portfolio Analysis
CDD broadly defined is an approach that gives control over planning decisions and
investment resources to community groups and local governments. CDD programs
operate on the principles of local empowerment, participatory governance, demand-
responsiveness, administrative autonomy, greater downward accountability, and
enhanced local capacity. The CDD Anchor in the Social Development Department
has been tracking the amount of Bank (IBRD/IDA) lending going toward CDD
approaches since 2000 using the following CDD typology developed by a cross-
regional/cross-sectoral group of CDD practitioners.
This is the universe of projects for the evaluation period. This review uses the Social
Development group definition of CDD and used their database of CDD projects. A
few CDDs (3 percent of total) are embedded in development policy operations. Since
the review will assess the functioning of World Bank procurement systems in its
operations, we will focus on Investment Lending, hence on CDDs delivered in that
form. The universe of CDDs is then 692 projects. It should be noted that projects that
have some CDD components are also included in the CDD database, per the Social
Development group definition and classification.
The social development database includes 732 projects classified as CDDs, 711 (97
percent) are IBRD/IDA financed and the rest are financed by Global Environment
Facility and other trust funds that do not consistently record implementation status
report (ISR) ratings. Only IBRD/IDA financed operations are included in this
analysis. As a first attempt to assess the effects of World Bank procurement systems
on the development effectiveness of CDD projects, we will use two sets of data
stemming from ISRs and from IEG project evaluations (Implementation Completion
Report Reviews—ICRRs).
From ISRs we will extract procurement ratings and outcome ratings. For simplicity
we will use ratings from the last ISR, although ratings from all ISRs are also
available and could be used to track changes in projects' procurement performance.
It should be noted that ISRs are only done for main projects and not for subsequent
additional financing projects. For this reason, the number of CDD projects with ISR
ratings is smaller, as seen in Table B.1. In tables where ISR ratings are shown, CDD
total will be 549 projects because of the existence of a few unrated projects.
ICRRs do not have procurement ratings, but outcome ratings and risk to
development outcome ratings can be extracted, either to confirm ISR outcome
assessment and its sustainability. Only a subset of the CDDs reviews are closed (47
percent, see Table B.2) and even fewer have been reviewed by IEG as of today (21
33
APPENDIX B
CDD PROCUREMENT
percent). For this reason the tables with IEG ratings will show a CDD total of 143
projects. RDO totals are lower because this rating was introduced in the middle of
our evaluation period (around FY08).
Table B.1. World Bank Project Approvals (all projects—CDD and non-CDD), FY02-11
IBRD/IDA Trust funds Total
IL Dev Pol IBRD/IDA and others
Main Additional Total Operation total
project financing
CDD 552 140 692 19 711 21 732
Non-CDD 1,492 313 1,805 563 2,368 1,256 3,624
Total 2,044 453 2,497 582 3,079 1,277 4,356
Source: IEG analysis and World Bank data.
Note: As of November 8, 2012.
Table B.2. World Bank Project Approvals (investment lending projects), FY02-11
CDD Non- Total
Main Project Additional Total CDD CDD
financing
Active 265 58 323 1052 1,375
Closed 287 82 369 753 1,122
Closed o/w have IEG 153 - 153 348 501
Evaluation
Total 552 140 692 1,805 2,497
Note: As of November 8, 2012. Includes investment lending projects financed by IBRD/IDA and excludes those financed by Trust Funds
and others.
34
APPENDIX B
CDD PROCUREMENT
Table B.3. IEG Outcome Ratings and Risk to Development Outcome Ratings for Investment
Lending Projects, by Fiscal Year
Outcome ratings Risk to development outcome ratings
Approval FY MS+ S+ Total % Sat % Sat Moderate or Total % Moderate or
(MS+) (S+) Lower lower
CDD
FY02 38 21 50 76 42 26 44 59
FY03 32 7 48 67 15 20 47 43
FY04 21 14 33 64 42 12 33 36
FY05 9 2 15 60 13 11 15 73
FY06 3 2 4 75 50 3 4 75
FY07 1 1 3 33 33 2 3 67
FY08-11
Total CDD 104 47 153 68 31 74 146 51
Non-CDD
FY02 72 38 97 74 39 46 79 58
FY03 65 30 83 78 36 56 81 69
FY04 49 19 69 71 28 48 67 72
FY05 36 15 55 65 27 32 55 58
FY06 19 8 26 73 31 14 26 54
FY07 8 2 12 67 17 5 12 42
FY08 2 2 4 50 50 3 4 75
FY09 1 2 50 0 1 2 50
FY10-11
Total Non- 252 114 348 72 33 205 326 63
CDD
CDD and Non-CDD
FY02 110 59 147 75 40 72 123 59
FY03 97 37 131 74 28 76 128 59
FY04 70 33 102 69 32 60 100 60
FY05 45 17 70 64 24 43 70 61
FY06 22 10 30 73 33 17 30 57
FY07 9 3 15 60 20 7 15 47
FY08 2 2 4 50 50 3 4 75
FY09 1 0 2 50 0 1 2 50
FY10-11
Total 356 161 501 71 32 279 472 59
Note: As of November 8, 2012. Totals refer only to those projects with available or valid ratings.
35
APPENDIX B
CDD PROCUREMENT
Table B.4. ISR Procurement Ratings for Investment Lending Projects, by Fiscal Year
Approval FY HS S MS MU U HU Total % Sat % Sat
(MS+) (S+)
CDD
FY02 48 12 3 3 66 91 73
FY03 44 19 5 4 72 88 61
FY04 1 45 18 4 1 69 93 67
FY05 29 27 6 1 63 89 46
FY06 33 22 4 59 93 56
FY07 1 22 22 7 52 87 44
FY08 20 27 4 51 92 39
FY09 1 21 17 9 2 50 78 44
FY10 12 20 5 37 86 32
FY11 16 12 2 30 93 53
Total 3 290 196 49 11 0 549 89 53
Non-CDD
FY02 1 80 35 4 4 1 125 93 65
FY03 3 75 34 8 3 1 124 90 63
FY04 2 88 31 16 4 141 86 64
FY05 1 80 67 6 4 1 159 93 51
FY06 1 93 62 10 2 168 93 56
FY07 81 49 10 5 1 146 89 55
FY08 74 63 16 4 157 87 47
FY09 61 61 8 3 133 92 46
FY10 87 69 11 2 169 92 51
FY11 2 117 53 13 3 188 91 63
Total 10 836 524 102 34 4 1510 91 56
Note: As of November 8, 2012. Totals refer only to those projects with available or valid ratings. Latest ISR ratings were used. Additional
financing projects don't have ISR ratings. HS = highly satisfactory; S = satisfactory; MS = moderately satisfactory; MU = moderately
unsatisfactory; U = unsatisfactory; HU = highly unsatisfactory.
36
APPENDIX B
CDD PROCUREMENT
Table B.5. ISR Procurement, ISR Outcome Ratings, and IEG Outcome Ratings
ISR Procurement IEG outcome ratings ISR outcome ratings
Ratings MS+ MU- Total MS+ MU- Total
CDD
MS+ 447 36 483 100 36 136
MU- 33 27 60 4 13 17
Total 480 63 543 104 49 153
% of Total
MS+ 93% 7% 100% 74% 26% 100%
MU- 55% 45% 100% 24% 76% 100%
Total 88% 12% 100% 68% 32% 100%
Non-CDD
MS+ 1,193 145 1,338 240 71 311
MU- 53 84 137 10 24 34
Total 1,246 229 1,475 250 95 345
% of Total
MS+ 89% 11% 100% 77% 23% 100%
MU- 39% 61% 100% 29% 71% 100%
Total 84% 16% 100% 72% 28% 100%
CDD & Non-CDD
MS+ 1,640 181 1,821 340 107 447
MU- 86 111 197 14 37 51
Total 1,726 292 2,018 354 144 498
% of Total
MS+ 90% 10% 100% 76% 24% 100%
MU- 44% 56% 100% 27% 73% 100%
Total 86% 14% 100% 71% 29% 100%
Note: As of November 8, 2012. Latest ISR ratings were used. Data only includes those projects with available or valid ISR and ICRR
ratings. Includes only investment lending projects. ISR = Implementation Status Report. Ratings: MS+ = moderately satisfactory or better;
MU– = moderately unsatisfactory or worse.
37
APPENDIX B
CDD PROCUREMENT
Table B.6. ISR Procurement and IEG Risk to Development Outcome Ratings
Risk to DO ratings Procurement ratings
S+ MS+ MU- Total
CDD
Moderate or lower 55 70 4 74
Total 98 129 17 146
% Moderate or lower 56 54 24 51
Non-CDD
Total 202 294 31 325
CDD and Non-CDD
Total 300 423 48 471
Note: As of November 8, 2012. Data only includes those projects with available or valid ISR and ICRR ratings. Includes only investment
lending projects. DO = development outcome; S+ = satisfactory or better; MS+ = moderately satisfactory or better; MU- = moderately
unsatisfactory or worse.
Table B.7. CDD Projects: ISR Procurement and IEG Outcome Ratings by Region
Region IEG outcome ratings ISR procurement ratings
MS+ MU- Total % MS+ MS+ MU- Total % MS+
AFR 37 19 56 66 177 23 200 89
EAP 5 2 7 71 61 5 66 92
ECA 22 8 30 73 63 6 69 91
LCR 18 9 27 67 79 13 92 86
MNA 8 4 12 67 29 5 34 85
SAR 14 7 21 67 80 8 88 91
Total 104 49 153 68 489 60 549 89
Note: As of November 8, 2012. ISR = Implementation Status Report. Regions: AFR = Africa; EAP = East Asia and Pacific; ECA =
Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia.
Table B.8. CDD Projects: ISR Procurement and IEG Outcome Ratings by Sector Board
Sector Board IEG outcome ratings ISR procurement ratings
MS+ MU- Total % MS+ MS+ MU- Total % MS+
Agriculture and Rural 30 12 42 71 165 16 181 91
Development
Education 17 8 25 68 45 8 53 85
Health, Nutrition and 17 12 29 59 62 7 69 90
Population
Social Development 13 4 17 76 35 2 37 95
Social Protection 12 7 19 63 48 6 54 89
Urban Development 4 4 100 41 3 44 93
Water 7 4 11 64 33 7 40 83
Other 15 5 20 75 60 11 71 85
Total 115 52 167 69 489 60 549 89
Note: As of November 8, 2012. ISR = Implementation Status Report. S+ = satisfactory or better; MS+ = moderately satisfactory or better;
MU- = moderately unsatisfactory or worse.
38
APPENDIX B
CDD PROCUREMENT
Table B.9. CDD Projects: ISR Procurement and IEG Outcome Ratings by Loan Size
Loan size IEG outcome ratings ISR procurement ratings
($ millions) MS+ MU- Total % MS+ MS+ MU- Total % MS+
< 20 33 12 45 73 99 16 115 86
20≤ L <50 39 21 60 65 183 19 202 91
50≤ L <100 23 10 33 70 101 15 116 87
≥100 20 9 29 69 106 10 116 91
Total 115 52 167 69 489 60 549 89
Note: As of November 8, 2012. ISR = Implementation Status Report. S+ = satisfactory or better; MS+ = moderately satisfactory or better;
Table B.10. CDD Projects: ISR Procurement and IEG Outcome Ratings by Fragile State Status
and CPIA Rating
Rating Fragile state status CPIA ratings
FCS Not FCS Total ≤3 >3 Total
IEG outcome ratings
MS+ 22 93 115 13 100 113
MU- 9 43 52 6 45 51
Total 31 136 167 19 145 164
% MS+ 71 68 69 68 69 69
ISR procurement ratings
MS+ 73 416 489 46 437 483
MU- 16 44 60 14 45 59
Total 89 460 549 60 482 542
% MS+ 82 90 89 77 91 89
Note: As of November 8, 2012. The harmonized list of fragile situations for FY13 was used. Overall CPIA ratings for 2010 were used.
CPIA = Country Policy and Institutional Assessment; FCS = fragile and conflict-affected situation; MS+ = moderately satisfactory or better;
.
Table B.11. CDD Projects: ISR Procurement and IEG Outcome Ratings by CDD Concentration
CDD IEG outcome ratings ISR procurement ratings
Concentration MS+ Total % MS+ MS+ Total % MS+
≤ 25% 32 44 73 157 177 89
25% < R ≤ 50% 20 33 61 85 100 85
50% < R < 90% 39 54 72 193 215 90
≥ 90% 7 12 58 53 57 93
Total 98 143 69 488 549 89
Source: Social Development group database of CDD projects.
Note: As September 24, 2012. CDD concentration is measured as a share of total IBRD/IDA lending amount. CDD = community-driven
development; ISR = Implementation Status Report; MS+ = moderately satisfactory or better; MU- = moderately unsatisfactory or worse.
39
APPENDIX B
CDD PROCUREMENT
40
APPENDIX B
CDD PROCUREMENT
Table B.12. CDD Projects: ISR Procurement and IEG Outcome Ratings by CDD component
CDD component IEG Outcome Ratings ISR Procurement Ratings
MS+ Total % MS+ MS+ Total % MS+
Decision making by participatory 28 45 62% 161 178 90%
elected local governments
Community Control and Management 52 74 70% 196 224 88%
of Investment Funds
Community Control without direct 53 75 71% 240 268 90%
Management of Invest. Funds
Enabling environment 53 86 62% 224 253 89%
Total CDD Projects 104 153 68% 478 540 89%
Source: Social Development group database of CDD projects.
Note: As of September 24, 2012. The categorization is not mutually exclusive. Each CDD category shows the number of projects that had
a non-zero amount assigned to that component. Total CDD amounts counts the number of projects that had an IEG outcome rating or an
ISR procurement rating, respectively. CDD = community-driven development’ ISR = Implementation Status Report; MS+ = moderately
satisfactory or better.
Table B.13. CDD Projects: Procurement and Development Outcome by Main Project Objective
Main project objective Procurement ratings Outcome ratings
MU- Total % MU- MU- Total % MU-
Service delivery 16 138 12 11 36 31
Empowerment 4 68 6 8 29 28
Improving local governance 6 43 14 4 16 25
Livelihood security prioritization 11 111 10 7 25 28
Total CDD projects 33 281 12 24 81 30
Note: As of September 24, 2012. Includes investment lending only. To identify the main project objective, only those projects that rated
each area with 1 were considered. Procurement ratings are those of the last ISR. Development outcome ratings are those of IEG's
ICRRs. CDD = community-driven development; MU- = moderately unsatisfactory or worse.
41
Project Reviews
Table B.14. Sample of Projects Reviewed
Approval Country Project ID Project name Amount
FY ($ millions)
2002 Central African P073525 Multisectoral HIV/AIDS Project 17
2002 Ethiopia P050383 Food Security Project 85
2002 Jordan P076961 Horticultural Exports Promotion and Technology Transfer 5
Project
2002 Morocco P073531 Support for the Social Development Agency Project 5
2002 Senegal P074059 HIV/AIDS Prevention & Control Project 30
2002 Sierra Leone P073883 HIV/AIDS Response Project 15
2002 Tanzania P047762 Rural Water Supply and Sanitation Project 26
2003 Indonesia P059931 Water Resources & Irrigation Sector Management Program 70
2003 Kenya P078058 Arid Lands Resource Management Project Phase Two 60
2003 Rwanda P071374 Multi-Sectoral HIV/AIDS Project 31
2003 Tanzania P059073 Dar es Salaam Water Supply and Sanitation Project 62
2004 Angola P081558 3rd Social Action Fund (FAS III) 55
2004 Azerbaijan P076234 Rural Investment Project (AZRIP) 15
2004 Bangladesh P086791 Bangladesh - Reaching Out of School Children Project 51
2004 Brazil P080830 Maranhao Integrated Program: Rural Poverty Reduction 30
Project
2004 Honduras P083244 Nuestras Raices Program 15
2005 Brazil P076924 Amapa Sustainable Communities 5
2005 Congo, P086874 Democratic Republic of Congo Emergency Social Action 60
Democrat Project
2005 Guinea-Bissau P083453 Coastal and Biodiversity Management Project 3
2005 Indonesia P078070 Support for Poor and Disadvantaged Areas Project 104
2005 Peru P082588 Agricultural Research and Extension APL Phase 2 25
2006 Indonesia P085375 Third Water Supply and Sanitation for Low Income 138
Communities Project
2007 Afghanistan P0102288 Afghanistan: Emergency National Solidarity Project II 120
2007 Kenya P074106 Western Kenya CDD and Flood Mitigation Project 86
2008 Bolivia P101298 Second Participatory Rural Investment 20
2008 Ethiopia P108932 Pastoral Community Development Project II 80
2008 Haiti P106699 Haiti - Urban Community Driven Development Project / 16
PRODEPUR
2009 Brazil P110614 Sergipe State Integrated Project: Rural Poverty 21
2009 Ethiopia P106855 General Education Quality Improvement Project - APL 1 50
(GEQIP)
2010 India P101650 Andhra Pradesh Rural Water Supply and Sanitation 150
2010 Indonesia P115052 Third National Program for Community Empowerment in 785
Rural Areas (PNPM-Rural)
2011 India P102329 Rajasthan Rural Livelihoods Project (RRLP) 163
2011 Mozambique P107598 MZ PROIRRI Sustainable Irrigation Development 70
2011 Nicaragua P121779 Nicaragua Social Protection 20
EXAMPLES OF REVIEWS UNDERTAKEN FOR EACH CDD PROJECT
(Available for all 34 projects from IEG)
Sample Project 1: Afghanistan - Emergency National Solidarity Project -
43
Extracts from Project Appraisal Document
Project Objective: “Building on the current National Solidarity Project, the proposed
Grant will finance expansion to new areas to lay the foundations for strengthening
of community level governance, and to support community-managed subprojects
comprising reconstruction and development that improve access of rural
communities to social and productive infrastructure and services.”
Highlighted Components: The proposed IDA grant will finance about 23 percent of
the total projected cost of the National Solidarity Program in the next three years for
the following components:
• Block grants for communities ($94.3 million) to carry out subprojects

involving reconstruction and development activities through a facilitated
participatory planning process.
• Community Facilitation and Subproject Preparation ($9.7 million) to support

local communities through: (i) facilitation exercises to establish inclusive
community institutions, and identify local development needs and priorities;
and (ii) assistance for preparing subproject proposals.
• Community Development Council Capacity Development ($14.4 million) to

assist Community Development Councils in carrying out subprojects.
Procurement Arrangements: A large majority of the works to be procured under

this project would include community subprojects. These subprojects are
implemented by the communities using block grant provisions made by the project.
Each community will get a block grant of $200 per family with an upper ceiling of
limit of $60,000 per community. Therefore, the estimated value of most of the
community subprojects would be less than $60,000 per project. The community shall
carry out procurement following paragraph 3.17 of Bank Guidelines.
The overall procurement risk of the National Solidarity Project is moderate. The
project includes a large number of community level subprojects that are widely
spread over the country and will be procured by the communities. IDA will be
unable to carry out procurement reviews of a reasonable sample of subprojects as
would be the case in a normal investment project. Regarding the mitigation plan,
procurement administration is overseen by an international procurement specialist
who is also building in house capacity at the Ministry of Rural Rehabilitation and
Development (MRRD) level. IDA’S procurement team will continue to supervise the
procurement administration closely and carry out post-procurement reviews
periodically. The National Solidarity Project operational manual also provides step
44
by step procedures to be followed by communities for community procurement. The
procurement procedures at the community level have been further simplified. The
project will continue to provide training on community procurement to FP staff and
Community Development Councils.
Nature of Procurement Problems – Extracted from ISR, ICRs, ICRRs, and So Forth
ISR 7: Anomalies in the MRRD Incremental Operating Cost Expenditure. The

review of MRRD Incremental Operating Cost expenditure finds that civil servants
resigned from government service and joined as consultants. This is against Article
15 of the Regulation of Personal Affairs of Civil Servants.
ICR
• Lack of familiarity by MRRD staff of Bank Procurement Guidelines and

procedures contributed to delays.
• There was one major contract with the oversight consultant that was
amended five times. The amendments resulted in (i) time extensions from an
initial period of 12 months to 42 months, and (ii) price modifications from an
initial contract price of $5.3 million to $32.5 million.
• Under the District School Window, 18 contracts were awarded to construct 8

to 16 classrooms for 58 schools in seven provinces. These contracts could not
be completed during the project period and were extended to the second
project. There were several contractual modifications to increase the
quantities due to lack of diligence in the calculation of bills of quantities by
MRRD’s engineering staff working on the District School Construction and
Rehabilitation Window.
• Careful engineering planning combined with more effective liaison with MoE
and other donors could have contributed to avoid such incidents.
IEG Comments
- Based on the project documents, it appears that the community implemented

subprojects had a smooth implementation reflecting extensive efforts at
capacity building and technical assistance for the communities. No indication
from the project documents of wastage or misuse of resources used for
community-managed procurement. This would suggest that the Bank
guidelines bearing on the CDD procurement were not an issue in this case.
Transparency of the procedures at the local level as well as informal systems
45
of accountability among communities were key to assuring satisfactory use of
project funds by the communities.
- Procurement problems almost exclusively related to non-CDD components

(for example, hiring of consultants and contracting of classroom construction
under the district window) requiring procurement by a central agency.
Design of CDD projects requires adequate attention to capacity for ancillary
procurement needed to complement the procurement at the community level.
- The project scope and coverage experienced major changes during

implementation necessitating numerous ad hoc amendments to the
procurement plans and change orders to previous awarded contracts. Up to a
point this was unavoidable in a program of this nature but perhaps the task
team could have done a better job in anticipating some of the changes as well
as through more timely adjustments of the procurement plans.
- Given the country context, procurement risk rating of “moderate” may have
been too optimistic.
Sample Project 2: Brazil—Maranhão Integrated Program: Rural Poverty Reduction Project
Extracts from Project Appraisal Document
• Project objective: “The proposed project will help Maranhão to achieve the
goal of reducing poverty by increasing its [Human Development Index
(HDI)] from 0.647 to 0.700 by the year 2007, through …(c) financing demand-
driven community investments for income-generation, health and sanitation,
education, culture, environmental management and others impacting the
HDI and environmental sustainability …”
• Highlighted Components: Community Subprojects ($36 million or 90 percent

of total project cost) will support community matching grants for
approximately 1,200 small-scale socioeconomic infrastructure, education,
health, culture, productive, environmental and other investments aimed at
raising the United Nations HDI. All 216 municipalities in Maranhão (other
than Sao Luis) will be covered by this component, because of the importance
of encouraging cross-sectoral integration throughout the state, getting all
municipalities to focus on the HDIs, and strengthening local and municipal
institutional arrangements. However, about 60 percent of total component
cost will be concentrated in the 80 municipalities with the lowest HDIs.
46
Procurement Arrangements: In early 2003, an independent procurement review was
conducted of the predecessor project and concluded that the procurement
procedures followed under the project complied with the loan agreement. The State
Technical Unit (NEPE) procurement staff was well-versed in Bank procurement
policies and procedures. It was anticipated that all procurement financed by the
project would be carried out by the beneficiary community associations.
Procurement of goods and works under subprojects costing less than $50,000
implemented in remote areas would be carried out mainly through direct
contracting and community participation. This procurement method is appropriate
because most subprojects: (i) would be small and/or implemented in scattered or
remote areas and therefore it will be difficult to obtain competitive proposals; (ii) can
be managed directly by rural communities, who will also contribute directly to the
work through the donation of unskilled labor and local materials; (iii) will be
selected on the basis of willingness of the beneficiary communities to contribute to
and physically supervise works execution; and (iv) would provide means by which
communities could play an active role in the local development process.
Additional procurement of goods estimated to cost less than $100,000 equivalent, up

to an aggregate amount of $1.5 million, would follow national shopping procedures.
Contracts estimated to cost more than $100,000 equivalent would be awarded on the
basis of NCB procedures satisfactory to the Bank. Works contracts (other than
contracts costing less than $50,000 in scattered or remote areas) estimated to cost less
than $100,000 equivalent, up to an aggregate amount of $1.0 million, would be
procured under lump sum, fixed price contracts awarded on the basis of quotations
obtained from a minimum of three qualified local contractors in response to a
written invitation. The invitation shall include a detailed description of the works,
including basic specifications, the required completion date, a basic form of
agreement acceptable to the Bank, and relevant drawings, where applicable. The
award would be made to the contractor who offers the lowest price quotation for the
required work, and who has the experience and resources to complete the contract
successfully. The standard bidding documents for NCB agreed between the Bank
and the federal government of Brazil would be used. No ICB is anticipated for any
goods or works under the project.
47
Nature of Procurement Problems
ISR 12: This ISR provides an update on recent developments related to the findings
of the Procurement Post Review (100 percent coverage) carried out in August 2008.
The task team leader has reviewed the PAD/LA for consistencies and reported to
the SM accordingly. Following the conclusions of the Post-Procurement Review
(evidence of systematic misapplication of Bank procurement guidelines, and red
flags of possible fraud/corruption), several measures were taken: (i) INT was
promptly notified of the Post-Procurement Review findings, (ii) the Borrower was
notified (at the highest level) and given a chance to explain findings (letter attached),
and (iii) the Borrower was asked not to approve further community subprojects
until a corrective action plan was agreed to with the Bank. The borrower's reply to
the findings (from October 17) did not significantly change the conclusions of the
Post-Procurement Review. Consequently, the Bank will declare misprocurement.
Given the above situation, project implementation has basically stalled.
ICR: The Bank’s Post-Procurement Review revealed prima facie evidence that
bidding quotations did not meet Bank requirements. The Mid-Term Review found
overly-complex procurement procedures and lack of understanding of procurement
rules and processes by associations. NEPE was found not to be providing adequate
supervision and guidance to communities on problems to be avoided in
procurement, and there was a lack of rigorous analysis by NEPE of problematic
subproject documentation in order to make corrections. The Mid-Term Review
recommended that NEPE (i) prepare procurement procedures designed to facilitate
associations’ involvement, including model documents; (ii) send technicians to the
subprojects to review their status; and (iii) expand the use of the MIS as a
procurement control and supervision instrument. NEPE was also tasked with
updating the Operational Manual, which at the time of the Mid-Term Review had
no specific section on procurement. NEPE’s response to these recommendations was
slow and incomplete and was in any case overtaken by issues related to the
imminent closing of the project on December 31, 2008. Since procurement issues
were first flagged only in April 2008, by then it was too late to take meaningful
corrective actions.
Concurrent Bank missions on financial management and regular field supervision at

the time noted that (i) these procurement irregularities were not related to financial
management practices, and (ii) the investment subprojects identified in April 2008
physically had been completed to the satisfaction of the beneficiaries. In other
words, the problems identified were related specifically to the misapplication of
procurement procedures.
48
The type of problem found in Maranhão (systematic misapplication of Bank
procurement guidelines and confusion about how Brazilian Law 8666 on
procurement must be applied under the project) has not been identified in the other
CDD projects under implementation in Northeast Brazil. Nevertheless, one
important lesson from this experience is that the application of Bank-approved
procurement procedures for CDD projects requires considerable initial and periodic
training of project implementation agencies which, in turn, need to transmit this
training to beneficiary community associations. In the particular case of Brazil, this
requires more pro-active engagement on the part of the Bank to periodically clarify
to implementing agencies when and how Bank-approved procedures, as specified in
the loan agreement and operational manual, should be used by communities, and
how these may or may not relate to Brazilian Law 8666 on procurement.
IEG Comments
- Built on good experience with an earlier project.
- Problems arose with compliance with procedure. No evidence of systematic
fraud. Subprojects completed to the satisfaction of beneficiaries.
- Importance of post reviews to ensure compliance.
- Need for clear guidance to implementing agencies when conflict between
Bank and government procedures.
49
ICT Procurement: Summary of Quality Assurance Group, IEG, and Bank Reports
IEG reviewed evaluative and other lessons that emerge from earlier reviews of ICT
that mention procurement, including a Quality Assurance Group assessment, an
IEG evaluation, a Bank study on Financial Management Information Systems
(FMIS), and the Bank’s ICT Strategy. Three of these were completed in the past two
years, indicating that the effectiveness with which the Bank approaches ICT
investment is an important topic in the Bank.
The Quality Assurance Group assessment reviewed 23 projects with a combined

procurement of ICT components valued at $680 million. 17 Its procurement
discussion focuses primarily on resource and skills issues, and procurement
planning. With respect to the latter, a key finding is that the special procurement
requirements of project ICT components are not properly recognized. This is treated
as a project design issue. The report concludes that the root of the problem is
weakness in the procurement strategy, as less than one fourth (23 percent) of
projects are rated as satisfactory or better in this respect. Today, the Quality
Assurance Group assessment is somewhat dated (it covers projects that were under
implementation eight or more years ago) but it points toward a broad spectrum of
skills and resource issues, and weak procurement planning, that are constraints in
achieving project outcomes.
The IEG ICT evaluation covered the Bank’s work in the ICT sector in several broad
areas (IEG 2011b). A part of the evaluation dealt with ICT applications such as the
use of ICT for delivery of public services. Procurement was assessed for its
contribution to their effective delivery. The evaluation offered the following
observations and recommendations regarding procurement.
• Procurement is a major constraint to implementing ICT projects. Procurement

delays, attributable in part to risk-averse procurement specialists, slow down
project implementation, in some cases stalling projects for years.
• The Bank’s procurement rules are designed for infrastructure projects; for
example, separating contracting from professional (consulting) services. They
are cumbersome and not suited for a rapidly evolving sector like ICT. The
Bank needs to better adapt its procurement rules to meet the needs of ICT
projects.
• The Bank lacks necessary ICT procurement-related expertise, leading to

inconsistencies of approach.
51
• Change management is inherent in projects that have a significant ICT
component (modernization of a tax system, for example). This further
complicates specifying system requirements and managing implementation.
This is a common theme of this study, reported by task team leaders
(Indonesia) with direct experience, as evidenced in IEG’s field visits.
The Bank’s recent ICT strategy followed the IEG ICT evaluation and reflects many of
its findings and recommendations. In particular it notes a lack of ICT-related
procurement expertise at the Bank, inconsistency of approach, and a procurement
regime derived from infrastructure projects, not well suited for ICT procurement.
Additional points raised by the strategy include:
• A possibly higher incidence of corruption in information technology

procurement than in other project components, based on a study in the
Europe and Central Asia Region.
• A need to build ICT-related procurement skills within the Bank, but also to
build capacity in client agencies, particularly in areas related to developing
technical specifications and bidding documents.
• Acknowledgement of recent improvements and agreed areas for future

change. The former include the 2011 revisions to the guidelines, which
provide for two-stage bidding, framework contracts and continuation of
engagement with a brand name supplier. Proposed future actions include the
update of the Bank’s ICT standard bidding document, revisions to
procurement specialist training materials, and allocation of large complex ICT
to the most experienced procurement specialists.
The FMIS study reviewed 55 closed and 32 active FMIS treasury-related projects
covering the period 1984 and 2010 (World Bank 2011d). All include large ICT
systems, implementation of which was a primary focus of the study. The study
pointed toward 18–24 months of typical delay caused by lengthy procurement
processes, stretching in rare cases to over two years. 18 This contributed to major
delays and dissatisfaction on the part of the borrower. It also pointed towards
technical capacity constraints on the Bank’s project teams and the project
implementing unit staff of the borrower. The study recommends ensuring ICT
expertise on the Bank project team and building capacity of the project
implementing unit, with the latter focused on building organic capacity. The study
also pointed to the need for appropriate procurement planning and recommends
combining procurement into a smaller manageable number of packages, typically
52
three or fewer. Finally, it alluded, in one of its case study countries (Turkey) where
procurement using country procurement systems was completed in six months. 19
Box B.1. FMIS Packages

The Bank’s FMIS study reported from its case studies that, in most instances, three variants describe
the design of procurement packages and procurement methods.
Option 1 (Georgia, Moldova, and Tajikistan): One single-responsibility contract package using two-
stage ICB was bid covering the implementation of all ICT software, hardware and network
components.
Option 2 (Albania, Azerbaijan, and Russian Federation): Two linked contracts were bid
simultaneously involving (1) two-stage ICB procurement of a contract for the development of
application software, including demonstration, and (2) single-stage ICB bidding of a contract for the
installation of all hardware and network equipment. To ensure compatibility, implementation of the
second contract depends on the inputs to be provided by the software developer under the first
contract. The second contract was delayed due to this linkage.
Option 3 (Kyrgyz Republic and Ukraine): Two separate independent contracts were bid
simultaneously involving (i) two-stage ICB bidding of a contract for the development of applications
software and installation of central hardware, including demonstration, and (ii) single-stage ICB
bidding for the installation of standard field hardware, engineering support systems and network
equipment.
The study concluded with suggestions to improve the design of FMIS procurement:
It is advisable to have two separate independent packages (Option 3). The study found that this
reduced the complexity of ICT implementation and resulted in faster implementation. Another
possibility is to use Option 1 for a well-defined and relatively small-scale implementation of the FMIS
system.
Bidders should be required to comply with all ‘mandatory’ requirements listed in the bidding
documents. This ensures that minimum technical requirements are met. Technical evaluation/merit
points should be considered only when there is a need to define and evaluate ‘desirable’
requirements that will add value to the minimum mandatory requirements—where there is
quantifiable value expected from offers that exceed the minimum requirements.
Where the technical requirements are well defined, the weight of price should be kept as high as
possible (80 percent or more) to benefit from price competition, while ensuring the delivery of good
quality solutions based on mandatory requirements. The weight of technical evaluation for desirable
features can be up to 20 percent, if there is a need to consider some optional high-value technical
requirements in addition to mandatory requirements. IEG agrees that the maximum level of 20
percent is reasonable. A higher level leads to more subjectivity in the evaluation and is also an
indicator that technical requirements are not well understood.
Source: IEG 2008a.
53
PPP Procurement: Comparison of the Bank and Other Organizations
Over the past few years, multilateral development banks (MDBs) have been
working to elaborate procurement requirements for PPP projects through their
Heads of Procurement. There is an established working group to study PPP-related
procurement matters that provides guidance on the matter of “acceptable
practices,” and harmonizes approaches among the MDBs. Outputs are meant to
guide the different MDBs as their policies develop. They include:
• Procurement & PPP Transactions Guidance for MDB Public-Sector Engagements

(2012). This guide aligns closely with the Bank Guidelines and the Guidance
Note, although with less detail (for example, in treatment of bidding
procedures). It elaborates the role of the MDBs in providing advisory services
and guarantees.
• Procurement Principles Applicable to Private Sector Transactions—Guidance for

MDBs (2012). As with the guide described above, it reflects existing Bank
practice, but expands into other areas. For example, it emphasizes the
importance of due diligence, recognizing that private sector projects are often
not subject to the same oversight as public sector projects (they might, for
example fall outside the jurisdiction of a national procurement law). Typical
due diligence requirements and documentation are detailed in attached
checklists. The guide also addresses the need for arms-length arrangements
(between a client and its affiliates, who might also act as contractors) and
provides guidance on guarantees and unsolicited proposals.
The approach of other MDBs is documented in the heads of procurement PPP

working group guidance notes described above. As with the Bank, MDB policies
are typically spelled out in the procurement guidelines of the organization, and
supplemented by interpretation notes and guides. With the exception of the Black
Sea Trade and Development Bank, requirements of the other MDBs are very similar
to the Bank’s. 20 When it comes to on-lending arrangements, where the beneficiary is
a PPP party, MDBs leave procurement to the intermediary to oversee, saying that
procurement must follow well-established methods and commercial practices
“acceptable” to the organization, or similar wording. For selection of concessions,
all MDBs specify open and competitive tendering, and all provide for the exception
to open tendering by requiring competitive tendering for downstream
procurement. There are a few variations between the organizations:
55
• The European Bank for Reconstruction and Development requires a formal
competitive process but defines it in a separate concession policy, rather than
in its guidelines.
• The African Development Bank (AfDB), Inter-American Development Bank
(IDB), and Islamic Development Bank (ISDB) all specify ICB for selection of
the concession, as did the Bank prior to the 2011 revision to its guidelines. The
European Investment Bank (EIB) specifies “formal international” tender.
• Except for the Bank, ADB, and ISDB, remaining MDBs employ a market
comparison to contract terms, saying that costs must be in line with the
market and the project budget, and that contract terms should be fair and
reasonable.
• EIB invokes compliance to local legislation. It also requires that European
Union treaty conditions be met where a concession is not awarded through
formal international tendering. These conditions include adequate
advertising, international competition, and a fair and nondiscriminatory
process.
There is very little variation in the PPP procurement policies of the MDBs, as
reflected in their procurement guidelines. Furthermore, through the heads of
procurement working group, the MDBs have worked to harmonize such things as
interpretations of “acceptability,” further aligning their practices. The policy
similarities are unsurprising. Over the past 10 years there has been a push to align
procurement guidelines. Differences that exist (AfDB, IDB, and ISDB specifying ICB
for selection of concessions, for example) probably reflect one organization getting
ahead of the other, rather than any fundamental disagreement. With policies so
similar, it is differences in application and interpretation of those policies that most
likely differentiate approaches to PPP procurement.
In contrast to the Bank, the International Finance Corporation (IFC) does not have
documented requirements regarding procurement in its PPP investments. IFC
reviews a concession agreement (and key contracts in a typical transaction) as part of
its overall assessment of credit, development and reputational risks in the
transaction, rather than as part of a self-standing procurement exercise. IFC takes a
commercial approach, assessing if the PPP arrangement was the result of an open
and competitive process, prices are reasonable, the project complies with local laws,
the project is financially viable, and the parties to the project are reputable,
competent, and financially sound. This reflects the commercial orientation of IFC.
Unlike the World Bank Group, which separates private and public sector lending
operations between two different organizations (IFC and IBRD), most other MDBs
have an in-house private sector lending window. Their operations include
56
departments dedicated to private sector lending, yet they manage to accommodate
this private sector lending function within the terms of procurement policies that are
almost identical to the Bank’s. Within these organizations, procurement policies,
including PPP-specific provisions tend to be developed by public sector procurement
specialists (MDB representatives on the heads of procurement PPP working group
are all public sector procurement people, except for some participation of IFC). It is
understandable then that the PPP-related procurement policies of other MDBs, like
the Bank, address public accountability concerns and are not fully private-sector
oriented, as are the IFC’s.
Box B.2. Public Accountability frames Government Regulations for PPP Transactions
Like the MDBs, governments, and those who may course financing through them with a sovereign
guarantee, are constrained by their public accountability. A guide produced by the World Bank
Institute gives examples of PPP procurement requirements for six countries (Brazil, Chile, Egypt,
Mexico, the Philippines, and South Africa) and the European Union. These examples describe the
legal requirements typically contained in a PPP or similar law. Also described are the methods used
in the different countries for establishing qualifications of clients, bidding processes to be followed,
conditions of negotiations, and basis of awarding concessions (such as price and technical capability).
Refer to Table 3.5.1 of the WBI guide for a detailed description of the different PPP procurement
procedures.
Sources: World Bank Institute, IEG.
Of the different country approaches described in the World Bank Institute (WBI) PPP
reference guide countries following the European Union procedures have the most
options (World Bank 2012j). The options allow open or restricted bidding (restricted
to a minimum number of prequalified bidders), single or multistage bidding (where
full proposals are invited at one time, or different aspects of the proposal are invited
in sequence), and different approaches for negotiation, ranging from no negotiations
at all to simultaneous negotiations with all bidders throughout the bid and
evaluation process. In all cases, European Union regulations allow for contracts to be
awarded to the best financial proposal (technically compliant) or the best
combination of technical and financial scores. The following variations of country
policies as compared to the Bank’s are noted:
• All countries, including those following European Union procedures, use a

competitive tendering process to arrive at a PPP concession contract. All
permit prequalification as an option, except Philippines, which states it is the
“norm” and South Africa, which mandates it (and further limits the number
of prequalified bidders to between three and four). Prequalification restricts
participation in the bidding to prequalified firms. This is not at odds with
Bank policies; the Bank Guidelines mandate competitive bidding and its
57
guidance encourages prequalification. However, the prequalification process
itself must be advertised and opened to all eligible bidders, and may not limit
the number of bidders as does South Africa.
• All countries require a single-stage bidding process, except Egypt and those
using the European Union competitive dialog procedures. South Africa
allows the addition of a second stage to bidding where the single stage
process does not produce a clear preferred bid (“best and final offer”
approach). All of these arrangements are fully in line with the Bank’s
guidance, which provides for either single- or two-stage bidding, at the
option of the government.
• Negotiation, to arrive at a concession contract, is an area where there appears
to be considerable divergence. The policies of three countries (Brazil, Chile,
and Mexico) are silent on the topic. The Philippines and South Africa do not
allow negotiation, apart from exceptions (where there is only one bid, for
example), consistent with Bank policy. Egypt and the European Union
provide for a “competitive negotiation” procedure, as an option. The Bank’s
guidance note (and also the MDB guidance) does not accept competitive
negotiation, where contract award is based on negotiations with several
bidders. It is therefore at odds with the policies of Egypt and any country
following European Union procedures. It is not clear if the other countries,
whose policies are silent on the matter, accept the practice or not.
• Except for Brazil and the Philippines, all countries, including those applying
European Union policies, have two options for deciding who is awarded the
concession contract. One option is to award on the basis of best financial offer
among technically compliant offers (“lowest evaluated bid” in Bank
terminology). The second is to use a point scoring system that combines
financial and technical proposals to come up with an overall best offer. Brazil
and Philippines do not allow the second option. The Bank, in its policies and
guidance, accepts both methods, as does the guidance from heads of
procurement. The WBI document notes that South Africa applies a preference
for black-owned bidders. Similar preferences (for domestic bidders, for
example) are not uncommon--the Bank allows some in its public procurement
policies and would likely accommodate certain country-specific preference
schemes when assessing “acceptability” of the procurement system.
The WBI guide addresses the matter of unsolicited proposals, proposals from a
private sector entity to government to enter into a PPP. These are seen positively, as
a way to encourage the private sector to promote its ideas and expertise to aid the
delivery of public services. Seven countries were surveyed (Chile, Indonesia, Italy,
Korea, the Philippines, South Africa, and the U.S. state of Virginia). All accept
58
unsolicited proposals, but then conduct a competitive bidding if the proposal is
worthwhile. To incentivize such proposals, the proponent is given an advantage
during evaluation, compensated for its development costs if it loses the bidding, or
given an opportunity to match a competing winning bid. The heads of procurement
group guidance recommends this approach. The Bank’s guidance is vague, saying
that awarding a contract to an unsolicited proposal may be considered as an
exception under the guideline provisions for direct contracting, or that the Bank may
require a competitive process, possibly “adding necessary flexibility.”
59
Appendix C. Procurement Risk Concepts and
Thresholds
Risk Concepts
Box C.1. Procurement Risk in Relation to the Bank’s Principles of Procurement—Process and
Outcome Risk
Transparency in procurement process sees to the first injunction in the Bank’s Articles: that loan funds be used
for the purposes intended. Sometimes less emphasized (even though measures to explicitly address fraud and
corruption in procurement have become more visible) is the need for transparency in contract management and
execution, so transparency in outcome is also needed. Transparency (widespread advertising of tenders) also
serves the equity principle by ensuring tender information is broadly disseminated.
The economy principle has been intended to embody a prime purpose of procurement, namely to secure for the
Borrower optimal value for money in the use of Bank loan funds, as the outcome of a competitive tender, against
the risk that many factors may work to impede that optimal outcome. While the achievement of an optimal
outcome may depend on an effective procurement process, the optimal economic result itself is an outcome
concept. As increasingly recognized today, “economy” on its own may not be adequate, especially if narrowly
interpreted, for a best value outcome.
The efficiency principle relates exclusively to the effectiveness of the procurement process (in pursuit of the
economic outcome) and addresses the risk that the procedural requirements of procurement policy may be such
as to be overly cumbersome, costly and slow to an extent that could slow project implementation. To that extent
highly inefficient procurement processes or systems may be a factor which impacts project outcomes, but the
efficiency principle relates exclusively to process. Too seldom emphasized in Bank discourse is the fact that there
is a key risk-efficiency trade-off embodied in the balancing of efficiency and economy, which is dealt with in
more detail in the main text.
The equity principle requires that, in the execution of its procurement policies, the Bank ensure that suppliers
from all eligible countries be given equal knowledge and an equal opportunity to bid on the tenders for contracts
under projects being financed by the Bank. Clearly, the equity principle cannot be extended to outcomes, since
there can be only one winner per bid. To that extent, the equity principle relates also only to process.
The local development principle historically supported the purpose of World Bank lending to bring
development to developing countries markets. (As data from procurement outcomes in recent years show an
increasing degree to which Part II country suppliers have won ICB tenders on Bank projects —now in excess of
60 percent — suggest that the risk that favoring local suppliers may conflict with the economy principle have
probably declined.
Procurement Principle Process Outcome Risk Addressed

Transparency X X Loans not used for purposes intended
Economy X Procurement gives sub-optimal value for money,
Efficiency X PR process too cumbersome, slow, delays outcome
and project implementation
Equity X Uneven playing field between eligible bidders
Local development Local suppliers, contractors shut out by international
- Goods, contractors, consultants X bidders
- Local procurement systems Local PR systems not compatible with Bank policy
X
61
62
Box C.2. Procurement Risk Management—Different Levels in the Bank
Procurement risk is dealt with at different levels in the Bank, including the entity level, country and
sector levels, and, most importantly, at the level of individual project transactions. The extent to
which procurement risk is treated as a separate set of risks, as against being integrated into more
general operational risk, varies both across levels and within them.
At the level of the Bank as a whole (entity level): There are three key functions at the entity level
which view procurement risk in real time, as part of ongoing operations:
• Integrated Risk Management Report: This report, which is the Bank’s most aggregated report
on risks facing the institution as a whole is based on a survey of Bank-unit “appetites and
tolerances” for 41 identified risk items. One of these items is “FM, Procurement and
Disbursement risk,” which views procurement risk mainly from the perspective of how it
may impact the Bank’s overall disbursement rate. The Chief Risk Officer unit is currently
engaged in a brainstorming pilot analysis which is attempting to seek answers to a number of
procurement risk related questions, with a view to establishing the appetites and tolerances
for this risk factor. Some key questions being explored include: How is procurement risk
managed? Beyond the taxonomy of risk management tools, how much procurement risk
should the Bank take on? Does the Bank follow a risk-return model in procurement where
higher risks maybe tolerated where returns to development (measured either by IRR,
progress towards PDOs, or IEG outcomes) are higher? How should procurement risk be
flagged? Has the bar for procurement protection been set too high or too low? It is
anticipated that this process could be completed by end FY13, but this will depend on senior
management (VPU/EVPU) sign-off, and could also be potentially impacted by the arrival of
a new Bank CFO in March.
• The Chief Procurement Policy Officer: This is the Bank’s chief authority on procurement
policy, and in that capacity is the apex of the Bank’s central and operational (Regional and
Network) based procurement specialists who administer the function throughout the Bank,
including a significant contingent of decentralized specialist staff located in the Bank’s field
offices in many countries. This team represents the core of the Bank’s procurement
implementation capability, it upholds procurement policy, it advises operational staff at the
level of project transactions and their procurement requirements, it acts to settle complaints
and disputes, and it offers a training component to both Bank staff and staff within
borrowing countries involved in implementing Bank projects. In that sense this officer
commands a team that is the backbone of the Bank’s capability in managing procurement
risk at all levels of the institution. He or she also serves as an ultimate arbiter on procurement
policy matters—often involving real time disputes on specific procurement contracts—where
these cannot be resolved at a unit or regional level.
• The Operations Procurement Review Committee (OPRC): The OPRC plays a key oversight
role by reviewing and clearing the highest-value and most complex contracts financed under
Bank-supported operations. All contracts above a defined threshold level must be reviewed
by the OPCR, and some complex or sensitive contracts may be reviewed, regardless of their
value. The thresholds for mandatory review were increased in January 2009 (unchanged
since 1992) to (i) $50 million or more for civil works; (ii) $30 million or more for goods; (iii)
$15 million or more for consultancy services; (iv) $5 million or more for all direct contracting
and single-source contracts with firms; and, (v) $1 million or more for single-source contracts
for individual consultants.
63
Source: IEG interviews with Bank staff.
Box C.3. Procurement Risk Management Relative to Other Operational Risks in Bank Projects
An important question in managing procurement risk at the level of individual projects is how this
risk is viewed in comparison with other operational risks facing Bank projects. The Bank addresses
this issue by distinguishing four categories of risk that affect projects, two of which (Country Risk
and Sector Risk) are external to the projects, while the remaining two (Inherent Risk and Control
Risk) refer specifically to factors arising from the project itself. Risks are rated at four levels: high,
substantial, moderate, and low. The categories are as follows:
External to the Project
• Country Risk: Country risk relates to the general standing of the country in terms of business
environment, operational risk, strength of institutions, stability of the economy, status of
procurement reform, record on governance and general risk of fraud and corruption.
Country risk assessments, including relevance to procurement (for example, as linked to
CPIA results, and/or governance and fraud and corruption risk) should be contained in the
Country Assistance Strategy for each borrowing country.
• Sector Risk: This relates to risks generic to specific sectors (including those derived from
strength of institutions, investment types and commercial risks) that should be reflected in
Sector Strategy Notes compiled by the networks.
Specific to the Project
• Inherent Risk (Project Risk): For a given project the relevant elements of Country, Sector
and Project risk, including the overall business environment and procurement capacity of the
country, design of the project, its technical complexity, its operational and/or commercial
objectives, market conditions, technology and financial risks, all combine to form the
Inherent Risk of the project. These are independent of specific procurement risk factors,
which are treated separately (see below) but, once identified, procurement risk should be an
integral part of overall project risk.
• Control Risk: Control risk is related to the internal control structure and the capacity of the
implementing agency for the project to competently handle three aspects of procurement
execution: Governance, Capacity in policies and procedures (knowledge of Bank Guidelines and
systems), and Oversight of procurement implementation. In brief, Control Risk reflects the
capability of the implementing agency to manage procurement process risk. Part of this
capability involves the capacity to develop a dynamic Procurement Risk Mitigation Plan
containing specific measures to address key procurement risk factors, and to monitor their
application over time. In standard Bank practice, all Project Appraisal Documents (PADs)
contain a procurement annex (written by the procurement specialist, but drawn from
material produced by the borrower) built around the two key dimensions of control risk: An
implementing agency capacity assessment; and the quality and effectiveness of the
Procurement Risk Mitigation Plan.
• Procurement Risk: Procurement risk is the risk deriving from the specific nature of the
supply and service contracts that have to be procured to implement the various components
of the project, as guide by the Bank’s or other procedural guidelines, with due regard to the
four overriding principles contained in Box C.1. (As described below, there is a need to
clarify this usage within the risk taxonomy currently used by the Bank)
• Residual Risk: The Residual Risk is the level of procurement risk remaining in the project (in
the judgment of the procurement specialist) after implementation of the procurement risk
mitigation plan.
Sources: OP/BP11.00, the P-RAMS template; OPCS Reports, for example, Use of Country Procurement Systems Proposed Piloting
64
Program. OPCS May 20, 2008 Annex D.
Box C.4. Managing Procurement Risk in Projects—Summary of Tools and Procedural Devices
Available to Procurement Staff to Manage Procurement Risk
• Project Design: Project design is mainly driven by project • Timing of Procurement: In large and/or complex
content and objectives, but quite often project components are procurements task team leaders/procurement
chosen because their procurement features set them apart specialists may elect to conduct advanced
from other components. Many projects also have project procurement (sometimes with retroactive financing) in
implementation components designed to finance the order to ensure procurement does not delay project
enhancement of implementation capacity of the implementation. Frequently pre-appointment of
implementing agency. Projects with highly decentralized procurement staff to project implementing/managing
objectives often have procurement delivery as part of the unit may be a condition of project approval or
project design. effectiveness.
• Procurement Plan: Written by the Borrower, under Bank • Frequency of Supervision: In higher risk projects
supervision, and mandatory for every project, this lays out procurement specialist may call for more frequent
the whole procurement strategy and content and highlights than annual supervision.
the risks to be addressed. It is intended to be updated at least
annually, and in that sense is both a planning and a dynamic • Use of website: In pursuit of the Transparency
monitoring tool. principle many projects now use web sites as a means
to display procurement tenders, awards and
• Selection of Implementing Agency: The Borrower may outcomes. E-procurement is being encouraged but is
propose but the Bank must agree on who will be the not yet widespread.
implementing agency. The choice is most often between a
Ministerial unit or agency, and project • Financing of Incremental Operating costs: As a
implementation/management unit (project means to ensure that project implementation is
implementing/managing unit) required by the Bank if it properly resourced projects are often designed to
deems the Ministry unit to be lacking in capacity. include the salary and other operating costs of
implementing agencies (excluding civil servant
• Staffing: It is very common for measures to be taken to salaries where these units are staffed by government
enhance the skills and experience of the implementing agency staff).
staff (whether in the project implementing unit or another) by
either or both training and hiring of consultants to the agency • Addressing Fraud and Corruption: The requirement
as a means of enhancing capacity to implement procurement. to address ethical behavior and fraud and corruption
risk in projects was introduced into the Procurement
• Procurement Method: Within the broad thresholds set Guidelines in 1996, but was not widely practiced until
Bank-wide by OPCSOR (and within that framework a decade later, following the governance and anti-
separately by the Regional RPM) project staff are free to corruption program and IDA internal controls review.
define which procurement method will be used for which Various tools are available:
project components, based on their judgment of risk and
efficiency in achieving an economic outcome. ICB + Prior o Ethics Pledge: Some projects require an
Review (and/or NCB) remains the default procurement ethical conduct pledge from project
method for all larger contracts (both goods and works), but counterparts.
newer forms of lending (particularly in the social sectors) o GAAP/IS: Recent IL reform and the
have increased the share of other methods. governance and anti-corruption have
introduced Implementation Support (IS) and
• Use of Pre-/postqualification: In large, complex projects Governance and Accountability Action Plan to
with substantial technology dimensions task team focus and resource implementation and
leaders/procurement specialists may choose to require pre- explicitly address transparency and fraud and
qualification of bidders, to reduce know-how risk in corruption risk.
procurement. o Civil Society Oversight: Increasing use is
being made in project design of a role for CS in
• Thresholds for Procurement Method and Review: oversight, often involving fraud and
Maximum Prior Review thresholds are set institutionally (in corruption risk and using the project website,
Annex C of BP 11.00), for goods, works and services, with among other means.
different levels for each risk category of the implementing o Establish Complaints process is a
agency. Regional Project task team leaders/procurement common means of providing access to red
specialists can set lower thresholds where they believe risks flags and indicators of fraud and corruption
are greater. Thresholds are a key tool for risk management at risk.
the project level. PS may also increase the sample of contracts
65
to be post-reviewed (generally 20 percent); they may call for • New Tools –P-RAMS/ORAF: Mandatory since July
Post-Procurement Reviews annually or twice a year, and may 2010 the P-RAMS provides a detailed checklist in the
call for Independent Procurement Reviews. implementing agency capacity assessment of
questions around 11 risk factors, and a Mitigation
Measure Action Plan aimed at dynamically assessing
all forms of procurement risk, including fraud and
corruption risks and intends to inform the ORAF and
the PAD.
Source: IEG review.
66
Thresholds Analysis – “Methods” Thresholds
SETTING METHOD THRESHOLDS TO OPTIMIZE COMPETITION
The Implementing Agencies of borrowing countries can use a variety of

procurement methods in World Bank funded projects. The method selected
depends on a number of factors including the type and value of good, work or
service procured and the potential interest of domestic and foreign bidders. 21 The
Bank sets monetary threshold above which certain methods have to be used. In the
case of goods and civil works these thresholds indicate, among others, above which
contract value ICB shall be used. 22 Contracts below this value are most prominently
procured through national competitive bidding (NCB).
Monetary procurement method thresholds are set for each borrowing country by the
regional procurement managers (RPMs), drawing on analyses of procurement risk
undertaken as part of Country Procurement Assessment Reviews (CPARs) or other
means, and including foreign bidders’ interest in participation in domestic markets;
conditions, size and depth of the market and capacity of the local industry. 23
ICB thresholds therefore show a significant degree of variation across countries and
within regions. More sophisticated clients and middle-income countries tend to have
higher thresholds. The maximum ICB threshold used by the Bank is $25 million, for
civil works contracts in Brazil. In East Asia and the Pacific ICB thresholds for civil
works vary from $100,000 in Kiribati to $20 million in China within the same region.
Some country-specific thresholds are set with a monetary range for risk categories
low to high, as in India, where ICB is used for works contracts starting from $10 to
20 million.
One feature distinguishing ICB and NCB lies in the requirements for the publication
of Specific Procurement Notices. Paragraph 3.4 of the Bank’s procurement
guidelines stipulates that borrowers may limit advertising to the local press for NCB
processes. Foreign firms cannot be barred from competing for NCB contracts; the
different treatment of advertising for NCB and ICB reflects the position that foreign
firms would not normally bid for NCB contracts given their value and nature. 24
With its procurement guidelines the Bank aims to allow borrowing countries to buy
high quality goods and services as economically as possible and expresses its
position that this objective is best achieved through transparent, formal competitive
bidding. For the procurement of equipment and civil works, ICB is the procurement
method the World Bank encourages its borrowers to use in the majority of cases. 25
67
The setting of ICB threshold is critical given the expected benefits in competition
and economy in the tender process. There is, however, a possible tradeoff with
processing time and efficiency. ICB procurement tends to take longer than contracts
awarded with NCB procedures, as IEG’s analysis of processing times in
procurement in Chapter 4 suggests.
ANALYSIS OF DISTRIBUTION OF ICB CONTRACTS
IEG has attempted an analysis similar to the one undertaken by the AfDB (as
described in Chapter 3). Using contract data from its budget tracking system, AfDB
has shown that in works contracts foreign bidders entered bids on contracts for
mostly those contracts in the top quintile (20 percent) of contract value. The AfDB
concluded, accordingly, that to set the ceiling contract for ICB at any level lower
than 80 percent of highest value would involve no increase in competitive bidding
but would involve significantly more review work.
To analyze the trends in methods used for procuring civil works in World Bank-
funded projects, IEG used recent years of the Bank’s Form 384 database on prior
reviewed contracts financed by the Bank. 26 Between 2007 and 2012 the Bank
financed 3,618 prior reviewed civil works contracts procured through ICB (out of
12,572 Civil Works contracts total). This gives an average of around 50 prior
reviewed civil works contracts procured through ICB per month. On the other hand
9,453 goods contracts were procured through ICB during the same period, an
average of around 130 contracts per month.
IEG undertook an aggregate analysis for procurement of civil works contracts only,
but suggests a closer look at cluster of countries with similar parameters to
determine the impact of current method thresholds. Based on their value civil works
contracts procured through ICB between 2007 and 2012 were divided into quintiles
and disaggregated into national and foreign contractors.
Participation of foreign firms, in terms of numbers of contracts, is relatively low in

the first four quintiles, ranging from 15.5 percent in the first quintile to 24.2 percent
in the fourth quintile (Table C.1). These results vary only minimally when looking at
the distribution by total value in those quintiles, though they are slightly higher than
what is found in AfDB’s analysis; that is, more foreign firms win contracts in the
lower quintiles of World Bank funded projects.
Table C.1. Distribution of Civil Works Contracts Procured through ICB (2007–12)
By number
Quintiles Value of National contractors % Foreign contractors (regional %
contracts ($m) (nos) and nonregional) (nos)
68
1 0 - 0.6 612 84.5 112 15.5
2 0.6 - 1.5 574 79.3 150 20.7
3 1.5 - 4.1 570 78.8 153 21.2
4 4.1 - 11.0 549 75.8 175 24.2
5 >11.0 436 60.3 287 39.7
Upper Percentiles
90th percentile >23.3 189 52.4 172 47.7
95th percentile >41.1 75 41.7 105 58.3
By value
Quintiles Value of National Contractors % Foreign Contractors (Regional % Average Contract
Contracts ($ (total $ millions) and Non-Regional) (total $ Value ($ millions)
millions) millions)
1 0 - 0.56 169.0 85.3 29.1 14.7 0.3
2 0.56 - 1.54 569.0 78.9 152.0 21.1 1.0
3 1.54 - 4.11 1,500.0 78.7 406.0 21.3 2.6
4 4.11 - 11.00 3,710.0 75.7 1,190.0 24.3 6.8
5 >11.00 13,300.0 50.6 13,000.0 49.4 36.3
Upper Percentiles
90th percentile >23.30 9,460.0 46.0 11,100.0 54.0 56.9
95th percentile >41.10 5,990.0 39.9 9,010.0 60.1 83.3
Source: IEG analysis.
The findings of IEG’s analysis suggest that about 80 percent of prior reviewed ICB
contracts for civil works in the lowest four quintiles, with a maximum value of $ 11
million, were won by national firms.
Although the results of IEG’s analysis are broadly similar to findings of the AfDB,
there are some differences: in the highest quintile of contract sizes (with values
greater than $ 11 million) in World Bank projects more contracts tend to be won by
national contractors than in the case of the AfDB. In the case of the World Bank the
majority of these high value contracts were won by national firms, with foreign
firms winning only 40 percent. 27 The picture changes for the World Bank in the
highest five percent of contracts (with values greater than $41 million), where more
contracts were awarded to foreign firms.
It has to be noted that the AfDB paper assumes a uniform level of ICB threshold for
all countries while the World Bank uses country-specific thresholds that vary
significantly as described above. Also, in practice, at the World Bank, the ICB
procurement method may frequently be used even below the threshold set for the
country; indeed the analysis suggests that average contract values are relatively low
for the first two quintiles where 1,448 contracts with an average value below one
million were procured through ICB (see also section on prior threshold analysis
earlier in this chapter). In FY12 half of the civil works contracts procured through
ICB used this procurement method for contract values below the threshold set on
the country level.
69
If the upper limit of the fourth quintile was to be considered as the ICB threshold for
civil works for the contracts in this aggregate analysis, the number of contracts with
mandatory ICB use would have reduced significantly. This would represent an
improvement in efficiency in terms of procurement processing time. In a previous
section IEG analyzed a sample of contracts and extracted average processing times
for procurement. The results found suggest savings of 93 days on average per
contract when moving from ICB to NCB. However, given the fact that they are
established on the country level, analysis of method thresholds for cluster of
countries with similar characteristics is key. It would be important to quantify the
trade-off between procurement efficiency and risk especially for groups of countries
with lower ICB thresholds as explanatory factors might not be uncovered with an
aggregate analysis.
A caveat to the above analysis is the known limitations, in terms of quality and
reliability of Form 384 data. Once the Bank’s proposed electronic no objection goes
fully into effect, such limitations will pose fewer problems.
70
Thresholds Analysis – “Prior Review” and “Clearance”
DATA SOURCES
Data Sources – Bank-Wide Prior Review and Clearance Thresholds
Data for the Bank-wide prior review and clearance threshold analysis were compiled
from archived and current OP/BP 11.0 files and corresponding memoranda on the
Bank’s intranet. There is no central repository for this information and it may be
incomplete.
Prior review threshold information from 1997 by clearance level were the earliest
available to IEG (Sanchez 1997). Reference to Bank-wide maximum prior review
thresholds are found in 2008 and 2009. 28 Clearance thresholds were revised in
2002. 29 Further revisions of some clearance thresholds are found in the 2008 prior
review threshold updates. 30 Mandatory review thresholds for RPMs and the OPRC
were further updated in 2009, as were regional and country level revision of prior
review thresholds. Some regions, for example, Africa, adopt further levels of
delegation for clearance (with corresponding thresholds) between procurement
specialists, senior procurement specialists and hub coordinators.
Data Sources—Regional Prior Review and Procurement Method Thresholds
Regional and country level information on previous years’ prior review and method
thresholds is not maintained on a uniform basis across all regions and is readily
available only for a few regions. Historical threshold information was readily
available for the East Asia and Pacific Region (from 2006), the Europe and Central
Asia Region (from 2006), the South Asia Region (from 2008), and for Africa (from
2004). Historical threshold information was not readily available for the Latin
America region or for the Middle East North Africa Region. Current regional
threshold information was taken from the extranet, and as this is not up to date, it
was supplemented by internal sources. 31
Data Sources - Project Level Prior Review Thresholds and Project Level Risk
Project Appraisal Documents and procurement plans were scanned for project level
prior review thresholds for goods and works, project risk, including residual project
risk, procurement risk and residual procurement risk, on a sample basis for 59
projects, based on overall country selectivity criteria. IEG also drew upon additional
project level prior review data in the Middle East and North Africa dashboard and
in the procurement plan execution system (SEPA), adding eight more projects to the
sample. IEG used the residual procurement risk rating when available, or, in its
71
absence, the procurement risk rating. Threshold data sources available to and used
by IEG are summarized below:
Table C.2. Prior Review and Procurement Method Thresholds Available to IEG
Region Prior Review Threshold Information and Source Procurement Method Threshold Information and Source
AFR Selected countries – 2004/2005 (RPM); 2012 (RPM) 2010 (external procurement website)
2012 (RPM)
EAP 2006, 2011(EAP RPO website); 2012 (external 2006, 2011(EAP RPO website); 2012 (external procurement website)
procurement website)
ECA 2006, 2007 (four revisions in 2007), 2010, 2011 (ECA RPO 2006, 2007 (four revisions in 2007), 2010, 2011 (ECA RPO website);
website); 2011 (external procurement website) 2011 (external procurement website)
LCR Brazil, Mexico, Colombia – 2007 (RPM) Brazil, Mexico, Colombia – 2007 (email from LCR RPM), 2012 (external
procurement website)
MNA 2011 (external procurement website) 2011 (external procurement website)
SAR 2008 (SAR RPO website); 2012 (external procurement 2008 (SAR RPO website); 2012 (external procurement website)
website)
Source: IEG.
Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA =
Middle East and North Africa; SAR = South Africa.
Data Sources—Governance Indicators
Governance indicators used in the analysis include Transparency International’s

Corruption Perceptions Index, the Kauffmann and Kraay Governance Indicators,
and the World Bank’s County Policy and Institutional Assessment (CPIA). The
Index scores provide a score, ranking a country on how corrupt its public sector is
perceived to be on a scale of 1 to 10 (with 10 indicating no corruption). 32 Two
Kauffmann and Kraay Governance indicators were used for this analysis:
Government Effectiveness and Control of Corruption. The overall CPIA score and
two sub indicators were used for this analysis: subindicator 13 Quality of Budget
and Financial Management and subindicator 16 Transparency, Accountability and
Corruption in Public Sector. 33
METHODOLOGY
Sample Construction – Correlations with Governance Indicators
Two samples were constructed, one from 2008 and one from 2011, for the
comparison of Bank thresholds with external measures of country risk/country
governance. The nonstandardized nature of the setting of thresholds across regions,
and gaps in historically available information that varied considerably across
regions are caveats to the sample construction.
72
Box C.5. Threshold Analysis – Sample Construction
2011 Sample:
Europe and Cental Asia: Data from thresholds effective as of 2011 were used. Prior review thresholds
are provided by procurement type (goods/works/consultants) and by procurement methods
(ICB/NCB/Shopping).
East Asia and Pacific: Data from thresholds effective as of 2011 were used. Prior review thresholds
(ICB/NCB/Shopping).
South Asia: Data from thresholds effective as of February 2012 were used. Prior review thresholds are
provided by procurement type (goods/works/consultants). Previous thresholds dated from 2008.
Latin America and the Caribbean: Current (2012) thresholds were used. It was assumed that prior
review thresholds were the same as NCB/ICB method thresholds in the absence of separate
information.
Middle East and North Africa: Current prior review thresholds were used (2012). Prior review
thresholds are provided by procurement type (goods/works/consultants) and by procurement
method (ICB/NCB/Shopping).
Africs: 2010 procurement method (ICB/NCB) thresholds were used as 2011 prior review thresholds.
2008 Sample:
Africa: RPM guidelines from 2005 outlining prior review thresholds for projects with three levels of
risk were used for the 2008 sample.
East Asia and Pacific: Data from thresholds effective as of 2006 were used. Prior review thresholds
(ICB/NCB/Shopping).
Europe and Central Asia: Data from thresholds effective as of 2007 were used. Prior review
thresholds are provided by procurement type (goods/works/consultants) and by procurement
methods (ICB/NCB/Shopping).
Latin America and the Caribbean: Threshold information for Brazil, Colombia, and Mexico. Other
countries in the region were not included in the 2008 sample because data were not available.
Middle East and North Africa: Historical threshold data unavailable –not included in the 2008
sample.
South Asia: Data from thresholds effective as of 2008 were used. Prior review thresholds are provided
by procurement type (goods/works/consultants).
Sample Construction Issues:
Diverse practices across Bank regions in terms of how prior review thresholds are set led to the
construction of subsamples by IEG to maintain homogeneity. In the 2011 sample, thresholds from the
Middle East and North Africa are divided into four categories by risk level of implementing agency
(mirroring the format for Bank wide prior review thresholds established in 2009). Africa countries
included in the 2008 sample had three risk levels (high, medium, and low risk), which also mirrors
the format of Bank wide prior review thresholds from 2008. Some countries from South Asia used a
range for their prior review threshold (for example, the 2008 prior review threshold for goods is
“300,000 to 500,000”). IEG divided the thresholds into equivalent risk categories – for example, the
300,000 threshold was interpreted as the threshold for a high risk implementing agency and the
73
500,000 threshold was interpreted as a low risk implementing agency.
For Europe and Central Asia and East Asia and Pacific Regions: Some thresholds were also set in
terms of the first few projects reviewed (for example, Turkey 2011 prior review threshold for NCB
goods is “First 2 & >500k”). In such cases the numerical threshold was considered the effective
threshold and the “First 2” prior review requirement is not reflected in the analysis and correlations.
Correlations between country level prior review thresholds and governance

indicators were undertaken in clusters, reflecting different practices in setting
thresholds across regions. For each coefficient, the vector of country level risk
thresholds was correlated with each vector of country level governance indicators. A
subgroup of countries was constructed to include those that did not differentiate
thresholds by risk level of implementing agency. These countries were pooled with
the overall sample, using three sets of assumptions, in the 2008 subsample, namely,
that risk thresholds in these countries corresponded, in turn, to the high, medium or
low risk implementing agency thresholds. Similarly, for the 2011 subsample, four
series were constructed to accommodate high, substantial, moderate and low risk
implementing agencies. Samples were also divided by category and method: ICB
goods and ICB works, NCB goods, NCB works, to ensure consistent comparisons.
RESULTS – ALL COUNTRIES
• There are no strong correlations. The majority of the correlation coefficients

are positive but weak/moderate in strength. There are a number of negative
correlations, all very small in magnitude.
• Correlations for NCB goods and works are lower than for ICB goods and
works.
• In most cases when comparing the correlation coefficients across the different
levels of risk, the strongest coefficient is on the high risk thresholds, with
lower coefficients on the substantial, moderate or low risk coefficient.
74
Table C.3. Correlations between Country Thresholds and Indices of Country Governance
CPIA Transparency
Subindicator 13. CPIA Subindicator International - Kauffmann and
Quality of Budget 16: Transpar. Corruption Kraay - Kauffmann and
and Financial Account.and Perceptions Regulatory Kraay - Control
CPIA - Overall Management Corrup.in Pub. Sec Index Quality of Corruption
2008
ICB goods
H (68) 0.51 (68) 0.45 (68) 0.32 (67) 0.46 (68) 0.57 (68) 0.25
M (66) 0.41 (66) 0.33 (66) 0.34 (65) 0.44 (66) 0.47 (66) 0.29
L (68) 0.13 (68) 0.09 (68) 0.19 (67) 0.28 (68) 0.18 (68) 0.22
ICB works
H (68) 0.46 (68) 0.41 (68) 0.18 (67) 0.30 (68) 0.43 (68) 0.16
M (66) 0.47 (66) 0.42 (66) 0.22 (65) 0.30 (66) 0.42 (66) 0.17
L (68) 0.44 (68) 0.38 (68) 0.19 (67) 0.30 (68) 0.40 (68) 0.18
2011
ICB goods
H (117) 0.33 (117) 0.32 (117) 0.20 (123) 0.12 (123) 0.28 (123) 0.15
S (110) 0.30 (110) 0.29 (110) 0.20 (116) 0.11 (116) 0.25 (116) 0.14
M (110) 0.16 (110) 0.14 (110) 0.12 (116) 0.04 (116) 0.13 (116) 0.05
L (117) 0.03 (117) 0.03 (117) 0.09 (123) 0.00 (123) 0.05 (123) -0.01
ICB works
H (116) 0.41 (116) 0.36 (116) 0.24 (120) 0.24 (122) 0.40 (122) 0.20
S (109) 0.35 (109) 0.30 (109) 0.23 (113) 0.23 (115) 0.35 (115) 0.21
M (109) 0.29 (109) 0.36 (109) 0.20 (113) 0.18 (115) 0.29 (115) 0.13
L (116) 0.22 (116) 0.32 (116) 0.18 (120) 0.13 (122) 0.23 (122) 0.08
2008
NCB goods
H (51) 0.39 (51) 0.39 (51) 0.25 (51) 0.27 (48) 0.45 (48) 0.14
M (49) 0.08 (49) 0.01 (49) 0.18 (49) 0.18 (46) 0.19 (46) 0.26
L (51) -0.11 (51) -0.19 (51) 0.06 (51) 0.04 (48) -0.02 (48) 0.20
NCB works
H (54) 0.39 (54) 0.36 (54) 0.00 (54) 0.11 (54) 0.35 (54) -0.04
M (52) 0.38 (52) 0.36 (52) 0.01 (52) 0.14 (52) 0.34 (52) 0.04
L (54) 0.30 (54) 0.26 (54) 0.02 (54) 0.15 (54) 0.25 (54) 0.10
2011
NCB goods
H (32) 0.29 (32) 0.09 (32) 0.26 (37) 0.14 (37) 0.44 (37) 0.29
S (28) 0.07 (28) -0.04 (28) 0.12 (33) 0.16 (33) 0.14 (33) 0.06
M (29) -0.1 (29) -0.14 (29) -0.02 (34) -0.08 (34) -0.09 (34) -0.09
L (32) -0.11 (32) -0.14 (32) -0.05 (37) -0.09 (37) -0.12 (37) -0.10
NCB works
H (33) 0.38 (33) 0.26 (33) 0.27 (38) 0.21 (38) 0.51 (38) 0.35
S (28) 0.13 (28) 0.06 (28) 0.14 (35) 0.07 (35) 0.22 (35) 0.12
M (29) 0.03 (29) -0.02 (29) 0.06 (35) 0.00 (35) 0.08 (35) 0.03
L (33) 0.00 (33) -0.07 (33) 0.01 (38) -0.03 (38) 0.00 (38) -0.04
Note: All countries for which risk level data are available. For those countries and years which did not differentiate country risk by
implementing agency, successive correlations have been undertaken assuming, in turn, that such risk levels correspond to each level on
the three- (2008) or four- (2011) point scale for risk used in other countries.
75
RESULTS – SUBSAMPLE OF COUNTRIES (AFRICA AND LATIN AMERICA AND THE CARIBBEAN)
IEG also analyzed results separately for the group of countries, in two regions
especially, for which country threshold data do not differentiate thresholds by risk
level of implementing agency (that is, there is only one threshold by procurement
method/type for the country).
Table C.4. Correlations between Country Thresholds and Governance Indices – Single Risk
Threshold Countries
CPIA
CPIA – Overall subindicator 13. CPIA
Quality of Subindicator
Budget and 16: Transpar., Transparency Kauffmann
Financial Account. and International - and Kraay - Kauffmann and
Management Corrup. in Pub. Corruption Regulatory Kraay - Control
Sec Perceptions Index Quality of Corruption
2008
ICB goods (43) 0.56 (43) 0.51 (43) 0.44 (43) 0.52 (43) 0.61 (43) 0.34
ICB works (43) 0.50 (43) 0.46 (43) 0.27 (42) 0.33 (43) 0.44 (43) 0.23
2011
ICB goods (102) 0.34 (102) 0.33 (102) 0.21 (105) 0.12 (105) 0.28 (105) 0.17
ICB works (101) 0.41 (101) 0.36 (101) 0.26 (103) 0.26 (104) 0.41 (104) 0.22
2008
NCB goods (26) 0.46 (26) 0.42 (26) 0.39 (26) 0.36 (25) 0.52 (25) 0.26
NCB works (29) 0.45 (29) 0.44 (29) 0.02 (29) 0.19 (29) 0.40 (29) 0.10
2011
NCB goods (20) 0.45 (20) 0.20 (20) 0.41 (22) 0.29 (22) 0.73 (22) 0.52
NCB works (22) 0.63 (22) 0.50 (22) 0.46 (24) 0.41 (24) 0.88 (24) 0.64
Note: These countries do not differentiate thresholds by risk level of implementing agency (ie there is only one threshold for the country),
at the country level. Most consist of countries in the Latin America and the Caribbean and Africa Regions.
• Although correlations for this subset of countries was generally higher than
for all countries taken together, only a few are over 0.5.
• As in the overall group, correlations for ICB goods and works are stronger
overall in 2008 than in 2011. For NCB goods and works there is variance – for
some indicators the correlation is higher in 2008, for others it is higher in
2011.
• The indicator that has the strongest correlations by year, procurement
category, and method is the Kauffmann and Kraay Regulatory Quality
Indicator, and may best reflect the implicit criteria in terms of county
governance/capacity that RPMs use in determining prior review thresholds.
These data are displayed in Figure C.1 for the years 2008 and for one governance
indicator, the CPIA score. These offer more granular country level insights and show
76
that while the highest prior review thresholds do tend to be found in countries that
have relatively high CPIA scores, there are many thresholds that are very low
among countries that have high CPIA scores, notably Armenia, Bhutan, Georgia,
Macedonia, and Thailand from the 2008 graph and Chile, Georgia, Macedonia, Peru,
Thailand, and Uruguay in the 2011 graph. Graphs based on other procurement
methods and governance indicators show a similar pattern.
Figure C.1. CPIA Scores and Correlation with Prior Review Thresholds (ICB Goods, 2008)
Source: IEG compilation.
A caveat to the analysis is that conclusions must take into account the quality of the
underlying data, which is heterogeneous by region and not prepared on the same
basis. Moreover, the dates at which revisions have occurred appear to be ad hoc, and
have not been uniform across regions. In some regions, prior review threshold are
also the method threshold (Latin America and the Caribbean, Africa, and some
countries in the Middle East and North Africa), and in others this is not the case but
relates to the challenges of interpreting threshold data. Sample sizes are small
because of the absence of better historical prior review threshold information.
77
Figure C.2. CPIA Scores and Correlation with Prior Review Thresholds
(ICB goods, 2011)
78
Post-Procurement Reviews
Table C.5. Post Procurement Reviews: Project Sample
Contract Value
Country Project ID Project Title PPR FY ($)
Africa
Tanzania P055120 Transport Sector Support 2011/12 64, 458
Tanzania P111155 Zanzibar Urban Service Project 2011/12 694,848
Tanzania P111556 East Africa Public Health Laboratory 2011/12 109,705
Tanzania P114866 Secondary Education Development 2011/12 306,052
Ethiopia P101556 Electricity Access Additional Financing 2012
East Asia and Pacific
Indonesia 7866 ID Empowerment of Urban Areas 2010
Lao PDR P120909 Upland Food Security Improvement 2012 989,981
Philippines P066076 Judicial Reform Support 2011 41,800
Europe and Central Asia
Turkey 7539-TU Fourth Export Finance Intermediary Ln. 2012 12,227,940
Ukraine P095203 Second Export Development Project 2011 4,682,626
Azerbaijan P100582 Real Estate Registration Project 2011
Azerbaijan P083126 Real Estate Cadastre and Registration 2012
Middle East and North Africa
Morocco P110833 Rural Roads II 2011
Morocco 7351 MOR Rural Potable Water and Sanitation 2011
Djibouti Morocco Consolidated Strategic Action Plans to Implement
Tunisia Follow-up Activities Addressing Findings of PPRs and 2009
Yemen, Rep. of IPRs
South Asia
Bangladesh Consolidated PPR/IPR Summary contained in FY2011
Report covering 34 projects and 174 contracts 2011 9.7m
India Consolidated Report on PPRs covering 24 projects and
1596 contracts 2012 113.84m
Note: In some cases (Bangladesh, India, and Yemen) consolidated reports were compiled by the region, in others there were no
consolidated reports and the analysis is based on individual PPR completions. PPR = Post-Procurement Review.
79
APPENDIX 3D
POST PROCUREMENT REVIEWS
Table C.6. Instances of Noncompliance in PPR/IPR Reports on Yemen (FY09–10)

Coverage: Projects PPR IPR
FY 09: FY09:
Public Sector Governance (4 contracts), Health sector (17 contracts)
1 Project in each of the following Water and Sanitation (11 contracts), Transport (4
sectors contracts), FY10:
Urban Development (2 contracts), Social Protection (11), Agriculture and Rural Development
Rural Development sector ( 2 projects: 8 contracts) sector (2 projects; 20 contracts)
FY10: Public Sector Governance sector (22
Education (2 contracts), contracts)
Transport (5 contracts),
Environment (8 contracts),
Private Sector (2 contracts)
Social Protection (16 contracts),
Water and Sanitation sector (2 projects; 16 contracts)
Total Number of contracts: 89
Total Number of contracts: 59

SUMMARY OF PROBLEMATIC FINDINGS
A: Relating to the Borrower/Implementing Agency
PPRs
Inaccurate cost estimates (referential sources of unit prices not reliable);
Recurrent overwriting of award decisions by the country’s High Tender Board (HTB) after the Bank NOs were granted;
Weak procurement and contract administration capacity in project implementing units (exacerbated by high turnover of procurement staff).
IPRs
(i) Generalized average procurement capacity and poor (xvii) A case of non-compliance with GL’s requirement to specify
procurement filing systems and contract management that in many liquidated damages for delayed completion/delivery; (xviii) a case of award
cases led to extensive time overruns; made for a lower-priced option of the lowest evaluated substantially
(ii) insufficiently comprehensive ToRs, Schedule of Requirements responsive bidder, not in accordance with the BER and the award
and technical specifications; recommendation accorded with the Bank;
(iii) unauthorized restriction in bidding documents for arithmetical (xix)A case where items re-advertised after cancellation due to change in
errors exceeding 3% of bid price resulted in disqualification of requirements on the bids’ submission deadline, differed from those in the
lowest priced/substantially responsive bidders; corresponding bidding documents; a case where currency of payment
(iv) unjustified use of the SSS method and unauthorized use of specified in the contract was inconsistent with the contract price currency
CQS and SSS in cases where the estimated contract value specified by the successful bidder in its bid;
exceeded the thresholds established in the LA for the use of these (xx) A case where procured items were long stored in a government
methods; location instead of distributed to the intended recipients;
(v) lack of procurement-related documentation on file (e.g., proof of (xxi) A case where, during physical inspection, procured items could not
advertisement in local newspapers, financial proposals and basis be found at the intended destination (and deficiencies noted with regard to
for calculation of consultants’ fees and reimbursable expenses, the location of other items procured which caused them to rust;
copies of performance securities when their furnishing was required (xxiii) In the particular case of one project implementing unit, prolonged
in the contracts; lack of procurement staff and new management unaware of prior
(vi) changes in the substance of bid (i.e., performance security, procurement activity; (xxii) poor to non-existent “after delivery” control of
delivery period, payment terms, etc.) occurred after the bids’ receipt procured assets; (xxiii) expired Bank guarantees not returned to bidders;
deadline; (xxiv) Required geographical spread of short-listed firms was not
(vii) unguaranteed advance payments over 10%; complied;
(viii) amount of performance security not increased as required by (xxv) Notification of award predated the Bank’s NO;
the performance security provision consequently with an increase of (xxvi) Unauthorized suppression of provision on qualification of the bidder
the contract price for additional works; as required alleging objective to develop local industry and due to low
(ix) performance security improperly obtained in Yemeni Rials average cost of sub-projects;
instead of the currency of the contract (US$) or a freely convertible (xxvii) A case where, though the bidding documents were accorded NO
currency; by the Bank, payment terms were unacceptably set differently for “goods
(x) required publication of evaluation results and contract award supplied from the purchaser’s country” from those for “goods supplied from
was not made; (xi) fraud and corruption and audit clauses in abroad” (allegedly to avoid the use of LC in the case of local suppliers by
bidding/contract documents were not in compliance with the agreed increasing slightly the advance payment);
legal provisions; (xxviii) Obligations not originally included in the RFQ (that is, furnishing of
(xii) modifications for aggregate increase by more than 15% of the performance security in the amount of 10% of the contract price), were
original contract amount made without seeking the Bank’s NO; imposed to successful suppliers at the time the POs/contracts were
80
(xiii) physical inspection revealed that payments were made against issued;
non-executed items of works and others with poor workmanship or (xxix) Opening of proposals occurred a week after the deadline for receipt
not in accordance with the technical specifications; of proposals;
(xiv) non-compliance with advertisement requirements and (xxx) Required prior reviews by the Bank not sought by Project
minimum 30-day period allowed for preparation/ submission of bids; Management Unit staff also implementing another Bank-financed project
(xv) a case of unauthorized use of the “shopping” method for works alleging not being aware that project-specific provisions applied;
exceeding the threshold established in the LA; (xxxi) Details relating to “liquidated damages clause” not properly specified
(xvi) a case where requirement to reduce the performance (four kinds of deficiencies found); delayed delivery of goods or completion
guarantee after delivery/acceptance of goods was not complied and of works or services;
2% of the contract price was improperly retained as cash deposit for (xxxii) A case where liquidated damages were not deducted from the
the duration of the warranty period; contract price payable on account of penalties despite a liquidated
damages provision in the contract;
(xxxiii) A case where changes to the scope of works to reduce the
contract amount were made after the Bank had accorded NO; and
(xxxiv) A case where extension of bid validity was requested from a limited
number of bidders instead of from all bidders as required in the GLs.
Source: Strategic Action Plan to Implement Follow-Up Activities Addressing Findings of FY09-FY10 PPRs and IPRs in the Middle East
and North Africa Region.
81
Analysis of P-RAMS
DATA DESCRIPTION
The data provided for analysis covers the period since the implementation of
Procurement Risk Assessment Management System (P-RAMS) in 2010 and ending in
FY12 and consists of 794 eligible projects, of which 607 had at least one P-RAMS.
Including multiple P-RAMS sequences, the number of eligible and actual P-RAMS
rises to 917.
COMPLIANCE AND COMPLETION
For the purpose of this analysis a completed P-RAMS is one that provides
information about both project and residual risk. There were 61 projects that listed
their P-RAMS status as “in progress” but did not provide any information about
project risk; therefore these observations were not counted as completed.
Compliance rates were calculated by project, looking at the ratio of projects with at
least one P-RAMS completed against the number of eligible projects (investment
lending projects that were in the Project Concept Note phase on or after July 19,
2010).
RISK MANAGEMENT AND DYNAMIC RISK MANAGEMENT
Overall procurement risk is determined after the analysis of procurement risk by

taking into account the 11 risk factors contained in the Procurement Risk
Assessment Questionnaire—as well as other possible project risk factors (political
climate, capacity, and so forth). After project risk is identified, the person
completing the P-RAMS designs a risk mitigation plan with steps to address the
risks defined in the previous stage, and enters each mitigation measure into the
Mitigation Measures Action Plan in the template. The risk that may remain after the
risk mitigation measures are carried out is called residual risk.
First and Last P-RAMS: “Bookend Analysis”
To analyze the risk management component of the P-RAMS, the sample of P-RAMS
was tabulated based on initial project procurement risk. Of the 607 P-RAMS
completed, 66 percent were categorized as having either high or substantial project
procurement risk. Only 7 percent of P-RAMS exercises reported low initial project
procurement risk. Risk was examined at the start and end of the available P-RAMs
sequences for the project, comparing project risk to residual risk. For projects with
82
two or more P-RAMS exercises the project risk for the first P-RAMS was compared
to the residual risk for the last P-RAMS, that is, using a “bookends” approach.
Table C.7. Residual Risk by Project Risk for All Projects with P-RAMS
Three Four or Five
One Sequence Two Sequences Sequences (N = Sequences
PROJECT RISK (N = 393) (N = 65) 13) (N = 11)
ASSESSED RISK : High Freq. % Freq. % Freq. % Freq. %
134 21 2 4
RESIDUAL RISK High 18 13% 5 24% 0 0% 0 0%
Substantial 91 68% 14 67% 2 100% 4 100%
Moderate 25 19% 2 10% 0 0% 0 0%
Low 0 0% 0 0% 0 0% 0 0%
ASSESSED RISK: Substantial Freq. % Freq. % Freq. % Freq. %

124 23 7 3
Substantial 34 27% 5 22% 3 43% 3 100%
Moderate 86 69% 18 78% 4 57% 0 0%
Low 3 2% 0 0% 0 0% 0 0%
ASSESSED RISK: Moderate Freq. % Freq. % Freq. % Freq. %

116 16 3 2
Substantial 1 1% 1 6% 0 0% 0 0%
Moderate 87 75% 9 56% 2 67% 2 100%
Low 28 24% 6 38% 1 33% 0 0%
ASSESSED RISK: Low Freq. % Freq. % Freq. % Freq. %

19 5 1 2
Substantial 0 0% 1 20% 0 0% 0 0%
Moderate 1 5% 0 0% 0 0% 0 0%
Low 18 95% 4 80% 1 100% 2 100%
Source: IEG calculations from P-RAMS database provided by OPSOR.
Note: These figures are for the “bookends” of risk, the initial risk rating for the first P-RAMS and the residual risk rating for the last P-
RAMS. (n = 482).
The main findings are:
• The exercise tends to show greater difference in risk levels (as measured by the
difference in initial and residual risk) for projects with high and substantial
initial project procurement risk. Seventy-eight percent of projects (249 out of
318) with an initial assessed risk of high or substantial had a difference in
residual risk rating of at least one risk category.
83
• Most projects with moderate risk tended to not see a difference in project
residual risk, although 25 percent of projects (35 out of 137) with an initial
risk rating of moderate had a residual risk rating of low.
• Projects with a low initial risk rating remained with low residual risk rating
93 percent of the time (though two saw an increased residual risk rating).
• The data does not show an increased decline in risk over multiple P-RAMS.
For example, zero projects with four or five P-RAMS that had an initial risk
level of high saw a reduction to moderate or low residual risk. This finding is
replicated for projects that have an assessed risk level of substantial as well.
Examination of Each P-RAMs in a Series: “Sequence Analysis”
The last unexpected finding raises some questions as to the value added of the
dynamic P-RAMS exercise. A closer examination was therefore undertaken looking
at the detailed sequencing of risk from the first P-RAMS to the last P-RAMS for each
given project where two or more sequences existed. Results are indicated in Table
C.8.
Table C.8. P-RAMS Risk Ratings for a Sample of Projects with Two or More P-RAMS
1st P-RAMS 2nd P-RAMS 3rd P-RAMS 4th P-RAMS
ProjectID ProjectRisk ResidualRisk ProjectRisk ResidualRisk ProjectRisk ResidualRisk ProjectRisk ResidualRisk
P094360 High Substantial High Substantial High Substantial High Substantial
P123923 High Substantial High Substantial High Substantial Substantial Moderate
P100530 Substantial Moderate Substantial Moderate Substantial Moderate Substantial Moderate
P120349 Moderate Moderate High High Moderate Moderate Moderate Moderate
P122319 Low Low Low Low Low Low Low Low
P125689 High Substantial High Substantial High Substantial
P123201 Substantial Moderate Substantial Substantial Substantial Substantial
P126537 Moderate Low Moderate Low Substantial Moderate
P122492 High Substantial High Substantial
P122008 Moderate Moderate Moderate Moderate
Source: IEG tabulation from P-RAMS database provided by OPSOR.
Note: Selected by IEG to show the variation in patterns between project and residual risk for projects with multiple sequences.
• Only one project showed the expected pattern of consistently reduced

residual risk with additional P-RAMS.
• The most common pattern seen was the same project risk rating through
every P-RAMS in the sequence as well as the same residual risk rating
through every P-RAMS.
• The data do not show an increase in reduction of risk over multiple P-RAMS.
For example, zero projects with four or five P-RAMS that had an initial risk
level of high or substantial saw a reduction to moderate or low residual risk.
84
Of the 65 projects that had two P-RAMS, 21 showed no difference between initial
project procurement risk and residual risk in any project phase. Four of these 21
projects with no change in risk were categorized as low risk projects; it can be
assumed that the risk mitigation plan could not have changed the residual risk
rating. Of the remaining 44 projects that did see some change in risk level between
initial project procurement risk and residual risk, 39 projects had identical P-RAMS
risk ratings in both sequences with the residual risk always being noted as lower
than the project risk (for example, an initial risk rating of high and residual rating of
substantial, for each project phase).
At first glance it seems counterintuitive that a project would have a residual risk
rating of moderate in one sequence followed by a project risk rating of substantial
(more than moderate) in the next sequence. One possible explanation for this is that
the risk mitigation plan from the previous phase was not completely implemented
and the risk was not reduced as planned. This could also reflect too frequent
changes in P-RAMS sequences. Another possibility is that changes in risk might be
expected to occur over the course of implementation, and projects with P-RAMS
have not been under implementation for any length of time, due to the newness of
the tool (see Table C.9, which shows that only a few projects are under supervision).
A third possibility is the inertia in terms of changing entries, and perhaps limited
scrutiny of P-RAMS sequences. These potential explanations for the two P-RAMS
sequence projects could apply to the three-P-RAMS projects, too.
Table C.9. Sequence and Phase Tabulations

Sequence 1 2 3 4 5 Total
Phase Freq % Freq % Freq % Freq % Freq % Freq %
Project Concept Note 215 40 0 0 0 0 0 0 0 0 215 32
Quality Enhancement 67 12 42 47 0 0 0 0 0 0 109 16
Review
Decision Meeting 97 18 20 22 13 52 0 0 0 0 130 19
Negotiations 60 11 17 19 8 32 10 100 0 0 95 14
Supervision 104 19 10 11 4 16 0 0 1 1 119 18
Total 543 81 89 13 25 4 10 1 1 0 668
Source: IEG calculations from P-RAMS database
One issue with analyzing the “spread” of risk rating data in the P-RAMS is that all
of the measures are subjective, assigned by the procurement specialist or task team
leader on the project. A complementary analysis looking at whether the risk
mitigation measures were carried out and effective would be useful in addition to
examining subjective assessments of risk provided in the P-RAMS. Table C.9 shows
the distribution of P-RAMS through the project cycle, as of FY12.
85
CONCLUSION
Overall it remains unclear whether the P-RAMS tool is performing as was intended.
Though the program has yet to achieve the desired 100 percent compliance, it seems
that goal will be achieved in the very near future. An analysis of the most important
element of the P-RAMS – the risk measurement and mitigation shows mixed results.
If residual risk over the cycle of the project is expected to decline, the analysis shows
it has yet to occur. If however, risk is basically static and the goal of the P-RAMS is
to mitigate risks where possible then the exercise is more successful (see the
“bookend” versus “sequence” analysis in preceding pages). The usefulness of P-
RAMS ultimately will depend on how much its findings are used for risk analysis.
Further complementary qualitative analysis is required before a more complete
conclusion can be drawn.
Box C.6. INT Guidance to Staff on Procurement Risk
INT’s draft paper on potential red flag issues, mostly alluding to procurement, offers a number of
suggestions to operations staff:
86
 Establish an integrity due diligence checklist for implementing agencies: National
procurement systems could be strengthened by adding integrity checklists as part of Bid
Evaluation Reports. OPCS could set standards, and the regions could offer training to local
implementing agency staff in operating the checklists.
 Define Bank Due Diligence in Prior Reviews: Especially as management is now suggesting
that transaction-level fraud and corruption due diligence be de-emphasized, it becomes
important that the general principles and what to look for in the prior review of the most
risky transactions be more clearly defined in a revised OP/BP11.00.
 Incorporate a Red Flag Checklist into Post-Procurement Reviews: The suggestion is to amend
the 2002 Office Memorandum which establishes the basis for handling integrity risks in Post-
Procurement Reviews to change the basis for the review from contract size to contract risk
rating; provide guidance on detection of high-risk transactions (for example, single bidder,
close to threshold amounts, existence of procurement complaint); attaching a red flag
checklist; ensuring that where consultants are used to conduct Post-Procurement Reviews
they are trained in red flag methodology.
 Disclose Past Performance in PADs and ICRs: The suggestion is that particularly in the case
of successor projects implemented by the same project implementing unit, the procurement
performance of contracted companies be reflected in the ICRs of one project and the PAD of
the ensuing project. A scorecard could be developed containing such items as: actual cost
compared to cost estimates at bidding; percentage of bids with only one or two bidders (not
clear if this refers to the bidders or the project implementing unit); number of transactions
with overruns above 6 months; number transactions with procurement complaints.
 Avoid Conflicts of Interest in Recruiting and Deploying STCs: Project staff should avoid
giving unchecked procurement discretion to STCs who may or may not be in a potential COI
situation.
87
Appendix D. Procurement Tracking Systems,
Time Analysis, and Africa Region Review
Systems
BANK-WIDE INFORMATION – EARLIER VERSIONS OF FORM 384
Earlier versions of the Form 384 template indicate a stronger emphasis on collecting
procurement process data, including a number of dates regarding the bidding
process as well as contract co-financing information. Before 2001 the Operations
Procurement – Form 384 selection screen provided data entry tabs such as Activity
Plan/Monitor, Project Plan/Monitor, Evaluation Report, Contract Evaluation, Form 384
and Project Information. A big part of the data populating the form was captured
from information entered via these tabs earlier in the procurement process. The
actual Form 384 data entry was divided into two separate sections for goods and
works and for consultant services, each providing sections for entry of basic data,
supplier/firm data and currency data. A number of contract dates were collected
such as bid opening/proposal submission, evaluation no objection, contract
received, contract signature and Form 384 date. In addition, fields for delivery
completion target/actual dates, completion percentage, exchange rate and co-
financing information were provided. Users were asked to enter the country codes
of the next three lowest bidders/firms considered for contract award in addition to
supplier/firm data of the winning bid.
The 2001 revision of Form 384 came embedded in a general overhaul of the
procurement information system aimed at reducing the amount of data entry in the
system. Operations Procurement – Form 384 selections screens were trimmed down to
Project Information, Form 384, Post Review and Activity Planning, with some of the tabs
capturing extended information to pick up data entered elsewhere prior to the
system revision. Cofinancing data were moved to their own tab and the following
fields removed: delivery completion target and actual dates, completion percentage,
completion as of date and one of three country codes to identify next lowest bidders.
While aiming for simplification of data entry, the incentive for procurement data
collection seemed to have remained a priority in 2001 as additional fields were
opened for data entry: implementing agency, procurement method, borrower bid
reference number, bid opening date, domestic preference indicator, award affected
by preference indicator, bid evaluation report no objection date, prequalification of
bidders required indicator, fixed price contract indicator, two stage bidding
indicator, bid number, bidding description and contract description. In the
89
consultant services section no fields were removed from data entry but new fields
were added such as the check box for misprocurement and allow for contract signature
date before no objection date. Overall it seems that while a few key variables have been
dropped from bank wide data collection in the 2001 revision, other indicators have
been added reflecting a possible shift in specific monitoring needs but not a general
reduction of data collection.
This seems to have changed significantly with the 2009 revision when Form 384
became a single data entry screen, delinked from procurement bids/activities
information and simplified with a broad stroke including the removal of all but key
data entry fields necessary for basic procurement reporting. Form 384 in its latest
version is divided into four sections: basic data, contract price, supplier
firm/individual details, and contract comments. No objection date and contract
signature date are the only contract dates collected in addition to basic information
like procurement method and sector allocation.
It is no longer necessary to enter the actual currency payment terms with the
contract information. While the form no longer requires bidding activity other than
the winning firm or individual, the most noticeable element added is the integrated
anti-money laundering check which must be executed in order to complete Form
384. On entry of data the new Form 384 provides a list of suppliers that have already
been cleared for the respective project and, if the supplier is not listed, the anti-
money laundering check process is activated. Discussions with Bank’s procurement
personal suggest that the 2009 revisions and simplifications of the Form 384 were in
line with a general move away from extensive control on the contract accounting
side within the Bank. However, recent Bank efforts indicate an increased interest to
put in place an integrated bank-wide procurement information system and invest in
more comprehensive data collection.
REGIONAL PROCUREMENT TRACKING SYSTEMS – NATURE OF REPORTS PRODUCED
Latin America and the Caribbean: SEPA—System Reports and Borrower Interaction 34
As a management information system SEPA allows users to create reports, approve

procurement plans and inform the parties concerned about the status of execution
thereof. The system provides a platform for interaction between the task team leader
and the implementing agency, while at the same time providing an audit trail of the
process. The borrower enters and updates the procurement plan according to the
loan or grant agreement, the Bank is then prompted by a system-generated
notification to furnish its no-objection or provide comments. 35 Once approved, the
procurement plan is automatically updated on the SEPA website.
90
SEPA helps both the Bank and its borrowers to ensure that detailed procurement
plans are prepared, and to identify variations between planned and actual
procurement activities. Upload of information into SEPA usually starts with the first
procurement plan of a project that covers the first 18 months. 36 According to the
Bank’s guidelines, procurement plans are to be updated by the Borrower every year.
Contracts previously awarded and those to be procured in the following twelve
months are required to be included (World Bank 2011e). SEPA staff indicated that
most executing agencies update their procurement plan in SEPA on average once
per month. Procurement plans, updates and modifications are subject to the Bank’s
prior review and no objection before they are posted and have to be published on
the Bank’s external website. Procurement plans approved in SEPA are
automatically published on the SEPA public website. They have to, however, be
manually migrated to show up in the public documents on the Bank’s external
website (http://www.worldbank.org), which SEPA staff do on a regular basis.
Data in SEPA are populated by the borrower. Some date fields, such as estimated
dates for procurement steps, and estimated amounts, are mandatory for
procurement plan approval while others, including actual dates and amounts, hinge
on the readiness of the borrower to enter additional information. Actual dates of
contract execution do not require new approval by the Bank and can be entered by
the borrower at any time.
Africa: The Procurement Cycle Tracking System—PROCYS
The principal function of PROCYS has been that of a management information

system that measures responsiveness of participants in the procurement process,
and monitors aggregate trends in countries and sectors to identify bottlenecks.
Unlike SEPA it channels communications including their attachments through the
system recording dates automatically. It is now partially integrated with other Bank
systems, in the sense that the filing of no objection notices and related documents
into the Bank system is also automated. It generates several managerial reports,
including monthly no-objections (by country/by stage/average time taken);
quarterly reports on RPM/OPRC No-Objections, status reports on all RPM/OPRC
cases, and reports on average time taken per project. Duration of process time is
tracked starting from initiation, i.e. when the first request arrives by email to
PROCYS, to approval, that is, issuance of no objection.
Box D.1. PROCYS Workflow

The process is initiated when the borrower sends a request for procurement review to PROCYS via
email; the Bank task team receives an email notification with a link to the request in the system. The
task team leader/team reviews and where necessary submits it to the appropriate level of
91
procurement specialist. The comments or no objection are then communicated to the borrower by the
task team. Both Bank staff and clients can retrieve project information via the web-based system.
PROCYS stores communications with a date and time stamp. However, the unique transaction
request number cannot be linked to contract identifiers used in other Bank systems tracking contract
data, such as the Bank-wide Form-384. At the time a no-objection is issued, for any stage of the
procurement process, all communications including attachments are filed under the project number.
Source: IEG.
Middle East and North Africa: The Procurement Portfolio Dashboard
The procurement portfolio dashboard is available to Bank staff for download from
the region’s intranet site in the form of Excel spreadsheets, available from October
2008. Database macros allow users to produce management reports filtered for
information for the entire region, for a selected country department, country, or
procurement staff. 37 A core element of the dashboard is information on project-level
procurement activities planned and executed by procurement specialists, such as
tracking compliance with scheduled procurement related activities, including
procurement post reviews, procurement plan updates, P-RAMS status and
complaints. Project-level data further include committed, disbursed, and cancelled
project amounts, procurement method and prior review thresholds, procurement risk
ratings, and task team leader and procurement specialist names. Data related to
specific contracts or procurement stages are not collected.
A main element of the dashboard is information on project level procurement

activities planned and executed by procurement specialists, such as tracking
compliance with scheduled procurement related activities, including procurement
post reviews, procurement plan updates, P-RAMS status, and complaints. Project-
level data further include committed, disbursed and cancelled project amounts,
procurement method and prior review thresholds, procurement risk ratings, task
team leader and procurement specialist names, among others. Data related to specific
contracts or procurement stages are not collected. However, the summary reports on
the dashboard are not presented in a way easily accessible for interested users, as
they lack explanatory information such as indication of the time period covered and
general legend.
SUMMARY
The preceding brief summary illustrates that at present from the management
information system point of view it is not possible for central procurement staff to
track procurement processes Bank-wide, in terms of parameters relating to efficiency
of implementation. Only Form 384 is uniform Bank-wide, but it provides a very
92
limited amount of information, on a subset of contracts, and Bank and borrower
contributions are not separated. Moreover its use as information of contract level
disbursement has been reduced. Form 384 does not provide a platform that is
accessible to borrowers or any persons external to the Bank, and its internal use is
not management friendly as it does not process any automated reports.
Other regional systems each have their aims and objectives, and reflecting these,
advantages and disadvantages. Although the procurement anchor is aware of these
difficulties, it has apparently been difficult to develop a system that is integrated
with the Bank’s other platforms.
Table D.1. Comparison of Procurement Monitoring Systems by Region

LCR / ECA: Procurement AFRICA: Procurement
MENA: Procurement
Bank-wide: Form 384 Plan Execution System Cycle Tracking System
Portfolio Dashboard
SEPA PROCYS
Database as clearinghouse
Borrower enters and updates for communications between
TTL or certified project Updated monthly by
Unit of Entry Procurement Plan for the the borrower and the bank on
team member staff in headquarters
project procurement processes,
initiated via email by borrower
Centralize information
based on information
Individual contracts subject Monitor Procurement Plan Individual contracts subject to
in Bank systems and
to prior review execution prior review
information on projects
by team
Approve Procurement Plan
Contract award information Track response times of
through the system; provide
on contracts above the prior Clients, Task Team,
public access to basic
review threshold Procurement Specialist
information
GENERAL PROJECT LEVEL DATA
Country yes yes yes yes
Loan / Credit No yes yes yes
Project ID yes yes yes yes
Project Status (Pipeline
yes yes
/Active / Closed)
Loan Amount/ Credit/ Grants
Total Committed AND Local counterpart amount Total committed and
(IBRD + Grants) recorded; including origin Grants
currency and exchange rate
Disbursed in FY and
Disbursed
Project Total
Amount Total undisbursed
Undisb.
balance
Cancelled Total cancelled
Value of PPR
Aggregated Contract Value
Contracts Reviewed
Loan Effective Date PCD Review Date
Project Original AND Actual
Appraisal Date
Key Project Dates Closing Date
Approval, Effective
Project actual closing date
and Closing Date
93
MENA: Procurement
Portfolio Dashboard
SEPA PROCYS
Estimated time per
P-RAMS Date
procurement stage
at least one and up to three
TTL TTL name per project TTL name per Project
TTL indicated
Primary and
Procurement Specialist up to three PS indicated PS name per project
Secondary PS
Agreement Type (IBRD,
yes
IDA, other)
Lending Instrument (DPL,
yes
P4R, IL, other)
Major Sector and Sector General sector code per
Sector Information Major Sector and Sub-Sector
Board project
Number of
Borrower, Executing Agency,
Implementing Agencies Implementing
Sub-Executing Agency
Agencies
PROCUREMENT PROJECT LEVEL DATA
Amounts and Location
Project Level Prior Review Threshold amounts set by the of Prior Review
Threshold Bank AND by the Borrower Threshold (Proc Plan
or Legal Agreement)
Amounts and Location
Project Level Procurement Threshold amounts for use of Review Threshold
Method Threshold with each procurement method (Proc Plan or Legal
Agreement)
Number for
Procurement Plan entered and Procurement Plans
updated in SEPA by executing status current,
agency; upon approval outdated and not
Procurement Plan Status
automatic publication on SEPA required, date of
external website; historic current and due date
procurement plans available next Procurement
Plan
Proposed ISR Rating,
Last ISR Rating from
Project Procurement Risk
SAP, Procurement
Rating
Risk Rating, # Projects
at Risk
#P-RAMS in progress,
archived and required;
Project P-RAMS Status Share of projects per
risk categories H, S,
M, L and “no info”
Procurement Flag yes
all contracts in Procurement only contracts marked as all contracts in
Contracts covered only prior review contracts
Plan prior review Procurement Plan
PPR completion actual
vs planned by country
and by person,
Project Level Procurement
indicates if PPR required for PPR/IPR action plan
Post Reviews (PPR)
project implementation,
(scheduled / completed)
Number of Post
Reviews and
Supervision Missions
94
MENA: Procurement
Portfolio Dashboard
SEPA PROCYS
PROCUREMENT CONTRACT LEVEL DATA
Contract Description yes yes yes yes
Contract ID assigned by
each contract transaction is
Contract ID SAP and Borrower country Contract Reference Number
assigned a request number
reference
Estimated contract value and
Total contract amount and
Individual Contract Value actual contract amount and Estimated contract value
amount used
payments
Procurement Method (ICB,
NCB; for consulting QCBS, yes yes yes
QBS,FBS,LCS,CQS, SSS)
Procurement Category
(Goods, Civil Works, Cons. yes yes yes
Services)
Clearance Level (PS, RPM,
yes
OPRC)
Bidders Nr of bids received and
Awarded: name, country of
firms considered, country of
nationality, bid amount/total
nationality of winning bid,
amount paid; Shortlisted:
bidder eligibility status
name, country of nationality &
(Anti-Money Laundering,
amount offer
list of debarred firms, etc.)
Key Dates and Elapsed Borrower issue of specific
Contract award date Date of first action
Times procurement notice
Borrower first submission to
Signature of contract Bank of draft bid (preQ) Date of most recent action
documents*
Borrower delay (Average time
Bank final no objection to draft
Form-384 sent date taken by borrower beyond
bid (PreQ) documents*
normal processing time)
Task team time (average time
Borrower issue of Bid (PreQ)
taken by task team to review
documents*
steps )
Procurement time (average
Bid Opening* time taken by PS/RPM to
review steps)
Borrower submission to Bank Average Response Time for
of Bid evaluation report* No Objection
Bank No-Objection to Bid Number of Stages Given No
Evaluation Report* Objection
idle time (duration between
today's date - date of last
Signature of contract* action, usually defined if
pending with borrower
between two stages)
Contract Completion* Total time client
*stages vary by procurement Total time with TTL / PS /
method and category Client
Estimated and actual contract Average time with TTL / PS /
dates can be entered Client
Total days of process between # iterations (Number
Bid Opening and Signing of interactions between TTL and
Contract borrower / TTL and PS/RPM)
95
MENA: Procurement
Portfolio Dashboard
SEPA PROCYS
Management type reports Procurement Plan monitoring Reports for the entire
Average Response Time for
on contract level for selected region, by country, by
No Objected Cases
project CMU, by person (PS)
Number and amount of ex post Number and Amounts
contracts initiated in specific Number of stages given No of supervision, lending
time period by category for Objection closed and total
selected project projects
Report of Management
Processes with elapsed times Response times by sector for Post Review Schedule
between stages on contract selected country (Planned and Actual)
level for selected country
Portfolio Review Report with
aggregate amounts for Cumulative PPR
Support Effort Hours per case
processing stage by project, Progress
country and province
Complaints: Number
Number of contracts by
of minimum,
category, method and stage Monthly reports on No-
maximum, average
(expected, in process, Objections (by country/by
days pending for
completed, canceled) for stage/average time taken)
status pending and
selected country and project
closed
Quarterly reports on
RPM/OPRC No-Objections
General Procurement Plan
report indicating total number PROCYS also provides an
of projects and procurement external website for clients for
plans by country for selected real-time status of all their
project status (in preparation, requests sent to the Bank
active, closed) and year
Public reports Publication of Procurement
Plans on SEPA public website
Aggregated Reports of
contract values by
procurement method/category
for selected values (country,
project, sector, province)
96
Procurement Efficiency at the World Bank – Elapsed Time Analysis
DATA COLLECTION AND SAMPLE CHARACTERISTICS
Information on processing times of procurement contracts in the Bank is not

available in any comprehensive centralized format that can be used by Bank
management to track the procurement process for all regions or contracts. Data are
highly decentralized and typically maintained by individual country procurement
offices. Although procurement checklists are required to be maintained, which
would serve as summary sheets, and could deliver dates on processing steps, these
are often not available. IEG therefore conducted its own data collection for this
analysis, linked to its field visits, and based on a template tracing critical dates in the
“no-objection” process. 38 Detailed contract processing data at the country/contract
level are in some cases maintained by the procurement coordinator’s offices and
sometimes by sector units. Each regional office has its own format in terms of
specific fields of information tracked, date conventions and, clearly, currency units.
To construct a representative stratified sample IEG suggested a specific list of prior

review contracts to each field visit country office, by procurement method and
category, by size, and across a sample of fiscal years from 2007 to the present.
However, information received is concentrated in 2010–12 with the year 2011
representing 32 percent of all observations in the sample. The response rate from
country offices was highly variable in quantity and quality, ranging from 107
contracts received from Azerbaijan to four from Tanzania.
Often, not all date fields were filled. Of 502 contracts received, only 201 provided the
requested information on all date fields from Issue of the Specific Procurement
Notice to Contract Signature. 39 By far the most frequently reported dates were the
no objection to the bid evaluation report and the contract signature which are also
provided to the Bank-wide Form 384. In regions where a central data repository for
the procurement process has not been established, staff in the field indicated that
dates for procurement steps prior to the no objection to the bid evaluation report
have to be extracted from procurement documents and communications with
country clients. It appears that information on all steps in the procurement process is
not readily available in all countries. Clearly this affected the scope and quality of
analysis that could be undertaken, as well as reliability of results obtained and will
have to be a caveat to their interpretation.
The request for contract sample data sent to the field visit countries was separated
by key processing dates for Goods and Works (ICB, ICB with preQ; NCB, NCB with
Pre-Q), Goods and Works (ICB/NCB Two Stage) and large values of consultant
97
services (quality- and cost-based selection, quality-based selection, Fixed Budget,
least cost selection, consultant qualifications section, single source selection). For
purposes of this analysis the respective dates for each procurement step were
merged across ICB/NCB, ICB/NCB two-stage and consultant services selection
allowing for global data set as well as sub set analysis. Due to a low response rate for
information on contracts involving ICB/NCB two-stage and pre-qualification
(evaluation reports), data provided for these groups was integrated with the overall
ICB/NCB data. For all contracts dates for final Bid Evaluation Reports were used to
construct time intervals. 40 Contract values were converted to dollars using monthly
average exchange rates. 41 The fiscal year of the contract was determined based on
the date of the Bank’s No Objection to the Bid Evaluation Report.
MEASURES OF AVERAGE ELAPSED TIMES
IEG’s analysis focuses primarily on two overall elapsed times: borrower issue of
specific procurement notice to contract signature and first submission to the Bank of
draft bid (preQ) documents to the Bank’s no objection to the bid evaluation report.
IEG also attempts to selectively review two intermediate steps in the procurement
process, to the extent that data permit: first submission to the Bank of draft bid
(preQ) documents to the Bank’s final no objection to the draft bid (PreQ) documents
and borrower submission to bank of bid evaluation report to the Bank’s no objection
to the bid evaluation report. Because of data limitations, results for different
intervals may refer to different subsets in the data set.
SELECT FINDINGS – AVERAGE TIME
The average days between the borrower’s first submission of the draft bidding
documents to the Bank’s final no objection to bid evaluation report of the contract
was 253 days. However, the variable displayed a high dispersion from the average,
with a standard deviation of 160. Though 50 percent of the contracts included in this
analysis go from the submission of the bidding documents to the No Objection to
the bid evaluation report in less than 208 days, IEG’s analysis indicates the existence
of contracts with significant longer duration, for two contracts even more than 900
days, driving up average processing times. Figure D.1 shows the distribution of data
points for borrower issue of specific procurement notice to contract signature and first
submission to the Bank of draft bid (preQ) documents to the Bank’s no objection to the bid
evaluation report compared to the same intervals without those contracts that make
for the highest five percent of elapsed times.
98
Figure D.1. Elapsed Time Distributions of Procurement Intervals
All Observations
0 500 1,000 1,500 2,000
Excluding upper 5 percent
0 200 400 600 800
Submission BD to Bank-No Objection BER Issue Specific Procurement Notice-Signature Contract
Source: IEG analysis of sample contract data.

Notes: Two intervals are examined in each box: (1) Borrower Issue of Specific Procurement Notice to Contract Signature and (2) First
Submission to the Bank of Draft Bid (preQ) Documents to the Bank’s No Objection to the Bid Evaluation Report
The vertical line in each box denotes the median; horizontal lines denote “adjacent values.” The offset of the median towards the left and
the ‘long tail’ of the distribution to the right describe the skewed distribution. Dots indicate outliers.
More than a quarter of the processing time from submission of bidding documents
until contract award is due to the preparation of the bid evaluation report after bid
opening. In the case of consultant services more than 40 percent of processing time is
dedicated to this step in the procurement process, likely because for the quality- and
cost-based selection procurement method two documents need to be prepared by
the borrower; an evaluation of technical proposals and, after the subsequent opening
of financial proposals, a combined evaluation report. Both reports require a no
objection by the Bank. This remains true even after excluding those contracts that
bring in the highest five percent of elapsed times per category. Results in terms of
processing times by category, method, and sector are given in Figure D.2.
99
Figure D.2. Average Elapsed Times in Procurement Steps from Submission of Bidding
Documents by Borrower to Contract Signature by Category, Method, and Sector (days)
SELECT FINDINGS: VARIATION IN ELAPSED TIMES
Comparing the arithmetic mean and the mode of some of the distributions, the
former is typically higher, due to a proportion of contracts that are more time
consuming. A key finding is that the procurement process from the time the
borrower issues the specific procurement notice to signature of the contract takes
more time, on average, for procurement method quality- and cost-based selection
used for the selection of consultant services, as compared to ICB and NCB. On
average, the processing times to procure consultant services are more than 100
100
percent greater than for contracts awarded using NCB. And differences are not
merely that of the average time taken, but also in the number of outliers.
Figure D. 3. Average Elapsed Days in Procurement Steps from Submission of Bidding

Documents by Borrower to Contract Signature, by Method (days)

Note: With controls for outliers.
The distribution of elapsed times for the intermediate procurement step from
submission of draft bid documents to the bank’s no objection to the final bid
evaluation report shows the existence of extreme values for all procurement
methods, with especially high values for ICB and quality- and cost-based selection
contracts (Figure D.3). Consultant services and ICB contracts differ only by a few
days in average processing time taken for this interval. The contracts that contribute
the upper five percent of processing time for consultant services selected by the
101
quality- and cost-based selection method take more than three times longer than the
bottom 50 percent of contracts.
REGRESSION ANALYSIS AND RESULTS

Dependent Variable: Elapsed Time Interval (1) Issue of the Specific Procurement Notice to Contract Signature
IEG undertook a simple regression analysis to obtain possible explanations for

factors affecting elapsed times in procurement, considering alternative dependent
variables in the regression specification. A first interval analyzed is the time taken
from the Issue of the Specific Procurement Notice to Contract Signature.
Explanatory variables included contract attributes: (contract value, procurement
method, category of good or service being procured and major sector); control
variables included country specific variables (gross domestic product, poverty rate).
Results indicate a statistically significant positive relationship between contract
value and processing time, robust to the introduction of controls (Table D.2).
Coefficients denote marginal effects, and results suggest (specification 5) that for
every $10 million increase in contract value, the number of days for processing
increases by 14.6, independent of all other contract attributes and across all countries and
regional offices. Results also indicate significantly lower elapsed times for NCB
contracts compared to ICB.
Dependent Variable: Elapsed Time interval (2) – Submission of Draft Bid Documents to No Objection to the Bid
Evaluation Report
Data on this time interval were available for 213 contracts. 42 The positive
relationship between contract value and elapsed time remains statistically
significant, and robust to the addition of controls (Table D.3). Marginal effects are
broadly similar - the regression coefficient in Specification (4) implies that for every
$10m increase in contract value, the number of days for Bank No Objection increases
by 10.6 days, independent of all other contract attributes and across all countries and
regional offices. The coefficient for method—NCB—as compared to other methods, is
also significant, and the coefficient implies that it takes the World Bank, on average,
65 days less to issue no objection for NCB method contracts as compared to ICB.
The second specification also adds country governance as a control variable, in the
form of the CPIA. Results indicate that country level governance factors have a
powerful influence on elapsed time, with lower elapsed times for countries with
higher CPIA scores.
Table D.2. Analysis of Processing Time from Issuing of Notice to Contract Signature
(1) (2) (3) (4) (5)
Variables SPN to Contract SPN to Contract SPN to Contract SPN to Contract SPN to Contract
102
Signature Signature Signature Signature Signature
Contract 1.42e-06*** 1.55e-06*** 1.52e-06*** 1.47e-06*** 1.46e-06***

Value (1.12e-07) (1.34e-07) (1.41e-07) (9.90e-08) (1.01e-07)
Social Sector 3.442 -18.05 -27.32
(51.06) (51.58) (52.15)
Infrastructure 95.80** 65.00 64.89
Sector (44.10) (47.22) (52.47)
Economic -65.66 -46.46 -49.65
Sector (46.38) (45.66) (47.34)
Civil Works -8.500 52.32 48.90
Contracts (56.46) (61.05) (66.33)
Consultant 135.1** 112.9* 70.51
Contracts (63.38) (66.28) (75.10)
Goods 14.62 37.19 34.09
Contracts (59.56) (60.09) (63.01)
NCB Method -112.1*** -113.5***
(39.27) (39.52)
Cons. Method 57.45 93.17
(63.32) (66.99)
GDP 0.0121*
(0.00626)
Average -0.725
Poverty Rate (4.488)
Constant 243.9*** 198.1*** 185.5*** 196.4*** 158.0*
(13.67) (31.30) (60.12) (59.08) (91.97)
Observations 272 272 272 272 272

R-squared 0.074 0.191 0.305 0.342 0.346
Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. N = 272. All country and regional controls have been
included in all specifications. GDP = gross domestic product; SPN = specific procurement notice.
One caveat to the interpretation of these results is that the distribution of residuals in
the regressions exhibits deviation from the normal distribution, possibly due to
some high leverage data points in the 'long tail' discussed in previous sections.
Although the non-normality of residuals does not affect the consistency of an
estimate, it may affect its statistical significance.
Table D.3. Analysis of Processing Time from Submission of draft Bid Documents to Bank’s No
Objection to Final Bid Evaluation Report
(1) (2) (3) (4) (5)
Variables Processing Time for Processing Time Processing Time Processing Time for Processing Time for
Bank No Objection for Bank No for Bank No Bank No Objection Bank No Objection
Objection Objection
NCB Method -68.19*** -47.92** -57.44** -79.32*** -64.61*

(23.05) (21.92) (76.47) (26.42) (27.69)
QCBS 28.41* 70.70*** 76.47*** 1.35 21.90
Method (24.00) (21.40) (22.36) (33.38) (34.87)
103
Contract 9.73e-07*** 1.16e-06*** 1.12e-06*** 1.00e-06*** 1.06e-06***
Value (2.34e-07) (2.13e-07) (1.96e-07) (1.75e-07) (1.78e-07)
Social Sector -59.92 -43.55 -36.42
(39.03) (40.75) (41.27)
Infrastructure -42.50 -36.87 -21.86
Sector (34.65) (36.00) (35.25)
Economic -27.70 -44.30 -58.11
Sector (39.97) (42.73) (44.20)
Civil Works -69.02* -51.14
Contracts (35.41) (35.93)
Goods -99.35*** -76.20*
Contracts (37.35) (40.35)
GDP Level 0.0352***
(0.00962)
Average -1.816
Poverty Rate (2.819)
CPIA -335.33***
score (112.82)
Constant 234.1*** 286.7*** 292.6*** 366.3*** 1,533***
(13.25) (40.59) (42.34) (54.00) (410.6)
Observations 213 213 213 213 213

R-squared 0.153 0.382 0.392 0.406 0.445
Note: Specifications (2) to (5) include country and regional level controls. Robust standard errors in parentheses, *** p<0.01, ** p<0.05, *
p<0.1. GDP = gross domestic product. N = 213.
104
Efficiency of Procurement Management in the Africa Region
Further insights into procurement efficiency are afforded by IEG’s analysis of
datasets made available by the Africa Procurement unit. Generated by PROCYS,
the system tracks each step in a procurement process that involves interaction
between the key players—borrowers, Bank task team leaders and Bank
procurement staff—to monitor the responsiveness and efficiency of each party in
the process. 43
ELAPSED TIMES IN THE PROCUREMENT PROCESS
Each step in a procurement process that involves an exchange between participants

in the procurement process is an entry in PROCYS. In 2012, 3,043 requests were
logged; a number that has increased over time. This likely reflects the fact that use
of PROCYS in terms of country coverage has increased, especially since 2010. 44 On
average, there are two to three iterations both between client and task team leader,
and task team leader and procurement specialist. In terms of numbers of days of
elapsed time, in each iteration, the client takes the most time with an average over
the 2009–12 period of almost 11 days; while the task team leader and procurement
specialist take around 8 days each. There is evidence of increased efficiency in all
parties; the number of days per iteration has shrunk considerably for clients, from
14 in 2009 to 5 days in 2012; from 10 to 5 days for task team leaders and from 8 to 7
days by the procurement specialist.
Table D.4. Africa: Numbers of Iterations and Average Response Times, 2009–12
Year Number of Average number of iterations Average number of days taken per iteration
records Client - TTL TTL – Procurement Client TTL Procurement
specialist specialist
2009 Q1 360 2 2 16 10 9
Q2 404 2 2 14 11 11
Q3 522 2 2 11 8 5
Q4 672 3 2 17 10 6
2010 Q1 655 3 2 12 8 9
Q2 606 3 2 10 10 8
Q3 626 3 3 12 8 6
Q4 653 4 3 13 10 9
2011 Q1 671 4 3 13 8 9
Q2 719 3 3 11 9 6
Q3 661 3 3 15 10 10
Q4 726 3 2 9 8 6
2012 Q1 723 3 2 8 7 8
Q2 898 3 2 6 6 7
Q3 772 2 2 5 5 6
Q4 647 2 1 1 3 5
105
APPENDIX 4C
EFFICIENCY OF PROCUREMENT MANAGEMENT IN THE AFRICA REGION
Avg/Total 10,315 3 2 10.3 8.0 7.5
Source: PROCYS database.
Note: TTL = task team leader.
Aside from average elapsed time, IEG reviewed the distribution of elapsed time.
Since the average numbers of days taken by client, task team leader, and
procurement specialist was considerably greater than the median of their
distributions, it suggests the presence of outliers that take much longer times per
iteration. 45 For instance, the average number of days taken by procurement specialist
per iteration is seven, but half the records (the median) take less than three days
and at the 75th percentile, it requires eight days to process a contract.
Table D.5. Africa: Response Times for Selected Procurement Categories by Clearance Level,
2009–12
Average numbers of days Procurement Category Clearance Level
per iteration PS RPM OPRC
(Number of records) Goods, works, and nonconsulting services 2690 221 40
Consulting services 4333 250 4
Client Goods, works, and nonconsulting services 11 16 8
TTL Goods, works, and nonconsulting services 7 7 13
Procurement specialist Goods, works, and nonconsulting services 7 8 16
Source: Africa procurement office PROCYS database.

Note: OPRC =Operational Procurement Review Committee; PS = procurement specialist; RPM = regional procurement manager; TTL =
task team leader.
Clearance at the OPRC level experienced the largest delays in terms of time taken
by Bank staff (task team leader and procurement specialist) with 27 days per
iteration. However, clients do not take longer to respond to OPRC, with just 8 days
per iteration, in contrast to 16 days on their iterations at the RPM level. There are
also clear differences in the numbers of client and task team leader iterations,
depending on the clearance level, with an average of five iterations for requests at
the OPRC level, four at the RPM level, and three at the procurement specialist level.
To estimate overall Bank performance (as separated from the client), IEG aggregated
the time taken by the task team leader and the procurement specialist. Best and
worst performers are indicated in Table D.6.
Table D.6. Africa: Procurement—Best and Worst Performers, 2009–12

Country Number Average number of Average number Average number Average
of days taken by client of days taken by of days taken by number of days
records (per iteration) TTL (per procurement taken by TTL
iteration) specialist (per +PS (per
iteration) iteration)
The longest numbers of days
Guinea-Bissau 30 9 32 7 39
Burkina Faso 220 19 13 23 36
106
APPENDIX 4C
Tanzania 878 14 9 12 21
The least number of days
Mauritius 17 15 2 3 5
Gabon 44 9 3 4 7
Mozambique 776 8 6 2 7
Note: PS = procurement specialist; TTL = task team leader.
During the 2009–12 period, there are no significant differences in response times
between procurement categories for any of the three sets of players in the process.
In terms of specific products procured, clients spend the most time on “plant design,
supply and installation” with 18 days on average, based on 93 requests recorded
from 2009 to 2012. For task team leaders, it takes the most time—16 days—to
process “Retroactive No-Objections,” based on 23 cases during the period of
analysis. For procurement specialists, “output and performance-based road
contracts” took the most time to process: 51 days; followed by “information
systems,” with 15 days. It should be noted that the products procured did not seem
uniformly defined. For example, in some cases procurement of consulting services
was indicated to be related to firms or individuals, in other cases it was not
specified.
In terms of procurement method, the time taken by clients to respond to ICB-related

requests was 15 days on average; compared to 8 days for NCB and 12 for quality-
and cost-based selection requests. Response times for these methods are similar for
task team leaders and procurement specialists; however, two-stage ICB takes 3
more days, per iteration, for task team leaders to process than for procurement
specialists. Along with shopping, two-stage ICB takes the most time to process (11
days) for task team leaders than any other procurement method. For procurement
specialists, the methods that take the most time are NCB, with 10 days; followed by
ICB and quality- and cost-based selection that take 9 days. 46 The procurement
methods that entail the shortest processing times are “force account” and
“commercial practices.” Although there are few of these cases in our sample, 20
records for “force account” and only 2 for “commercial practices,” the times that
clients, task team leaders and procurements specialists take for request processing
are lower than the rest. They vary in between zero and four days.
BORROWER DELAYS AND IDLE TIME
As is clear from the table below, borrower delays in between steps in the
procurement process are even greater than the actual time taken by all parties to
process the different requests. And delays appear to vary by quarter. In 2012, for
example, borrower delays in between steps averaged 44 days; while the total time
taken to process all requests was 27 days. The 27 days refer to the average time per
iteration taken by the borrower (15 days), task team leader (6 days) and by
107
APPENDIX 4C
procurement staff (6 days). Borrower delays vary by quarter but they are always
above Bank response times, on average, by more than three times.
Table D.7. Africa: Borrower Delays and Bank Response, 2010–12
Borrower Response times (days) within iterations

delay (days) Borrower Task team Procurement Bank Total
between time time staff time response response
Year-Quarter iterations time time
2010 Q1 18.3 9.2 3.8 5.1 8.9 18.1
Q2 38.1 16.1 5.5 4.6 10.1 26.2
Q3 49.6 19.3 9.3 5.9 15.2 34.6
Q4 38.6 12.4 5.7 7.3 13.0 25.4
2011 Q1 53.5 12.8 10.6 6.4 17.0 29.8
Q2 30.9 11.8 6.6 6.5 13.1 25.0
Q3 39.8 14.1 5.8 5.4 11.2 25.2
Q4 39.1 14.8 6.5 5.8 12.3 27.1
2012 Q1 38.1 15.7 6.9 6.1 13.0 28.7
Q2 63.0 17.5 6.9 7.2 14.1 31.6
Q3 38.1 15.1 6.6 6.0 12.6 27.6
Q4 38.4 11.4 4.8 5.8 10.6 22.0
Note: Delays recorded in PROCYS are defined as the average time taken by the borrower beyond normal processing time (70 days for
consultants and 90 days for goods, works, and services.
Three clearance levels were indicated in the database: procurement specialist, RPM,
and OPRC. Borrower delay was similar at the RPM and OPRC levels but much
lower with clearance at the procurement specialist level.
Table D.8. Africa: Borrower Delays by Clearance Level, 2010–12

Clearance level Number of records Borrower delay (days)
PS 1,026 40
RPM 68 69
OPRC 4 65
Not available 494 40
Total 1,592 42
Note: OPRC = Operational Procurement Review Committee; PS = procurement specialist; RPM = regional procurement manager.
According to the most recent report from the Africa procurement unit, the countries
with the largest borrower delays are Chad (66 days), Cameroon (63 days), and
Ethiopia (62 days). 47 The largest Bank response times per iteration correspond to
South Africa (21 days), Burkina Faso (19 days), and Central African Republic (16
days).
108
APPENDIX 4C
Figure D.4. Procurement Processing Delays and Bank Response by Country
Botswana
Comoros
Sao Tome and Principe
Sierra Leone
Central Africa
Gabon
Cape Verde
Mauritania Procurement
Swaziland Staff
Congo
Gambia
Task Team
South Africa
Lesotho
Central African Republic
Malawi Borrower Delay
Niger
Sudan
Madagascar
Mali
Togo
Nigeria
Mozambique
Burundi
Burkina Faso
DRC
Liberia
Angola
Benin
Africa
Zambia
Republic of Congo
Eastern Africa
Rwanda
Tanzania
Kenya
Uganda
Senegal
Ivory Coast
Ghana
Ethiopia
Cameroon
Chad
0 10 20 30 40 50 60
Source: Africa procurement office database.
Note: Delays recorded in PROCYS are defined as the average time taken by the borrower beyond normal processing time (70 days for
consultants and 90 days for goods, works, and services).
109
Bibliography
for Realistic Solutions. Cambridge:
Cambridge University Press 128-130.
APEC (Asian Pacific Economic Cooperation).
1999. “Government Procurement
Aberdeen Group. 2008. The E-Procurement Experts Group Non-Binding
Benchmark Report: Driving Year-Over- Principles on Government
Year Superior Performance. Boston: Procurement”. August 1999.
Aberdeen Group. Singapore: APEC.
ADB (Asian Development Bank). 2002. ASIL (American Society of International Law).
“Understanding Public Procurement.” 2013. Will the World Trade
Governance Brief, Asian Development Organization Finally Tackle
Bank. Manila: ADB. Corruption in Public Purchasing? The
———. 2007. Environmentally Responsible Revised Agreement on Government
Procurement—A Reference Guide. Procurement.
Manila: ADB. http//www.asil.org/insights130415.c
fm.
AfDB. (African Development Bank). 2011.
“Evaluation of Paris Declaration Blue, Richard, and John Eriksson. 2011.
Implementation at the African “Evaluation of the Implementation of
Development Bank.” Operations the Paris Declaration: United States
Evaluation Department. Tunis: AfDB. Government Synthesis Report”.
Arlington, VA: The United States
———. 2012. “Determination of Financial
Agency for International
Thresholds for Modes of Procurement
Development.
of Goods, Works and Non-Consulting
Services”. African Development Bank Burger, Philippe, and I. Hawkesworth. 2011.
Group, Procurement Policy and How to Attain Value for Money:
Fiduciary Services Department. Comparing PPP and Traditional
January 2012. Tunis: AfDB. Infrastructure Public Procurement.
OECD Journal on Budgeting, Volume
AfDB, IDB (Inter-American Development 2011/1. Paris, France: OECD.
Bank), World Bank, and Curtin
University of Technology. 2007. Casartelli, G., E. Wolfstetter. 2007. "Selection
International Survey of Procurement. and Employment of Consultants: the
Perth. Effectiveness of the World Bank's
Policy" Working Paper. Washington,
Agaba, Edgar, and Nigel Shipman. 2007l DC: World Bank.
“Public Procurement Reform in
Developing Countries: The Uganda Commonwealth of Australia. 2008.
Experience.” In G. Pig and K.V. Thai Commonwealth Procurement Guidelines,
(eds.) Advancing Public Procurement: December 2008. Issued under
Practices, Innovation, and Knowledge Regulation 7 of the Financial
Sharing (Boca Raton, FL: Pr. Management and Accountability
Academics Press), 373-91. Regulations, 1997. Canberra:
Commonwealth of Australia.
Al-Azar, Rima. 2011. “A Tale of Two CDDs:
Lessons Learned for Mitigating the Debevoise, E.W., and C. Yukins. 2010.
Risk of Fraud and Corruption in CDD “Assessing the World Bank’s
Operations.” PowerPoint Proposed Revision of Its Procurement
presentation, World Bank, Guidelines,” The Government
Washington, DC. Contractor, 52, (21).
Andrews, Matt. 2013. The Limits of Institutional Dehn, Jan, Ritva Reinikka, and Jakob
Reform in Development: Changing Rules Svensson. 2003. “Survey Tools for
110
Assessing Performance in Service Procurement of Goods and Services,
Delivery.” In Francois Bourguignon February 16. Mimeo.
and Luiz Pereira da Silva (eds.),
IEG (Independent Evaluation Group). 1977.
Evaluating the Poverty and
“The Role and Use of Consultants in
Distributional Impact of Economic
Bank Group Projects.” Report no.
Policies. New York: Oxford University
1824, Washington, DC: World Bank.
Press and World Bank.
———. 1981. “An Interim Report on
Dener Cem, Joanna. A Watkins and William.
Procurement Issues in Bank-Financed
L. Dorotinsky. 2011. “Financial
Projects.” Report no. 3557,
Management Information Systems—
Washington, DC: World Bank.
25 Years of World Bank Experience on
What Works and What Doesn’t.” ———. 1990. “A Review of Procurement in
World Bank, Washington, DC. Policy-Based Lending.” Report no.
8341, Washington, DC: World Bank.
De Silva, S. 2000. “Community-based
Contracting; A Review of Stakeholder ———. 2002. India Country Assistance
Experience.” Washington DC: World Evaluation. Washington, DC: World
Bank. Bank
DfID (Deparment for International ———. 2005. The Effectiveness of World Bank
Development). 2011. “Approach to Support for Community-Based and –
Value for Money.” London: Driven Development: An OED
Department for International Evaluation. Washington, DC: World
Development. Bank.
European Union, World Bank, and Inter- ———. 2006a. Debt Relief for the Poorest—An
American Development Bank. 2009. Evaluation Update of the HIPC Initiative.
Perú Informe de la gestión de las finanzas Washington, DC: World Bank.
públicas Basado en el marco de referencia
———. 2006b.Engaging with Fragile States: An
para la medición del desempeño (PEFA).
Evaluation of World Bank Support for
Lima: European Union, World Bank
Low-Income Countries Under Stress.
and Inter-American Development
Bank.
———. 2006c. Hazards of Nature, Risks to
Gartner Group. 2012. “Survey Shows Why Development – An IEG Evaluation of
Projects Fail.” World Bank Assistance for Natural
http://www.gartner.com/id=2034616 Disasters. Washington, DC: World
. Bank.
GSDRC. 2010. Helpdesk Research Report: ———. 2006d. Small States: Making the Most of
Public Financial Management and Development Assistance – A Synthesis of
Frontline Service Delivery. World Bank Evaluation Findings.
http://www.gsdrc.org/docs/open/ Washington, DC: World Bank.
HD653.pdf,.
———. 2007a. The Development Potential of
IDB (Inter-American Development Bank). Regional Programs – An Evaluation of
2010. “Guide for Acceptance of the World Bank Support of Multicountry
Use of Country Procurement Operations. Washington, DC: World
Systems.” June 2010. Washington DC: Bank.
IDB.
———. 2007b. Water Management in
IAEA (International Atomic Energy Agency). Agriculture—Ten Years of World Bank
2011. Agency for Financial Assistance, 1994-2004. Washington,
Regulations and Rules AM VI/1: DC: World Bank.
111
APPENDIX 4C
———. 2008a. Country Financial Accountability Support. Washington, DC: World
Assessments and Country Procurement Bank.
Assessment Reports: How Effective Are
———. 2010f. The World Bank in Nigeria 1998-
World Bank Fiduciary Diagnostics?
2007—Country Assistance Evaluation.
Washington, DC: World Bank
———. 2008b. Public Sector Reform: What Works
———. 2011a. “Aid Effectiveness 2005-10:
and Why? An IEG Evaluation of World
Progress in Implementing the Paris
Bank Support. Washington, DC: World
Declaration.” World Bank,
Bank.
Washington, DC.
———. 2008c. The World Bank in Indonesia
1999-2006--Country Assistance ———. 2011b. Capturing Technology for
Development – An IEG Evaluation of
Evaluation. Washington, DC: World
World Bank Group Activities in
Bank
Information and Communication
———. 2009a. Bangladesh Country Assistance Technologies. Washington, DC: World
Evaluation 2001-2008. Washington, Bank.
DC: World Bank
———. 2011c. Trust Fund Support for
———. 2009b. Improving Effectiveness and Development – An Evaluation of the
Outcomes for the Poor in Health, World Bank’s Trust Fund Portfolio.
Nutrition and Population--An Evaluation Washington, DC: World Bank.
of World Bank Group Support since 1997.
———. 2011d. World Bank Country-Level
Washington, DC: World Bank
Engagement on Governance and
———. 2009c. Review of IDA Internal Controls: Anticorruption: An Evaluation of the
An Evaluation of Management’s 2007 Strategy & Implementation Plan.
Assessment and the IAD Review. Washington, DC: World Bank.
———. 2011e. World Bank Progress in
———. 2009d. The World Bank Group Guarantee Harmonization and Alignment in Low-
Instruments 1990-2007 –An Independent Income Countries. Washington, DC:
Evaluation. Washington, DC: World World Bank.
Bank
———. 2012a. The Matrix System at Work: An
———. 2010a. Cambodia – An IEG Country Evaluation of the World Bank’s
Assistance Evaluation, 1999-2006. Organizational Effectiveness.
Washington, DC: World Bank Washington, DC: World Bank.
———. 2010b. Climate Change and the World ———. 2012b. “The World Bank and Public
Bank Group. Phase II: The Challenge of Procurement. An Independent
Low-Carbon Development. Washington, Evaluation.” Approach Paper.
DC: World Bank. Washington, DC: World Bank.
———. 2010d. IDA Internal Controls: Evaluation ———. 2012c. The World Bank Group’s Response
of Management’s Remediation Program-- to the Global Economic Crisis—Phase II.
Report on Management’s Implementation Washington, DC: World Bank.
and IAD’s Review of the Five-Point
———. 2013. “Comparison of the
Action Plan. Washington, DC: World
International Instruments on Public
Bank.
Procurement.” Background paper.
———. 2010d. Mozambique Country Program Washington, DC: World Bank.
Evaluation. Washington, DC: World
IMF (International Monetary Fund). 2012.
Bank.
Update on the Bank’s Business
———. 2010e. Poverty Reduction Support Modernization: Results, Openness,
Credits: An Evaluation of World Bank
112
and Accountability Spring 2012.”IMF ———. 2006a. “The Challenge of Capacity
World Bank Development Committee. Development: Working Towards
Good Practice.” DAC Reference
Kivistö, T., V.M. Virolainen, and E. Tella. 2003.
Document, Paris.
Consortia purchasing and logistics in
Kuopio area (Finland)--Lessons ———. 2006b. “Methodology for Assessment
learned from a 4-year project. IRSPP of National Procurement Systems.”
conference, Budapest, and in KV Thai Paris: OECD.
et al. (eds.), (2005): Benchmarking
———. 2009. “Compendium of Country
municipal public procurement
Examples and Lessons Learned from
activities in Finland. In Challenges in
Applying the Methodology for
Public Procurement: an International
Assessment of National Procurement
Perspective.
Systems: Sharing Experiences.” OECD
Korpela, J., and M. Tuominen. 1994. Journal on Development 9(4).
Benchmarking Logistics Performance with
———. 2011a. International Engagement in
an Application of the Analytical
Fragile States: Can’t We Do Better?
Hierarchy Process. Lappeenranta,
Paris: OECD.
Finland: Lappeenranta University of
Technology. ———. 2011b. A Practical Guide for
Transforming Procurement Systems.
Kruger. K, 2009. “Ban on negotiations in
Paris: OECD.
tender procedures – Undermining the
best value for money.” Journal of ———. 2011c. Strengthening Country
Public Procurement, 4. Procurement Systems, prepared for the
4th High Level Forum on Aide
Mackie, Andrew, and Giovanni Caprio. 2011.
Effectiveness, November 29 –
Assessing the Impact of the FEFA
December 1, 2011, Busan, Korea.
Framework. Washington, DC: PEFA
OECD/DAC Working Party on Aid
Secretariat.
Effectiveness Task Force on
Ministry of Foreign Affairs of Denmark. 2008. Procurement.
“Evaluation of the Implementation of
OECD-DAC and World Bank. 2004.
the Paris Declaration: Phase One
“Mainstreaming and Strengthening
Synthesis Report.” Denmark: Ministry
Public Procurement: A Strategic
of Foreign Affairs.
Framework.” Draft report of the 3rd
OECD (Organization for Economic Joint Round Table on Strengthening
Cooperation and Development). 2003. Procurement Capacities in Developing
Mainstreaming the Procurement Countries. Washington, DC, and
Function into the Public Expenditure Paris.
Policy and Effectiveness Dialogue. Paris:
OECD. Pallas, Christopher L., and Jonathan Wood.
2009. “The World Bank’s Use of
———. 2005a. “Paris Declaration on Aid Country Systems for Procurement: A
Effectiveness--Ownership, Good Idea Gone Bad?” Development
Harmonization, Alignment, Results Policy Review 27 (2): 215-230.
and Mutual Accountability.” Paris,
France: OECD. PEFA Secretariat. 2011. Independent Evaluation
of the Public Expenditure and Financial
———. 2005b. “Strengthening Procurement Accountability Program. Washington,
Capacities in Developing Countries.” DC: PEFA.
Volume 3 of the OECD-DAC
Pollit, C., and G. Bouckaert. 2004. Public
Guidelines and Reference Series,
Management Reform: A Comparative
Harmonising Donor Practices for
Effective Aid Delivery. Paris: OECD.
113
APPENDIX 4C
Analysis. New York: Oxford ———. 2003. “Standard Bidding Documents:
University Press. Supply and Installation of Information
Systems.” Washington, DC.
Sanchez, A. 1997. The World Bank Procurement
Function – Adjusting to Emerging Needs. ———. 2004. “Country Procurement Issues
Washington, DC. World Bank. Paper—Rwanda.” Washington, DC:
Retrieved from: World Bank.
http://intresources.worldbank.org/I
———. 2005a. “Africa Capacity Building Task
NTPROCUREMENT/Resources/Proc
Force Report: Forging Effective States
urement-Strategy.pdf.
and Engaged Societies.” Washington,
Singh, J., Dilip R. LImaye, Brian Henderson, DC: World Bank.
and Xiaoyu Shi. 2012. “Public
———.2005b. “Expanding the Use of Country
Procurement Of Energy Efficiency
Systems in Bank-Supported
Services: Lessons from International
Operations: Issues and Proposals.”
Experience.” Washington, DC: World
Operations Policy and Country
Bank.
Services, Washington, DC.
Tonkin, Christine. 2012. “Procurement
———. 2005c. “Meeting the Challenge of
Governance Reform at the
Africa’s Development: A World Bank
International Atomic Energy Agency.”
Group Action Plan.” Africa Region
Mimeo.
Department, Washington, DC: World
Thai, K.V., et al. 2005. “Benchmarking Bank.
Municipal Public Procurement
———. 2006. “Quality of Information and
Activities in Finland.” In Challenges in
Communication Technology
Public Procurement: An International
Components in Bank Projects.”
Perspective. Academic Press.
Quality Assurance Group
UNCITRAL (United Nations Commission on Assessment, Washington, DC: World
International Trade Law) Secretariat. Bank.
(1994, 2011) “UNCITRAL Model Law
———. 2007a. “Detailed Implementation
on Public Procurement.” UNCITRAL,
Review. India Health Sector. 2006-
Vienna, Austria.
2007.” INT, Washington, DC: World
Volcker, Paul, Gustavo Gaviria, John Bank.
Githongo, Ben W. Heineman Jr.,
———. 2007b. “Mandatory Provisions for
Walter Van Gerven, and Sir John
NCB (for Bank/IDA financed
Vereker. 2007. Independent Panel
contracts).” Washington, DC: World
Review of the World Bank Group
Bank.
Department of Institutional Integrity.
Washington, DC: World Bank. ———. 2007c. “Operational Approaches and
Financing in Fragile States.”
World Bank. 1997. “The World Bank
Washington, DC.
Procurement Function- Adjusting to
Emerging Needs.” SecM97-854, ———. 2007d. “Strengthening World Bank
Washington, DC: World Bank. Group Engagement on Governance
and Anti-Corruption.” Washington,
———. 2002a. “Fiduciary Management for
DC: World Bank.
Community-Driven Development
Projects: A Reference Guide.” ———. 2008a. “Enhanced Support to Staff
Washington, DC: World Bank. Working in Fragile and Conflict-
Affected States.” Washington, DC:
———. 2002b. Guidelines for Country
World Bank.
Procurement Assessment Reviews.
114
———. 2008b. “Use of Country Procurement July 2011, November 2011, January
Systems in Bank-Supported 2012. Washington, DC: World Bank.
Operations: Proposed Piloting
———2010g. “Public Procurement of Energy
Program.” R008-0036/5. Washington,
Efficiency Services: Lessons from
DC: World Bank.
International Experience.”
———. 2009a. “Guidance to Staff for Using Washington, DC: World Bank.
the Company Risk Profile Database
———. 2010h. “Republic of Moldova Country
(CRPD).” World Bank, Washington,
Procurement Assessment Report.”
DC.
———. 2009b. “Handling Suspected and
———. 2011a. “Community-Driven
Alleged Fraud and Corruption in
Development Projects: A Review of
Procurement: OPCS and INT
Governance Issues.” OPCS Document,
Protocol.” Washington, DC: World
World Bank, Washington, DC.
Bank.
———. 2011b. Country Partnership Strategy
———.2009c. “Piloting Program in Use of
Progress Report for the Republic of
Country Systems: First Progress
Moldova for the Period FY09-FY13.
Report.” Operations Policy and
Country Services. Washington, DC:
World Bank. ———. 2011c. “Diversity Benchmarking Study
in Selected Countries.” OPSOR,
———. 2010a. Financial Management Sector
Mimeo. Washington, DC: World
Strategy FY11-13. 6, 9. Washington,
Bank.
DC: Financial Management Sector
Board. ———. 2011d. “Financial Management
Information Systems—25 Years of
———. 2010b. “Guidance Note on the
World Bank Experience on What
Operational Risk Assessment
Works and What Doesn’t.”
Framework (ORAF) – Risks to
Achieving Results.” Operations Policy
and Country Services, World Bank, ———. 2011e. “Green Procurement Policy
Washington, DC. Framework in Selected Countries.”
OPSOR, Mimeo. Washington, DC:
———. 2010c. “Guidance Note: Procurement
World Bank.
Arrangements Applicable to Public-
Private Partnership Contracts ———. 2011f. “Guidelines: Procurement of
Financed under World Bank Projects.” Goods, Works, and Non-Consulting
Washington, DC: World Bank. Services under IBRD Loans and IDA
Credits and Grants by World Bank
———. 2010d. Monitoring Report 2010. PEFA
Borrowers.” Washington, DC: World
Secretariat Report. World Bank,
Bank.
Washington, DC.
http://pefa.org/sites/pefa.org/files/ ———. 2011g. “Strengthening Country
attachments/ENG-%20MR-10- Procurement Systems.” Prepared for
FinalReport-May192011.pdf. the 4th High Level Forum on Aide
Effectiveness, November 29 –
———. 2010e. “Piloting Program in the Use of
December 1, Busan, Korea.
Country Procurement Systems—
Second Progress Report.”
Washington, DC: World Bank. ———. 2011h. “Strengthening World Bank
Engagement on Governance and Anti-
———. 2010f. “Procurement Risk Assessment
Corruption: Procurement Companion
and Management System User
Piece.” Washington, DC: World Bank.
Guide.” OPSOR, May 2010, Updated
115
APPENDIX 4C
———. 2011i. World Development Report: User Guide.” Washington, DC: World
Conflict, Security, and Development. Bank.
———. 2012j. “Procurement and PPP
———. 2012a. “Analysis of World Bank Transactions Guidance for MDB
Completed Cases of Fraud and Public-Sector Engagements and
Corruption from the Perspective of Procurement Principals Applicable to
Procurement.” Integrity Vice Private Sector Transactions—
Presidency, Washington, DC: World Guidance for MDBs.” Washington,
Bank. DC: World Bank.
———. 2012b. “Applying the Principles for ———. 2012k. “Procurement Risk Assessment
Good International Engagement in and Management System User
Fragile States and Situations to Guide.” Washington, DC: World
Strengthen and Transform Public Bank.
Procurement Systems.” Background
———. 2012l. “Project Appraisal Document
note, OPSOR. Washington, DC: World
on a Proposed Credit in the Amount
Bank.
of SDR 26.1 million ($40.0 million
———. 2012c. “The Bank's Procurement Equivalent to the Republic of
Policies and Procedures: Initiating Moldova for the Moldova Education
Discussion and Approach Papers-- Reform Project).” Washington, DC:
Proposed Review.” Report submitted World Bank.
to the Audit Committee and to CODE,
———.2012m. “Public Procurement of Energy
Efficient Products: Lessons from
———. 2012d. “Fiduciary Management for Around the World.” Energy Sector
Community-Driven Development Management Program. Washington,
Projects: A Reference Guide.” DC: World Bank.
———.2012n. “Review of the World Bank’s
———. 2012e. Financial Management and Procurement Policies and Procedures:
Procurement in World Bank Operations: Achieving Social Diversity Through
Annual Report for FY11. Washington, the use of Supplier Preferences in the
DC: World Bank. Procurement Process.” Background
paper. Washington, DC: World Bank.
———. 2012f. “Green Procurement in Selected
Environmental Policy Framework: ———.2012o. “Review of the World Bank’s
Review of the World Bank‘s Procurement Policies and Procedures,
Procurement Policies and The Use and Impact of the Bank’s
Procedures.” Background Paper. Policy of Domestic Preferences.”
Washington, DC: World Bank. Background Paper. Washington, DC:
World Bank.
———. 2012g. “Guidance Note for Design and
Management of Procurement ———. 2012p. “Update on the Bank’s Business
Responsibilities in Community- Modernization: Results, Openness,
Driven Development Projects.” and Accountability Spring 2012.”
Washington, DC: World Bank. DC2012-0005. Washington, DC: World
Bank.
———. 2012h. “Investment Lending Reform:
Modernizing and Consolidating ———. 2012q. “Virtual VP Review of the
Operational Policies and Procedures.” Procurement Policy Review Proposal-
OPCS, Washington, DC: World Bank. -An Approach Paper.” Washington,
DC: World Bank.
———. 2012i. “OPCPR Procurement Risk
Assessment and Management System
116
———.2013a. “Approach for Private Sector
Procurement in Projects financed by
IBRD/IDA, including Public Private
Partnerships.” Washington, DC:
World Bank.
———. 2013b. “Audit of the Bank’s
Operational Framework for Using
Investigation Results in Bank-Funded
Projects.” Report No. IBRD FY12-16.
———. 2013c. “Procurement in World Bank
Investment Operations. Phase I: A
Proposed New Framework.” OPCS.
———. 2013d. “Red Flags Learning Tool.”
Integrity Vice Presidency.
http://go.worldbank.org/D1K227ET8
1.
———. 2013e. “Situations of Urgent Need of
Assistance or Capacity Constraints:
Simplified Procurement Procedures
and Guiding Principles.”
———. 2013f. “Supporting Countries in
Strengthening their Public
Procurement Systems: Second
Generation Reforms” (draft).
Operations Risk Management
Department. Washington, DC: World
Bank.
———.2013g. “Use of Country Procurement
Systems.” Background Paper,
Washington, DC, World Bank.
World Bank Institute. 2012. “Public-Private
Partnership Reference Guide, Version
1.0.” Washington, DC: World Bank.
World Trade Organization. 1996. “Agreement
on Government Procurement (GPA).”
Geneva: World Trade Organization.
.
117
Endnotes
Appendix A
1 Others apply some value for money principles (such as life-cycle costing, for example) to varying degrees,
often without calling it “value for money.” Some agencies use the term but without defining a distinct
procurement process around it—for example, the United Nations procurement guide refers to value for moeny,
but only in limited terms and makes no recommendations to operationalize VfM in procurement policies and
practices.
2 Getting Value for Money from Procurement: How Auditors Can Help, Office of Government Commerce and
National Audit Office, UK, 2001; Regularity, Propriety and Value for Money, Treasury Officer of Accounts, HM
Treasury, UK, November 2004; Procurement Efficiency and Value for Money Measurement, Office of
Government Commerce, 2005; and An Introduction to Public Procurement, Office of Government Commerce,
2008.
3“Best Value” was ascribed its modern regulatory definition by FAC 97-02, Sept. 30, 1997, as part of the rewrite
of Federal Acquisition Regulation Part 15.
4Directive 2004/18/EC of the European Parliament and of the Council of March 31, 2004 on the Coordination of
Procedures for the Award of Public Works Contracts, Public Supply Contracts and Public Service Contracts. The
EU Directive says, “…Where the contracting authorities choose to award a contract to the most economically
advantageous tender, they shall assess the tenders in order to determine which one offers the best value for
money.”
5 MEAT must be used when competitive negotiation procedures are applied (Article 29).
6Commonwealth of Australia, Commonwealth Procurement Guidelines, December 2008. Issued under Regulation 7
of the Financial Management and Accountability Regulations 1997.
7 APEC Government Procurement Experts Group Non-Binding Principles on Government Procurement. August
1999.
8 Procurement in World Bank Investment Operations. Phase I: A Proposed New Framework. OPCS, March 29,
2013.
9 Although negotiation after contract award is not generally accepted good practice, with the possible exception
of in the United Kingdom.

10 By using benchmarks, for example, as illustrated in a simple example in Chapter 4.
11 See Chapter 6 for a description of competitive negotiation.
12 The proposal is currently under legislative review, between the Council and the European parliament. In
December 2012, the Council adopted a compromise text, which is now the basis for negotiations between council
and parliament. There is some expectation, at the European Union, that the new directive may be adopted before
the end of 2013. Implementation dates are unclear.
13Governance in public procurement is increasingly on the agenda of international organizations as well,
including the World Trade Organization, as witnessed by proposed amendments to its Government
Procurement Agreement (Chapter 5).
Appendix B
14
The 2000 Bank document Community-based Contracting—A Review of Stakeholder Experience highlighted local
shopping among the very simple contracting procedures.
15 Examples of incentives were, for example, savings reverting back to communities for financing additional
facilities.
119
ENDNOTES
16 Although this conclusion may not have been surprising, since the sample had purposively focused on
successfully completed subprojects. Construction cost-savings ranged from 12 percent to 56 percent compared to
original estimates. At the same time, the quality and sustainability of the completed projects was found to be
high.
consultant.
18 A Mongolia case study noted that procurement of the ICT system went through three procurement processes
over two years The study reports on three FMIS related projects in Mongolia (P051855, P077778, P098426), with
the first project approved in June 1998. Two were active at the time of the report (2011).
19 The case study covers one Bank funded project (P035759), approved in 1995 and closed in 2002, and a
subsequent Government-funded component. The cost of the Bank project was $78.5 million. In 1999, the
Government and the Bank agreed to restructure the project, and the government initiated the Public Expenditure
Management (PEM) component as a separate activity funded from the its own budget. Project funds were used
to develop a customs information system and improvement of debt management system, together with related
advisory support. The PEM component was implemented as a government led activity in parallel to the Bank
project, with a cost of approximately $15.7 million.
20Unlike the other MDBs, the Black Sea Trade and Development Bank takes a very much hands off approach. It
only requires procurement methods that “lead to sound selection of goods, works and services at fair market
prices.”
Appendix C
21Resource Guide: Procurement Methods
http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:20109695~pa
gePK:84269~piPK:60001558~theSitePK:84266~isCURL:Y,00.html.
22 Most countries also set monetary thresholds for the use of shopping procurement method.
23 World Bank Procurement Method Thresholds
http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:21038090~isC
URL:Y~menuPK:2926825~pagePK:84269~piPK:84286~theSitePK:84266,00.html.
24http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:21038037~is
CURL:Y~menuPK:2926825~pagePK:84269~piPK:84286~theSitePK:84266,00.html.
Bank guidelines do make publication of Specific Procurement Notices in UNDB and DG market electronic portal
mandatory for ICB processes.
25 Procurement of goods, works and non-consulting services under IBRD Loans and IDA Credits and Grants by
World Bank Borrowers, Paragraph 1.3, January 2011.

26 The field “amountUSE” of Form 384 was used as contract value.
27 When looking at the distribution by value a higher share is won by foreign firms, though still more than half of
contracts in the highest quintile go to national firms (Table 1, “Distribution by Value”).

28 In 2008, three risk levels were defined by implementing agency (high, medium, or low). In 2009, Bank-wide
maximum prior review thresholds were revised and categories expanded into high, substantial, moderate, or
low.
292002BP 11.0, Annex A Decision Authority Matrix. This document specifies clearance thresholds for task team
leaders/PS/PAS, RPA, Regional Vice Presidents, and OPRC.
http://intranet.worldbank.org/WBSITE/INTRANET/OPSMANUAL/0,,contentMDK:22819821~menuPK:51457
764~pagePK:51458737~piPK:51458739~theSitePK:210385,00.html
30Email from OPCPR retrieved from:
http://intranet.worldbank.org/WBSITE/INTRANET/INTCOUNTRIES/INTAFRICA/INTAFRPRO/0,,content
MDK:21772310~pagePK:64168332~piPK:64168299~theSitePK:1938585,00.html, this specified, for example, task
team leader/procurement specialist levels of justification for single source selection of consultants.
120
ENDNOTES
31http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:21038090~m
enuPK:2926825~pagePK:84269~piPK:84286~theSitePK:84266~isCURL:Y~DIR_PATH:WBSITE/EXTERNAL/PR
OJECTS/PROCUREMENT/,00.html.
32 http://www.transparency.org/research/cpi/
33http://www.worldbank.org/ida/IRAI-2011.html. Bank data on CPIA scores for IBRD countries were also
used.
Appendix D
34 Sistema de Ejecucion de Planes de Adquisiciones in Spanish.
35SEPA does not handle procurement documents or attachments through the system.
36To upload a procurement plan the government enters project information and data on the related contracts.
On the project level this includes procurement method and prior review thresholds, names of implementing
agencies, responsible task team leaders and procurement specialists, project amounts and key project dates,
among other things. On the contract level this includes contract description, estimated amount, the percentage of
the contract financed by the local counterpart to the project, and estimated dates for all procurement stages
related to the contract. Over the course of execution, actual amounts and actual dates are entered into the
system, after the estimated dates. Dates are entered according to procurement method and review method.
After the actual contract signature date is recorded, contracting data such as bidder’s names and their bids,
winning bidder and contract value can be entered.
37In the form of Excel spreadsheets, available from October 2008. Database macros allow users to produce
management reports filtered for information for the entire region, for a selected country department, country or
procurement staff.
38 The data template is included in Appendix A, which contains the overall questionnaire template.
39 Information was requested on the procurement steps borrower issue of specific procurement notice, borrower
first submission to Bank of draft bid (preQ) documents, Bank final no objection to draft bid (PreQ) documents,
borrower issue of Bid (PreQ) documents, borrower bid (PreQ) opening date/ minutes, borrower submission to
Bank of bid evaluation report, Bank no-objection to bid evaluation report, borrower submission to Bank of
negotiated contract, Bank no-objection to negotiated contract, date of signature of contract.
40 For consultant services selected by quality and cost the dates of submission of / no objection to the negotiated
contract were considered as the final bid evaluation report. For contracts flagged as ICB/NCB Two Stage the
first stage Submission of / No Objection to Bid Evaluation Report was used, only one observation was received
for the second stage. For contracts flagged as subject to Pre-Qualification the dates of Submission of / No
Objection to the final Bid Evaluation Report were used. Where Bid Evaluation Report Submission / No Objection
dates were not provided the “Award No Objection” date in the bank-wide Form 384 database was used if the
contract information received from the field could be matched with Form 384.
41 International Monetary Fund, International Financial Statistics: http://elibrary-data.imf.org/.
42For the regression analysis of average processing times from submission of draft bid documents to no
objection to final bid evaluation report only quality- and cost-based selection procurement is considered for
consultant services contracts.
43Two databases were provided by the regional procurement management office, on user responsiveness and
elapsed times. Data cover the period from 2009 to 2012; however, data for 2009 are incomplete and 2010 provides
a first reasonably full dataset.
44Numbers of iterations for 2011 were 2,778; for 2010, 2,540; and for 2009, 1,970. While 2009 was a startup year,
the system is now at “steady state.” Arguably it could also suggest an increase in the number of iterations.
However, data in Table B.14 suggest that this may not be the case.
121
ENDNOTES
45 The database shows some errors in recording times. For example, in 62 cases (of 10,331) the average number of
days that client take to respond has a negative values. However, these are not frequent.
46 For clients, limited international bidding takes 19 days to process. Other lengthier client response times
correspond to ICB and two-stage ICB with 15 and 14 days, respectively.

47The latest monthly report produced by the Africa Procurement Unit on April 2, 2013, that includes no
objections processed through July 1, 2011, and excludes 1 percent of outliers.
122

Procurement Vol2 Appendix

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Procurement Vol2 Appendix

Uploaded by

Copyright:

Available Formats

The World Bank and Public Procurement—An

Appendixes to Volume II: Achieving Development

Table A.1. Bank Procurement Methods—Adequacy of Competition

Source: IEG questionnaire.

Table A.3. Bank Procurement—Capturing Bidder Interest

Table A.5. Bank Procurement—Flexibility in Response

Table A.6. Bank Procurement—Incidence of Delay in Contract Award

Methodology for Assessing Procurement Systems Indicators and Scores

Table A.8. Bank NCB Requirements and Country Systems

Table A.9. Stage II Equivalence Analysis—Average Scores

Modifications as may be needed by

The United Kingdom

Focusing on cost reduction, notably both the transaction overhead costs of

The United States—“Best Value” Procurement

The tradeoff process is a competitive negotiation process where the purchaser

The European Union

Application of MEAT is a choice of the procuring agency—according to the

Asia-Pacific Economic Cooperation Forum

In 1999, APEC’s Government Procurement Experts Group identified the principles

Other aspects of APEC’s value for money approach include:

The International Atomic Energy Agency

• Integration of procurement strategies and planning with program

BROADER CONCEPTS OF VALUE FOR MONEY

Non-Price Factors and Preferences

The Australian provisions to use public procurement to strategically develop its

Reducing Cost through Competition and Negotiation

Competition is generally accepted, within value for money principles, as an effective

Strengthening Relationships with Suppliers

There is a concern that the traditional focus of public procurement on regulations

Supporting Contract Implementation

To summarize, the principle of value for money in procurement is being

Bank requirements do not accommodate the inclusion of social objectives through

The proposal is cautious in its ambitions. It says that it endeavors to retain

The toolbox approach is similar to the direction the Bank is suggesting – to

Several measures are proposed to promote and mandate e-procurement:

• Mandatory electronic notices

In addition to e-procurement, further suggestions to modernize, streamline,

• Shortened time limits for bidding

Summary of Previous Reports

Responding to INT’s concerns, Operations Policy and Country Services (OPCS)

EXAMPLES OF REVIEWS UNDERTAKEN FOR EACH CDD PROJECT

(Available for all 34 projects from IEG)

Sample Project 1: Afghanistan - Emergency National Solidarity Project -

• Block grants for communities ($94.3 million) to carry out subprojects

• Community Facilitation and Subproject Preparation ($9.7 million) to support

• Community Development Council Capacity Development ($14.4 million) to

Procurement Arrangements: A large majority of the works to be procured under

ISR 7: Anomalies in the MRRD Incremental Operating Cost Expenditure. The

• Lack of familiarity by MRRD staff of Bank Procurement Guidelines and

• Under the District School Window, 18 contracts were awarded to construct 8

- Based on the project documents, it appears that the community implemented

- Procurement problems almost exclusively related to non-CDD components

- The project scope and coverage experienced major changes during

Sample Project 2: Brazil—Maranhão Integrated Program: Rural Poverty Reduction Project

Extracts from Project Appraisal Document

• Highlighted Components: Community Subprojects ($36 million or 90 percent

Additional procurement of goods estimated to cost less than $100,000 equivalent, up

Concurrent Bank missions on financial management and regular field supervision at

The Quality Assurance Group assessment reviewed 23 projects with a combined

• Procurement is a major constraint to implementing ICT projects. Procurement

• The Bank lacks necessary ICT procurement-related expertise, leading to

• A possibly higher incidence of corruption in information technology

• Acknowledgement of recent improvements and agreed areas for future

Box B.1. FMIS Packages

Contract 1.42e-06* 1.55e-06* 1.52e-06* 1.47e-06* 1.46e-06***

NCB Method -68.19* -47.92 -57.44 -79.32* -64.61*