Quality in Design and Monitoring Frameworks (DMFS)

Quality in Design and Monitoring Frameworks (DMFs)
FINAL REPORT
Contract No. AO 8019 RSC No. C51894 (PHI)
Kenneth F. Smith Monitoring & Evaluation Consultant Operations Evaluation Department Asian Development Bank
March 31 2006
CONTENTS Page ACKNOWLEDGEMENTS ............................................................................................................. ii EXECUTIVE SUMMARY ............................................................................................................. iii I. II. III. IV. BACKGROUND & SCOPE ...............................................................................................1 FINDINGS .........................................................................................................................2 SUMMARY CONCLUSIONS ............................................................................................7 RECOMMENDATIONS .....................................................................................................7
APPENDIXES
1. 2. 3. 4. 5. 6. 7. 8.
Tables and Graphs Study Database Summaries Study Database Methodology and Limitations Rating System for Design and Monitoring Frameworks Quality Review and Assessment Checklist Terms of Reference Ballad of the Logframe
9 24 36 49 52 56 65 66
ii ACKNOWLEDGEMENTS I would like to thank Keith Leonard and Henry Tucker for their technical guidance, congeniality and flexibility in permitting me to conduct this relatively short-term study over a period of several months intermittently with some of my other consultancy activities. Id also particularly like to acknowledge the invaluable quality control assistance afforded me by Ms. Jocelyn Tubadeza. Her attention to detail caught several egregious errors in some earlier draft data so her feedback contributed immeasurably to the quality of this final product. Oscar Badiola and Brian Cafirma also deserve recognition for their administrative support throughout my study, which occasionally required considerable extensive and intensive efforts on their part to search, retrieve and/or reproduce numerous copies or extracts of documents for me at short notice. Thank you for the opportunity to work on this assignment, and again I look forward to future occasions to assist in this or related areas where my background in project design, implementation monitoring and/or evaluation techniques, skills and services can help further the objectives of ADB and its developing member countries in economic and social development.
iii EXECUTIVE SUMMARY The project/program design and monitoring framework (DMF) is the principal tool of the Asian Development Bank (ADB) for establishing quality-at-entry for its projects and programs. The DMF summarizes a proposed project/program design and is contained as an appendix in a key loan documentthe Report and Recommendation of the President (RRP) to the Board of Directors of ADB. The DMF also forms the basis for subsequent project/program monitoring of implementation activities and evaluation of results. The information contained in the DMF is also transcribed in other ADB project performance management system (PPMS) documentationnamely the project administration memorandum (PAM), project/program performance reports (PPRs) and technical assistance performance reports (TPRs), project completion reports (PCRs), and project/program performance evaluation reports (PPERs). During this consultation, the quality of ADB project and program loans, regional technical assistance (RETA) and advisory technical assistance (TA) loans & grants frameworks over a period of several yearsfrom 2000 to December 2005was assessed, compared, and trends noted. The prime focus in the review was the design summary statement for the project/program and TA at three levels: (i) impact (i.e. goal), (ii) outcome (i.e. purpose), and (iii) outputs; and related performance indicators and targets. Assumption and risk statements associated with each project/program proposal in both the DMF and RRP text were also reviewed, as well as the data source statements in the DMFs. In order to compare like with like, this study looked exclusively at DMFsnot DMFs and PPRs as in previous studies. The study also expanded the review to include assumptions & risks, and data sources. Otherwise the process, analytical procedure and rating scale used in this assessment was essentially the same as that used in similar ADB studies conducted in previous years. The findings are very encouraging. Since 2000, there has been continual improvement in the quality of almost all aspects of ADB DMFs for project/program loans and TAs. The weakest areas of DMFs are the developmental objective levelsi.e. Impacts and Outcomesand the same generic errors continue to appear as heretofore. Despite the commendable improvements noted, however, renewed efforts are still necessary on three fronts. (i) Quality assurance at entry for new project/program loans and technical assistance processing within Departments should be strengthened prior to ADB Board approval. Review and selective DMF retrofitting for on-going projects/programs is also necessary to enhance their utility for future results assessment, country and sector strategy programming. Concurrently, continual capacity building within ADB and the DMCs in PPMS concepts and related tools is also essential to ensure system sustainability for effective project/program management and measurement of development results.
(ii)
(iii)
I.
BACKGROUND & SCOPE
1. The project/program DMF is the principal tool of the Asian Development Bank (ADB) for establishing quality-at-entry for its projects and programs. The DMF summarizes a proposed project/program design and is contained as an appendix in a key loan documentthe Report and Recommendation of the President (RRP) to the Board of Directors of ADB. The DMF also forms the basis for subsequent project/program monitoring of implementation activities and evaluation of results. The information contained in the DMF is also transcribed in other ADB project performance management system (PPMS) documentationnamely the project administration memorandum (PAM), project/program performance reports (PPRs) and technical assistance performance reports (TPRs), project completion reports (PCRs), and project/program performance evaluation reports (PPERs).1 2. This study was undertaken for OED by the consultant during the period 7 Dec 2005 through 31 March 2006, under Contract No. A08019 RSC No. C51894 PHI, Quality in Design and Monitoring Frameworks. In order to compare like with like, this study looked exclusively at DMFsnot DMFs and PPRs as in previous studies. The study also expanded the review to include assumptions and risks, and data sources. Otherwise the process, analytical procedure and rating scale used in this assessment was essentially the same as that used in similar ADB studies conducted in previous years. A. Scope
3. During this consultation the quality of selected DMFsas of December 2005was assessed and compared to several prior years. Project/program loans, regional technical assistance (RETA) and advisory technical assistance (ADTA) loans & grants approved during years 2000, 2002, 2004, and 2005 were selected for review, as follows: Project/Program Loans 1. 2. 3. 4. Random sample of 25 project/program loans approved in 2000 Random sample of 25 approved in 2002 All (56) approved in 2004 All (57) approved in 2005 Regional Technical Assistance (RETA) 1. 2. 3. 4. Random sample of 25 approved in 2000 Random sample of 26 approved in 2002 All (66) approved in 2004 Random sample of 25 approved in 2005 Advisory Technical Assistance (ADTA) 1. 2. 3. 4. Random sample of 52 ADTAs approved in 2000 Random sample of 52 approved in 2002 All (130) approved in 2004 Random sample of 50 approved in 2005
Formerly Project Performance Audit Reports (PPARs).
2 4. The design summary statements and their related performance indicators and targets at three levelswere the prime focus of the review: (i) (ii) (iii) impact, (goal), long term objectives outcome, (purpose), immediate development objective outputs [i.e. deliverables]
5. The source of data for the indicators and targets, and the statement of assumptions and risks associated with each project/program proposal were also examined. 6. The prime objective of the review was to assess the current quality of the stated rationale for each project/program and technical assistance initiativei.e. the extent to which the output statements and resultant objectives reflected a coherent, measurable, approach to providing financial and technical assistance; as well as suitability of the DMF for subsequent implementation performance monitoring and objective analysis of results. 7. In accordance with item (vi) of the terms of reference, a checklist was also developed to facilitate OED review and quality assessment of DMFs. This was produced in two modes: (i) An MS word-file format for manual usewith 44 items that can be hand-tallied together with a supplementary DMF general reference criteria in a matrix format. An MS Excel worksheet formatwith the same 44 itemsfor automatic quantitative and qualitative assessment. II. FINDINGS
(ii)
8. Over the period of this study2000 to 2005the overall quality of DMFs2 for project and program design and monitoring has consistently improvedfrom 32% in 2000, to 83% in 2005 being deemed satisfactory or better (Figure 1). 9. In 2005, the improving trend in project/program DMF qualitycompared with 2004 continued, but was not statistically significant.3 10. Following some dissonance in 2002, a similar improvement trend occurred in the overall quality of both advisory and regional technical assistance DMFs. About half of technical assistance DMFsi.e. 64 % of RETAs, and 56 % of ADTAs respectivelyare now deemed satisfactory or better. 11. Improvement in advisory technical assistance DMFs in 2005 over 2004 was significant in almost all aspects. 12. On the other hand, significant improvement in 2005 was noted for only a few aspects of DMFs for regional technical assistance, compared with 2004.
2 3
And their precedent frameworks. Essentially due to the relatively high quality levels previously attained, coupled with the large margin for error that results from small population sizes.
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
Figure 1: ADB Design & Monitoring Framework Quality Percentage Satisfactory or Better OVERALL SUMMARY
2000
2002 RETA
2004 ADTA
2005 PROJECTS
13. However, despite these improvements in trends, the absolute quality levels for various aspects of technical assistance DMFs are still substantially lower than the rates attained by project/program loans. 14. Moreover, in the less-than-satisfactory DMFs of both projects/programs and TAs, the same generic errors continue to appear. Many design summary statements are at inappropriate levels, poorly articulated and/or inadequately supported i.e. lacking indicators, targets &/or timing for attainment. 15. Percentage resultsas well as further disaggregation, statistical significance, and rank order priority identification of areas for improvementof these findings are outlined in Tables 1 3. 16. The data are also graphed in a separate set of tables and graphs (Figures 126) contained in Appendix 1.
4 Table 1: DMF Quality Project and Program Loans

Percent Satisfactory or Better Rank order priority areas for improvement Statistically Significant
2000
2002
2004
2005
1 2 3 4 5 6 7
Sample Size Overall Summary Impact Summary Outcome Summary Output Summary Data Source Summary Assumptions and Risks Summary Impact Summary Impact Appropriate Level Impact Quality Impact Indicators Impact Targets Impact Timing Outcome Summary Outcome Appropriate Level Outcome Quality Outcome Indicators Outcome Targets Outcome Timing Output Summary Output Appropriate Level Output Quality Output Indicators Output Targets
25 32% 32% 32% 92% 84% 20%
25 52% 40% 52% 100% 92% 32%
56 71% 68% 68% 97% 91% 48%
57 83% 71% 79% 96% 77% 58%
2 4 5 3 1
3 8 9 10 11 12 4 13 14 15 16 17 5 18 19 20 21
32% 72% 60% 76% 32% 24% 32% 48% 32% 68% 40% 28% 92% 96% 92% 92% 88%
40% 72% 44% 60% 32% 36% 52% 40% 44% 64% 44% 48% 100% 100% 100% 100% 100%
68% 91% 66% 89% 63% 50% 68% 77% 72% 79% 61% 54% 97% 98% 98% 9% 77%
71% 88% 79% 84% 63% 61% 79% 74% 88% 82% 77% 68% 96% 100% 100% 91% 91%
2 5 3 4 2 1 4 2 5 4 3 1 5
1 1
Conclusion: General improvement trend but no statistically significant difference.
5 Table 2: DMF Quality Advisory Technical Assistance (ADTAs)

Percent Satisfactory or Better Rank order priority areas for improvement Statistically Significant
2000
2002
2004
2005
1 2 3 4 5 6 7
52 19% 21% 20% 66% 69% 23%
52 6% 12% 12% 74% 31% 21%
130 35% 29% 23% 75% 100 % 37%
50 64% 48% 60% 94% 100% 62% Yes 1 2 4 Yes Yes
Yes
3 8 9 10 11 12 4 13 14 15 16 17 5 18 19 20 21
21% 54% 54% 37% 19% 15% 20% 42% 31% 29% 19% 15% 66% 90% 87% 65% 37%
12% 67% 48% 33% 6% 12% 12% 46% 33% 27% 8% 10% 74% 96% 90% 73% 37%
29% 63% 48% 43% 19% 20% 23% 44% 34% 33% 20% 18% 75% 85% 85% 64% 48%
48% 82% 78% 66% 34% 40% 60% 74% 72% 84% 46% 44% 94% 98% 98% 90% 84%
1 5 4 3 1 2 2 4 3 5 2 1 4 3 3 2 1 Yes Yes
Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
CONCLUSION: General improvement trend and most are statistically significant differences.
Table 3: DMF Quality Regional Technical Assistance (RETAs)

Percent Satisfactory or Better 2000 2002 2004 2005 Rank order priority areas for improvement Statistically Significant
1 2 3 4 5 6 7
25 12% 12% 8% 48% 96% 28%
26 0% 0% 4% 57% 27% 35%
66 44% 14% 21% 85% 100% 42%
25 56% 44% 32% 80% 60% 76%
2 1 5 3 4
Yes
Yes Yes
3 8 9 10 11 12 4 13 14 15 16 17 5 18 19 20 21
12% 40% 20% 28% 8% 0% 8% 24% 36% 20% 8% 8% 48% 68% 64% 48% 28%
0% 38% 8% 23% 12% 8% 4% 19% 15% 23% 4% 0% 57% 88% 81% 54% 19%
14% 74% 47% 39% 6% 15% 21% 64% 45% 36% 12% 17% 85% 91% 91% 80% 59%
44% 96% 68% 48% 48% 28% 32% 60% 68% 56% 36% 20% 80% 100% 100% 72% 60%
2 5 4 2 2 1 1 4 5 3 2 1 5
Yes Yes
Yes
Yes Yes 2 1
CONCLUSION: General improvement trend and some are statistically significant differences.
7 III. SUMMARY CONCLUSIONS
17. Direct attribution of quality improvements in project/program and technical assistance DMFs over the past 2 years to particular ADB efforts is not feasible. Nevertheless, since the changes do coincide with several initiatives during this time period by ADB top management, OED, COSO and regional Departments to heighten staff awareness and utilization of the PPMSincluding continuing on-going traininga cause-effect linkage is plausible. 18. Since there is still need for some improvement in DMFsparticularly with identification of higher level development objectives to facilitate future evaluation of resultsthose efforts should be continued, and given renewed emphasis to achieve a higher plateau and ensure sustainability. IV. RECOMMENDATIONS
19. Continue initiatives for ADB departments to improve quality assurance-at-entry for new project/program loans, and technical assistance processing. (i) Require individuals responsible for preparing a DMF to certify its conformance with a standardized DMF checklist.4 (i.e. Individual self-check) Establish and staff a permanent PPMS specialist position within each department and RM to provide assistance to design teams and other members in preparing project/program DMFs. (i.e. Organizational Help Desk support function) Institute an authoritative organizational quality assurance point in the project/ program review process within each departmenteither the PPMS specialist acting alone, or a working group review board (with the PPMS specialist as one of the members)with recurrent responsibility to periodically review, screen and comment on DMF quality for action officers to rectify prior to writing the RRP text and circulating it for comment. (i.e. Organizational-check) Incorporate a standard requirement in the Terms of Reference for project appraisal preparation submissionsfor at least one consultant team member to have ADB DMF competency.
(ii)
(iii)
(iv)
20. Urge departments to conduct selective review and retrofitting of DMFs for on-going projects/programs as a special effortas well as in conjunction with mission reviewsto enhance their utility for country and sector strategy programming and future results assessment. 21. Continue to fosterand furthergeneral PPMS awareness; skills training; and competence in ADB, EAs, and TA consultants. (i) Make PPMS awareness training mandatory for all new personnel before they assume project/program design responsibilities, as well as incumbentsnot just on an ad hoc voluntary self-initiated basis, or only for selected individuals.
The checklist prepared for OED in this studyAppendix 6could also be utilized for this purpose.
8 (ii) Provide more extensive in-depth training in PPMS aspects to certifyor at least qualifydirect-hire personnel in each sector, department, and resident mission as PPMS specialiststo authoritatively assist other ADB and developing member country executing agency members with PPMS formulation requirements and review processes.
22. Utilize the checklist developed in this study to review, rate and provide feedback to department project officers during the project design phase.
Appendix 1
TABLES AND GRAPHS

Table A1.1: Project and Program Loans Statistically Percent Satisfactory, or Better 2000 2002 2004 2005 Rank Order (%) (%) (%) (%) Priority Areas for Significant Improvement 1 Overall Summary 2 Impact Summary 3 Outcome Summary 4 Output Summary 5 Data Source Summary 6 Assumptions and Risks Summary 2 Impact Summary 7 Impact Appropriate Level 8 Impact Quality 9 Impact Indicators 10 Impact Targets 11 Impact Timing 3 Outcome Summary 12 Outcome Appropriate Level 13 Outcome Quality 14 Outcome Indicators 15 Outcome Targets 16 Outcome Timing 4 Output Summary 17 Output Appropriate Level 18 Output Quality 19 Output Indicators 20 Output Targets 32 32 32 92 84 20 32 72 60 76 32 24 32 48 32 68 40 28 92 96 92 92 88 52 40 52 100 92 32 40 72 44 60 32 36 52 40 44 64 44 48 100 100 100 100 100 71 68 68 97 91 48 68 91 66 89 63 50 68 77 72 79 61 54 97 98 98 89 77 83 71 79 96 77 58 71 88 79 84 63 61 79 74 88 82 77 68 96 100 100 91 91
2 4 5 3 1
5 3 4 2 1
2 5 4 3 1 5
1 1
Conclusion: General Improvement Trend but No Statistically Significant Difference

Note: Sample sizes for 2000 is 25, 2002 is 25, 2004 is 56, and 2005 is 57. Source: Ken Smith, OED Project Evaluation Consultant, March 2006
10
Appendix 1
Table A1.2: Advisory Technical Assistance (ADTAs) Percent Satisfactory, or Better 2000 2002 2004 2005 Rank Order Statistically (%) (%) (%) (%) Priority Areas for Significant Improvement 1 2 3 4 5 6 2 7 8 9 10 11 3 12 13 14 15 16 4 17 18 19 20 Overall Summary Impact Summary Outcome Summary Output Summary Data Source Summary Assumptions and Risks Summary Impact Summary Impact Appropriate Level Impact Quality Impact Indicators Impact Targets Impact Timing Outcome Summary Outcome Appropriate Level Outcome Quality Outcome Indicators Outcome Targets Outcome Timing Output Summary Output Appropriate Level Output Quality Output Indicators Output Targets 19 21 20 66 69 23 21 54 54 37 19 15 20 42 31 29 19 15 66 90 87 65 37 6 12 12 74 31 21 12 67 48 33 6 12 12 46 33 27 8 10 74 96 90 73 37 35 29 23 75 100 37 29 63 48 43 19 20 23 44 34 33 20 18 75 85 85 64 48 64 48 60 94 100 62 48 82 78 66 34 40 60 74 72 84 46 44 94 98 98 90 84 Yes 1 2 4 3 1 5 4 3 1 2 2 4 3 5 2 1 4 3 3 2 1 Yes Yes Yes
Yes Yes
Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Conclusion: General Improvement Trend and Most are Statistically Significant Differences.
Note: Sample sizes for 2000 is 52, 2002 is 52, 2004 is 130, and 2005 is 50. Source: Ken Smith, OED Project Evaluation Consultant, March 2006.
Appendix 1
11
Table A1.3: Regional Technical Assistance (RETAs) Percent Satisfactory, or Better 2000 2002 2004 2005 Rank Order Statistically ($) ($) ($) ($) Priority Areas for Significant Improvement 1 Overall Summary 2 Impact Summary 3 Outcome Summary 4 Output Summary 5 Data Source Summary 6 Assumptions and Risks Summary 2 Impact Summary 7 Impact Appropriate Level 8 Impact Quality 9 Impact Indicators 10 Impact Targets 11 Impact Timing 3 Outcome Summary 12 Outcome Appropriate Level 13 Outcome Quality 14 Outcome Indicators 15 Outcome Targets 16 Outcome Timing 4 Output Summary 17 Output Appropriate Level 18 Output Quality 19 Output Indicators 20 Output Targets 12 12 8 48 96 28 12 40 20 28 8 4 57 27 35 44 14 21 85 100 42 14 74 47 39 6 15 21 64 45 36 12 17 85 91 91 80 59 56 44 32 80 60 76 44 96 68 48 48 28 32 60 68 56 36 20 80 100 100 72 60
2 1 5 3 4 2 5 4 2 2 1 1 4 5 3 2 1 5
Yes
Yes Yes Yes Yes
38 8 23 12 8 4 19 15 23 4
Yes
8 24 36 20 8 8 48 68 64 48 28
57 88 81 54 19
Yes Yes 2 1
Conclusion: General Improvement Trend and Some are Statistically Significant Differences.
Note: Sample sizes for 2000 is 25, 2002 is 26, 2004 is 66, and 2005 is 25. Source: Ken Smith, OED Project Evaluation Consultant, March 2007.
12
Appendix 1
Table A1.4: Percentage of Satisfactory or Better on Overall Summary
Item RETAs ADTAs Loans
2000 (%) 12 19 32
2002 (%) 0 6 52
2004 (%) 44 35 71
2005 (%) 56 64 83
Figure A1.1: Percentage of Satisfactory or Better on Overall Summary

100 90 80 70 60 50 40 30 20 10 0
2000
2002
RETA
2004
ADTA
2005
Loans
Source: Ken Smith, OED Project Evaluation Consultant, March 2006.
Table A1.5: Percentage of Satisfactory or Better on Overall Impact 2000 (%) 12 21 32 2002 (%) 0 12 40 2004 (%) 14 29 68 2005 (%) 44 48 71
Figure A1.2 Percentage of Satisfactory or Better on Overall Impact

100 90 80 70 60 50 40 30 20 10 0
2000 2002 2004 2005
RETAs
ADTAs
Loans
Appendix 1
13
Table A1.6: Percentage of Satisfactory or Better on Overall Outcome 2000 (%) 8 20 32 2002 (%) 4 12 52 2004 (%) 21 23 68 2005 (%) 32 60 79
Figure A1.3: Percentage Satisfactory or Better on Overall Outcome 100 90 80 70 60 50 40 30 20 10 0

2000 2002 2004 2005
RETAs
ADTAs
Loans
Table A1.7: Percentage of Satisfactory or Better on Overall Output 2000 (%) 48 66 92 2002 (%) 57 74 100 2004 (%) 85 75 97 2005 (%) 80 94 96
Figure A1.4: Percentage of Satisfactory or Better on Overall Output
100 90 80 70 60 50 40 30 20 10 0
2000 2002 2004 2005
RETAs
ADTAs
Loans
14
Appendix 1
Table A1.8: Percentage of Satisfactory or Better on Overall Data Source 2000 (%) 96 69 84 2002 (%) 27 31 92 2004 (%) 100 100 91 2005 (%) 60 100 77
Figure A1.5: Percentage of Satisfactory or Better on Overall Data Source 100 90 80 70 60 50 40 30 20 10 0

2000 2002
RETAs
2004
ADTAs
2005
Loans
Table A1.9: Percentage of Satisfactory or Better on Overall Assumptions and Risks 2000 (%) 28 23 20 2002 (%) 35 21 32 2004 (%) 42 37 48 2005 (%) 76 62 58
Figure A1.6: Percentage of Satisfactory or Better on Overall Assumptions and Risks

100 90 80 70 60 50 40 30 20 10 0 2000 2002 2004 2005
RETAs
ADTAs
Loans
Appendix 1
15
Table A1.10: Percentage of Satisfactory or Better on Impact Appropriate Levels 2000 (%) 40 54 72 2002 (%) 38 67 72 2004 (%) 74 63 91 2005 (%) 96 82 88
Figure A1.7: Percentage of Satisfactory or Better on Impact Appropriate Levels

100 90 80 70 60 50 40 30 20 10 0
2000
RETAs
2002
ADTAs
2004
2005
Loans
Table A1.11: Percentage of Satisfactory or Better on Impact Quality 2000 (%) 20 54 60 2002 (%) 8 48 44 2004 (%) 47 48 66 2005 (%) 68 78 79
Figure A1.8: Percentage of Satisfactory or Better on Impact Quality

100 90 80 70 60 50 40 30 20 10 0
2000
2002
RETAs
2004
ADTAs
2005
Loans
16
Appendix 1
Table A1.12: Percentage of Satisfactory or Better on Impact Indicators 2000 (%) 28 37 76 2002 (%) 23 33 60 2004 (%) 39 43 89 2005 (%) 48 66 84
Figure A1.9: Percentage of Satisfactory or Better on Impact Indicators

100 90 80 70 60 50 40 30 20 10 0
2000
2002
RETAs
2004
ADTAs
2005
Loans
Table A1.13: Percentage of Satisfactory or Better on Impact Targets 2000 (%) 8 19 32 2002 (%) 12 6 32 2004 (%) 6 19 63 2005 (%) 48 34 63
Figure A1.10: Percentage of Satisfactory or Better on Impact Targets

100 90 80 70 60 50 40 30 20 10 0 2000 2002 RETAs 2004 ADTAs 2005 Loans
Appendix 1
17
Table A1.14: Percentage of Satisfactory or Better on Impact Timing 2000 (%) 0 15 24 2002 (%) 8 12 36 2004 (%) 15 20 50 2005 (%) 28 40 61
Figure A1.11: Percentage of Satisfactory or Better on Impact Timing

100 90 80 70 60 50 40 30 20 10 0
2000
2002
RETAs
2004
ADTAs
2005
Loans
Table A1.15: Percentage of Satisfactory or Better on Outcome Appropriate Level Item RETAs ADTAs Loans 2000 (%) 24 42 48 2002 (%) 19 46 40 2004 (%) 64 44 77 2005 (%) 60 74 74
Figure A1.12: Percentage Satisfactory or Better on Outcome Appropriate Level

100 90 80 70 60 50 40 30 20 10 0
2000
2002
RETAs
2004
ADTAs
2005
Loans
18
Appendix 1
Table A1.16: Percentage of Satisfactory or Better on Outcome Quality 2000 (%) 36 31 32 2002 (%) 15 33 44 2004 (%) 45 34 72 2005 (%) 68 72 88
Figure A1.13: Percentage of Satisfactory or Better on Outcome Quality

100 90 80 70 60 50 40 30 20 10 0
2000
RETAs
2002
ADTAs
2004
2005
Loans
Table A1.17: Percentage of Satisfactory or Better on Outcome Indicators 2000 (%) 20 29 68 2002 (%) 23 27 64 2004 (%) 36 33 79 2005 (%) 56 84 82
Figure A1.14: Percentage Satisfactory or Better on Outcome Indicators

100 90 80 70 60 50 40 30 20 10 0
2000
RETAs
2002
2004
ADTAs
2005
Loans
Appendix 1
19
Table A1.18: Percentage of Satisfactory or Better on Outcome Targets 2000 (%) 8 19 40 2002 (%) 4 8 44 2004 (%) 12 20 61 2005 (%) 36 46 77
Figure A1.15: Percentage Satisfactory or Better on Outcome Targets

100 90 80 70 60 50 40 30 20 10 0 2000 RETAs 2002 ADTAs 2004 Loans 2005
Table A1.19: Percentage of Satisfactory or Better on Outcome Timing 2000 (%) 8 15 28 2002 (%) 0 10 48 2004 (%) 17 18 54 2005 (%) 20 44 68
Figure A1.16: Percentage Satisfactory or Better on Outcome Timing

100 90 80 70 60 50 40 30 20 10 0
2000 RETAs
2002 ADTAs
2004 Loans
2005
20
Appendix 1
Table A1.20: Percentage of Satisfactory or Better on Output Appropriate Level Item RETAs ADTAs Loans 2002 (%) 68 90 96 2002 (%) 88 96 100 2004 (%) 91 85 98 2005 (%) 100 98 100
Figure A1.17: Percentage of Satisfactory or Better on Output Appropriate Level

100 90 80 70 60 50 40 30 20 10 0
2000
2002
RETAs
2004
ADTAs
2005
Loans
Table A1.21: Percentage of Satisfactory or Better on Output Quality 2000 (%) 64 87 92 2002 (%) 81 90 100 2004 (%) 91 85 98 2005 (%) 100 98 100
Figure A1.18: Percentage of Satisfactory or Better on Output Quality

100 90 80 70 60 50 40 30 20 10 0
2000 RETAs
2002 ADTAs
2004 Loans
2005
Appendix 1
21
Table A1.22: Percentage of Satisfactory or Better on Output Indicators 2000 (%) 48 65 92 2002 (%) 54 73 100 2004 (%) 80 64 89 2005 (%) 72 90 91
Figure A1.19: Percentage of Satisfactory or Better on Output Indicators

Table A1.23: Percentage of Satisfactory or Better on Output Targets 2000 (%) 28 37 88 2002 (%) 19 37 100 2004 (%) 59 48 77 2005 (%) 60 84 91
Figure A1.20: Percentage of Satisfactory or Better on Output Targets

22
Appendix 1
Table A1.24: Percentage of Satisfactory or Better on Impact for 2005 App Level Quality (%) Indicators (%) (%) 96 68 48 82 78 66 88 79 84 Targets (%) 48 34 63 Timing (%) 28 40 61
Figure A1.21: Percentage of Satisfactory or Better on Impact for 2005

100 90 80 70 60 50 40 30 20 10 0 App Level Quality RETAs Indicators ADTAs Targets Loans Timing
Table A1.25: Percentage of Satisfactory or Better on Outcome for 2005 App Level Quality (%) Indicators (%) (%) 60 68 56 74 72 84 74 88 82 Targets (%) 36 46 77 Timing (%) 20 44 68
Figure A1.22: Percentage of Satisfactory or Better on Outcome for 2005

100
80
60
40
20
0 App Level Quality RETAs Indicators ADTAs Targets Loans Timing
Appendix 1
23
Table A1.26: Percentage of Satisfactory or Better on Outputs for 2005 App Level (%) 100 98 100 Quality (%) 100 98 100 Indicators (%) 72 90 91 Targets (%) 60 84 91
Figure A1.23: Percentage of Satisfactory or Better on Outputs for 2005

100 90 80 70 60 50 40 30 20 10 0 App Level Quality Indicators ADTAs Targets LOANS
RETAs
% of 3s % of 2s % of 1s % of 0s
% Impact Statement Level Impact Statement Quality Impact Indicators Impact Targets Impact Timing Outcome Statement Level Outcome Statement Quality Outcome Indicators Outcome Targets Outcome Timing Output Statement Level Output Statement Quality Output Indicators Output Targets No MOVs (Data Sources) Identified for Impact, Outcome or Output level Very Generic non-specific MOVs for Outcome & Impact Levels Inappropriate MOV for Impact &/or Outcome Level Frequency of Data Collection not cited for Outcome Level Surveys # of ASSUMPTIONS # of RISKS No Risks; or Risks are not specifically ID'd in Framework Some Duplication of Assumptions or Risks Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints Assumptions or Risks are internal to project management or at the Wrong Level Structure* FRAMEWORK A&R QUALITY RATING condition exists Average # condition exists
Count of 3s Count of 2s Count of 1s Count of 0s 17 1 4 3 25 14 1 7 3 25 19 0 2 4 25 8 0 0 17 25 6 0 1 18 25 10 2 9 4 25 5 3 12 5 25 17 0 2 6 25 IMPACT RATING
% % % % % % % %
24
Appendix 2
OUTCOME RATING
10 0 1 14 25 7 0 0 18 25 24 0 0 1 25 23 0 1 1 25
% % % % %
OUTPUT RATING
68 56 76 32 24 40 20 68 40 28 96 92 92 88 4 4 0 0 0 8 12 0 0 0 0 0 0 0 16 28 8 0 4 36 48 8 4 0 0 4 4 4 12 12 16 68 72 16 20 24 56 72 4 4 4 8 100 100 100 100 100 100 100 100 100 100 100 100 100 100 23 22 0 0 1 1 1 2 25 25
% % % %
STUDY DATABASE SUMMARIES
Table A2.1: Database 2000 Project Loans
FRAMEWORK Means of Verification ASSUMPTIONS & RISKS (Data Source) RATING RATING Enter "1" if Count A&R Then Enter "1" if
% 21 2 Check Check % of 3s % of 2s % of 1s % of 0s
% % % % % %
Key Risks in RRP Text are not in Framework
FRAMEWORK & RRP ASS & RISK COMPATIBILITY
Enter "1" if condition exists A&R FRAMEWORK & RRP COMPATIBILITY RATING GOAL LEVEL RATING
Count of 3s Count of 2s Count of 1s Count of 0s 4 4 13 4 25 0 22 8 1 12 1 5 1 25 25
OUTCOME LEVEL RATING OUTPUT LEVEL RATING MOV (DS) RATING
16 0 88 28 16 32 4 56 52 48 4 16 16 20 4 0 100 100 100 100 12 0 8 32 12 68 68 0 100 100

% % % %
NOTE: Five Factors Weighted Equally
7 14 4 0 25 3 2 3 17 25 0 8 17 0 25
Assumptions & Risks RATING OVERALL SUMMARY RATING (All Five Factors)
% of 3s % of 2s % of 1s % of 0s Check Count of 3s Count of 2s Count of 1s Count of 0s Check

Impact Statement Level Impact Statement Quality Impact Indicators Impact Targets Impact Timing Outcome Statement Level Outcome Statement Quality Outcome Indicators Outcome Targets Outcome Timing Output Statement Level
17 1 3 4 25 11 0 10 4 25 15 0 5 5 25 8 0 2 15 25 9 0 0 16 25 9 1 7 8 25 10 1 6 8 25 16 0 4 5 25 IMPACT RATING
% 68% 4% 12% 16% 1 % 44% 0% 40% 16% 1 % 60% 0% 20% 20% 1 % 32% 0% 8% 60% 1 % 36% 0% 0% 64% 1 % 36% 4% 28% 32% 1 % 40% 4% 24% 32% 1 % 64% 0% 16% 20% 1 11 0 3 11 25 % 44% 0% 12% 44% 1 12 0 0 13 25 25 0 0 0 25 25 0 0 0 25 25 0 0 0 25 25 0 0 0 25 % % % % % 48% 100% 100% 100% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 52% 0% 0% 0% 0% 1 1 1 1 1 % % % % % Average # 19 2 Check Check % of 3s % of 2s % of 1s % of 0s % 16% 24% 36% 24% 1 % % % % % 20% 100% 60% 8% 0% % 32% 0% 32% 24% 52% % 16% 0% 8% 28% 48% % 32% 0% 0% 40% 0% % 1 1 1 1 1 % Count of 3s Count of 2s Count of 1s Count of 0s 4 6 9 6 25 5 8 4 8 25 25 0 0 0 25 15 8 2 0 25 2 6 7 10 25 0 13 12 0 25
# of RISKS No Risks; or Risks are not specifically ID'd in Framework Some Duplication of Assumptions or Risks Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints Assumptions or Risks are internal to project management or at the Wrong Level Structure* FRAMEWORK A&R QUALITY RATING Output Indicators Output Targets No MOVs (Data Sources) Identified for Impact, Outcome or Output level Very Generic non-specific MOVs for Outcome & Impact Levels Inappropriate MOV for Impact &/or Outcome Level Frequency of Data Collection not cited for Outcome Level Surveys # of ASSUMPTIONS GOAL LEVEL RATING OUTCOME LEVEL RATING OUTPUT LEVEL RATING MOV (DS) RATING Assumptions & Risks RATING OVERALL SUMMARY RATING (All Five Factors) Output Statement Quality Key Risks in RRP Text are not in Framework A&R FRAMEWORK & RRP COMPATIBILITY RATING
Source: Ken Smith, OED Project Evaluation Consultant, March 2006
OUTCOME RATING OUTPUT RATING Means of Verification (Data Source) RATING Enter "1" if condition exists FRAMEWORK ASSUMPTIONS & RISKS RATING
Count A&R Then Enter "1" if condition exists Enter "1" if condition exists NOTE: Five Factors Weighted Equally
Table A2.2: Database 2002 Loans

Appendix 1
25

Impact Statement Level Impact Statement Quality Impact Indicators Impact Targets Impact Timing Outcome Statement Level Outcome Statement Quality Outcome Indicators Outcome Targets Outcome Timing Output Statement Level Output Statement Quality Output Indicators Output Targets No MOVs (Data Sources) Identified for Impact, Outcome or Output level
IMPACT RATING
47 37 50 35 28 4 0 0 0 0 1 17 2 5 0 4 2 4 16 28 56 56 56 56 56 % % % % % 84% 66% 89% 63% 50% 7% 0% 0% 0% 0% 2% 30% 4% 9% 0% 7% 4% 7% 29% 50% 1 1 1 1 1 33 10 8 5 56 % 59% 18% 14% 9% 1 34 43 6 1 12 8 4 4 56 56 % % 61% 77% 11% 2% 21% 14% 7% 7% 1 1 33 29 55 55 50 1 1 0 0 0 11 0 1 1 6 11 26 0 0 0 56 56 56 56 56 % % % % % 59% 52% 98% 98% 89% 2% 2% 0% 0% 0% 20% 0% 2% 2% 11% 20% 46% 0% 0% 0% 1 1 1 1 1 43 0 6 7 56 % 77% 0% 11% 13% 1 % % % % % Average # 8 2 Check Check % of 3s % of 2s % of 1s % of 0s Count of 3s Count of 2s Count of 1s Count of 0s 20 18 13 5 56 % 36% 32% 23% 9% 1 11 27 12 6 56 % 20% 48% 21% 11% 1 43 27 4 0 11 24 23 40 1 4 6 16 1 1 23 0 56 56 56 56 % % % % 77% 48% 7% 0% % 20% 43% 41% 71% % 2% 7% 11% 29% % 2% 2% 41% 0% % 1 1 1 1 %
# of RISKS No Risks; or Risks are not specifically ID'd in Framework Some Duplication of Assumptions or Risks Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints Assumptions or Risks are internal to project management or at the Wrong Level Structure* FRAMEWORK A&R QUALITY RATING MOV (DS) RATING Assumptions & Risks RATING # of ASSUMPTIONS Count A&R Then Enter "1" if condition exists GOAL LEVEL RATING OUTPUT LEVEL RATING
26
Appendix 2
Source: Ken Smith, OED Consultant, Feb 2005
OUTCOME RATING OUTPUT RATING

Very Generic non-specific MOVs for Outcome & Impact Levels Inappropriate MOV for Impact &/or Outcome Level Frequency of Data Collection not cited for Outcome Level Surveys
Means of Verification (Data Source) RATING Enter "1" if condition

exists Enter "1" if condition exists Key Risks in RRP Text are not in Framework
FRAMEWORK ASSUMPTIONS & RISKS RATING FRAMEWORK & RRP ASS & RISK COMPATIBILITY
A&R FRAMEWORK & RRP COMPATIBILITY RATING
OUTCOME LEVEL RATING
OVERALL SUMMARY RATING (All Five Factors)
% of 3s % of 2s % of 1s % of 0s
Count of 3s Count of 2s Count of 1s Count of 0s 48 2 6 1 57 45 0 11 1 57 48 0 7 2 57 36 0 6 15 57 35 0 0 22 57 IMPACT RATING
% % % 84% 79% 84% 4% 0% 0% 11% 19% 12% 2% 2% 4% 1 1 1 40 2 13 2 57 OUTCOME RATING 49 1 5 2 57 47 0 9 1 57 44 0 2 11 57 39 0 0 18 57 OUTPUT RATING 57 0 0 0 57
Output Statement Quality
Source: Ken Smith, OED Consultant, Dec 2005.
57 0 0 0 57 52 0 3 2 57 52 0 3 2 57
Output Indicators Output Targets No MOVs (Data Sources) Identified for Impact, Outcome or Output level
% % % % % % % % % % % 63% 61% 70% 86% 82% 77% 68% 100% 100% 91% 91% 0% 0% 4% 2% 0% 0% 0% 0% 0% 0% 0% 11% 0% 23% 9% 16% 4% 0% 0% 0% 5% 5% 26% 39% 4% 4% 2% 19% 32% 0% 0% 4% 4% 1 1 1 1 1 1 1 1 1 1 1 % % % % % Average # 14 6 Check Check % % % % of 3s 46% 44% 89% % of 2s 25% 35% 7% % of 1s 21% 18% 4% % of 0s 9% 4% 0% 1 1 1 % 30% 47% 19% 4% 1 % % 11% 4% 47% 77% 25% 18% 18% 2% 1 1 % % % % % Count of 3s Count of 2s Count of 1s Count of 0s 26 14 12 5 57 25 20 10 2 57 51 4 2 0 57 17 27 11 2 57 6 27 14 10 57 2 44 10 1 57
# of ASSUMPTIONS # of RISKS GOAL LEVEL RATING OUTCOME LEVEL RATING OUTPUT LEVEL RATING MOV (DS) RATING Assumptions & Risks RATING
Very Generic non-specific MOVs for Outcome & Impact Levels Inappropriate MOV for Impact &/or Outcome Level Frequency of Data Collection not cited for Outcome Level Surveys

exists
No Risks; or Risks are not specifically ID'd in Framework Some Duplication of Assumptions or Risks Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints
FRAMEWORK ASSUMPTIONS & RISKS RATING Count
A&R Then Enter "1" if condition exists Assumptions or Risks are internal to project management or at the Wrong Level Structure* FRAMEWORK A&R QUALITY RATING
Enter "1" if condition exists
A&R FRAMEWORK & RRP COMPATIBILITY RATING NOTE: Five Factors Weighted Equally OVERALL SUMMARY RATING (All Five Factors)
Appendix 1
27
% of 3s % of 2s % of 1s % of 0s Check % 54% 0% 2% 44% 1

Impact Statement Level Impact Statement Quality Impact Indicators Impact Targets Impact Timing Outcome Statement Level Outcome Statement Quality Outcome Indicators Outcome Targets Outcome Timing Output Statement Level Output Statement Quality Output Indicators Output Targets
Count of 3s Count of 2s Count of 1s Count of 0s Check 28 0 1 23 52 IMPACT RATING 16 12 0 24 52 19 0 1 32 52 10 0 1 41 52 8 0 0 44 52
% % % % 31% 37% 19% 15% 23% 0% 0% 0% 0% 2% 2% 0% 46% 62% 79% 85% 1 1 1 1 21 1 5 25 52 % 40% 2% 10% 48% 1 14 2 11 25 52 % 27% 4% 21% 48% 1 15 0 5 32 52 10 0 1 41 52 8 0 0 44 52 47 0 0 5 52 45 0 1 6 52 % % % % % 29% 19% 15% 90% 87% 0% 0% 0% 0% 0% 10% 2% 0% 0% 2% 62% 79% 85% 10% 12% 1 1 1 1 1 34 0 7 11 52 % 65% 0% 13% 21% 1 % 37% 0% 15% 48% 1 % % % % % Average # 9 1 Check Check % % % % of 3s 13% 12% 37% % of 2s 8% 8% 29% % of 1s 21% 23% 23% % of 0s 58% 58% 12% 1 1 1 % 31% 38% 15% 15% 1 % 4% 19% 12% 65% 1 % 0% 19% 50% 31% 1 % % % % % Count of 3s Count of 2s Count of 1s Count of 0s 7 4 11 30 52 6 4 12 30 52 19 15 12 6 52 16 20 8 8 52 2 10 6 34 52 0 10 26 16 52 19 0 8 25 52
# of RISKS No Risks; or Risks are not specifically ID'd in Framework Some Duplication of Assumptions or Risks Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints Assumptions or Risks are internal to project management or at the Wrong Level Structure* FRAMEWORK A&R QUALITY RATING condition exists MOV (DS) RATING Assumptions & Risks RATING OVERALL SUMMARY RATING (All Five Factors) # of ASSUMPTIONS GOAL LEVEL RATING OUTCOME LEVEL RATING OUTPUT LEVEL RATING
28
Appendix 2
OUTCOME RATING OUTPUT RATING
No MOVs (Data Sources) Identified for Impact, Outcome or Output level Very Generic non-specific MOVs for Outcome & Impact Levels Inappropriate MOV for Impact &/or Outcome Level Frequency of Data Collection not cited for Outcome Level Surveys
Table A2.5: Database 2000 ADTAs
Means of Verification FRAMEWORK ASSUMPTIONS (Data Source) & RISKS RATING RATING Enter "1" Count A&R Then Enter "1" if
if condition exists
A&R FRAMEWORK & RRP COMPATIBILITY RATING NOTE: Five Factors Weighted Equally

Impact Statement Level
35 0 1 16 52
Impact Statement Quality
% 67% 0% 2% 31% 1 25 0 11 16 52
Impact Indicators
IMPACT RATING
% 48% 0% 21% 31% 1 17 0 9 26 52

Impact Targets
% 33% 0% 17% 50% 1 3 0 3 46 52 % 6% 0% 6% 88% 1 % 12% 0% 0% 88% 1

Impact Timing
6 0 0 46 52
Outcome Statement Level
23 1 3 25 52
Outcome Statement Quality
% 44% 2% 6% 48% 1 16 1 10 25 52
Outcome Indicators
OUTCOME RATING
% 31% 2% 19% 48% 1 14 0 6 32 52 4 0 2 46 52 5 0 0 47 52 50 0 0 2 52 47 0 2 3 52 38 0 7 7 52 19 0 9 24 52 % 27% 0% 12% 62% 1 % 8% 0% 4% 88% 1 % 10% 0% 0% 90% 1 % 96% 0% 0% 4% 1 % 90% 0% 4% 6% 1 % 73% 0% 13% 13% 1 % 37% 0% 17% 46% 1 % % % % %
Average # Output Targets Outcome Targets Outcome Timing Output Statement Level Output Indicators
OUTPUT RATING
No MOVs (Data Sources) Identified for Impact, Outcome or Output level Very Generic non-specific MOVs for Outcome & Impact Levels Inappropriate MOV for Impact &/or Outcome Level Frequency of Data Collection not cited for Outcome Level Surveys
if condition exists
Means of Verification FRAMEWORK ASSUMPTIONS (Data Source) & RISKS RATING RATING Enter "1" Count A&R Then Enter "1" if
10 1 Check
# of ASSUMPTIONS # of RISKS No Risks; or Risks are not specifically ID'd in Framework Some Duplication of Assumptions or Risks Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints Assumptions or Risks are internal to project management or at the Wrong Level Structure* FRAMEWORK A&R QUALITY RATING condition exists
Check % of 3s % of 2s % of 1s % of 0s % 4% 8% 52% 37% 1 % 6% 6% 35% 54% 1 % 37% 37% 21% 6% 1 % 8% 23% 42% 27% 1 % 4% 17% 21% 58% 1 % 0% 6% 65% 29% 1 % % % % %
Count of 3s Count of 2s Count of 1s Count of 0s 2 4 27 19 52 3 3 18 28 52 19 19 11 3 52 4 12 22 14 52 2 9 11 30 52 0 3 34 15 52
A&R FRAMEWORK & RRP COMPATIBILITY RATING GOAL LEVEL RATING
OUTPUT LEVEL RATING
MOV (DS) RATING
Assumptions & Risks RATING
OVERALL SUMMARY RATING (All Five Factors)
Appendix 1
29

Means of Verification (Data Source) RATING Enter "1" if condition IMPACT RATING OUTCOME RATING OUTPUT RATING
exists Very Generic non-specific MOVs for Outcome & Impact Levels Frequency of Data Collection not cited for Outcome Level Surveys
30
FRAMEWORK ASSUMPTIONS & RISKS RATING

Count A&R Then Enter "1" if condition exists Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints

Enter "1" if condition exists A&R FRAMEWORK & RRP COMPATIBILITY RATING NOTE: Five Factors Weighted Equally OVERALL SUMMARY RATING (All Five Factors)
Appendix 2
Inappropriate MOV for Impact &/or Outcome Level
Assumptions or Risks are internal to project management or at the Wrong Level Structure*
No Risks; or Risks are not specifically ID'd in Framework
No MOVs (Data Sources) Identified for Impact, Outcome or Output level
Some Duplication of Assumptions or Risks
FRAMEWORK A&R QUALITY RATING
Count of 3s 80 61 56 25 26 54 39 43 25 24 110 110 83 62 Count of 2s 1 1 0 0 0 3 5 0 1 0 0 0 0 0 Count of 1s 7 29 19 17 0 10 24 25 16 1 3 3 23 21 Count of 0s 42 39 55 88 104 63 62 62 88 105 17 17 24 47 Check 130 130 130 130 130 130 130 130 130 130 130 130 130 130 % 62% 1% 5% 32% 1 % 47% 1% 22% 30% 1 % 43% 0% 15% 42% 1 % % % 19% 20% 42% 0% 0% 2% 13% 0% 8% 68% 80% 48% 1 1 1 % 30% 4% 18% 48% 1 % 33% 0% 19% 48% 1 % % % % 19% 18% 85% 85% 1% 0% 0% 0% 12% 1% 2% 2% 68% 81% 13% 13% 1 1 1 1 % 64% 0% 18% 18% 1 % 48% 0% 16% 36% 1
Average # Check 9 % % % % % 2
Count of 3s Count of 2s Count of 1s Count of 0s
11 10 60 125 3 0 27 20 38 5 46 45 43 30 13 0 33 67 49 70 19 0 48 18 130 130 130 130 130 130 % 2% 35% 25% 37% 1 % 0% 35% 52% 14% 1
% of 3s % of 2s % of 1s % of 0s Check
Check
% % % % % of 3s 8% 8% 46% 96% % of 2s 21% 15% 29% 4% % of 1s 33% 23% 10% 0% % of 0s 38% 54% 15% 0% 1 1 1 1
Assumptions & Risks RATING
OUTPUT LEVEL RATING
Output Statement Level
GOAL LEVEL RATING
# of ASSUMPTIONS
MOV (DS) RATING
Outcome Indicators
Outcome Targets
Impact Indicators
Output Indicators
Outcome Timing
Impact Targets
Output Targets
Impact Timing
# of RISKS
% % % % %
Source: Ken Smith, OED Consultant, Feb 2005.
% of 3s % of 2s % of 1s % of 0s Count of 3s Count of 2s Count of 1s Count of 0s

41 0 4 5 50
% 82% 0% 8% 10% 1 39 0 6 5 50
Impact Indicators
IMPACT RATING
% 78% 0% 12% 10% 1 33 0 11 6 50

Impact Targets
% 66% 0% 22% 12% 1 17 0 6 27 50 % 34% 0% 12% 54% 1 % 40% 0% 0% 60% 1

Impact Timing
20 0 0 30 50 34 3 8 5 50 35 1 9 5 50 42 0 5 3 50 23 0 5 22 50 22 0 1 27 50 49 0 0 1 50 49 0 0 1 50 45 0 3 2 50 42 0 3 5 50
Outcome Indicators Outcome Targets Outcome Timing Output Indicators Output Targets
% 68% 6% 16% 10% 1 % 70% 2% 18% 10% 1 % 84% 0% 10% 6% 1 % 46% 0% 10% 44% 1 % 44% 0% 2% 54% 1 % 98% 0% 0% 2% 1 % 98% 0% 0% 2% 1 % 90% 0% 6% 4% 1 % 84% 0% 6% 10% 1 % % % % % Average # 7 3 Check Check % of 3s % of 2s % of 1s % of 0s % 26% 22% 38% 14% 1 % % % 26% 84% 60% 34% 10% 40% 26% 4% 0% 14% 2% 0% 1 1 1 % % 18% 8% 44% 56% 12% 36% 26% 0% 1 1 % % % % % Count of 3s Count of 2s Count of 1s Count of 0s 13 11 19 7 50 13 17 13 7 50 42 5 2 1 50 30 20 0 0 50 9 22 6 13 50 4 28 18 0 50
# of RISKS
OUTCOME RATING
Output Statement Level
OUTPUT RATING
No MOVs (Data Sources) Identified for Impact, Outcome or Output level Very Generic non-specific MOVs for Outcome & Impact Levels Inappropriate MOV for Impact &/or Outcome Level Frequency of Data Collection not cited for Outcome Level Surveys # of ASSUMPTIONS exists No Risks; or Risks are not specifically ID'd in Framework Some Duplication of Assumptions or Risks Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints Assumptions or Risks are internal to project management or at the Wrong Level Structure* FRAMEWORK A&R QUALITY RATING

Means of Verification (Data Source) RATING Enter "1" if condition FRAMEWORK ASSUMPTIONS & RISKS RATING
Count A&R Then Enter "1" if condition exists
A&R FRAMEWORK & RRP COMPATIBILITY RATING GOAL LEVEL RATING OUTCOME LEVEL RATING OUTPUT LEVEL RATING MOV (DS) RATING Assumptions & Risks RATING OVERALL SUMMARY RATING (All Five Factors) NOTE: Five Factors Weighted Equally
Appendix 1
31

Impact Statement Level Impact Statement Quality Impact Indicators Impact Targets
8 2 2 13 25 IMPACT RATING 5 0 6 14 25 7 0 2 16 25 2 0 1 22 25
Impact Timing
% 32% 8% 8% 52% 1 0 0 0 25 25 % % % % 20% 28% 8% 0% 0% 0% 0% 0% 24% 8% 4% 0% 56% 64% 88% 100% 1 1 1 1 % 20% 4% 32% 44% 1 5 1 8 11 25 7 2 4 12 25 5 0 6 14 25 2 0 1 22 25 2 0 0 23 25 17 0 0 8 25 15 1 1 8 25 % % % % % % 28% 20% 8% 8% 68% 60% 8% 0% 0% 0% 0% 4% 16% 24% 4% 0% 0% 4% 48% 56% 88% 92% 32% 32% 1 1 1 1 1 1 12 0 4 9 25 % % 48% 28% 0% 0% 16% 8% 36% 64% 1 1 % % % % % Average # 6 2 Check Check % % % % % of 3s 0% 4% 24% 52% % of 2s 12% 4% 24% 44% % of 1s 16% 28% 20% 0% % of 0s 72% 64% 32% 4% 1 1 1 1 % 0% 28% 48% 24% 1 % 0% 12% 56% 32% 1 % % % % % Count of 3s Count of 2s Count of 1s Count of 0s 0 3 4 18 25 1 1 7 16 25 6 6 5 8 25 13 11 0 1 25 0 7 12 6 25 0 3 14 8 25 7 0 2 16 25
Output Targets No MOVs (Data Sources) Identified for Impact, Outcome or Output level Very Generic non-specific MOVs for Outcome & Impact Levels Inappropriate MOV for Impact &/or Outcome Level Frequency of Data Collection not cited for Outcome Level Surveys # of ASSUMPTIONS # of RISKS No Risks; or Risks are not specifically ID'd in Framework Some Duplication of Assumptions or Risks Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints Assumptions or Risks are internal to project management or at the Wrong Level Structure* FRAMEWORK A&R QUALITY RATING Outcome Statement Level Outcome Statement Quality Outcome Indicators Outcome Targets Outcome Timing Output Statement Level Output Statement Quality Output Indicators Key Risks in RRP Text are not in Framework GOAL LEVEL RATING OUTCOME LEVEL RATING OUTPUT LEVEL RATING MOV (DS) RATING Assumptions & Risks RATING OVERALL SUMMARY RATING (All Five Factors)
32
Appendix 2
OUTCOME RATING OUTPUT RATING Means of Verification (Data Source) RATING Enter "1" if condition exists FRAMEWORK ASSUMPTIONS & RISKS RATING
Count A&R Then Enter "1" if condition exists Enter "1" if condition exists A&R FRAMEWORK & RRP COMPATIBILITY RATING NOTE: Five Factors Weighted Equally
Table A2.9: Database 2000 RETAs


Impact Statement Level Impact Statement Quality Impact Indicators Impact Targets Impact Timing Outcome Statement Level Outcome Statement Quality Outcome Indicators Outcome Targets
10 0 4 12 26 2 0 12 12 26 6 0 7 13 26 3 0 0 23 26 2 0 0 24 26 IMPACT RATING
% % 38% 8% 0% 0% 15% 46% 46% 46% 1 1 % % % 23% 12% 8% 0% 0% 0% 27% 0% 0% 50% 88% 92% 1 1 1 5 0 9 12 26 % 19% 0% 35% 46% 1 4 0 10 12 26 % 15% 0% 38% 46% 1 6 0 6 14 26 1 0 1 24 26
Outcome Timing
OUTCOME RATING
0 0 0 26 26 23 0 0 3 26 21 0 2 3 26 14 0 9 3 26 5 0 4 17 26
Output Statement Level Output Statement Quality Output Indicators Output Targets No MOVs (Data Sources) Identified for Impact, Outcome or Output level Very Generic non-specific MOVs for Outcome & Impact Levels Inappropriate MOV for Impact &/or Outcome Level Frequency of Data Collection not cited for Outcome Level Surveys # of ASSUMPTIONS
% % % % % 23% 4% 0% 88% 81% 0% 0% 0% 0% 0% 23% 4% 0% 0% 8% 54% 92% 100% 12% 12% 1 1 1 1 1 % 54% 0% 35% 12% 1 % 19% 0% 15% 65% 1 % % % % % Average # 8 2 Check Check % % % of 3s 0% 0% % of 2s 0% 4% % of 1s 42% 19% % of 0s 58% 77% 1 1 % 19% 38% 31% 12% 1 % 8% 19% 62% 12% 1 % % 8% 0% 27% 0% 12% 65% 54% 35% 1 1 % % % % % Count of 3s Count of 2s Count of 1s Count of 0s 0 0 11 15 26 0 1 5 20 26 5 10 8 3 26 2 5 16 3 26 2 7 3 14 26 0 0 17 9 26
# of RISKS GOAL LEVEL RATING MOV (DS) RATING
OUTPUT RATING Means of Verification (Data Source) RATING Enter "1" if condition
exists No Risks; or Risks are not specifically ID'd in Framework Some Duplication of Assumptions or Risks Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints Assumptions or Risks are internal to project management or at the Wrong Level Structure* FRAMEWORK A&R QUALITY RATING

A&R FRAMEWORK & RRP COMPATIBILITY RATING
OUTCOME LEVEL RATING OUTPUT LEVEL RATING
Appendix 1
33

Impact Statement Level Impact Statement Quality Impact Indicators Impact Targets Impact Timing Outcome Statement Level Outcome Statement Quality Outcome Indicators Outcome Targets Outcome Timing Output Statement Level Output Statement Quality Output Indicators
49 0 3 14 66 30 1 21 14 66 26 0 18 22 66 4 0 10 52 66 10 0 0 56 66 IMPACT RATING
% 74% 0% 5% 21% 1 % 45% 2% 32% 21% 1 % % % 39% 6% 15% 0% 0% 0% 27% 15% 0% 33% 79% 85% 1 1 1 35 7 4 20 66 % 53% 11% 6% 30% 1 26 4 18 18 66 % 39% 6% 27% 27% 1 24 0 23 19 66 % 36% 0% 35% 29% 1 8 0 9 49 66 11 0 0 55 66 60 0 1 5 66 60 0 1 5 66 53 0 7 6 66 % % % % % % 12% 17% 91% 91% 80% 59% 0% 0% 0% 0% 0% 0% 14% 0% 2% 2% 11% 14% 74% 83% 8% 8% 9% 27% 1 1 1 1 1 1 39 0 9 18 66 % % % % % Average # 9 1 Check Check % % % % % of 3s 3% 6% 58% 95% % of 2s 11% 15% 27% 5% % of 1s 58% 41% 8% 0% % of 0s 29% 38% 8% 0% 1 1 1 1 % % 9% 2% 33% 42% 15% 50% 42% 6% 1 1 % % % % % Count of 3s Count of 2s Count of 1s Count of 0s 2 7 38 19 66 4 10 27 25 66 38 18 5 5 66 63 3 0 0 66 6 22 10 28 66 1 28 33 4 66
Output Targets No MOVs (Data Sources) Identified for Impact, Outcome or Output level Very Generic non-specific MOVs for Outcome & Impact Levels Inappropriate MOV for Impact &/or Outcome Level Frequency of Data Collection not cited for Outcome Level Surveys # of ASSUMPTIONS # of RISKS No Risks; or Risks are not specifically ID'd in Framework Some Duplication of Assumptions or Risks Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints Assumptions or Risks are internal to project management or at the Wrong Level Structure* FRAMEWORK A&R QUALITY RATING Count A&R Then Enter "1" if condition exists exists MOV (DS) RATING Assumptions & Risks RATING OVERALL SUMMARY RATING (All Five Factors) GOAL LEVEL RATING OUTCOME LEVEL RATING OUTPUT LEVEL RATING Key Risks in RRP Text are not in Framework A&R FRAMEWORK & RRP COMPATIBILITY RATING NOTE: Five Factors Weighted Equally
34
Appendix 2
OUTCOME RATING OUTPUT RATING Means of Verification (Data Source) RATING Enter "1" if condition FRAMEWORK ASSUMPTIONS & RISKS RATING FRAMEWORK & RRP ASS & RISK COMPATIBILITY
% of 3s % of 2s % of 1s % of 0s Count of 3s Count of 2s Count of 1s Count of 0s

24 0 1 0 25 IMPACT RATING 17 0 8 0 25 12 0 12 1 25 12 0 1 12 25 7 0 0 18 25
% % % % % 96% 68% 48% 48% 28% 0% 0% 0% 0% 0% 4% 32% 48% 4% 0% 0% 0% 4% 48% 72% 1 1 1 1 1 13 2 7 3 25 % 52% 8% 28% 12% 1 17 0 5 3 25 % 68% 0% 20% 12% 1 14 0 8 3 25 9 0 1 15 25 5 0 0 20 25 25 0 0 0 25
Output Statement Quality Output Indicators Output Targets
OUTCOME RATING
OUTPUT RATING
25 0 0 0 25 18 0 7 0 25 15 0 3 7 25
% % % % % % % 56% 36% 20% 100% 100% 72% 60% 0% 0% 0% 0% 0% 0% 0% 32% 4% 0% 0% 0% 28% 12% 12% 60% 80% 0% 0% 0% 28% 1 1 1 1 1 1 1 % % % % % Average # 7 2 Check Check % % % % % of 3s 12% 16% 60% 52% % of 2s 32% 16% 20% 8% % of 1s 52% 48% 20% 40% % of 0s 4% 20% 0% 0% 1 1 1 1 % % 36% 0% % 40% 56% % 12% 44% % 12% 0% % 1 1 % Count of 3s Count of 2s Count of 1s Count of 0s 3 8 13 1 25 4 4 12 5 25 15 5 5 0 25 13 2 10 0 25 9 10 3 3 25 0 14 11 0 25
# of RISKS MOV (DS) RATING
No MOVs (Data Sources) Identified for Impact, Outcome or Output level Very Generic non-specific MOVs for Outcome & Impact Levels Inappropriate MOV for Impact &/or Outcome Level Frequency of Data Collection not cited for Outcome Level Surveys # of ASSUMPTIONS exists
No Risks; or Risks are not specifically ID'd in Framework Some Duplication of Assumptions or Risks Some Assumptions/Risks are Technical "Givens" or Rationale rather than potential constraints Assumptions or Risks are internal to project management or at the Wrong Level Structure* FRAMEWORK A&R QUALITY RATING

A&R FRAMEWORK & RRP COMPATIBILITY RATING GOAL LEVEL RATING OUTCOME LEVEL RATING OUTPUT LEVEL RATING
Appendix 1
35
36
Appendix 3
STUDY DATABASE
Table A3.1: 25 Sample DMFs - 2000 Project/Program Loans Seq # Loan Country Title 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 1753 1758 1748 1814 1738 1770 1810 1779 1786 1749 1743 1787 1796 1807 1798 1768 1812 1739 1746 1772 1785 1800 1737 1773 1799 CAM IND PRC PRC INO INO INO KAZ KGZ LAO MON PAK PAK PAK INO PNG PNG PHI PHI PHI SAM SRI UZB UZB UZB Stung Chinit Irrigation and Rural Infrastructure Housing Finance II Project Hefei-Xi'an Railway West Henan Agricultural Development Industrial Competitiveness and Small and Medium Enterprise Development Program Marine and Coastal Resources Management Decentralized Health Services Farm Restructuring Sector Development Program (Policy) Skills and Entrepreneurship Development Primary Health Care Expansion Second Financial Sector Reform Program North-West Frontier Province Area Development Phase II Small-and Medium-Size Enterprise Trade Enhancement Finance Energy Sector Restructuring Program Road Rehabilitation Sector Microfinance and Employment Provincial Towns Water Supply and Sanitation Grains Sector Development Program - Program Loan Pasig River Environmental Management and Rehabilitation Sector Development Program - Project Loan Infrastructure for Rural Productivity Enhancement Sector Small Business Development Private Sector Development Program (Subprogram I) Senior Secondary Education Railway Modernization Small and Medium Enterprise Development
Source: Ken Smith, OED Consultant, Dec 2005
Appendix 3
37
Table A3.2: 25 Sample DMFs - 2002 Project/Program Loans Seq # Loan Country Title 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 1920 1943 1939 1969 1953 1924 1985 1944 1974 1909 1964 1949 1948 1928 1937 1938 1950 1955 1988 1925 1971 1976 1911 1904 1960 BAN BAN CAM CAM CAM PRC PRC IND IND INO INO LAO RMI PAK PAK PAK PAK PAK PAK PNG VIE UZB SRI TON UZB Road Network Improvement and Maintenance Dhaka Clean Fuel Tonle Sap Environmental Management Mekong Tourism Development (Cambodia) Commune Council Development Efficient Utilization of Agricultural Wastes Hebei Province Wastewater Management East-West Corridor Modernizing Government and Fiscal Reform in Kerala (Subprogram I) Poor Farmers' Income Improvement through Innovation Sustainable Capacity Building for Decentralization Smallholder Development Outer Island Transport Infrastructure Punjab Road Development Sector Local Government Performance Enhancement (TA Loan) Gender and Governance Mainstreaming (TA Loan) Punjab Community Water Supply and Sanitation Sector Financial (Nonbank) Markets and Governance Program Rural Finance Sector Development Program (Project Loan) Coastal Fisheries Management and Development GMS: Mekong Tourism Development (Viet Nam) Regional Power Transmission Modernization (Uzbek Component) Aquatic Resource Development and Quality Improvement Economic and Public Sector Reform Program Education Sector Development Program (Program Loan)
38
Appendix 3
Table A3.3: 56 Sample DMFs - 2004 Project/Program Loans Seq # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 Loan 2076 2079 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2099 2101 2102 2105 2106 2107 2110 2111 2112 2115 2116 2117 2118 2123 2124 2125 2126 2128 2129 2133 2134 2135 2140 2143 2146 2147 2151 2152 2103/ 2104 2113/ 2114 2119/ 2120 2121/ 2122 2130/ 2131/ 2132 2136/ 2137 2138/ 2139 2141/ 2142 2144/ 2145 2148/ 2149/ 2150 Country Title VIE PNG PRC AFG SRI LAO LAO MON TUV PRC UZB AFG NEP UZB PRC VIE SRI NEP FSM BAN NEP AFG KGZ PAK IND NEP PRC VIE PRC BAN VIE VIE TAJ PRC INO VIE CAM PAK PAK PAK AFG NEP PRC BAN IND IND PAK KGZ AZE CAM SRI PHI SRI IND PAK BAN Health Care in the Central Highlands Community Water Transport Fujian Soil Conservation and Rural Development II Agriculture Sector Program North East Community Restoration and Development Extension Roads for Rural Development Northern Community-Managed Irrigation Sector Regional Road Development Maritime Training Hunan Roads Development II Woman and Child Health Development Afghanistan Investment Guarantee Facility Decentralized Rural Infrastructure and Livelihood Second Textbook Development Guangxi Roads Development II Small and Medium-sized Enterprise Development Secondary Education Modernization II Subregional Transport Facilitation Omnibus Infrastructure Development Teaching Quality Improvement in Secondary Education Community-Managed Irrigated Agriculture Sector Regional Airports Rehabilitation Ph 1 Southern Transport Corridor Road Rehabilitation Balochistan Resource Management Program Torrent Combined-Cycle Power Skills for Employment Liaoning Environmental Improvement Second Lower Secondary Education Development Dali-Lijiang Railway Secondary Towns Integrated Flood Protection Phase 2 Second Financial Sector Program Support to Implementation of Poverty Reduction Program Irrigation Rehabilitation Gansu Roads Development State Audit Reform Sector Development Northern Power Transmission Sector Small and Medium Enterprise Development Program Restructuring of the Technical Education and Vocational Training System (Balochistan Province) Sustainable Livelihoods in Barani Areas Restructuring of the Technical Education and Vocational Training System (Northwest Frontier Province) Andkhoy-Qaisar Road Gender Equality and Empowerment of Women Coal Mine Methane Development Chittagong Port Trade Facilitation Multisector Project for Infrastructure Rehabilitation in Jammu and Kashmir Power Grid Transmission (Sector) North-West Frontier Province Road Development Sector and Subregional Connectivity Regional Customs Modernization and Infrastructure Development Urban Water Supply and Sanitation Second Education Sector Development Program Fiscal Management Reform Program Health Sector Development Financial Markets Program for Private Sector Development Assam Governance and Public Resource Management Sector Development Punjab Devolved Social Services Small and Medium Enterprise Sector Development Program
Appendix 3
39
Table A3.4: 57 Sample DMFs - 2005 Project/Program Loans Seq # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 Loan 2175 2207 2218 2181 2191 2196 2208 2221 2182 2183 2205 2171 2172 2225 2166 2202 2220 2224 2180 2158 2226 2201 2217 2195 2188/2189 2163/2164 2157 2159 2219 2162 2165 2222 2200 2199 2176 2223 2190 2167/2168 2178 2192 2227 2186 2211 2210 2187 297 2215 2228 2174 2170 2184 2160 2156 2194 2185 2216 2213 Country Title PRC PRC MON PRC UZB TAJ UZB INO PRC REG AZE PAK BAN VIE IND PAK SAM KGZ VIE FIJ IND SRI SRI VIE BAN INO PRC IND PRC LAO AFG VIE BAN PHI PRC VIE BAN SRI PAK INO AFG PHI PAK PAK BHU SRI AFG INO COO MLD INO MLD BAN VIE CAM PAK PAK Jilin Watere Supply and Sewerage Development Henan Wastewater Management and Water Supply Sector Financial Regulation and Governance Program Central Sichuan Roads Development Information and Communications Technology in Basic Education Dushanbe-Kyrgyz Border Road Rehabilitation Project (Phase II) Kashkadarya and Navoi Rural Water Supply and Sanitation Sector Rural Infrastructure Support Zhengzhou-Xi 'an Railway Establishment of the Pacific Aviation Safety Office East-West Highway Improvement Agribusiness Development Second Urban Primary Health Care Northern Power Transmission Expansion Sector Tsunami Emergency Assistance (Sector) Balochistan Devolved Social Services Program Education Sector Project II Banking Sector and Capital Market Development Program Preventive Health System Support Alternative Livelihoods Development Kerala Sustainable Urban Development Project Local Government Infrastructure Improvement National Highways Sector Project Central Region Transport Networks Improvement Sector Project Gas Transmission and Development Community Water Services and Health Sanjiang Plain Wetlands Protection Chhattisgarh Irrigation Development Hunan Roads Development III Project Greater Mekong Subregion: Nam Theun 2 Hydroelectric Power Transmission and Distribution Kunming-Haiphong Transport Corridor Noi Bai-Lao Highway Technical Assistance Project Southwest Area Integrated Water Resources Planning and Management Microfinance Development Program Fuzhou Environmental Improvement Central Region Water Resources Project Agribusiness Development Tsunami Affected Areas Rebuilding / Northeast Community Restoration and Development II Infrastructure Development Local Government Finance and Goverenance Reform Sector Development Program Western Basins Water Resources Management Small and Medium Enterprise Development Support Rawalpindi Environmental Improvement National Highway Development Sector Investment Program Road Network Technical Education Development Project Fiscal Management and Public Administration Development Policy Support Program Cyclone Emergency Assistance Regional Development Project Phase II - Environmental Infrastructure and Management Road Rehabilitation-2 Tsunami Emergency Assistance Emergency Flood Damage Rehabilitation Support the Implementation of the Poverty Reduction Program II Financial Sector Program (Subprogram III) Punjab Resource Management Program Subprogram 2 Earthquake Emergency Assistance
40
Appendix 3
Table A3.5: 52 Sample DMFs - 2000 ADTAs Seq # TA No. Country Title 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 1903 3110 3391 3396 3402 3407 3411 3415 3419 3424 3429 3433 3438 3445 3450 3454 3460 3468 3472 3478 3481 3484 3493 3498 3503 3510 3515 3521 3522 3523 3528 3529 3533 3534 3541 3543 3549 3550 3554 3555 3558 3560 3566 3573 3577 3582 3586 3592 3600 3607 3611 3616 NEP SRI FIJ LAO VIE LAO PHI CAM SOL FIJ PHI PAK UZB IND SRI CAM IND PRC INO LAO INO INO PRC SAM PRC FSM TIM PRC RMI INO VIE UZB BAN PRC KAZ PRC SAM KAZ COO TAJ PAK VIE SAM IND CAM BAN IND TIM TAJ PRC RMI LAO Group Formation and Training of Women Beneficiaries (Supplementary) Reengineering of Road Sector Institutions (Supplementary) Strengthening Debt Management Assessing a Concession Agreement for the Lao PDR Component of the Chiang Rai to Kunming Human Capital of the Poor in Viet Nam: Policy Options Participatory Assessment of Poverty in the Lao PDR Joint ADB, UNDP. And World Bank Poverty Consultations Education Strategic Support Strengthening Public Sector Management (Phase 2) Capacity Building of the Native Land Trust Board in Preparing Land Maps and Establishing Grains Policy and Institutional Reforms Strengthening of Institutional Capacity for Judicial and Legal Reform Development of the Insurance Industry Establishing a Public Private Joint Venture for the West Bengal North-South Economic Corridor Promotion of Private Sector Involvement in Oil and Gas Exploration Building Capacity in Tourism Planning Policy and Operational Support and Capacity Building for the Insurance Regulatory and Policy Support for PRC 2020 Project (Phase III) Governance Audit of the Public Prosecution Service Capacity Building for Primary Health Care Outer Island Electrification Corporate Governance Reform Development of a Small and Medium-sized Enterprise Credit System Education Support Establishing an Enterprise Sample Survey System Improving Access to Laws Capacity Building for Local Government Assessment of Small and Medium Cities Urban Infrastructure Development Community-Based Coastal Marine Resources Development Capacity Building for Decentralized Natural Resources Management Capacity Building for Water Resources Management (TA Cluster) Facilitating Development of the Railway Sector Capacity Building of the Securities and Exchange Commission and Selected Capital Development of Financing Policies and Mechanisms for Small and Medium-Sized Deepening of Agricultural Reforms and Development Programs Private Sector Development Capacity Building of Financial and Business Advisory Intermediaries A Comprehensive Medium-Term Poverty Reduction Strategy Review of the Superannuation and Insurance Schemes Support to Rural Financial Systems Development Institutional Strengthening of Export Promotion Bureau (TA Cluster) Secondary Education Sector Master Plan Capacity Building for Urban Planning and Management Reorganization Plan for Guharat Electricity Board Implementation of Land Legislation Strengthening the National Accounts and Poverty Monitoring System Building HUDCO's Capacity for Lending to Community-Based Finance Institutions Economic Policy Forum Improving Barki Tajik's Billing and Collection System Policy Support for Social Security Reform under Tenth Five-Year Plan Reviewing the Health Management Information System Participatory Poverty Monitoring and Evaulation
Appendix 3
41
Table A3.6: 52 Sample DMFs - 2002 ADTAs Seq # TA No. Country Title 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 3621 3830 3834 3833 3840 3838 3846 3850 3859 3866 3873 3880 3884 3891 3902 3910 3920 3925 3926 3932 3933 3936 3937 3940 3941 3944 3946 3952 3954 3957 3958 3962 3963 3967 3969 3972 3976 3980 3987 3993 4003 4009 4015 4020 4025 4030 4040 4044 4053 4061 4067 4072 PAK VIE NEP VAN PRC KIR INO INO KGZ IND TON IND VIE PRC IND BHU MON MLD PAK PHI PRC SAM TKM PHI RMI PAK PNG CAM PHI INO PRC FSM PRC INO LAO IND THA PRC TAJ CAM BAN LAO PAK CAM CAM CAM SRI BAN IND PRC INO KAZ Fiscal Decentralization (Supplementary) Assessment and Strengthening of Coastal Management Institutions Support for the Focal Point for Financial Sector Reforms Institutional Strengthening of the National Statistics Office Opportunities for the Clean Development Mechanism in the Energy Sector Community Development and Sustainable Participation Gender Equity in Policy and Program Planning Establishment of a Financial Services Authority Strengthening Capacity in the Office of the President, Phase Two Secured Transactions Reform Building a Performance Based Public Service Integrating Poverty Reduction in Programs and Projects Eligibility of State-Owned Enterprises in ADB-Financed Projects Study of Control and Management of Rural Nonpoint Source Pollution North Eastern Region Urban Sector Profile Institutional Development of the Bhutan Development Finance Corporation Strengthening Public Sector Administration and Financial Management Revenue Diversification Support to the Implementation of Decentralization Improving the Climate for Investment and Productivity in the Philippines Improving Corporate Governance and Financial Performance of State-Owned Enterprises Strengthening of Economic Sectors Planning and Management Improving the Statistical Methodology of the National Institute of State Statistics Capacity Building of the Mindanao Economic Development Council to Enhance Private Sector Mobilizing Land Industrial Environmental Management Improving Economic and Social Statistics Integrated Social Sectors Study Organizational Development of the Commission on Higher Education Integration of Poverty Considerations in Decentralized Education Management Improving Basic Education in Underdeveloped Areas Through Information and Communication Capacity Building in Public Sector Financial Management Study of Carrying Capacity of Water Resources Renewable Energy Development in Small Towns and Rural Areas Local Government Provision of Minimum Basic Services for the Poor Northern Region Strategic Action Plan Strengthening Consumer and Stakeholder Communication for Madhya Pradesh Power Sector Strengthening Accountability Mechanisms II Strengthening Public Debt Management Strengthening Corporate and Financial Governance Reforms Improving the Regulatory and Management Framework for Inland Fisheries Supporting Urban Government Reform Social Protection in the Lao PDR: Issues and Options Enhancing Capacity for Resource Management and Poverty Reduction in Punjab Improving Insurance Supervision Capacity Building of the Inland Fisheries Research and Development Institute Private Sector Assessment Needs Assessment in Conflict-Affected Areas Efficiency Enhancement of Fiscal Management II Management Consulting Services to Indian Railways Songhua River Water Quality and Pollution Control Management Managing Regional Disparity in Economic and Poverty Reduction Programs Under Decentralization Capacity Building of National and Local Governments to Implement the Poverty Reduction Program
42
Appendix 3
Table A3.7: 130 Sample DMFs - 2004 ADTAs SEQ # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 TA No. 2080 3874 4131 4283 4315 4316 4319 4320 4321 4322 4325 4326 4327 4328 4329 4330 4331 4333 4334 4335 4337 4338 4339 4341 4342 4345 4346 4349 4350 4351 4352 4353 4354 4358 4359 4362 4363 4364 4364 4365 4370 4375 4376 4383 4386 4388 4389 4390 4392 4393 4395 4396 4397 4398 4399 4400 4401 4402 4403 4404 4405 4406 4408 4410 4412 Country Title SRI AFG CAM CAM SRI CAM PAK BAN PHI PRC PRC NEP PRC UZB TIM FIJ VIE INO AFG PRC MLD MLD LAO INO PRC AFG PAK PRC PRC PRC MON NEP AFG PRC MON VAN PRC MON MON PRC IND KAZ CAM MON BHU PNG PRC PHI LAO MON VIE UZB NEP BHU PRC PAK PRC PRC FIJ PRC KGZ LAO KGZ VIE BHU TA Road Project Preparatory Facility Capacity Building for Reconstruction and Development (Supplementary) Preventing Poverty and Empowering Female Garment Workers Affected by the Changing Participatory Poverty Assessment of the Tonle Sap (Supplementary) Road Sector Master Plan Harmonizing Loan Project Implementation Procedures Determinants and Drivers of Poverty Reduction and ADB's Contribution in Rural Pakistan Social Protection of Poor Female Workers in the Garment Sector in the Context of Changing Strengthening Governance of Securities Trading Markets Poverty Impact of Area-Wide Road Networks World Trade Organization Policy Reform Support to the Ministry of Railways Education Sector Development Policy and Strategy Flood Management Strategy Study Agricultural Sector Review and Planning Capacity Building in the Ministry of Planning and Finance to Monitor the National Development Plan Strengthening Public Sector Banking and Cash Management Support for Pro-Poor Health Policies Gas Generation from Waste Capacity Building for Agriculture Policy Reform Town-Based Urbanization Strategy Study Commercialization of Agriculture Strengthening the Framework of Education Toward Vision 2020 Study of Gender Inequality in Women's Access to Land, Forests, and Water Strengthening the Capacity of the Commission for Eradication of Corruption in Indonesia National Food Safety Regulatory and Strategic Framework Security of ADB-Financed Projects in Afghanistan Coordination of Devolved Social Services Programs Strengthening of the Statistical System of the China Banking Regulatory Commission Development of Small and Medium-Sized Enterprise Credit Guarantee Companies Policy Reform in Road Transport Developing and Urban Development and Housing Sector Strategy Promoting Pro-Poor and Gender-Responsive Service Delivery Establishing a Gas Regulatory Framework Capacity Building to Combat Land Degredation Agriculture Sector Strategy Study Development of a Medium-Term Strategic Framework Support for Reforms in Compulsory Education Financing Regional Road Development Awareness and Prevention of HIV/AIDS and Human Trafficking Technical Assistance Facility for Policy Reform West Bengal Development Finance Environmental Monitoring and Information Management System for Sustainable Land Use Capacity Building for the Tonle Sap Poverty Reduction Initiative Participatory Poverty Assessment and Monitoring Strengthening of the Payment and Settlement System Strengthening the Capacity of the Parliamentary Accounts Committee Waste Coal Utilization Study Preparing the Agrarian Reform Communities Project II Marketing Support for Organic Produce of Ethnic Minorities Establishing an Effective Anti-Money Laundering Regime Improving Risk Management in the Viet Nam Bank for Agriculture and Rural Development Woman & Child Health Development Decentralized Rural Infrastructure and Livelihood Strengthening the Debt Management Capacity of the Department of Aid and Debt Management Agricultural Taxation Reform Transport Policy Support Rural Income and Sustainable Development Heating Supply for Urban Poor in Liaoning Fisheries Sector Review Implementation of the National Strategy for Soil and Water Conservation The Study on Pricing Systems and Cost-Recovery Mechanisms for Irrigation Capacity Building for Smallholder Livestock Systems A Study of the Impact of Land Reform on Agriculture, Poverty Reduction, and Environment Development of the Transport Sector, Review of Policy and Regulations Small and Medium Enterprise Development
Appendix 3
43
Table A3.7: 130 Sample DMFs - 2004 ADTAs (con't) SEQ # 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 TA No. 4414 4415 4416 4417 4418 4419 4421 4422 4423 4424 4426 4427 4428 4430 4431 4432 4434 4435 4439 4441 4442 4443 4444 4445 4446 4447 4450 4451 4452 4453 4454 4455 4457 4459 4464 4465 4468 4471 4472 4476 4480 4481 4486 4488 4489 4490 4491 4494 4496 4497 4503 4504 4506 4507 4510 4513 4518 4519 4521 4523 4527 4536 4543 4544 4557 Country Title PAK AFG PRC SAM VIE LAO SRI NEP TAJ BHU FSM CAM CAM PRC KAZ PAK LAO PAK RMI CAM SRI PAK KGZ KGZ BAN PRC KGZ TAJ VIE VIE PRC PRC VAN CAM FSM AZE CAM MON TAJ CAM VIE LAO PRC TON AZE CAM AZE SOL IND IND VIE PAK BAN BAN TON SAM IND TIM LAO PHI SOL AFG INO PHI PHI Non-Formal Primary Education and Functional Literacy for Rural Women in Selected Barani Areas of Punjab Kabul Air Quality Management Capacity Strengthening of Power Planning Process Privatization Support Capacity Building for Small and Medium-Sized Enterprise Development Preparing the Forest Plantations Sector Implementing Products and Services for the Donestic Debt Market (TA Cluster Preparation of National Resettlement Policy Framework Development of Community Based Micro-Hydropower Supply in Remote Rural Areas Strengthening the National Statistical System Phase 2 Omnibus Infrastructure Development Establishment of the Tonle Sap Basin Management Organization II Strengthening National Program Budgeting for the Agriculture Sector Rural Finance Reforms and Development of Microfinance Institutions Financial Sector Governance Capacity Building for Environmental Management in Sindh Poverty Reduction through Land Tenure Consolidation, Participatory Natural Resources Management, and Local Communities Skills Building Water Section Irrigation Development Improving the Environment for Private Sector Development Support to Public Financial Management Reform Psychosocial Health in Conflict-Affected Areas Mobilization of Grassroots Stakeholders for Pro-Poor Social Service Delivery (Sindh) Southern Transport Corridor Road Rehabilitation Southern Transport Corridor Road Rehabilitation Support to the Roads and Highways Department for Safeguard Policy Compliance Evaluation of Environmental Policy and Investment for Water Pollution Control in the Huai River Basin and the Taihu Lake Basin Regional Customs Modernization and Infrastructure Development Regional Customs Modernization and Infrastructure Development Gender Mainstreaming Action Plan for Agriculture and Rural Development Supporting the Preparation of the Law on Gender Equality Developing a Poverty Monitoring System at the Country Level Dali-Lijiang Railway Secured Transaction Reforms Implementation of the Action Plan for Gender Mainstreaming in the Agriculture Sector Strengthening Public Sector Audit Function Urban Water Supply and Sanitation Second Education Sector Development Formulating a Transport Strategy Irrigation Rehabilitation Small and Medium Enterprise Development Developing Agricultural Insurance Integrating the Poor in Regional Trade Through Industrial Standard Development Phase II Formulation of the Regulation for Selection and Engagement of Consultants for GovernmentFinanced Projects Youth Microenterprise Development Capacity Building at the Ministry of Economic Development Enhancing the Resettlement Legal Framework and Institutional Capacity Developing Collateral Framework and Microfinance Regulations Institutional Strengthening of the Ministry of Infrastructure and Development Capacity Building for Clean Development Mechanism Capacity Building for Municipal Service Delivery in Kerala Monthly Report on Small-Scale Technical Assistance Projects Not Exceeding $150,000 per Project Punjab Devolved Social Services Program Chittagong Port Trade Facilitation Small and Medium Enterprise Sector Development Integrated Strategic Planning, Medium-Term Fiscal Framework and Budgeting Strengthening State-Owned Enterprise Corporate Governance Capacity Building for Kerala Sustainable Urban Development Strengthening Microfinance Operations Institutional Strengthening for Poverty Monitoring and Evaluation Strengthening SME Credit Management Systems Diagnostic Assessment of Inter-Island Transport Cross-Border Trade and Transport Facilitation Sustaining Decentralization and Local Governance Reforms Enhancing Access of the Poor to Microfinance Services in Frontier Areas Institutional Strengthening of Energy Regulatory Commission and Privatization of National Power Corporation
44
Appendix 3
Table A3.8: 50 Sample DMFs - 2005 ADTAs Seq # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 TA No. 4594 4675 4701 4688 4599 4636 4658 4283 4645 4755 4580 4604 4621 4680 4724 4630 4697 4687 4561 4618 4627 4655 4737 4572 4590 4752 4391 4207 4588 4620 4546 4749 4686 4693 4717 4712 4738 4700 4597 4736 4748 4558 4667 4613 4565 3528 4603 4612 4670 4695 Country Title AFG AFG PRC BAN BHU BHU BHU CAM CAM CAM PRC PRC AZE PRC PRC IND IND INO BAN PRC LAO LAO MON FIJ NAU NEP INO PAK SOL PAK TAJ PAK PHI PHI PHI SAM SAM SOL THA SRI SRI THA THA THA UZB VIE VIE VIE VIE VIE Capacity Strengthening of the Civil Aviation Sector Capacity Building for Road Sector Institutions Railway Passenger and Freight Policy Reform Improving National Accounts, Price, and Wage Statistics Capacity Building for the Bhutan Power Corporation Capacity Building to Implement Environmental Assessment Procedures Capacity Building in Road Safety and Road Asset Management Participatory Poverty Assessment of the Tonle Sap (Supplementary) Restructuring of the Railway in Cambodia Developing Deposit Services in Rural Cambodia Nongovernment Organization-Government Partnerships in Village-Level Poverty Alleviation Nanjing Water Utility Long-Term Capital Finance in Commercial Markets Participatory Strategy Development for Millenium Development Alternative Livelihood Options to Facilitate Coal Sector Restructuring Application of Public-Private Partherships in Urban Rail-Based Transportation Uttaranchal Power Sector Capacity Building Development of Road Agencies in the North Eastern States Natural Resources Management in a Decentralized Framework Emergency Flood Damage Rehabilitation Asset-backed Securities Market and Restructuring of Asset Management Companies Public Expenditure Planning for National Growth and Poverty Eradication Strategy Capacity Building for Gender Mainstreaming in Agriculture Capacity Building for Financial Sector Reforms Alternative Livelihoods Development Reform of the Nauru Phosphate Corporation Regional Development Strategy Decentralized Education Management II Support to Implementation of Decentralization II (Supplementary) Implementation of Interisland Transport Reforms Developing Social Health Insurance Improving AID Coordination and Portfolio Management Results-Based Monitoring of Projects Harmonization and Managing for Results Support the Implementation of the Microfinance Development Program Debt and Risk Management Promoting Economic Use of Customary Land National Teacher Development Framework Supporting Business Law Reform Promoting International Cooperation on Anti-Money Laundering and Combating the Financing of Terrorism Capacity Building of the Environmental and Social Division of the Road Development Authority Independent External Monitoring of Resettlement Activities of the Southern Transport Development Commercial Financing for Local Government Units Capacity Building for Pollution Taxation and Resource Mobilization for Environment and Natural Resources Sectors Ph II Subregional Development Plan for the Tsunami-Affected Andaman Region Financial Sector Infrastructure Development Capacity Building for Water Resources Management (Supplementary) Strategic Secondary Education Planning and Cooperation Results-Based Monitoring of Poverty Reduction and Growth Air Pollution, Poverty, and Health Effects in Ho Chi Minh City Expressway Network Development Plan
Appendix 3
45
Table A3.9: 25 Sample DMFs - 2000 RETAs Seq # RETA No. Title 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 5824 5892 5905 5908 5911 5914 5915 5918 5921 5924 5927 5930 5933 5936 5939 5942 5945 5948 5951 5954 5957 5960 5963 5966 5969 Regional Study of Nutrition Trends, Policies and Strategies in Asia and the Pacific An International Conference on Poverty Second Asia Development Forum Pacific Governance and Public Sector Management Training Program Sixth Joint ADB-OECD Forum on Development Ninth International Congress of the World Federation of Public Health Associations Establishment of Backbone Telecommunications Network Project Phase I Study on Potential Use of Biotechnology in Reducing Poverty and Achieving Food Security Third ADB-Colombo Plan Training Program Development of a RETA Homepage 2000-2002 Orientation Program for Officials of ADB's Member Countries ADB-ADBI Capacity Building Seminar on Poverty Reduction Issues Seminars on the Use of Consulting Services in 2000-2001 Identification and Prioritization of Subregional Projects in South Asia Strategies for Poverty Reduction through Urban Environmental Improvement Regional Economic Cooperation in Central Asia (Phase II - Year Two) Fifth Agricultural and Natural Resources Research at CGIAR Centers Combatting Trafficking of Women and Children in South Asia Small and Medium-Size Enterprise Growth and Development in the Mekong Region Seventh Joint ADB-OECD Forum on Asian Perspectives Technology and Poverty Economic Cooperation in Asia Regional Power Transmission Modernization Project in Central Asian Republics Private Sector Development in the Pacific Capacity Building for Financial Regulation and Supervision Strategic Study on Development Options for Economic Cooperation between the People's Republic of China and Mongolia
46
Appendix 3
Table A3.10: 26 Sample DMFs - 2002 RETAs Seq # RETA No. Title 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 5945 6022 6023 6026 6027 6030 6031 6034 6035 6038 6039 6042 6043 6046 6050 6054 6058 6059 6062 6066 6070 6074 6078 6082 6086 6090 Fifth Agriculture and Natural Resources Research at CGIAR Centers Thematic Evaluation Studies of Asian Development Bank Operations in Developing Member Countries Regional Gas Transmission Improvement Project in the Central Asian Republics Promoting Urban Poverty Reduction through Participation in the Cities Alliance Fourth Asian Development Forum Eighth Joint ADB-OECD Forum on Asian Perspectives - Asia and Europe: Services Liberalization Promoting Effective Water Management Policies and Practices Study on Subregional Issues in the Agriculture Sector in the Greater Mekong Subregion Capacity Building for Developing Member Countries on the World Trade Organization Trading System Networking with the Asian Institute of Technology Formulation of the Pacific Region Environmental Strategy Poverty Mapping in Selected DMCs Promoting Good Governance in East and Central Asia Regional Public Goods and Regional Development Networking with the Asian Institute of Management Training Workshops on Financial Governance and Management of Investment Projects Trade Facilitation and Customs Cooperation Regional Trade Facilitation and Customs Cooperation Program (Kyrgyz Republic & Tajikistan) Coordinating the Revival of Cooperation Activities in BIMP-EAGA Feasibility Studies of the Turkmenistan-Afghanistan-Pakistan Natural Gas Pipeline Capacity Building for Implementing Early Warning Systems in ASEAN-3 Countries Technical Monograph on Participatory Development Rural Finance in Central Asia Supporting Country-Driven Private Sector Development Strategies Public-Private Infrastructure Advisory Facility Private Sector Cooperation in the SASEC Subregion
Appendix 3
47
Table A3.11: 26 Sample DMFs - 2004 RETAs Seq # RETA No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 6016 6056 6077 6088 6108 6109 6166 6167 6168 6169 6170 6171 6172 6173 6174 6175 6176 6177 6178 6179 6180 6181 6183 6184 6185 6186 6187 6188 6189 6190 6191 6192 6193 6194 6195 6196 6197 6198 6199 6200 6201 6202 6203 6204 6205 6206 6207 6208 6209 6210 6211 6212 6213 6214 6215 6218 6219 6220 6222 6223 6224 6225 6226 6227 6228 6229 Department Title RSDD/RSES MKRD/MKSS RSDD/RSFI ERD/ERDI RSDD/RSAN RSDD/NGOC PARD/PAHQ MKRD/MKAE RSDD/RSPR OED/OEOD RSDD/RSGR MKRD/MKAE REMU/REMU RSDD/RSAN SERD/SEID PARD/PAHQ RSDD/RSPR ECRD/ECSS PARD/PAHQ MKRD/MKSS RSDD/RSES BPMS/BPHR ERD/EROD ECRD/ECOC SPD/SPPI MKRD/MKOC SARD/SAOC OGC/OGC SARD/SAOC MKRD/MKAE REMU/REMU MKRD/MKAE MKRD/MKID MKRD/MKSS MKRD/MKID RSDD/RSGR RSDD/RSPR MKRD/MKAE ECRD/ECOC ERD/ERMF REMU/REMU PARD/PAHQ ECRD/ECGF PARD/PAHQ ERD/ERMF PARD/SPSO PARD/PAHQ RSDD/RSAN PARD/PAHQ MKRD/MKGF COSO/COPP SERD/SEGF MKRD/MKAE MKRD/MKAE RSDD/RSFI RSDD/RSGR RSDD/RSAN ERD/EROD PARD/PAHQ ECRD/ECTC RSDD/RSPR SARD/SAAE PARD/PAHQ MKRD/MKID MKRD/MKGF RSDD/RSFI Clean Air Initiative for Asian Cities Greater Mekong Subregion Phnom Penh Plan for Development Management (Supplementary) Road Safety in the Association of Southeast Asian Nations Strengthening and Collection of Purchasing Power Parity Data in Selected DMCs Emergency Regional Support to Severe Acute Respiratory Syndrome (Supplementary) NGO Partnerships for Poverty Reduction (Supplementary) Pacific Regional Transport Analysis Study on Cooperation Opportunities between ADB and Mekong River Commission Upscaling Successful Poverty Reduction Initiatives Selected Evaluation Studies for 2004 Pilot Project on Institutionalizing Civil Society Participation to Create Local Pro-Poor Projects Reviewing the Poverty Impact of Regional Economic Integration in the Greater Mekong High-Level Conference on Asia's Economic Cooperation and Integration Strengthening the Response to HIV/AIDS in Asia and the Pacific Support to Strategize Regional Cooperation in Southeast Asia Making Resources Allocation Pro-Poor and Participatory in the Pacific Building Capacity for Participatory Approaches to Poverty Reduction in ADB Operations Mainstreaming Gender into Poverty Reduction Strategies in Four Central Asian Republics E-Rural Demonstration in the Pacific Greater Mekong Subregion Tourism Sector Strategy Preparation of Asian Environment Outlook for 2005 2004-2006 Orientation Program for Officials of the Asian Development Bank's Developing Fourteenth Tax Conference Central Asia Regional Cooperation in Trade, Transport and Transit Regional Workshops in Preparation for the Second High-Level Forum on Harmonization and Strengthening the Greater Mekong Subregion Economic Cooperation Promoting South Asian Regional Economic Cooperation Establishing Legal Identity for Social Inclusion Supporting Network of Research Institutes and Think Tanks in South Asia Preventing the Trafficking of Women and Children and Promoting Safe Migration in the Greater Capacity Building of Selected Credit Rating Agencies in Asia Phase III Transboundary Animal Disease Control in the Greater Mekong Subregion Greater Mekong Subregion Infrastructure Connections in Northern Lao PDR Preparing the Greater Mekong Subregion Regional Communicable Disease Control Project Greater Mekong Subregion Transport Sector Strategy Study Implementation of the ADV/OECD Anticorruption Initiative for Asia-Pacific Supporting the Achievement of the Millennium Development Goals in the Asia-Pacific Region Capacity Building for Promoting Sustainable Developmennt in the Greater Mekong Subregion The Greater Silk Road Initiative 2005 Asian Developmennt Outlook 2005 Technical Training and Capacity Building in Support of the Asean Economic Surveillance Process Diagnostic Studies for Secured Transactions Reforms in the Pacific Region Regional Trade Facilitation and Customs Cooperation Program (Phase II) Mainstreaming Environmental Considerations in Economic and Development Planning Processes in Selected Pacific Developing Member Countries Long-Term Scenarios for Asian Growth and Trade, 2005-2020 Pacific Financial Technical Assistance Centre 2004 Pacific Regional Social Protection Systems Assessment Ninth Agriculture and Natural Resources Research at International Agricultural Research Centers HIV/AIDS in the Pacific: ADB's Response Small and Medium-sized Enterprise Growth and Development Phase II Capacity Building for Project Implementation and Administration 2005-2006 Southeast Asia Workers' Remittance Study Biodiversity Conservation Corridors Initiative Strengthening Capacity and Regional Cooperation in Advanced Agricultural Science and Technology in the Greater Mekong Subregion Public-Private Infrastructure Advisory Facility Decentralization in Social Sectors Promoting Efffective Water Management Policies and Practices Phase 4 Capacity Building of Developing Member Countries on Selected World Trade Organization and DOHA Leadership Enhancement and Advancement Program (LEAP) Formulating and Implementing an Intergovernmental Agreement of the Shanghai Cooperation Organization Member States on Facilitation of International Road Transport Pilot Testing Participatory Assessment Methodologies for Sustainable and Equitable Water Supply South Asia Subregional Economic Cooperation Human Resource Development and Capacity Building Developing and Implementing the Pacific Plan for Strengthening Regional Cooperation and Integration Coordinating the Greater Mekong Subregion: North-South Economic Corridor Bridge Project Facilitating Cross-Border Trade and Investment in the Greater Mekong Subregion Improving the Outreach of Formal Financial Institutions
48
Appendix 3
Table A3.12: 25 Sample DMFs - 2005 RETAs Seq # RETA No. Department Title 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 6085 6113 6194 6222 6234 6236 6237 6243 6246 6250 6256 6259 6262 6265 6268 6270 6280 6282 6291 6292 6293 6294 6296 6297 6298 PARD MKRD MKRD PARD MKRD ECRD MKRD MKRD OED RSDD COSO PARD MKRD RSDD PARD RSDD OREI MKRD RSDD RSDD RSDD ECRD RSDD SARD SERD Preparing a Pacific Governance Strategy Making Markets Work Better for the Poor (Supplementary) Greater Mekong Regional Communicable Diseases Control Leadership Enhancement and Advancement Project (Supplementary) Environmental Compliance and Enforcement Network Central Asian Countries Initiative for Land Management Greater Mekong Subregion Phnom Penh Plan for Development Management II Strengthening Malaria Control for Ethnic Minorities Technical Assistance for Selected Evaluation Studies for 2005 Preparation of International Public Sector Financial Audit Guidelines Results-focused Project Design and Management Aviation Legislative and Regulatory Review Enhancingthe Development Effectiveness of the Greater Mekong Subregion Economic Cooperation Program Implementing Pilot Projects for Small Piped Water Networks Implementation of Pacific Education Strategy: Skills Development Facilitating Knowledge Management for Pro-Poor Policies and Projects Strengthening Economic and Financial Monitoring in Selected ASEAN+3 Countries A Study on Economic Cooperation Between East Asia and South Asia Rolling Out Air Quality Management in Asia Promoting Environmental Investment in Asia and the Pacific Managing the Cities in Asia Facilitation of Transport Cooperation among Central Asia Regional Economic Cooperating Countries Ph 1 Developing Pro-Poor Governance Capability and Knowledge South Asia Subregional Economic Cooperation III Supporting Enhanced Cooperation among Southeast Asian Equity Markets
Ken Smith, OED Consultant, Dec 2005
Appendix 4
49
METHODOLOGY AND LIMITATIONS A. Methodology
1. The review was confined to a documentary examination and subjective assessment of static output deliverables and development objectives for the loans and TAs i.e. those items that reflect the key project design items in terms of their description, and intended results, together with quantifiable / monitorable targets. 2. Listings of loans and TAs were obtained from OED and random sampling conducted (using a skip interval method) to select documents for the study. [Note: The minimum sample sizes for the advisory and regional TAs were contractually predetermined by OED.] 3. The appropriate loan and TA documents were then obtained by OED and provided to the consultant. 4. The DMFs for each of these documents was reviewed and various factors for each level rated on a four point scale, using the same criteria as employed in previous studies. The factors rated were as follows: Impact (Goal) for conformity with ADB DMF criteria: (i) (ii) (iii) (iv) (v) Appropriate Level Quality (i.e. Succinctness and Clarity) Indicators Targets Timing
Outcome (Purpose) (i) (ii) (iii) (iv) (v) Appropriate Level Quality (i.e. Clarity) Indicators Targets Timing
Outputs (i) (ii) (iii) (iv) Appropriate Level Quality (i.e. Clarity) Indicators Targets
5. A four point quantitative ordinal/nominal rating scale was employed (3 = Highly Satisfactory, 2 = Satisfactory/Acceptable, 1 = Major Shortcomings, 0 = Totally Inadequate) with subjective criteria for each rating as shown in the Criteria for Ratings in the checklist. 6. Assessment was in terms of each DMFs stated design summary objectives, indicators and targetsnot substantive quality or actual/reported implementation performance. Nevertheless, Major Shortcomings or Totally Inadequate ratings indicate that the objectives
50
Appendix 4
were too imprecisewhich would impair ability to objectively monitor implementation effectively, and weaken subsequent evaluation of achievements. 7. Several Excel files and worksheets were created to list the pertinent DMFs. Columns were then added to each worksheet to accommodate quantitative ratings in terms of the aforementioned DMF attributes for each level. 8. Aggregate scores for each level were derived as follows: Impact level. All items were weighted equally. Outcome level. All five items were weighted equally. Output level. All four items were weighted equally. [Note: Timing of Outputs was not considered in this assessment because in practice unless specifically statedtiming is considered to be concurrent with the project completion date. Furthermore, in rating the Output level, as long as the implied deliverables were apparent, grammatical distinction in the framework between Activities (i.e. active verb) and Outputs (completed action) was ignored.] 9. Other columns were added to tabulate and rate risks and data sources.
10. The risks in the loan project/program DMFs were compared with those cited in the main body of the RRP text. 11. 12. The data sources were reviewed for appropriateness. All other aspects of the loan and TA documents were ignored.
13. An overall summary rating was derived by weighting each of the five factorsImpact, Outcome, Output, Risks and Data Sourcesequally. 14. The scores for each item were tabulated directly.
15. Since the five factor summaries and the overall summary for each loan/TA were averaged scoresand not integersin the final tabulation they were treated differently as follows: 3 = 2.99 = 1.99 = 0.99 = 3 2 1 0
Range 2 Range 1 Range 0
16. Finally, percentages were computed, and the top two levels (2 and 3)i.e. at least Satisfactory or betterwere combined, tabulated and graphed.
Appendix 4
51
B.
Limitations
17. This study was a desk documentary review. No follow-up clarification was sought from those responsible for, or more familiar with, the documents. The assessment was conducted in its entirety by one consultant using a simplified checklist of qualitative nominal/ordinal criteria, and standard statistical techniques. 18. The variables and criteria guidelines have face validity, and were used by a single rater in an attempt to standardize ratings and limit subjectivity. Subjective ratings were converted to numerical integer scores. However, rating narrative statements is subjective, and unconscious bias and halo effect are possible even with one objective rater knowledgeable in the subject matter but uninvolved in the actual implementation, the documents authors, or the outcome. 19. Impact (Goal) level statistics are somewhat misleading as most are poorly articulated. Economic development and/or poverty reduction meet both criteria of appropriate level and succinctness, but without indicators are relatively vapid statements. 20. No separate/independent verification was conducted to check the raters reliability or accuracy in rating. Furthermore, the actual substantive quality of each loan and TAin terms of its political desirability, economic necessity, and/or administrative priority; technical appropriateness, resource adequacy, sufficiency, and/or feasibilitywas not questioned or assessed. Neither was any attempt made to verify or judge implementation performance either reported and/or actual. C. A Note re: Numbers and Percentages in this report
21. The 0 3 rating scale employed was ordinal rather than interval, so strictly speaking the data are not open to rigorous statistical manipulation with results less-than integers. 22. Averages are useful for summarizing data to make comparisons between different sets of data, and to depict trends. However variation within the ranges of the data sets may also be useful to identify patterns, &/or other performance by meaningful groupings, and for possible follow-up action. 23. Percentages facilitate comparisons between data sets of differing sizes. However, where a set of data is less than 100, percentages drawn from the set are exaggerated. Widely disparate percentages on small sets may not be really significantly different from each other because they are also subject to differing standard errors, standard deviations as well as any predetermined margin of error tolerated for comparison. Thus comparisons between different size data setsespecially small sets such as employed heremay lead to inappropriate conclusions as any apparent difference may be statistically insignificant. 24. Nevertheless, if comparisons are made and conclusions drawn with caution, comparisons of overall averages and percentage ratings at the ADB level are still useful descriptive statistics.
52
Appendix 5
RATING SYSTEM FOR DESIGN AND MONITORING FRAMEWORKS 1. The following rating system was used to review and rate Project loan design and monitoring frameworks (DMFs). 2. Points were awarded on the following quantitative/nominal qualitative rating scale: 3 = Completely Satisfactory 2 = Satisfactory/Acceptable with some shortcomings 1 = Major Shortcomings 0 = Totally Inadequate 3. The thrust of this assessment was the quality of each framework in terms of its stated objectives, performance indicators and targets in conformity with ADB guidelines and structural requirements. In addition, the quality of assumption and risk statements, and monitoring mechanisms (data sources) were examined to determine their adequacy for monitoring purposes. 4. The desirability, substantive technical quality, and subsequent implementation performance of the project were not examined. Nevertheless, any structural quality ratings of either 1 Major Shortcomings, or 0 Totally Inadequate are early-warning flags that the stated performance objectives, monitoring mechanisms, assumptions or risks are too imprecise, thereby limiting ability to effectively monitor (or review) project implementation objectively. Thus any subsequent evaluation of achievements at the PCR stage is likely to be impaired, the findings questionable, and assertions indefensible. 5. The criteria for rating each of the framework factors and attributes are outlined in the tables on the following pages. A. Various Levels 3 = Highly Satisfactory Goal (Impact) Level
1.a. Statement All at Appropriate Level Succinct Most at Appropriate Level Some at Appropriate Level Run-on Statement (i.e. by . . . through . . . etc.); or unclear Vague None at Appropriate Level (i.e. Not Long Term Objectives) None at Appropriate Level
2 = Satisfactory/ Acceptable
1 = Major Shortcomings
0 = Totally Inadequate
1.b.
Quality
2.
Indicators
Measurable
None (or at Inappropriate Level) None (or at Inappropriate Level) Not Stated (or Inappropriate: Timing
3.
Targets
Measurable
Vague
4.
Timing
Stated
Appendix 5
53
3 = Highly Satisfactory Purpose (Outcome) Level

1.a. Statement Only one, and at Appropriate Level
or Level
One or Two, and both at Appropriate Level
More than Two at Appropriate Level, or some at wrong level
1.b.
Quality
Succinct
Run-on Statement (i.e. by . . . through . . . etc.); or Unclear Some or all Vague
None at Appropriate Level (Not Immediate Objectives/Outcomes/ i.e. the Reason Why the project is being done) None at Appropriate Level
2.
Indicators
All Measurable
None (or at Inappropriate Level; i.e. Not Measures of Changed Behavior or Conditions) None (or at Inappropriate Level) Not Stated (or Inappropriate: Timing or Level)
3.
Targets
All Measurable
Some or all Vague
4.
Timing
Stated
3 = Highly Satisfactory Output Level

1.a. Statement Some at Appropriate Level (May include Measurable Activities) Succinct
Unclear
None at Appropriate Level (i.e. Not Deliverables or Activities of the Project) None at Appropriate Level
2.
Indicators
All Outputs have Measurable Indicators
Only Some Outputs have Measurable Indicators
No Indicators (or at Inappropriate Level i.e. Not Output Level Indicators) None (or at Inappropriate Level i.e. Not Output Level Targets)
3.
Targets
All Outputs have Measurable Targets
Only Some Outputs have Measurable Targets
54 B.
Appendix 5
All Levels of the Framework 1. Quality of Means of Verification (Data Sources)
6.
Four typical shortcomings are used to rate MoV (DS) framework quality: (i) (ii) (iii) No MOV (DS) given for either impact-, outcome-, or output-level indicators. Very generic, non-specific, sources provided for outcome and impact levels. Inappropriate MOV (DS) given for impact- &/or outcome-level indicatorsi.e. citing project reports for measuring higher-level achievements long after the project has been completed. Frequency of data collection not provided (i.e. annual, semi-annual, quarterly, etc.) for Outcome or Output levels Criteria for Rating Framework Means of Verification (Data Sources):
(iv)
Condition
None of the above shortcomings One of the above shortcomings Two of the above shortcomings Three or Four of the above shortcomings
Rating
3 = Highly Satisfactory 2 = Satisfactory / Acceptable 1 = Major Shortcomings 0 = Totally Inadequate
2a. 7.
Quality of Framework Assumptions & Risks (A&R)
Four typical A&R shortcomings are used to rate framework quality: (i) (ii) (iii) (iv) No risks are stated in the framework, or risks are not clearly identified for monitoring. Some duplicationassumptions &/or risks are repeated at different levels in the framework. The stated assumptions and/or risks are really rationale (or technical givens) rather than potential constraints for monitoring. The assumptions and/or risks are at internal to project management or at the wrong level in the framework. Criteria for Rating Framework Quality of Assumptions & Risks
Condition
None of the above shortcomings One of the above shortcomings Two of the above shortcomings Three or four of the above shortcomings
Rating
2b. 8.
Compatibility of Framework Assumptions & Risks (A&R) with the RRP Text:
Three typical A&R shortcomings are used to rate framework quality: (i) (ii) (iii) Key risks in RRP text are not in the framework either as risks or assumptions. Key risks in the framework are not in the RRP text. No risks identified in either the RRP text or the framework.
Appendix 5
55
Criteria for Rating Framework Compatibility of Assumptions & Risks with RRP Condition
None of the above shortcomings Either one or two above Both one and two above item three above
Rating
56
Appendix 6
QUALITY REVIEW AND ASSESSMENT CHECKLIST Project Title:________________________________________ Loan Number:________________________________________ This checklist contains forty-four (44) generic reminders about project design criteria that should be incorporated in a Project Design & Monitoring Framework (DMF). Use this checklist as an aid to rapidly review, assess and comment on the RRP (DMF Appendix).
Assessment
U Unsati sfactory P Partly Satisfactory S Satisfactory
Topic
A. General
Design Summary (DS) 1st Column (DS) The summaries at each level are precise descriptive narrative statements (i.e. no numbers); one sentence (or less) at each level, without run-on conjunctions such asand, as well as, also, through, by means of, in order to, etc. Indicators and Targets (IT) 2nd Column (IT) The indicators (direct and/or proxy) are valid and reliable measures of accomplishmenti.e. an increase in the level would indicate improvement; no change would represent stability; and a decrease would indicate the situation was deteriorating [Note: for some classes of indicatorssuch as in public health, crime, or safetythe reverse may be true:i.e. increased incidence of a disease, crime rate, or number of accidents, could indicate deterioration in the situation, while a decrease would be considered an improvement.] Data Sources (DS) 3rd Column (DS3) A reasonable source &/or method for collecting the data required for each indicator is identifiedestablished secondary sources, special surveys (frequency also cited for Outcome and Impact levels), and/or specific reports. Assumptions and Risks (AR) 4th Column (AR) Assumptions are pre-requisites for attaining the project's Design Summary at the same level; NOT the rationale or a working hypothesis. (AR) Risks are major constraints to successful attainment of the
Appendix 6
57
Assessment
Topic
Design Summary at the same level. (AR) Risks are NOT simply restatements of the Assumptions. (AR) The AR cited are independent of and external to the project. i.e. outside the project management's implementation responsibility, reach, or influence. B. Specific Considerations at Each Level
6 7
Impact levelthe long term objective to which the project contributes.

8
9 10 11 12 13 14 15 16 17 18
(DS) This project is only one of several initiatives contributing to the Impact. The Impact will not be attained by this particular project alone. (DS) The Impact statement does NOT describe what the project is going to do. (IT) The Impact is likely to be attained only a considerable time after completion of this project. (IT) There are Indicators of achievement. (IT) The Indicators have Quantitative (or measurable qualitative) Targets. (IT) Target attainment will NOT be solely the result of this project. (IT) Targets have an estimated time (year) for attainment. (DS3) Performance data are external to the project, and will be obtained from secondary sources. (AR) Impact-level assumptions are necessary to attain the Impact. (AR) Risks identified in the RRP text are adequately addressed in the DMF either as Risks or restated positively as Assumptions. (AR) If Impact-level risks occur, they will impair attainment of the Impact. Outcome levelthe immediate objective; i.e. why the project is being done
19
20
(DS) The project has only one Outcome. [NOTE: Projects sometimes have multipleand even contradictoryOutcomes imposed on them by various stake-holders with different agendas. To maximize the potential for unambiguous success, the projects prime Outcome should be narrowly defined.] (DS) The Outcome is the project objective, but it is not a deliverable that the project managerial team can produce. [NOTE: Institutional strengthening is only an Output, not an Outcome. The Outcome resulting from institutional strengthening should be an increased level of effectiveness &/or efficiency in service
58
Appendix 6
Assessment
Topic
delivery performance.] (DS) The Outcome represents the result of an increased level of activity, a different method for doing things, and/or a fundamental change in behavior by the targeted beneficiaries as a direct consequence of the project. (i.e. more / better.) (IT) There are Indicators at the Outcome leveldifferent from Impacts and Outputs. Indicators are not a summary or restatement of Outputs. (IT) All Indicators have measurable quantitative &/or qualitative Targets. (IT) All Targets have an estimated date for attainment. (IT) The baseline situation (or benchmark for comparison) at the beginning of the project status (BOPS) is identified in numerical quantitative terms. (If percentages are used, an actual value is provided to establish a linkage. (IT) The nature, extent and incidence of the targeted beneficiary population is identified, in addition to aspects (such as level of income) intended to be changed by the project. (IT) The categories of indicators for a specific project are limited to not more than five. [Note: An appropriate simple proxy indicator obtained periodically through an opinion survey may be preferable to regular collection, processing and reporting of numerous technically-detailed service statistics or other measurements.] (DS3) In addition to PCRs, other Outcome-level data will be obtained by special survey, and/or derived from secondary source analysis. (AR) Outputs and Outcome-level assumptions are necessary and sufficient to attain the Outcome. (AR) Outcome-level risks are major constraints to attaining the Outcome. Output level what the project will do.
21
22
23 24 25
26
27
28
29 30
31
32 33 34 35 36
(DS) Outputs are results to be produced by the project. (i.e. kilometers of road built, farmers trained, hectares planted with high yielding variety seed, studies conducted, etc.) (DS) The Outputs define the projects key areas of implementation management responsibility. (IT) All Outputs have Indicators. (IT) All Indicators have measurable quantitative &/or qualitative Targets. (IT) To the maximum extent possible, interim Output targets and accomplishmentsif anyare cumulative. (DS3) Reporting frequency (i.e. annual, quarterly, monthly, or as
Appendix 6
59
Assessment
Topic
occurs) of utput targets is identified. (DS3) Most of the Output-level data will be obtained by regular reporting. (AR) Assumptions about activities of collateral organizations necessary to support the project are identified. (AR) The Activities, and Output assumptions are necessary and sufficient to assure delivery of the outputs. (AR) Output-level risks are major constraints to completing timely delivery of the outputs. Activity levelKey jobs or steps to accomplish the Outputs.
37 38 39 40
41
42
(DS) Key Activities to be undertaken by the project are stated as actions (i.e. recruit personnel, procure equipment, train farmers, etc.). (IT) Time-related Milestones are identified for completing each Activity. Input levelKey resources required to accomplish the activities.
43
44
Donors/contributors to the project (i.e. ADB and other donors, such as the World Bank, AUSAID, USAID, etc; the DMC, and others (i.e. NGO matching grants)); and/or resource categories (i.e. technical assistance level of effort, equipment and supplies, training, and funding) are identified. Funding levels for each categoryi.e. personnel (and/or level of effort in person-months), and other resources are identified. Summary Quality Assessment Total Numeric Score Relative Percentage Summary (Rounded off) % = (# / 44) x 100 [See conversion table below]
A B
DMF prepared by:___________________ OED Review & Assessment by:_______________ ___________________________ __________________________________ Position :___________________________ Position: _________________________________ Dept/Division: ______________________ Date:____________ Date:_____________
60
Appendix 6
Number to Percentage Conversion Table Based on 44 items

1 = 2% 2 = 5% 3 = 7% 4 = 9% 5 = 11% 6 = 14% 7 = 16% 8 = 18% 9 = 20% 10 = 23% 11 = 25% 12 = 27% 13 = 30% 14 = 32% 15 = 34 % 16 36 % 17 = 39 % 18 = 41 % 19 = 43 % 20 = 45 % 21 = 48 % 22 = 50% 23 = 52% 24 = 55% 25 = 57% 26 = 59% 27 = 61% 28 = 64% 29 = 66% 30 = 68% 31 = 70% 32 = 73% 33 = 75% 34 = 77% 35 = 80% 36 = 82% 43 = 98% 37 = 84% 44 = 100% 38 = 86% 39 = 89% 40 = 91% 41 = 93% 42 = 95%
Source: Ken Smith, OED Monitoring & Evaluation Consultant, March 2006.
Appendix 6
61
General Reference Criteria

Design Summary Indicators and Targets Data Sources Assumptions and Risks
Impact The Impact is the long term objectivei.e. the project visionand may also be expressed as sustainable. Impacts are only indirectly relatedand partially attributable (if at all)to the project. Impacts are also likely to occur only a considerable time period after the project Outcome has been attained The Impact is not something that the project will be expected to achieve alone, or a deliverable during implementation. Outcome The Project should have only one Outcome statement. The Outcome should summarize the immediate reason why the project is being proposed: NOT what the project is going to do.] The Outcome should describe the improved situation for the target beneficiaries after the project has been satisfactorily completedor the changed conditions when the outputs to alleviate a perceived problem have been Outcome indicators and targets should reflect changes in processes, results and/or behavior by target beneficiaries that are expected by the end of the project, or shortly after the project outputs have been delivered. The desired state, or expected improvement in the situation should be stated in measurable termsi.e. quantity and/or quality, and timeand if possible, compared to an earlier time period. Project Completion Reports are one source of data. Some other sources, means and/or frequency for collecting data should also be external to the project. Assumptions and risks at the Outcome level are those that relate to attainment of Outcome targets. Indicators are the preselected means to measure performance. The measurement unit for each indicator should be specified. A target is a predetermined measurement level on the indicator for a particular timeframe that will serve as a proxy for success. Every Impact should have a related indicator with target and timing stated. The source, means &/or frequency for collecting data at the Impact level should be external to the project, and should already be institutionalized. Every indicator should have at least one source of data, &/or means for collecting it. Only assumptions and risks beyond the control of the projectbut essential to success at each levelshould be cited. Assumptions and risks at the Impact level are beyond the control of the project, but essential to attainment of the Impact.
62
Appendix 6
Design Summary
Indicators and Targets
Data Sources
Assumptions and Risks
delivered. Outputs Outputs are the key categories of results that should be produced/ delivered during project implementation. Outputs are deliverables within the control of the project. Outputs should all be expected to be completed by the end of the project The source, means and/or frequency for collecting data should be within the project implementation managements control. PPRs should be at least one source of data. Assumptions and Risks at the Output level are those that are external, and beyond the control of the project implementers but essential for successful attainment of the Outputs. [Do not state the project rationale, or cite implementation management expectations or concerns.] Inputs A rough outline of the project implementation schedulein terms of key milestonesshould be provided here. [A milestone is an unequivocal descriptive checkpoint that indicates the activity has been completed.] Identify a milestone and target timing for each Activity. The various resource categories required to undertake the project should be identified. Indicators and targets should all be expressed in terms of money, and/or level of effort (person months)
Activities Activities are the key tasks, steps, or stages to be undertaken by the project to accomplish each of the Outputs above. Several key Activities should be enumerated for each Output.
Appendix 6
63
Additional DMF Reference Criteria

Design Summary Indicators & Targets Monitoring Mechanisms Monitoring mechanisms either exist already, or have been planned and budgeted The same source, or means for collecting data can be cited for some indicators at different levels More than one source can be cited for some indicators The monitoring mechanism is only a brief description. Assumptions & Risks Assumptions and risks in the hierarchy: inputs activities outputs outcomes impacts relate to attainment of targets at the same level. Every identified risk should be prefaced with an R for easy identification. Each A&R statement in the framework is unique. Each statement only describes a single item.
Each level in the design summary hierarchy: inputs activities outputs outcomes impacts should contribute to the next higher level. There may be more than one sentence at any level, but each sentence should only describe a single item. There should be no run-on conjunctions The statements at each level should only be descriptive Each statement should be uniquei.e. the only one of its kind in the framework.
Every statement in the design summary column has at least one measurable indicator. There may be more than one indicator for each design summary statement Every indicator has a target (or predetermined standard) that is measurable in terms of quantity &/or quality, and time Each indicator and target should be uniquei.e. the only one of its kind in the framework. Every indicator should be sufficiently descriptive and quantitative (&/or qualitative) to permit subsequent performance measurement and comparison, by independent monitoring and evaluation personnel Where direct measurement indicators are impractical, indirect proxy indicators should be used The following format is a guide to indicator and target definition: Quantitative: Increase (indicator kkk) at least: x% from: XX units (baseline) to: YY units, by: ZZ (date) [Round-off monetary values. Express level of effort in person- /man-months]]
64
Appendix 6
Design Summary
Indicators & Targets
Monitoring Mechanisms
Assumptions & Risks
Qualitative: Improve (indicator kkk) at least: x% from: XX satisfaction level (baseline) to: YY satisfaction level [on a xx-point scale] by: ZZ (date)
Appendix 7
65
TERMS OF REFERENCE
Contract Project Expertise Source Prof. Group A08019 RSC C51894 (PHI): Quality in Design and Monitoring Frameworks Design and Monitoring Frameworks Specialist International C Category Job Level Independent 5
Objective/Purpose of the Assignment: Assess the quality of the DMF prepared for each loan approved during 2005 and to ensure that the DMFs prepared for all loan and TA projects are of an acceptable standard. Scope of Work: To review the quality of the DMFs prepared during 2005 (and to evaluate a sample of 2000 and 2002 loan approval DMFs to provide a more robust basis for historical comparison than the previous used figures based on PPR data), to ascertain whether those for loan projects have continued to be relatively well prepared, and whether or not there has been any improvement in those for TAs. Detailed Tasks: (i) Assess the quality of the DMF prepared for each loan approved during 2005, following the same methodology and criteria (but incorporating all five criteria assessed at the goal [impact] level in the goal [impact] level rating) as used for the previous assessments of loans approved during 2000, 2002, and 2004, but extended to include a rating of the monitoring mechanisms [data sources], and the statements of assumptions and risks, in addition to the design statements and the performance targets and indicators. This will include a check to determine whether or not the assumptions and risks identified in the text are reflected in the DMF, and whether or not they have been used as the basis for sensitivity testing in the financial and economic analysis for the loan. (ii) Rate a random sample of 25 project frameworks for loans from each of 2000 and 2002 using the same methodology as used for (i), extending the analysis of goal (impact), purpose (outcome), and output statements to include an assessment of the adequacy of the proposed monitoring mechanisms (data sources), and the quality of the statements of assumptions and risks. (iii) Assess the quality of the DMFs prepared for a random sample of 50 advisory TAs and 25 regional TAs approved during 2005, again using the same methodology and criteria as used for (i). (iv) Analyze the trends in the overall quality of the DMFs prepared for loans and TAs during 2000-2005 and determine the statistical significance of the differences. (v) Discuss any apparent generic shortcomings in the DMFs assessed, including the underlying causes, and discuss possible remedies to rectify the problems. (vi) Prepare a standard format for use by OED staff to assess the DMF quality when reviewing and commenting on draft RRPs and TA papers, including the criteria by which each aspect of the DMF is to be evaluated. (vii) Prepare a report presenting the findings of the assessments and the analyses. Output/Reporting Requirements: To prepare a report presenting the findings for the assessments and the analysis. Places of Assignment PHI, Manila OTH, Home Office, Virginia, U.S.A. TOTAL DAYS (Intermittent) Days 15 15 30 Estimated Dates (dd/mm/yyyy) 28/11/2005 31/03/2006 28/11/2005 31/03/2006 Equivalent to 30WD
66
Appendix 8
BALLAD OF THE LOGFRAME The ADB Design & Monitoring Framework (DMF)
If youve chosen the management game to seek out your fortune and fame To preclude going bust, a frameworks a must to map out the project terrain. Its not really nuclear fission to outline your Mission and Vision But big bangs for the buck are more than dumb luck; Rather, ideas honed to precision. Though plannings an onerous task when the details all slip through your grasp The framework is here to make it all clear; Each cell has key questions to ask. An essential tool of the trade, whose values not likely to fade, The framework will show where you all plan to go Why; When; and How Much is your aid. Although its a bit of a bore to constantly monitor score To determine your fate On Time, Early or Late? The framework will open the door. To measure a projects success takes more than simply a guess, But youre not in the dark with a framework benchmark, Work Your Plan best you can, with less stress. The frameworks a ready-made briefing whenever youre holding a meeting Concise and compact, with fact after fact Youll Wow them wherever theyre seating! Thus embarked on a project career, with a framework youve nothing to fear. You dont need to sweat; Its the best you can get And still better yet its not dear! Kenneth F. Smith March 2006

Quality in Design and Monitoring Frameworks (DMFS)

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Quality in Design and Monitoring Frameworks (DMFS)

Uploaded by

Copyright:

Available Formats

Quality in Design and Monitoring Frameworks (DMFs)

BACKGROUND & SCOPE

Formerly Project Performance Audit Reports (PPARs).

4 Table 1: DMF Quality Project and Program Loans

25 32% 32% 32% 92% 84% 20%

25 52% 40% 52% 100% 92% 32%

56 71% 68% 68% 97% 91% 48%

57 83% 71% 79% 96% 77% 58%

Conclusion: General improvement trend but no statistically significant difference.

5 Table 2: DMF Quality Advisory Technical Assistance (ADTAs)

52 19% 21% 20% 66% 69% 23%

52 6% 12% 12% 74% 31% 21%

130 35% 29% 23% 75% 100 % 37%

50 64% 48% 60% 94% 100% 62% Yes 1 2 4 Yes Yes

Table 3: DMF Quality Regional Technical Assistance (RETAs)

25 12% 12% 8% 48% 96% 28%

26 0% 0% 4% 57% 27% 35%

66 44% 14% 21% 85% 100% 42%

25 56% 44% 32% 80% 60% 76%

7 III. SUMMARY CONCLUSIONS

TABLES AND GRAPHS

Conclusion: General Improvement Trend but No Statistically Significant Difference

Yes Yes Yes Yes

Table A1.4: Percentage of Satisfactory or Better on Overall Summary

Item RETAs ADTAs Loans

Figure A1.1: Percentage of Satisfactory or Better on Overall Summary

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans

Figure A1.2 Percentage of Satisfactory or Better on Overall Impact

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans

Figure A1.3: Percentage Satisfactory or Better on Overall Outcome 100 90 80 70 60 50 40 30 20 10 0

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans

Figure A1.4: Percentage of Satisfactory or Better on Overall Output

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans

Figure A1.5: Percentage of Satisfactory or Better on Overall Data Source 100 90 80 70 60 50 40 30 20 10 0

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans

Figure A1.6: Percentage of Satisfactory or Better on Overall Assumptions and Risks

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans

Figure A1.7: Percentage of Satisfactory or Better on Impact Appropriate Levels

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans

Figure A1.8: Percentage of Satisfactory or Better on Impact Quality

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans

Figure A1.9: Percentage of Satisfactory or Better on Impact Indicators

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans

Figure A1.10: Percentage of Satisfactory or Better on Impact Targets

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans

Figure A1.11: Percentage of Satisfactory or Better on Impact Timing

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Figure A1.12: Percentage Satisfactory or Better on Outcome Appropriate Level

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans

Figure A1.13: Percentage of Satisfactory or Better on Outcome Quality

Source: Ken Smith, OED Project Evaluation Consultant, March 2006.

Item RETAs ADTAs Loans