Prepared for CARE Nepal Projects

Handbook for Participatory Evaluation (For Internal Project Evaluation)

Prepared by: Bishnu Pokhrel April 2001

TABLE OF CONTENTS
ABBREVIATION........................................................................................................................................................5 ACKNOWLEDGEMENTS ........................................................................................................................................6 1. INTRODUCTION AND BACKGROUND............................................................................................................7 2. EVALUATION MODEL AND CONCEPT OF INTERNAL EVALUATIONS...............................................7 2.1 AN EVALUATION MODEL ....................................................................................................................................7 2.1 PROCESS OF INTERNAL EVALUATION ..................................................................................................................8 3. EVALUATION WORKS........................................................................................................................................9 3.1 DECISION TO EVALUATE......................................................................................................................................9 3.2 PREPARING THE TERMS OF REFERENCE (TOR) ..................................................................................................10 3.3 SELECTION OF EVALUATION TEAM ...................................................................................................................10 3.4 PREPARING EVALUATION PLAN .........................................................................................................................11 3.5 CONDUCTING THE EVALUATION ........................................................................................................................11 3.5.1 Steps to be followed: .................................................................................................................................11 3.5.2 Writing the report......................................................................................................................................12 4. RESPONSIBILITY OF PROJECT STAFF AND COMMUNITY TEAM MEMBERS ................................13 4.1 STAFF SELF EVALUATION ..................................................................................................................................13 4.1.1 Facilitator/Team Leader:..........................................................................................................................13 4.2 COMMUNITY EVALUATION ................................................................................................................................14 4.2.1 Project Staff (Observer and Co-facilitator): .............................................................................................14 4.2.2 Facilitator from the communities ..............................................................................................................14 5. QUALITY REQUIREMENTS.............................................................................................................................14 6. DATA COLLECTION METHODS.....................................................................................................................14 6.1 PROJECT STAFF SELF EVALUATION ...................................................................................................................14 6.2 COMMUNITY EVALUATION ................................................................................................................................15 7. USE OF EVALUATIONS.....................................................................................................................................16 8. DISSEMINATION OF LESSONS LEARNED...................................................................................................16 9. FEEDBACK OF LESSONS LEARNED .............................................................................................................17 REFERENCES ..........................................................................................................................................................18 ANNEXES ..................................................................................................................................................................19 ANNEX 1. INTERNAL EVALUATION MODEL .................................................................................................19 ANNEX 2. PROCESS OF INTERNAL EVALUATIONS .....................................................................................20 ANNEX 2.1 USE OF DIFFERENT PRA TOOLS .............................................................................................................20 ANNEX 2.2 DIAMOND RANKING PROCESS ................................................................................................................21 ANNEX 2.3: SECONDARY DATA REVIEW .................................................................................................................22

3

ANNEX 3. IMPORTANCE –SUCCESS MATRIX AND FLOW DIAGRAM ....................................................23 ANNEX 4. FLOW DIAGRAM................................................................................................................................26 ANNEX 5. DIAMOND RANKING TO FIND OUT STRENGTHS AND WEAKNESSES................................28 5.1 DIAGRAM OF DIAMOND RANKING .....................................................................................................................29 ANNEX 5.1 PROJECT APPROACHES AND PROCESSES EVALUATION ..........................................................................30 ANNEX 5. 2 EVALUATION OF PROJECT SUPPORT SYSTEM .......................................................................................32 ANNEX 6. SOCIAL AND RESOURCE MAP .......................................................................................................34 ANNEX 7. RESOURCE LOCATION AND PIE DIAGRAMMING ....................................................................37 ANNEX 8. TIMELINE..............................................................................................................................................41

4

Abbreviation
AHW ANR BTRT CBO FCHV IGA LF MG NFE NGO PRA SFCP TBA UC VDC WG Animal Health Worker Agriculture and Natural Resource Management Begnas Tal Rupa Tal Community Based Organization Female Community Health Volunteer Income Generating Activities Leader Farmer Mother Group Non Formal Education Non Governmental Organization Participatory Rural Appraisal Small Farmers Community Project Traditional Birth Attendent User Committee Village Development Committee Women Group

5

ACKNOWLEDGEMENTS
I wish to express my sincere gratitude and profound appreciation to CARE Nepal project staff who have kindly and generously extended their help in the preparation of this Participatory Evaluation Handbook (an Internal Evaluation Handbook for CARE Nepal supported projects). I am deeply indebted to the communities who helped to test the various PRA tools. I am grateful to Ms Shova Gurung, Mr. Padma N. Poudel, Mr. Gopal Shrestha and Om Ghimire for their support during field-test. I am also grateful to Ms. Marcy Vigoda, Dr. Balaram Thapa and Mr. Govinda Rajbhandari and other Senior Management Team members. Similarly, I am grateful to Specialists and Training Officers for their support. Bishnu Pokhrel

6

1. Introduction and Background
Internal Evaluation is relatively a new system in CARE Nepal. For the first time, community representatives were invited in the final evaluations of two projects viz. Begnas Tal Rupa Tal Watershed Management Project (BTRT), Kaski District and Small Farmers Community Project (SFCP), Rapti Zone. These were the first events whereby the community representatives got opportunities to observe and evaluate their projects. In fiscal year 1993/94, CARE Nepal introduced internal evaluation in Remote Area Basic Needs Projects of Bajura and Solukhumbu Districts. Then after Mahottari Natural Resource Management Project, Mustang Natural Resource Management Project, Remote Area Basic Needs Projects (Gorkha) and Upper Andhi Khola Watershed Management Project (Syangja) conducted internal evaluations. There are two types of internal evaluations–Self-Evaluation and Community Evaluation. The project self evaluation is a process in which the project staff and /or others involved in a project analyze and/or measure the achievements and limitations over a period of time. The community evaluation involves the evaluation of a project, process or activities by community members who are involved in the project. It may involve some facilitation by project staff, but this should be minimal1. In the past, internal evaluations were conducted separately and reports were prepared in different ways. In some cases, the recommendations of staff and community evaluations were different and therefore it was found difficult to implement them. Efforts weren’t made to compare the recommendations of internal evaluations for the easy implementation. The evaluation findings were not shared with communities. The new thoughts for internal evaluations are different than in the past. Both the evaluations are conducted separately as before but there must be a single report having common conclusions and recommendations so these could be implemented easily for better programming. In case of community evaluation, no external evaluators (except project staff) are involved. The beneficiaries themselves evaluate their programs.

2. Evaluation Model and Concept of Internal Evaluations

2.1 An Evaluation Model
Internal evaluations are based on three-dimensional evaluation model as given in Annex 1. The overall concept of evaluations is to assess development factors against each evaluation components. The evaluation factors, which are applicable to all CARE Nepal projects, are as follows:

1

This definition is taken from allnepal communication #871, which was written by Ms. Marcy Vigoda.

7

Immediate outputs or achievements of the project Local capacity building Local human resource development Gender aspects in development Use of appropriate technologies Financial and economic aspects Other factors: Depend upon project's goals (impact and effect goals) Some other development factors, which could be project specific, are as follows: Environmental protection Institutional coordination Natural resource management, etc. The internal evaluations must focus on the following components of Output, Effect and Impact goals2: i. Efficiency (achievements of results): How the results stand in relation to the effort expanded. How economically inputs are converted into outputs. Whether the same results could have been achieved in another, better way. Effectiveness (achievements of objectives): The extent to which the purpose (this is called effect goal in CARE) has been achieved; whether this can be expected to happen on the basis of the outputs of the project. Impact (other effects of the project): The changes and effects positive and negative, planned and unforeseen of the project, seen in relation to target groups and others who are affected. Relevance (the direction and usefulness of the project): The degree to which the project can still be justified in relation to local and national development priorities. Sustainability (the long-term viability of the project): An assessment of the extent to which the positive effects of the project will still continue after external assistance has been concluded.

ii.

iii.

iv. v.

A development program is sustainable when it is able to deliver an appropriate level of benefits for an extended period of time after major financial, managerial, and technical assistance from an external donor terminated (Guidelines for Evaluation, Danida, 1994).

2.1 Process of Internal Evaluation
The two types of internal evaluations are very much integral parts of each other and hence cannot be separated. These are conducted separately and may have different findings, but the
2

The definitions of evaluation components are taken from Handbook for evaluators, Royal Ministry of Foreign Affairs, Norway, November 1993

8

conclusions and recommendations should be compared and presented together in one section. The advantage of preparing single report is that the communities and project will have better understanding. This process also helps to provide feedback to the external evaluators and project management team. The process of internal evaluation is given in Figure 1. Figure 1. Process of Staff Self and Community Evaluations Self Evaluation- by project staff and Counterpart Representatives Community Evaluationsby beneficiaries and participating communities

Evaluation findings

Evaluation findings

Conclusions Recommendations Dissemination of lessons learned

3. Evaluation Works

3.1 Decision to Evaluate
Some of the reasons for internal evaluations are as follows:
• •

• • • • •

There is a need of periodic evaluation, which should be done by beneficiaries and project staff to determine and document impacts (both positive and negative). The projects are planned with specific phases and therefore it is necessary to internal evaluations just before the mid term or final evaluations. This may also provide feedback to external evaluations. The plan of future components depends on experience from preceding-phases. To assess whether some of the project activities should be changed, terminated or continued. To identify the problems, their causes and solutions. This is a process of community empowerment and local capacity building. There is a need for building staffs’ capacity to do internal evaluation. 9

3.2 Preparing the Terms of Reference (ToR)
It is recommended to prepare a single ToR for internal evaluation, which shall satisfy the requirements of staff and communities. The following elements are included in a standard ToR. i. ii. iii. iv. v. vi. vii. viii. ix. Project background: project context and rationale. Purpose (objectives) of evaluation: reasons related to project performance and intended use of results. Scope and methodologies: scope of evaluation, type of analysis, methods to be used and degree of details. Issues to be covered: issues related to efficiency, effectiveness, impacts and sustainability. Key questions related to these factors are also written. Evaluation team: number of team members and their roles, required qualification and experiences. Timetable: approximate date and allocation of time for different tasks. Field work: areas to be covered and institutions or groups or individuals to be consulted during and at the end of fieldwork. Reporting: size of report and deadlines for draft version and final report. Reference materials: the recommended documents. paper, publications, books, reports, etc. are included.

These elements are included in a standard ToR. Since project staff and beneficiaries conduct the internal evaluations a short ToR (2-3 page long) covering relevant elements as mentioned above is recommended. This can enable the team to carry out the evaluations smoothly. Depending upon the needs project management can make decision on contents.

3.3 Selection of Evaluation Team
Since staff self evaluation is done by the project staff, it is recommended to ask all project staff to participate the evaluation. Regarding the community evaluation, the experience of evaluation team members is vital to the success. Other personal qualities like the ability to communicate, analytical skill and teamwork capabilities are also important. Therefore, in case of community evaluation, literacy is not an important criterion. While selecting the evaluation team for community members the following factors are considered. i. Experience in rural development: a Team Leader or a Team comprising two to three members should lead the evaluations. S/he should have experience in rural development. The ideal persons for leading the community evaluations would be Community Based Organization (CBO) member, other social workers, Village Development Committee (VDC) members and Local NGO representatives. They would be from the evaluating community or outside the community. It depends upon the project management’s decision. 10

Project Coordinator, Project Manager, Sector Heads, Training Officer or Specialists can lead the evaluation. This also depends upon project decision. ii. Professional expertise: The team members should have professional expertise on the areas planned for the evaluations. For example, a traditional birth attendant (TBA) is not a right choice to ask for evaluating community forestry program. Cross-disciplinarity: The team members should consist of persons with the different professional background. In community evaluation, Leader Farmers, Animal Health Workers, Mother Group (MG)/ Women Group (WG) representatives, Caretakers of rural infrastructure, TBAs, Female Community Health Volunteer (FCHVs), Village Development Committee (VDC) representatives, etc. are involved. This also depends upon the nature of a project to be evaluated. Gender balance: Since the evaluation factors also include gender aspects in development, team members also require having gender analysis skill. The team should comprise of both women and men.

iii.

iv.

3.4 Preparing Evaluation Plan
The evaluation plan provides a clear description of the evaluation. The methods should also be clearly written. The evaluation Plan should describe what the evaluation team is supposed to do, as well as when, where, how and why. Separate evaluation plans can be prepared for staff and community evaluations. Some of the checklists are given below: i. Review of available documents and relevant literature: This includes baseline survey reports and maps/diagrams, project information management system reports, project implementation reports, etc. ii. iii. iv. v. Evaluation questions: discussions of main problem areas. Methods: methodological approaches. Workplan: timetable and work division among the team members. Documentation: The documentation process should also be planned. This includes note taking during evaluation work, writing, drawings, maps and diagram, etc.

3.5 Conducting the Evaluation
3.5.1 Steps to be followed: The following steps are recommended for conducting the evaluations: 11

i.

Orientation to the team members: All evaluation team members should be oriented about the evaluation objectives, methodologies and responsibilities. In case of staff evaluation, an orientation in the beginning is necessary. Initial talks: This includes discussion on ToR, area selection including sample size and other logistical arrangements. Data collection and fieldwork: This includes review of documents, interviews, field studies, field visits, group discussion, etc. Ideally it is not relevant to conduct fieldwork for staff self evaluation. In case of staff self evaluation, field-based staff are key informants and therefore it may or may not be necessary to do field observations. Project progress report review is necessary for assessing the achievements. Participating communities by using visual diagrams and maps does community evaluation. Interviews and group discussions are combined with other Participatory Rural Appraisal (PRA) tools such as resource map and social map to make it more participatory and visual.

ii. iii.

iv.

Data analysis: The team leaders are responsible for data analysis. For the community evaluations, the community members may or may not have analytical skills and therefore project staff have to facilitate to analyze information and draw conclusions and recommendations. In case of staff self evaluation, Project Manager and Sector Heads are responsible for data analysis. Final stage: Finalization of evaluation report and presentation of conclusions and recommendations is the final stage. Project Manager, selected Sector Heads or field staff and selected community representative are responsible for finalization of the report. Debriefing: After finalizing international evaluation report, a debriefing among project staff and selected community members is needed to organize. Field-based staff can debrief evaluation findings and recommendations to community members.

v.

vi.

3.5.2 Writing the report Two separate brief reports are recommended to prepare. Project staff self-evaluation report can be written either in Nepali or in English. But the community evaluation reports are written in Nepali. Then after both reports should be combined to draw common conclusions and recommendations. In the past, most projects had written reports separately and therefore it was difficult to draw conclusions and to implement them for better programming as well. The following elements are recommended to include in evaluation report: 12

i. ii.

Executive summary Summary of conclusions and recommendations Introduction Methodology and approach Scope and limitation Project description Background for the evaluation Project areas Project components Evaluation findings: Efficiency of the project: progress and resource use in the project; achievements of results; results in relation to resource use (if possible) Effectiveness of the project: success of project achieving its goals/objectives; factors affecting the achievement of these objectives Impacts: Foreseen and unforeseen impacts (at individual level and community level); Factors and processes, which explain project impacts. Sustainability: Factors affecting sustainability- economic, institutional, technological, socio-cultural, environmental factors Conclusions and recommendations Lessons learned Annexes : Some of the annexes are as follows: ToR Map of project area and area evaluated. Team Members- community and staff members participated Evaluation schedule Maps and diagrams Other relevant documents

vii.

iii.

iv. v. vi.

It is recommended that the main document of combined report should not exceed 20 pages. Additional information should be kept in annexes.

4. Responsibility of Project Staff and Community Team Members

4.1 Staff Self Evaluation
4.1.1 Facilitator/Team Leader: Prepare ToR. Design evaluation plan Orient project staff who are participating the evaluation. Select and orient community evaluation facilitators Assist to organize evaluation Finalize evaluation report Debrief the findings to staff and other relevant stakeholders 13

4.2 Community Evaluation
4.2.1 Project Staff (Observer and Co-facilitator): Prepare ToR (It could be a combine ToR with Staff Self Evaluation). Design evaluation plan. Assist to select and orient community evaluation facilitators Assist to design evaluation plan and organize evaluation Assist to write evaluation report 4.2.2 Facilitator from the communities Participate in community evaluation orientation Select participants in consultation with communities Facilitate or assist to facilitate community evaluation Assist to site staff to design evaluation plan. Assist to site staff to write evaluation report. Assist to site staff for debriefing. After final report, the evaluation team leaders are responsible for debriefing the outcomes to project office, local level counterpart and communities.

5. Quality Requirements
It is recommended to lower both the level of ambition and level of precision. Obtaining reliable information may be difficult and time/resource consuming. In general, the information obtained and conclusions drawn from the evaluation works should be reliable and valid. The descriptions and conclusions should be valid for more than specific situation i.e. within the project area. It should also be impartial.

6. Data Collection Methods
PRA tools are recommended for internal evaluations. Different PRA tools and their relation for internal evaluations have been put in logical order in Annexes. It is recommended to use appropriate tools depending upon the specific situation and nature of information to be gathered. The following tools are recommended for internal evaluations.

6.1 Project Staff Self Evaluation
The following methods are recommended for staff self evaluation: i. Success-Importance Matrix: To find out successful and failure activities and core activities of the project, Success-Importance Matrix is used. Refer Annex 3 for method of using the matrix. 14

ii. iii.

Flow Diagram: To find out positive and negative impacts of core activities of a project, the Flow Diagram is drawn. For detail refer Annex 4. Diamond Ranking: It is also a PRA tool, which is used to find out strengths and weaknesses of a project or an intervention. The possible areas for analysis by using Diamond Ranking in internal evaluation are program core activities; process and approaches of project and project support system (administrative and finance) See Annex 5 for the detail. Analysis of Gender Aspects: Gender aspect is a cross cutting issue and therefore this is included in all methods. Review of Secondary Data (achievements against the planned targets): Evaluators should also analyze the achievement against project goals. This can be presented in logframe format. By reviewing project database and Project Implementation Report, the achievements can be reported. Refer Annex 2.3.

iv. v.

6.2 Community Evaluation
The following methods are recommended for community evaluation: i. Social and Resource Mapping: Social and Resource maps are useful tools of internal evaluations. The following areas can be covered by using these maps: • Location of various groups in the community. • Location and coverage of human resources such as Animal health Workers (AHW), Leader Farmer (LF), Maintenance Workers, Female Community Health Volunteers (FCHV), etc. • Various development activities implemented with the support of project. Comparison of before and after project can also be done. • The changes in communities can be identified. • Coverage of the project can be seen. The details are found in Annex 6. ii. iii. iv. Importance - Success Matrix. See section 6.1. Diamond Ranking. See section 6.1. Resource Location: Resources allocated for core activities is evaluated by using this tool. Based on resource allocation, sustainability of project activities can be assessed. See Annex 7 for details. Pie diagrams are drawn after Resource Location exercises. Trend analysis of contribution on the core activities by CARE, community, VDC, counterpart and other 15

agencies can be found. The trend analysis (past, present and future), gives the picture of sustainability of project activities. See Annex 7 for details. v. Timelines: (only for the core activities): This tool is used to assess impacts of the project and communities’ capacity at the local level to manage development activities. See Annex 8 for details. Analysis of gender aspects: Since it is a cross cutting issue, it has to be incorporated in all exercises of internal evaluations. It is expected that the evaluation team members will cover the following areas: Gender related needs − Practical needs: access to basic commodities such as water; access to basic social services; access to IGAs, etc. − Strategic needs: Reducing inequalities in the division of labor between sexes; reducing the burden of work connected to home and child-care; access to credit, etc. • Gender role analysis: − Community management: Work in maintaining societal functions such as water supply, health services, education, etc. − Reproductive roles − Productive roles

vi.

7. Use of Evaluations
Major uses of internal evaluation are as follows: Evaluation findings have wider use. If the internal evaluations are conducted just before external evaluations (mid-term or final), the outcomes can influence the external evaluation. Project management team can use the findings and recommendations for improving program qualities. This is a process for empowering the local communities and building the local capacity in monitoring and evaluation. It is also a process of building skills and increasing knowledge of project staff.

8. Dissemination of Lessons Learned
The lessons learned could be used in the following areas: Annual reports and Project Implementation Reports preparation. Annual Implementation Plan and Annual Operating Plan development. Seminars and workshops. Counterpart reporting. Project briefing papers. 16

Feedback for external evaluations. Revision of project approaches and strategies.

9. Feedback of Lessons Learned
The lessons learned can be used as feedback for: New projects or re-designed projects Improving program qualities Improving management support Changing the strategies and guidelines Improving the service delivery system. Feedback for communities Others

17

References
1. Evaluation of Development Assistance, Handbook for Evaluators and Managers, Royal Ministry of Foreign Affairs, Norway, November 1993 2. Guidelines for Evaluation, DANIDA, February 1994 3. Ideas for Project self-evaluation and community Evaluation, Marcy Vigoda, June 1996 4. Community Evaluation Report, Buffer Zone Development Project, Bardia, June 1999 5. Various PRA notes.

18

Input

Efficiency

Process

Sustainability

Output

Annexes Annex 1. Internal Evaluation Model

Relevance

Effectiveness

Effect Impact

Impact

Achievements of the project Local capacity building Local human resource development Gender aspects in development Use of appropriate technologies Financial and economic aspects Environmental protection Institutional coordination Development Factors

Note: This model is adapted from Hand Book for Evaluators, Royal Ministry of Foreign Affairs, Norway, 1993

Annex 2. Process of Internal Evaluations

Annex 2.1 Use of different PRA tools
Social and Resource map Importance – Success Matrix Most important activities Least important activities Most successful activities Least successful activities Core activities Flow Diagram

Coverage by Groups, local HR and activities Changes- baseline vs. current Major problems and their causes and solution

Positive impacts, negative impacts, their causes and solution

Timelines

Resource location of core activities Sustainability of core activities

Pie Diagram or Bar Chart Trend analysis of contributions/ subsidy

Communities capacity to continue the core activities

Note: It is not always necessary to combine all the tools together. Evaluators decide the right tools for internal evaluation.

20

Annex 2.2 Diamond ranking process
Processes and approaches of the project Strengths Causes & solutions Weaknesses

Core activities

Strengths Causes & solutions

Diamond Ranking

Weaknesses

Project support system (administrative and finance)

Strengths Causes & solutions Weaknesses

21

Annex 2.3: Secondary Data Review

Project logical framework progress report

Project Achievements

Synthesis of Annual Project Information (API)

Synthesis of Project Information Management System (PIMS)

22

Annex 3. Importance –Success Matrix and Flow Diagram
The objectives of drawing Importance –Success matrix is to find out successful and important activities of the project, which will enable to develop flow diagram for evaluating impacts (positive and negative). The following steps are recommended: A. Finding important and successful activities and thereby core activities i. ii. iii. iv. v. Divide the participants into Community Organization group (including NFE and IGA), ANR, Infrastructure and Health. The minor activities are not recommended to include in the evaluation. Ask the participants to list the activities implemented. The facilitator should have a checklist of activities. The participants can use symbols to indicate the activities. Ask the participants to categorize the activities in order of importance. Then again ask them to put into three categories- Most Important, Average Important and Least Important. Ask the participants to categorize the activities in order of success (as most, average and least successful). Then again ask them to put into three categories- Highly Successful, Average Successful and Least Successful. Summarize the activities in the following matrix

Importance - Success Matrix Importance High I Average (Medium) IV Low VII

Successful

Average (Medium)

Most

II

V

VIII

Least

III

VI

IX

vi. vii.

After completing the matrix, start to identify core activities. The core activities should be from cell I, II and III. Activities fall in cell IV and V can also be considered as core activities. Identify the core activities and share with the participants. Make conclusion based on the matrix result.

B. Finding of positive and negative impacts of the core activities: viii. ix. x. xi. xii. Divide the participants into different groups. Let them to choose the core activities (at least two activities per group). Brief the task. Ask the participants to list positive and negative changes/impacts of the selected activities. The impacts should be presented in a flow diagram as given in Annex 4. Give specific examples for each change. After finding the positive and negative impacts by the small groups, share it in plenary. Again, ask the small groups to go back and do the analysis on negative impacts. The causes and solutions should be identified for the negative impacts. Conclude and present.

24

An Example of Importance - Success Matrix Importance High

Average (Medium)

Low

Most

• • • • • •

Average (Medium)

• • • • • • • • •

Pit latrine Slab culvert Boring Kitchen garden Dalit Group Formation Plantation Nursery establishment Demonstration Women Group formation User Committee formation Leader Farmers Selection Hand pump Goat raising Improved Cook stove Forest Management Plan preparation Agroforestry farm management training

• •

Successful

Non formal education Cross visit

Source: Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15 June, 1999

Before filling out the matrix, the participants had set criteria for Important Activities and Successful Activities. Based on the set criteria, the participants discussed the reasons for high importance, Most Successful, and so on. The final outcomes were also included in the report. Core Activities of the User Committee: 1. Pit latrine 2. Slab culvert 3. Boring 4. Kitchen garden 5. Plantation 6. Nursery establishment 7. Demonstration 8. Group formation 9. Leader Farmers Selection 10. Hand pump 11. Goat raising 12. Non formal education 13. Improved Cook stove 14. Agroforestry farm management training 15. Management training

Source: Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15 June, 1999

In this example, the evaluation concluded that Cross Visit program is not the core activity.

Least

25

Annex 4. Flow Diagram
DRINKING WATER SUPPLY SYSTEM ESTABLISHED Piped Water available Less water for crop land irrigation (One lps decreased) Water User Committee (Out of 11 EC members, 3 are women and 2 are Dalits) Crop production deceased Conflict raised (Out of 15 taps, 2 closed due to conflict) Maintenance fund raised Workload of children increased Instability of system Maintained & repaired the system regularly (one paid caretaker exist) Water system sustainable Use of surplus time in productive works (villagers reported) Production of meat & milk increased Causes of disease (villagers reported) Livestock health improved Availability of green vegetables (villagers reported) Time saved (average 30min) Water for cattle (from 8 tapstands) Water for kitchen garden (from 7 tapstands) Safe drinking water available Poor drainage (5 tapstand posts have) Personal hygiene improved

Other community development works affected

Less food available (farmers perception)

Income increased

HEALTH STATUS: improved/ not improved

Negative impacts

Positive impacts

An example of Flow Drawing
Irrigation Boring Water available for farm land irrigation Due to insufficiency of irrigation water, conflict raised during cropping season Drainage problem raised during off season

Production increased Conflicts and other problems raised Family income increased

Income increased

Source: Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15 June, 1999

27

Annex 5. Diamond ranking to find out strengths and weaknesses
Diamond ranking is a tool, which can be used to identify strengths and weaknesses of a project. Three areas of the project can be analyzed. They are: 1. Project core activities. 2. Project processes and approaches. 3. Project support system (administration and finance). After finding strengths and weaknesses, further analysis can be done. The analysis of results gives the efficiency and effectiveness of the project. The following steps are recommended; i. ii. iii. iv. v. vi. vii. Brief the task to the participants. Divide the participants into three groups- project core activities, process and approach and project support system. In the small group, ask the participants to list strengths of the given topic. Let them to rank the strengths in order. Again ask the participants to list the weaknesses of the given topic. Let them to rank the weaknesses in order. Put the strengths and weaknesses in Diamond Matrix. Share outcomes in plenary: Find out the reasons of weaknesses and their solutions, and supporting factors for strengths. Conclude and present.

28

5.1 Diagram of Diamond Ranking
I

II

II

III

III

III

IV

IV

IV

IV

4

4

4

4

3

3

3

2

2

1

Shape of Diamond Ranking is given in the figure above. Ideally it would be nice if the strengths and weaknesses can be presented in a diamond shape. But it is not necessary that the project should always have equal number of strengths and weaknesses. Therefore some of the boxes can be kept blank. For example a project may have ten strengths and five weaknesses. In such case five boxes in weaknesses are blank.

29

Weaknesses

Strengths

Annex 5.1 Project Approaches and Processes Evaluation
Some of the areas that can be analyzed by using Diamond Ranking are given below. 1. Beneficiaries participation in project − Planning- Infrastructure, Agriculture, Forestry, NFE, IGAs, Group formation, Training, others (specify) − Project implementation- Infrastructure, Agriculture, Forestry, NFE, IGAs, Group formation, Training, others (specify) − Project monitoring (participatory monitoring)- Infrastructure, Agriculture, Forestry, NFE, IGAs, Group formation, Training, others (specify) 2. Maintenance of facilities − Infrastructure (drinking water system, irrigation system, others) − Agriculture (kitchen garden, agroforestry farm, orchards, etc) − Forestry (Community plantation, private plantation, community forestry, etc) − IGAs 3. Human Resource Development − Agriculture sector (Leader Farmers, Animal Health workers) − Infrastructure (Caretakers) − Forestry sector (Nursery Manager, Heralu, etc) − NFE sector (Facilitators, Supervisors) − IGA (Masons, Carpenters, etc) 4. Institutional coordination 4.a CARE

UC/CBOs

Counter part

4.b Institutional coordination CARE VDC

Counter part 30

4.c Institutional coordinationCARE DDC

Counter part 4.d Institutional coordinationCARE Other agencies

Counter part 5. Local Institutional capacity building − User Committee/User Groups − Women Group − Disadvantage Support Group/ Dalit Groups − Other Groups (specify) 6. Evaluate sectoral activities 7. Implementation modalities − Partnership with NGOs, CBOs and Government Counterpart. − Direct implementation 8. Counterpart staff deputation to the project and its positive impact

31

Annex 5. 2 Evaluation of Project Support System
Under the project support system the following areas can be covered. i. ii. iii. Administrative support Financial support Logistic support

The facilitators should identify the areas that have to be analyzed. Since the evaluation depends upon the nature of project, it cannot be generalized. Administrative and financial support systems can be lumped together for analysis. This depends upon participants’ choice.

32

An Example of Diamond Ranking
Construction materials are available on time

For plantation and agroforestry, the saplings are available on time.

Technical support available in time

Good quality seeds distributed for increasing crop production.

People participated in latrine and DWS construction

Financial and technical support for tube well boring provided

Clean drinking water available from hand pump

Goat raining program is effective for income generation

Private pipes have not been provided to Dalits

Projects has not led the forest protection works

Delay in project planning and decision making

Programs have not been not launched for poor as Dalits

Weaknesses

Conflicts raised in the communities due to insufficiency of water

Project does not work for river training works

Failure in plantation protection

Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15 June, 1999

33

Strengths

Awareness program and infrastructure projects are implemented.

Availability of irrigation water through tube well boring and drinking water system

Annex 6. Social and Resource Map
Social and Resource maps can be used for internal evaluations. Both of the maps can be combined together. This is an easy way of involving illiterate people in the participatory community evaluation process. The following steps are recommended for evaluation: i. Choose appropriate place for drawing the maps. Cleaned ground without grass is the best place for drawing maps. Since the whole process may last for about three hours it is recommended to choose comfortable place. Ask participants to select the persons who want to draw social and resource map. Let the participants draw the maps on ground by using locally available materials. Check whether all the visible things such as village road, forest, physical infrastructure, school, VDC building, stream, etc. are included in the map or not. Ensure that the following things, if applicable are also located in the maps − Location of Community Based Organizations (CBOs) and NGOs in the community − Available human resources such as Animal Health Worker (AHW), Leader Farmer (LF), Caretakers, Female Community Health Volunteers (FCHV), etc − Coverage of core activities and mapping the coverage of activities − Baseline vs. current activities. vi. vii. Ask the participants to list major problems such as deforestation, landslide, etc. and locate in the map, if possible. Do analysis of the problems- (problems-causes-consequences) and find possible solutions

ii. iii. iv. v.

viii. Conclude and copy the map in book.

34

An Example of Use of Social and Resource Map After preparing the Social and Resource Map, the participants listed the major activities of their village. Then, before the project and after the project has been compared. The outcomes of Social and Resource Map preparation is given below: SN Activity Before Project After Project 1 Hand pipe 21 25 2 Kitchen Garden (improved) 26 3 Treadle pump 3 4 Boring for irrigation 3 5 Improved Cook Stove 5 6 Latrine 38 7 Slab culvert 1 8 Nursery 1 9 Non formal education 5 6 10 Leader Farmer 4 11 Household covered by Leader 43 Farmers 12 Demonstration 7 13 Goat Raising (IGA) 15HHs 14 Community plantation 4 ha 15 Private plantation 15 Kattha by 9 HHs 16 Agroforestry farm 2 nos 17 Study visit 6 persons 18 Women Group formation 1 19 User Committee formation 1 20 Forest Management Plan 1 preparation and discussion 21 Orientation workshop for User 1 event for 11 members Committee 22 Management training 3 members 23 Leader Farmers training 4 24 ICS training 1 25 Goat Raising training 14 26 Nursery Naike training 1 27 NFE facilitator training 1 28 NFE Supervisor training 1 29 Fruit tree pruning training 3 30 Dalit Sub Committee formation 11 31 Provision of Caretaker (Heralu) 1 for plantation 32 Gender awareness training 11
Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15 June, 1999

35

Problems, their causes and the solutions Based on the Social and Resource Map, the participants were asked to find out the major problems, their causes and the solutions. An example is given below: SN 1 Activity Hand pump Problems The hand pump does not work Causes Unavailability of good quality washer in the village Breaking of hand Lack of maintenance pump parts training Low discharge of the Unavailability of pump experienced persons for pump installation Open grazing system exist No fencing Solutions Management of good quality materials Provision of maintenance training Provision of experienced persons for pump installation Control of open grazing system. Provision of barbed wire or live fence

2

Plantation Damage by the domestic animals Low survival rate

Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15 June, 1999

36

Annex 7. Resource Location and Pie Diagramming
Resource location analysis is a PRA tool, which is used to find out the resources used for an activity. The resources can be divided into three types as follows: − Community resources − CARE/the Project resources − Outside resources The following steps are recommended to find out various resources allocated for an activity: i. Choose the activities to be analyzed and share with the participants. ii. Ask the communities to list the resources required for each activity. iii. Distribute the resources in the matrix given in figure below.

Symbols in visual form

Symbols in visual form
ooo

Symbols in visual form
o

ooo ooo

Community resources Name Locally available materials Symbol

CARE/the Project resources Name Symbol Technical support Money

Other agencies' resources, specify Name Symbol Money $

$

iv. v.

To know the contribution percentage of different agencies give Soya bean or maize seeds or any types of locally available materials to the participants and ask them to distribute to different segments as per the actual situation. Take note simultaneously.

37

vi. Do analysis for each activity by asking the following questions: − Has the activity created dependency on the project? What was the past trend? − If the project has stopped to support the resources, would it be continued? − What should be the activity phase-out strategy? vii. Conclusion: Draw conclusion for each activity in the following areas: − Appropriateness of use of project resources − Sustainability of the activity after the project − Use/ mobilization of local resources viii. Repeat the above processes (step i to vii) for other activities. Drawing the Pie Diagram: To analyze the contribution status of core activities by different stakeholders, Pie Diagram or bar chart are used. Take the information from resource location exercise, and do the analysis as follows: i. SN 1 2 3 4 5 6 7 8 9 10 ii. Draw the Pie Diagram as follows: • To draw Pie for row values, it is straightforward. • To draw the Pie for column values, take five activities (maximum) at a time, calculate average percentage and draw Pie. Put the information in the following table. Core activities Contribution by (percentage or other appropriate unit) Project Communities VDCs Counterpart Other agencies

38

Pie row

Pie column

iii. Analyze the findings in the following patterns: For this pie diagram should be related with Importance-Success matrix, which gives core activities. Subsidy/contribution High Low Important Most Least Successful Highly Least

The possible combinations are: I. High subsidy, most important and highly successful II. High subsidy, most important and least successful III. High subsidy, least important and highly successful IV. High subsidy, least important and least successful V. Low subsidy, most important and highly successful VI. Low subsidy, most important and least successful VII. Low subsidy, least important and highly successful VIII. Low subsidy, least important and least successful ii. Type I: Type II: Type III: Type IV: Type V: Type VI: Type VII: Type VIII: Conclusions: Relatively better than the other categories. These are the areas of strengths. Find out the reasons of least successful activities. Does the project want to move it to type I? Need to rethink and further analysis. Drop it. The best areas. Continue. Find out the reasons of least successful activities and gradually improve the program. Need to rethink Why do we want to continue this activity? Drop it.

Note: This exercise can also be done during staff self evaluation by using secondary data from the project file.

39

Example of Resource Location and Pie Diagramming Activities Boring for Irrigation Community • Unskilled labor contribution • Meals for skilled labor Project • Pipe • Rent for pumping set • Skilled labor cost • All other imported materials needed for the tube well boring • Copy and pencil • Kerosene • Lantern • Blackboard • Tin Trunk • Training • Cash for facilitator and supervisor
• •

Other Agencies • Not involve

Non Formal education

• •

Room for class Copy and pencil

Kitchen garden

• • • • •

Plantation

• • • • •

Labor work Land Fertilizer Tools Others Transportation Tools Labor work Fencing (temporary) Others

Seed Technical training

• •

Saplings Technical support

• •

Delegation of authority Management plan

SN 1 2 3 4

Core activities Project Boring for Irrigation Non Formal education Kitchen garden Plantation 87% 90% 5% 25%

Contribution (percentage) Communit VDCs Counterpart ies 13% 0 0 10% 95% 70% 0 0 0 0 0 5%

Other agencies 0 0 0

Source: Community Evaluation Report, Asneri User Committee, Buffer Zone Development Project, 14-15 June, 1999

40

Annex 8. Timeline
Timeline can be used to assess the inputs of the project and communities' capacity building at the local level to manage development activities. The following steps are recommended: i. ii. iii. Choose an activity or process or approach to be analyzed. Choose appropriate variables. Check the knowledge of participants about the chosen topic. Such as if the topic is "People's participation in development works", then the participants should know the past trend, present status and expected trend in future. iv. Prepare appropriate place for doing the exercise and choose appropriate materials such as maize seeds. v. Draw boxes for each variable. An example is given below: People's participation in development works Project Started People Participation

Men Women

oo oo oo

oooo o ooo ooo

ooooo ooooo ooooo

ooooo ooooo ooooooo

10 yrs ago Men Villagers would like to participate Women
ooooo ooooo ooooo oo

5 yrs ago
oooooo oooooo ooooo ooooo

2 yrs ago
oooooo oooooo oooooo oooooo

now

now

2 yrs in future End of Project

5 yrs in future

i.

Draw conclusion and make recommendations.

41

An Example of Timeline Time Line of Pit latrine construction 5 years ago 0 2 0 2 5 years ago 30 53 2 years ago 0 9 0 9 2 years ago 25 28 At present 2 36 2 36 At present 10 15 2 years after 10 50 10 45 2 years after 5 10 5 years after 28 60 28 60 5 years after 0 0

Number of Household s Use of Latrine

Dalits Others Dalits Others

Open grazing

Dalits Others

Source: Community Evaluation Report, Ranjha Bij Tole User Committee, Buffer Zone Development Project, 14-15 June, 1999

42

Sign up to vote on this title
UsefulNot useful