1

“ Rapid Appraisal Methodologies for Assessing Impact”
Notes edited by the consultant and the FAO Evaluation Service

A. General The discussion focussed on Rapid Appraisal Methodologies (RAM) to be used (1) at field level and (2) during a project evaluation mission. The basic feature distinguishing RAMs from more "impressionistic" methodologies is the use of "triangulation" in order to cross-validate all findings. Triangulation means gathering data from (at least) 2 different sources - preferably using 2 different techniques (e.g. key informant interviews, focus groups, observation) – for each variable or issue being studied. The following steps of a rapid appraisal were outlined. They are to be used in approximately this chronological order or, if time and resources allow, concurrently: 1. Literature Review = from technical journals; case studies, demographic data and general reference documents and maps; and locally available project reports. 2. Quantitative data gathering = from statistical documents (central bank, statistical office) available locally and at national levels; project reports; analysis of existing data. 3. Interviews with key informants from both an outsider’s and insider's angle of vision: • • outsiders may include other donors, competing NGOs, other institutions; insiders can be national and local authorities overseeing the project; project management and front line field and project staff.

During this phase, the objective is to identify underlying issues and develop a first topic list to be used for field investigations. 4. Interviews with focus groups and key informants at regional, local and village/field levels. At this point, Ms. Blumberg emphasized the need for: • small (5 being the ideal number), homogenous focus groups. Homogeneity within the group being interviewed means avoiding groups composed of people with conflicting interests and/or resources within the community (e.g. large landowners and landless families and, in most places, men and women should be interviewed separately); the use of control groups1 selected from non-project’s clients but with salient features similar to the project's clientele (focus groups);

Control groups can help ensure that the results being verified are truly the results of project activities and not due to ongoing events or other externalities. They can also serve as the basis for comparison to measure effects, impact, and changes in living/working conditions brought about by the project (see also para. 4 below). Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999

1

At this first level. religious and racial-ethnic variations within the community. gender. FAO. at the end of the investigative process. quantify findings and/or provide quantitative information in support of these findings. obstacles to overcome. Social. Mini-surveys would be used as a way to: • • • establish parametric data about the people in the project. shop. 8. and. administrative. religious and racial-ethnic variations tend to be linked and usually can be correlated with class differences. was noted. and discern measures to ensure the sustainability and/or "institutionalization" of results. traditional. Discussions here focused on the use of multiple stakeholder brainstorming sessions and inter-institutional workshops to: • • • • evaluate the validity. those that possess a moderate level of resources (land. verification and discussion of mission's findings at institutional level. Field Observation = used to verify what is actually being done by the participants themselves (e. In the absence of marked social divisions involving caste. ideally. and should preferably involve random sampling and control groups. livestock. findings and issues that have emerged are shared at a village (large) group meeting to seek confirmation. answer questions that remain open. Notes based on the Seminar given in FAO th By: Professor Rae L. class differences could be initially identified at three levels: the landless/semi-landless. use being made of outputs). household possessions). and. Homogeneity can be achieved by identifying different categories of focus groups distributed according to: agro-ecological areas.g. These would be composed of a few well-defined/focused questions. lists of clients that can be used for eventual random sampling (see mini-surveys below).2 • the use of outsider key informants such as staff at health centres. and the elite . 6.rich and/or village authorities (religious. 7. credit/lending institutions and/or schools who are knowledgeable about the community and possess both qualitative information about the local situation and. Feedback and verification of mission's findings back and throughout the pyramid of stakeholders starting at village level. spiritual. social. the possible use of mini-surveys. relevance and potential use of results with insiders and outsiders. Rome. assent and ideas for potential use and/or reforms. what has been achieved versus what is being reported. Finally. May 18-19 1999 . tangible measures of standard of living (such as type of housing. identify additional success criteria. assist in future work-planning and detect administrative. or other. the level/quality of use of land conservation practices. etc). 5. race-ethnicity and/or religion. Blumberg. especially those with complex social settings where it is difficult to identify the right weight to give to the various social components and focus group distribution. etc). Feedback. age groups.

a system of semi-annual or annual focus group meetings can be established to collect information/data for indicator measurement (results and impact) and to detect problems. unanticipated negative effects of the project not found in control groups. These focus groups should help suggest the M&E data to be collected to followup progress. They can also be chosen at random from adjacent villages not covered by the project if sampling lists are available. results need to measured according to success criteria that come. The choice is to be made on a case-by-case basis either to ensure continuity of the information gathered. At this level.g. etc. a general economic downturn or boom. or drought. such as identifying broad socio-economic features of the project clients. or to guarantee the validity of the findings by enlarging the number of people covered by the reporting system. Control groups need to be utilized to serve as the basis for comparison from which to measure changes brought about by the project. Blumberg. at least partially. very early in the project cycle. This helps ensure similarity in wealth.B. 2. Instead. or be maintained as panel members. affecting both project and non-project households. FAO. e. mini-surveys should be focused on specific issues and targeted on pertinent groups. from within the target groups. 3. However. In either case. village focus groups need to be selected to develop indicators of success and the consequent M&E system. Rapid Appraisal Methodologies for Assessing Results (for M&E at Village level) 1. they can provide comparative information on indicators such as: • • • the spread (use being made) of results into non-project areas. and to help select new data to be included in the system as activities develop. For this purpose.3 N. N.B. the incidence of externalities in project results. Impact indicators to be gathered at ultimate target group/village level can estimate three main impact areas: • • what has happened to the business/individual plot area? what has happened to the participant/stakeholder including increases/decreases in empowerment and wellbeing? Notes based on the Seminar given in FAO th By: Professor Rae L. there are generally more drawbacks to the panel method in these rural settings. B. in the whole area. May 18-19 1999 . Therefore. education and other characteristics with the focus groups. Mini-surveys should not be used for general information gathering. Participants of the focus groups may change at each data gathering exercise. Control groups can be derived from focus groups in a “snowball sample” by asking one or more focus group participants to suggest persons similar to themselves but who are not involved in the project. Rome.

Village reporters should work with project staff to convert findings and data gathered into the more formal institutional M&E system. a designated M&E village representative. can take notes on the focus group discussion. Men tend to spread increased income under their control over a wide range of activities. e.4 what has happened within the households including increases/decreases in children’s (male and female) nutrition. May 18-19 1999 . If no M&E staff person is present at the focus group. as well as to join the staff of M&E people in orally presenting the results of the latest focus groups to project management. if at all possible. time constraints. • C. MULTIPLE STAKEHOLDER ANALYSIS The steps and criteria for identifying multiple stakeholders are somewhat similar to the selection of focus groups. N. Data collection should be done regularly (e. dynamic participants should be recruited and trained to help facilitate subsequent waves of focus groups. certificates. education and health. improved healthcare.g. Data gathering should encompass some form of remuneration and recognition. or other interested and literate volunteer. changes in the division of labour. Villagers should progressively become stakeholders in the M&E system.? N.B. improved family diet b. improved education for children c. Rome. FAO. 4. Blumberg. in order to convert him/her into a stakeholder. honours. • • • Interested and potential candidates for the data collection exercises should be identified and trained (while on-the-job) at each field visit. etc. transport costs. a school teacher. However. and the village facilitator is not totally literate. The outsider helping to set up the results-oriented M&E system should be accompanied to the field by at least two project staff members who will be in charge of maintaining the system and. 5. their boss. several intelligent. by semester).g. whereas women with increased income under their control tend to spend it in a much more focussed manner on: a. who can be a child or youth of the village. etc. stakeholders start at the top of the pyramid at national/donor level and proceed down to the local/village levels as follows: Notes based on the Seminar given in FAO th By: Professor Rae L. During the focus group phase.B.

FAO. Regional/local level 3. D. Authorities 2. PROGRAMMING RAM INTO EVALUATION MISSIONS Notes based on the Seminar given in FAO th By: Professor Rae L. Finally.poor/landless families 9. Staff from NGOs. Local front line staff – extension workers and local health workers. Lower stratum -men . Other donors/NGOs 2. private sector. the minisurvey questionnaire to be carried out if and when authorities at country level so warrant it. Youth (male and female) – from within the project 10. National/donor level – line ministry/donor – admin. It is important that the information given by project-related focus groups be crosschecked through group meetings with a control group (or groups). in consultation with management and project staff and/or the various "outsider" categories. May 18-19 1999 . if time and resources permit. Community leaders & large landowners – traditional. Rome. Middle/lower stratum women 5. Project management & front line staff – extension workers and local health/school workers For each agro-ecological zone: 4. Other minorities/ethnic classes (covered by the project) – if not represented in the above classifications N. attention should be paid to on-the-job training of a followup M&E team (at least two staff members. Middle stratum -women . Blumberg. Middle stratum -men . Additional stakeholders can be identified as the fieldwork progresses. religious 5. school teachers From nearby and similar areas: 1. other/similar projects 3. Middle/lower stratum men 4. This minimum of 15 categories of focus groups containing a wide range of stakeholders needs to be covered.B. Other minorities/ethnic classes (covered by the project) – if not represented in the above classifications 3. Lower stratum women . and selected stakeholders at village level). a member of project management.average households 6.5 TABLE 1: Category Insider Stakeholders (Control group) Outsiders 1. early in the project cycle. Additionally.poor/landless families 7.average households 8.

At national level.e. on day 2) can bring together all of the authorities and donor representatives engaged in project activities and reduce the number of purely "formal" visits. N. 3. if possible. Rome. all focus groups in the first round of data collection will also be used to familiarize the project staff who will manage the ongoing M&E system. Discussions at these initial meetings focus on the objectives and scope of the project and its relative comparative advantages and relevance to the country/sector/region. From the above. with the techniques involved in running successful focus groups. Blumberg. At village level. an initial "brainstorming" session with all of the project staff can be organized. Its purpose is to review together the weaknesses and strengths/benefits and achievements of the project. 5. After travelling to the field. These questions can be modified based on the results of early interviews.4). • Focus group interviews can be carried out by one or more of the mission members in an iterative.6 Nearly all of the requirements for the use of RAM can be planned and written into the TORs and mission field visits. a list of topics for further investigation can be drawn. per agro-ecological zone. should be programmed into the field visits. May 18-19 1999 . together with the main findings and issues emanating from the literature review (see section A. who will have a continuing commitment to an ongoing M&E system. a similar regional/local initial meeting can be arranged to bring together a large number of stakeholders.1) will form the basic list of questions to serve as a guideline for subsequent key informant interviews and focus group meetings (see below). question and answer mode with all of the focus group members (see section A. 1. interactive. in close consultation with project and national authorities. It can include government and NGO staff. FAO. Notes based on the Seminar given in FAO th By: Professor Rae L. Once in the project's headquarters. in most cases at least the 10 main categories of "local stakeholders" (as well as 5 control groups).B. it remains necessary to complement this briefing meeting with a few key informant interviews to further delineate the important issues. This list. private sector representatives and local authorities. selected focus group and key informant interviews need to be planned beforehand to encompass the various categories and classes of stakeholders. 4. Ideally. However. This is of course to be done on a case-by-case basis according to the nature and geographical coverage of project interventions. Another objective is to recruit people who will eventually become the stakeholders in the evaluation process i. 2. • However. a large briefing meeting (to take place. as well as the selected villagers.

Rome. presented and facilitated by the team members. the mission should seek to provide on-the-job training to at least two staff and two stakeholders at village level in order to leave behind a follow-up team for continuous M&E. Feedback throughout the various layers of the "pyramid" needs also to be programmed through a series of group meetings to be used for restitution.e.7 • Specifically. opportunities and limitations to ensure continuity. an inter-institutional evaluation workshop can be planned with representatives of insider and outsider organizations. • At both levels. The specific success criteria by which each of the stakeholder groups judge the success of the project. potential effects. project results are discussed to identify: a. May 18-19 1999 . b. At village level a town/local meeting can be programmed with members and non-members of project activities. cross-checking. in order not to taint the self-evaluation process and initiate a process by which stakeholders have an interest in the continuous review of project results. The potential. Discussions should lead to an understanding of weaknesses and strengths present in the project. sustainability and/or institutionalization. • In both cases. Feed-back 6. Notes based on the Seminar given in FAO th By: Professor Rae L. and c. FAO. The use being made (or to be made) of project results i. as well as project personnel ranging from field staff to top management. 8. • In both cases. At national/regional level. discussions should lead to a better understanding of evaluation requirements and a commitment to maintaining an M&E process as feedback to both the community and the implementing institutions. 7. validation and eventually negotiation of the mission's main findings. PRA visualisation techniques may be introduced. Blumberg. as well as to the planning of consequent and agreed upon follow-up actions.

leaders) Outsiders (schoolteacher. production systems • • • Key Informant Interviews at Community Level Leaders (traditional. Blumberg. youth.) • • • • • Focus Group Interviews with Control Group ** (non-participants in project.8 Annex 1 Impact Evaluation PRA Sequence • • Review of Secondary Data Desk review In-country sources and grey literature • • • • Key Informant Interviews National level (insider/outsider) Regional (government. civil society) Front-line community workers (extension. Rome. ethnic.) Inter-institutional Seminar * • Zoning and Purposive Sampling of Communities Agro-ecological. private sector. etc.) • • • Supplementary Cross-checks Field Observation Content analysis of newspaper articles PRA exercises within a Community Setting Notes based on the Seminar given in FAO th By: Professor Rae L. non-beneficiaries) Middle stratum (average households) Male Middle stratum (average households) Female Lower stratum (poor households) Males Lower stratum (poor households) Females Other (ethnic minorities. modern) Upper stratum (rich households. FAO. health post) • • • • • • Focus Group Interviews with Project Beneficiaries Middle stratum (average households) Males Lower stratum (poor households) Males Middle stratum (average households) Females Lower stratum (poor households) Females Youth groups (Male and Female) Representatives of Other (ethnic minorities. May 18-19 1999 . etc. etc.

etc. Blumberg. UNDP. etc. Rome.) Heads of line departments Project Management • • • • Quantitative Survey ("last Step") Parametric data To cover unanswered questions or To fill gaps in data To Substantiate Controversial findings • • • • Feedback to National Level National project team Minister Donors (FAOR. May 18-19 1999 .) Inter-institutional Seminar * * PD .9 Household Interviews (few) • • • Feedback to Community at Village Meeting For Cross-checking and validation Restitution of Findings Negotiation of possible solutions • • • Feedback to Project Staff To Front-line workers (extensionists. FAO.The Inter-institutional Seminar can be held ex-ante for impact evaluation to review: • validity and pertinence of project goals and timing • level of dissemination & institutionalisation of project results • the probable effects of externalities and the nature of these on project results • success criteria to be considered in addition to foreseen criteria. The Inter-institutional Seminar can be held ex-post to the field visits to review: • validity and pertinence of project goals to national policies and within the national context • validity and pertinence of project activities to ongoing efforts in partner institutions • level of dissemination & potential for institutionalisation of project results • success criteria to be considered in addition to foreseen criteria Notes based on the Seminar given in FAO th By: Professor Rae L.

through a semi-structured interview with open and closed questions derived from the former 4 steps. Rome. FAO. etc. Notes based on the Seminar given in FAO th By: Professor Rae L. participants/non-participants.10 ** Focus Group Modality = no more than 5 group members. May 18-19 1999 . Blumberg. adults/youth. Facilitator / Interviewer Seasonal Calendar Village Diagram/Transect Value Matrix Focus Group male/female.

Much on-farm research falls in this category. It is a unilateral announcement by an administration or project management without any listening to people’s responses. Blumberg. FUNCTIONAL PARTICIPATION People participate by forming groups to meet predetermined objectives related to the project. which can involve the development or promotion of externally initiated social organization. as farmers provide the fields but are not involved in experimentation or the process of learning. Rome.11 Annex 2 How People May Participate in Development Programmes and Projects2 1. These external agents define both problems and solutions. FAO. London. cash or other material incentives. The information being shared belongs only to external professionals. but may become self-dependent. People do not have the opportunity to influence proceedings. These institutions tend to be dependent on external initiators and facilitators.Policies and Practices for Sustainability and Self-Reliance”. PASSIVE PARTICIPATION People participate by being told what is going to happen or has already happened. Notes based on the Seminar given in FAO th By: Professor Rae L. PARTICIPATION IN INFORMATION GIVING People participate by answering questions posed by “extractive” researchers using questionnaire surveys or similar approaches. as the findings are neither shared or checked for accuracy. 7. PARTICIPATION BY CONSULTATION People participate by being consulted and external agents listen to their views. 1995. INTERACTIVE PARTICIPATION People participate in joint analysis. PARTICIPATION FOR MATERIAL INCENTIVES People participate by providing resources. Earthscan Publications Ldt. 6. Jules N. 3. and may modify these in the light of people’s responses. These groups take control over local decisions and so people have a stake in maintaining structures or practices.. which leads to action plans and the formation of new local institutions or the strengthening of existing ones. Excerpt from ”Regenerating Agriculture . for example labour. yet people have no stake in prolonging activities when the incentives end. Pretty. in return for food. Such a consultative process does not concede any share in decision making and professionals are under no obligation to take on board people’s views. Such self-initiated mobilisation and collective action may or may not challenge existing inequitable distributions of wealth and power. May 18-19 1999 2 . It is very common to see this called participation. It tends to involve interdisciplinary methodologies that seek multiple perspectives and make use of systematic and structured learning processes. but retain control over how resources are used. 5. SELF-MOBILISATION People participate by taking initiatives independent of external institutions to change systems. They develop contacts with external institutions for resources and technical advice they need. 4. 2. Such involvement does not tend to be at early stages of project cycles or planning but rather after major decisions have been made.