A N ASSESSMENT OF THE SOCIO-ECONOMIC IMPACT OF THE WORKING FOR W ATER PROGRAMME ON THE HOUSEHOLDS OF

BENEFICIARIES

By the COMMUNITY AGENCY FOR SOCIAL ENQUIRY

The objectives of the study

objective of this study is to measure the socioeconomic impact on households being employed on WfW projects and to recommend a methodology which can form the basis of a regular assessment of household level impact of the programme. The study had a number of objectives including: To measure changes in income and expenditure of households with members employed on the WfW programme; To investigate the household asset base of programme and non-programme respondents; Explore the monetary and non-monetary differences for households where team members are employed; To investigate intra household dynamics, including: gender, age; number of dependents and whether household members earn an income. METHODOLOGY INTRODUCTION

A variety of research methodologies were used in the study, these included A survey Focus groups In-depth interviews THE SURVEY A national representative sample of Working for Water Projects was selected for the study. A multistage sampling strategy was used, involving stratification by province and respondent type. Table 1 shows the sample distribution by programme and non-programme respondents. The study aimed to compare the different programme groups in respect of objective measures of poverty and subjective measures, involving a set of standard Quality of Life Indicators. The following groups were selected for the sample: A programme group - comprising people who are employed by Working for Water or Working on Fire). The programme group comprised workers on contractor projects and workers on collective projects4. A non-programme or control group comprising respondents not employed on the programme.

In the absence of baseline data, a control group comprisingMETHODOLOGY INTRODUCTION A variety of research methodologies were used in the study, these included A survey Focus groups In-depth interviews THE SURVEY A national representative sample of Working for Water Projects was selected for the study. A multistage sampling strategy was used, involving stratification by province and respondent type. Table 1 shows the sample distribution by programme and non-programme respondents. The study aimed to compare the different programme groups in respect of objective measures of poverty and subjective measures, involving a set of standard Quality of Life Indicators. The following groups were selected for the sample: A programme group - comprising people who are employed by Working for Water or Working on Fire). The programme group comprised workers on contractor projects and workers on collective projects4.

A non-programme or control group comprising respondents not employed on the programme. In the absence of baseline data, a control group comprising people not employed on the programme was included in the study for comparative purposes. The same set of objective and subjective measures of quality of life were collected for this group via the survey. 4 Collective projects differ from contract projects. Collective projects can entail a system of profitsharing by workers on the projects and workers appointing a teamleader to manage the project and decide jointly for example on issues relating to salaries for example. In contrast contractor projects are managed by contractors appointed by Water Affairs. Workers are paid by the Department of Water Affairs, through the contractor, on the basis of working contracts and there is no profit sharing. COMMUNITY AGENCY FOR SOCIAL ENQUIRY FINAL REPORT 2007 -WORKING FOR WATER IMPACT EVALUATION 16 Group

Sample Size Working for Water Contract 650 Working on Fire 100 Working for Water Collective 100 Non-programme Group 155 Total Sample 1005 Table 1: Sample Groups SAMPLE SELECTION In order to achieve the combined sample size of 1005 interviews, the following process was undertaken: A random selection of six Working on Fire Projects and the distribution of the sample size proportionally to the number of people employed in those projects; For the collective Working for Water projects, there were two projects (Lower Kwena and Hex River) that were identified in the dataset. The allotted 100 interviews were split evenly between the two projects; With respect to the contract projects within the WfW programme, 38 projects were randomly selected. The sample size of 650 interviews was split proportionally amongst these projects;

The non-programme respondents were selected by matching the same demographic characteristics as specific programme respondents5. This was a process undertaken in-field. Age6, level of education and gender were taken into account in the selection of nonprogramme respondents. Respondents in this group were unemployed although they may have engaged in casual, informal or piecework. Housing type was not taken into consideration although respondents had to be living close to (in the same residential area) as respondents employed by the programme. 5 For example, in Project X, if a Control respondent (non programme respondent)
How satisfied are you with the leadership/management of the programme that you are working on

Evaluation of DFID Support to Poverty Reduction DRAFT Spin-off study: Impact Assessment for Poverty Reduction

Alicia Herbert
with Andrew Shepherd? [

6: Impact Assessment Tools and Methods at Project Level

Project impact assessments, generally, and for poverty reduction specifically, have over the years, moved from a single method to multi-method approach. Indeed, the introduction of participatory approaches to impact assessment in the last few years have expanded the methodological menu for data collection and knowledge creation (Hulme, 1997) (see table 6.1). While sample surveys are still common, rather than being used on their own, they are now often combined with participatory and other qualitative approaches. It is also the case, particularly among NGO implemented projects, that assessments are increasingly carried out using qualitative methods (rapid appraisal, participant observation, PLA) only. Each of the key methods has its strengths and weaknesses (see table 6.2) and this has led to a growing consensus among impact assessors and analysts that the central methodological question is no longer what is the optimal method for the study? but rather what mix of methods is most appropriate for the study and how should they

be combined? By mixing the different methods, the studies now benefit from the advantages of sample surveys and statistical methods ( quantification, representativeness and attribution) and the advantages of the qualitative and participatory approaches (ability to uncover approaches, capture the diversity of opinions and perceptions, unexpected impacts etc.). The choice of method(s), the extent to which they are mixed and the scale of their application will depend on a number of factors, namely the nature of the project, the type of information which is needed (or given priority), the context of the study and the availability of resources (time, money, human). For example, for an IAs which is well resourced and required to provide independent corroboration of a small scale programme and strengthen aspects of its implementation, a comprehensive mix of a small scale survey and qualitative approaches may be more appropriate. Boxes 6a and b give case examples. There is a series of basic questions which the assessor/analyst should ask in deciding which method(s) to choose, the answers to which should be matched with the information given in tables 6.2 and 6.3. These questions include: What are the objectives of the IA? - Proving/Improving?? How complex is the project, what type is it (blue print or process), what is already known about it? What information is needed? When is the information needed? How is the information to be used and by whom? What level of reliability is required? What resources are available (time, money and human)? Who is the audience of the IA study?
35

Table 6.1 Common Impact Assessment Methods

Method Key Features

Sample Surveys Collect quantitative data through questionnaires. Usually a random sample and a matched control group are used to measure predetermined indicators before and after the intervention Rapid Appraisal A range of tools and techniques developed originally as rapid rural appraisal (RRA). Involves the use of focus groups, semi-structured interviews with key informants, case studies, participant observation and secondary sources Participant Observation Extended residence in a programme/project community by field researchers using qualitative techniques and mini-scale sample surveys

Case Studies Detailed studies of a specific unit (a group, locality, organisation) involving open-ended questioning and the preparation of histories. Participatory Learning and Action The preparation by beneficiaries of a programme of timelines, impact flow charts, village and resource maps, well being and wealth ranking, seasonal diagrams, problem ranking and institutional assessments through group processes assisted by a facilitator. Specialised methods Eg. PhotogTable 6.2 : Strengths and weaknesses of key methods of IAs for poverty reduction
Method Criteria Surveys Rapid Appraisal Participant Observation Case Studies Participatory Learning and Action 1. Coverage (scale of applicability) High Medium Low Low Medium 2. Representativeness High Medium Low Low Medium 3. Ease of data standardisation, aggregation and synthesis High Medium Medium to Low Low Medium to Low 4. Ability to isolate and measure nonintervention causes of change High Low Low Low Low 5. Ability to cope with the problem of attribution High Medium Medium Medium Medium 6. Ability to capture qualitative information about poverty reduction Low High High High High 7. Ability to capture causal processes of poverty and vulnerability Low High High Medium High 8. Ability to capture diversity of perceptions about poverty Low High High Medium High 9. Ability to elicit views of women, minorities and other disadvantaged groups about poverty Low Medium?? High High (if targeted) Medium?? 10. Ability to capture unexpected negative impacts on the poor Low High Very High High High 11. Ability to identify and articulate felt needs Low High High Medium to Low High 12. Degree of participation of the poor encouraged by the method Low High Medium Medium Very High

13. Potential to contribute to building capacity of stakeholders with respect to poverty analysis Low High Low Medium to Low Very High 14. Probability of enhancing downwards accountability to poor groups and communities Low High Medium Medium High 15. Ability to capture the multidimensionality of poverty Low Medium High Medium Very High 16. Ability to capture poverty impact at different levels eg individual, household, community Low Medium High Low High 17. Human resource requirements Specialist supervision, large numbers of less qualified field workers High skilled practitioners who are able to analyse and write up results Medium skilled practitioners, with good supervision, who are prepared to commit for a lengthy period Medium skilled practitioners with good supervision High skilled practitioners 18. Cost range Very high to Medium High to Medium Medium to Low Medium to Low High to Medium 19. Timescale Very high to Medium

Medium to Low High High to Medium Medium to Low4 Surveys Rapid Appraisal Participant Observation Case Studies Participatory Learning and Action Adapted and extended from Montgomery 1996 and Hulme, 1997 4 It is important to note that participatory methods could consume a lot of poor peoples time. 38 Box 6a: The classic mix The Andra Pradesh Primary Education Project (APPEP)