Oxfam GB

Impact Measurement for NGOs: Experiences from India and Sri Lanka Author(s): Linda Kelly, Patrick Kilby, Nalini Kasynathan Reviewed work(s): Source: Development in Practice, Vol. 14, No. 5 (Aug., 2004), pp. 696-702 Published by: Taylor & Francis, Ltd. on behalf of Oxfam GB Stable URL: http://www.jstor.org/stable/4029898 . Accessed: 09/01/2012 18:16
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Taylor & Francis, Ltd. and Oxfam GB are collaborating with JSTOR to digitize, preserve and extend access to Development in Practice.

http://www.jstor.org

(Narayan 1999) It has been difficult for NGOs to demonstratethe worth and relevance of their work in a manneracceptableto scepticaloutsiders. limited in their outreach and corrupt.a number effects on the lives of poor andmarginalised of studies of NGO work point to a perceived lack of 'evidence' from which to establish measurableimpact. and impact Defining measuring first what changes had been experiencedby people. is a summaryof the researchundertakenand some of the conclusions about Oxfam CAA's impact that were reached throughthis project.ODI (1996).. The 696 ISSN 0961-4524 print/ISSN1364-9213 online 050696-14 C)2004 OxfamGB DOI: 10. The project ran over an 18-monthperiod.one areathathas gained considerablecurrencyhas been the impactdeliveredby NGOs. Patrick Kilby. and then two specific activities in measuringimpact in different countries where Oxfam CAA worked-India and Sri Lanka.self-serving. Kenall and Knapp 1999). Thereis also a view thatthe NGO sectoras a whole tendsto exaggeratethe impactof its workto supportthe case for increaseddonorsupport(Roche 1999. how the work of Oxfam CAA had or had not contributedto those changes. was not only in response to the wider critiquesabout the failure to establish the impact of developmentworkbut was also a recognitionby OxfamCAA itself thatits own externaland more clearly what the impact of its work internalaudiences were interestedin understanding was and how that impact may be measured.. Development Practice. But perhapsthe most relevantchallengeto NGOs comes frompoor people themselves: . andthe AusAID NGO EffectivenessReview (1996). among others.... the Danish NGO Impact Study (Oakley 1999). The final stage brought the findings of this research together with the agency's needs and This report issues.and Nalini Kasynathan Introduction In modem developmentdiscourse. second. The impact project. It startedwith several debates and discussions about how the agency might define and measure impact.1080/0961452042000239841 Carfax Publishing . mainlybecause theirwork is understoodto have directandobvious people. Despite these high expectations. NGOs touch relativelyfew lives . Given the scale of poverty. to look at how it might develop an ongoing model of impact measurement. Furthermore.Most of the studies cited above tend to reportthatthe evidence and frameworksfor demonstrating impactare lacking in the examples of NGO work. poor people give NGOs mixed ratings. The impact project sought to understand and. nor agreed methods for measuringit. . although to a much lesser extent than the State . developmentwork ratherthan suggest that their there are neither agreed definitions of what impact is.Volume Number August2004 in FrPubishing CTaylof& Impactmeasurement for NGOs: experiences from Indiaand Sri Lanka Linda Kelly. 14. some NGOs are largely irrelevant.These studiesinclude. in work has no impact as such. which ran into 2001.5. It was within this context that Oxfam CommunityAid Abroad(CAA) decided to undertake an 'impact project' in 2000 aimed at developing a frameworkand process for the ongoing measurementof the impact of the organisation'swork. Riddell 1999.

At the broaderor macro level. It value whattheyhave been doing andhave valid reasonsfor consideringan intervention is difficultto assumethatwe startwith a blankslate andthatall thatimpactassessmentwould be aboutis looking afreshat whatpeople have experiencedandhow theirlives have changed. which had as their goal the Developmentin Practice.It was clear thatinvestigatingonly the changes thathad occurredfor people. one. The second is the 'context in' approach. in the absence of an had of appreciation the changesthe respectiveprogrammes been tryingto achieve. openedup A furtherstep was then to test the relevanceof specific interventions againstthese changes.a 'projectout' approach whether thatoccurredin domesticor local powerrelationsas a consequenceof the intervention. The overall aim was to furtherthe organisationalunderstanding whathad been achieved.education.It looks at the changes that are happeningin people's lives.It startswith the aim of the projector programmeand then finds ways of measuringthe outcomes of that aim or objective from a variety of perspectives.The key questions that were being investigated were 'had peoples' lives improved?' and 'had we contributed to this improvement?' Moving from definition to measurementcreated furtherdifficulties.The first is the 'projectout' approach. This needed to be done in a fresh way that the significanceof such changefromthe perspectiveof the beneficiariesthemselves. Oxfam CAA adopteda blend of both approaches its study of impact.for Impact measurement NGOs organisationrequired an understandingof impact that allowed it to focus on experiences interventionand make linkages to the beyond the more immediateoutcomes of any particular broadercontexts in which developmentactivities were occurring. impactcould be appliedto programmes in At a practicallevel. which presents a significant step forward in addressing the relevance of development interventions. Volume14. Number5. the main one being whose perspective should be taken into account when measuringimpact.the 'context in' approachis resourceintensive and ultimatelyvalue laden and perspectivedependent.A 'context in' approachwould look at the causes of disempowermentthat an individual or community is facing (the context) and relate the interventionto thatcontext. is that the starting point for measurementis the change in the conditions of a particularcontext. and also to assess how to measuresuch changemore effectively in the future.one in Indiaand one in to Sri Lanka. Withinthis framework. Oxfam CAA was also looking to understand change from both the perspectiveof the people concerned.each designed to try differentapproaches defining andmeasuringimpactfor poor of and marginalisedpeople. would be too artificial.and from other perspectives.or health.Part of the value of the impact enquiry is to assess what organisationswere trying to achieve in comparison to the changes experienced by people. and then assesses the usefulness or effect of any interventionin relation to these changes. Roche (1999) has to identifiedtwo broadapproaches deal with the issue of perspectivein impactassessment. what is significantaboutthese changes. For example. August 2004 697 .to considerthe implicationsof this learningfor ongoing programming. such as broaderlocal and regional changes that had more indirectandlonger-term impactson the people. these two approachesto measuring such as nationwideadvocacycampaigns. and understandwhy the two might be different.in can These two approaches be appliedat a very basic level or at a broader anempowerment wouldlook at the intervention the changes and project. ratherthan the interventionitself. it be microfinance.Oxfam CAA undertooktwo researchstudies. The Indian study in A study was undertaken India that looked at programmesof interventionover long periods (up to ten years) with long-standing Oxfam CAA partners.But the difference in this approach.Partof the tension with the 'context in' approachis that staff and constituentsboth useful. Like the 'project out' approach.

the researcherwas not identified with a donor agency. The issue of the natureof the leadershipof groups appearedto be very important(an area often poorly covered in NGO practice). For the purposes of the study. these findings can give some insight into how Oxfam CAA develops and supportsit partners.The results showed thatthreefactorswere statisticallysignificantin determiningempowermentoutcomes: 'downward'NGO accountabilityto local groups. The approachthen set out to test the degree to which the work of the partnerNGOs had facilitated this change.how long the group had been meeting. and recognises that seeking short-term outcomes may not lead to empowerment. and Nalini Kasynathan empowermentof women. Care was taken to ask open-endedquestions that did not suggest certain answers.Again. and formal accountability mechanisms the NGO had established with respect to its partnerorganisations.e. A process for measuringempowerment was adoptedbased upon the experiencesof poor and marginalised women and how they answereda series of open-endedquestionsrelatingto (any) changes in their lives (Hines 1993). village social capital. It also has implications for how Oxfam CAA develops and fosters partnerships. The resultsof the answersto the questionson the change that women had experiencedwere ranked.The broadrankingsthat emergedfrom the women themselves rangedfrom being able to go out of the house and engage with people in authoritysuch as bank managers and the police. Related to these specific findings was the broad observationthat even the most successful community development programmes should be aware of the context in which they are 698 Developmentin Practice. Volume14. Other questions were asked about village life and women's participationin it to provide some idea of the broadercontext and the disempoweringinfluences women generally faced.Linda Kelly. A total of 15 NGOs were studied. and whether the change was in fact empowering. semi-formal. education. The finding that formal accountabilityof the NGO to its constituency is related to strong outcomeshas implicationsfor how Oxfam CAA field partners empowerment engage with their constituency in terms of the level of control they give their constituency in determiningthe direction of their work. throughto engaging with the local political processes.It points to a much greater focus on local control at all levels. correspondingto an increased range of choices and action in the lives of women. of which half were Oxfam CAA partners. land. and the leadershipof the group.These approachesenabled the women to make an assessment of the perceived changes that had occurredin their lives and gave them the opportunityto attributethese changes to the work of the NGO and explain how that occurred. Patrick Kilby. Similarly. and the range of informal. Hindess 1996). The empowermentchanges were then statistically tested in relation to several factors such as caste. but ratherwith a researchinstitution. August 2004 .. The second point is that empowermentof the most disadvantagedtakes time and requires to a long-termfocus.and how it manages its development programmes. and on sensitive issues questionswere asked in the 'thirdperson' so thatrespondentsdid not feel they were being personally 'put on the spot'. It also has implications for developing and supportingpartnersfor the long term. Those groups that had a rotated or collaborative leadershipapproachshowed strongerempowermentresults than those groups who depended on a single leader or even on the NGO to provide leadership. This finding has implicationsfor the 'project'approach developmentand tends to favour more strategic longer-term interventions. i. empowermentwas defined as an expansion of choice and a person's enhancedcapacity or opportunityto act on those choices (Kabeer 1999. Number5.From each of these NGOs a sample of self-help groups (the constituentgroup of around20 women that NGOs generallyfoster and work with) was chosen and a total of 77 women's groups were interviewed. etc.

and subcommitteesof stakeholders.As in the Indian study. each of the partnerorganisations. a group of waste-pickerwomen who had enormous success in empowermentthat led to significant changes to their working and living conditions were having their livelihoods threatenedby a proposedprivatisationof waste management in their city. living in the areas subjectto civil war between the governmentand Tamil separatist groups. The methodology developed for the study had two parts that roughly to correspond the notions of 'projectout' and 'context in' approachesto impactassessment. the relevance of the programme in terms of the overall impact experiencedby the poorest and most marginalisedpeople was not as great as expected. Finally. that focused upon the long-termchanges that the programme was trying to achieve. The study suggested that the CAA programmein Sri Lankahad been highly relevant to some of Development in Practice. inclusion of the most marginalised. the centralcommittee.These included the effects of the war and issues related to incompatibility and competition among donors. A broader advocacy campaign on waste managementmay be requiredto preserve the rights of these women. This change in urban waste management policy had the potential of underminingthe strong empowermentresults that had been achieved. The dimensions that were examined were the levels of: * * * * . . For the changes sought by the programme. Data were sought from various sources and involved a large number of meetings with includingcommunitygroups. empowerment/capacity genderjustice. institutionalstrengthening.OtherNGOs. This may require some integration with advocacy to ensure that there is a supportivelocal economic. and social environmentat the broaderlevel to ensure the sustainabilityof empowermentoutcomes. While the findings were generallypositive and the changes experiencedwere relatedto the work of Oxfam CAA. social awareness.so that change could be understoodat several differentlevels beyond that of the communitiesthemselves. people were asked 'What is going on in your lives? What have been significant changes in your lives? What are the significant problems?' Second. Number 5. The Sri Lankanstudy The Oxfam CAA programme in Sri Lanka works with poor and marginalised people. governmentofficials. externalexperts. especially women. August 2004 699 . each 'change dimension' was consideredfor the seven partnerNGOs studied. and either an assessment of change over time or an overall evaluation of progress was made. building. and the like were also interviewed. provision of services and benefits.the 'project out' assessment. a study on the national and regional changes that influenced developmentin Sri Lanka was undertaken. political. For example. developed externallyby Oxfam CAA staff. Volume 14. The issue-based assessment also identified other key factors that were outside the direct questioning but clearly related to the possible impact of the programme. the study sought information against a set of impact indicators or 'change dimensions'.for Impact measurement NGOs situated. * communityparticipation.staff.

The direction of the programme was improvedso that the objectives and indicatorsmore closely matchedthe areas of change requiredfor real empowerment. processes to bring the relationshipinto more of a partnership transparent promoting 'downward' accountabilitymechanisms. Third. with Oxfam CAA establishing more participatory and arrangement-i. People were more able to take advantageof situationsof change (for example. to responding to the local expression of these problems and were unable to address the problemsof war. An example of this limited level of change came from looking at the degree to which women were empoweredby the programme. national.and Nalini Kasynathan poor problemsfaced by people-human rightsviolations. learning Organisational The two case studies identifiedissues thathave relevance for Oxfam CAA as an organisation. and awareness raising were all tested and proven. New activities and approaches were introduced by programme staff and communities. the findings indicated the need for strongerlinkages between development programmes at the local level and advocacy efforts at a broader national level. many of the important decisions were still being made by men and. project. advocacy and policy work should be integrated with field programmesand capacity building. First is that a process of 'downward' accountability to partner organisations and their constituenciesinvolves mutual learning and the ability to listen to them. in some instances. The staff knew what constituted women's empowerment and what the outcomes were. Action for change occurs at local. the findings of the study clearly indicatedthat Oxfam CAA had a long way to go. and they felt that the programmeswere moving in the right direction. and lack of civil society structures. as in the case of Oxfam CAA's work in India detailed above.and the failure of governance. caste. in field programmessuch as Oxfam CAA's programmesin India to measureshort-termgains and Sri Lanka.regional. however. the particular The interventionstended. action needs to take place at more than one 'level' of interventionfor any impact to be sustainable.e. Volume14.Second was a revision of the use of training as a tool for communitydevelopment. While women reported a decrease in domestic violence and an increase in household income. August 2004 .Linda Kelly. genderand ethnic discrimination. Second.The content and approachesto the trainingoffered by Oxfam CAA were revised with a greaterfocus on group and issues-based training. In this respect it is mechanismsbuilt into the design of the to important have formalor semi-formalaccountability including how local people can assert their rights vis-'a-visthe NGO. regular training. What became evident is that the empowerment of women in a complex war-zone situation requiresmore sophisticatedinterventionsthan those used in other settings. That is. women were willingly handing over decision-making power to the men. Finally.The strategiesadopted to improve effective participationof women such as policies of positive discrimination. This finding led to changes in programming.it is clear that any impacttakes time and the variouscomponentsof change that 700 Developmentin Practice. However. Number5.humanrightsviolations. to be limited services. they know more about their rights and how to organise). genderdiscrimination. following discussion and analysis of the research findings. and be monitoredto account for the changes While it is possible andensurethese changes arerelatedto the overall goals of the programme.it appearsthat interventionsare dynamic and should change with a changing context and over time. Patrick Kilby. fundamental economic disadvantage. The relationship between Oxfam CAA and the community-based organisations they supported was also revised. and in some cases intemationallevels. constitutionalchanges. but the changes were not happeningat a level that ensured sustained difference.

Conclusions The final lesson from the impact project was that the process of trying to measure impact taught Oxfam CAA more about the meaning and process of impact than many of the earlier discussions and debates had. However. it would be a mistake to allow this to stop an organisationfrom attempting to learnmore aboutthe contribution is makingto sustainedand positive changes in people's it lives. M. (1999) 'Can anyone hear us? Voices from 47 countries'. New York. Riddell. London: Earthscan. R. Narayan. 2. (1999) ImpactAssessmentfor DevelopmentAgencies. achievements:reflections on measures of women's empowerment'. Briefing Paper No. C. Kai Schafft. D.) InternationalPerspectives on Voluntary London:Earthscan. Kenall. (1996) Discourses of Power: From Hobbes to Foucault.' References AusAID (1996) A Review of the Effectiveness of AusAID Support to NGO Programs. (1999) 'Resources. Number5. London: ODI. (1999) Danish NGO Impact Study: A Review of Danish NGO Activities in Developing Countries-Synthesis Report. Managingthis process requiresregularattentionto measurementof change and difference.and Sarah Koch-Schulte. and M. Roche. B.ratherthan just test the attainmentof objectives. Narayanwith Raj Patel. C. in D. (1999) 'Evaluating NGO development interventions'. P. Canberra: Hindess. Developmentin Practice.) International Perspectives on VoluntaryAction: Reshaping the Third Sector. NY: OUP (for the WorldBank).we might end up doing it. In many ways the struggle to develop and test an appropriate methodology was as important as the specific conclusions. Hines. August 2004 701 . Poor people know what is helpful to them and what is not. ODI (1996) 'The Impact of NGO Development Projects'.Impact measurement NGOs for need to occur will take place in differentways and at differenttimes. American Journal of Community Psychology 21(3):729-746. (1993) 'Linkingqualitativeand quantitativemethods in cross-culturalresearch: techniques from cognitive science'. Volume14. Knapp (1999) 'Evaluationand the voluntarysector: emerging issues'. N. Oxford and Copenhagen: INTRAC/BECH Distribution. the attemptto learn about the impact of interventions. Oakley. One problemwith these types of field studies is the difficulty of drawingfirm conclusions about attribution. Finally. was a powerful challenge to Oxfam CAA and its staff.Developmentand Change 30(3):435-464. in D. Anne Rademacher. Oxford: Oxfam GB. Kabeer. A. Lewis (ed. D. J. Unless the poor participatein defining the impact being sought and their perspective is recognised as central to assessing impact. Lewis (ed. Oxford:Blackwell. Voices of the Poor: Can Anyone Hear us?.As one staff membernoted. in Action: Reshaping the ThirdSector. While it seems that there is no one approachto measuring or defining impact. NGOs (or indeed any aid-delivery mechanism) run the real risk of being considered 'largely irrelevant'. AusAID. 'Perhapsif we keep trying to measure what's really important.both studies indicatedthat people know what impact they want. agency.

Patrick Kilby lectures in programmemanagementand empowermentand rights-baseddevelopment at the Australian National University. Volume14. <patrick. The potential for benefits from collaborationbetween a researchorganisationand an operationalNGO seems large. August 2004 .and programmeassessment in Peru and Ethiopia.3114. Australian National University. a partnershipbetween CARE and the InternationalFood Policy Research Institute(IFPRI). Number5. organisationsare not common.Examiningtheir collaborativeefforts.including livelihood assessments in Tanzania.This can in turnstrengthenits show of impact and innovationto donors.kilby@anu. Park Orchards.technical assistance in Mozambique. <Nalinik@caa.therebyincreasingthe probabilitythat others will actively make use of their findings. Bridginggaps: collaborationbetween research and operational organisations James L. Garrett The potential of collaboration Informationis essential to improving organisationaleffectiveness.By working with NGOs.The two have worked in a numberof countries.org.au>. Contact details: Asia Pacific School of Economics and Government. The NGO can tap into the latest knowledge and learnhow to improveits own survey and analytical methods. Australia.au>.and gender and aid management. this paper provides a concrete illustrationof how to build bridges and profit from synergies between two such organisations while highlightingpotentialbumps to expect along the way and what to do about them. explicit collaborations between research and operational perceptionsthrowup barriersto workingtogether. ACT 0200. Institutional cultureand intellectual The roots of the problemmay be primarilydifferencesin organisational 702 Developmentin Practice. 156 George St.edu. CARE and IFPRI have collaboratedon increasingknowledge abouturbanlivelihoods that will be of use to programmedevelopment. Contactdetails: Oxfam CommunityAid Abroad. <paulnichols@onaustralia. This Practical Note builds on the insights of Laura Roper's 2002 article on 'Achieving researchcollaborations'by reviewing one example of such successful academic-practitioner collaboration. researcherscan get a bettersense of criticalpolicy and programmequestions and shape their work to demand. The gap Despite the apparent benefits.Contactdetails: 37 North Valley Road. Since 1997. Australia.Nalini Kasynathanhas taught at University Peredeniyain Sri Lankaand is currentlyresponsiblefor managingOxfam CAA programmeswithin South and East Asia. Crawford Building.Vic 3065. Australia. Garrett The authors Linda Kelly is an independent practitionerwho specialises in community development.au>. Fitzroy. monitoringand evaluation.com.James L.issue-based research in Bangladesh.