United Nations Inter-agency Resource Pack on Research, Monitoring and Evaluation in Communication for Develo ment

!e"ruary #$%%

United Nations Inter-agency Pro&ect to'ard a Resource Pack on( Research, Monitoring and Evaluation in Communication for Develo ment

Part I

Researching, Monitoring and Evaluating Communication for Develo ment( )rends, Challenges and * roaches

Re ort on a literature revie' and Consultations 'ith E+ ert Reference ,rou and UN !ocal Points on C-D Prepared for the United Nations Inter-agency Group on Communication for Development by June Lennie and Jo acchi

!" #ebruary "$!!

"

Contents
%c&no'ledgements List of acronyms ()ecutive summary !* !* Introduction and methodology "* "* Principles for effective+ appropriate and sustainable research+ monitoring and evaluation of Communication for Development ,* .* ,* %pproaches to Communication for Development /* .* Ne' thin&ing and trends in 0+ 12( and impact assessment 3* /* Challenges+ issues and strategies 4* 3* (valuation capacity development in C.D -* 4* 5ey approaches+ methodologies and methods for 0+ 12( of C.D 6* -* Indicators of C.D impacts 6* Conclusion and 'ays for'ard 0eferences %ppendi) !7 List of ()pert Panel members+ C.D #ocal Points and other participants .ist of ta"les able !7 General and specific principles for effective+ appropriate and sustainable 0+ 12( of C.D able "7 %pproaches to Communication for Development able ,7 %pproaches to 0+ 12( of C.D able .7 %pproaches to assessing the impacts of C.D able /7 %pproaches+ methodologies and methods for 0+ 12( of C.D able 37 %pproaches to evaluation capacity development able 47 8trengths and limitations of &ey 12( methodologies able -7 8trengths and limitations of &ey impact evaluation planning approaches able 67 0atings for methods considered effective for assessing C.D vi i) ) )i )iii )i) -6, 6. iii iv v vi ! !. ". ,4 /3 4, !$$ !!, !"" !,,

impacts able !$7 8trengths and limitations of &ey evaluation methods 6/

iv

*"out the authors

his report 'as 'ritten by June Lennie and Jo acchi* Dr June Lennie is a 8enior 0esearch %ssociate in the Creative Industries #aculty+ 9ueensland University of echnology+ in :risbane+ %ustralia* Jo acchi is Professor and Deputy Dean of 0esearch and Innovation in the 8chool of 1edia and Communication+ 01I University+ 1elbourne+ %ustralia*

*ckno'ledgements
he research upon 'hich this report 'as based 'as conducted in close collaboration 'ith Dr eresa 8tuart and Paula Claycomb in UNIC(#;s Communication for Development Unit in Ne' <or&* Paavani 0eddy at UNDP;s =slo Governance Centre also provided some input* >e 'ould li&e to than& eresa and Paula very much for providing constant support and encouragement+ useful resources and information+ and valuable feedbac& and advice* #unding for the development of this report+ 'hich is part of a 0esource Pac& for 0esearch 1onitoring and (valuation of Communication for Development Programmes+ 'as provided by UNIC(#* 8ome additional funding 'as provided by the Creative Industries #aculty+ 9ueensland University of echnology* his report 'as prepared 'ith the assistance of Dr 5irsty 1artin+ 'ho undertoo& an e)tensive literature search+ developed and maintained a large electronic bibliography of publications and resources and assisted in compiling the reference list in this report and some of the feedbac& from those 'e consulted* % !/ member ()pert Panel and !! C.D #ocal Points in seven UN agencies+ funds or other bodies contributed relevant resources and suggestions and provided valuable comments on an earlier draft of this report* hey also provided e)tensive feedbac& on drafts of the Principles that are set out in 8ection " of this report* heir names are listed in %ppendi) !* ?arious UN staff provided feedbac& on a draft of a detailed online @uestionnaire 'hich 'as used in the research process* %long 'ith si) members of the ()pert Panel and five members of the C.D #ocal Point group+ !, other specialists from UNIC(# and UNI#(1 provided feedbac& on a draft of this report* % number of them also assisted in developing strategies and plans to ta&e this 'or& further at a series of meetings in Ne' <or& in December "$!$* >e are most grateful to everyone 'ho contributed to this report in some 'ay and than& you all very much* In particular+ 'e 'ould li&e to than& %ilish :yrne from the Communication for 8ocial Change Consortium for her e)tensive feedbac& and useful edits to an earlier draft of this report* June Lennie and Jo acchi !" #ebruary "$!!

.ist of acronyms
*C-/C 0CC C-D C!/C E*N E*R ECD !*1 ICD IC) I.1 2*0P MD,s M3E M/C P, M3E PRC* PR/Ps RR* R, M3E //M2 UNDP UNE/C1 UN!P* UNICE! 4CCD %ssessing Communication for 8ocial Change proAect :ehaviour Change Communication Communication for Development Communication for 8ocial Change (@ual %ccess Nepal (thnographic %ction 0esearch (valuation capacity development #ood and %griculture =rganisation Information and Communications for Development Information and Communication echnology International Labour =rganisation 5no'ledge+ %ttitudes+ :ehaviours and Practices surveys 1illennium Development Goals 1onitoring and (valuation 1ost 8ignificant Change techni@ue Participatory 1onitoring and (valuation Participatory 0ural Communication %ppraisal Poverty 0eduction 8trategy Papers 0apid 0ural %ppraisal 0esearch+ 1onitoring and (valuation Saathi Sanga Manka Khura BCChatting 'ith my best friendDE radio programme United Nations Development Programme United Nations (ducational+ 8cientific and Cultural =rganisation United Nations Population #und United Nations ChildrenDs #und >orld Congress on Communication for Development vi

Researching, Monitoring and Evaluating Communication for Develo ment( )rends, Challenges and * roaches
E+ecutive summary
his report highlights a number of important trends+ challenges and approaches associated 'ith researching+ monitoring and evaluating Communication for Development BC.DE 'ithin the UN conte)t* It is a &ey component of the 0esearch+ 1onitoring and (valuation B0+ 12(E 0esource Pac& for C.D Programmes* his 0esource Pac& is being developed as part of an ongoing series of strategies that aim to institutionalise C.D 'ithin the International Development %genda+ demonstrate the contributions and impacts of C.D+ and thereby strengthen C.DDs institutional position 'ithin the UN* o prepare this report+ 'e undertoo& a maAor literature revie' and consultations 'ith !! C.D #ocal Points or 12( specialists from seven UN agencies and a !/ member ()pert Panel+ 'ho provided e)tensive inputs into the proAect+ including suggested revisions to a draft of this report 'hich 'as discussed at a series of meetings at UNIC(# head@uarters in Ne' <or& in December "$!$ Bsee Pamer et al*+ "$!!E*

Princi les for effective, a

ro riate and sustaina"le R, M3E of C-D

:ased on the &ey themes and issues in the literature revie' and our consultations+ this report details a number of principles for effective+ appropriate and sustainable 0+ 12( of C.D+ 'hich are summarised in able !* hese principles provide a foundation for the approach that is advocated in this report and in the 0esource Pac&* )a"le %( ,eneral and s ecific rinci les for effective, a M3E of C-D ,eneral rinci les
Is consistent 'ith the underlying values and principles of C.D and appropriate for different types of C.D initiatives

ro riate and sustaina"le R,

/ ecific rinci les

Uses a participatory approach that respects+ legitimises+ conte)tualises and dra's on the &no'ledge and e)perience of local participants* 1ethodologies and methods selected are culturally appropriate+ complementary+ and the most appropriate for different issues and purposes+ different types of C.D initiatives+ and the aims of the evaluation* #acilitates active participation of sta&eholders in all stages of the evaluation and o'nership of the initiative and its evaluation* Is as inclusive as possible of a diversity of groupsF addresses issues of gender+ class+ race and une@ual po'er and voice* Uses openness+ freedom and fle)ibility in developing evaluation frame'or&s and plans* his process begins by agreeing on the initiativeDs obAectives and outcomes and clarifying the evaluationDs purpose and sta&eholdersD e)pectations* a&es a long-term vie' of the process and the benefits of a participatory+ mi)ed methods approach and the use of longitudinal studies to assess impacts and outcomes* he

Is meaningfully participatory and inclusive

Is integrated into the programme cycle from the conception+ design+ and planning stages

Is provided 'ith sufficient funding+ time and resources

vii

evaluation is proportionate to the scale of the programme and timeframes to achieve outcomes are realistic* %ims to lead to sustainable outcomes+ including strengthened organisational systems and evaluation capacities and more effective policies+ strategies and programmes that address development goals* Goes beyond a focus on individual behaviour to consider social norms+ policies+ gender and po'er relations+ culture and the specific and general development conte)t %ttempts to understand ho' and 'hy social change happens+ using a rigorous participatory+ mi)ed methods approach 8taffs at all levels are involved in long-term capacity development 'hich focuses on organisations as a 'hole and the development of learning organisations* (valuation capacity development aims to increase understanding of the fundamentals of 0+ 12(+ and provides practical guidance and simple+ user-friendly information for managers+ field staff and community members* a&es the 'ider social+ economic+ political+ cultural and communication conte)t+ issues and barriers into account+ the comple)ity of human systems+ and the interrelationships bet'een different interventions* Data is disaggregated by gender+ age+ ethnicity and other relevant differences* Involves developing locally and e)ternally derived indicators and a dynamic+ moving theory of change* Considers the short-term+ intermediate and long-term impacts of initiatives* %dAusts baseline information to recognise changes in the conte)t* riangulates findings to increase rigour and ensure that a diversity of voices and vie's are included* (ngages in continuous monitoring of the communications environment* Identifies process outcomes to learn lessons* Learns from negative findings+ 'ea&nesses and CfailureD and loo&s for unplanned and une)pected results* %ppropriate+ effective and open communication and feedbac& systems are used to share findings+ learnings+ outcomes and e)periences* Positive+ negative+ intended and une)pected findings are shared openly and honestly 'ith participants+ donors and funders and the larger development community*

Involves continuous critical reflection and learning* Includes an action component and a process of ongoing improvement through feedbac& loops as a programme is implemented* he evaluation is based on a high level of independence+ integrity and honesty

Challenges, issues and tensions

=ur research identified a number of significant and fundamental challenges+ tensions and issues that need to be addressed in order to better meet the UNDs development goals Bparticularly the 1illennium Development GoalsE and the aims and obAectives of C.D and related development activities* hese have been summarised belo' in a series of tables that aim to highlight the &ey characteristics of+ and the critical differences bet'een dominant and alternative approaches to C.D+ 0+ 12( for C.D and evaluation capacity development+ in the interest of ensuring that more informed and effective decisions are made* %lthough the dichotomies presented in these tables may appear suggest a strong polarisation bet'een the t'o approaches+ 'e emphasise in the report ho' a mi)ed methods approach that dra's on contrasting methodologies and methods can richly complement each other to give a fuller and more realistic picture of change* >hile our research demonstrates the benefits of alternative approaches+ this report highlights the strengths and limitations of both dominant and alternative approaches and methods* >e also recognise the need to be realistic about the process of changing current policies and practices* (ffective use of alternative participatory approaches is clearly more difficult 'ithin hierarchical and infle)ible organisational structures and cultures* Go'ever+ 'e viii

believe that there is a need to criti@ue ineffective practices such as donors setting unrealistic timeframes for outcomes to occur and the use of processes that tend to discourage learning from negative findings and CfailuresD in order to improve initiatives* It is not the approaches to 0+12( themselves+ but how they are applied and that is critical here* #or e)ample+ participatory approaches can be implemented in very top-do'n 'ays* hey can also present many challenges for time and resource poor UN agencies and their partners 'hose staff may need to strengthen their evaluation capacities in order to produce more useful and rigorous research* he findings and issues identified by this research are summarised under the follo'ing headings7 %pproaches to communication for development %pproaches to research+ monitoring and evaluation of C.D %ssessing the impacts of C.D Indicators for C.D impact 5ey approaches+ methodologies and methods for 0+ 12( of C.D (valuation capacity development

roaches to Communication for Develo ment

here has been a shift Bin rhetoric at leastE from vertical one-'ay+ top-do'n models of communication for development to horiHontal models that aim to facilitate participation+ inclusion and empo'erment* Go'ever+ many approaches refer to both perspectives in contradictory 'ays+ resulting in confusion and inappropriate compromises that limit the effectiveness of C.D initiatives* #or e)ample+ communication is often marginalised+ 'hile at the same time+ it is heralded as a maAor pillar for development and change* In practice+ communication+ as understood by decision-ma&ers+ is often reduced to vertical information delivery or public relations+ rather than part of a process of meaningful engagement in development processes* Diverse approaches to C.D are ta&en across UN agencies but the follo'ing four main CstrandsD have recently been identified7 :ehaviour Change Communication B:CCE Communication for 8ocial Change BC#8CE Communication for advocacy 8trengthening an enabling media and communication environment B1cCall et al*+ "$!$E

Long-term research highlights a recurring problem 'ith decision ma&ers in development organisations not appreciating 'hat C.D means+ or its important role in development* Decision ma&ers in the UN often do not understand that C.D includes t'o-'ay communication systems that enable dialogue+ Callo' communities to spea& out+ e)press their aspirations and concerns and participate in the decisions that relate to their developmentD BUN 0esolution /!I!4"+ !664E* Inclusion of people on the ground in all stages of development processes is seen as a fundamental principle by progressive proponents of C.D* Go'ever+ institutions 'hich i)

e)cluded communities might engage 'ith through communication are often structurally unsuited for listening+ and development generally positions the poor and marginalised as listeners rather than spea&ers* In addition+ participatory approaches are often considered costly and time consumingF they challenge entrenched po'er structures+ and are seen by some as incompatible 'ith dominant organisational cultures and dominant 0+ 12( and planning approaches* >e have therefore argued that a long-term perspective on the use of participatory approaches is needed+ given the many benefits that they bring* )a"le #( * Dominant a roaches roaches to Communication for Develo ment Partici atory a roaches
GoriHontal models based on meaningful participation+ engaging people in dialogue+ sharing &no'ledge and learning from each other+ in a multi-'ay process* Communication is seen as maAor pillar for development and change* 8ome UN agencies strongly support horiHontal communication for social change and participatory development approaches*

)ensions and issues

1any C.D approaches refer to both perspectives in contradictory 'ays* his results in confusion and inappropriate compromises* Institutions are often structurally unsuited for listening to the community* 1any C.D units are located in corporate communication and e)ternal relations departments* he 'ide range of C.D approaches and meanings suggest a need to focus on common C.D goals and to reduce confusion about the meaning of C.D* #ull and direct participation is incompatible 'ith dominant organisational cultures and practices* It challenges entrenched po'er structures and is difficult to achieve+ given issues of po'er and inclusion of a diversity of sta&eholders+ politics and perceptions of the greater time and resources re@uired*

?ertical+ top do'n models+ sending messages or disseminating information via one-'ay communication or public relations Communication is seen as marginal in the development process* Lac& of high level support and understanding+ in some UN agencies+ of C.D as dialogue and community participation in decisions that affect their lives*

CParticipationD in development is often only rhetoric+ not put into practice+ or implemented in top-do'n 'ays*

?ie' that people on the ground need to be included in all stages of the development processes* Participation and o'nership are seen as vital for sustainability*

roaches to research, monitoring and evaluation of C-D

% 'ider and more holistic perspective based on systems thin&ing and participatory approaches to 0+ 12( is increasingly seen as important to address comple) social problems. his has significant implications for the evaluation of C.D+ 'hich is seen as re@uiring more organic and fle)ible strategies and methodologies and methods that match the dynamics of the system in 'hich they are implemented* he impact of 'ider conte)tual+ structural+ institutional+ and organisational issues also needs to be ta&en into account more in the evaluation of C.D* Less dominant approaches to development and social change ma&e e)plicit connections among po'er+ 'hat is legitimiHed as C&no'ledgeD and the social hegemonies that constitute )

social norms* :urns B"$$47 ,3E suggests that if interventions do not attend to local social norms+ Cmany policy initiatives 'ill fail to 'in community support+ rendering them unsustainableD* his has maAor implications for C.D initiatives 'ithin programmes that aim to change harmful social and cultural practices and prevent the spread of maAor health problems such as GI?I%ID8* Compelling arguments have been made for a long time about the value of participatory and mi)ed methods approaches in the 0+ 12( of development programmes* =ur research suggests that the evaluation of C.D needs to be based on an appropriate combination of @ualitative and @uantitative techni@ues+ complementary approaches and triangulation+ in recognition that different approaches are suitable for different issues and purposes+ and different approaches to C.D* Go'ever+ there is a often lac& of appreciation+ funding and support for alternative+ innovative 0 12( approaches among management and mainstream 12( specialists in the UN* Commitment to participatory processes often remains rhetoric rather than translating into meaningful or appropriate practice* #unders tend to place greater value on narro'+ @uantitative measurement-oriented approaches and indicators that better fit their o'n management systems and tools+ but that ta&e insufficient account of the comple)ity of culture and the conte)t of particular C.D and development initiatives* he benefits and rigour of alternative 0+ 12( approaches need to be better demonstrated* )a"le 5( * Dominant a roaches roaches to R, M3E of C-D roaches )ensions and issues
here is a lac& of funding and support for alternative 0+ 12( approaches that are often more appropriate for C.D* 8ince policy ma&ers and managers often have a hard science bac&ground+ they tend to lac& an understanding and appreciation of the potential of alternative approaches and @uestion their rigour and validity*

*lternative a

Narro' focus on individual behavior change+ short-term changesF use of linear causeeffect models of social change that do not foster understanding of the comple)ity of culture and the conte)t of development programmes*

he comple)ity of social change is seen as re@uiring a participatory+ holistic approach+ incorporating insights from systems thin&ing and comple)ity theories and including a focus on gender+ po'er and 'ider social norms* his approach dra's attention to the underlying dynamics of social change* % pragmatic+ participatory+ mi)ed methods approach+ guided by appropriate principles and &ey evaluation @uestions+ is needed to move beyond unhelpful @ualitativeI@uantitative dichotomies* his 'ould increase the strength and rigour of evaluation and impact assessment findings*

Dominance of @uantitative+ measurement-oriented approaches that limit the ability to as& important @uestions about social and behavior change*

#unders typically have a preference for numerical data but do not appreciate the value of participatory tools for eliciting information that is often more credible and useful to communities themselves* % participatory+ mi)ed methods approach re@uires a 'ider range of s&ills and &no'ledge to use effectively* here are other particular challenges to rigorously using this approach in some developing countries*

*ssessing the im acts of C-D

)i

Demonstrating the impact of C.D is a crucial part of moving C.D up the development agenda and achieving institutionalisation of C.D* Go'ever+ this is often much more comple) and difficult than for other development initiatives* Challenges identified in our research included7 he challenge of attribution of impact in a comple) and rapidly changing 'orld* Donors often 'anting to see results in an unreasonably short time frame* >hile the best 'ay to assess lasting and sustainable change is to use longitudinal studies underta&en some time after proAects end+ donors are reluctant to fund such studies as there is Cno strongly established evidence base of past e)perience on 'hich to buildD B8outer+ "$$-7 !3.E* he high cost of impact assessment J there are problems 'ith inade@uate funding+ 'ea& capacity and inade@uate resources+ including time to underta&e impact assessment of C.D programmes* %pproaching 12( in a vertical rather than an integrated manner - monitoring+ evaluation and impact assessment need to be integrated into the overall proAect cycle+ including programme and proAect design* he comple)ity of change J social and behavioural change often needs to be assessed against a moving baseline+ 'hich is inconsistent 'ith dominant organisational practice*

Participatory+ fle)ible and holistic approaches to evaluation based on systems thin&ing+ comple)ity theory and action research are advocated since they enable us to consider the multiple paths to achieving impacts and the contributions of the intervention to'ards achieving impact+ in relation to the macro and micro conte)ts* 8uch approaches encourage ongoing critical reflection and learning* his ne' 'ay of thin&ing has led to shifts a'ay from the dominant focus on measuring and proving impacts+ to'ards understanding of the actual process of social change and continually improving initiatives and practices* his implies a greater focus on programme delivery+ innovation+ sustainable results+ progress towards social change+ the contribution of C.D to outcomes and impacts+ and the impacts of evaluation processes themselves* In addition+ this approach suggests that it is often more important to understand negative and une)pected impacts and 'hat has not been achieved+ as this leads to better learning and programme improvement* Go'ever+ the politics of aid means that implementing agencies are Coften tempted to claim credit for impacts because that is 'hat those they are accountable to 'ant to hearD B8outer+ "$$-7 !3"E and reporting focuses more on CsuccessesD than on CfailuresD* )a"le -( * Dominant a roaches roaches to assessing the im acts of C-D *lternative a roaches )ensions and issues
Demonstrating the impact of C.D is notably more comple) and difficult than for other types of development initiatives* Dominant accountability-based approaches

Dominance of instrumental+ accountability-based approaches that focus on proving impacts+ using linear cause-effect logic and

#le)ible+ holistic interdisciplinary approach based on ongoing learning+ improvement and understanding* a&es the

)ii

reporting results* %lternative approaches are not ade@uately resourced or supported and are often criti@ued for lac&ing CobAectivityD+ CrigourD and CvalidityD*

comple)ity of social change and the particular conte)t into account*

discourage o'nership of the evaluation process and learning from evaluation and a focus on une)pected and negative impacts and CfailureD that are often more significant for learning and improvement* 0esults are often biased to'ards positive outcomes and evaluations are not independent* Longitudinal studies are re@uired but they are costly and one of the most difficult challenges in impact assessment* Donors are reluctant to fund them* his means that there is a lac& of strong evidence on 'hich to build C.D research+ 'hich fuels s&epticism* Indicators are unable to capture comple) realities and relationships and the reasons behind social change* In some C.D evaluations it may be more useful to use alternatives to indicators*

Pressure to produce shortterm results 'ithin rigid and unrealistic timeframes* his results in a focus on more tangible+ short-term changes that are not good indicators of long-term social change*

Indicators are often set 'ithout input from &ey participants+ are @uantitative and unrealistic and do not fit C.D outcomes*

8een as more important to focus on progress towards long-term social change and the contribution made by C.D* his is a more realistic measure of effectiveness and provides practical recommendations for the implementation of policies and programmes* Indicators need to be selected and developed through dialogue 'ith &ey sta&eholders+ to be most useful and appropriate* 9ualitative C.D indicators are often most effective and appropriate*

Indicators of C-D im act

here are many challenges and issues associated 'ith developing and implementing indicators of C.D impact* =ur research suggests that indicators themselves are largely unable to capture comple) realities and relationships* hey can be useful 'ays of measuring some change+ but not of capturing the reasons behind social change* In C.D+ and in particular the Communication for 8ocial Change approach+ indicators should be developed through dialogue and negotiation bet'een &ey participants+ so that they are chosen based on local assessments of 'hat &ey participants need to &no' and 'hy+ and they are more realistic and useful* >hile @uantitative indicators are emphasised in mainstream 12( approaches+ in C.D @ualitative shifts are often most appropriate to capture* %lternatives to indicators 'hich are fle)ible and 'hich better encompass comple)ity+ such as stories of significant change and Cverifying assumptionsD+ are often more appropriate and effective*

2ey a

roaches, methodologies and methods for R, M3E of C-D

here is a need for openness+ freedom and fle)ibility in the selection and use of 0+ 12( approaches+ methodologies and methods to ensure that they are appropriate and fit the underlying aims and values of the C.D initiative* hey also need to ta&e into account various constraints such as time+ resources and organisational challenges* Participatory approaches to 12( have been advocated given their many benefits+ including strengthened capacity in 0+ 12(+ greater utilisation of findings and learnings+ and the empo'erment of participants* hey are also seen as Copen approaches that can be adapted locallyD*

)iii

Participatory 1onitoring and (valuation BP+ 12(E is recommended as an effective 'ay of actively engaging &ey sta&eholders in all stages of the evaluation of C.D and strengthening evaluation capacities and o'nership of the process* =ur consultations found that the follo'ing participatory+ @ualitative or mi)ed methods approaches and methodologies 'ere considered to be the most effective for assessing the impacts of C.D7 Case studies Participatory 0ural Communication %ppraisal 0apid 0ural %ppraisal =utcome 1apping 1ost 8ignificant Change techni@ue (thnographic %ction 0esearch

=ther participatory+ @ualitative or mi)ed methods methodologies that 'ere nominated by respondents as effective included Developmental (valuation+ rights-based approach methodologies+ contribution assessment and %ppreciative In@uiry* 9uantitative survey-based methodologies and cost benefit analysis 'ere also seen as effective for assessing the impacts of C.D* he follo'ing approaches and methods 'ere generally considered to be the most effective for planning impact evaluations of C.D 'ithin programmes7 Causal analysisIproblem analysis+ the heory of Change approach and the logical frame'or& approach* Commonly used @ualitative methods such as in-depth intervie's and focus group discussions and participatory tools such as communityIvillage mapping 'ere seen as particularly effective in assessing the impacts of C.D* Go'ever+ as the table belo' indicates+ all of these 12( approaches+ methodologies and methods have particular strengths and limitations or constraints* )a"le 6( * roaches, methodologies and methods for R, M3E of C-D Dominant a roaches *lternative a roaches )ensions and issues
Dominant @uantitative survey-based methodologies are limited in their ability to as& important @uestions about the social+ cultural and political conte)t 'ithin 'hich development problems are embedded* hey miss the level of detail re@uired to understand the nuances of impact* hey do not allo' for @ualitative analysis and change over time in a given conte)t+ and are more suited to short-term activities* he logframe has been 'idely criticised as infle)ible and unable to capture une)pected outcomes or changes* It represents the simplification of comple) social processes and avoids the importance of process* here is a gro'ing hey highlight openness+ freedom and fle)ibility in the selection and use of various 0+ 12( approaches+ methodologies and methods+ 'hich should fit 'ith the underlying aims and values of the C.D initiative* Participatory+ @ualitative and mi)ed methods approaches and methods are seen to have much value+ if they are effectively used* % heory of Change approach to evaluation enables a more detailed analysis of different sta&eholders+ communication flo's and processes* It ma&es e)plicit the values that underpin the perspectives of more and less po'erful here is a need to strengthen capacities in using participatory+ @ualitative and mi)ed methods approaches so as to increase the rigour of 12( for C.D* (ach of the approaches+ methodologies and methods that 'ere considered effective for evaluating C.D have strengths and limitations or constraints that need to be considered* %ll are best used in tandem 'ith complementary approaches* >hile adaptations of the logframe have been developed+ and some have called for more participatory use of the logframe+ @uestions have been raised about 'hether this tool can be reconciled 'ith goals of empo'erment and giving voice to the most marginalised*

)iv

a'areness that the logframe and similar tools s@ueeHe out data related to local culture and conte)t+ and do not provide a space for an analysis of informal interactions and e)ternal influences that can be important to successful interventions*

sta&eholders and enables targeted proAect design and 12(* It is good for e)pressing assumptions about causal changes and for deeper analysis of 'hat is 'or&ing or not+ for 'hom etc*

% number of practical+ political+ theoretical and systemic limitations have been identified to applying the heory of Change approach in practice* Particular conte)t and programme specifics 'ill help to determine 'hich approaches and methods are most appropriate*

Evaluation ca acity develo ment

=ur research identified a significant need to strengthen capacity in C.D and 0+ 12( at all levels* (valuation capacity development B(CDE can be seen as part of the process of institutionalising evaluation and developing an evaluation culture 'ithin UN agencies and other organisations involved in C.D* his process is anticipated to generate more high @uality 12( and impact assessments of C.D and to improve the design and outcomes of C.D initiatives* =ur research also highlights the value of adopting a holistic+ participatory+ learning-oriented approach to managing and improving capacity development 'ithin organisations and initiatives+ one 'hich aims to develop planning and evaluation capacities at all levels+ from community to management level* Go'ever+ the use of participatory evaluation methods for (CD raises various critical challenges+ issues and contradictions* Developing+ implementing and sustaining (CD often present particularly difficult challenges and issues for time+ s&ill and resource-poor organisations in developing countries* Challenges and issues that have a particular impact on the effectiveness and sustainability of (CD in the C.D and development conte)t+ include7 he comple)ity inherent to assessing the impact of C.D* %ttitudes to 12( among donors+ C.D organisations and NG=s* 8enior managers ta&ing a short-term vie' of evaluation and not being open to alternative learning-based approach to evaluation* Unrealistic timeframes+ proAect cycles and the demand for C@uic&D results* he need for practical+ fle)ible and sustainable impact assessment frame'or&s for C.D* he diversity of C.D approaches* 1aintaining+ supporting and sustaining evaluation capacity* he need to facilitate 'ide participation in 12( for C.D* Coordinating 12( 'ith C.D programme content and improvement processes* he 'ide range of s&ills re@uired to effectively monitor and evaluate C.D programmes*

)a"le 7( *

roaches to evaluation ca acity develo ment )v

Dominant a

roaches

*lternative a

roaches

)ensions and issues

% participatory approach to (CD is often effective and appropriate+ but can re@uire greater time and planning* Developing evaluation capacities and achieving a high level of participation in (CD and Cbuy-inD and o'nership of 12( can be particularly difficult in pressured and resource constrained organisational conte)ts* here is a need for management to act as models of learning and organisational change+ and greater funding and support for long-term+ sustainable capacity development* Go'ever+ this is often difficult to achieve+ particularly for organisations based on hierarchical or bureaucratic structures*

Narro'+ short-term focus on training in &ey tools such as the logframe and development of individual staff members* % one-off C'or&shop cultureD dominates+ as opposed to longer term capacity development and sustained support over time*

Golistic+ participatory+ longterm approach to capacity development that see&s to develop learning organisations and strengthen capacities Bincluding in a 'ide range of 0+ 12( s&illsE at all levels+ from community to management* hese processes are long-term+ interspersing field'or& 'ith des& based studies* he effective development of learning cultures in organisations re@uires good communication+ cooperation+ collaboration and trust bet'een 12( and other staff and the integration of 12( into the 'hole programme cycle+ 'hich participatory and systemic approaches can offer*

12( is separated from other functions in organisations and lac&s status and po'er* here are issues 'ith communications and programme staff not seeing the value of 12(* his reduces the overall effectiveness of 12( and discourages the development of a learning culture*

!rame'ork for research, monitoring and evaluation of C-D

Dra'ing on ne' thin&ing+ approaches and trends in this area+ this report presents an emerging #rame'or& for 0+ 12( of C.D 'hich incorporates the principles for effective+ appropriate and sustainable 0+ 12( of C.D outlined in this report* his can be summarised as follo's7 %8 Conce tual and theoretical frame'ork % holistic perspective based on comple)ity and systems thin&ing a&es the 'ider conte)t into account #ocuses on gender+ po'er and social norms a&es a holistic approach to evaluation capacity development that aims to develop learning organisations (valuation is seen as an ongoing learning and programme improvement process a&es a long-term+ sustainable perspective on evaluation and evaluation capacity development*

#8 Methodological and re orting frame'ork %dopts an open+ fle)ible approach to designing evaluations and selecting 0+ 12( approaches+ methodologies and methods )vi

Uses participatory approaches as appropriate Uses a mi)ed methods approach and triangulation Impact assessment uses contribution assessment and a dynamic+ moving theory of change and involves longitudinal studies 1a&es more use of @ualitative and participatory indicators (valuation is independent and learns from CfailuresD and negative results (stablishes open communication and feedbac& systems*

Implementing both the conceptual and methodological parts of the frame'or& re@uires a clear strategy* he strategy is re@uired to address a number of challenges7 here is a clear need for advocacy across the UN and 'ith other organisations and donors+ to highlight the importance of C.D and 0+ 12( in development* here is a need for greater understanding of the appropriateness and long-term benefits of participatory approaches* here is a need to create a common understanding of C.D and its various benefits* here is a need to provide sufficient budgets+ resources and time+ including for longitudinal studies* here is a need to improve capacity in conceptualising+ managing and planning 0+ 12( of C.D 'ithin the UN+ and 'ith partners* Long-term capacity development for staff at all levels is re@uired+ 'ith high @uality and yet accessible training and reference resources* Creative and innovative strategies need to be employed to develop the frame'or&+ using participatory and collaborative methods* In addition to the need to collect and present good e)amples of highly effective 0+ 12( for C.D+ there is also benefit in underta&ing meta-evaluations of these e)amples* he open sharing of positive and negative+ intended and une)pected findings needs to ta&e place 'ithin an environment that understands the benefit of learning from success and failure* here is a need to establish a community of practice 'ith online access to e)pert advice*

/trategies and lans for develo ing the Resource Pack

%t the Ne' <or& consultation it 'as agreed that as part of the first phase of the development of the 0esource Pac&+ the report on the literature revie' and consultations 'ould be completed+ 'ith the e)ecutive summary 'idely circulated as a stand-alone document* %

)vii

revised outline of the Guide to Designing the 0+ 12( Process for C.D in the UN 'ill also be completed* =nce funding has been obtained+ Phase " of the development of the Pac& 'ill ta&e place bet'een 1ay and November "$!! and 'ill concentrate on developing and testing a 'or&able version* he version 'ill be developed around the theme C%dvancing the 0ights of %dolescent Girls through Communication for DevelopmentD+ 'hich is the theme for the !"th UN 0ound able on C.D* ()tensive consultation and feedbac& 'ith UN programme staff 'ill be sought* It is hoped that a regional Aoint UN meeting in 5athmandu in mid "$!! might ta&e place 'here the Pac& can be discussed and further developed* 0egional 0ound able meetings might also be used to further develop and obtain feedbac& on the Pac&* % final 'or&able version of the 0esource Pac& 'ill be presented to the !"th UN 0ound able meeting in Delhi in November "$!!*

)viii

)i)

Researching, Monitoring and Evaluating Communication for Develo ment( )rends, Challenges and * roaches
%8 Introduction and methodology
his report highlights a number of important trends+ challenges and approaches associated 'ith researching+ monitoring and evaluating Communication for Development BC.DE 'ithin the UN conte)t* It is a &ey component of the 0esearch+ 1onitoring and (valuation B0+ 12(E 0esource Pac& for C.D Programmes 'hich is currently being developed* :ased on a 'ide-ranging literature revie'+ and an e)tensive research and consultation process+ this report sets out a frame'or& for 0+ 12( of C.D+ 'hich incorporates a proposed set of Principles for effective+ appropriate and sustainable 0+ 12( of C.D* hese principles 'ere developed 'ith significant input from an ()pert Panel and C.D #ocal Points or 12( specialists from seven UN agencies 'ho collaborated in this proAect Bsee list in %ppendi) !E* hrough this process+ strategies and plans 'ere developed that aim to promote a greater appreciation of the role of C.D in international development+ further develop the 0+ 12( 0esource Pac& for practical use at country and field level in order to strengthen evaluation capacities+ and begin the process of further refining and implementing the 0+ 12( of C.D frame'or& outlined in the conclusion to this report* >e hope that this report 'ill be of interest to 12( and C.D specialists+ academics and consultants 'or&ing in this field and others 'ith an interest in the latest thin&ing in this area* 8pecific sections 'ill have different uses and 'ill be of more interest to some groups than others* his report 'as largely 'ritten for a UN audience and this should be &ept in mind*

0ackground
8ince !6--+ the UN has been promoting C.D as an approach to development 'or& that facilitates the inclusion and participation of people in decision ma&ing* UN 0ound ables have advocated that C.D be mainstreamed as an integral part of the UNDs 'or& and mindset+ and that communication is central to achieving the 1DGs Bsee #%=+ "$$/E* he !!th 0ound able Bheld in 1arch "$$6E discussed the institutionalisation of C.D 'ithin the UN system under t'o main themes7 !* %ssessing and demonstrating the impact of C.D and "* Institutionalising C.D* he UN agencies and &ey non-UN partners at the 0ound able reinforced the importance of demonstrating the impact of C.D for furthering the institutionalisation of C.D 'ithin the UN system* % bac&ground discussion paper+ prepared by %ndre' Puddephatt and others for the !!th 0ound able+ provided an overvie' and analysis of the monitoring and evaluation B12(E approaches of UN agencies* It recommended indicators and mechanisms for monitoring and evaluating the impact of C.D* he paper concluded that there is no systemic use of 12( to demonstrate C.D impact among UN agencies* It recommended7 he need to identify the strategic intent of C.D initiatives* Using 12( to establish all the outcomes of the initiative* Development of a 0esource Pac& of 12( %pproaches in C.D Busing @uantitative and @ualitative methodsE to fit the different C.D obAectives of various UN agencies at different stages of programming* !

he paper also put for'ard a recommendation for developing five categories of indicators for measuring C.D* #ollo'ing the !!th 0ound able+ UN agencies agreed to develop a C.D 0+ 12( frame'or& based on the suggestions made in the bac&ground paper* UNIC(# has led this process in collaboration 'ith other UN agencies*

1vervie' of the Resource Pack

*ims and e+ ected results Designed to benefit C.D practitioners and programme staff in the UN as 'ell as their partners+ the original aims of the 0+ 12( 0esource Pac& 'ere to7 %id in demonstrating the impact of C.D approaches ranging from :ehaviour Change Communication B:CCE to community-led Communication for 8ocial Change BC#8CE to 8ocial 1obilisation and %dvocacy* Provide a revie' of literature and bibliography of hyperlin&ed state of the art references+ tools and e)amples of completed research and 12( that demonstrate outcomes and impact of C.D programmes* Provide a roster of specialists B'ith resume on relevant 'or&E 'ho can be tapped to provide technical assistance and training on 0+ 12( in C.D* 1a&e available a 0esource Pac& on C.D 0+ 12( that could be used as a learning resource by UN agencies and partners*

Current contents and ro&ect colla"orators he 0esource Pac& currently includes7 % literature revie' and findings from the research and consultation process Bi*e* this reportE* he e)ecutive summary to this report+ 'hich UNIC(# plans to publish as a stand-alone document* %n outline of a guide to designing the 0+ 12( process for C.D in the UN+ 'hich 'ill be further developed during "$!!* %n e)tensive electronic bibliography of publications+ reports+ tool&its and other resources related to C.D and 0+ 12( of C.D* %n initial directory of consultants 'ho can provide training and advice on 0+ 12( of C.D*

his initial version of the 0esource Pac& 'as developed by Jo acchi B01I UniversityE and June Lennie+ 'ith the assistance of 5irsty 1artin Bboth from 9ueensland University of echnologyE* hey 'or&ed on this proAect in close collaboration 'ith eresa 8tuart and Paula Claycomb from UNIC(#Ds C.D Unit in Ne' <or&* Paavani 0eddy at UNDPDs =slo Governance Centre also provided some input* %n ()pert Panel 'hich included t'o 12( specialists from UN agencies and C.D #ocal Points or 12( specialists from seven UN agencies+ funds or other bodies contributed relevant resources and suggestions and provided comments on and revisions to the draft outputs*

Develo ment of the literature revie' and this re ort

he original brief for this proAect indicated that the 0esource Pac& 'ould include the follo'ing7 "

!* 0evie' of the literature to detail various @ualitative and @uantitative 0+ 12( approaches and tools that can be used in C.D approaches* "* Detail various @ualitative and @uantitative 0+ 12( approaches that can be used in all four C.D approaches to7 establish the strategic intent Bi*e* recognise &ey @uestions that the 12( approaches should ans'er to demonstrate the impact of the C.D initiativeE measure the outcomes of the C.D initiative and establish baselines ,* (laborate on the selection of the follo'ing five categories of result indicators and ho' they fit into the mandates of UN agencies and the C.D focus* hese categories include7 !* (vidence of local a'arenessF "* (vidence of direct impact on behaviour change and social change related to programmesF ,* Local participation and empo'ermentF .* 8cope and scale of coverage of media and communication strategiesF /* Country capacity o'nership and resources for C.D for sustainability and scalability* Propose indicative process indicators for each of the five categories and propose appropriate methodologies to measure the result indicators that agencies can adapt* .* Clearly lay out the challenges of each of the 0+ 12( approaches in C.D* he shape and content of the literature revie' emerged from a 'ide literature search and e)tensive consultations* #ollo'ing phone meetings and discussions 'ith our UNIC(# contacts+ feedbac& and input from the UN #ocal Points and ()pert Panel members+ and initial 'or& on the literature revie'+ the content+ scope and focus of this original outline 'as revised* %s 'ell as a revie' of the literature+ this report also includes e)tensive data and responses gathered from t'o online surveys and intervie's about approaches+ challenges and issues in 012( for C.D* It retains the original focus on various 0+ 12( approaches+ methodologies and methods and includes some feedbac& about the five categories of result indicators and issues related to C.D indicators+ and the challenges and issues in using various 0+ 12( approaches in C.D* Go'ever+ it became clear that a broader focus on the trends+ challenges and issues in C.D and 0+ 12( in C.D+ and in the development conte)t more generally+ 'as essential* his 'as partly due to our findings about inconsistent understandings of C.D and the lac& of appreciation of the important role of C.D in development and partly due to the lac& of understanding of the comple)ity of the process of assessing the impacts of C.D and the need to outline ne' thin&ing about more effective and appropriate 'ays to address these significant challenges and issues* his process led to the development of a set of draft Principles for effective+ appropriate and sustainable 0+ 12( of C.D 'hich 'ere revised based on e)tensive feedbac& from the UN #ocal Points and ()pert Panel* hese principles guided the focus of this report and are a &ey component of the 0esource Pac&* It 'as also clear that the literature revie' needed to ta&e 'ider conte)tual+ structural+ institutional+ and organisational issues into account+ given the many comple)ities+ tensions and contradictions bet'een dominant and alternative approaches to 0+ 12( that 'ere emerging* %nother significant issue that emerged from our consultations 'as the urgent need to strengthen capacities in 0+ 12( of C.D from community to management level* % revie' of the literature on evaluation capacity development B(CDE in the international

development conte)t 'as therefore also included* 8pecific issues related to (CD in the C.D area are identified in this section*

Methodology used in the research

o prepare this report 'e undertoo& a 'ide-ranging literature revie' and consultations 'ith a !/ member ()pert Panel from UNIC(# and various research and consulting organisations and universities from around the 'orld+ and !! C.D #ocal Points or 12( specialists from seven UN agencies funds or other bodies B#%=+ IL=+ UN(8C=+ UNDP+ UN#P%+ UNIC(# and the >orld :an&E* %s 'ell as providing references and relevant literature for the bibliography+ this consultation and data gathering process involved analysis of the follo'ing data and feedbac&7 9ualitative and @uantitative data from t'o detailed online surveys completed by !. ()pert Panel members and !$ UN #ocal Points* Intervie's conducted 'ith five UN #ocal Points Bone face to face and four by telephoneE* #eedbac& received from seven ()pert Panel members and t'o UN #ocal Points on the bac&ground paper for the !!th UN 0ound able on 12( for C.D by Puddephatt et al* B"$$6E* #eedbac& on a draft of the principles for 12( of C.D set out in 8ection " of this report+ provided by nine ()pert Panel members and four UN #ocal Points* #eedbac& on a draft of this report! provided by si) members of the ()pert Panel and five members of the C.D #ocal Point group+ and other specialists from UNIC(#+ UNI#(1 and else'here at a series of meetings in Ne' <or& in December "$!$ Bsee list of participants in %ppendi) ! and Pamer et al*+ "$!!E* 8ome further feedbac& 'as provided follo'ing this meeting*

Process used to develo the literature revie' %n e)tensive literature search 'as underta&en as a first step in preparing this literature revie'* he follo'ing process 'as used to identify and gather relevant literature+ tools and resources7 !* >e prepared a list of relevant literature and other resources that had previously been gathered by the consultants BJo acchi and June LennieE as part of the Assessing Communication for Social Change B%C.8CE research proAect* "* =ur 0esearch %ssistant 5irsty 1artin then conducted searches of the follo'ing7 "6" references in an electronic (ndnote bibliographic library developed by the %C.8C proAect* 0eference lists in &ey documents and reports+ including the bac&ground paper by Puddephatt B"$$6E* >ebsites of a range of relevant organisations including7 Institute for Development 8tudies+ Communication for 8ocial Change Consortium+ International Development (valuation %ssociation+ IN 0%C+ UN(8C=+ UNDP+ >orld :an&+ and %ID8 %lliance*
!

It should be noted that this draft did not include a conclusion to the report*

=ther relevant resource-based 'ebsites such as 1y12(+ he Communication Initiative+ 1edia Development K 1onitoring and (valuation+ (ldis and 1and(Ne's* Google 8cholar - to find information on 12( for C.D 'ith a specific focus on the follo'ing areas of interest to the UN7 changing harmful social norms such as child marriage and female genital mutilation+ governance+ climate change+ employment+ GI?I%ID8+ agriculture+ and child and adolescent participation* ?arious academic library databases and the 8age Aournal search engine*

,* >e also obtained a large number of UN publications and tool&its and other relevant materials from our &ey contacts in UNIC(#+ eresa 8tuart and Paula Claycomb+ as 'ell as from members of the ()pert Panel and some C.D #ocal Points* %s this literature 'as collected it 'as entered into in an electronic (ndnote library* he literature search 'as mainly focused on literature published 'ithin the past seven years* Go'ever+ the bibliography also includes some older publications and tool&its that 'ere seen as relevant or useful* Given the consultantsD and UNIC(#Ds commitment to participatory approaches to development+ and the often+ more 'idely e)pressed importance of participation in C.D and development by all mainstream development agencies+ the bibliography includes a predominance of publications about the use of participatory approaches+ methodologies and methods*

.imitations
>e identified a vast amount of literature in our searches+ far too much for us to cover in detail in this report* It is possible that 'e have missed some &ey te)ts* It 'as also not possible+ due to time constraints+ to revie' and include a range of good e)amples of 0+12( of C.D*" he current aim is that a 'ider range of good e)amples 'ill later be added to the practical CGuide to designing the 0+ 12( process for C.D in the UND section of the 0esource Pac& 'hich is e)pected to be further developed and piloted during "$!! and beyond* 8ome useful e)amples of C.D in practice in the UN conte)t+ including the use of various participatory methodologies and the outcomes of these C.D activities+ can be found in UNDP B"$$6aE and 1cCall et al* B"$!$E*

Definitions of some key terms

=ur consultations highlighted the need for clear definitions of the terms CapproachD+ CmethodologyD+ and CmethodsD as they relate to evaluation+ since they are sometimes used in conceptually confusing 'ays* In this report+ 'e have used the follo'ing definitions of these terms7, :y Ca roachD 'e mean conceptually distinct 'ays of thin&ing about+ designing and conducting evaluations* ()amples of evaluation approaches are the results-based management approach Bsee+ for e)ample+ UNDP+ "$$6cE and sta&eholder-based participatory approaches*
"

Go'ever+ 'here relevant+ 'e have dra'n e)tensively on e)amples and learnings from the Assessing Communication for Social Change proAect 'hich 'e have 'or&ed on over the past four years in collaboration 'ith %ndre' 8&use and 1ichael >ilmore from the University of %delaide and (@ual %ccess Nepal* , hese definitions of methodology and methods 'ere also used in the online surveys for the UN #ocal Points and ()pert Panel+ follo'ing a consultation process*

:y CmethodologyD 'e mean the process+ design or frame'or& that underpins your choice of methods* In some cases you may be informed by more than one methodology* he methodology you use affects decisions about the most appropriate methods to use in achieving your desired outcome* :y CmethodsD 'e mean techni@ues or tools used to plan an evaluation+ gather and analyse data* 1ore than one method 'ill usually be used in an evaluation plan+ chosen according to the methodology* #or e)ample+ your methodology might be Participatory 0ural Communication %ppraisal+ and you might use a mi) of participatory techni@ues and focus group discussions as your methods*

1vervie' of this re ort

his report consists of nine sections+ 'hich are described belo'7 /ection %( Introduction and methodology( his sets out the bac&ground to and aims and scope of the literature revie' and consultations and locates them 'ithin the 'ider aims of the 0+ 12( 0esource Pac&* >e e)plain the process used to search for and revie' the literature and conduct the consultations+ included 'ho 'as involved+ the methods used+ and the process involved in developing the contents of this report* >e also e)plain some limitations and issues that need to be ta&en into account+ and provide definitions of some &ey terms* /ection #( Princi les for effective, a ro riate and sustaina"le R, M3E of C-D( :ased on other related sets of principles+ our research+ and an e)tensive consultation process+ this section presents a detailed set of principles or criteria for effective+ appropriate and sustainable 0+ 12( of C.D programmes* hese principles provide a foundation for the approach that is advocated in this report and in the 0esource Pac&* /ection 5( * roaches to Communication for Develo ment( his e)plores and summarises approaches ta&en to C.D* It dra's on survey and intervie' data 'e collected as part of this proAect+ discussions 'ith C.D #ocal Points and the ()pert Panel around &ey documents+ literature in the field+ and the maAor themes in C.D described in the recent UN publication Communication for Development: Strengthening the Effectiveness of the nited !ations B1cCall et al*+ "$!$E* >e present and discuss definitions of C.D used in the UN+ e)plore issues concerning the institutionalisation of C.D+ reflect on the lin& bet'een communication and participation in C.D 'or&+ and briefly consider some of the issues raised for 0+ 12( of C.D* /ection -( Ne' thinking and trends in R, M3E and im act assessment( >e revie' a number of ne' trends and 'ays of thin&ing about research and evaluation and their value for the assessment of C.D* hey are7 the value of ta&ing a holistic+ systems approach and the significance of comple)ity thin&ing in addressing comple) social problemsF the benefits of participatory and mi)ed methods approaches to 0+ 12(F the need to focus on po'er relations+ gender and social normsF seeing evaluation as an ongoing learning process and the related need to develop Clearning culturesD 'ithin organisations and programmersF and

the shift from measuring and CprovingD impacts to CimprovingD and better understanding programmes* /ection 6( Challenges, issues and strategies( his outlines some of the many comple) challenges and issues surrounding the evaluation of C.D* In addition to revie'ing relevant literature+ this section e)tensively dra's on findings from our online surveys+ as 'ell as and some strategies suggested in consultation and planning meetings in Ne' <or&* >e begin by considering the need for a greater appreciation of the importance of ta&ing the 'ider conte)tual+ structural+ institutional+ and organisation issues into account and issues related to the attitudes and policies of funders and management* 5ey challenges in conceptualising+ managing and planning 0+ 12( of C.D programmes are then briefly considered* his is follo'ed by a more in-depth revie' of the many challenges and issues involved in assessing the impacts of C.D programmes* #inally+ 'e outline suggested strategies for overcoming these challenges+ and some strategies that could be used at different stages in the programme cycle* /ection 7( Evaluation ca acity develo ment in C-D( his begins 'ith an overvie' of various definitions of evaluation capacity development B(CDE 'ithin the development conte)t and a discussion about the various levels+ groups and net'or&s that are seen as important to engage in this process* his is follo'ed by a brief discussion about the increased focus on (CD in the development conte)t and a summary of the &ey issues and findings from our consultations about the need for (CD among UN staff and others involved in C.D* Ne)t 'e revie' the benefits of ta&ing a participatory+ holistic approach to (CD and some of the challenges and issues that need to be considered* >e then outline a number of challenges and issues that are particular to (CD in the C.D conte)t* #inally 'e revie' some more general (CD challenges and issues and outline some recent learnings about increasing the effectiveness of (CD in the C.D conte)t* /ection 9( 2ey a roaches, methodologies and methods for R, M3E of C-D7 >e begin this section by revie'ing four &ey themes and issues related to 0+ 12( frame'or&s and approaches that are important in C.D programmes* >e then present a detailed overvie' of the &ey approaches+ methodologies and methods that 'ere considered effective for planning impact evaluations and assessing the impacts of C.D programmes by the UN #ocal Points and ()pert Panel members 'e consulted and in the literature* % summary of the strengths and limitations of some &ey approaches+ methodologies and methods+ identified by those 'e consulted+ is also presented in a series of tables* #inally+ 'e briefly outline factors and @uestions that need to be considered in selecting the more useful and appropriate approach+ methodologies and methods in 0+12( of C.D* /ection :( Indicators of C-D im acts( >e revie' the literature on indicators of C.D impacts* >e begin by introducing the idea of indicators+ and their roles* Different types of indicators are presented before loo&ing at some general indicators developed specifically for C.D+ and the &inds of indicators that might suit the four main approaches to C.D used across the UN* 8ome of the challenges in the areas of ne' thin&ing in the field are presented+ follo'ed by a summary of some &ey ideas on indicators in C.D* /ection ;( Conclusion and 'ays for'ard7 In this final section+ 'e begin by presenting a summary of the &ey challenges+ tensions and issues that 'ere identified in the report* Dra'ing on ne' thin&ing+ approaches and trends in this area and the principles set out in 4

8ection "+ 'e then present a proposed #rame'or& for 0esearch+ 1onitoring and (valuation of C.D* Ne)t 'e list various strategies that aim to address the many challenges and issues that 'ere identified in this research+ and to gradually refine and implement elements of this frame'or&* #inally+ 'e outline plans for the further development and implementation of the 0+ 12( for C.D 0esource Pac& and related capacity development strategies*

#8 Princi les for effective, a ro riate and sustaina"le research, monitoring and evaluation of Communication for Develo ment
Introduction
:ased on &ey themes and issues in the literature revie' and our consultations 'ith the UN #ocal Points and ()pert Panel+ in this section 'e outline a number of principles for effective+ appropriate and sustainable 0+ 12( of C.D programmes B'ithin the conte)t of the relevant UN General %ssembly resolutionE* In developing these principles 'e have dra'n on 12( principles set out in Chavis et al* B"$$!E+ 1ayou) and Chambers B"$$/E+ Par&s et al* B"$$/E and 0egeer et al* B"$$6E* % set of draft principles 'as circulated to the UN #ocal Points and ()pert Panel* #eedbac& and suggested revisions 'ere received from four UN #ocal Points and nine ()pert Panel members* he draft principles 'ere then revised based on this feedbac&* 1ost of those 'ho responded thought the draft principles 'ere CcomprehensiveD+ CholisticD+ CusefulD or CimpressiveD* #urther feedbac& and suggested revisions 'ere provided during and after consultation meetings in Ne' <or& in December "$!$ Bsee Pamer et al*+ "$!!E* % simplified list of these principles 'as also prepared* hey are included in the Guide section of the 0esource Pac& and in the ()ecutive 8ummary to this report* hey 'ill be further revised and refined as part of the process of developing the 0esource Pac&*

%8 ,eneral rinci les

he research+ monitoring and evaluation process7 Is consistent 'ith the underlying values and principles of C.D in the conte)t of the relevant UN General %ssembly 0esolution Is meaningfully artici atory* he aim is that participants develop an o'nership of the initiative and its evaluation and are active and e@ual partners in decision ma&ing+ 'hich is an honest and transparent process* %s 'ell as local o'nership+ the process should aim to foster national o'nership of the initiative and the evaluation* Participatory approaches can be lin&ed to human rights such as the right to be heard and to be empo'ered+ based on various UN conventions*. It also means being open 'ith communities and the researcherIevaluator having an obligation to e)plain the evaluation and feedbac& results to participants* Involves participants 'or&ing together to actively integrate R,M3E into the ro&ect< rogramme cycle from the conce tion, design, and lanning stages+ rather than seeing these as separate processes* his means that 0+ 12( becomes a responsive and integral part of the iterative process of developing+ implementing+ improving and adAusting C.D initiatives*

hese include UN conventions such as the Convention on the 0ights of the Child+ BC0CE+ the Convention on the (limination of all forms of Discrimination %gainst >omen BC(D%>E+ and the Convention on the 0ights of Persons 'ith Disabilities BC0PDE*

(nables participants to effectively engage in initial discussions about the meaning of fundamental C-D and R, M3E conce ts Bincluding participation+ o'nership+ sustainability+ e@uality and e@uityE* 0esearchers and evaluators ta&e responsibility for being e)plicit and clear about the meaning of concepts used so as to demystify the theory and practices of research and evaluation* 8ees people as actors and agents of their o'n change 8ees research and evaluation as a continuous critical reflection and learning rocess that focuses on policy+ programme and organisational improvement+ process outcomes and capacity strengthening+ as 'ell as assessing the contributions that C.D programmes ma&e to 'ider impacts and outcomes* 0esearchers and evaluators ta&e responsibility to gather lessons from across evaluations and contribute to the 'ider body of &no'ledge about ho' C.D is underta&en and 'or&s* Is provided 'ith sufficient funding, time and resources to be done effectively and appropriately* his re@uires ta&ing a long-term vie' of the 0+ 12( process and the long-term benefits of adopting a participatory approach* %ims to develo ca acity and ensure that findings are used to inform learning and programme improvements 'hich lead to sustaina"le outcomes* his includes nurturing longer-term evaluation and learning processes that are an integral part of 'ider organisational development and change processes+ in 'hich process and content findings are actively used to inform programme improvement+ and 0+ 12( practices strengthen evaluation capacities and capabilities* Is o en to negative findings, 'eaknesses and =failures>+ as 'ell as Csuccess storiesD* his means that the process should see& to ensure a high level of inde endence, integrity and honesty of the evaluation* Loo&s for proAectIprogramme specific obAectives and intended and e)pected results+ but is also o en to un lanned and une+ ected results at both the community level and the institutional level* Includes an action com onent in order to be useful to the initiativeDs end users* In a participatory evaluation approach+ this 'ould entail evaluation participants producing Caction-oriented &no'ledge about their reality+ clarifyLingM and articulateLingM their norms and values *** reachLingM a consensus about further action *** and constructing ***Da common vision of a desirable futureD B:runner 2 GuHman+ !6-67 !!E* >here appropriate+ goes "eyond a focus on individual "ehaviour to consider local social norms+ current policies+ gender and po'er relations+ culture and the general development conte)t* >here appropriate+ attempts to understand ho' and 'hy social change ha ens*

#8 Conte+tual issues

!$

he follo'ing conte)tual issues are ac&no'ledged in the 0+ 12( process and in the choice of methodologies and methods used7 he 'ider social+ economic+ political+ cultural+ communications+ technological and environmental conte)t and the macro and micro issues and "arriers that have an effect on the initiative and are of concern and interest to end users of the initiative* he com le+ity of human systems - this means loo&ing at problems 'ith multiple perspectives+ studying the micro and macro issues+ and understanding ho' they are interdependent BLacayo+ "$$37 ",E* 8ystems thin&ing and comple)ity theory are seen as important to addressing comple) social problems and understanding the dynamics of social change* he impact of other programmes or interventions and the interrelationshi s bet'een them and the initiative being evaluated*

58 Partici ation, o'er and inclusion

he 0+ 12( process aims to7 Develop ositive relationshi s bet'een those involved+ based on high levels of interpersonal and organisational trust and open communication* #acilitate the active artici ation of programme participants and others 'ith an interest in the programme in all stages of the evaluation cycle+ as appropriate* :e as inclusive as possible of a diversity of social groups and ma&e every effort to include the voices and e)periences of the very poor and most marginalised* %ctively and e)plicitly address issues related to gender, class, race and une?ual o'er and voice among participants* (nsure that the vulnera"ility of the most vulnera"le is not increased8 1 enly ackno'ledge the differences bet'een those involved* 1 enly communicate the principles+ values and commitments of the researchersIevaluators to the goals of the initiative+ so as to increase transparency and trust* @alidate evaluation results through the participatory process*

-8 Ca acity develo ment

Capacity development focuses on develo ing learning organisations and strengthening the ca acity of organisations as a 'hole* his involves participants and staff at all levels being actively involved in strengthening their 0+12( capacities+ including staff 'ho are involved in conceptualising+ planning and managing 0+12( of C.D programmes and field staff and community members 'ho participate in the process* It also involves managers acting as models for learning and organisational change* !!

%ppropriate and effective long-term ca acity develo ment in a range of &ey s&ills is provided as part of the evaluation process* hese &ey s&ills include collaboratively planning 0+ 12(+ developing a theory of change+ facilitation+ active listening+ the collection+ management and analysis and triangulation of a range of @ualitative and @uantitative data and report 'riting Bas set out in the Guide part of the 0esource Pac&E* Capacity development aims to increase understanding of the fundamentals of 0+ 12(+ and rovide ractical guidance and sim le, user-friendly information on evaluation*

68 Develo ing evaluation frame'orks and lans

he development of an evaluation frame'or& involves7 Using an a roach that is not rushed+ allo'ing dialogue to begin the process*

%t the start of the evaluation process+ participants reaching agreement a"out the o"&ectives and outcomes of the initiative and their roles in achieving these outcomes* %t the start of the process+ participants clarifying the ur ose of the evaluation and making their e+ ectations for the evaluation clear8 his includes 'hat information they need to &no'+ 'hen they 'ant it by+ the forms in 'hich they 'ant it+ for 'hom they 'ant it and ho' they plan to use it* he aim is that the evaluation is useful to the end users of the initiative and that results and findings are used to improve initiatives and understandings about social change* a&ing the scale of the ro&ect or rogramme into account 'hen planning the evaluation so that the evaluation is proportionate to the programme* Considering the e)tent to 'hich it involves using untested approaches+ 'hich might be high ris&+ high cost and 'eighing these against potential gains if it goes 'ell* Considering the ossi"le risks and "enefits of the evaluation* In consultation 'ith a range of participants+ developing fle+i"le and realistic lans and timeframes for the 'hole 0+ 12( process+ using an organic approach that is responsive to unfolding developments* 8eeing indicators as Aust one part of an 0+ 12( strategy* .ocally derived indicators Bincluding indicators of social changeE are developed using participatory methods+ as 'ell as e)ternally derived indicators* Programme res onsiveness and ada tation can also be seen as measures of success* Indicators should be meaningful and fle)ible+ &ept @uite small in number+ and strongly lin&ed to programme aims and obAectives* hey should reflect the need for gender-disaggregated data and aim to encompass comple)ity* *lternatives to indicators, such as most significant change stories+ should be used as appropriate*

!"

78 Design and methodology

he 0+ 12( process involves7 Using a participatory approach that respects+ legitimises+ conte)tualises and dra's on the kno'ledge and e+ erience of local artici ants8 he aim is to enable the voices of diverse groups of participants to be given an e@ual share and the evaluatorIfacilitator sharing po'er 'ith participants* Participants create shared meaning of their e)periences over time* #ocussing on "oth intended "eneficiaries and other social grou s or communities that may be affected by the initiative+ either directly or indirectly* Using o enness, freedom and fle+i"ility in selecting frame'orks+ approaches+ methodologies+ methods and tools* his means that if some methodologies and methods prove unsuitable others are readily available for use in the evaluation* hey should be the most a ro riate for different issues and purposes+ different types of C.D initiative+ and the aims of the evaluation+ and match the dynamics of the system in 'hich they are implemented* hey should also be culturally a ro riate for the people involved+ used in culturally sensitive 'ays+ and as sim le and ractical as possible* Constraints of the programme and the organisational conte)t and resources need to be ta&en into account* Clarifying the articular research and evaluation aradigm that is being used since mi)ing paradigms can result in confusion and inappropriate compromises* Using a mi+ed methods a roach that combines complementary and varied 'ays to collect+ analyse and interpret data* Continuous monitoring of the communications environment*

98 Im act assessment rocess

he impact assessment process includes7 Dra'ing on revious research and the kno'ledge and e+ erience of community mem"ers to inform the process of identifying indicators+ outcomes and impacts+ including indicators of social change+ community dialogue+ participation+ empo'erment and capacity development* he use of longitudinal studies to assess lasting and sustainable change* If appropriate+ developing a dynamic, moving theory of change 'hich is trac&ed and adapted as part of the ongoing evaluation process* Considering the short-term+ intermediate and long-term outcomes and impacts of initiatives+ based on the programDs vision of success and theory of change+ 'hich is regularly revie'ed and revised by participants* !,

0eaching agreement 'ith participants on a realistic timeframe that is li&ely to be needed to e)pect some evidence of the proposed outcomes* his timeframe may need to be adAusted over time+ given changes to local+ national and global conte)ts and other factors* Identifying rocess outcomes Bassociated 'ith the implementation of an initiativeE so that lessons can "e learned about ho' the obAectives of the initiative 'ere achieved and the conditions re@uired to achieve them* Loo&ing for une+ ected, indirect and negative im acts as 'ell as intended+ direct and positive impacts* *d&usting "aseline information as necessary to recognise changes in the social and communication conte)t* Using a moving baseline as necessary to trac& change*

:8 Data organisation, analysis and validation

12( data is disaggregated by gender+ age+ caste+ education and income level and other relevant differences+ 'hich are ta&en into account in the analysis* his is built into the design of 12( systems and plans from the beginning* Data analysis and interpretation is conducted in collaboration 'ith &ey participants and sta&eholders such as programme staff in order 'herever possible+ to increase the rigour, trust'orthiness and utilisation of findings* #indings from the use of different methods and researchers and other aspects of the evaluation are triangulated to increase the rigour of the results and to ensure that a diversity of voices and vie's are included in reports and other feedbac& methods*

;8 Communication and information sharing rocesses

=pen+ appropriate and effective communication and feedbac& systems and processes are established at the beginning of an evaluation to regularly kee artici ants informed and involved in the 0+ 12( process* % range of communication methods are used to feedbac& findings to participants+ senior management+ funders and others+ and to share learnings and e)periences* Creative and engaging communication methods such as digital storytelling and sharing stories of significant change are used 'here possible* Positive and negative+ intended and une)pected impact assessment findings are shared o enly and honestly 'ith participants+ donors and funders and the larger development community* !.

%$8 1utcomes of the R,M3E rocess

he 0+12( process aims to contribute to7 Demonstrating the im act of C.D programmes+ based on e)pected and une)pected outcomes* 1ngoing ca acity "uilding+ and increasing empo'erment+ human rights+ gender e@uity+ dialogue and other C.D aims* Im roving C.D initiatives+ and broader development initiatives that C.D activities are attached to* his 'ould include creating feed"ack loo s of outcomes to help improve a proAect as it continues to be implemented* /trengthening organisational cooperation+ collaboration+ coordination and performance and the develo ment of learning organisations8 More effective decision making and learning about C.D and related development programmes* he development and implementation of more effective policies+ strategies+ programmes and initiatives that address development goals*

!/

58 *

roaches to Communication for Develo ment

/ummary of key findings

Communication for Development is about people rather than technologies+ and 'hile it has multiple meanings+ is generally understood to be about the use of communication in participatory processes for social change* Current approaches and understandings of C.D define communication as a t'o-'ay system+ promote the importance of enabling dialogue and discussion and the sharing of &no'ledge and s&ills+ rather than information or message delivery* 1any C.D approaches confuse+ or use both vertical and horiHontal models+ in contradictory 'ays* Participation is an essential and intrinsic component of C.D* Participation in development is often only rhetoric+ or implemented in top-do'n 'ays* #ull and direct participation is often hard to achieve 'ithin dominant organisational cultures and 0+ 12( approaches* C.D can be a mechanism for achieving participation in development more broadly* 8uccessful C.D challenges ine@uitable po'er structures* %cross the UN+ understanding and support for C.D varies* %s an outcome of the !!th 0ound able+ the UN has set out the four main approaches it ta&es7 :ehaviour Change CommunicationF Communication for 8ocial ChangeF Communication for advocacyF and 8trengthening an enabling media and communication environment* here is a need to institutionalise C.D 'ithin the UN+ and to strengthen its institutional position by finding 'ays to demonstrate impact* he 12( needs of C.D are different to the mainstream* It is important therefore to demonstrate the rigour of participatory and mi)ed methods approaches* here is a need to build capacity in 0+ 12( of C.D+ and to advocate for C.D 'ith senior staff and donors*

Introduction
his section e)plores and summarises approaches ta&en to C.D* It dra's upon the survey and intervie' data collected as part of this proAect+ other discussions 'ith the UN C.D #ocal Points and the ()pert Panel around &ey documents+ the literature in the field+ and the maAor themes in C.D described in the recent UN publication Communication for Development: Strengthening the Effectiveness of the nited !ations B1cCall et al*+ "$!$E* It presents and discusses the definitions of C.D used in the UN+ e)plores issues concerning the

!3

institutionalisation of C.D+ reflects on the lin& bet'een communication and participation in C.D 'or&+ and considers some of the issues raised for 0+ 12( of C.D* Communication for development or development communication is essentially about people rather than technologies+ and is both a field of &no'ledge and of practice B>aisbord+ "$$-F >il&ins+ "$$$F >il&ins 2 1ody+ "$$!E* >aisbord B"$$!E+ surveying its multiple meanings+ describes development communication as Ca sort of umbrella term to designate research and interventions concerned 'ith improving conditions among people struggling 'ith economic+ social political problems in the non- >estern 'orldD B>aisbord+ "$$!7 "-E* 8ervaes suggests+ in his introduction to Communication for Development and Social Change B"$$-E that 'hile the 'ords used to define C.D might change over time+ since the mid !64$s the intent is reasonably constant* a&ing some of the e)amples 8ervaes B"$$-E cites+ and adding others+ there is a sense of a general trend7 0ogers B!643E described development communication as the study of social change brought about by communication research+ theory and technologies to bring about development+ 'ith development understood as a participatory process of social change* #%= B!6-.E defined C.D as a social process to'ards common understanding and concerted action of all involved in a development initiative Bin 8ervaes+ "$$-E* #raser and ?illet B!66.E considered the planned use of communication techni@ues+ activities and media that allo' people to both guide and e)perience change and intensify the e)change of ideas+ can bring people together in a common cause* his+ they state+ is a fundamental re@uirement for appropriate and sustainable development* he UN resolution /!I!4" B!6647 "E+ stresses Cthe need to support t'o-'ay communication systems that enable dialogue and that allo' communities to spea& out+ e)press their aspirations and concerns and participate in the decisions that relate to their developmentD* he >orld Congress on Communication for Development B>CCDE "ome Consensus B"$$37 "E defines C.D as a Csocial process based on dialogueD+ as Cabout see&ing change at different levels including listening+ building trust+ sharing &no'ledge and s&ills+ building policies+ debating and learning for sustained and meaningful changeD*

he underlying theories of development communication have changed @uite significantly over time+ 'ith modernisation theory and diffusion approaches being replaced by dependency+ and participatory theories and approaches* %lthough participatory communication can no'adays be considered a dominant paradigm+ older modernisation paradigms have not been completely displaced* >hile+ in !643+ as >aisbord B"$$-E points out+ even (verett 0ogers+ the most prominent proponent of the diffusion paradigm+ recognised the limitations of diffusionism and the importance of an approach that foregrounds community participation B0ogers+ !643E+ in his revie' of recent trends in empirical research on C.D+ Inaga&i B"$$4E sho's that the modernisation paradigm and diffusion approach have a persistent influence* =ver time+ the idea that communication is about meanings and about processes+ rather than about the transmission of messages+ has concretised+ even if it is yet to be as 'idely or fully practiced as generally thought B#raser 2 0estrepo-(strada+ !66-F Inaga&i+ "$$4E* % !4

participatory communication approach is highly complementary to a human development approach+ as it promotes horiHontal and participatory models of development rather than vertical+ one-'ay+ top do'n+ or tric&le do'n models+ more suited to modernisation and gro'th theories of development B8ervaes+ "$$-F >aisbord+ "$$!E* 1odernisation and diffusion models of development and of development communication are generally considered to be outdated B8ervaes+ "$$-F >aisbord+ "$$!+ "$$-E+ and yet they still appear in practice BInaga&i+ "$$4E* =ne of the UN #ocal Points for the 0esource Pac& proAect said that Cacademics are !$ years ahead of us J 'hat they &no'F only about !$N of it is applied by us J 'e simply donDt have the tools to apply itD* :alit suggests that 'hile C.D practitioners have reached a common understanding on the principles governing their discipline+ it is a Csoft and social science that has to do 'ith listening+ building trust and respecting local cultures J not easy concepts to understand for policy ma&ers and program managers 'ith a bac&ground in hard sciencesD B:alit+ "$!$a7 /E* :alit B"$!$aE also suggests that there are po'er issues at sta&e and that enabling full and direct participation challenges entrenched po'er structures* Participation in development is considered by some as a false participation+ a buHH 'ord+ rhetoric+ incompatible 'ith procedures and goals of aid organisations+ a threat to those in positions of po'er B:ailur+ "$$4F :alit+ "$!$aF Coo&e 2 5othari+ "$$!F #raser 2 0estrepo-(strada+ !66-F Leal+ "$$4F 1c5ee+ !66"F >hite+ !663E* %nd yet+ dialogue+ debate+ the t'o-'ay flo's of information+ and the co-creation of &no'ledge are regularly put for'ard as important pieces of the development Aigsa'+ and intrinsic to the idea of participatory development J as evidenced in the processes promoted for the development of the Poverty 0eduction 8trategy Papers BP08PsE and the >orld :an&Ds insistence that the strategic use of communication tools and concepts is essential to its success*/ his ac&no'ledges a move a'ay J in rhetoric at least J from vertical models of communication for development to horiHontal modelsF in other 'ords+ a shift from sending messages to providing an opportunity for people to engage in dialogue+ share &no'ledge and as& @uestions+ 'hich+ coincidentally+ the Cne' communications environmentD is ideally able to facilitate BDeane+ "$$.E* %t our consultation meetings in Ne' <or&+ there 'as felt to be a need to stress the follo'ing t'o particular aspects of C.D7 !* Participation in C.D refers to engagement 'ith various sta&eholders at all points of a development process* Indeed+ C.D can be demonstrated to be a mechanism for achieving the levels of participation that development more broadly strives and often struggles to achieve B acchi+ "$$6E "* Participatory approaches to development+ to C.D+ and to 0+ 12( inevitably brings up issues of po'er* It is important in C.D to recognise this+ and to be alert to po'er dynamics* 8uccessful C.D 'ill al'ays+ to some e)tent+ involve challenging po'er structures* his is because it depends on actively engaging a range of people+ not only encouraging everyone to have an e@ual voice+ but also encouraging active listening across difference B=DDonnell et al*+ "$$6E*
/

http7IIgo*'orldban&*orgI/1C?,J-48$* Unsurprisingly to C.D academics and practitioners+ P08Ps have been criticised for reinforcing e)isting structures and politics of representation Bsee Gould+ "$$/E*

!-

Defining C-D in the UN

Understanding of and support for C.D across UN agencies varies Bsee belo' discussion on institutionalisationE* #or one of our #ocal Point respondents+ C.D in her agency conte)t Cis neither institutionalised+ nor an accepted practiceD* 0ather a fe' people are interested in C.D in LO agencyM+ and to her mind+ Csome of it is Aust plain common sense* :ut it is not embedded in given programmes or resources in the same 'ay as it is+ for e)ample+ in L< agencyM*D It is important to recognise that different UN agencies ta&e different approaches to C.D+ as one #ocal Point e)plained+ CC.D should not have one specific approach in the UN J other'ise agencies 'ould be obstacles to each otherP in each case the approach 'ill relate to their particular mandate+ and each has their o'n mandateD* 8ervaes B"$$4E also @uestions the obAective of a common UN approach on C.D+ in part because of their different mandates* In preparation for the !!th 0ound able+ Puddephatt et al* B"$$6E described C.D under t'o headings7 one-to-many+ or diffusion communicationF and+ t'o-'ay+ or participatory communication* hey add that the former approach does not preclude+ and is in fact often enriched+ by the second approach* 8ervaes agrees that the various approaches used in UN agencies can be grouped into either diffusionist or participatory models of C.D+ but that since the underlying theories and conse@uences are rarely considered+ Cmany approaches contain references to both diffusionist and participatory perspectives in obvious contradictory and illogical 'aysD+ 'hich has been described as Cparticipatory diffusion or semantic confusionD B8ervaes+ "$$47 .-6E* >aisbord B"$$!E e)amines cases 'here these t'o approaches are brought together+ and 'hile not 'anting to pass Audgment+ he ac&no'ledges that C he fundamental issue continues to be that definitions of the problem are different+ and e)pectedly+ theories+ strategies and techni@ues still offer essentially opposite analyses and recommendationsD B>aisbord+ "$$!7 "-E* his approach to C.D 'as also considered problematic by some of the ()pert Panel for the 0esource Pac& proAect+ since it dra's together t'o essentially contradictory approaches+ 'hich presents an overly cautious statement about C.D+ and results in inappropriate compromises* he UN resolution /!I!4" B!664E gives a clear but broad ranging definition+ 'hich includes support for t'o-'ay communication systems that enable dialogue+ Callo' communities to spea& out+ e)press their aspirations and concerns and participate in the decisions that relate to their developmentD* here is concern that decision ma&ers 'ithin the UN fail to understand this aspect of C.D* %t the same time+ a UN respondent 'as &een to point out that 'hile it is Cgood to give parameters to decide 'hat C.D means across the UND+ it is very important that no single definition dominates* >hile the !$th 0ound able set out to develop a common approach to C.D across the UN BUN(8C=+ "$$4E+ the emphasis for the !!th 0ound able 'as on the need to institutionalise C.D+ to ensure that decision ma&ers understand the importance of communication for development and give it due attention* Participants at the !!th 0ound able agreed to 'or& together to produce a document that 'ould describe the diverse approaches to C.D across UN organisations BUNDP+ "$$6bE* Communication for Development: Strengthening the Effectiveness of the nited !ations B1cCall et al*+ "$!$E+ sets out to e)plain and e)plore the main approaches in play across the agencies* It recognises that there is a significant crossover*

!6

he four main CstrandsD are described as follo's7 :ehaviour Change Communication B:CCE Communication for 8ocial Change BC#8CE Communication for advocacy 8trengthening an enabling media and communication environment*

:CC is described as envisioning social and individual change+ having evolved from information+ education and communication BI(CE programmes+ to incorporate greater dialogue and o'nership* It is particularly relevant to health-related programmes Bsee http7II'''*globalhealthcommunication*orgIstrategiesE* C#8C promotes dialogue through participatory and empo'ering approaches* It stresses the role of people as change agents+ and long-term social change Bsee :yrne et al*+ "$$/F #igueroa et al*+ "$$"E* Communication for advocacy aims to change governance+ po'er relations+ social relations+ attitudes and institutions* =ngoing advocacy see&s to build enabling environments for positive change Bsee 1orariu et al*+ "$$6E* 8trengthening an enabling media and communication environment is about capacity building for7 !* a free+ independent and pluralistic mediaF "* broad access to communication and media channelsF ,* a non-discriminating regulatory environmentF .* accountability systemsF and /* freedom of e)pression and participation in decision ma&ing processes Bsee 1edia 1atters+ '''*interne's*frII1GIpdfI1ediamatters*pdfQpageR63E* In our survey 'ith UN respondents the most common C.D approach ta&en by their agencies is Communication for Policy %dvocacy B4/NE+ follo'ed by :CC B3"*/NE* =ther approaches 'ere7 C#8C+ 8ocial 1obilisation and 1edia %dvocacy and 1obilisation* =f the agencies represented+ UNIC(# and the >orld :an& appear to ta&e the broadest range of C.D approaches* In terms of 'hat C.D means+ respondents gave a 'ide range of definitions and meanings of 'hat C.D meant in their o'n 'or&* he main themes 'ere that C.D is a process that involves participation+ community engagement+ dialogue+ access to &no'ledge and information+ building capacities+ and some form of advocacy* It see&s or promotes human rights and change+ including behaviour or social change+ and involves using communication to achieve development goals* 8urvey responses from the ()pert Panel 'ere some'hat different+ 'ith the most common C.D approach ta&en by the UN agencies and other organisations that respondents 'or& 'ith being C#8C B-.*3NE+ follo'ed by :CC B3!*/NE+ then Communication for Policy %dvocacy B.3*"NE* =ther approaches ta&en are 8ocial 1obilisation and 1edia %dvocacy and 1obilisation* %s 'ith the UN respondents+ a 'ide range of definitions and meanings of 'hat C.D meant 'ere given* he main themes 'ere that C.D involved the use of communication to advance development+ it 'as a process that involves the use of a range of communication tools and methods+ dialogue+ debate and participation+ and sharing ideas+ &no'ledge and s&illsF it see&s or promotes change+ including behaviour or social change+ and is a planned+ evidence-based process* "$

he agencies in 'hich the UN #ocal Points 'or&ed tended to ta&e a more policy-oriented approach to C.D compared 'ith the UN agencies and other organisations that the ()pert Panel 'or& 'ith+ 'hich appear to ma&e much more use of the more participatory and change-oriented C#8C approach* he responses highlight the range of terminology used+ as 'ell as the similarities in the underlying themes or meanings of C.D* Common themes 'ere that C.D involved the use of communication to advance or achieve developmentF it is a process that involves dialogue+ debate and participation+ and access to &no'ledge and information+ and see&s or promotes change+ including behaviour or social change* Go'ever+ some of the ()pert Panel also commented that C.D involves the use of a range of communication tools and methods and is a planned+ evidence-based process+ 'hile some UN respondents also noted that C.D involves promoting human rights and capacity building*

Institutionalisation
he !!th United Nations Inter-%gency 0oundtable on Communication for Development had a dual and complementary focus J it set out both to e)plore 'ays to effectively institutionalise C.D 'ithin the International Development %genda B#ee& 2 1orry+ "$$6E and to find 'ays to demonstrate impact and thereby strengthen C.DDs institutional position BPuddephatt et al*+ "$$6E* #ee& and 1orry conducted a survey and intervie's in late "$$- and early "$$6 as preparation for the !!th 0ound able meeting+ 'hich set out to e)plore the level of CUN agency understanding+ acceptance+ and implementation of C.D Lor its e@uivalentsM as a central+ critical+ and core element of their policy frame'or&s and programming strategiesD B#ee& 2 1orry+ "$$67 .E* heir findings are to some e)tent echoed in our survey and intervie's conducted in mid to late "$!$ in relation to the current situation of understanding and institutionalisation of C.D+ although this 'as not the main focus of our survey and intervie' @uestions* he message from UN respondents that C.D is undervalued and not 'ell understood in some of the UN agencies 'as very clear* #ee& and 1orryDs survey findings included7 C.D lac&s central status in policy+ strategy and planning* here is a lac& of demonstrated impact data* here is an inade@uate number of s&illed C.D staff* here is a need to learn across UN agencies+ and support each other* here is a lac& of dedicated funding* C.D is supported in some agencies and not in others+ and varies across time* Corporate communications is prioritised over C.D*

In fact+ #ee& and 1orryDs survey from "$$-I"$$6+ and the survey that this literature revie' dra's upon+ reinforces surveys from !66. B#raser 2 #Aortoft for UNIC(# and >G=E+ "$$, B0amireH 2 9uarry for ID0CE+ and "$$3 B#raser 2 0estrepo for the >orld :an&E Bcited in

"!

:alit+ "$!$a7 .E* (ach highlight the recurring problem of decision ma&ers in development organisations not appreciating 'hat C.D means+ or its important role in development*

Communication and artici ation

Current thin&ing on C.D intrinsically lin&s communication 'ith participation* :oth are simultaneously promoted 'ithin development agendas and agencies+ at least at the rhetorical level+ and at the same time+ often misrepresented and misunderstood* #or one of our #ocal Point respondents+ CC.D is based on participatory approaches+ it 'ouldnDt be C.D 'ithout it+ it 'ould be public relations or some other form of communicationD* Communication is on the one hand considered to be marginalised in the development process+ the fifth 'heel on the cart B:alit+ "$!$aE+ not even allocated the importance of the spare tire on the development car BGumucio Dagron+ "$$-E* :alit B"$!$aE also notes that communication is absent from the 1illennium Development Goals B1DGsE* =n the other hand+ there is recognition that communication needs to be harnessed to achieve the 1DGs BUN(8C=+ "$$4E+ and communication is heralded as a maAor pillar for development and change* he >CCD+ held in 0ome in =ctober "$$3+ produced a set of recommendations to policy ma&ers based on an understanding that communication is a CmaAor pillarD for development and social change+ placing community participation and o'nership on the part of the poor and e)cluded at the heart of communication for development* %mong the Cstrategic re@uirementsD specified in the "ome Consensus are7 access to communication tools so that people can communicate amongst themselves and 'ith decision ma&ersF recognition of the need for different approaches depending on different culturesF and+ support to those most affected by development issues to have a say* here is a stress on the need to build capacity for development communication at all levels+ from community members to development specialists* It is not to be reduced to public relations or corporate communications+ 'hich 'ould fail to appreciate that good communication implies a t'o-'ay relationship* In practice+ communication as understood by decision-ma&ers+ is often reduced to vertical information delivery+ public relations+ or dissemination+ rather than used for meaningful participation B:alit+ "$!$aF Gumucio Dagron+ "$$3F 9uarry+ "$$-E* Indeed+ a revie' of the literature commissioned by the >orld :an& demonstrated that 'hile participatory approaches are Cofficially sanctionedD by most of the maAor development organisations+ the remnants of modernisation theory persist BInaga&i+ "$$4E* >hile C.D+ particularly the C#8C approach Bsee communicationforsocialchange*orgF Gray-#elder 2 Deane+ !666E+ and participatory development more broadly places dialogue at the centre of development+ meaning a participatory engaged relationship 'hich involves valuing voice+ recognition and respect+ 'e are still fundamentally lac&ing an understanding of the information+ communication and net'or&ing needs and aspirations of people 'ho are marginaliHed or socially e)cluded* >e need to more effectively listen across difference and ine@uality B=DDonnell et al*+ "$$67 .",E* <et the very institutions 'hich e)cluded communities might usefully try to engage 'ith through communication technologies and activities are often structurally unsuited for listening+ and indeed development itself generally positions the poor and marginalised as listeners rather than spea&ers Breceivers rather than senders of messagesE* he Listening ProAect+ conducted 'ithin the CD% Collaborative Learning ProAects+ have held Clistening ""

conversationsD 'ith more than .+/$$ people since "$$/ in aid recipient societies across the globe* hese people include ordinary people+ community leaders+ government officials+ civil society and religious leaders+ people from education+ business+ health and NG= sectors* hey found that the aid agenda and its systems might be far more effective if ClisteningD happened 'ith people on the ground+ building relationships at that level* hey found that the Csystems of international assistance bias the 'ays that agencies and aid 'or&ers listen and do not listen+ 'hat they listen to+ 'here and 'hen they listen+ and to 'hom they listenD B he Listening ProAect+ "$$6E* Progressive proponents of C.D consider it a fundamental principle that people on the ground need to be included in development processes at all stages* Communication+ understood as a t'o-'ay relationship that not only ac&no'ledges the right for people to be heard+ but includes prioritising effective listening+ and recognising and respecting alternative forms of &no'ledge+ is needed to achieve this B:alit+ "$!$aF Gumucio Dagron+ "$$-F 9uarry 2 0amireH+ "$$6F 8ervaes+ "$$-F acchi+ forthcomingE* Giving people a voice on the &ind of scale the >orld :an& undertoo& through their consultations 'ith the poor proAect Bmore often referred to as ?oices of the Poor J see http7IIgo*'orldban&*orgIG!N-4.3O!$ and Narayan et al*+ "$$$aF Narayan et al*+ "$$$bF Narayan 2 Petesch+ "$$"E can be considered a part of such a process J hearing the voices of 3$+$$$ poor people across the 'orld as a consultation e)ercise on 'hat constitutes poverty has helped us modify our ideas about poverty and development* :ut in terms of developing effective development programmes and implementation+ ongoing participation+ dialogue and o'nership of the development process and goals is re@uired* Indeed+ the voices of the poor proAect highlights the high importance that people living in poverty place on having a voice+ a say in decisions that affect them BNarayan+ "$$$aEF clearly indicating that Ccommunication is &ey and centralD to development B:alit+ "$!$a7 "E* he horiHontal and dialogic components in content flo's+ the essential components of CcommunicationD BGumucio Dagron+ "$$-E+ are often reduced+ in development+ to information delivery+ or to allo'ing people to spea& but not entirely engaging 'ith them and including them in the processes of development* he participatory approaches that allo' dialogue and engagement are often considered costly+ to ta&e a lot of time+ and difficult to accommodate in 'ell defined development proAect plans and logframes B:alit+ "$!$aE* 8uch development planning approaches are considered by some to be an e)ample of the stifling of participation+ as they Creinforce relationships of po'er and control*** LembodyingM a linear logic associated 'ith things rather than peopleD BChambers 2 Pettit+ "$$.7 !./E+ they often do not give space for styles of communication and 'or&ing that are more appropriate than >estern styles+ in non->estern conte)ts B1arsden+ "$$.E* %nd yet+ C.D proponents insist that C'ithout peoplesD participation+ no proAect can be successful and last long enough to support social changeD BGumucio Dagron+ "$$-7 4$E+ participation and o'nership are crucial for sustainability B9uarry 2 0amireH+ "$$6E+ and+ therefore+ long-term perspectives for participatory communication are not as costly as the failure of e)pensive proAects due to lac& of the involvement and o'nership on the part of the communities concerned B:alit+ "$!$a7 4E*

C-D and research, monitoring and evaluation

",

Demonstrating the impact of C.D through research+ monitoring and evaluation is a crucial part of moving C.D up the development agenda and achieving the &ind of institutionalisation that is desired* =ne of the recommendations of the "ome Consensus+ is that C.D programmes Cshould be re@uired to identify and include appropriate monitoring and evaluation indicators and methodologies throughout the processD B>CCD+ "$$,7 ,E* Go'ever+ as 'e discuss further in 8ection /+ demonstrating the impact of C.D is often much more comple) and difficult than for other development initiatives* %nother issue identified by many of our UN respondents is that 12( approaches that are suitable for C.D re@uire different s&ills and frame'or&s to mainstream and accepted UN evaluation e)pertise* his is emphasised in the comments of one of the UN #ocal Points+ 'ho described+ A fault line in the ! whereby M#E specialists don$t understand the M#E needs of C%D& so we have to get e'ternal specialists in( )hese *funky$ participatory M#E consultants demonstrate impact in interesting ways& but the ! M#E specialists don$t recognise their work& findings or approach( +e$re therefore not able to make this approach more central across the !( It is important+ therefore+ to demonstrate the rigour of participatory approaches+ and to be able to @uantify results+ using a mi)ed methods approach to 0+ 12(* he area 'here there is most cross-over bet'een the t'o 12( approaches Bmainstream and participatoryE is in the area of changing social norms+ and this is 'here this respondent sees the most productive conversations ta&ing place+ Cfor e)ample+ in maternal health+ there is recognition that targets 'onDt be met unless 'e can help to change social norms+ and that is 'here 'e can get the C.D approach+ and participatory 12( approaches on the agenda+ because they are other'ise unable to achieve thisD* %nother UN respondent echoed the concern that in order to underta&e 12( for their C.D 'or& they 'ould need to contract in e)ternal evaluators+ since they do not have the s&ills and e)pertise for participatory style 12( internally+ and this is the &ind of 12( that she felt 'as appropriate* %nother respondent stressed the need to locate C.D 0+ 12( into the results-based management approach BUNDP+ "$$6cE 'hich is the basis for UN programme planning* C.D can become a broad+ supporting strategy 'ith its o'n outcomes* Currently the 'ay C.D interventions are planned are scattered+ and there is a lac& of organised &no'ledge about tools and approaches* Planning is an important part of the process+ and all parts of the process need to be monitored and evaluated* =ther respondents felt the emphasis in terms of improving capacity to conduct 12( for C.D needs to focus on the methods and tools+ and less on planning aspects* Go'ever+ many respondents agreed on the particular need to demonstrate validity of both C.D itself+ and the 12( approaches that are associated 'ith it* #rom the survey 'ith the UN #ocal Points and ()pert Panel+ a set of common challenges 'ere identified+ such as problems 'ith evaluation design+ &no'ledge of suitable methodology and methods+ lac& of budget and e)pertise to underta&e 12( of C.D+ and unrealistic demands+ targets and time frames* he ()pert Panel also commented on problems 'ith demonstrating impact+ dominant assumptions+ CbiasesD and a lac& of openness of funders and commissioners regarding valid methodologies and methods+ lac& of planning and foresight+ and lac& of a'areness and &no'ledge of impact assessment and Cthe practical application of different methodologiesD*

".

:oth groups suggested more capacity building in 0+ 12( and the need for advocacy 'ith senior staff and donors about the importance of C.D and appropriate 0+ 12(* he ()pert Panel also suggested the need for7 Greater focus on innovative+ Cnon-dominantD approaches and e)perimentation* 1ore attention to understanding Cthe fundamentals of evaluationD and provide Cpractical guidanceD and simpler information on evaluation* Provide sufficient budgets and time for proAects*

>e discuss these issues in more depth in the remainder of this report*

Conclusion
C.D is 'idely understood today as a field of &no'ledge and of practice that uses communication and media in participatory processes of social change* he underlying theories of development communication have changed @uite significantly over time* >hile participatory communication can no'adays be considered a dominant paradigm+ older modernisation paradigms have not been completely displaced* =ver time+ the idea that communication is about meanings and about processes+ rather than about the transmission of messages+ has concretised* % participatory communication approach promotes horiHontal and participatory models of development rather than vertical+ one-'ay+ top do'n+ or tric&le do'n models* >hile C.D practitioners have reached a common understanding on the principles governing their discipline+ it is sometimes challenging for policy ma&ers and programme managers schooled in the hard sciences to understand* 1any C.D approaches confuse+ or use both vertical and horiHontal models+ in contradictory 'ays* Participation+ in C.D+ refers to engagement 'ith various sta&eholders at all points of a development process+ and can be demonstrated to be a mechanism for achieving the levels of participation that development more broadly strives and often struggles to achieve* It inevitably brings up issues of po'er* It is important in C.D to recognise this+ and to be alert to po'er dynamics* 8uccessful C.D 'ill al'ays+ to some e)tent+ involve challenging po'er structures* Understanding of and support for C.D across UN agencies varies+ 'ith the four main strands understood as7 :ehaviour Change Communication Communication for 8ocial Change Communication for advocacy 8trengthening an enabling media and communication environment*

C.D is undervalued and not 'ell understood in some UN agencies* It lac&s status in policy and strategy* here is a lac& of impact data* here is a persistent problem of decision ma&ers in development organisations not appreciating 'hat C.D means+ or its important role in development* Demonstrating the impact of C.D through research+ monitoring and "/

evaluation is a crucial part of moving C.D up the development agenda and achieving the &ind of institutionalisation that is desired*

"3

-8 Ne' thinking and trends in R, M3E and im act assessment

/ummary of key findings
% holistic perspective based on systems and comple)ity thin&ing and participatory 0+ 12( is increasingly seen as important to address comple) social problems* Comple)ity thin&ing can help us to loo& at things differently and better understand ho' and 'hy social change happens* Participatory 12( approaches are very appropriate and effective for C.D and have numerous benefits over the long-term* Go'ever+ there is a lac& of investment in these approaches* % mi)ed methods approach to 0+ 12( can shed light on different issues and can increase the strength and rigour of evaluation and impact assessment findings* 0esearch and evaluation needs to consider the gender and po'er relations inherent in social interactions and organisations+ along 'ith local social norms+ in order to ma&e interventions more successful and sustainable* (valuation is increasingly seen as an ongoing learning process and an important means of strengthening capacity and improving organisational performance* his re@uires the development of learning organisations* >hen these alternative approaches to evaluation are used+ there is a shift from measuring and proving impacts+ to'ards better understanding and improving programmes*

Introduction
%s this report clearly sho's+ a top-do'n donor driven approach to 0+ 12( tends to ignore many significant conte)tual issues and outcomes through the use of dominant+ mainly @uantitative-based+ evaluation methodologies* his often results in a 'arped or incomplete picture of the outcomes of C.D interventions* >e therefore need to use more appropriate and effective alternative methods for researching and evaluating C.D+ based on ne' thin&ing in this area* %t a recent meeting 'hich discussed ne' methods for evaluating social change communication+ participants reaffirmed the need to Cfocus on approaches and methods that 'ill complement Brather than replaceE those that dominate at present+ 'hile &eeping Sbigger pictureT issues to the fore and avoiding becoming bogged do'n in stereotypes and narro'er methods-oriented debates of limited valueD* hey agreed that+ to be constructive+ C'e have to stay focused on the bigger picture and larger issues+ 'hich inform choice of method and approachD B:yrne+ "$$6b7 3E* 8imilarly+ Inaga&i B"$$47 ./E suggests that Cdevelopment communication research needs to address the gaps among different methodological paradigms in order to advance more holistic understanding of communication processes in international development settingsD* Ge goes on to argue that7 C he e)isting division bet'een methodological paradigms must be replaced by constructive dialogues bet'een "4

different approaches so that the empirical evidence generated in the scholarship 'ill achieve greater legitimacy and substanceD BInaga&i+ "$$47 .3E* Go'ever+ concerns have been raised about fundamental incompatibilities bet'een t'o distinctive trends in 12(7 results-based management 'hich is focussed on evaluation for accountability and a fle)ible approach based on understanding the cultural conte)t of interventions and the differing values and perceptions of local sta&eholders+ based on evaluation for learning BCrac&nell+ "$$$F (arle+ "$$"E* his highlights the need to give more consideration to the Cbigger pictureD issues in evaluation and ne' thin&ing and trends in this field that are li&ely to be of great benefit to strengthening and improving C.D and the evaluation of C.D* In her overvie' of ne' thin&ing about ho' the evaluation of social change communication processes can be strengthened+ :yrne B"$$6a7 "E comments7 Considering gaps and biases in dominant practice has highlighted the value of systemic thin&ing+ comple)ity thin&ing Bincluding comple) adaptive systemsEF associated methodologies+ such as large-system action researchF and more participatory approaches+ among others* 8ome of these ne' trends and 'ays of thin&ing about research and evaluation and their value for the assessment of C.D programmes 'ill no' be considered*

* holistic, systems a

roach

% holistic perspective based on systems thin&ing and participatory approaches to 0+ 12( is increasingly seen as important to address comple) social problems B:urns+ "$$4F :yrne+ "$$6a and "$$6bF Gearn et al*+ "$$6E* his type of approach is becoming more 'idely adopted in the fields of planning and community development as a strategy to facilitate sustainable community and economic development BGearn et al+ "$$67 ,.E and has significant implications for the evaluation of C.D* 8everal of these implications have been identified by :yrne B"$$6a and "$$6bE and are discussed belo'* :urns B"$$47 !E argues that Ca holistic approach to intervention is crucial because comple) issues cannot be ade@uately comprehended in isolation from the 'ider system of 'hich they are partD* Go'ever+ he believes that it is not enough to see things holistically+ Ceffective 'hole system change has to be underpinned by processes of in-depth in@uiry+ multi-sta&eholder analysis+ e)perimental action and e)periential learning+ enacted across a 'ide terrainD* Ge ma&es a compelling case for the use of systemic action research to generate action that supports 'hole system change* 8ystemic thin&ing means CSta&ing into account the 'holeT+ and see&s meaning in the comple) patterning of interrelationships bet'een people and groups of peopleD* his Chighlights dynamics that are not al'ays visible through the scrutiny of individual interactionsD B:urns+ "$$47 "!E* :urns sees this as crucial because Coutcomes Bpositive or negativeE 'ill often have more to do 'ith the interrelationships bet'een interacting interventions than the effect of any individual action* %ction rarely impacts in a linear 'ayD B:urns+ "$$47 "!E* :yrne B"$$6a7 "E argues that Cstaying focused on the 'hole dra's attention to the deeper+ underlying dynamics of social changeD* "-

In contrast to linear+ reductionist approaches to policy ma&ing and policy implementation 'hich try to isolate variables+ a systemic perspective Ctries to understand the relationships bet'een the different elementsD and 'hat happens 'hen they combine B:urns+ "$$47 "6E* Gearn et al* B"$$67 ,3E point out that one of the implications of adopting a holistic vie' Cis a recognition that any e)planation of a phenomenon 'ill not be able to point to single causes and effectsD* ()amples of this ne' 'ay of thin&ing cited by Gearn et al* B"$$67 ,.E are the C'hole systemsD approaches to involving the community in sustainable planning and development advocated by =leari B"$$$E+ the Csustainable livelihoodD approach to rural development in 8outh %sia outlined by 0aAbhandari B"$$3E and 0ice and #ooteDs systems approach to communication campaigns for improving health conditions of young children in developing countries B0ice 2 #oote+ "$$!E*

Com le+ity thinking

he comple)ity of human systems is a &ey concept in ne' approaches to research and evaluation of development programmes* Lacayo B"$$37 ",E points out that7 Contrary to the cause-effect Ne'tonian paradigm+ comple)ity provides us 'ith the opportunity to loo& at problems 'ith multiple perspectives+ studying the micro and macro issues+ and understanding ho' they are interdependent* 8o+ instead of describing ho' systems should behave+ comple)ity science focuses the analysis on the interdependencies and interrelationships among its elements to describe ho' systems actually behave* :yrne B"$$-7 6E suggests that comple)ity thin&ing is very significant in the social change communication conte)t since Cit highlights the lin&s bet'een conte)t-specific social processes+ standards+ norms and values and+ therefore+ the danger of assuming replicability or scale-upD* Puntos de (ncuentro is a valuable case study of the potential and strengths of adopting a systems and comple)ity perspective* his initiative is a multi-media social change and feminist movement and organisation in Nicaragua 'hich 'as Cfounded BunconsciouslyE on principles of comple)ityD B:yrne+ "$$6b7 4E* In her case study of Puntos+ Lacayo B"$$37 ./E describes the difficulties that emerged in underta&ing the impact assessment of the proAect+ based on both donor re@uirements and their comple)ity science approach* 8he highlights the need for Ca supportive social+ political+ and economic environment *** %n environment that encourages alternative thin&ing+ construction of comple)ity-based indicators+ and evaluation methodologies that can test ne' theoretical propositions that e)plain rather than Austify+ understand rather than measure social change processesD* 8he goes on to highlight the benefits of comple)ity science for innovative C.D programmes such as Puntos that see& social change7 LItM can open our mind+ and help us to loo& for different 'ays to do thingsF to as& different @uestionsF to get different ans'ersF to try different strategiesF and to better understand 'hat does 'or& and 'hat doesnDt in each conte)t+ but most important+ ho' "6

and 'hy social change happens BLacayo+ "$$37 .-E* 0ogers B"$$67 "/E refers to a three-part typology - simple+ complicated or comple) J that has been found useful in planning and analysing evaluations and is beginning to be dra'n upon in the evaluation of development programmes* Lacayo e)plains that in this typology CComple)D refers to appropriately dynamic and emergent aspects of interventions+ 'hich are adaptive and responsive to emerging needs and opportunities* 8imple aspects of interventions can be tightly specified and are standardiHed - for e)ample+ a specific product+ techni@ue or process* Complicated aspects of interventions have multiple components+ are part of a larger multi-component intervention+ or 'or& differently as part of a larger causal pac&age+ for e)ample in particular implementation environments+ for particular types of participants+ or in conAunction 'ith another intervention* hese different aspects of interventions have significant implications for ho' interventions operate+ ho' 'e can understand them+ and ho' 'e can use this understanding* he use of this typology in an impact evaluation re@uires considering the multiple paths to achieving impacts and the role of the intervention in achieving the impacts+ in relation to the conte)t - i*e* 'hether the conte)t 'as favourable or other'ise B0ogers+ "$$6E* 'o ()pert Panel members for the 0esource Pac& proAect emphasised the value of comple)ity thin&ing for the evaluation of C.D proAects* =ne pointed out that this is a Ccrucial difference bet'een the thin&ing behind logical frame'or& analysis and the thin&ing behind =utcome 1appingD*3 (choing arguments made by :urns B"$$4E+ he also commented that Cit doesn;t ma&e sense to tal& about SbestT practices+ unless you are tal&ing about very simple conte)ts* C.D interventions are decidedly not SsimpleTD* :yrne B"$$-7 !!E points out significant implications of the &ey features of comple) adaptive systems BC%8E for evaluation* 8he cites (oyang and :er&as B!66-E 'ho state that7 hese principles+ consistent 'ith C%8 behaviour+ shift the focus+ tools and techni@ues of evaluation from the structures+ lo'-dimension+ predictable patterns of much of traditional research to more organic and fle)ible strategies* hey also provide more structure and pre-designed rationality than many of the individualistic and constructivist methods of @ualitative evaluation* :y including a 'ide range of approaches+ C%8 methods of evaluation integrate the best of many disciplines and methods that 'ere previously irreconcilable* % further implication is that the evaluation of programmes Cmust match the dynamics of the system to 'hich it is appliedD B(oyang 2 :er&as+ !66-+ cited in :yrne+ "$$-7 !$E* he comple)ity+ diversity and unpredictability of C.D processes and initiatives also has serious implications for the concept of Cone heory of ChangeD for these initiatives+ as :yrne B"$$6b7 3-4E suggests*

Partici atory a
3

roaches

>e discuss these approaches further in 8ection 4 of this report*

,$

Participatory forms of 0+ 12( have been used in the development conte)t for many decades and are 'idely ac&no'ledged as effective and appropriate in C.D programmes B:alit+ "$!$bF :yrne+ "$$6a+ "$$6bF Gearn et al+ "$$6F Lennie et al+ "$$6F Par&s et al*+ "$$/F Puddephatt et al*+ "$$6E* Indeed+ both the UN #ocal Points and the ()pert Panel 'e surveyed considered participatory approaches to 12( Cvery importantD for C.D+ 'ith 6$N of the UN #ocal Points and 46N of the ()pert Panel ma&ing this assessment* =ne of the ()pert Panel commented7 , think there is currently a groundswell of interest in alternatives to mainstream approaches ((( )hey offer numerous strengths and help to complement and balance some of the weaknesses of dominant& more traditional approaches to evaluation& in C%D as elsewhere( Chambers B"$$6a7 .E argues that C'e are living in a time of e)plosive innovation 'ith participatory methodologies+ including for monitoring and evaluationD* Ge points out that Cmuch of the *** creativity and innovation 'ith participatory methodologies has come and is coming from the 8outh+ from %sia+ from Latin %merica+ and+ notably+ from %fricaD* he ubi@uity and mainstreaming of participatory methods is noted by Leeu' and ?aessen B"$$67 ,!E 'ho suggest that CNo'adays+ participatory methods have become mainstream tools in development in almost every area of policy interventionD* Puddephatt et al* B"$$67 !$E highlight the importance of using participatory approaches to 12( of C.D programmes* hey comment that the tendency of commonly used approaches to evaluation to focus on @uantitative data and statistics Coften fails to provide the depth necessary for understanding more comple) C.D initiatives and does not al'ays allo' for other une)pected outcomesD* hey recommend that Cin the conte)t of promoting dialogue and building capacity to'ards community empo'erment and o'nership+ C.D initiatives should al'ays aim to include a level of participatory analysis BPuddephatt et al*+ "$$67 !$E % further rationale for using participatory approaches to evaluation is provided by :yrne B"$$6a7 ,E+ 'ho suggests that Cconte)ts of multiple actors and multiple+ diverse perspectives and types of &no'ledge call for participatory approachesD* In a similar vein+ Chambers B"$$6a7 .E e)plains that Cparticipatory approaches and methods fit in a paradigm that is pluralist+ evolutionary and iterativeD* %t a recent international conference4 on impact evaluation for development effectiveness+ Chambers B"$$6bE argued that participatory methods open up more possibilities for impact assessment+ in terms of creativity and improvisationF they resonate 'ith comple)ity science and can e)press local diversity* Ge also suggested that participatory methods open us up to learning and being in touch 'ith the community in 'ays that other methods donDt allo'* % &ey theme at this conference 'as that the development discourse recognises that many partners+ dra'n from different sectors+ contribute to the development of people and communities* Given this increasingly broad range of sta&eholders+ beneficiaries+ and multiple-goals+ greater consideration must be given to 'ays of including their voices in evaluations BLennie+ "$$6aE* ?aluing diversity and difference and ta&ing an inclusive approach to research and evaluation can also enable a more ade@uate understanding of problems and issues and can provide ne' insights and
4

his maAor conference 'as entitled7 -erspectives of ,mpact Evaluation: Approaches to Assessing Development Effectiveness* It 'as conducted in Cairo+ (gypt from "6 1arch J " %pril "$$6*

,!

understanding of other perspectives B1organ and 0amireH+ !6-.+ cited in Gearn et al*+ "$$67 "!"E* >illiams and Iman B"$$4E highlight the characteristics of comple)ity theory and systems thin&ing that have fundamental similarities to participatory monitoring and evaluation* hese include7 !* % shift in focus to interrelationships and processes rather than snapshots+ 'hich seriously challenges dominant linear e)planations of systemic phenomena* "* %n understanding of development as comple)+ emergent and transformative* ,* % shift to the bigger picture and interconnections+ 'ith much focus on boundaries and the values they reflect Bcited in :yrne+ "$$-7 6E* he numerous benefits of participatory 0+ 12( approaches and methodologies that have been identified include7 #le)ibility of the process and responsiveness to changes in the research and organisational conte)ts* Can increase utilisation of evaluation results and recommendations* 8een as a 'ay to ensure the @uality of an evaluation* Can facilitate better decision ma&ing+ programme improvement and sustainability* Provide an effective 'ay of strengthening sta&eholder and staff capacities in 12(* Can foster a sense of o'nership of the evaluation process and the initiative being evaluated* Can generate mutual trust and understanding bet'een participants and development of a shared vision and shared understanding of programme obAectives* Can create more e@ual partnerships bet'een participants+ and through the use of democratic and inclusive processes+ can produce various forms of empo'erment* BDiaHPuente et al+ "$$-F #etterman 2 >andersman+ "$$/F #orss et al*+ "$$3F Gibbs et al*+ "$$6F Leeu' 2 ?aessen+ "$$6F Lennie+ "$$/F 1ayou) 2 Chambers+ "$$/F Papineau 2 5iely+ !663F Par&s et al*+ "$$/E* Go'ever+ from positivist perspectives+ @uestions are still raised about the rigour and validity of these approaches BChambers+ "$$-F Par&s et al*+ "$$/E* Participatory research is essentially a challenge to positivist research paradigms since it is built upon the ideas of democratic practice and transformative relationships BGall+ !66,E* %nalysis of the online surveys 'e conducted for the 0esource Pac& proAect identified as &ey challenges the lac& of openness of funders and commissioners Cabout 'hat constitute appropriate+ UrigorousU and UvalidU methodologies and methodsD and a lac& of understanding or appreciation of the ne'er+ more participatory approaches to 0+ 12(+ based on comple)ity theory and 'hole systems approaches*

>hile participatory approaches to 12( are particularly 'ell-suited to C.D programmes+ they may appear to cost more than non-participatory approaches and the political 'ill to invest in this is often 'ea& or absent BPar&s et al*+ "$$/7 !,E* here are also issues 'ith the ,"

dominance of @uantitative approaches and the entrenched use of tools such as the logframe+ 'hich are seen by some as incompatible 'ith alternative+ participatory approaches to evaluation B(arle+ "$$"E* herefore+ 'e 'ould suggest that there is a need to ta&e a longterm vie' of the 0*+ 12( process and the benefits of adopting a participatory+ mi)ed methods approach*

Mi+ed methods a

roach to R, M3E

Johnson and =n'uegbuHie B"$$.7 !4E suggest that a mi)ed methods approach to research is the Cthird 'aveD or Cthird research movementD+ 'hich Cmoves past the paradigm 'ars by offering a logical and practical alternativeD* hey define mi)ed methods research as Cthe class of research 'here the researcher mi)es or combines @uantitative and @ualitative research techni@ues+ methods+ approaches+ concepts or language into a single studyD BJohnson 2 =n'uegbuHie+ "$$.7 !4E* 5ey features of mi)ed methods research 'hich they identify include7 uses the pragmatic method and system of philosophy uses induction+ deduction+ and abduction Buncovering and relying on the best of a set of e)planations for understanding resultsE it is an e)pansive and creative form of research it is inclusive+ pluralistic+ and complementary+ and suggests an eclectic approach to method selection and thin&ing about and conducting research the research @uestion is most fundamental J research methods should follow research @uestions in a 'ay that offers the best chance to obtain useful ans'ers BJohnson 2 =n'uegbuHie+ "$$.7 !4-!-E*

%rguments for the use of multiple methods and triangulation have been put for'ard for @uite some time+ Cfrom convergentIdiscriminant matri) and unobtrusive research methods to triangulationD BGearn et al+ "$$67 ,6E* riangulation is the process of combining multiple methods and perspectives 'ith various types of data sources in order to cross-chec& the results of 0+ 12(* It can also mean using several different researchers or evaluators BPatton+ "$$"E* :amberger et al* B"$$37 "4E suggest that triangulation offers the follo'ing advantages7 !* Increases the validity of conclusions by providing t'o or more independent estimates of &ey indicators* "* Permits the researcherIevaluator to dra' on the 'idest possible range of research and evaluation methods and tools* ,* Permits a deeper and richer analysis and interpretation of the programmeDs results and its conte)t+ lending greater rigor to the research process* :amberger et al* B"$!$E ma&e a compelling case for using mi)ed methods in the monitoring and evaluation of international development programmes* In their criti@ue of @uantitatively oriented development economists and researchers+ :amberger et al* B"$!$7 "E point out that7 :y restricting themselves to the econometric analysis of survey data+ development economists are bo)ed into a Cartesian trap7 the @uestions they as& are constrained ,,

by the limitations inherent in the process by 'hich @uantitative data from closedended @uestions in surveys are collected B0ao 2 >oolcoc&+ "$$,E* %s such+ they are limited in their ability to as& important @uestions about the social+ cultural and political conte)t 'ithin 'hich development problems are embedded *** % related criticism *** is that many &inds of econometric analysis fail to e)amine 'hat actually happens during the process of proAect implementation *** and conse@uently are unable to determine the e)tent to 'hich failure to achieve intended impacts is due to Sdesign failureT or to Simplementation failureT* In other 'ords+ their research @uestions are being shaped by their data instead of their data by the @uestions* % strong case can be made that such @uestions re@uire a more eclectic approach to data+ one that mi)es participation+ observation+ the analysis of te)t-based information *** free-ranging open-ended intervie's 'ith &ey informants and focus groups+ and other such types of information that can be loosely called S@ualitative dataT* >hile :amberger et al* B"$!$E are referring to the international development conte)t in general+ their arguments can e@ually be applied to the limitations that result from a heavy reliance on @uantitative survey data in the evaluation of many C.D programmes* :alit B"$!$b7 !E points out that @uantitative approaches predominate in evaluating C.D programmes Csince decision ma&ers in development institutions are usually hard scientists and demand evidence of results and cost effectiveness based on numbers and statisticsD* #indings from our survey sho'ed that @uantitative survey-based methodologies 'ere one of the most fre@uently used methodologies in the evaluation of C.D programmes 'ithin the agencies of our UN respondents* (ighty percent of UN respondents indicated that @uantitative survey-based methodologies 'ere CoftenD used by their agency+ along 'ith various @ualitative+ participatory and mi)ed methods approaches* :amberger et al* highlight a number of challenges that face evaluators 'ho 'ant to use a mi)ed methods approach in international development proAects7 % first challenge is the fact that mi)ed methods have been the evaluation design of choice for many development agencies for many years* Go'ever+ many of these evaluations used some'hat ad hoc approaches and most do not apply the &inds of methodological and conceptual rigor that is re@uired by academic Aournals such as the Journal of 1i)ed 1ethod 0esearch* 8o the mi)ed method approach is not ne' per se+ but the professional+ financial and other resources have usually not been available to increase methodological rigor B:amberger et al*+ "$!$7 ",E % pragmatic+ mi)ed methods approach to social research and evaluation often results in superior research compared 'ith mono-method research B:amberger et al*+ "$!$F Greene+ "$$"F Greene 2 Caracelli+ "$$"F Johnson 2 =n'uegbuHie+ "$$.E* % &ey theme at a recent international conference on impact evaluation for development effectiveness 'as the need to use fle)ible+ multi-disciplinary frame'or&s and methods that enable people to learn from each other+ along 'ith a culturally appropriate+ mi)ed methods approach BLennie+ "$$6aE* In their boo& Action "esearch and !ew Media+ Gearn et al* B"$$67 .$E point out that a &ey feature of the action research approach is its methodological pluralism* Go'ever+ they e)plain that7 >hat brings different approaches together in a meaningful 'ay is the guidance offered by the principles 'e have outlined here in formulating the in@uiry and the ,.

design process* In particular+ the guiding underlying principle is that action research is al'ays cyclical+ 'ith all action able to be evaluated* hat is+ action research learns from itself* It is increasingly ac&no'ledged that+ among many benefits+ an appropriate combination of complementary methods 'ill shed light on different issues and increase the strength and rigour of evaluation and impact assessment findings B:amberger et al*+ "$!$F :yrne+ "$$6aF Leeu' 2 ?aessen+ "$$6F Lennie+ "$$3F Puddephatt et al*+ "$$6F >hite+ "$$6E* :yrne B"$$6a7 ,E calls for approaches to the evaluation of social change communication programmes that adopt an Cethos of complementarities and triangulationD+ recognising that different approaches are suitable for different issuesD* Puddephatt et al* B"$$6F !"E highlight the value of using a mi)ed methods approach in the evaluation of C.D programmes7 >here more traditional+ @uantitative evaluation techni@ues fail to appreciate the increasingly comple) nature of many development initiatives+ LparticipatoryM methods focus on innovative 'ays to assess less tangible outcomes alongside the principal obAective and often use more @ualitative analyses* here is some difference of opinion as to the value of @uantitative against @ualitative data+ but increasingly there are calls for an appropriate combination of both* In terms of statistical evidence and securing funding from donors there is still a preference for ShardT data produced from standard+ @uantitative evaluation techni@ues* %s 'ith the approaches to C.D themselves+ these modes of 12( should not be seen as mutually e)clusive+ nor as rigidly defined in every case+ rather as a complementary set of methods that can be adapted and 'hen used in the right combination can provide a suitable strategy for pragmatic evaluation and clear reporting* % good e)ample of the need to use a mi)ed methods approach 'as identified in a revie' of published studies of C.D impacts 'hich found that Cthe evaluations using @uantitative methods and those investigating vertical communication strategies did not+ or failed to+ capture proAect failures caused by po'er ine@ualityD BInaga&i+ "$$47 ./E* >e outline a range of other benefits of adopting a mi)ed methods approach in 8ection 4 of this report* In evaluating C.D+ 'e need to &eep in mind that different C.D approaches and interventions 'ill have different 0+ 12( re@uirements*

!ocus on o'er relations, gender and social norms

1any contemporary and participatory approaches to 0+ 12( openly ac&no'ledge and ta&e into account the gender and po'er relations that are an inherent part of social interactions and organisations+ as 'ell as the political nature of research and evaluation practices B:urns+ "$$4F Gosling 2 (d'ards+ "$$,F Gearn et al+ "$$6F Lennie+ "$$/+ "$$6bF 1artin+ !663F 1ertens 2 Chilisa+ "$$6F acchi et al*+ "$!$E* :urns B"$$47 ,6E ma&es the important point that Cchange emerges from the spaces in bet'een+ in the interrelationships and in the discussion+ and it is mediated by comple) po'er relationsD* In a recent international 'or&shop on mi)ed methods and social transformation+ 1ertens and Chilisa B"$$6E argued that 'e need to recognise po'er differentials bet'een those involved in evaluations and to as& C>hose version of reality is being privilegedVD hey suggested that using mi)ed methods allo's dialogue to begin the research process* his 'ould include as&ing @uestion such as7 >ho do I need to involveV+ Go' do I involve themV >hose values do I representV hese

,/

issues clearly have significant implications for the 'ay in 'hich 0+ 12( of C.D is conducted and the e)tent to 'hich the processes used are inclusive and empo'ering or other'ise* =ne of the implications of this approach is that+ rather than ta&e an idealistic vie' that assumes that all participants in a participatory 12( proAect are the same and e@ual+ it is more useful to openly ac&no'ledge the differences bet'een those involved+ particularly those related to gender+ po'er and &no'ledge BGearn et al*+ "$$67 ,4E* Lennie B"$$/7 .!$E found that a participatory evaluation capacity building proAect she evaluated had a number of unintended and disempo'ering impacts due to Cine@ualities in po'er and &no'ledge+ the different values and agendas of the participants and researchers+ the pre-e)isting relationships and net'or&s 'ithin the communities+ and other comple) issuesD* In addition+ a revie' of published studies on the impact of C.D initiatives found that Cthe issue of po'er is a common cause of unsuccessful outcomes in these interventionsF po'er imbalances in political+ economic+ occupational and gender domains created bloc&ages to communication across social boundariesD BInaga&i+ "$$47 .$E* Gosling and (d'ards B"$$,7 ,,E point out that Cthe full participation of 'omen is a fundamental principle for any development programme* Go'ever+ this principle is often difficult to put into practice because of ine@ualities bet'een genders in many cultures and societiesD* Clearly+ the monitoring and evaluation of C.D and other development programmes re@uires a high level of a'areness of gender issues+ given that the improvement of 'omenDs status Cis essential if 'e are to move the 'orld to'ards a better life for all individualsD B1ongella+ !66/7 !"!E* Indeed+ successive UN conferences have Crepeatedly articulated the pivotal role and needs of 'omenD B1ongella+ !66/7 !"!E and many of the 1DGs are aimed at improving the 'ellbeing and opportunities of 'omen and girls* Go'ever+ Corn'all B"$$$7 !E argues for the need to rethin& the concept of CgenderD and more directly address Cthe issues of po'er and po'erlessness that lie at the heart of both Gender and Development BG%DE and participatory developmentD* #oucaultDs conceptualisation of po'er has underpinned a considerable amount of contemporary systems thin&ing B:urns+ "$$47 ,3E and has been dra'n on by many social science and feminist researchers BLennie+ "$$6bE* his approach sees po'er as Cconstantly in motion+ multi-directional and systemic in patterningD B:urns+ "$$47 ,3E* #rom this perspective+ po'er is something that e)ists in action+ in a net'or& of interconnected relations* It is enacted in everyday social practices+ rather than 'ielded by po'erful groups such as corporations or large institutions* #oucaultDs 'or& sho's Cho' obAects of &no'ledge are not natural+ but are ordered or constructed by discourses 'hich determine 'hat is Sseeable and sayableTD BJennings 2 Graham+ !6637 !4!E* his po'er-&no'ledge ne)us highlights the po'er relations that are enacted in all interactions+ 'hether those involved have an emancipatory intent or other'ise BLennie+ "$$6bE* In this frame'or&+ po'er is intimately connected to &no'ledge+ including the technical &no'ledge of specialists and the tacit &no'ledge of community members or 'or&ers BGearn et al*+ "$$6E* In this conceptualisation+ po'er is seen as embodied in social hegemonies that constitute social norms J Cthe attitudes and behaviours that people regard as normal for their peer groupD B:urns+ "$$47 ,/E* :urns B"$$47 ,3E highlights the urgent need to Cradically refocus attention on the importance of local social normsD+ and suggests that if interventions do not attend to local social norms+ Cmany policy initiatives 'ill fail to 'in community support+ ,3

rendering them unsustainableD* his has maAor implications for C.D programmes that aim to change harmful social and cultural practices such as child marriage and female genital mutilation and prevent the spread of significant health problems such as GI?I%ID8* :urns B"$$47 ,--,6E suggests that 'hile top-do'n interventions focussed on public health issues have had very mi)ed results+ Csystemic action research offers the opportunity to develop bottom up interventions in relation to local social normsD*

Develo ment of learning cultures

In the current era of rapid change+ evaluation is increasingly seen as an ongoing learning process and an important means of strengthening capacity and improving organisational performance B1orariu et al*+ "$$6F Gorton et al*+ "$$,7 4E* his is due to the need for people and organisations to engage in ongoing learning and to adapt to changing conditions* It is no' recognised that the process of participating in an evaluation can often result in positive changes to an organisation+ including to its capacity+ processes and culture BDiaH-Puente et al*+ "$$-F Gorton et al*+ "$$,F Patton+ !66-E* % CpedagogicalD approach to evaluation entails a teaching and learning process+ one that is Cmore about learning than AudgingF more about participants becoming critically a'are of their o'n positions on issues and developing an understanding and appreciation of ne' and different perspectives* P his learning process is made possible by dialogues of several &indsD B8ch'andt+ "$$!+ in :yrne+ "$$-7 !.-!/E* 8outer B"$$-7 !-!E suggests that impact assessment+ Cabove all *** re@uires an honest and self-critical approach and a learning culture 'ith good and trusting relations bet'een partnersD* Li&e'ise+ Gosling and (d'ards B"$$,7 "$E suggest that one ingredient for a successful evaluation 'ith a specific implementing partner is that Cboth the implementer and funder are Clearning organisationsD* Go'ever+ Crac&nell B"$$$E and (arle B"$$"E have identified basic incompatibilities bet'een the aims of accountability and lesson learning from evaluation* In addition+ the development of a learning culture re@uires the active support+ leadership and involvement of senior management 'ithin organisations+ an openness to critical reflection and learning from the things that did not 'or&+ then putting these learnings into practice B#orss et al+ "$$3F aut+ "$$4E* Go'ever+ moving to this &ind of learning culture in the current conte)t of results-based management is seen by some as problematic BChambers 2 Petit+ "$$.F (arle+ "$$"E*

* shift from = roving> to =im roving>

8an&ar and >illiams B"$$47 !E ma&e the important point that the increasing emphasis on CprovingD the impact of programmes

can undermine and sometimes even distract from program delivery efforts+ 'here a focus on CimprovingD could be more meaningful* It is not easy to design evaluations that both CproveD and CimproveD* %lso can an overemphasis on impact limit options for innovation and gradual achievement of sustainable resultsV

,4

>hen the evaluation of development initiatives is underpinned by a holistic perspective based on meaningful participation+ critical reflection and learning+ there is a shift a'ay from Cmeasuring and Sproving.+ to'ards understanding and improving$& as :yrne B"$$-7 6E points out* In this conte)t+ progress towards long/term social change and the contribution being made is increasingly considered a more realistic measure of effectiveness B:yrne+ "$$-7 6E* his shift from proving to improving is e)emplified by the emergence of relatively ne' planning and 12( methodologies such as =utcome 1apping+ 'hich focuses on constant improvement+ understanding and the creation of &no'ledge+ rather than on proving+ reporting and ta&ing credit for results B(arl et al*+ "$$!E* 8imilarly+ 1ayou) and Chambers B"$$/7 "4,E e)plain that the ne' impact assessment agenda of pro-poor development has moved from a focus on Cproving impactD to Cimproving practiceD* his means that Csimple SrigorousT measurement of before and after situations for random samples 'ith control groups is no' rarely sufficient* It re@uires producing more credible practical recommendations and thin&ing about ho' they can be implemented+ that is+ the policy and practical impact of impact assessment itselfD* his implies ne' @uestions to include the priorities of very poor people+ loo&ing in detail at differences bet'een the e)perience of specific groups of poor people and crucially going from @uestions about 'hat is happening to 'hom+ to @uestions of causality and attribution and the implications for future change* 1oreover the ne' agenda re@uires not only ne' @uestions+ but ne' processes and methods because poor people themselves are no' central actors at all stages of the assessment process Bp*"4.E* 1ayou) and Chambers+ B"$$/7 "6"E outline the &ey principles of the ne' paradigm of impact assessment7 Prioritises the voices+ vie's and interests of poor 'omen and men+ particularly the poorest and most vulnerable* Involves these people throughout the process of impact assessment from indicators+ to representation in sampling to analysis and recommendations* (nsures that the vulnerability of those most vulnerable is not increased* Increases the s&ills+ &no'ledge and net'or&s of poor people and communities as part of the assessment process*

hese principles and others 'ere dra'n on to identify principles for effective+ appropriate and sustainable 0+ 12( of C.D 'hich 'ere set out in 8ection " of this report*

Conclusion
here is a need to consider bigger picture issues and ne' thin&ing and trends in 0+ 12( and impact assessment that can strengthen and improve both C.D initiatives and the evaluation of C.D* % holistic perspective based on comple)ity and systems thin&ing and participatory approaches to 0+ 12( 'as seen as important to address the type of comple) social problems that C.D aims to address* his is crucial because these issues cannot be understood in isolation from the 'ider conte)t and system of 'hich they are part* Comple)ity ,-

thin&ing is very significant for C.D since it highlights the lin&s bet'een conte)t-specific social processes+ norms and values and can help us loo& at things differently and better understand ho' and 'hy social change happens* Participatory approaches to 0+ 12( have been sho'n+ over many decades+ to be very appropriate and effective for C.D* Go'ever+ the political 'ill to invest in these approaches is often 'ea& or absent+ since they tend to be perceived as too time consuming and costly* % long-term perspective is re@uired in relation to the use of participatory methodologies+ given their numerous benefits+ including fle)ibility of the process+ increased o'nership of the evaluation+ better utilisation of evaluation results and recommendations+ and strengthened evaluation capacities* % pragmatic+ mi)ed methods approach to research and evaluation 'as argued to often result in superior research* %n appropriate combination of complementary methods can shed light on different issues and increase the strength and rigour of evaluation and impact assessment findings* Go'ever+ research and evaluation needs to consider the gender and po'er relations that are inherent in all social interactions and organisations and to attend to local social norms in order to ma&e interventions more successful and sustainable* (valuation is increasingly seen as an ongoing learning process and an important means of strengthening capacity and improving organisational performance* 8uccessful evaluation is associated 'ith the development of learning organisations* his re@uires the active support and leadership of senior management+ and an organisational culture that is open to critical reflection and learning from things that did not 'or&+ and then puts these learnings into practice* >hen these alternative approaches to evaluation are used+ there is a shift a'ay from measuring and proving impacts+ to'ards better understanding and improving programmes* his ne' approach re@uires impact assessment to include the development of practical recommendations and effective 'ays of implementing them+ ta&ing the vie's+ needs and interests of community members+ especially the poor and vulnerable+ into account*

,6

68 Challenges, issues and strategies

/ummary of key findings

he many 'ider conte)tual+ structural and communication-related challenges+ issues and barriers related to C.D and 0+ 12( of C.D re@uire more consideration* here are also significant country level and institutional challenges+ including lac& of coordination bet'een head office and field staff+ and confusion about the meaning of C.D* he assumptions+ attitudes and policies of funders and management can result in lac& of appreciation+ funding and support for C.D and innovative evaluation practices+ and problems 'ith effectively applying participatory processes* Numerous challenges 'ere identified in conceptualising+ managing and planning 0+ 12( of C.D+ including lac& of time+ funding and resources+ lo' levels of capacity+ 'ea& planning and design+ and the dominance of @uantitative methodologies* 8imilar challenges 'ere identified in assessing the impacts of C.D* =ther challenges for impact assessment included demonstrating and attributing impacts+ unrealistic demands+ targets and time frames+ poor baseline data+ and facilitating sta&eholder participation in all stages of the evaluation* 8trategies to overcome these challenges included7 advocacy 'ith UN and other organisations and donors+ identifying creative and innovative 0+ 12( approaches and e)amples+ providing sufficient budgets and time+ long-term capacity development and institutionalisation+ enhancing sta&eholder engagement+ and encouraging open sharing of all findings*

Introduction

In this section 'e outline some of the many comple) challenges and issues surrounding the evaluation of C.D and suggest some strategies that aim to overcome these challenges and issues*- %s 'ell as revie'ing relevant literature+ 'e have e)tensively dra'n on our consultations and findings from a survey of UN #ocal Points and ()pert Panel members 'ho collaborated in this proAect*

4ider conte+tual and structural challenges and issues

-

=ur consultations indicated that the challenges set out in this section are not Aust C.D related+ they are across the board and also apply to areas such as early childhood development*

.$

1any comple) social+ economic+ political+ cultural+ environmental and technological factors and barriers Bincluding issues related to gender ine@uality and po'er relationsE affect the sustainability and success of C.D and other development programmes and the evaluation of these programmes* :alit B"$!$a7 3E points out that both development and communication are basically political and this is 'hy Cpolitical 'ill to put into practice on the part of governments and local authorities is often lac&ing* %fter all+ enabling poor communities to participate directly challenges e)isting po'er structuresD* :yrne B"$$6a7 /E argues that Cimpacts of the 'ider environment in 'hich any evaluation ta&es place need to be better appreciatedD* 8imilarly+ Inaga&i B"$$47 ..E emphasises the need for Cacademic research in development communication *** to engage more fully 'ith larger structural issues that may not be ade@uately addressed in proAect evaluations conducted as part of individual assistance proAectsD* Ge notes the e)istence of Ca peculiar disAuncture bet'een empirical research and critical theoretical discourses+ at least among the empirical studies published in the mainstream development studies AournalsD and comments that 8uch a disAuncture may put the legitimacy of the scholarship into @uestion* If reduced to purely instrumental research interests+ academic research 'ill lose its authenticity as a voice @uestioning fundamental development problems such as racism+ structural poverty+ political economy of malnutrition and diseases+ commercial e)ploitation of indigenous and natural resources+ and international conflicts BInaga&i+ "$$47 ..E* %nother factor is that many C.D and Information and Communications for Development BICDE initiatives are implemented in an information and communication conte)t 'hich is changing very rapidly* 8outer B"$$-7 !46E states that

by the time that a BsayE three-year initiative has been completed+ the technology originally used has become obsolete+ or at least 'ould no longer be the most appropriate for deployment *** %t the same time+ changes in the availability and use of IC s 'ithin target communities are most li&ely to suggest a very different approach to facilitating access and use than occurred at proAect initiation*

?arious issues related to access to communication and information technologies clearly affect the outcomes of C.D programmes and their evaluation+ especially if methods are used that employ IC s such as online surveys or multimedia* In addition+ in many developing countries there are specific problems associated 'ith geographic+ communication and cultural barriers and local political issues that can significantly affect communication among sta&eholders and evaluators and travel to research sites+ ma&ing field research and data collection more time consuming and difficult* #or e)ample+ Lennie et al* B"$$6E have described maAor communication and travel problems that arose in the four year Assessing Communication for Social Change proAect in Nepal that involved developing a participatory impact assessment methodology for C.D programmes* hese problems 'ere due to the 'ide cultural and linguistic diversity in the country+ internet access being fairly limited outside the 5athmandu ?alley+ and the countryDs high mountain .!

terrain and poor roads* In addition+ ongoing political instability and discontent in the country fre@uently involved stri&es that included disruptions to the transport net'or&* hese communication and travel problems greatly affected field research 'or& and capacity development activities conducted as part of the proAect+ 'hich involved a net'or& of community researchers and 12( staff in (@ual %ccess Nepal+ a development communication NG=*

Country and institutional level challenges

In a recent paper+ :alit outlines a number of significant obstacles that affect the development and implementation of the UNDs recently developed C.D advocacy strategy+ including its 12( and capacity development strategies* 8he points out that the concept of C.D is a social process based on dialogue+ it is a Csoft and social science that has to do 'ith listening+ building trust and respecting local cultures - not easy concepts to understand for policy ma&ers and programme managers 'ith a bac&ground in hard sciencesD B:alit+ "$!$a7 .E* %s 'e noted in 8ection .+ this means that @uantitative approaches predominate* Go'ever+ the problem is that Ccounting and hard data cannot truly capture the comple)ity of social change processes over longer periods of timeD B:alit+ "$!$b7 "E* =ur consultations found that there is an emphasis in some UN agencies on the use of @uantitative approaches 'hich are unli&ely to provide the most meaningful and useful data on C.D impacts* =ne of the ()pert Panel also noted that7 A key issue underlying the challenges and difficulties is that the M#E of C%D 0like much other development1 is typically approached in a vertical& non/integrated manner& rather than being an integral part of programmes( An add on& for 2M#E e'perts2( )his reinforces the tendency towards top/down& 2e'pert driven2 approaches and actively works against participatory approaches 0skills for which the former do not typically have1( % further obstacle identified by :alit is that many C.D units are still located in corporate communication and e)ternal relations departments* >hile some policy ma&ers understand the advantage of using C.D Cthey tend to be interested only in producing messages to disseminate information and tell people 'hat to do+ rather than listening and giving people a voiceD* :alit suggests that Celiminating the confusion bet'een communication for development+ publicity and corporate communication is essential for the correct understanding of the disciplineD B:alit+ "$!$a7 /E*

:alit also points out that

Countries that foster dialogue+ debate and inclusion 'hile encouraging free and open media are more li&ely to engage in participatory communication practices than more centrally controlled countries and authoritarian governments* he 'hole notion of good governance+ transparency and accountability is political and depends on the 'illingness of those in po'er to share &no'ledge and information 'ith citiHens and civil society B:alit+ "$!$a7 3E*

."

%gain+ this has significant implications for the effective use of participatory evaluation methods that are more congruent 'ith the underlying ethos of C.D* 1any organisational challenges also affect the sustainability and effectiveness of 0+ 12( of C.D* =ne of the challenges in planning and conducting 0+ 12( of C.D programmes identified by an ()pert Panel member 'as C he lac& of co-ordination bet'een central G9 policy staff 'ho 'ant evaluations and field staff for 'hom evaluation is an irritationD* Puddephatt et al* B"$$6E also highlight the need for greater coordination of C.D programmes and continual strengthening of country capacity* =ur consultations emphasised the need for a long-term+ sustained focus on capacity development in 0+ 12( for staff at all levels* Go'ever+ our consultations also suggested that 'ithout the understanding+ funding+ support and commitment of senior UN managers and donors+ improvements to capacity and moves to'ards greater use of more innovative and participatory approaches and methods are li&ely to be less successful*

*ttitudes and olicies of funders and management

%s 'e have already noted+ a &ey challenge identified in our online survey 'as that senior managers and funders 'ere seen as lac&ing an appreciation of the value and importance of C.D and 0+ 12( and often did not support more innovative or participatory approaches* =ne ()pert Panel respondent listed the follo'ing as a &ey challenge in planning and conducting 0+ 12( of C.D programmes7 )he assumptions and biases of funders3those commissioning research and evaluation& combined with a lack of openness to less mainstream& more innovative& less prescriptive and predictable approaches( 4oth conceptually and in terms of resourcing these processes& an un5uestioning 2more of the same2 is all too commonplace& regardless of the suitability and fit with the aims of and values underlying the particular programme involved( %s 'e highlight belo'+ such assumptions and attitudes appear to be reflected in the lac& of ade@uate funding and resources provided for 0+ 12( of C.D* :yrne B"$$-7 .E highlights difficulties 'ith funding innovative evaluation practice in the C.D area and the frustrations of many at the field level 'ith having to fit their achievements into e)ternally imposed C81%0 D obAectives and indicator tables li&e logframes* GuiAt B"$$4E suggests that CIn practical terms+ donors need to rethin& the principles on 'hich they base their models of evaluation and learningD Bcited by :yrne+ "$$-7 3E* :alit also points to the problem of applying participatory processes 'ithin the rigid timeframes of logframes and results-based management7 CParticipatory processes 'ill upset the 'ell defined plan* Donors 'ant @uic& results* hus it is easier to implement an information campaign than develop a long-term communication process 'ith the local peopleD B:alit+ "$!$a7 4E* 8outer B"$$-7 !-!E argues that impact assessment of ICD programmes re@uires Csustained commitment on the part of implementing agencies+ from proAect design through to proAect .,

completion and beyond$* Ge suggests that donors also need to understand and be 'illing to recognise Cthat une)pected and even negative impacts need to be identified and understoodF and that impact assessment is not about validation of past decisions but about the improvement of those that 'ill be made in futureD B8outer+ "$$-7 !-!E* =ne of the challenges identified by Puddephatt et al* B"$$6E 'as the tendency for 0+12( studies to be published that report on successful initiatives rather than those 'hich have been less successful but could provide valuable learnings+ and a lac& of reporting on the long-term effects of communication programmes*

Challenges in conce tualising, managing and lanning R, M3E of C-D

=ur online survey identified a 'ide range of challenges in conceptualising+ managing and planning 0+ 12( of C.D programmes+ some of 'hich have already been identified in this literature revie'*6 #ive of the UN #ocal Points and !" of the ()pert Panel listed challenges in these areas*!$ 0esults from these survey responses are summarised belo'* Lack of sufficient funding and resources for R, M&E: #our UN respondents listed as either their second or third most important challenge Clo' priority of funding for 0+ 12(D+ Cappropriate fundingD+ Cinade@uate resources for 12(D or Clac& of resources+ 'hich could be addressed partially by applying a SpercentageT system to all programme sectorsD* en ()pert Panel respondents listed lac& of funding or resources for various aspects of 0+ 12( as a challenge* =ne respondent nominated CInsufficient time and budget for research+ monitoring and evaluationD as their most important challenge+ si) listed this as their second most important challenge+ three said it 'as their third most important challenge+ 'hile one listed it as their fourth most important challenge* Comments on this from the ()pert Panel included7 "esources needed for research& if available 0which they are usually not1 would be disproportionate to the scale of the pro6ect3programme( nder resourcing the effort& e'pecting impact results from what is really 6ust 7a drop in the ocean. case study( )he second most important challenge is to convince the contractor that it also takes money to do it well( A fi'ed percentage of budgets should be allocated to research& monitoring and evaluation from the design phase& instead of adding the activity at the end and looking for funding when the pro6ect or programme funds are already e'hausted( his issue appears to be related to the lac& of appreciation and understanding of the value and importance of C.D and 0+ 12(* %nother factor is that participatory approaches to 0+ 12( can ta&e more time and resources to do 'ell compared 'ith other approaches* Leeu' and ?aessen B"$$67 ,"E comment that Cin general+ the higher degree of participation+ the more costly and difficult it is to set up the impact evaluationD* Go'ever+ if issues of capacity
6

he survey for UN #ocal Points as&ed respondents to name+ in order of importance+ 'ith the most important first+ up to four challenges that they face in conceptualising+ outsourcing and managing 0+ 12( in their C.D programmes* % similar @uestion 'as as&ed in the survey for ()pert Panel members about challenges that they face in planning and conducting 0+ 12( of C.D programmes*
!$

'elve of the ()pert Panel respondents named one challenge+ !! named t'o challenges+ nine named three challenges+ and seven named four challenges*

..

development+ local o'nership and sustainability could be costed or factored in+ participatory approaches 'ould be seen to offer value for money as 'ell as having other strengths*

Low capacity, skills or awareness7 %ll of the UN #ocal Points 'ho responded to this @uestion listed challenges related to lo' levels of capacity+ understanding or a'areness or Climitation of staffDs s&illsD* hey 'ere listed as either the first or second most important challenge in conceptualising+ outsourcing and managing 0+ 12( of C.D programmes* 'o respondents also listed lac& of Cavailability of s&illed resource personsD 'ithin their agency 'hile another listed as her second most important challenge7 CUneven understanding of behaviour and social changeD* 'o respondents to the ()pert Panel survey listed 'ea& e)pertise or capacity as their second most important challenge 'hile another listed C#e' s&illed practitioners in many countries to conduct 0+12(D as their third most important challenge* Comments on this 'ere7

+eak or non/e'istent e'pertise of personnel within the national agency0ies1 with whom the ! agency is working(

+eak capacity for research and evaluation& especially at organisational levels& and inade5uate resources to strengthen capacity at all levels& over a realistic timeframe(

Problems with objecti es, indicators, lack of baseline data and results e!pected7 8i) ()pert Panel respondents listed problems in these areas 'hile one of the UN #ocal Points named C>ea& design of indicators+ baseline information+ and conceptual approach to assessing impact at start of implementationD as their second top challenge* he top challenges for t'o ()pert Panel respondents 'ere7 CLac& of clarity about obAectives for the commissioned research+ monitoring and evaluationD+ and the Cdiffuse+ long-term and hard-tomeasure results e)pected from our proAects and programmesD* he second most important challenge of another respondent 'as7 CIndicators for some aspects of C.D programming are not 'ell defined or understoodD+ 'hile another listed as their fourth most important challenge indicators being Ctoo difficult for field or local staff to applyD* % further challenge is that a Cculture of measurable indicators and baseline information is not 'idespread in the C.D NG= sectorD and that Cbaseline data are lac&ing and often C.D components are not programmed e)plicitly 'hich limit their evaluabilityD*

Lack of importance of R, M&E7 'o of the ()pert Panel respondents named a lac& of appreciation of the value and importance of 0+ 12( as the most important challenge* Comments on this 'ere7

Convincing decision/makers and pro6ect managers that "& M#E of C%D is important( ./

8ow level of realisation among partners of importance and value of "ME for C%D(

In addition+ one of the UN #ocal Points listed the follo'ing as their fourth most important challenge7

8ack of interest among programme staff& governments& other stakeholders in activities and use of evaluation results( "nsufficient time7 % related issue 'as a lac& of appreciation of the time re@uired for effective 0+ 12( and pressure to CproveD results 'ithin a certain timeframe* =ne UN respondent listed her most important challenge as C#inding the time to design evaluations for diverse programmes+ 'here each re@uires specialised analysisD + 'hile anotherDs third most important challenge 'as7 C ime constraints vis-W-vis proAect durationD* In addition+ an ()pert Panel respondent commented7 C he most important challenge is convincing the contractor that research+ monitoring and evaluation are processes that ta&e time+ and they are not Aust @uic& mechanical operationsD* #or t'o other respondents+ lac& of time to Cget good baseline dataD and to Censure effective follo'-up+ over time+ to help ensure that capacity is developed in sustainable 'ays at individual+ institutional and collective levelsD 'as their fourth most important challenge* %nother commented that C12( is very time- and resource-intensive+ for everyone involvedD* Poor planning or lack of long#term planning: this is related to 'ea& e)pertise or capacity in conceptualising+ managing and planning 0+ 12( of C.D programmes+ and lo' levels of understanding and a'areness of effective planning of 0+ 12(* hree ()pert Panel respondents listed problems 'ith planning as a challenge* =ne nominated CplanningD as the top challenge since CC.D is often an organic process that follo's opportunities 'hich means the evaluation needs to fle)ible enough to move 'ith a program as it develops 'ithout missing opportunities for baselinesD* <et this is not ho' evaluations are typically approached or designed* =thers nominated CLac& of long-term planningD and CPoor C.D planning 'ith unclear resultsD as their second or fourth most important challenge*

Lack of understanding of $%&7 'o ()pert Panel respondents listed this as their most important challenge* =ne said there 'as a CPoor understanding of C.D concepts+ its role and ade@uate use in programmesD+ 'hile the other thought there 'as a CLac& of understanding of C.D by agencies commissioning research+ monitoring and evaluationD* 'ttitudes to methods and problems with the e aluation process7 =ne UN respondent listed as her top challenge7 C(mphasis in the organisation on @uantitative methodologies and methodsD* In addition+ three ()pert Panel respondents identified various problems and issues related to evaluation methodologies+ methods and processes* Challenges listed included7

.3

)o convince the contractor that 5uantitative methodologies will not provide the necessary information on how peoples$ lives changed( 9nly 5ualitative methodologies which allow people to participate and speak can provide 5uality information about social change( )he apparent obsession with methods and tools& to the neglect of deeper& fundamental 5uestions like: +ho is the evaluation for: +hat is it for: +ho are the intended users of the evaluation: +hat are the intended uses: ;ow will the process itself empower those involved and strengthen wider communication for development processes:

)oo much 6argon and mystification of the process& lack of simplicity and lack of clarity about what is being evaluated( =ther challenges in conceptualising+ managing and planning 0+ 12( of C.D programmes identified in the surveys included7 Poor documentation of C.D impact through 0+ 12( in reports+ case studies* Lac& of institutional or corporate guidance* %ppropriate follo'-up and accountability* Insufficient support for communication offices* 1ostly G9s based and donor driven* Permanent rotation of personnel*

Challenges in assessing the im acts of C-D

Crac&nell B"$$$7 ",4E points out that Cdespite the obvious need for them+ impact studies are still not all that common* his is because they absorb a lot of resources+ ta&e time to implement+ and need to be conducted in full cooperation 'ith the beneficiary country 'here the proAect is locatedD* >hile the number of impact evaluations in the development sector has increased since then+ our literature revie' indicates that there is a lac& of published reports on high @uality impact assessments of C.D* % maAor study by Inaga&i B"$$47 .,E found that Cthe volume of empirical research on the impact of communication for development is not as large as one might e)pectD 'hile :alit B"$!$a7 !E comments that Conly a limited number of LC.DM interventions have been ade@uately monitored and evaluatedD* 8ome of the difficulties in demonstrating the impacts of C.D programmes 'ere aptly summarised by an ()pert Panel respondent7 ,mpact is a holy grail& it re5uires considerable funding and effort to gain credible results because communication impact is challenging( ,t is not counting latrines that have been built& it is about assessing changes in how people think and respond to issues and conte'ts and this can be impacted by many variables(

.4

)he challenge of attri"ution %s 'e have already indicated+ there are many problems and issues 'ith demonstrating and attributing impacts of C.D programmes compared 'ith other development initiatives 'here it can be easier to isolate individual changes in the environment* 8outer B"$$-7 !4/E suggests that Can open-minded approach Lis neededM to analysis and interpretation of findings+ particularly to issues of attribution+ aimed at learning from e)perienceD* :alit B"$!$b7 !E notes that Ccause and communication effect are difficult to measure+ and one of the most serious obstacles to evaluating communication activitiesD* >hile some outcomes such as Cempo'ermentD and CdialogueD can be difficult to define and measure+ :alit B"$!$b7!E suggests that C in some cases it may be possible to measure changes in &no'ledge+ behaviour+ attitudes and access and use of servicesD* #our ()pert Panel respondents listed various challenges associated 'ith demonstrating the impact of C.D programmes+ attribution or assessment of impact not being Cprovided forD* =ne commented7 C he impact of improving media is a 'idely-held assumption but very hard to demonstrateD* %nother respondent listed Cdifficulty of attributing impact in a comple) 'orldD as his third most important challenge* % further comment on this issue 'as7 Much of our work is to do with improving processes 0eg 5uality of relationship between 6ournalists and CS9s1( ,mpact of this is likely to be improvement in other processes 0eg 5uality of 6ournalists< research1( 8outer B"$$-7 !3"E e)plains that the challenge of attribution

stems from the comple)ity of causality *** In practice+ change may 'ell result from a number of factors+ either contributing independently or acting upon one another* Different factors in causality maybe more or less important at different times during the proAect cycle*

his problem is difficult because of the politics of aid 'hich means that implementing agencies are Coften tempted to claim credit for impacts because that is 'hat those they are accountable to 'ant to hearD B8outer+ "$$-7 !3"E* his issue is emphasised in the study by Inaga&i B"$$4E 'hich found that the maAority of evaluations of C.D programmes that they revie'ed reported positive impacts* Go'ever+ as 'e have noted+ it is often more important to understand negative and une)pected impacts and 'hat has not been achieved as this contributes to learning and improvement* 8uch &no'ledge also allo's Cpractitioners and researchers to empirically identify sources of failureD and can Cfeed bac& into more effective planning and implementation of proAects in the futureD BInaga&i+ "$$47 ,6E* Leeu' and ?aessen B"$$67 ,"E also point out that >ithin the light of the attribution issue+ sta&eholder perspectives can help improve an evaluatorDs understanding of the comple) reality surrounding causal relationships among interventions and outcomes and impacts* In addition+ insight into the multiple and BpotentiallyE contrasting assumptions about causal relationships bet'een an

.-

intervention and processes of change can help enrich an evaluatorDs perspective on the attribution issue* Par&s et al* B"$$/7 "$E suggest that Cthe initial C#8C evaluation challenge is to determine ho' best to measure short-term increases in communication capacity+ and in turn+ ho' to attribute this increasing capacity to broad+ longer-term social change and more narro'ly defined improvements in GI?I%ID8 prevention and careD* (ime constraints, unreasonable demands and the )longitudinal* problem 8outer B"$$-7 !3.E argues that+ in relation to the ClongitudinalD problem+ CIn many 'ays this is the most difficult of all challenges for impact assessmentD* 8ocial change impacts often ta&e a long time to occur* Go'ever+ our consultations identified that there are often unrealistic demands+ targets and time frames for the impact assessment process and donors often 'ant to see results in an unreasonably short time frame*

ime constraints 'ere listed by t'o UN respondents as either their first or second most important challenge in assessing C.D impacts+ 'hile another listed CDonors 'anting to see results in an unreasonably short time frameD as her third most important challenge* In addition+ t'o ()pert Panel respondents listed unrealistic demands or targets as their third most important challenge* =ne commented7 CIf donors demand SresultsT+ it is hard to lin& improvements in public &no'ledge and discourse 'ith actual changes on the groundD 'hile the other challenge 'as CImprecise results and unrealistic targets and dubious indicators Bdata can;t be collectedE planned at the beginningD*

hree ()pert Panel respondents also mentioned issues 'ith unrealistic time frames* =ne commented on7

,nade5uate time and resources to do 6ustice to effort e'pended& to achievements and to learning potential and interest( )his typically accounts for limited follow/up opportunities& including those for collective critical reflection and learning& which should lie at the heart of evaluation processes(

% related challenge is that Curgency to implement displaces good 12(D* 8outer B"$$-7 !3.E ma&es the important point that Cif an intervention leads to Slasting and sustainable changeT+ the maAority of that change must still lie in the future 'hen proAect managers are closing the accounts and 'riting up their notesD* Go'ever+ impact assessment is usually underta&en immediately after the end of proAect implementation* :alit B"$!$b7!E points out that Cdonors 'ant visible results 'ithin a relatively short timeframe B, to / yearsE of programmes+ 'hile communication processes ta&e much longer to achieve* Inaga&i B"$$47 .!E found that only four of the ,4 published studies on the impact of C.D programmes offered any type of insights into the long-term impacts of communication interventions+ and .6

even among these studies impacts going beyond the immediate timeframe of the proAect are discussed through anecdotal accounts rather than systematic analysesD* =ne of the factors identified by Inaga&i B"$$47 .!E 'as that 1ost of the proAect implementation schedules are too short if one tries to gauge longterm impacts during or 'ithin the timeframe of the proAects* he average length of the proAects evaluated in the revie'ed studies is t'o years+ and the active proAect period in a little over half of these proAects had lapsed in one year or less* Par&s et al* B"$$/E suggest that assessment of the impact of C#8C programmes should loo& at short-term+ intermediary and long-term impact* >hile 8&use B"$$37 "/E suggests that understanding the behavioural impact of radio programmes is Cnotoriously difficult and can only occur over the long-termD+ he argues that Cthere is scope to set interim behaviour change indicators 'ithin ICD programmes that can and should be evaluatedD* 8outer B"$$-7 !3.E suggests that the best 'ay of assessing Slasting and sustainable changeT is to use longitudinal studies Cunderta&en some time Bsi) months+ t'o years+ five yearsE after proAect closureD* Go'ever+ he notes that the reluctance of donors to fund such studies is a particular problem in areas li&e ICD C'here there is no strongly established evidence base of past e)perience on 'hich to buildD B8outer+ "$$-7 !3.E* Inade?uate funding and resources

% &ey challenge identified by our online surveys 'as inade@uate funding and resources+ including time to underta&e impact assessment of C.D programmes* #our UN respondents listed as either their second or third most important challenge7 Clo' priority of funding for 0+ 12(D+ Cappropriate fundingD+ Cinade@uate resources for 12(D or Clac& of resources+ 'hich could be addressed partially by applying a SpercentageT system to all programme sectorsD* Lac& of budget 'as also the top challenge for three ()pert Panel respondents and the second top for another*

In ICD conte)ts+ 8&use B"$$37 "/E points out that Cformal impact evaluation places significant transaction costs on ICD practitioners+ especially at the community level 'here financial and technical resources are fe'D* Go'ever+ he suggests that Cthe challenge for donors is to ma&e evaluation neither comple) nor costly+ and support the development of easy to use tools that 'ill allo' broadcasters to @uic&ly and easily assess the impact of their outputsD B8&use+ "$$37 "3E* )he challenge of conte+t and artici ation %s 'e have previously suggested+ a bottom-up approach+ 'hich implies a holistic+ participatory approach to impact assessment+ highlights the importance of ta&ing the social+ economic+ cultural and political conte)t of C.D programmes and their evaluation into account* Par&s et al* B"$$/E suggest that the results of C#8C programmes must go beyond individual behaviour and consider social norms+ current policies+ culture and the general development conte)t BPar&s et al*+ "$$/7 .E* /$

he participation of proAect users at all stages of the cycle is increasingly considered crucial to the design+ implementation+ monitoring and evaluation and impact assessment of development interventions BGosling 2 (d'ards+ "$$,F Leeu' 2 ?aessen+ "$$6F Puddephatt et al*+ "$$6F 8outer+ "$$-E* Gosling and (d'ards B"$$,7 !"3E argue that Cit is important that impact assessment is embedded in all stages of the programme spiralD* Go'ever+ there is a need to strengthen an understanding of the conte)t of C.D proAects Cincluding the relationship bet'een the intervention under consideration and other interventions underta&en by themselves or other development actorsD B8outer+ "$$-7 !4/E* %s Puddephatt et al* B"$$67 !,E highlight+ it is important that C.D specialists Band other sta&eholdersE be involved at every stage of the development process+ from the conception through to the evaluation of an initiative in order to integrate fully the communication process into the development frame'or&* It is essential to ta&e positive steps in this direction in order to both demonstrate the value communication can add to development systems on a global scale and to embed its principles 'ithin the UN operating frame'or&* Li&e'ise+ Crac&nell B"$$$7 ,.$E suggests that Cthe &ey issue is to ensure full participation of all the sta&eholders right from the start of the proAectDs life *** unless the primary sta&eholders are fully involved+ progress could be slo' and success problematicD*

1ther key challenges and issues

=ther &ey challenges and issues in assessing the impacts of C.D programmes identified in our online survey and in the literature revie' include7 Lack of planning and foresight7 'o ()pert Panel respondents listed the follo'ing issues related to lac& of planning and foresight as their second most important challenge7 Evaluation is not really conceptuali=ed at the beginning of programmes( ,t is not foreseen that beneficiaries participate in planning and implementing research& M#E( (he comple!ity of social change: 8ince social change is a comple)+ uncertain+ unpredictable and long-term process+ change often needs to be assessed against a moving baseline and is rarely continuous* Understanding change re@uires Can in-depth understanding of both conte)t and the baseline against 'hich change is being measuredD B8outer+ "$$-7 !3!E* >e discuss this issues further in 8ection 4* (he challenge of the baseline: Gosling and (d'ards state that baseline studies are @uite rare and 'here they have been carried out Cthey have often loo&ed at factors 'hich seemed important initially but have become less relevant as the programme has developedD BGosling 2 (d'ards+ "$$,7 !.$E* his highlights the need for the relevance of baseline data to be periodically revie'ed* 8outer B"$$-7 !3"E points out that C'ithout some sort of baseline it is not really possible to assess impact at allD* >hile baseline data can be recovered retrospectively+ Cthese are poor substitutes for genuine baseline data ac@uired at the proper timeD* Go'ever+ as our consultations suggest+ baseline data is often @uite poor or absent in /!

C.D proAects* Indeed+ only .$N of the UN e)perts surveyed as part of the proAect conducted evaluation via Cbaseline+ formative and summative studiesD* "ssues with the indicator setting process and the selection of appropriate indicators have been identified by 8outer B"$$-E and Par&s et al* B"$$/E as some of the fundamental methodological challenges associated 'ith the measurement of ICD and C#8C* Par&s et al* B"$$/7 !4E argue that selecting indicators is one of the most difficult steps in setting up a P12( approach as it Chighlights+ more than any other+ the different information needs and e)pectations that the different sta&eholders have of the monitoring 'or&D* Challenges identified by Par&s et al* B"$$/7 "$-"!E include7 >hat is the optimum methodology for developing C#8C indicatorsV >ho sets the criteria for indicator developmentV >ho is involved and 'ho decided 'hat type of information 'ill be collected and ho' it 'ill be usedV % lac& of a good @uality evidence base for impacts of ICD and C.D programmes ma&es it more difficult to decide 'hat outcomes might be e)pected and 'hat indicators may be useful to enable assessment of une)pected outcomes B8outer+ "$$-7 !4-E* % particular challenge is to Cdistinguish bet'een indicators 'hich are primarily technological and those 'hich are primarily concerned 'ith attitudes and behaviourD B8outer+ "$$-7 !4-E* Par&s et al* B"$$/E suggest that as 'ell as C'hat indicatorsD 'e need to as& C'ho should develop and use these indicatorsVD* Go'ever+ this is often overloo&ed* 1ayou) and Chambers B"$$/7 "4-E also raise concerns that 81%0 indicators may Ccompletely miss the most significant @uestions for respondentsD* % &ey concern 'ith indicators is that they are typically imposed from above* Go'ever+ there are e)amples 'hich demonstrate the benefits of actively involving local people in the development of indicators B#ontalvo-GeraHo et al*+ "$$4F 0eed et al*+ "$$3E* >e discuss indicators of C.D impacts further in 8ection - of this report* (he challenge of aggregation and scaling up: 8outer B"$$-7 !3"E points out that CIn practice *** it may not be individual interventions at all to 'hich change should be attributed+ but the cumulation of a group of interventions 'hich interact 'ith one another+ so that their collective impact is more substantial than the sum of their impact as individual interventions* Ge suggests that it may sometimes be easier Cto establish the aggregate impact of changes in the information and communication technology BIC E environment than the impact of specific components in that aggregate changeD B8outer+ "$$-7 !4-E* his attribution challenge is notable in the C.D field+ 'here the impact of a communication initiative cannot be considered in isolation from 'ider conte)tual factors that 'ill inevitably have an impact* % related problem identified by Par&s et al* B"$$/7 "$E is7 ho' best can micro-level data for participatory 12( be generalised and used to inform national and macro-level strategies and policiesV In response to this issue+ Lennie et al* B"$$-E have suggested that national initiatives must adapt the C#8C approach to e)trapolate findings to the broader conte)t and that this could be done through community-based research in representative case study sites* (he challenge of disaggregation: to be effective+ impact evaluations need to disaggregate data by gender or other categories such as different levels of income+ education+ age+ ethnicity or caste* Go'ever+ as 8outer B"$$-7 !3,E points out+ it is often difficult for interventions to reach the most marginalised groups such as the very poor so Cparticular attention therefore also needs to be given to identifying the e)periences of the most marginalised* he capacity to monitor+ evaluate and assess impact 'ith this level of disaggregation needs to be built into proAect design from the start+ including baseline dataD* /"

his emphasises the need for effective planning of impact assessments and the establishment of good @uality systems for the collection+ organisation and analysis of useful monitoring and evaluation data from the very beginning of programmes* (he need to focus on both primary audiences and wider communities: C.D interventions do not only impact target audiences J their impact can also be on other communities and social groups+ either directly or indirectly* 8o impact assessment needs to also consider ho' the intervention affects the lives of 'ider communities and groups B8outer+ "$$-7 !3,E* %lso+ it is sometimes difficult to specify a primary audience 'hen mediums such as radio and television are used to influence social change*

(he challenge of the une!pected: :yrne B"$$6a7 ,E suggests that a &ey concept in ne' thin&ing about ho' C.D programmes are evaluated is to Csee& the une)pected+ 'hich can emerge to be the most significantD* Li&e'ise+ 8outer B"$$-7 !3.E points out that all development interventions have une)pected outcomes+ because

it is never possible to predict everything that is li&ely to happen in the future *** hey may be positive or negative+ in terms of proAect obAectives or the overall development needs of target communities *** Une)pected outcomes may turn out to be more significant than anticipated outcomes *** Impact assessment needs to pic& them up+ to be honest about them+ and to learn from the e)perience ho' to capitalise on the opportunities of the positive and avoid the problems of the negative*

(he challenge of capturing di erse perceptions: he importance of using multiple methods and ensuring the inclusion of a range of sta&eholders in an impact assessment is highlighted by 8outer 'ho points out that since different sta&eholders are li&ely to have different perceptions of impact Cit is important for impact assessment to capture the diversity of e)perience and perceptions around an interventionD B8outer+ "$$-7 !3.E* :yrne suggests that Cconte)ts of multiple actors and multiple+ diverse perspectives and types of &no'ledge call for participatory approachesD* Go'ever+ she notes that the participation of diverse sta&eholders Cinvolves diverse and at times contradictory perspectivesD B:yrne+ "$$6a7 ,E* +eed for stronger capacity and skills: here is a lac& of a'areness and &no'ledge of impact assessment and Cthe practical application of different methodologiesDF a lac& of e)pertise+ ongoing learning+ guidance and e)amples* 5ey challenges identified by Par&s et al* B"$$/7 "$E are7 Go' to add or integrate P12( Cinto conventional 12( systems already challenged by data @uality and resource constraintsD* >hat type of capacity building in P12( is needed+ for 'hom+ and at 'hat levelV Go' 'ill P12( capacity be maintainedV :alit B"$!$b7 /E comments on the need for in-service training of C.D staff to develop ne' s&ills+ including s&ills in Cparticipatory situation analysis+ @ualitative research methods+ group facilitation+ group dynamics and conflict resolutionD* 1ayou) and Chambers B"$$/7 "-"E also note the need for high level s&ills in Cfacilitating the participatory process in a balanced+ e@uitable and ethical mannerD* >ell facilitated participatory research also enables Ca more systematic discussion of priorities and trade-offs+ cross-chec&ing 'ith many individuals+ /,

rather than imposing e)ternal interpretationD Bp*"-"E* >e discuss evaluation capacity development in more detail in 8ection 3* =ther challenges to effective impact assessment identified in our consultations include7 Problems 'ith the evaluation design or methods used* Issues 'ith poorly defined outcomes and the challenges of ensuring rigour+ 'hatever methods are used* % preference for @ualitative or @uantitative methods* =vercoming assumptions about and the dominance of C@uantitative evaluationD approaches* he challenge of estimating the cost and impact of 'hat did not happen* %s :alit notes+ there is a need to highlight Cthe immense cost of not investing in communicationD B:alit+ "$!$b7 4E*

here are also problems 'ith poor documentation of C.D impact through 0+ 12( in reports and case studies and capacity to effectively disseminate impact assessment findings to various sta&eholders needs to be strengthened B8outer+ "$$-E* he outcomes of our literature revie' and consultations clearly demonstrate that there are many significant challenges+ issues and barriers that affect the planning and implementation of evaluation and impact assessment of C.D* his research suggests that a more fle)ible+ realistic and sustainable approach is re@uired in relation to the timeframe+ resources and approaches used in the evaluation of C.D+ and in strengthening evaluation capacity at all levels* he follo'ing section outlines some strategies that could help to address these comple) challenges and issues*

1vercoming the challenges

1embers of the ()pert Panel and the UN #ocal Points identified a number of strategies that could be used to overcome some of the challenges outlined above* 8imilar strategies 'ere also identified in previous bac&ground papers for the !!th UN 0ound able on C.D* *dvocacy 'ith organisations, donors and others :oth the ()pert Panel and the UN #ocal Points suggested underta&ing advocacy 'ith UN and other organisations and donors to highlight the importance of C.D and 0+ 12( and develop a greater appreciation of C.DDs contribution to achieving strategic results* :alit B"$!$b7 /E argues that C'hat is really missing are high level communication planners and managers7 the &ind of professional 'ho can advocate for the discipline 'ith senior decision ma&ersD* In terms of the best advocacy strategy+ :alit B"$!$b7 3E suggests that as 'ell as using a mi) of approaches such as case studies and a position paper or boo&let+ a multimedia approach should be used since Cseeing is believing+ and decision ma&ers do not have much time to readD* /.

% further advocacy strategy+ suggested at the Ne' <or& consultation meetings+ 'as to establish lin&s 'ith high profile thin&ers+ academics+ thin& tan&s and high profile CbloggersD* It 'as felt that lin&s could also be made 'ith aid critics 'ho 'ant development assistance to be more accountable+ since there could be a philosophical overlap bet'een these critics and C.D as a participatory method of achieving development Bsee Ne' <or& consultation meeting report+ Pamer at al*+ "$!!7 6E* Identify creative and innovative a roaches and e+am les

=ther survey respondents suggested a greater focus on innovative+ Cnon-dominantD approaches and e)perimentation* :yrne B"$$6a7 /E suggests that 'e urgently need Cmore honest and reflective stories of innovation in practice+ in social change+ communication for social change and their evaluation+ from across the 'orld and across the development sector* he Nicaraguan initiative Puntos de (ncuentro+ 'hich 'as outlined in 8ection .+ is a good case study of innovation in this field* Li&e'ise+ :alit B"$!$b7 3E suggests that Ccreative and innovative e)amples from the past /$ years Lcan be selectedM to illustrate 'hat 'or&s best and sho' resultsD* It is also important to include e)amples of proAects that have failed due to lac& of a C.D element* Digital story telling through participatory content creation has been found to be a po'erful 'ay for development programme participants to tell their stories in their o'n voices and to facilitate the process of social change BGearn et al*+ "$$6F >at&ins 2 acchi+ "$$-E and is li&ely to be another effective means of advocating to donors and senior management* Provide sufficient funding and time he above gaps highlight another &ey issue raised by nearly all of those 'e consulted - that is the need for donors and programme implementers to provide sufficient budgets and time for 0+ 12( of C.D proAects+ including for longitudinal studies of impact* :alit B"$!$b7 !$E highlights the importance of advocating 'ith donors Cto foresee sufficient time and resources for research and monitoring and evaluation efforts+ especially 'hen dealing 'ith participatory approaches for social changeD* %nother advocacy strategy suggested at the Ne' <or& consultation meetings 'as to produce short information sheets on C.D activities+ 'ith 012( evidence+ and to promote these activities as 'orth'hile investments* %nalysis of cost effectiveness 'ould reveal the benefits of incorporating C.D versus not incorporating it* 8uch evidence can help to convince UN Country 0epresentatives and Deputy 0epresentatives to set aside fi)ed percentages of programme budgets for C.D and for 0+ 12( Bsee report of Ne' <or& consultation meeting+ Pamer et al*+ "$!!7 6E* .ong-term ca acity develo ment =ur consultations and literature revie' emphasise the need for long-term capacity development in 0+ 12( for UN staff and their partners at all levels of organisations+ particularly at the country level* >e have suggested that this process should particularly focus on staff involved in planning+ designing and implementing 0+ 12(+ and their government and NG= partners* %s 'e noted in the previous section+ participatory

//

approaches to 12( are particularly good at strengthening capacities in this area BPar&s et al*+ "$$/F Lennie et al*+ "$!$E* his is discussed further in 8ection 3* 8ome respondents also suggested that more attention should be given to understanding Cthe fundamentals of evaluationD+ providing Cpractical guidanceD and simpler information on evaluation+ and sta&eholder engagement in interpretation* %t the consultation meetings in Ne' <or&+ the follo'ing suggestions 'ere also made in relation to capacity development7 Institutionalisation of ca acity develo ment of C-D R, M3E7 %s a first step7 ()amine the different cultures of 0+ 12( across agencies* 8ystematise 0+ 12( practices across agencies* Develop materials e)plaining 'hat developing capacity for C.D and 0+ 12( can do for a particular programme or agency* he materials should focus on demystifying C.D and providing basic s&ills to staff* ()isting resources for developing capacity for 12( outside of C.D such as the >orld :an& resource on system diagnosis and building effective 12( systems and UNIC(# publications on country-led 12( systems and real 'orld evaluation should be resourced Bsee :amberger et al*+ "$$3F 1ac5ay+ "$$4F 8egone+ "$$6E*

Ca acity develo ment in C-D and R, M3E of C-D o strengthen understanding of C.D in some agencies+ future cooperation should focus on general C.D capacity development 'ith specifics on capacity development in 0+ 12( to come later* o increase understanding about C.D+ UN agency leaders at the decision ma&ing level should receive concise information on 'hat it is and 'hy they should invest in its use* Practitioners re@uire information e)plaining the bac&ground of and Austification for C.D that reiterates the difference bet'een C.D and corporate communications and public relations* 1aterials for practitioners also should include simple ans'ers to their most common @uestions and provide capacity development resources 'ith methods and Cho' toDs*D C.D 0+ 12( should also be included in broader 0+ 12( 'or& at the country level to help build capacity* Create a mechanism for practitioners to as& an e)pert+ for e)ample through establishing an online community of practice* his 'as seen as a &ey to capacity strengthening BPamer et al*+ "$!!7 !$-!!E*

Colla"orative research % further strategy identified by 8am :ic&el+ =fficer in-Charge of UNIC(#Ds (valuation =ffice+ in his official closing remar&s to the Ne' <or& consultation meetings 'as to underta&e collaborative research on pilot programmes+ since he considered that Aoint evaluations are of higher @uality than those conducted by UNIC(# Bor other UN agenciesE alone* 8uch

/3

collaboration 'as also seen as creating an opportunity to e)change ideas on programming and 12( Bsee Pamer et al*+ "$!!7 !/E*

*ddressing the challenges at different rogramme stages

=ther 'ays of addressing the challenges listed above at various stages in the programme cycle+ 'hich have been suggested by 8outer B"$$-E and others are outlined belo'* Design stage

8outer B"$$-7 !46E suggests using the follo'ing strategies at the design stage7

Conducting an information and communications audit Bassessment of the communications environmentE at the proAect design stage* Giving attention to sta&eholder mapping of initiatives to gain input throughout the monitoring period* his should Csignificantly help to ensure full sta&eholder engagement in the final impact assessment processD* (stablishing indicators and baselines and disaggregation of data bet'een different social groups 'ith particular attention on very poor or marginalised groups*

Participants in the Ne' <or& consultations thought that (thnographic %ction 0esearch B(%0E 'as an important approach to needs assessment that fits 'ell 'ith formative research and the systems approach advocated in this report* >e describe (%0 in 8ection 4 of this report* Par&s et al* B"$$/E also suggest that combinations of locally-generated measurements and P12( processes 'ith e)ternally derived indicators and 12( approaches are at times the most appropriate 'ay of monitoring and evaluating C#8C initiatives*

Im lementation stage

8outer B"$$-7 !46E advises using the follo'ing strategies at the implementation stage7

Conducting Ccontinuous monitoring of the communications environment affecting an intervention in order to adAust implementation to changing needsD* %dopt a rolling baseline 'here necessary+ i*e* adAusting the baseline Cat various time points 'ithin the implementation phase in order to recognise the scale of changes 'hich have occurred in the communications environmentD

/4

*nalysis and inter retation

In terms of analysis and interpretation of data+ 8outer B"$$-7 !-$E suggests the follo'ing strategies7

a&ing care to Cdistinguish bet'een technological and behavioural change *** o accommodate the long-term nature of impact+ it may be necessary to use attitudinal indicators as pro)ies for subse@uent behavioural change Balthough this techni@ue re@uires cautionED* Dra'ing on the evidence of proAect participants and other sta&eholders+ since this C'ill be crucial to understanding 'hat is going on *** it may be particularly important to understanding impacts on the very poor and most marginalisedD* Dra'ing on Ce)pertise and e)perience in comparable interventions *** provided *** that LtheyM have been assessed 'ith sufficient rigourD* Underta&ing Cadditional longitudinal studies or trac&er studies *** the @uality of insight 'hich they can contribute to understanding *** has very high potential valueD*

%s 'e discuss further in 8ection 4+ =utcome 1apping and the 1ost 8ignificant Change techni@ue are particularly useful methodologies for engaging sta&eholders in this phase*

Use of meta-evaluation

During our consultations in Ne' <or&+ meta-evaluation 'as seen by one UN participant as a very po'erful 'ay of identifying and framing issues that emerge from the evaluation of development programmes* It can also be an important means of improving evaluations and increasing the utilisation of evaluation results* he concept of meta-evaluation is used in different 'ays by evaluators+ researchers and policy ma&ers* >hile it is seen by some as an aggregation of evaluation findings across a number of studies+ others+ such as Patton B!664E+ see it as a process of evaluating evaluation studies based on the professionDs standards and principles J this can be seen as an approach to @uality control for a single evaluation study BUusi&yla 2 ?irtanen+ "$$$7 /!E*

Uusi&yla 2 ?irtanen argue that meta-evaluation should also be concerned 'ith the utilisation of evaluation results* hey suggest that Cthe continuous and refle)ive interpretation of

/-

evaluation findings is the only 'ay to enhance organisational learning and thus increase the utilisation of evaluation resultsD BUusi&yla 2 ?irtanen+ "$$$7 /"E* %n ongoing meta-evaluation of the %C.8C proAect 'as found to have many benefits+ particularly7 increased evaluation s&ills+ &no'ledge and capacity+ including improved critical reflection and revie' s&illsF development of ne' &no'ledge and learnings+ including about the comple) organisational+ social and cultural conte)t of the proAect and the effects of such conte)tual factors on the outcomes of the proAectF forming effective collaborative relationships 'ith participants in the proAectF development of an impact assessment methodology+ and 12( systems that are li&ely to be practical and sustainable and more useful than if 'e had not underta&en our meta-evaluation of the proAect BLennie et al*+ "$!$7 4E* 1 en sharing of all findings

8outer B"$$-7 !-!E also suggests that there is a need for

8tronger collaboration bet'een donors and implementing agencies to share impact assessment findings J positive and negative+ intended and une)pected J in a more open-minded fashion* Donors and implementing agencies might also do more to share e)perience of impact assessment methodologies*

8imilarly+ one of the ()pert Panel respondents noted the need for Copenness to admitting and learning from 'ea&nesses and SfailuresTD 8he suggested that

+hat these highlight is the importance of transparency and integrity,honesty throughout research and evaluation processes( )he critical point is not to deny or seek to hide weaknesses but& rather& to learn from them and for those involved to document how they are attempting to do this(

>e discuss the importance of developing open and effective communication and feedbac& systems and processes in order to share 12( findings and enhance the success of participatory 12( in C.D organisations in 8ection 3*

Conclusion

/6

1any comple) conte)tual+ institutional and methodological factors and barriers affect the sustainability and success of C.D and its evaluation that need to be better understood and appreciated* 8pecific problems associated 'ith geographic+ communication and cultural barriers and local political issues can significantly affect the outcomes of 0+ 12( of C.D in certain developing countries* In addition+ greater a'areness is needed of the many country level and institutional challenges+ including the predominance of @uantitative approaches+ lac& of coordination bet'een head office and field staff involved in evaluations+ and confusion bet'een C.D+ publicity and corporate communication* =ur research suggests that the assumptions+ attitudes and policies of funders and management can result in a lac& of appreciation+ funding and support for C.D and innovative evaluation practices+ problems 'ith effectively applying participatory processes+ and lac& of learning from evaluations* >e identified a 'ide range of challenges in conceptualising+ managing and planning 0+ 12( of C.D+ including lac& of time+ funding and resources+ lo' levels of evaluation capacity+ 'ea& planning and design of 0+ 12(+ and the dominance of @uantitative methodologies* 8imilar challenges 'ere identified in assessing the impacts of C.D* %dditional challenges and issues for impact assessment included7 demonstrating and attributing impacts+ problems 'ith indicator development+ donors setting unrealistic demands+ targets and time frames+ facilitating sta&eholder participation in all stages of the evaluation cycle+ poor baseline data+ and the need to see& une)pected and negative outcomes+ as 'ell as e)pected and positive outcomes* % range of strategies to overcome these challenges 'ere identified* hey included7 %dvocacy 'ith UN and other organisations and donors to highlight the importance of C.D and 0+ 12( and to develop a greater appreciation of C.DDs contribution to results* Identifying creative and innovative 0+ 12( approaches and e)amples to illustrate 'hat 'or&s best and to demonstrate the results of C.D* Providing sufficient budgets and time+ including for longitudinal studies* Long-term capacity development for staff at all levels and institutionalisation of capacity development* Using various strategies to enhance sta&eholder engagement* Underta&ing meta-evaluations to identify and frame issues+ improve the @uality of evaluations and (CD+ and increase the utilisation of evaluation results* =pen sharing of positive and negative+ intended and une)pected findings*

% more fle)ible+ realistic and sustainable approach is clearly re@uired in relation to the timeframe+ resources and methodologies used in the evaluation of C.D+ and in strengthening evaluation capacity to enable more effective evaluation and impact assessment of C.D*

3$

78 Evaluation ca acity develo ment in C-D

/ummary of key findings
here is a significant need to strengthen capacity in C.D and 0+ 12( at all levels and to ta&e a long-term approach to evaluation capacity development B(CDE* Institutionalising evaluation and developing an evaluation culture in organisations is e)pected to generate more high @uality 12( of C.D and to improve C.D design and outcomes* Professional evaluation associations and net'or&s+ universities and research institutions can play important roles in (CD* Developing+ implementing and sustaining (CD can present challenges and issues for time+ s&ill and resource-poor organisations in developing countries* %dopting a holistic+ participatory+ learning-oriented approach to managing+ implementing and improving (CD can be very effective* Developing good 'or&ing relationships bet'een those involved is crucial for success* Particular challenges and issues that can affect the effectiveness and sustainability of (CD in the C.D and development conte)t include7 the comple)ity of assessing the impact of C.D+ the diversity of C.D approaches B'hich affects (CD needsE+ the need for practical+ fle)ible and sustainable C.D impact assessment frame'or&s+ and the range of s&ills re@uired* =ther (CD challenges and issues include7 the need for a readiness for organisational learning+ po'er and conflict issues+ language+ literacy and cultural issues+ and developing effective data collection and management and communication and feedbac& systems* Learnings about increasing the effectiveness of (CD in the C.D conte)t include7 designing (CD that is fle)ible and open to change+ actively demonstrating the value of 12( to programme staff+ and &eeping evaluation methodologies practical and simple*

Introduction
% &ey theme in this report is that developing and strengthening evaluation capacity is an important priority in the C.D area* 8trengthening capacity in both C.D and 12( 'ere identified as &ey needs in our consultations+ in the bac&ground paper by Puddephatt et al* B"$$6E+ and in the report of the !!th UN 0ound able on C.D by UNDP B"$$6bE* =ne of the &ey challenges identified by participants in the !!th 0ound able 'as Cthe lac& of a'areness+ understanding and capacity of 12( in C.DD BUNDP+ "$$6b7 ",E* Participants put for'ard various strategies to strengthen 12( capacity* =ne of the recommendations made by :yrne B"$$-E in her paper on evaluating communication for social change is the need to redress imbalances at all levels by providing ade@uate resources and support in order to strengthen evaluation capacity* 3!

Developing evaluation capacity can be seen as part of the process of institutionalising evaluation and creating an evaluation culture 'ithin UN agencies and their government and NG= implementing partners 'ho are involved in C.D activities* his process is anticipated to generate more high @uality 12( and impact assessments of C.D and to improve C.D initiatives and development initiatives that include C.D elements* hese processes are vital components of the strategy to 'iden appreciation of the value and significance of C.D in reaching development goals*

1vervie' of evaluation ca acity develo ment

C(valuation capacity developmentD B(CDE and Cevaluation capacity buildingD B(C:E !! have been conceptualised in various 'ays* In their paper on supporting partner country o'nership and capacity in development evaluation+ Lundgren and 5ennedy B"$$67 -!E define (CD as Cthe process of unleashing+ strengthening and maintaining evaluation capacityD* hey see evaluation capacity as Cthe ability of people and organisations to define and achieve their evaluation obAectivesD and e)plain that capacity involves three interdependent levels7 Cindividual+ organisational and the enabling environmentD* (CD is seen as Ca long-term process+ targeted in the conte)t of strengthening capacity in related systems of management+ accountability and learningD BLundgren 2 5ennedy+ "$$67 -!E* Pres&ill and :oyle provide a very detailed definition of evaluation capacity building 'hich reflects various concepts in their comprehensive multidisciplinary model of (C:7 (C: involves the design and implementation of teaching and learning strategies to help individuals+ groups+ and organisations+ learn about 'hat constitutes effective+ useful+ and professional evaluation practice* he ultimate goal of (C: is sustainable evaluation practice - 'here members continuously as& @uestions that matter+ collect+ analyse+ and interpret data+ and use evaluation findings for decision-ma&ing and action* #or evaluation practice to be sustained+ participants must be provided 'ith leadership support+ incentives+ resources+ and opportunities to transfer their learning about evaluation to their everyday 'or&* 8ustainable evaluation practice also re@uires the development of systems+ processes+ policies+ and plans that help embed evaluation 'or& into the 'ay the organisation accomplishes its mission and strategic goals BPres&ill 2 :oyle+ "$$-7 ...E* his definition highlights the need for capacity building to lead to sustainable evaluation practices 'ithin organisations and the importance of ta&ing a learning-oriented approach+ gaining support from management+ the provision of ade@uate resources+ and the development of strong systems and processes for 12( 'ithin organisations* %s 'e noted earlier in this report+ there is an increasing trend to'ards seeing evaluation as an ongoing learning process and as a means of strengthening capacity and improving organisational performance BGorton et al*+ "$$,E* Patton Bin Gorton et al*+ "$$,7 viiiE argues that aiming for multiple levels and &inds of impacts from evaluation is crucial 'hen resources are scarce+ such as in the developing 'orld*
!!

In this section 'e have referred to Cevaluation capacity developmentD since this is seen as the most appropriate term* Go'ever+ 'e also refer to Cevaluation capacity buildingD since this term is commonly used in the literature on this topic*

3"

:a all B"$$67 !",E notes that capacity Cincludes different realities from individual to institutional levelD and is usually defined as Cthe po'er of something to perform or to produceD* Li&e Lundgren and 5ennedy B"$$6E and Pres&ill and :oyle B"$$-E+ she suggests that it is Ca continuing process of learning and change managementD B:a all+ "$$67 !",E and that+ li&e development+ it is not a short-term process* :a all sees evaluation guidelines+ principles and ethical codes of conduct+ 'hich evaluation associations are deeply engaged in developing+ as a &ey tool for developing capacity* Ger paper highlights the effectiveness of evaluation associations and net'or&s in building capacity in various developing countries and beyond* Go'ever+ to be effective+ they Cmust play this role of organising the national dialogue amongst all development sta&eholders in the country+ and ma&e the bridge to the international community of evaluationD B:a all+ "$$67 !,,E* Li&e'ise+ :oyle suggests that professional associations for evaluators can play a role as facilitator of a net'or& of evaluation practitioners and users+ in the promotion of good practice standards and ethical guidelines+ in raising a'areness of Cmethodological and s&ills developments and innovationsD+ in the provision of education and training+ and in offering Cuseful support for developing the evaluation resourceD B:oyle+ !6667 !.!-!."E* In addition+ :amberger B"$$6E argues that the active involvement of leading national universities and research institutions is critical for (CD in the development conte)t* 8imilarly+ :alit B"$!$b7 /E comments on the role of universities in increasing the number of @ualified C.D professionals in development institutions and the need for the development of human resources at all levels J Cfrom field 'or&ers up to communication planners and managersD* :amberger B"$$6E advises that an (CD strategy for impact evaluation BI(E must target at least five main sta&eholder groups7 Cagencies that commission+ fund+ and disseminate I(sF evaluation practitioners 'ho design+ implement+ and analyse I(sF evaluation usersF groups affected by the programmes being evaluatedF and public opinionD* Ge e)plains that users include Cgovernment ministries and agencies that use evaluation results to help formulate policies+ allocate resources+ and design and implement programmes and proAectsD B:amberger+ "$$67 ,$E* his brief revie' of the literature highlights the need to consider the follo'ing elements of (CD in the C.D area7 he enabling conte)t of (CD activities and the readiness 'ithin organisations for change to'ards an evaluation culture that is focussed on learning+ improvement and accountability* he diverse organisational levels and sta&eholders groups that need to actively participate in and ta&e o'nership of the process+ in order to be most effective* he range of capacities and s&ills that need to be developed* he need to embed evaluation into all aspects of C.D activities in organisations and initiatives to increase the sustainability of (CD and evaluation practices* he role of professional evaluation net'or&s and universities in the (CD process*

%s 8chiavo-Campo B"$$/7 "-,E suggests+ Ccapacity-building in general and 12( in particular is far more than Aust trainingD* Ge advises against an over-reliance on one-off 12( 3,

'or&shops since Csustained capacity-building efforts are re@uired to improve the performance of the public sector on a lasting basisD B8chiavo-Campo+ "$$/7 -E* 8chiavo-Campo B"$$/7 !,E notes that a &ey lesson from the (CD e)perience is that Cbuilding an effective capacity for monitoring and evaluation is neither @uic& nor easyD and emphasises the need for Csteady and sustained support by international donorsD* he UN and ()pert Panel members 'e consulted also emphasised the importance of a long-term+ sustained focus on capacity development in 0+ 12( for staff at all levels* hey indicated that this process needs to include UN staff 'ho are conceptualising+ planning and managing 0+ 12( of C.D programmes and staff and sta&eholders of NG=s and government partners 'hich UN agencies are 'or&ing 'ith to implement and evaluate C.D activities*

/ome issues in ECD 'ithin develo ing countries

Naccarella et al* B"$$47 ",!E state that7 Cinternationally+ evaluation capacity-building activities have mushroomed as demands have increased for government-funded programmes to demonstrate that they are effective and efficientD* %s 'e have noted in this report+ similar demands are behind the increasing focus on more effectively evaluating the impacts of C.D programmes* ?arious strategies are no' being implemented in many developing countries to help sta&eholders and staff at different organisational levels Bincluding those involved in C.DE to learn about evaluation and to regularly engage in effective evaluation practices* % &ey aim of these strategies is to promote the adoption of an evaluation culture 'ithin organisations+ a process &no'n as Cthe institutionalisation of evaluationD B5hadar+ "$$,7 6,E* :amberger advises that institutionalisation of impact evaluation BI(E at the sector or national level occurs under the follo'ing conditions7 It is country-led and managed by a central government or a maAor sectoral agency* here is strong Sbuy-inT from &ey sta&eholders* here are 'ell-defined procedures and methodologies* I( is integrated into sectoral and national 12( systems that generate much of the data used in the I( studies* I( is integrated into national budget formulation and development planning* here is a focus on evaluation capacity development B:amberger+ "$$67 !.E*

:amberger notes that impact evaluation Ccan only be successfully institutionalised as part of a 'ell-functioning 12( systemD and that the 'ay in 'hich impact evaluation is institutionalised 'ill vary+ depending on Cdifferent political and administrative systems and traditions and historical factorsD B:amberger+ "$$67 !.E* his indicates the need to develop more effective systems for the evaluation of C.D initiatives as part of the process of developing a stronger evaluation culture 'ithin organisations involved in implementing C.D* Go'ever+ developing+ implementing and sustaining (CD can present particularly difficult challenges and issues in resource and capacity poor developing countries such as %fghanistan and Nepal 'ith high levels of poverty+ ill health+ illiteracy+ gender discrimination+ and ongoing political instability or violence* his suggests that the success of (C: is highly conte)t dependent B?alery 2 8ha&ir+ "$$/E* ?alery and 8ha&ir B"$$/7 -$E point out that 'hile 3.

donors and NG=s have been supporting evaluation capacity building activities for at least three decades+ most of these activities Coccur in developing countries and not in conflict or post-conflict settingsD such as %fghanistan* %mong the numerous conte)tual factors that affected access to and e)tension of health services and capacity building in %fghanistan+ ?alery and 8ha&ir B"$$/E note the erosion of human capacity+ the lac& of personnel 'ith managerial and technical s&ills throughout the country+ cultural constraints that limit access to health care for rural 'omen+ high illiteracy levels+ the absence of telecommunications in rural areas+ harassment of the international community+ violence and political instability* 8ome similar conte)tual challenges 'ere identified in the Assessing Communication for Social Change proAect that involved strengthening the evaluation capacity and 12( systems in a development communication NG= in Nepal* his process included using a range of participatory research tools and methods to engage listeners of the CChatting 'ith my best friendD B8815E and CNe' NepalD radio programmes and other community members in 12( and impact assessment processes* his process is being facilitated by 12( staff and a net'or& of community researchers in various rural and regional areas in Nepal BLennie et al*+ "$$6+ "$!$F acchi et al*+ "$!$E* Go'ever+ a number of challenges and issues arose 'hich affected the success of the capacity development process+ including the regular turnover of 12( managers+ loss of &ey leaders+ champions and change agents+ language and communication problems+ travel restrictions related to political instability+ the hierarchical culture that affected the use of participatory action research+ and various other factors related to the comple)ity of the cultural conte)t BLennie et al*+ "$$6+ "$!$F acchi et al*+ "$!$E*

)he value of taking a artici atory, holistic a

roach to ECD

he increased attention to (CD in recent times can be attributed+ in part+ to the gro'ing interest in the use of participatory and collaborative forms of evaluation BPres&ill 2 :oyle+ "$$-E+ and to increased a'areness of the benefits of incorporating evaluation into programmes to facilitate better decision ma&ing and ongoing organisational and programme improvement B#etterman 2 >andersman+ "$$/F Gorton et al*+ "$$,F Pres&ill 2 :oyle+ "$$-F Lennie et al*+ "$!$E* Crac&nell B"$$$7 ,/.E highlights the need to strengthen evaluation capacities in developing countries Cas part of the trend to'ards more participationD+ 'hile 5han B!66-7 ,!"-,!,E notes that some donors favour the use of participatory evaluation as the basis for all future (C: 'or&* % &ey benefit of participatory approaches to research and evaluation is that they can demystify these processes and ma&e them more accessible to a 'ider range of participants+ including community members* Indeed+ one of the ()pert Panel respondents thought that @ualitative and participatory methodologies held Cfar greater potential in terms of strengthening the research+ evaluation and development-capacity of individuals+ organisations and communities themselves+ in lasting and empo'ering 'aysD+ compared 'ith @uantitative and survey-based methodologies and methods* %s 'ell as the many benefits of participatory 0+ 12( methodologies that 'ere listed in 8ection .+ such as the fle)ibility of the process and the potential to foster greater levels of staff and sta&eholder o'nership and empo'erment+ other benefits and strengths of participatory and action research methodologies for (C: include7

3/

 %dopts a Clearning by doingD approach 'hich is recommended in particular for adult learners* Can provide rapid feedbac& about the success or failure of an (C: intervention* Can be a cost-effective method of (C: B#orss et al*+ "$$3F aut+ "$$4F ?alery 2 8ha&ir+ "$$/E Gorton et al* B"$$,7 .3-.-E highlight the value of adopting participatory+ learning-oriented self assessment processes for managing and improving organisational capacity development* :ased on their learnings from a maAor international proAect on the evaluation of capacity development in research and development organisations+ Gorton et al* B"$$,E advocate ta&ing a holistic approach to organisational capacity development* Its principles include7 a&e o'nership of your organisationDs capacity development initiative* #ocus on the needs and priorities of the organisation as a 'hole* 1anagement of capacity development processes is crucial for success* Prepare for monitoring and evaluation at the outset of a capacity development initiative* Capacity development is more than a one-off event* (ngage sta&eholders in the capacity development process* (stablish an environment conducive to learning and change BGorton et al*+ "$$,7 //E*

Gorton et al* B"$$,7 /3E e)plain that organisational capacity building is a process that Cevolves over a number of years *** the development and maintenance of good 'or&ing relationships bet'een the various parties involved in a capacity development effort is crucial to its overall successD* his approach is highly congruent 'ith the participatory+ holistic+ learning-based approach to the 12( of C.D that 'e have consistently advocated throughout this report* Go'ever+ as research by Lennie B"$$/E+ Lennie et al* B"$$6E+ #orss et al* B"$$3E+ acchi et al* B"$!$E and aut B"$$4E has sho'n+ the use of participatory evaluation methods for (CD raises various challenges+ issues and contradictions that should be ta&en into account* hose identified by acchi et al* B"$!$7 !E include7 Cthe po'er relations bet'een donors+ outside evaluation and development specialists and internal 12( staffDF cultural factors that can lead to dependency on research and evaluation specialists+ and Cthe time re@uired to build relationships and effective communication and engage sta&eholdersD* =ther challenges and issues are outlined in the ne)t section*

Particular ECD challenges and issues in the C-D conte+t

=ur research has suggested a number of challenges and issues that have a particular impact on the effectiveness and sustainability of evaluation capacity development in the C.D conte)t+ including7

he diversity of C.D approaches he comple)ity of assessing the impact of C.D 33

he need for practical and sustainable impact assessment frame'or&s for C.D %ttitudes to 12( among donors+ C.D organisations and NG=s 1aintaining and sustaining evaluation capacity #acilitating 'ide participation in 12( for C.D Coordinating 12( 'ith C.D programme content and improvement needs he 'ide range of s&ills re@uired in 12( for C.D*

=ur literature search indicates that there are fe' published papers about evaluation capacity development in the C.D area* Go'ever+ in this section 'e have dra'n on &ey issues raised in this literature revie' and in our consultations+ as 'ell as 'or& on the %C.8C proAect+ that has identified a number of challenges and issues and valuable learnings about (CD in the C.D field* Diversity of C-D a roaches

%s 'e noted in 8ection ,+ there is a 'ide range of C.D approaches+ 'ith the four main CstrandsD across the UN described as7 :ehaviour Change Communication Communication for 8ocial Change Communication for advocacy 8trengthening an enabling media and communication environment*

>hile they have a number of similar aims+ these approaches dra' on different underlying theories and concepts* his suggests that some'hat different 12( approaches and methodologies and different types of indicators may be needed+ depending on the C.D approach ta&en* his 'ould affect the capacity development and training needs of UN agencies and their partners 'hich employ these different C.D approaches* #or e)ample+ programmes that adopt a C#8C approach are more li&ely to be interested in building capacities in bottom-up participatory forms of 0+ 12( compared 'ith programmes that adopt a :CC approach+ 'hich have tended to use 5%P: surveys and more top-do'n 12( methods* In addition+ the strengthening of an enabling media and communication environment strand 'ould have @uite specific capacity building needs related to researching and monitoring issues such as communication and media access and participation+ compared 'ith other approaches that are more focussed on social and behaviour change* Com le+ity of assessing the im act of C-D %s 'e have noted+ assessing the impact of C.D raises many methodological challenges and comple)ities that are less evident in other development areas* Issues here include the need for a better understanding of the communication conte)t B'hich is often comple) and rapidly 34

changingE and the challenges involved in CprovingD impacts to donors and the attribution of impacts to specific C.D elements of larger development programmes* Given the difficulty of predicting C.D outcomes and impacts+ the use of baseline measures and the development of 81%0 indicators can be @uite problematic* #or e)ample+ 12( staff involved in the %C.8C proAect found it difficult to develop some indicators since the obAectives of their radio programmes 'ere constantly changing+ based on funding for ne' radio programme topics that 'as regularly received from donors* his means that mainstream planning tools such as the logframe are often more difficult to apply to the evaluation of C.D programmes* (CD programmes also need to be based on a good understanding of the particular challenges and issues in the C.D conte)t* 1any C.D programmes at the country and NG= level are not 'ell e@uipped to deal 'ith these challenges and comple)ities+ given other constraints and difficult conte)tual challenges that they face* %s a paper on the %C.8C proAect by Lennie et al* B"$$67 -E points out7 he comple)ity of the C#8C model+ 'ith its focus on media interventions stirring dialogue and creating the impetus and self-efficacy for social change implies a research focus on aspirations of social change+ 'hat is said about constraints to social change and shifts in public opinions+ social organisation and patterns of inclusion* =bserving and trac&ing social change demands a deeply @ualitative and participatory approach+ yet the ability of organisations such as (%N L(@ual %ccess NepalM to roll out the comple) participatory 12( systems that the C#8C approach demands is fraught 'ith constraints such as lac& of and turnover of 12( human resource+ irregular supply of electricity+ fre@uent stri&es that hamper travel+ as 'ell as natural disasters that impede travel and the regular flo' of data from community researchers*

Need for ractical and sustaina"le im act assessment frame'orks for C-D hese issues emphasise the need for (CD in the C.D area to be based on more fle)ible+ realistic+ practical and sustainable 12( frame'or&s and approaches* Go'ever+ the challenge is to develop 12( systems and capacities that all staff can understand and use+ 'hile+ at the same time+ ta&ing the comple)ity of C.D and the impact assessment of C.D into account* 0esearch by Lennie et al* B"$$6E and acchi el al* B"$!$E has highlighted a number of challenges raised by evaluation capacity development proAects in developing countries Bsuch as the %C.8C proAectE that have both academic and practical aims* #ollo'ing feedbac& from the 12( staff at (@ual %ccess Nepal about the comple)ity of the %C.8C methodology+ the process 'as revised in an attempt to ma&e it more streamlined and immediate in terms of outcomes* his included focussing the 12( 'or& on the more general social change indicators that the C#8C model is associated 'ith+ such as increased dialogue and inclusion of e)cluded groups+ better grounding the approach in the content and obAectives of (%NDs broader development programmes and more clearly linking the
participatory research process to these particular areas of media output (Lennie et al*+ "$$67 3E*

*ttitudes to M3E 3-

%s this report has indicated+ 12( of C.D is often donor driven and underta&en for up'ard accountability rather than for learning and improvement purposes* It is not al'ays accorded a high level of importance by management and programme staff* Crac&nell B"$$$7 //E points out that there is a fundamental tension and incompatibility bet'een the t'o competing obAectives of evaluation Caccountability on the one hand+ and lesson learning on the otherD* Ge notes that 'hile governments and pressure groups favour accountability+ 'hich emphasises the degree of success or failure of a proAect+ aid agencies and developing countries tend to favour lesson learning to improve future performance and to identify the reasons for success or failure* Go'ever+ it is difficult to ade@uately satisfy both obAectives* In addition+ many developing countries complain that evaluation research is Ctoo costly and too time consuming to be of any use to managementD B5han+ !66-7 ,".E* % small number of respondents in the study by Napp et al* B"$$"7 .,E thought that the e)pectations of the funding agency 'as the main reason for underta&ing the evaluation and most C:=s Cfelt burdened by evaluation re@uirementsD* 8ome sa' it as a Snecessary evilT to meet the re@uirements of their funding agencyD* hey also felt that funding agencies needed to understand the challenges inherent in evaluating GI? prevention programmes+ and that 'ithout this understanding Cthe evaluation e)pectations of funding agencies 'ere li&ely to be unrealistic and more of a hindrance to than a facilitator of evaluationD BNapp et al*+ "$$"7 ./E* %s 'e have previously indicated+ donors and managers 'ere also seen by some of the ()pert Panel as not valuing CalternativeD 12( approaches that are li&ely to be more appropriate for C.D* his can affect the amount of time and the ade@uacy of resources provided for (CD and the effectiveness of strategies that aim to develop an evaluation culture 'ithin organisations that implement C.D initiatives* 0esearch sho's that a lac& of support for evaluation among programme staff and management is a &ey barrier to effective (CD and evaluation* In the study by Napp et al*+ respondents reported that time spent on evaluation 'as seen as compromising service @uality and that staff Cresist evaluation because it encroaches on their opportunity to provide prevention servicesD BNapp et al*+ "$$"7 ..E* his suggests a need for specific funding to support evaluation efforts and for long-term funding for such prevention programmes BNapp et al*+ "$$"7 .3E* Crac&nell B"$$$7 !-"E also points out that staff attitude to evaluation findings C'ill differ according to their role in the organisationD* he %C.8C proAect found that 12( staff 'ithin (@ual %ccess Nepal had a lo'er status 'ithin the organisation than programme production staff+ some of 'hom 'ere high profile presenters of the very popular 8815 radio programme 'hich has a large follo'ing of young listeners around Nepal* :efore the %C.8C proAect began+ a fe' staff+ including programme production staff+ had received training in (thnographic %ction 0esearch* Go'ever+ there 'as no system in place to effectively analyse (%0 data+ provide feedbac& on it+ and ma&e effective use of it beyond the 'or& of the individuals underta&ing it* he programme production staff 'ere therefore unable to utilise this data in reports to donors* >hile the research helped them to improve some aspects of their 'or&+ it did not help them to prove impact* Contrast this 'ith the measurable feedbac& received through large numbers of listener letters and other feedbac& about the 8815 programme 'hich indicated that the programme 'as highly successful+ but did not directly help them to understand ho' or 'hy* he letters 'ere easy to @uantify+ and the 8815 team therefore felt C'hy do 'e need to evaluate the impact of our 'or&VD %s a result+ at the 36

beginning of the %C.8C proAect+ most staff thought that (%0 'as not used very 'ell in their organisation and they had not been able to demonstrate its usefulness to sta&eholders BLennie et al*+ "$$67 .E* Lennie et al* B"$$6E found that one of the most important outcomes of interactive (CD 'or&shops that the %ustralian research team conducted as part of the %C.8C proAect 'as improved Cteam buildingD+ Cteam spiritD and communication and appreciation of the need for a Cculture of sharingD among 12( and radio programme production staff* Go'ever+ a critical revie' of 12( systems in June "$$- identified that >hile the Naya Nepal program team 'as cooperating 'ell 'ith the 12( team+ the 8815 team 'as seen as CresistantD to changing the 'ay it does 12( since they thought their current system 'as 'or&ing 'ell* hey had also been reluctant to provide content themes to the community researchers in case this affected the number of listeners* 12( staff thought that the 8815 team 'as very insulated and not open to others entering their group BLennie et al*+ "$$67 3E %gain+ this e)ample highlights the importance of understanding the organisational dynamics and conte)t and attitudes to 12( before (CD strategies and processes are developed and implemented* =ne of the ()pert Panel recommended using a diagnostic tool developed by the >orld :an& B1ac5ay+ "$$4E 'hich can help to build better 12( systems* Maintaining and sustaining ca acity 1aintaining and sustaining evaluation capacity is a &ey issue 'hen there is high staff turnover in C.D organisations and &ey CchampionsD leave organisations* (mployee turnover is a persistent challenge in developing countries 'here there is often a shortage of people 'ith good evaluation capacities* his can undermine (C: efforts due to problems 'ith maintaining capacity and s&ills and varying levels of commitment to the (CD process from ne' staff B%t&inson et al*+ "$$/F Napp et al*+ "$$"E* %s 'e noted earlier+ one of the &ey challenges 'hich affected the success of the %C.8C proAect and the (CD process 'as the regular turnover of 12( coordinators+ and loss of &ey leaders and change agents 'ithin (@ual %ccess Nepal* his created problems 'ith continuity and time 'as needed to bring each ne' coordinator up to speed 'ith all the facets of the proAect* =nly the first 12( coordinator too& part in the initial (CD 'or&shops 'ith staff 'hich included developing initial theory of change matrices and practicing various participatory tools 'ith community groups* his situation put pressure on remaining staff 'ho then had less time to devote to the proAect BLennie et al*+ "$$6E* Gorton et al* B"$$,E suggest that rather than focussing on building the capacities of individuals and parts of an organisation+ it is more effective to focus on building the capacity of the organisation as a 'hole and to encourage the active participation of a broad range of staff and sta&eholders in the process* his latter strategy can cushion the impact of staff turnover BGibbs et al*+ "$$6E* he %C.8C proAect attempted to use this more holistic approach to (CD* %s acchi et al* B"$!$E e)plain+ this included adopting a partnership approach to capacity building by+ for e)ample+ encouraging (%N staff to ta&e some responsibility for facilitation of 'or&shops and meetings+ and organising meetings that engaged various sta&eholders Bincluding (%NDs competitorsE in the proAect* he research team also encouraged staff to see participatory evaluation as an ongoing action learning and 4$

programme improvement process that could facilitate the development of a learning organisation and research and evaluation culture 'ithin (%N* !acilitating 'ide artici ation %s 'e have emphasised+ effective 12( of C.D re@uires a high level of participation from a range of staff+ participants and programme or outreach partners* hey also need an ade@uate understanding of C.D and 12( concepts+ a range of relevant methodologies and methods+ and a 'illingness to devote time to the process of planning+ designing and implementing 12( processes* Given other demands on their time and energy and other factors+ achieving this level of participation and commitment to the process is not easy* %lthough there are many benefits to involving sta&eholders in the evaluation process+ 5han B!66-7 ,".E points out that C'here beneficiaries are restricted by une@ual po'er relationships+ the ability of evaluators to reach a cross section of beneficiaries 'ill continue to remain a problemD* %s 8outer B"$$-E points out+ it is often difficult for development interventions to reach the most marginalised groups such as the very poor* Participatory approaches to evaluation and (CD re@uire greater planning and higher levels of participation and engagement than other approaches BDiaH-Puente et al*+ "$$-E* ime and resources are therefore needed for ade@uate planning+ diagnosis of an organisationDs strengths+ 'ea&nesses and capacity building needs+ development of trust+ and encouraging participation BDiaH-Puente et al+ "$$-F Gorton et al*+ "$$,E* % study of self-evaluation capacity building in a large international development organisation identified a lac& of management support Cthrough engaged participationD in the (C: 'or&shops B aut+ "$$47 /"E* Go'ever+ #orss et al* B"$$3E suggest a need to be realistic about the level of input and involvement in an evaluation that should be e)pected from senior managers* hey also highlight the fact that for deeper learning in evaluation to have occurred they 'ould have had to have Cspent considerable time 'ith programme staff+ and to ensure that the interaction bet'een programme staff and the evaluation team made learning possibleD B#orss et al+ "$$37 !,-E* he comple)ity of assessing the impact of C.D programmes can be a barrier to the engagement and participation of a 'ide range of staff and sta&eholders* In addition+ there are often issues related to o'nership+ po'er+ control and participation* his is illustrated by the follo'ing account of a process that the authors engaged in 'ith staff of (@ual %ccess Nepal to develop theories of change B =CE for t'o of the (%N radio programmes+ 'hich involved 'or&ing 'ith teams that included programme producers+ management and 12( staff7 Initially this 'or&ed e)tremely 'ell+ and staff from across the organisation invested time and energy* Go'ever+ at some stage+ this 'as follo'ed by the realisation of the amount of time and 'or& that 'ould be re@uired to complete the =C 'ith a range of sta&eholders e)ternal to (%N* his 'as the thing that had most attracted us as academics to the approach+ and ended up being the factor that made the 12( team the most uneasy about it+ ma&ing it difficult to pursue* %lready stretched in terms of the demands of the organisation on their time+ the 12( team felt over'helmed by the seemingly Ce)traD 'or& this tas& re@uired+ and 'ere unable to 'eigh this positively against perceived benefits* >hile initially the =C seemed to offer an ideal 4!

mechanism for Aoining up 12( 'or& 'ith agreed change obAectives+ it shifted to an additional tas& 'ith ill-defined benefits* B acchi et al*+ "$!$7 4--E* Coordinating M3E 'ith C-D rogramme content and im rovement needs % further issue is that the time re@uired to plan+ organise and conduct participatory 12(+ analyse large volumes of @ualitative data+ and prepare reports on research and evaluation may not match 'ell 'ith the needs of C.D programme ma&ers 'ho 'ant data available more @uic&ly in order to inform programme content and ma&e decisions about improvements to ongoing programmes* his 'as another issue raised in the %C.8C program+ 'hich found that the 8815 production team Cneeded to obtain data immediately to inform their 'ee&ly letter revie' processD BLennie et al*+ "$$67 3E* Range of skills re?uired %s 'e have previously indicated+ &no'ledge and understanding of a range of theories+ frame'or&s+ methodologies and methods is needed to underta&e effective 12( and impact assessments of C.D initiatives* % number of papers highlight the 'ide range of s&ills re@uired to underta&e research and evaluations+ particularly those using participatory methods B:oyle+ !666F Gearn et al+ "$$6F Napp et al*+ "$$"F aut+ "$$4E* %s 'ell as technical s&ills+ they include7 Cstrong s&ills in facilitation+ as 'ell as humility+ respect for others and the ability to listenD BNarayan+ !66,+ cited in :oyle+ !6667 !.,E* =ther s&ills include7 Cresponsiveness to user needs *** acceptance of diverse vie's+ Land theM ability to establish rapport and trustD BGreen+ !6-- cited in aut+ "$$47 .6E* Gigh level conflict management and facilitation s&ills are also needed 'hen sta&eholders have contradictory perspectives about the initiative or there are une@ual po'er relations bet'een participants* In addition+ some popular methods such as 1ost 8ignificant Change are not necessarily as simple to use as their handboo&s indicate* he 18C guide describes the process as Ca simple means of ma&ing sense of a large amount of comple) informationD BDavies 2 Dart+ "$$/7 !!E and recommend conveying the message that C18C is simple and straightfor'ard to implementD BDavies 2 Dart+ "$$/7 !/E* Go'ever+ research has sho'n that the full 18C techni@ue can be @uite comple) to use effectively B>illetts 2 Cra'ford+ "$$4E* >e discuss the strengths and limitations of 18C in 8ection 4 of this report* In order to effectively use a mi)ed methods approach to 0+ 12(+ 12( staff re@uire s&ills+ &no'ledge and e)perience in the analysis of @ualitative and @uantitative data* Go'ever+ as the %C.8C proAect found+ the management+ analysis and interpretation of large volumes of @ualitative data can be particularly difficult for 12( staff in conte)ts such as Nepal 'ho lac& e)perience in this area and do not have access to the type of face to face training and local support that 'ould more easily and rapidly enhance their capacities* he %C.8C proAect also demonstrated that+ as Jallov B"$$/E has sho'n+ community members can be trained to collect useful @ualitative data about the impacts of community radio programmes* Go'ever+ as Lennie et al* B"$$67 -E note7 here is a need to ensure that community researchers are very clear about the conte)t and focus of this research 'or&* hey also re@uire continuous mentoring and support+ and both formal and informal training to increase their capacities and the @uality of the data they collect* (ffective feedbac& systems are also needed to 4"

maintain motivation and to share learnings and e)amples of good @uality data* his process can ta&e over a year or more*

1ther ECD challenges and issues

he ne)t section provides an overvie' of some more general (CD issues that need to be considered+ including issues related to the use of participatory evaluation and participatory approaches to (CD* )aking the organisational culture, dynamics and conte+t into account %s 'e have indicated+ for (CD to be effective+ an organisation needs to have a readiness for organisational learning from evaluation and the environment and culture needs to be conducive to success B#orss et al*+ "$$3F Naccarella et al*+ "$$4F aut+ "$$4E* Carlsson et al* B!66.E in Crac&nell B"$$$7 !-!E state that Cthe 'ay aid activities are being evaluated is primarily a function of organisational dynamics and not a technicalIadministrative matterD* hey go on to argue that7 effective feedbac&+ and getting people to change their habitual 'ay of doing things in response to evaluation findings+ has to involve a thorough understanding of ho' organisations ta&e decisions+ ho' they set obAectivesF ho' they resolve internal conflictsF and ho' they learn* In other 'ords+ their thesis is that effective feedbac& is a function of organisational dynamics BCrac&nell+ "$$$7 !-!E* #orss et al* B"$$37 !,-E cite a UN(8C= survey 'hich indicates a number of organisational factors that can hinder learning from evaluation+ including Clac& of transparent communication and decision-ma&ing+ lac& of managers as models of learning+ lac& of re'ard for innovation and learning from mista&es+ and a largely missing collaborative cultureD* hey suggest that if such conte)tual prere@uisites are not addressed+ Clearning from evaluation 'ill encounter too many obstacles to really ta&e offD* Leadership is particularly important here* 8everal studies demonstrate the need for leaders to support (C: and evaluation+ and to be seen as strong models for learning B#orss et al*+ "$$3F 5hadar+ "$$,F aut+ "$$4F ?alery 2 8ha&ir+ "$$/E* his 'as also clearly demonstrated in the %C.8C proAect BLennie et al*+ "$$6E*

Considering differences in o'er, kno'ledge and status =rganisations form net'or&s of people 'ith different agendas and interests and varying levels of po'er+ status+ authority+ e)perience and e)pertise BCrac&nell+ "$$$7 !-"E* %s 'e indicated earlier+ the degree of conflict and cooperation among these groups has an impact on (C: and evaluation activities in the C.D area* he study by aut B"$$4E highlights the political nature of evaluation and the need to ta&e the organisational 'or& environment and the potentially negative effects of the self-evaluation process into account* Lennie B"$$/7 .!$E found that an %ustralian (C: proAect she evaluated B'hich aimed to build capacities in evaluating community-based IC initiativesE had a number of unintended and disempo'ering impacts due to Cine@ualities in po'er and &no'ledge+ the different values and agendas of the participants and researchers+ the pre-e)isting relationships and net'or&s 4,

'ithin the communities+ and other comple) issuesD* hese issues included Ca perceived lac& of o'nership and control of some proAect activities+ and confusion and misunderstandings about the proAect and the L(%0N(08 processD B"$$/7 .!$E* Lennie also suggested that gender and po'er issues need to be ta&en into account in such evaluations+ Cparticularly issues related to leadership+ communication and controlD B"$$/7 .!$E* 8imilar issues 'ere raised in the %C.8C proAect+ as 'e have already indicated* .anguage, literacy and cultural issues ?alery and 8ha&ir B"$$/7 6,E point out that evaluation capacity building is ClanguagedependentD* his is particularly the case 'hen not all of the participants in an (CD activity or an evaluation can spea& the same language* he diversity of local languages and the literacy levels of community participants are &ey issues that can affect peopleDs participation in certain 12( activities and 'as often raised in the %C.8C proAect* Client literacy and language 'as also fre@uently mentioned by respondents in the study by Napp et al* B"$$"E as something that hindered their data collection activities* 8ince client literacy 'as lo'+ they reported problems identifying appropriate data collection tools* It 'as also difficult to find tools appropriate for those 'ho do not spea& (nglish+ and designing their o'n tools+ or translating e)isting tools Coften re@uired s&ills and resources beyond their meansD BNapp et al*+ "$$"7 ."E* Go'ever+ many participatory research and evaluation tools have been specifically designed for groups 'ith lo' levels of 'ritten literacy and have been sho'n to be very effective in engaging a 'ide range of community members in development proAects+ as Chambers B"$$-E and others have sho'n* ?arious cultural issues can also affect the (CD process* #or e)ample+ Lennie et al* B"$$6E and acchi et al* B"$!$E have identified factors related to the hierarchical culture in Nepal+ and the comple)ity of the cultural conte)t that affected the outcome of the %C.8C proAect* acchi et al* B"$!$E note that the relative informality of the %ustralian academic institutions 'ithin 'hich the %C.8C research team 'or&ed contrasted greatly to the e)pectations of deference that characterise the relationships of Nepali students to their teachers* %nother factor is the rigid pedagogical methods used in Nepal in 'hich critical @uestioning or ta&ing the type of critical approach used by evaluators is not encouraged BLennie et al*+ "$$6E* his raises @uestions about the influence of cultural conte)ts on (CD because participantDs e)pectations of their o'n and each othersD roles clearly vary+ depending on their previous e)periences and bac&grounds* )aking communication and =evaluation language> issues into account Crac&nell B"$$$7 !-3E comments that Cevaluation is all about communicationD+ 'hile Pres&ill and :oyle B"$$-7 .//E suggest that an evaluation culture is reinforced Cthrough intense and sustained communication about evaluationD* Communication systems 'ithin organisations and the communication processes and language used in an (C: initiative are therefore &ey issues that need to be considered* Critical by-products of participatory action research are Cmethodological innovation favouring collaboration+ and locally driven theories and models for changeD+ and yet in practice this is Cfraught 'ith challenges and contradictionsD B8chensul et al*+ "$$-7 !$"-!$,E* #or e)ample+ acchi et al* B"$!$E reported difficulties 'hen the %C.8C proAect attempted to develop shared understandings of the theories of social change and the language of donor4.

influenced development evaluation* his indicates that there are severely limiting factors at play in the field of development 'here strong paradigms of participation e)ist in star& contrast to structures that tend to prioritise certain forms of &no'ledge BCorn'all+ "$$3F Corn'all 2 Coelho+ "$$4E* 1c5ie emphasises the need to focus on the communicative and relational dimensions of participatory evaluations 'hich can affect their outcomes in unintended 'ays and argues that C>hether 'e li&e it or not evaluation has created a language and modus operandi that can be e)cludingD B1c5ie+ "$$,7 ,"$E* Lennie B"$$/7 .!$E suggests that including a diversity of community members in participatory evaluations forces us to pay attention to issues related to the appropriateness of language as 'ell as the perceived value and relevance of participation and evaluation to various groups* In addition+ 5hadar B"$$,E and 5han B!66-E highlight the confusion generated by the multiple definitions of &ey evaluation terms in the literature* Develo ing o en and effective feed"ack systems 0elated to these communication issues is the need for open and effective communication and feedbac& systems and processes* Crac&nell B"$$$7 !63E notes that although feedbac& is vitally important+ Cironically this is the branch of evaluation 'hich has so far received least attentionD* Ge offers several possible reasons for this+ including the different s&ills re@uired+ the amorphous nature of feedbac&+ and 'ay that it Cinvolves seeming to put oneself in a superior position to colleaguesD BCrac&nell+ "$$$7 !6/E* Go'ever+ Crac&nell advises that it is no' 'idely recognised that feedbac& must be planned for and organised C'ith as much care and determination as 'as re@uired for the evaluation itselfD B"$$$7 !63E* =ur o'n 'or& in this area indicates that effective communication and feedbac& systems are essential to the success of participatory 12( in C.D organisations* *ddressing the need for good ?uality data and data management systems 5han B!66-7 ,!,E states that in developing countries C'ea& institutional and methodological capacities Bstaffing logistics+ &no'ledge+ s&ills etc*E affect the @uality of evaluation findings and conse@uently+ their credibilityD* hese are also issues for more developed countries* :oyle et al* B!6667 6E point to the need for Cgood reliable data 'hich can be trustedD to enable sound conclusions to be dra'n from evaluations* hey suggest that if such systems are not available initial efforts 'ill need to be put into establishing and developing sound data* Pres&ill and :oyle B"$$-7 .//-./3E emphasise that sustainable evaluation practice Cis in many 'ays dependent on the organisationDs ability to create+ capture+ store and disseminate evaluation-related data and documents *** as 'ell as processes+ procedures+ and lessons learned from evaluative effortsD* he %C.8C proAect used a range of strategies to improve the @uality and depth of data gathered by community researchers+ including developing a detailed community researcher manual+ periodic intensive and refresher training+ and regular follo' up visits and mentoring* 1any respondents in the study by Napp et al* B"$$"7 .,E e)pressed concern about Cthe validity of evaluation measures and 'ere less li&ely to support evaluation data 'hen they lac&ed confidence in the accuracy of evaluation dataD* hey also @uestioned the truthfulness of the self-reported behaviour of clients in their GI? prevention programmes and Ctheir ability to use evaluation designs 'ith sufficient scientific rigour to accurately measure programme 4/

outcomesD* he respondents 'ere concerned that 'ithout such scientific rigour+ the evaluation may produce spurious results that did not accurately reflect the programDs 'orth* 8imilar issues about the reliability of some 12( data collected about C.D programmes and the need for CindependentD evaluations 'ere raised in our consultations 'ith the ()pert Panel for this proAect* Go'ever+ CindependentD 'as sometimes used to refer to evaluators 'ho 'ere not briefed and managed by donors+ 'hich 'as seen as leading to reports that tend to cater to shared e)pectations bet'een consultant and donor+ rather than useful and insightful engagements on the ground and meaningful results*

/ome learnings a"out increasing the effectiveness of ECD in the C-D conte+t
:oth this literature revie' and the outcomes of the %C.8C proAect have provided a number of valuable learnings about (CD and creating an evaluation culture in development organisations+ including those involved in C.D* hey include learnings about the importance of leaders being committed to the process+ and the need to create a learning organisation and a collaborative organisational culture+ based on mutual trust and understanding* his re@uires a sustained effort over a number of years* 8ome of those involved in our consultations in Ne' <or& also suggested that bringing evaluation and C.D professionals together in dialogue is a useful strategy* his strategy 'as found to be very effective in the %C.8C proAect* =ther learnings reported by Lennie et al* B"$$67 6E and Lennie et al* B"$!$E include7 'll rele ant staff need to be in ol ed in E$& acti ities: 1anagement+ programme staff and 12( staff need to ta&e an active part in (C: activities so that evaluation is not seen as solely the tas& of the 12( section* his can help to reduce the impact 'hen &ey staff members leave the organisation* E$& initiati es need to be fle!ible and open to change: he design of (CD initiatives must be fle)ible and open to change or revision+ based on regular feedbac& from staff and sta&eholders* -ngoing meta#e aluation of E$. is aluable7 1eta-evaluation of (CD can help to improve evaluation capacity 'ithin organisations* he outcomes of meta-evaluations can also be effectively used to enrich and enliven practical evaluation tool&its and to pass on the learnings to others* =ngoing meta-evaluation and critical revie's are also important to gradually increasing the @uality and trust'orthiness of 0+ 12( findings BLennie+ "$$3F Lennie et al*+ "$!$E* /ood communication and feedback systems are important: he language and forms of communication used in (CD need to be appropriate and clear to all staff* Definitions of &ey concepts need to be clarified and agreed to by all relevant staff as early as possible* #eedbac& systems need to be 'ell thought out and timely so that they can be used to improve programmes and evaluation systems most effectively* (he alue of e aluation needs to be acti ely demonstrated to programme and management staff: Programme and management staff may be reluctant to spend resources and time on evaluations due to pressure to develop or deliver programmes 'ithin a set timeframe and budget* hey may also be reluctant to change 12( systems 'hich they believe are 'or&ing 'ell* he best 'ay to address these challenges is to actively 43

demonstrate the benefits and value of underta&ing ongoing evaluation to these staff members in such a 'ay that it is integral to all their 'or&+ and not seen as an add-on* 0eep participatory e aluation methodologies simple and practical: In the initial phase of (CD initiatives that are trialling ne' participatory 0+ 12( methodologies and methods+ it is important to &eep evaluation methodologies and methods+ and 12( systems+ as simple and practical as possible* his should help to reduce confusion or lac& of motivation and interest among staff and participants+ and to increase their usefulness+ effectiveness and sustainability* his may re@uire spending more time in the initial planning phase on ensuring that the (C: obAectives and process is clear to everyone and not too ambitious or unrealistic in its scope* he roles and responsibilities of everyone involved also need to be very clear*!"

Conclusion
=ur research has identified a significant need to strengthen capacity in C.D and 0+ 12( at all levels+ from community and field levels to planning and management levels* >e have also emphasised the importance of ta&ing a long-term approach to evaluation capacity development* (CD 'as seen as part of the process of institutionalising evaluation and developing an evaluation culture 'ithin countries+ organisations and initiatives that use C.D* his process is anticipated to generate more high @uality 12( and impact assessments of C.D and to improve the design and outcomes of C.D initiatives* Professional evaluation associations and net'or&s+ universities and research institutions can play important roles in (CD in developing countries* >e also highlighted the value of adopting a holistic+ participatory+ learning-oriented approach to managing and improving capacity development* Go'ever+ 'e ac&no'ledged that using participatory evaluation methods for (CD raises various challenges and issues* Developing+ implementing and sustaining (CD 'as seen as presenting particularly difficult challenges and issues for time+ s&ill and resource-poor organisations in developing countries* Challenges and issues that have a particular impact on the effectiveness and sustainability of (CD in the C.D conte)t include7
!"

he comple)ity inherent to assessing the impact of C.D* %ttitudes to 12( among donors+ C.D organisations and NG=s* Lac& of practical and sustainable impact assessment frame'or&s for C.D* he diversity of C.D approaches+ 'hich affects (CD needs* 1aintaining+ supporting and sustaining evaluation capacity 'hen there is high staff turnover and loss of change agents and champions* Coordinating 12( 'ith C.D programme content and improvement processes* he 'ide range of s&ills re@uired for effective 0+12( of C.D*

>e 'ould li&e to clarify that+ as one of the ()pert Panel pointed out+ these learnings apply to all evaluation methodologies and (CD processes+ 'hether participatory approaches are used or not*

44

=ther (CD challenges and issues that 'e identified included7 the need for a readiness for organisational learning and an organisational culture that is conducive to successF po'er and conflict issues 'ithin organisations and (CD proAectsF language+ literacy and cultural issuesF and developing effective data collection and management and communication and feedbac& systems* his literature revie' and our o'n research in this area have provided some valuable learnings about increasing the effectiveness of (CD in the C.D conte)t* hey included7 involving all relevant staff in (CD activities+ ensuring that (CD is fle)ible and open to change+ actively demonstrating the value of 12( to programme and management staff+ and &eeping evaluation methodologies and 12( systems as practical and simple as possible*

4-

98 2ey a roaches, methodologies and methods for research, monitoring and evaluation of C-D
/ummary of key findings
#our &ey themes and issues related to research+ monitoring and evaluation frame'or&s have emerged from this research7 !* "* ,* .* he need for a more fle)ible approach in selecting and using 0+ 12( approaches+ methodologies and methods* he value and importance of a participatory approach* he benefits of a mi)ed methods approach* he importance of using approaches and methodologies that consider the 'ider conte)t and structural issues*

Participatory 1onitoring and (valuation is recommended as an effective 'ay of engaging sta&eholders in all stages of the evaluation+ strengthening evaluation capacities+ and increasing o'nership of the process* ?arious participatory+ @ualitative or mi)ed method approaches and methodologies 'ere considered effective for assessing the impacts of C.D+ including case studies+ Participatory 0ural Communication %ppraisal+ =utcome 1apping+ and contribution assessment* 9uantitative survey-based methodologies and cost benefit analysis 'ere also seen as effective* Causal analysisIproblem analysis+ the heory of Change approach and the logical frame'or& approach 'ere considered effective for planning impact evaluations of C.D* Commonly used @ualitative methods such as in-depth intervie's and focus group discussions+ and participatory tools such as communityIvillage mapping+ 'ere seen as particularly effective in assessing the impacts of C.D* Go'ever+ each of the approaches+ methodologies and methods that 'ere assessed as effective have strengths and limitations or constraints that need to be considered* 5ey factors in selecting the approach+ methodologies and methods for 0+ 12( of C.D include7 the e)tent to 'hich they 'ill provide the most useful and desired outcomes and are consistent 'ith the particular C.D approach+ and the resources and support available*

Introduction( 2ey themes and issues related to R, M3E frame'orks and a roaches
#our &ey themes and issues have emerged from this research7

46

!* "* ,* .*

he need for a more fle)ible approach in selecting and using 0+ 12( approaches+ methodologies and methods* he value and importance of a participatory approach* he benefits of a mi)ed methods approach* he importance of using approaches and methodologies and that ta&e the 'ider conte)t and larger structural issues into account*

)he need for a fle+i"le a

roach

% &ey finding from our consultations 'as that more openness+ freedom and fle)ibility is needed in the selection and use of various 0+ 12( approaches+ methodologies and methods to ensure that they are appropriate and fit the aims of the C.D programme* %s one of the ()pert Panel noted7 )he research and evaluation methodologies and processes selected should fit with the underlying aims and values of the development initiative or programme involved( 9ften this is not the case and people are frustrated by attempts to fit 7s5uare pegs into round holes.& without having the freedom& support and know/how to choose approaches and methods that best fit the particular conte't and the research or evaluation 5uestions being considered(

% 'ide diversity of C.D initiatives is implemented 'ithin the UN system and the approaches used to evaluate these different initiatives vary 'idely* In order to deal 'ith the Ccomple)ities and challenges of monitoring and evaluating C.D proAects or proAect componentsD Puddephatt et al* B"$$67 "4E recommended adopting a Ctailored tool&itD approach to selecting methodologies and methods* hey suggest that

he typeBsE of methodology selected 'ill depend both on the type of proAect+ but also the restraints of the evaluation in terms of time+ resources and organisational challenges* #or this reason+ each methodological approach 'ill vary* %s such+ suggesting best practice+ in terms of selection of methodologies that can apply to all C.D evaluations+ 'ould be impossible as 'ell as futile* Instead+ the evaluator should decide at the outset of each evaluation 'hat 'ould be the appropriate set of methodologies and tool&its for the specific intervention* Li&e'ise+ Par&s et al* B"$$/7 ""E argue that CP'e should be 'ary of forcing Communication for 8ocial Change initiatives into e)isting or pre-determined monitoring and evaluation frame'or&s and systems* #rame'or&s are seen by Chapman and >ameyo B"$$!E as Cuseful for giving an overvie' of areas to loo& at but should be seen as tools for facilitating -$

creative thin&ing* he challenge is to remain open to unintended outcomes that fall outside the frame'or& of assessment that may have been adoptedD Bcited in Par&s et al*+ "$$/7 ""E* In addition+ 8outer B"$$-7 !44E states that CContinual rapid change adds considerably to the difficulty of proAect design for interventions that ma&e use of IC s because it ma&es many of the assumptions involved unstableD* Ge points out that the proAect cycle is often , to / years for many development interventions and that ICD interventions Care therefore li&ely to need greater fle)ibility than is often found in other development initiativesD B8outer+ "$$-7 !44E* )he value and im ortance of artici atory a roaches

hroughout this report 'e have emphasised the value and importance of participatory approaches to 0+ 12(* hey are 'idely ac&no'ledged as highly effective and appropriate for C.D* =ne of the ()pert Panel made the important point that ltimately& how 7effective. a particular methodology or method is deemed to be& should depend on the degree of fit between the aims and values that underlie the particular initiative and those witnessed in the approach and process selected to research and evaluate such an initiative( Gighlighting the value of @ualitative and participatory approaches to evaluation+ another ()pert Panel member commented that7 Cthey are not models+ they are not tool&its or toolbo)es P they are open approaches that can be adapted locally+ that should be adapted locallyD* Participatory 0+ 12( frame'or&s and methodologies have a 'ide range of benefits that can greatly strengthen C.D programmes+ and their outcomes+ including the improvement of C.D programmes and policies+ increased capacities in 0+ 12(+ greater utilisation of findings and learnings+ and the empo'erment of participants* =ther benefits of participatory methods+ identified by Chambers B"$$6a7 .E are that7 >ell designed and facilitated+ participatory methods are rigorous+ and besides offering @ualitative insights can count the uncountable+ and generate statistics for relevant dimensions that 'ould other'ise be overloo&ed or regarded as purely @ualitative* hey open studies to the voices of those most affected by a proAect in 'ays not possible using more conventional methods and can ma&e the realities and e)periences of poor people count more* %s discussed in 8ection .+ Chambers B"$$6a7 3E has highlighted the Clargely unrecognised abilityD of participatory methods to generate numbers C'hich can also be commensurable and treated li&e any other statisticsD* Ge e)plains that7 hrough Audgement+ estimation+ and e)pressing values+ people @uantify the @ualitative* he potential of these methods is overdue for recognition* %s al'ays that there are ethical issues* >ell facilitated+ participatory methods can be 'in-'in J empo'ering people as 'ell as providing credible and reliable insights for policyma&ers*

-!

%s 'e have noted+ there are a number of challenges and issues associated 'ith participatory methodologies that need to be ta&en into account* hey include7 the greater level of time+ capacity development in necessary s&ills and resources+ issues related to po'er ine@uities and inclusion and the potential for dependence on the facilitatorI evaluation consultant+ and for conflicting agendas and perspectives of various sta&eholder groups to hinder success BGregory+ "$$$F Lennie+ "$$/F Gearn et al*+ "$$6F 1c5ie+ "$$,F acchi et al*+ "$!$E* =ther challenges and limitations of participatory 12( methodologies are outlined in the section belo' on effective methodologies* Go'ever+ as 'e have previously argued+ in the long run participatory approaches can be less costly 'hen their many benefits are factored into the costs involved* Puddephatt et al* B"$$67 !,E ma&e the important point that Csta&eholder communication+ involvement and o'nership should not be limited to methodologies that are more participatory in nature7 efforts should be made to build this element into any evaluation practiceD* )he "enefits of a mi+ed methods a roach

%s 'e have highlighted throughout this report+ using a mi)ed methods approach to 0+ 12( provides much needed fle)ibility+ sheds light on different issues+ and increases the strength and rigour of evaluation and impact assessment findings B:amberger et al*+ "$!$F :yrne+ "$$6aF Gearn et al*+ "$$6F Leeu' 2 ?aessen+ "$$6F Puddephatt et al*+ "$$6F >hite+ "$$6E* % &ey finding from our surveys of the UN and ()pert Panel groups 'as that -$N of the UN respondents and 46N of the ()pert Panel assessed a mi)ed methods approach to 0+ 12( for C.D as Cvery importantD in their 'or&* In comparison+ @uantitative approaches 'ere considered Cvery importantD by /$N of the UN #ocal Points and ,6N of the ()pert Panel and @ualitative approaches 'ere considered Cvery importantD by -$N of the UN #ocal Points and nearly -/N of the ()pert Panel* :amberger et al* B"$!$E ma&e a compelling and comprehensive case for the use of mi)ed methods in the monitoring and evaluation of international development programmes* 1any of their arguments also apply to the evaluation of C.D* he numerous benefits of a mi)ed methods approach that they identify include7 %re more appropriate for Cmore comple)+ multi-component programmesD* Can be more effective 'hen impact assessments have to be done @uic&ly+ 'ith a modest budget+ and have other constraints* he approach can Cprovide a number of rapid feedbac& techni@uesD* Can help to provide detailed analysis of local conte)ts* (nables information on sensitive topics such as domestic violence to be collected and helps to locate and include difficult to reach groups* Can enable more effective reconstruction of baseline data* Can Cprovide the fle)ibility to integrate good monitoring practices 'ithin evaluation designsD and to ma&e more creative use of administrative records and analysis of communication and information such as emails+ ne'sletters and radio announcements*

-"

Can significantly strengthen the rigour and validity of @uantitative approaches by triangulating various data sources and ensuring that survey respondents are interpreting @uestions in the same 'ay* 9ualitative information on the implementation process enables evaluations to assess 'hether certain outcomes are due to design or implementation failure* Can Ccontribute a range of @ualitative indicators as 'ell as generating case studies and in-depth intervie's to help understand the meaning of the statistical indicatorsD* (nables the specific information needs of different sta&eholders to be met B:amberger et al*+ "$!$7 ,- !3E*

)he im ortance of taking conte+t into account >e have also consistently emphasised the importance of using 0+ 12( frame'or&s+ approaches and methodologies that ta&e into account the 'ider social+ economic+ political+ cultural+ communications and environmental conte)t and larger structural issues that affect C.D initiatives and are of concern and interest to sta&eholders* 8outer B"$$-7 !3!E suggests that a Cgreater depth of understanding of conte)t is re@uired for impact assessment than at other levels Be*g* evaluationED* Ge argues that Cconte)tual factors are often highly important in determining 'hether particular results ariseD* Go'ever+ one of the principles of Participatory 1onitoring and (valuation BP12(E+ 'hich is highly recommended for measuring Communication for 8ocial Change initiatives+ is that Cthe evaluation must be conte)t-specific+ rooted in the concerns+ interests and problems of the programDs end users BPar&s et al*+ "$$/7 !"E* his suggests that a good understanding of conte)t is important for both evaluation and impact assessment* >e describe P12( in more detail in the ne)t subsection* Gosling and (d'ards B"$$,7 !"3-!"4E point out that non-linear models of change 'hich sho' ho' the conte)t of programme inputs and outputs affects the resulting changes is Ccloser to the reality of development interventions *** %ccording to this model+ change is brought about by a combination of specific events+ people and conditions present in a given situation+ as 'ell as by the proAect or programme underta&enD* %s many others have noted+ those 'ho understand the local conte)t best are the people living in the communities affected by the initiative* he different conte)t of C.D initiatives also affects the selection of particular 0+12( approaches methodologies and methods* 8outer B"$$-7 !4.E argues that it Cis not possible to construct a single impact assessment model or frame'or& for ICD proAects *** because different methodologies are re@uired for the very different conte)ts and types of obAective involvedD* 'o particularly important differences in the ICD conte)t that he identifies are7

!*

he difference bet'een technologies J CDifferent approaches to impact assessment are li&ely to be re@uired depending on 'hich technologies are involvedD*

-,

"*

he difference bet'een interventions concerned 'ith media or traditional C.D obAectives and those concerned 'ith the use of IC s to deliver services to proAect users B8outer+ "$$-7 !4/E*

1vervie' of the key R, M3E methodologies

>e begin this section by presenting an overvie' of P12(+ 'hich has been recognised as a &ey approach to the evaluation of development programmes and C#8C initiatives B(strella+ "$$$F Par&s et al*+ "$$/F ?ernooy et al*+ "$$,E* P12( is an umbrella term for some of the more focusedIspecific methodologies that 'e subse@uently outline+ 'hich are considered effective for 0+ 12( of C.D* hese methodologies 'ere identified in our literature revie' and survey results* #inally+ 'e summarise a number of strengths and 'ea&nesses or limitations of the &ey methodologies that 'ere identified by our survey respondents as effective for assessing the impacts of C.D programmes* Partici atory Monitoring and Evaluation P12( has been recommended as an effective approach to actively engaging sta&eholders in all stages of the monitoring and evaluation of C#8C and ICD initiatives and strengthening evaluation capacities and o'nership of the process B:yrne et al*+ "$$/F 1yers+ "$$/F Par&s et al*+ "$$/E* It emerged because of the limitations of conventional 12(+ 'hich is seen as mainly serving the needs and interests of proAect implementers and donors B?ernooy et al*+ "$$,7 "6E* P12( has been described as Ca set of principles and a process of engagement in the monitoring and evaluation endeavour* he process is at least as important as the recommendations and results contained in P12( reports or feedbac& meetingsD BPar&s et al*+ "$$/7 4E* %nother definition of P12( is7 any process that allo's all sta&eholders - particularly the target audience - to ta&e part in the design of a proAect+ its ongoing assessment and the response to findings* It gives sta&eholders the chance to help define a programmeDs &ey messages+ set success indicators+ and provides them 'ith tools to measure success B1yers+ "$$/7 !6E* P12( differs from traditional 12( by Cattempting to include all sta&eholders in all aspects of the processD BGolte-1c5enHie et al*+ "$$37 ,3/E* %nother distinguishing feature of P12( is its fundamental values of Ctrust& ownership and empowerment$ BPar&s et al*+ "$$/7 !!E* P12( is seen as an integral component of a proAect that is Cclosely 'oven into the 'hole proAect cycle *** It provides information that can be fed bac& into the proAect immediately to improve subse@uent performanceD B?ernooy et al*+ "$$,7 "6E* he participatory tools and techni@ues of P12( Chave evolved as useful tools for involving local people in developing strategies for learning about their communities and for planning and evaluationD BPar&s et al*+ "$$/7 !!E* Go'ever+ there are many local forms of P12( that go unrecognised* he concept of P12( is not ne'* It dra's from over ,$ years of participatory research traditions+ including Cparticipatory action research BP%0E+ participatory learning and action Bincluding participatory rural appraisal or P0%E+ and farming systems research B#80E or farming participatory research B#P0ED B(strella+ "$$$7 ,E* Par&s et al* B"$$/E point out that P12( entered the policy-ma&ing domain of large donor agencies and development organisations Bincluding #%=+ U8%ID and the >orld :an&E during the !64$s* -.

'o main streams of P12( have been identified7 !* Practical PM3E 'hich is focused on the pragmatic and 'ith fostering evaluation use* his is seen as similar to Developmental (valuation B'hich is described belo'E and Csta&eholder-based evaluationD* "* )ransformative PM3E 'hich is based on emancipation and social Austice activism and focuses on the empo'erment of oppressed groups* his has similarities to transformative versions of (mpo'erment (valuation B'hich is described belo'E and Cdemocratic evaluationD BPar&s et al*+ "$$/7 !$-!!E* In practice+ there are overlaps bet'een these t'o streams* Par&s et al* B"$$/7 !.E list si) essential ingredients that are needed to ma&e P12( 'or&7 !* 0eceptive conte)t J P12( 'or&s best 'hen the organisational climate and political conte)t is fairly open and democratic* "* he evaluatorDs or evaluation teamDs commitment to participation and faith in the inherent capacity of people to contribute meaningfully to the P12( process* ,* 0ecognition that P12( ta&es time and resourcesF it cannot be rushed *** .* People s&illsXparticularly facilitationXare a &ey part of the participatory evaluatorDs tool&it* >illingness to share e)periences+ &no'ledge+ insights+ and perhaps most difficult+ po'er* /* Capacity building should be a P12( obAective* Capacity building is consistent 'ith P12( goals and principles* Capacity building enhances accountability and supports sustainability through community and leadership development+ creating a core of participants 'ho are committed to the programIinitiative and &no'ledgeable about it* 3* he process should be structured in such a 'ay that ensures participation of the different interest groups but must be easy to facilitate because local facilitators may be themselves ine)perienced in participatory techni@ues*

Useful artici atory, ?ualitative and mi+ed methods a

roaches and methodologies

>hile assessments bet'een the UN and ()pert Panel members 'e surveyed varied+!, the follo'ing participatory+ @ualitative or mi)ed methods approaches and methodologies!. 'ere generally considered to be the most effective for assessing the impacts of C.D programmes7
!,

Case studies

% comparison bet'een the findings from the surveys found that the UN #ocal Points rated most of the methodologies and methods as more effective than the ()pert Panel+ sometimes to @uite a large e)tent* his could indicate that the #ocal Points tended to ta&e a less critical perspective or that they had less on the ground or in-depth &no'ledge of the actual use of some of these methodologies and methods* his needs to be ta&en into account 'hen considering the assessments provided belo' of various methodologies and methods*
!.

hese 'ere included in a list of methodologies that survey respondents 'ere as&ed to assess*

-/

Participatory 0ural Communication %ppraisal BP0C%E 0apid 0ural %ppraisal B00%E =utcome 1apping 1ost 8ignificant Change B18CE techni@ue (thnographic %ction 0esearch B(%0E

(ach of these methodologies+ other than case studies+ 'as outlined in the bac&ground paper by Puddephatt et al* B"$$6E* hat paper highlighted 18C and =utcome 1apping as t'o of the methodologies 'hich they thought represented Cstate of the art techni@ues that practitioners should consider adding to their e)isting toolbo) of evaluation approachesD BPuddephatt et al*+ "$$67 "4E* 18C 'as also included as a &ey methodology for use in a P12( frame'or& in Par&s et al* B"$$/E and has become 'idely used and &no'n in the development field* (mpo'erment (valuation 'as also considered @uite effective+ especially by some of the UN respondents+ and has been successfully used in a variety of programmes around the 'orld B#etterman+ "$!$+ "$$!F #etterman 2 >andersman+ "$$/E* =ther participatory+ @ualitative or mi)ed methods methodologies that 'ere nominated by respondents as effective 'ere7 Developmental (valuation 0ights-based approach methodologies Contribution assessment %ppreciative In@uiry Usability studies for online platforms

>hen used appropriately+ P0C%+ (%0+ the 18C techni@ue+ =utcome 1apping+ (mpo'erment (valuation+ Developmental (valuation+ and %ppreciative In@uiry are all highly participatory methodologies* 8everal of these methodologies are becoming more 'idely used or recommended for 0+ 12( purposes in different development conte)ts and programmes* %s 'e discuss belo'+ most of these methodologies include the use of a variety of methods and tools that can provide different forms of @ualitative and @uantitative data+ therefore enabling a mi)ed methods approach to 12( to be ta&en* he follo'ing provides an overvie' of these methodologies and the UN #ocal Points and ()pert Panel memberDs assessments of these methodologies for assessing the impacts of C.D programmes* $ase studies Case studies 'ere assessed as Cvery effectiveD by -$N of the UN #ocal Points and -/N of the ()pert Panel+ ma&ing it the methodology that 'as considered the most highly effective of all those listed in our surveys for assessing the impacts of C.D programmes* It is also the methodology that 'as most often used by both groups B-$N of the UN respondents and -/N of the ()pert Panel CoftenD used case studies in their C.D 'or&E* Case studies fre@uently involve the use of participatory+ @ualitative and @uantitative methods* Lacayo B"$$37 !!E points out that Cseveral scholars argue that the case study approach is appropriate for e)ploratory+ descriptive+ or e)planatory purposesD* Lacayo B"$$37 !!E also states that comple)ity theorists Cfavour the choice of a case study approach as it enables the researcher to study a phenomenon as an integrated 'holeD* -3

%t a recent international conference on impact evaluation for development effectiveness+ 5hagram suggested that case studies 'ere the best method+ empirically+ for loo&ing at causal path'ays about the effects of an intervention* Go'ever+ he noted that they are often poorly done* 5hagram argued that case studies provide the ability to focus on conte)t+ multiple outcomes from the same initiative+ and unintended causes* Ge argued that the comparative case study approach provides a rigorous method that enables learning and &no'ledge development from impact evaluations B5hagram+ "$$6E* Participatory Rural $ommunication 'ppraisal P0C% 'as assessed as CveryD or Cfairly effectiveD by 4/N of the UN #ocal Points and 4,N of the ()pert Panel* his methodology 'as adapted from 0apid 0ural %ppraisal* P0C% is described as Ca @uic&+ multidisciplinary and participatory 'ay to conduct communication research* It actively involves the people concerned in the research process to ensure that Communication for Development programmes are effective and relevant to themD B%nyaegbunam et al*+ "$$.7 !E. P0C% is Ca methodology that combines participatory approaches 'ith communication methods aimed at investigating issues+ especially in rural settings+ 'hile building the capacities of the individuals involved in the processD B1efalopulos+ "$$/7 ".6E* his process Callo's sta&eholders to play an active role in defining their realities and prioritiesD B1efalopulos+ "$$/7 "/$E* %s 'ell as using a range of participatory and @ualitative tools and methods+ P0C% includes @uantitative 5%:P baseline surveys for 8ituation and Communication %nalyses* =ne ()pert Panel member thought that this made it Ca very po'erful and comprehensive approach+ especially for :CC L:ehaviour Change CommunicationM benchmar&ingD* Rapid Rural 'ppraisal 00% 'as assessed as either CveryD or Cfairly effectiveD by 4/N of the UN #ocal Points and /-N of the ()pert Panel Bho'ever+ /$N of the ()pert Panel assessed 00% as Cfairly effectiveDE* 00% is an approach to rural development research that uses multiple methods to enable outsiders to learn about and understand rural conditions and rural poverty in a timely and cost-effective 'ay BChambers+ "$$-E* Chambers B"$$-7 4.-4-E outlines ten disparate 00% methods 'hich include7 using e)isting information+ &ey indicators+ local researchers+ direct observation+ &ey informants and group intervie's* =ne ()pert Panel respondent considered that 00% 'as Ca good @ualitative methodology accepted also by policy ma&ersD 'hich is C@uite 'ell &no'n and practiced in many countriesD* Chambers B!66"7 4E e)plains that 00% emerged in the late !64$s due to dissatisfaction 'ith the Cbiases *** of rural development tourism *** disillusionment 'ith the normal processes of @uestionnaire surveys and their results *** LandM *** the gro'ing recognition that rural people 'ere themselves &no'ledgeable on many subAects 'hich touched their livesD* It 'as also found that Ce)cept 'hen rushed and unself-critical+ 00% came out better by criteria of cost-effectiveness+ validity and reliability 'hen it 'as compared to more conventional methodsD BChambers+ !66"7 -E* -utcome Mapping

-4

=utcome 1apping 'as assessed as either CveryD or Cfairly effectiveD by 3,N of the UN #ocal Points and 34N of the ()pert Panel* It is an integrated approach to planning+ monitoring and evaluation that Cprovides a programme 'ith a continuous system for thin&ing holistically and strategically about ho' it intends to achieve resultsD B(arl et al*+ "$$!7 Chapter !7 .E Dra'ing on comple)ity thin&ing+ it recognises that multiple+ non-linear events lead to change* he originality of this approach is seen as its Cshift a'ay from assessing the development impact of a program Bdefined as changes in state X for e)ample+ policy relevance+ poverty alleviation+ or reduced conflictE and to'ard changes in the behaviours+ relationships+ actions or activities of the people+ groups+ and organisations 'ith 'hom a development programme 'or&s directlyD B(arl et al*+ "$$!7 Chapter !7 !E* (arl et al* B"$$!E detail the !" steps involved in using this methodology* 0ather than focussing on the impacts of a programme+ =utcome 1apping focuses on the often subtle changes that are clearly 'ithin a programmeDs sphere of influence+ C'ithout 'hich the large-scale+ more prominent achievements in human 'ellbeing cannot be attained or sustainedD B(arl et al*+ "$$!7 Chapter !7 4E* his approach is based on the concept of ongoing learning+ Cconsciousness-raising+ consensus-building+ and empo'erment *** for those 'or&ing directly in the development programD B(arl et al*+ "$$!7 Chapter !7 ,E* Its focus is on constant improvement+ understanding and creating &no'ledge rather than on proving+ reporting and ta&ing credit for results* Most 1ignificant $hange techni2ue 18C 'as assessed as either CveryD or Cfairly effectiveD by 4/N of the UN #ocal Points and /-N of the ()pert Panel* 18C is a highly participatory approach to 12( that is becoming very 'idely used or recommended in development conte)ts Bsee for e)ample :hattacharya+ "$$4F Jallov+ "$$4a+ "$$4bF Leeu' 2 ?aessen+ "$$6F Par&s et al*+ "$$/F >illetts 2 Cra'ford+ "$$4F >rigley+ "$$3E* =ne of the ()pert Panel listed a number of important strengths of 18C7 -eople love telling and hearing stories& if the environment is safe and trust and rapport well established( Caters to the une'pected and unpredictable( Enables people to tell their own stories& in their own words& and to have these listened to by an interested outsider& in a safe environment( )he process can be empowering both for the interviewer and the interviewee( +hen the MSC techni5ue is followed through& there is great group learning potential / both from the stories themselves as well as from their active participation in the process( E(g( having to prioritise and select particular stories& and 6ustify the rationale in each case& fuels important discussion& debate and learning( ,t can be a very satisfying process for all involved( MSC can really capture the rich detail of changes in the lives of people& communities and organisations involved( 18C should be used in combination 'ith other methodologies and 12( methods* It involves assessing the changes and impacts that have happened as a result of a programme from the perspective of participants* Programme participants and sta&eholders are involved in deciding 'hat sort of change should be recorded+ and in analysing the stories that are collected* he 18C process happens throughout the programme cycle and provides monitoring information that can help staff to improve a program* It also contributes to evaluation by providing information about the impacts and outcomes of a programme that --

can be used to assess ho' 'ell the programme as a 'hole is 'or&ing BDavies 2 Dart+ "$$/E* % &ey aim is to encourage continuous dialogue up and do'n the various levels of an organisation+ from field level to senior staff and bac& again* >hen this process 'or&s 'ell+ it can be a po'erful tool for ongoing evaluation and learning BDavies 2 Dart+ "$$/F >illetts 2 Cra'ford+ "$$4E* %lthough 18C emphasises @ualitative monitoring and reporting of change+ it can also enable some @uantification of changes* >hile the 18C techni@ue has proven popular and effective+ and has an Capparent simplicityD+ there are a number of deeper comple)ities and challenges in using the full 18C approach in development conte)ts that must be considered+ as >illetts and Cra'ford B"$$4E demonstrate* hey include7 the need for rigorous planning of each stage in the 12( cycle and to ensure ade@uate representation of Cdata sourcesD+ the need for higher-order s&ills than many conventional 12( methods+ problems 'ith conveying the concept of Cmost significant changeD to villagers+ issues 'ith po'er imbalances and the translation of stories+ and the e)tensive time re@uired for story selection* hey also identify a number of Cbroad enabling conte)tual factorsD that are important to the successful implementation of this techni@ue+ including support from senior management and Can organisational culture that prioritises learning and reflectionD B>illetts 2 Cra'ford+ "$$47 ,44E* %n additional issue is that the structure of 18C Cdoes not focus on+ or ans'er @uestions about+ overall proAect impactD B>illetts 2 Cra'ford+ "$$47 ,4-E* 8ome of these challenges and issues in using 18C 'ere also identified in the %C.8C proAect* Ethnographic 'ction Research (%0 'as assessed as either CveryD or Cfairly effectiveD by ,-N of the UN #ocal Points and 3.N of the ()pert Panel+ 'ho tended to ma&e more use of (%0* his methodology 'as mainly designed for use in community-based IC or media proAects and has been applied in a number of maAor development proAects conducted in 8outh and 8outh (ast %sia and else'here BGearn et al*+ "$$6F acchi et al*+ "$$,F acchi 2 5iran+ "$$-E* (%0 is similar to participatory action research+ 'ith three &ey distinctions+ as Gearn et al* B"$$67 -4E e)plain7 #irst+ the ethnographic refers not only to the &ey methods that are used L'hich are mostly @ualitative and participatoryM *** but also to the ethnographic approach that is a fundamental plan& of (%0 and the 'ay it is both integrated into the development of media initiatives and is ongoing* (%0 is designed to build the capacity of media initiatives to monitor and evaluate+ and *** to alter practices as part of their ongoing development *** 8econd+ (%0 'or&s 'ith the conceptual frame'or& of the communicative ecology* his involves paying &een attention to the 'ider conte)t of information and communication flo's and channels J formal and informal+ technical and social J and monitoring opportunities for both intervention and the changes that result* #inally+ the media itself are used as tools for action research+ for e)ploring issues in a community as 'ell as archiving+ managing and collecting data and facilitating online net'or&s of (%0 researchers* % &ey aim of (%0 is to develop a research culture through 'hich &no'ledge and reflection become integral to an initiativeDs ongoing development B acchi et al*+ "$$,F acchi et al*+ "$$4E* >hile (%0 is mainly focussed on the collection of @ualitative data+ several participatory (%0 tools enable the production of charts+ maps and diagrams 'hich can -6

provide valuable @uantitative information Bsee (%0 training handboo& at http7IIear*findingavoice*orgIE* 8hort @uestionnaire surveys can also be used to collect some statistical data from larger numbers of people that can be triangulated 'ith various @ualitative data to add rigour to 12( findings* he four-year %C.8C proAect further developed the (%0 methodology for assessing the social change impacts of community radio programmes in Nepal* >hile the proAect 'as successful in building an evaluation culture and strengthening 12( systems and capacities 'ithin (@ual %ccess Nepal+ a number of significant challenges and issues arose that affected proAect outcomes and impacts BLennie et al*+ "$$6 and "$!$F acchi et al*+ "$!$E* hey included issues related to the hierarchical culture in Nepal+ regular staff turnover and loss of change agents+ communication barriers+ po'erJ&no'ledge relations+ and the time and resources re@uired for proAect activities* Go'ever+ an ongoing mi)ed methods metaevaluation of the proAect 'as considered effective in increasing evaluation s&ills+ &no'ledge and capacities+ forming effective collaborative relationships 'ith participants+ and developing a practical impact assessment methodology and 12( systems that are li&ely to be sustainable BLennie et al*+ "$!$E* % tool&it on the %C.8C methodology and methods+ 'hich includes numerous e)amples contributed by 12( staff of (%N+ is currently being completed* Empowerment E aluation (mpo'erment (valuation 'as assessed as either CveryD or Cfairly effectiveD by ,3N of the ()pert Panel group and /4N of the UN #ocal Points 'ho 'ere able to ma&e an assessment*!/ his methodology aims to increase programme success by CB!E providing program sta&eholders 'ith tools for assessing the planning+ implementation+ and selfevaluation of their program+ and B"E mainstreaming evaluation as part of the planning and management of the programIorganisationD B>andersman et al*+ "$$/7 "-E* Communitybased proAects and organisations in many countries have successfully used this methodology to improve programmes and interventions in a 'ide range of fields B%ndre's+ !663F #etterman+ "$!$+ "$$!F #etterman 2 >andersman+ "$$/F 1iller 2 Lennie+ "$$/E* his methodology is distinguished by its clearly articulated principles7 B!E improvement+ B"E community o'nership+ B,E inclusion+ B.E democratic participation+ B/E social Austice+ B3E community &no'ledge+ B4E evidence-based strategies+ B-E capacity building+ B6E organisational learning+ and B!$E accountability B>andersman et al*+ "$$/E* he steps of (mpo'erment (valuation are7!E ta&ing stoc& or determining 'here the programme stands+ including strengths and 'ea&nessesF "E focusing on establishing goals *** 'ith an e)plicit emphasis on programme improvementF ,E developing strategies and helping participants determine their o'n strategies to accomplish programme goals and obAectivesF and .E helping programme participants determine the type of evidence re@uired to document progress credibly to'ard their goals B#etterman+ "$!$7 -E* :oth @ualitative and @uantitative data are collected as part of an (mpo'erment (valuation* 1iller and Lennie B"$$/7 ".E suggest that (mpo'erment (valuation has several strengths that ma&e it Ca practical and valuable methodology for improving and assessing the impacts of community based programmes *** and increasing their long-term sustainability and successD* Go'ever+ they advise considering a number of limitations and issues before
!/

It should be noted that 3"N of the UN #ocal Points had a lo' to moderate &no'ledge of this methodology 'hile it 'as un&no'n to ,3N of the ()pert Panel*

6$

deciding to implement this methodology* Common to all participatory approaches+ these include the funding+ time and resources re@uired to Cbuild evaluation capacities and include a diversity of programme staff and community members in designing and conducting the evaluationD* %lso+ to be effective+ Ca strong commitment is re@uired to the principles of (mpo'erment (valuation by senior management+ staff and community participants and ade@uate resources are re@uiredD B1iller 2 Lennie+ "$$/7 ". J "3E* &e elopmental E aluation Developmental (valuation 'as suggested as a useful methodology by t'o of the ()pert Panel* =ne of them e)plained that he uses this methodology Cto gather data Bsituation analysisE and inform criticalIcreative thin&ing during an ongoing process of CdevelopingD a model or proAect Bfor e)ample+ if there is a big change in conte)t and you have to adaptE - or 'hen one has to respond and adAust in comple) environments+ for e)ampleD* He further e)plained that this type of evaluation respects and incorporates comple'ity thinking and provides new ways to think about lines of accountability( )his type of thinking poses important 5uestions and challenges to results/based management( Developmental (valuation 'as designed for use by initiatives 'ith Cmultiple sta&eholders+ high levels of innovation+ fast paced decision-ma&ing+ and areas of uncertainty LthatM re@uire more fle)ible approaches BDoHois et al*+ "$!$7 !.E* his methodology refers to Long-term+ partnering relationships bet'een evaluators and those engaged in innovative
initiatives and development* LItsM processes include as&ing evaluative @uestions and gathering information to provide feedbac& and support developmental decision-ma&ing and course corrections along the emergent path* he evaluator is part of a team 'hose members collaborate to conceptualise+ design and test ne' approaches in a long-term+ on-going process of continuous improvement+ adaptation+ and intentional change* he evaluatorDs primary function in the team is to elucidate team discussions 'ith evaluative @uestions+ data and logic+ and to facilitate data-based assessments and decision-ma&ing in the unfolding and developmental processes of innovation BPatton+ "$$- cited in DoHois et al*+ "$!$7 !/E*

%s 'ith other forms of participatory evaluation+ some specialised s&ills are needed to effectively
underta&e Developmental (valuation* %s 'ell as s&ills such as communication+ facilitation+ active listening+ and fle)ibility+ DoHois et al* B"$!$7 ""E suggest that Ca t a minimum+ a D( needs to have

some facility 'ith strategic thin&ing+ pattern recognition+ relationship building+ and leadershipD* Rights#based approach methodologies =ne of the UN respondents nominated C0ights-based approach methodologiesD as an additional methodology that is effective for assessing the impacts of C.D programmes* 8he commented that Cthis is loosely understood and operationally ends up applying already &no'n methods* Go'ever+ it deserves to be treated as a SmethodologyTD* % number of evaluation frame'or&s that incorporate a human rights approach have been identified and analysed by Porter B"$$6E 'ho notes that C:eing specialised means that in many instances human rights is ne' to evaluators+ 'hile evaluation is ne' for human rights 6!

practitioners* Curiously a middle ground has been found in development practice+ 'here the interplay bet'een the t'o is especially importantD BPorter+ "$$67 !E* =ne of the frame'or&s revie'ed by Porter is the United Nations (valuation Group J Draft Guidance on Integrating Guman 0ights and Gender (@uality Perspectives in (valuations in the UN 8ystem* %s 'e noted earlier in the Principles section+ participatory approaches can be lin&ed to human rights such as the right to be heard and to be empo'ered+ based on various UN conventions* Li&e most genuinely participatory approaches+ the transformative mi)ed methods frame'or& associated 'ith Donna 1ertens is rooted in human rights and social Austice* It places priority on partnership-based indigenous-rooted approaches and uses methodologies that are culturally appropriate and employ mi)ed methods B1ertens 2 Chilisa+ "$$6E* he 8ave the Children tool&it by Gosling and (d'ards B"$$,7 4--E e)plains ho' a rights-based approach to development affects planning+ 12( and impact assessment7 %ttention shifts from the needs of people to the duties and responsibilities of those around them to respect+ protect and fulfil their rights* 0esponsibility for this is distributed bet'een family+ community+ civil society organisations+ national and local government institutions+ business and the media* %ll those 'ith responsibility to respect+ protect and fulfil childrenDs rights are defined as duty-bearers* he implications of this approach to child rights programming+ for e)ample+ include7 .roader analysis7 % broader situation analysis is re@uired+ focussing on peopleDs rights and responsibilities* .roader impact7 Programmes should have an impact on the root causes of rights violations+ as 'ell as immediate problems* Participation7 People Lincluding childrenM have a right to participate in decisions that affect them* +on#discrimination and e2uality7 %ll people have rights and this emphasises the importance of non-discrimination and e@uality* he best interests of children should come first BGosling 2 (d'ards+ "$$,7 6E*

$ontribution assessment %nother UN respondent nominated Ccontribution assessmentD as another useful impact assessment methodology* Ge described this as Can analysis of the contribution of a C.D intervention to an overall development outcome+ as opposed to a cause-effect relationship *** LItM sho's ho' C.D contributes to an outcome 'ithout being directly responsible for itD* =ur literature revie' has highlighted some of the benefits of this approach* Contribution analysis is a performance measurement approach that 'as developed by 1ayne B!666E* 0ather than attempting to definitively lin& a programDs contribution to desired results+ contribution analysis see&s to provide plausible evidence that can reduce uncertainty regarding the CdifferenceD a program is ma&ing to observed outcomes B1ayne "$$!+ cited in 6"

5otvoAs 2 8hrimpton+ "$$47 "4E* In addition+ contribution analysis recognises that it ta&es time for results to occur+ and Cso does not attempt to prove an impact before impacts can realistically be achieved B5otvoAs 2 8hrimpton+ "$$47 "-E* %n assessment of the use of contribution analysis in the #iAi (ducation 8ector Program Bfunded by %us%IDE found that its most notable benefits 'ere Cimprovements to the e)isting #(8P program logic+ monitoring against performance indicators that better demonstrate progress to'ards outcomes+ donor harmonisation and increased support for monitoring and evaluation activitiesD B5otvoAs 2 8hrimpton+ "$$47 ,.E* 'ppreciati e "n2uiry %ppreciative In@uiry 'as suggested by another UN respondent 'ho thought the strengths of this approach 'ere that it Cfacilitates dialogue and participationD* his process 'as developed by Cooperrider and Whitney (2005) and Cooperrider et al. (2003). It was Cdesigned to create democratically based+ visionary changeD in organisations BGergen+ "$$,7 /,E* It is based on an understanding of organisations as living human systems that are so ially onstru ted! and is seen as useful in formulating positive futures"oriented plans in a tion resear h pro#e ts ($earn et al.! 200%). In this approa h! problem identifiers and bringers are valued as they tell us that things ould be better. &rganisations are seen as being made up of many voi es! all of whi h have valid perspe tives ($earn et al.! 200%). Useful ?uantitative methodologies 9uantitative survey-based methodologies and cost benefit analysis 'ere also seen as effective for assessing the impacts of C.D programmes by several of the UN and ()pert Panel members 'e surveyed* Puddephatt et al* B"$$6E describe a number of CdiffusionbasedD @uantitative methodologies that are often associated 'ith top-do'n approaches to C.D and 12(* Go'ever+ as Leeu' and ?aessen B"$$67 ,"E suggest+ Cmany methods not commonly associated 'ith sta&eholder participation *** can also be used in more or less participatory 'aysD* 3uantitati e sur ey#based methodologies 9uantitative survey-based methodologies 'ere assessed as either CveryD or Cfairly effectiveD by 36N of the ()pert Panel and 6$N of the UN respondents+ 'ho made much more use of these methodologies in developing and evaluating their agencyDs C.D programmes* Puddephatt et al* B"$$6E outline the strengths and 'ea&nesses of a number of @uantitative and survey-based techni@ues that are commonly used in the 12( of development programmes+ and have an application to C.D programmes* hey include7 behaviour change comparison surveys+ behavioural surveillance surveys and 5no'ledge+ %ttitudes+ :ehaviours and Practices surveys* he strengths and 'ea&nesses of 5%:P surveys are discussed in the methods section belo'* %s 'e have consistently emphasised in this report+ @uantitative methodologies should be used in combination 'ith @ualitative methodologies to produce more useful and rigorous 12( findings* $ost benefit analysis

6,

Cost benefit analysis 'as assessed as either CveryD or Cfairly effectiveD by /.N of the ()pert Panel and 34N of the UN #ocal Points+ 'ho made much more use of this methodology in their agencyDs C.D 'or&* Puddephatt et al* B"$$67 -E suggest that cost benefit or cost effectiveness analysis can be a useful tool in demonstrating success Cin a results-based environment 'here competition for funds is tight and donors 'ant to see the most efficient use of their resourcesD* Go'ever+ one of the ()pert Panel argued that in Ccomple) C.D environments cost benefit analysis is too reductionist to tell us very much unless it is complemented 'ith in-depth @ualitative researchD*

Im ortant note a"out the ta"les in this section

he vie's e)pressed in the tables in the follo'ing section reflect the diverse perspectives of the UN C.D #ocal Points and ()pert Panel members 'e consulted* heir responses indicated that these perspectives ranged from tending to favour dominant 12( approaches and methodologies to strongly advocating for the use of alternative participatory or @ualitative approaches* his should be ta&en into account 'hen reading these tables* Go'ever+ as one of the respondents noted7 Cthe common criticism of participatory evaluation being subAectiveIless obAective can be applied to almost any methodology or methodD* It should also be noted that the same respondents 'ho described strengths of various methodologies+ approaches and methods also outlined their limitations or 'ea&nesses* heir responses indicate that even those 'ho favoured particular methodologies and approaches generally too& a critical approach that openly ac&no'ledged their 'ea&nesses as 'ell as their strengths* >e thin& this is an important point+ and that it is advisable to al'ays ta&e a critical approach and consider the 'ea&nesses along 'ith the strengths of the approaches+ methodologies or methods chosen* /trengths and limitations of the key methodologies It 'as beyond the scope of this literature revie' to revie' all of the literature on the and of each of the approaches+ methodologies and methods % strengths 'ide range of limitations strengths and 'ea&nesses Bor limitations and constraintsE of various 12( discussed in this section* Go'ever+ such information can be found in various evaluation methodologies 'ere described by several respondents to the UN and ()pert Panel surveys* and impact assessment guides+ tool&its and compendiums such as Clar& and 8artorius hey are summarised in able 4* B"$$.E+ Gosling and (d'ards B"$$,E+ Gee&s and 1olla B"$$6E and >estat B"$$"E* )a"le 9( /trengths and limitations of key M3E methodologies
Methodology Case studies /trengths Provide an in-depth revie'+ understanding or analysis of an issue or comple) situation and detailed information over space and time about the comple)ities of the research conte)t and change in the community+ including une)pected change* his allo's a holistic description of the process and outcomes+ &ey issues and trends+ and the ability to compare various scenarios to find out S'hyT* Case studies are concrete and provide real life information for replication and an understanding of the type of dynamics re@uired to achieve results* .imitations or 'eaknesses hey are limited in terms of their representation+ and their inability to generalise to a larger population and prove large-scale results* hey are vie'ed by some as anecdotal and not sufficiently CscientificD and it may be difficult to e)trapolate learning* Case studies can be designed for or against a single principle and need continuous careful attention* Preparing case studies re@uires certain e)pertise+ time+ support and financial resources* heir @uality depends on the @ualities of the documenter and sources of information - 'hether from direct or

6.

Participatory 0ural Communication %ppraisal and Participatory Learning and %ction BPL%E

0apid 0ural %ppraisal

=utcome 1apping

1ost 8ignificant Change

hese methodologies provide a @uic& means of ta&ing stoc& of needs+ preferences+ perceptions+ vie's and opinions of various target groups* hey allo' participants to define results and measure change and they stimulate and encourage participation and discussion 'ith sta&eholders at all levels* 9uite 'ell &no'n and practiced in many countries7 more local staff are beginning to be trained in these techni@ues* % good @ualitative methodology that is also accepted by policy ma&ers* Useful for planning and evaluation* his methodology puts people+ relationships and behaviours at the heart of the process and differentiates levels of change* It chec&s assumptions behind e)pected results+ and focuses on results that are directly related 'ith a particular programme or proAect* =1 does this by measuring ho' peopleDs lives have changed and the sense of o'nership they ac@uired over the programme or proAect* (valuates from the participantsD point of vie' and gathers various vie'points among sta&eholders to appreciate their perceptions and priorities* 18C can capture the rich detail of changes in the lives of people+ communities and organisations+ including the une)pected and unpredictable* his techni@ue enables people 'ho are usually voiceless to tell often comple) stories about ho' their individual lives have been affected* 18C can also catalyse participatory reflection+ and fuel important discussion+ debate and learning* he process can be empo'ering for those involved* (%0 captures relevant data through the active involvement of researchers 'ith different participant groups to document their conte)t and realities* It delves into underlying problems and loo&s closely at the 'hole process+ from the planning stage+ and is responsive to programmatic changes* (%0 facilitates timely feedbac& and significant+ ongoing learning+ and

indirect means* 0esults cannot be generalised and they are difficult to use for largescale resultsF re@uires particular s&ills and resources*

8een as not systematic+ and not measured rigorously enough to provide credible baseline reference*

=utcome 1apping is seen as overly detailed+ 'ith a terminology that can be e)clusive and a process that can be long* It does not focus on impacts and 'as seen as not very suitable for proAects aimed at changing processes Be*g* ne's-ma&ingE and much more diffuse results*

18C 'as seen as having similar problems to =utcome 1apping - the full process is time consuming+ demands sustained support and commitment and can be comple) and difficult to implement and use* Li&e case studies+ it may not provide robust evidence and be credible to certain sta&eholders* In addition+ 18C does not enable an understanding of the magnitude of phenomena+ less prominent factors and effects are not captured+ and it is difficult to @uantify*

(thnographic %ction 0esearch

(%0 'as seen as resource intensive+ time consuming+ and re@uires necessary e)pertise* It also captures depth and nuances of conte)t+ over time+ 'hich is hard or impossible to do at scale+ and may be merely descriptive 'ithout attempts to measure* he longitudinal nature of an (%0 study 'as seen as unsuited to the

6/

allo's immediate participatory engagement 'ith people in search for a common solution* his process can be enriching and can catalyse ne' ideas and behaviours* (mpo'erment (valuation Participatory (valuation Is people-focussed and allo's for longer term sustainability* 5ey actors and sta&eholders have a maAor say+ their voices+ perspectives and e)periences are elevated* P( can be a significant learning e)perience+ fostering constructive and critical reflection and learning among and bet'een all involved* It can be a richly re'arding e)perience+ as those involved see and feel the benefits of their o'n active participation* P( fosters commitment to and use of evaluation findings+ as those involved have a greater degree of o'nership of the process and findings+ both of 'hich therefore mean more to them* hey enable a 'ide coverage+ and provides a considerable amount of data* >ith the right sampling+ allo's for broad generalisations of findings to larger populations* 8urvey data gives a reality basis from 'hich to plan* hey are seen as crucial for gaining a basic understanding about certain problems+ and can effectively test assumptions about @ualitative and observed behaviour+ and gauge levels of public participation+ and monitor usage of tools etc* hese methodologies can provide more e)act or solid evidence or mar&ers+ particularly of short-term progress to'ards change and can be less costly and resource intensive than other methodologies* hey are also good for policy ma&ers* Cost benefit analysis %long 'ith 5%:P effects+ this is seen by one of the ()pert Panel as the most important method for bottom-line decision ma&ing about the 'orth of C.D* he method can be instrumental in deciding 'hether or not to support large-scale C.D initiatives* %nother ()pert Panel respondent considered this a Cgood methodD for policy ma&ers

UN;s short- term allocation of funds for research* his approach re@uires long-term planning 'ith defined and agreed benchmar&s+ the nature of 'hich may shift and change over time+ sometimes necessitating constant monitoring and adAustment* Can be difficult to Audge in the shortterm* 0e@uires time+ sustained support and resources* he common criticism of P( being subAective or less obAective can be applied to almost any methodology or method*

9uantitative surveybased methodologies

hey can be costly+ time consuming and labour intensive* heir use re@uires e)pertise in application and the participation of a team of people+ hence significant and careful pre and post planning and scheduling+ some financial investment and strong leadership are necessary* hey do not capture conte)t-specific realities and the level of detail re@uired to understand the nuances of impact Bincluding perceptions+ opinions and e)perienceE* In addition+ they do not allo' for @ualitative analysis and change over time in a given conte)tF and are best suited for short-term activities* here may also be problems 'ith conducting appropriate baseline surveysF they may be unsuitable for all situationsF they are often less engaging for participants* his methodology usually re@uires an economist to establish credibility and appropriate data gathering and analysis+ in short+ time and effort to produce results* %lso+ it does not al'ays provide an analysis of @ualitative aspects*

63

and donors*

1vervie' of key im act evaluation a

roaches and methods

In this section 'e outline the &ey approaches+ methods and tools identified in our survey results and in the literature 'hich 'ere considered effective for planning and implementing impact assessments of C.D programmes* >e then outline a number of strengths and 'ea&nesses of the &ey approaches and methods 'hich need to be ta&en into account* * roaches for lanning im act evaluations

he follo'ing approaches and tools 'ere generally considered by the UN #ocal Points and ()pert Panel to be the most effective for planning impact evaluations of C.D programmes7 Causal analysisIproblem analysis heory of Change approach Logical frame'or& BlogframeE approach

he follo'ing provides a brief overvie' of these approaches and tools and their assessment by the UN #ocal Point and ()pert Panel respondents7 $ausal analysis,problem analysis Causal analysisIproblem analysis 'as assessed as Cvery effectiveD by /3N of the UN #ocal Points and .3N of the ()pert Panel* % causal analysis frame'or& aims to identify the follo'ing7 !* "* ,* he maAor problem and conditionBsE that the proAect see&s to change* he factors that cause the conditionBsE* he 'ays to influence the causal factors+ based on hypotheses of the relationships bet'een the causes and li&ely solutions* .* he interventions to influence the causal factors* /* he e)pected changes or desired outcomes BChaplo'e+ "$$-7 "E Chaplo'e B"$$-7 ,E advises that causal analysis should be based on a careful study of local conditions and available data as 'ell as consultation 'ith potential beneficiaries+ program implementers+ other sta&eholders+ and technical e)perts* 8uch information may be available in needs assessments+ feasibility studies+ participatory rapid appraisals+ community mapping+ and other forms of analysis* =ther forms of problem analysis+ such as problem trees can be useful for Cisolating conditions and conse@uencesD that help to identify proAect obAectives and strategies BChaplo'e+ "$$-7 ,E* =ne of the ()pert Panel members found that the most effective tool in Participatory 0ural Communication %ppraisal is the use of Problem ree %nalysis for setting the focus of @uantitative baseline surveys* (heory of $hange approach

64

he heory of Change approach 'as assessed as either CveryD or Cfairly effectiveD by ,,N of the UN #ocal Points and -"N of the ()pert Panel*!3 C heories of changeD is Cone manifestation of the theory-driven approach used to evaluate comple) public policy interventionsD B8ullivan et al*+ "$$"7 "$3E* his is due to its apparent capacity to accommodate multi-sector activity BdiversityE+ its concern 'ith the relationship bet'een process and outcomes BdynamicsE and its emphasis on 'holesale change at individual+ organisational and system levels Bcomple)ityE B8ullivan et al*+ "$$"7 "$3E* his approach also ma&es e)plicit the values that underpin the perspectives of more and less po'erful sta&eholders B8ullivan et al*+ "$$"E and is highly compatible 'ith participatory 12( methods* % structured heory of Change tool has been developed by 5eystone %ccountability* Using this tool+ the process of developing a theory of change is seen as Can e)citing and often liberating process of interaction and discovery that helps organisations see beyond their familiar frames and habits P understand the full comple)ity of the change they 'ish to see+ and imagine ne' solutions in dialogue 'ith othersD B5eystone %ccountability+ "$$67 .E* he strengths of the theory of change approach include that it adds value to processoutcomes evaluations by Cre@uiring the lin& bet'een process and outcomes to be articulated at the beginning of the processD+ its capacity to lin& the Cparticipation of all relevant sta&eholders 'ith a ma)imisation of learningD+ and its emphasis on Cthe dynamic nature of conte)tD in the evaluation B8ullivan et al*+ "$$"7 "$-E* Go'ever+ 8ullivan et al* B"$$"E also identify a number of practical+ political+ theoretical and systemic limitations to applying theories of change in practice* hey include7 the problem of including Cdissenting voicesD in the process+ its lac& of reference to ho' po'er differentials may need to be addressed+ and the potential that a bottom up approach to theory could limit other e)planations from broader theoretical perspectives B8ullivan et al*+ "$$"7 "$6-"!$E* Logical framework approach he logical frame'or& approach or ClogframeD 'as assessed as either CveryD or Cfairly effectiveD by /-N of the ()pert Panel and -$N of the UN #ocal Points+ 'hich made much more use of this approach* %long 'ith 18C and =utcome 1apping+ Puddephatt et al* B"$$6E nominated the logframe as one of the 12( methodologies that they thought represented Cstate of the art techni@ues that practitioners should consider adding to their e)isting toolbo) of evaluation approachesD hey argue that Ca logical model of change should drive any 12( systemD and that the logframe Ccould be one 'ay to progress impact assessment of C.D initiatives as an interim measure and lay the ground'or& for more nuanced evaluation over time BPuddephatt et al*+ "$$67 "-ED* he logframe has been described as Cthe single most 'idely used device for presenting a summary description of 'hat aid programmes are trying to achieveD BGasper+ !664+ cited in Davies+ "$$.7 !$,E and is one of a larger class of tools &no'n as programme logic models+ 'hich are fre@uently used by organisations and evaluators* =ver the past "$ years the logframe has come to play a dominant and central role in the planning+ design+ implementation+ evaluation and management of development proAects B:a&e'ell 2 Garbutt+ "$$/F Dearden+ "$$!E and its use is no' stronger than ever*

!3

It should be noted that ..N of the UN respondents did not &no' this approach but -6N of them had a very high or high level of interest in learning more about it*

6-

Proponents of the logframe claim that it provides a Cstructured+ logical approach to setting priorities and determining the intended results and activities of a proAectD+ and the basis for Cevaluating the effectiveness+ efficiency and relevance of a proAectD BDearden+ "$$!7 ,E Logframes have also been praised for Cthe 'ay in 'hich they can encourage strategic thin&ing at different SlevelsT of a proAectD B(arle+ "$$"7 "E and for Cencouraging clear thin&ingD B:a&e'ell 2 Garbutt+ "$$/7 !"E* % maAor strength of this approach+ identified through research 'ith international development NG=s+ is that Cit forces development actors to thin& through the relationship bet'een 'here they 'ant to go Bthe impactE and 'hat they are going to do Bthe inputs and activitiesE and the intermediate steps on the 'ayD B:a&e'ell 2 Garbutt+ "$$/7 !-E* Go'ever+ the logframe has also been 'idely criticised as infle)ible+ reductionist and unable to capture une)pected outcomes or changes* It represents the simplification of comple) social processes and avoids the importance of process B(arle+ "$$"E* In addition+ there is a gro'ing a'areness that Cthe logframe and similar tools s@ueeHe out data related to local culture and conte)t+ and thus do not provide a space for an analysis of informal interactions and e)ternal influences that can be the lifeblood of a successful development interventionD B(arle+ "$$"7 /E* It is also seen as grounded in a C'orldvie' largely associated 'ith >estern positivist thin&ing+ and alien to the rest of the 'orldD B:a&e'ell 2 Garbutt+ "$$/7 !"-!,E* (ven 'hen a participatory approach to logframe development is used+ this can prove problematic due to it being difficult to change after sta&eholders have gone through a very thorough planning process B:a&e'ell 2 Garbutt+ "$$/E+ and the 'ay in 'hich it can foster Can e)tractive approach to participationD B(arle+ "$$"7 .E* >hile adaptations of the logframe have been developed+ and some have called for more participatory use of the logframe+ (arle has @uestioned 'hether Cthe problems of language+ >estern concepts of linearity and its fundamentally hierarchical nature LcanM ever be reconciled 'ith goals of empo'erment and giving voice to the most marginalisedDB(arle+ "$$"7 !.E* % number of other strengths and 'ea&nesses Bor limitations and constraintsE of these three approaches and tools+ described by respondents to our surveys+ are summarised in able belo'* )a"le :( /trengths and limitations of key im act evaluation lanning a * roach /trengths
Can provide solid insights into &ey issues re@uiring attention* It is seen as a reliable 'ay to describe determinants of and barriers to behaviour and social change at different levels of the problem tree+ particularly from analysis of intermediate and root causesIdeterminants of behaviour and social norms* It allo's problem identification to lead the 'ay and helps people to see the logic of an intervention* It can also determine the nature of a problem+ and provide an understanding of 'ays to address it that resonate 'ith

roaches

.imitations or 'eaknesses
he @uality of information depends on the reliability of sources Bthis 'as seen as a challenge+ not a 'ea&nessE* It can dilute the overall obAective+ and 'hen the intervention is too concrete and the outcome is too broad+ it is not easy to sho' the logic of an intervention* It can also be difficult to sho' the causal lin&s bet'een the proAect+ program+ regional and global levels of results* It re@uires good s&ills and e)tensive data and does not al'ays assist Band can even diminishE communication design if it is done

Causal analysis

66

local communication practices and interests* heory of change 8een as enabling more detailed analysis of different sta&eholders+ communication flo's and processes* It enables targeted proAect design and 12(* It is also good for e)pressing assumptions of causal changes and for deeper analysis of 'hat is 'or&ing or not+ for 'hom etc* (nables more effective planning and monitoring by helping people to clearly see the lin&ages of interventions+ and to thin& things through from the beginning in an easy to follo' systematic+ step-bystep method* his process helps to identify the se@uence from activities to outcomes and obAectives* Logframe is useful for setting performance obAectives and Audging 'hether they have been achieved* his tool can give clarity and simplicity to 'hat can be an overly comple) design+ and is seen as essential for identifying 'hat 'ill be trac&ed+ and 'hat baseline data is needed*

poorly* It is often designed to fit the scope and resources of the proAect Bby small NG=sE rather than ta&ing in all the e)ternal conte)t and has the same problems as logframe in that 'hen the conte)t changes the theory has to adAust* Go'ever+ it is intended to be adAusted over time* he process is generally not participatory+ has lo'-levels of fle)ibility+ can be time consuming+ does not e)press underlying causal theory and is not impact focused* It also re@uires good databases for setting achievable performance targets* It cannot al'ays account for the numerous factors that contribute to social change and does not sho' the level of commitment and contributions from sta&eholders to the attainment of results* It is very difficult to use the full logframe for comple) programmes and proAects due to the large number of indicators re@uired and the difficulty of formulating 81%0 indicators* he logframe can become a bo) tic&ing e)ercise that is not properly utilised once completed+ and can reduce communication creativity*

Logframe

Effective methods for assessing im acts of C-D able 6 belo' sets out the methods listed in our survey that 'ere generally considered by the UN #ocal Points and ()pert Panel respondents to be the most effective for assessing the impacts of C.D programmes* It sho's the percentage of respondents 'ho assessed these methods as either CveryD or Cfairly effectiveD for assessing the impacts of C.D programmes* )a"le ;( Ratings for methods considered effective for assessing C-D im acts Method In-depth intervie's #ocus group discussions CommunityIvillage mapping ChannelImedia usage and preference analysis ChannelImedia coverage UN C-D !ocal Points !$$N -6N --N 4/N 34N E+ ert Panel 6!N -,N 4-N 3.N /$N !$$

analysis Communication (nvironment %nalysis ParticipantIaudience analysis 5no'ledge+ %ttitudes+ :ehaviours and Practices B5%:PE surveys

..N /$N 3$N

3.N 34N /-N

his indicates that commonly used @ualitative methods such as in-depth intervie's and focus group discussions and participatory tools such as communityIvillage mapping 'ere seen as particularly effective in assessing the impacts of C.D* =ther methods nominated by respondents that 'ere often used or considered effective 'ere7 1edia content analysis estimonies and feedbac& from proAect participants ParticipantInonparticipant observation =bservational analysis =bservations of on-the-ground impacts such as audience reaction to a particular radio programme Participatory tools such as body mapping+ photo-novella+ poc&et or voting charts+ story 'ith a gap 5ey Informant Panel intervie's %nalysis of relevant secondary @uantitative and @ualitative data National and regional mapping of regulatory policy and legislative frame'or&s 1apping of institutional capacity+ media environment assessments Peer revie'

% 'ide range of strengths and 'ea&nesses Bor limitations and constraintsE of some of the &ey impact evaluation methods+ described by the respondents+ are summarised in able !$* )a"le %$( /trengths and limitations of key evaluation methods Method
In-depth intervie's

/trengths
hey enable more in-depth and closer e)ploration and understanding of issues and impacts of a communication initiative+ and the nuances of a programme from the participantsD point of vie' that can be very rich and telling* hey allo' probing and discussion of sensitive and personal information that could be more difficult to discuss 'ith 'ider audiences*

.imitations or 'eaknesses hey ta&e time+ are labour and resource intensive and therefore generally not suitable for use 'ith large numbers of people* 8ince they are necessarily individual and inevitably selective it may be difficult to e)trapolate to a general finding and may provide biased vie's* hey should be conducted 'ith focus group discussions in order to validate results*

!$!

#ocus group discussions

Can elicit and capture rich+ in-depth and varied input+ feedbac&+ vie's+ problems and information from a number of people in a short time and can usefully inform 'ider surveys and intervie's* hey can validate information through in-depth intervie's or triangulate related data and information obtained @uantitatively as 'ell as those from other @ualitative methods* #GDs also provide an opportunity for connecting people+ and enable participants to debate or discuss issues and learn from each other* hey open up communication flo's+ stimulate participation and commitment+ and can foster an openness to ne' ideas in a safe and supportive environment*

Can be time consuming and costly and need to be done 'ith several groups* hey re@uire systematic preparation and strong facilitation+ interpersonal and listening s&ills and s&ills in documentation and analysis+ an ability to deal 'ith sensitive issues+ to be open+ and to deal 'ith conflict and po'er differentials* he @uality of information may vary according to conte)t+ @uality of facilitator and attitude of the focus group on the issueItopic* #GDs can be manipulated to'ards a particular outcome+ there can be some constraints on 'hat can be said+ there are issues 'ith influence+ and the group may be s&e'ed to'ards more important members of a society* hey can rarely be used on a very large scale but can inform larger-scale data gathering* 8hould only be used 'ith other @ualitative methods and re@uires e)pertise in facilitation to ensure that the process achieves its potential+ that differences are ade@uately captured+ and that the environment is conducive to honest sharing and reflection*

CommunityIvillage mapping

Communication environment analysis

his 'as seen as an important method to identify social strata in a village and for introducing the spatial perspective+ and ho' a community relates to the e)ternal 'orld* %llo's for the une)pected and unpredictable+ enables comparisons bet'een different time periods+ groups+ genders etc*+ and has benefits for children and less literate groups* Participants usually enAoy and learn a lot through the process* he picture created can facilitate o'nership and use of the process and findings* his is seen as a holistic approach to addressing political+ socio-cultural+ economic+ geo-physical determinants of communication environment at family+ community and national levels* It can help identify+ as a priority+ 'hich communication channels or vehicles not only e)ist but are open and better positioned to use to achieve results* his method puts the subAects in the conte)t of their environment and uses @uantitative and @ualitative methods* If they are planned and e)ecuted 'ell+ provides baseline data for setting 81%0 C.D obAectives as 'ell as for developing 1 2 ( outcome indicators* 5%P: levels are

It needs thorough research+ using both primary and secondary sources Ba challenge and not a 'ea&nessE and may not be able to account for a @uic&ly changing environment* he value of this method depends on the nature of the research @uestion*

5%:P surveys

hey do not capture underlying causes of behaviour and issues around social norms+ may not include enough bac&ground to place responses in conte)t and may be

!$"

seen as essential to establishing the 'orth of a C.D effort and as a prere@uisite activity for cost benefit analysis* hey are more people centred than other types of survey methods and allo' the researcher to cross-chec& subAectsD responses*

difficult to @uantify* 0e@uire research s&ills normally ac@uired through PhD studies+ thereby limiting the pool of researchers available to carry them out*

/election of R, M3E a

roaches, methodologies and methods

his section considers some of the &ey factors that need to be considered in selecting the approaches+ methodologies and methods for the 0+ 12( of C.D including7 he e)tent to 'hich the approach+ methodology and methods 'ill provide outcomes and information that 'ill best meet the e)pectations of &ey sta&eholders and be most useful to them* he particular research and evaluation @uestions+ 'hich 'ill determine best methodologies and methods and the types of data that need to be collected* %chieving clarity about the particular paradigm that underpins the 0+ 12( 'or&* Go' consistent the 0+ 12( approach is 'ith the C.D approach and the 0+ 12( for C.D principles set out in this report* Go' 'ell the methodologies and methods 'ill engage the primary sta&eholders and participants in the 0+ 12( process* >hich particular mi) of methodologies and methods 'ill best provide the desired evaluation outcomes* he resources and support available* he fle)ibility and robustness of the evaluation design*

Meeting e!pectations and usefulness of the results: %n important initial step in an 0+ 12( process is to clarify the e)pectations of &ey participants and sta&eholders Bincluding funders and beneficiariesE* he aim is that the results of the evaluation are useful to the end users of the C.D initiative and that the results and findings are used to improve the initiative and understandings about the process of behaviour and social change and the role of C.D in bringing about change* hese factors 'ill also affect the selection of appropriate methodologies and methods* #or e)ample+ if results are needed @uic&ly+ planning to implement the full 1ost 8ignificant Change techni@ue 'ould be inappropriate since this is @uite a lengthy process* In addition+ 18C is not a stand-alone techni@ue and should be used 'ith other evaluation methods* $larifying the research and e aluation 2uestions: % further issue is that @ualitative or action research approaches do not usually begin 'ith clearly specified research @uestions+ but rather formulate @uestions after open-ended field research has been completed* =ne solution to this issue is for @ualitative 'or& that uses open-ended data collection methods !$,

such as in-depth intervie's to be embedded in structured research B>estat+ "$$"7 ./E* he @uestions that guide 0+ 12( 'or& 'ill determine the type of data that needs to be collected and the approaches and methods that can best be used* %s Johnson and =n'uegbuHie B"$$.7 !4 - !-E suggest in relation to using a mi)ed methods approach to research7 C>hat is most fundamental is the research @uestion - research methods should follow research @uestions in a 'ay that offers the best chance to obtain useful ans'ersD* $larifying the research and e aluation paradigm: his report has advocated the use of a participatory mi)ed methods approach to 0+ 12( of C.D* Go'ever+ 'e 'ould argue that mi)ing research and evaluation paradigms is problematic and tends to result in confusion and inappropriate compromises that limit the outcomes of evaluations* his means that it is necessary to be very clear about the particular paradigm that underpins 0+ 12( 'or&* Use of an empirical+ interpretive+ holistic+ systems-based+ participatory action research-based or pragmatic paradigm 'ill clearly affect your choice of methodologies and methods and the 'ay in 'hich they are implemented* $onsistency with $%& approach and M&E for $%& principles: he process of selecting methodologies and methods should be fle)ible+ participatory and consistent 'ith the C.D approach and the principles of 0+ 12( for C.D proposed in this report* Engaging primary stakeholders and audiences: Consideration also needs to be given to ho' 'ell the methodologies and methods 'ill engage primary sta&eholders and audiences in the 0+ 12( process+ are appropriate for the groups involved+ and 'ill lead to findings that they 'ill see as useful+ trust'orthy and credible* .est mi! of methodologies and methods( %nother step is to consider 'hich particular mi) of methodologies and methods 'ill best fit the evaluation outcomes being sought* his re@uires becoming familiar 'ith the main purpose of various methodologies and methods and understanding their strengths and limitations or constraints* he aim here is to consider the e)tent to 'hich any limitations or constraints out'eigh the strengths of the methodology or method and ho' 'ell different methodologies and methods balance or complement each other* Resources and support a ailable: 8everal different types of resources and constraints also need to be considered before finalising the @uestions that guide the 0+ 12( process and selecting the methodologies and methods* % &ey @uestion here is C>ill the particular approach+ methodologies and methods provide the type of information that 'e 'ant+ 'hen 'e 'ant it+ and help to ans'er our particular @uestionsV =ther factors include7 he budget and costs involved+ including staff time+ training and travel costs* ime constraints Bespecially if participatory and @ualitative methods are used and a large amount of data is collectedE* #inding 'ays of integrating 12( into e)isting systems and processes can help to reduce time constraints* he scale and scope of the evaluation J this should be proportionate to the scale of the program*

!$.

he level of s&ills+ &no'ledge and e)perience in using particular approaches and methods+ and any additional training that may be needed* he level of organisational support Bespecially for non-mainstream or unfamiliar 0+ 12( approachesE*

4le!ibility and robustness of the design( he design of an evaluation needs to be fle)ible and open to revision as data is gathered and those involved learn from the process* If some methodologies or methods prove unsuitable or do not yield the results that 'ere sought+ other methodologies methods need to be readily available for use* he 0+ 12( design should allo' further e)ploration of &ey @uestions or identification of gaps in the information that is being collected* 8uch a robust design allo's for this type of gro'th and change+ yet preserves the overall intent of the evaluation BUniversity of asmania+ "$$,E* #urther details and &ey @uestions to consider in selecting the most appropriate and effective approach+ methodologies and methods used in 0+ 12( of C.D 'ill be provided in the practical Guide section of the 0esource Pac&+ 'hich 'ill be further developed in "$!!*

Conclusion
here is a need for openness+ freedom and fle)ibility in selecting and using 0+ 12( approaches+ methodologies and methods to ensure that they are appropriate and fit the underlying aims and values of the C.D initiative* hey also need to ta&e into account various constraints such as time+ resources and organisational challenges* Participatory approaches to 12( have been advocated given their many benefits+ including strengthened evaluation capacities+ greater utilisation of findings and learnings+ and the empo'erment of participants* hey are also seen as Copen approaches that can be adapted locallyD* >e provided more Austification for using a mi)ed methods approach to 0+ 12(+ and emphasised the importance of using methodologies and approaches that can ta&e account of the 'ider conte)t and larger structural issues* Participatory 1onitoring and (valuation 'as recommended as an effective 'ay of actively engaging &ey sta&eholders in all stages of an evaluation and increasing o'nership of the process* he follo'ing participatory+ @ualitative or mi)ed methods approaches and methodologies 'ere considered the most effective for assessing the impacts of C.D7 Case studies Participatory 0ural Communication %ppraisal 0apid 0ural %ppraisal =utcome 1apping 1ost 8ignificant Change techni@ue (thnographic %ction 0esearch*

=ther participatory+ @ualitative or mi)ed methods methodologies that 'ere nominated as effective included Developmental (valuation+ rights-based approach methodologies and contribution assessment* 9uantitative survey-based methodologies and cost benefit analysis 'ere also seen as effective for assessing the impacts of C.D* Causal analysisIproblem analysis+ the heory of Change approach and the logical frame'or& approach 'ere !$/

considered the most effective for planning impact evaluations of C.D* Commonly used @ualitative methods such as in-depth intervie's and focus group discussions and participatory tools such as communityIvillage mapping 'ere evaluated as particularly effective in assessing the impacts of C.D* Go'ever+ strengths and limitations or constraints 'ere identified for each of the methodologies+ approaches and methods that 'ere seen as effective 'hich need to be ta&en into account* % number of &ey factors 'ere identified that need to be considered in selecting the approach+ methodologies and methods used in 0+ 12( of C.D* hey included7 the e)tent to 'hich they 'ill provide the most useful and desired outcomes and are consistent 'ith the particular C.D approach and the principles outlined in this report+ the particular research and evaluation @uestions being as&ed+ and the level of resources and support available*

!$3

:8 Indicators of C-D im acts

/ummary of key findings
Indicators measure progress against pre-determined aims and obAectives or development targets* hey are a &ey element in a results-based approach to 12(* Indicator setting is often challenging+ comple) and time consuming* In participatory 12( approaches+ indicators should be developed through participation and dialogue to be locally meaningful* Indicator setting can highlight the different information needs and ideas of change of different sta&eholders* Conventional evaluation stresses pre-determined indicators of success principally related to cost and outputs* % participatory approach re@uires ac&no'ledgement of different registers of success* here are a range of types of indicators+ 'hich can roughly be described as baseline+ process and outcomeIimpact indicators* here are variations of these+ depending on the application and purpose* Indicators need to measure physical and visible change+ but also less tangible attitudinal and behavioural chance* hey can be @ualitative as 'ell as @uantitative* In C.D+ @ualitative indicators are often the most effective and appropriate* Different approaches to indicators are re@uired for the four main UN C.D approaches* Indicators are unable to capture comple) realities and relationships and the reasons behind social change* In some C.D evaluations+ alternatives to indicators+ such as Csignificant change storiesD and Cverifying assumptionsD+ can be more useful*

Introduction
his section revie's the literature on indicators of C.D impacts* It starts by introducing the idea of indicators+ and their roles* Different types of indicators are presented before loo&ing at some general indicators developed specifically for C.D+ and the &inds of indicators that might suit the four main approaches to C.D used across the UN* 8ome of the challenges in the areas of ne' thin&ing in the field are presented+ follo'ed by a summary of some &ey ideas on indicators in C.D*

)he conce t of indicators

Indicators are+ @uite simply+ CobAective 'ays of measuring BindicatingE that progress is being achievedD+ 'ith CprogressD determined by the aims and obAectives of a particular initiative BGosling 2 (d'ards+ "$$,7 ,,-E* In international development+ indicators are used to measure the impact of development interventions and monitor the performance of proAects in relation to pre-determined targets B:ennett 2 0oche+ "$$$E* his apparent simplicity obscures the fact that indicator setting is considered by some to be Cthe most difficult step in establishing a reliable evaluative approachD BGuiAt+ !66- and 1i&&elsen+ !66/+ cited in Classen+ "$$,7 ".E* %s one of the ()pert Panel commented7 !$4

one of the points about indicators >is? that they might seem 0misleadingly1 simple to develop and use& but in reality if you try and cater to the *Checklist of ,ndicators$ and the SMA") framework& plus multiple conte'ts and diverse initiatives& even developing one common& meaningful indicator is a challenge( %s Gosling and (d'ards B"$$,E point out+ indicators simply indicate progress+ Cthey are not proofD* hey also cannot tell us ho' change occurs+ or 'hy communication made a difference BD%NID%+ "$$/E* Indicators can be @uantitative or @ualitative* #urthermore+ many Cso-called SintangibleT @ualitative impacts can be measured 'ith @uantitative indicators+ or vice versaD BGuiAt+ "$$$7 "$.E* Clarification and agreement on programme obAectives is essential before beginning the process of indicator setting* 9uantifiable and CobAectively verifiableD indicators are no' a &ey element of the results-based management approach to 12( in 'hich logframes are used* his approach is no' a &ey re@uirement of many international donors* %s 'e have previously noted+ a number of issues have been raised about the incompatibility of this approach 'ith the comple)ity of assessing the impact of C.D+ compared 'ith other development interventions+ given the comple)ity of behaviour and social change processes and the rapidly changing communication conte)ts 'ithin 'hich C.D initiatives are implemented* Go'ever+ the participants at the Ne' <or& consultation felt that creatively approaching and developing indicators of social change impacts could help mainstream evaluation e)perts in the UN to obtain the &inds of data that are other'ise difficult to obtain* Creating channels for discussions across evaluation approaches might help to create more complete understandings of social change* In C.D+ and in particular Communication for 8ocial Change+ 'here dialogue and participation are stressed+ it is 'idely considered that indicators themselves should be developed through dialogue and negotiation bet'een &ey participants+ so that indicators are chosen based on local assessments of 'hat participants 'ant to &no' and 'hy B:alit+ "$!$bF :yrne et al*+ "$$/F D%NID%+ "$$/E* his helps to identify 'hat information is critical+ clarify goals+ vie's on change+ information needs and values BGuiAt+ "$$$7 "$.E* his in turn is considered to be empo'ering Cas it allo's local vie's to dictate 'hat constitutes success or changeD* CGo'ever+ for indicator development to be empo'ering is an impressive feat and one that fe' 12( efforts can correctly claim to have achievedD BGuiAt+ "$$$7 "$.E* GuiAt B"$$$7 "$"E points out that Cindicator and method selection are intert'ined* %n ideal indicator may be selected+ but if no feasible method e)ists to assess it+ then the indicator must be adAustedD* 8he goes on to ma&e the important point that7 Cmuch innovation can occur 'hen principles such as participation and usefulness drive the choice of methods+ rather than fi)ed ideas about 'hat others Bnotably scientists and policy ma&ersE 'ould consider acceptableD* >e consider some alternatives to indicators later in this section*

)y es of indicators
here are different types of indicators+ and many 'ays to thin& about them* Indicators need to measure physical and visible BmeasurableE outcomes+ but also changes in attitudes and behaviour+ 'hich is often less tangible and not al'ays amenable to counting* >hile @uantitative indicators are emphasised in mainstream 12( approaches+ for C.D they often need to be @ualitative to be most effective and appropriate* :yrne et al* B"$$/7 -E e)plain that Cthe most important indicators are often not @uantifiable* #or e)ample+ the number of people participating in a social net'or& is relatively unimportant compared to the 5uality of !$-

relationships and dialogue 'ithin that net'or&D* 9ualitative indicators are generally more descriptive* Indicators can be thought of as input+ process+ output and outcomeIimpact indicators BD%NID%+ "$$/7 !"E+ or as baseline+ process+ short-term+ intermediate and long-term outcomeIimpact indicators B>ebb 2 (lliott+ "$$"E+ or more simply as process and outcome indicators B#igueroa et al*+ "$$"E+ or output and impactIoutcome indicators BGuiAt+ !66-+ cited in Classen+ "$$,7 ".E* >ebb and (lliott B"$$"E define the indicators used in different stages of an GI?I%ID8 intervention7 0aseline indicators( Used at the pre-intervention+ participatory e)ploratory research or e)isting data revie' stage* ()amples include e)isting attitudes and self-reported behaviours* Process indicators( Used at the stage of trainingIparticipation+ information distribution or service provision* ()amples include the number of people trained* he initial identification of process indicators Cis essential+ to allo' documentation of inputs+ activities+ outputs+ numbers of beneficiaries and coverage* he &ey point is that impact assessment is virtually impossible 'ithout good information as to 'hat proAects have actually done$* Intermediate indicators( Used at the short-term+ post-activity stage* ()amples include short-term changes in &no'ledge and attitudes 'ithin the target group* C he purpose of intermediate indicators is not to measure the behaviour+ but to be predictors of the behaviourD* .ong-term<outcomes<im acts indicators( Used at the long-term+ sustained activity stage* ()amples include maintenance of positive self-reported behaviours and changed socialIpeer norms* Go'ever+ Cmeasuring the long-term impact of GI?-prevention programmes is not easyD B>ebb 2 (lliott+ "$$"7 ,4-,-E* ()amples of different types of indicators for monitoring and evaluating communication for social change provided by D%NID% B"$$/E are7 In ut indictor( #unds covering the planned communication activities 9ualified staff Process indicator( Number of participatory radio programmes aired Number of people reached through popular theatre activities 1ut ut indicator( Percentage of participants by men and 'omen ()posure to needed informationImessages ()panded public and private dialogue 1utcome<im act indicator( !$6

 IC increasingly used for dialogue and debate Percentage of men and 'omen 'ho &no' about voting procedures (strella B"$$$7 6E e)plains that+ C'hile there are no set rules to select indicators+ one guideline is to use the acronym C81%0 D7 indicators should be specific+ measurable+ actionoriented+ relevant+ and time-boundD* his tends to suit @uantitative indicators in particular* %nother contrasting acronym recently offered is C8PIC(DD7 CsubAective+ participatory+ interpreted+ communicable+ empo'ering and disaggregatedD B(strella+ "$$$7 6E* %s (strella points out+ the acronym 8PIC(D Creflects a shift to'ards placing greater emphasis on developing indicators that sta&eholders can define and use directly for their o'n purposes of interpreting and learning about changeD B"$$$7 6E+ rather than simply measuring or attempting to demonstrate impact for donors* 8outer B"$$-7 !3-E considers that 81%0 describes the properties of the indicators themselves+ 'hile 8PIC(D relates more to ho' indicators should be used7 /M*R) indicators 8pecific Bto the change being measuredE 1easurable Band unambiguousE %ttainable Band sensitiveE 0elevant Band easy to collectE ime bound B'ith term dates for measurementE /PICED indicators 8ubAective Participatory Interpreted Band communicableE Cross-chec&ed (mpo'ering Diverse and disaggregated 8outer ma&es the point that many of the terms in the table are ambiguous+ and these meanings need to be e)plored by agencies both 'ithin their o'n organisation and the conte)t of the intervention* (ssentially+ indicator choice depends on 'hat the sta&eholders 'ant to measure* #urthermore+ the obAectives and information needed depends on the conte)t BPar&s et al*+ "$$/7 !-E*

Indicators for C-D im act

In preparation for the !!th UN Inter-%gency 0ound able on Communication for Development+ Puddephatt et al* B"$$67 "--,"E drafted a set of outline indicators for assessing the effectiveness of C.D* hey advocate a 81%0 frame'or&+ and a toolbo) through 'hich the most relevant indicators and approaches could be developed* #ive principal C.D CresultsD and some &ey @uestions are presented as a mechanism for developing relevant indicators* he C.D results are7 Result %7 the level of local a'areness about the development program and the issues covered by the initiative Result #7 evidence of direct impact Result 57 participation and empo'erment !!$

Result -7 level of media coverage Result 67 country capacity In the Ctool&itD approach they ta&e+ these results are placed in a table+ along 'ith &ey @uestions for each result+ and suggestions for the focus of the indicators* 1ethodologies thought to be appropriate for each are also suggested* #or e)ample+ the first result is lin&ed to three &ey @uestions7 !* %re local sta&eholders a'are that the development program is in placeV "* %re they a'are 'hat the goals of the program areV ,* Gas &no'ledge of the issues covered by the program increasedV #or each &ey @uestion a possible focus is suggested+ so for @uestion ! the suggested focus is7 (vidence of local communication about the program J e*g* survey data* %n appropriate methodology put for'ard for this focus is C5%:P surveys using mar&et research tools such as surveys+ and e)perimental impact evaluation studiesD BPuddephatt et al*+ "$$67 ,!E* In our consultations in preparation for this revie'+ 'e consulted 'ith the UN #ocal Points and ()pert Panel about this approach to indicators* >hile t'o ()pert Panel members suggested that the list of indicators 'as CfineD or C@uite comprehensiveD+ one of them suggested that Ctypical agenciesD and specific e)amples should also be givenD* %nother thought the list corresponded 'ith the principle of defining simple indicators but emphasised the need to define indicators 'ith local sta&eholders* =ne of them also appreciated the advice about using indicators sparingly and fle)ibly* Go'ever+ some problems 'ere raised 'ith the C.D results areas+ the indicators+ and the assignment of appropriate methodologies* It 'as thought by some that the C.D results could be better lin&ed to the definitions of C.D identified earlier in the Discussion Paper* =ne of the ()pert Panel also commented that these result areas do not seem aligned to the definitions of C.D put for'ard in the 0ome Consensus+ 'hich emphasised Systematic& planned& evidence/based& strategic& integral to programmes& participation& networks etc@ Almost all of these definitions relate to the ;9+ 'hich this respondent does not see reflected in the results areas+ and that Key indicators intended to be universally applied across the field should be based around those elements@ 8pecific comments on the five results areas included7 Result %7 .evel of local a'areness a"out the develo ment rogram and the issues covered "y the initiative( Doesn$t this depend on the scale of the program& the intended primary beneficiaries and other stakeholders to be involved: ,mportantly& is it enough to 7be aware of. the program& its aims and ob6ectives etc ((( the more important 5uestion is shifts in the awareness of those directly involved with the program on certain critical and focus issues( !!!

Result #7 Evidence of direct im act7 So much impact of programmes can be indirect ((( e(g(& from our community dialogues on ;,A3A,DS issues& if young people participating change their attitudes and behaviour apparently 0but not solelyB1 as a result of the initiative& good( 4ut what about their family and friend networks whom they might influence positively: ,n communities where community dialogues have been running for some time there are often significant knock/on effects that result( Another problem with *direct$ impact is that it automatically downplays the significance of wider conte'tual factors( )his is the point about attribution vs( contribution towards( ,n all likelihood& the dialogues contributed towards positive change& but not in isolation from other factors( Result 57 Partici ation and em o'erment7 )he precise stakeholders cannot be specified in advance& it will depend on conte't and program aims( CD indicators could be developed around each of these dimensions& if they are key to the program ((( needs lots more thought to come up with nuanced and widely/appropriate indicators of participation and empowerment that might be relevant to all C%D programmes( )his could also include empowerment more broadly: ;as the pro6ect built people$s capacities to speak out and participate more in other areas: ;as it established new information sources or new communication flows between classes of stakeholders which might improve accountability and dialogue generally: ;as it contributed to changing cultural norms around communication and power: Etc( Result -7 .evel of media coverage7 )he definitions of C%D above would not suggest that this is of overriding importance for every C%D initiative& given all the other how dimensions mentioned in the definitions( )his is one of the easier/to/measure aspects& but not necessarily one of the most significant( Result 67 Country ca acity7 )his might be critical for C%D in !/wide C%D initiatives& but this is not apparent from the definitions above( +hat about critical issues of capacity at every level& including country: And how would each of these best be assessed over time& given how long effective and sustainable capacity development takes: %nother approach to C.D indicators is provided by an integrated model for measuring the process for C#8C and its outcomes+ published by the 0oc&efeller #oundation in "$$" B#igueroa et al*+ "$$"E* he Communication Initiative 'ebsite features a set of si) indicators for measuring social change communication+ to e)plore the development of indicators from a social change and social movement perspective Bhttp7II'''*comminit*comIenInodeI!4.4I,.4E+ dra'n from this 'or& 'ithin the communication for social change arena* he C#8C Consortium continues to put for'ard this set of si) indicators BGrey-#elder 2 Deane+ !6667 "!-""F Par&s et al*+ "$$/7 ,!E7 !* ()panded public and private dialogue and debate* "* Increased accuracy of the information that people share in the dialogueIdebate* ,* 8upported the people centrally affected by an issueLsM voicing their perspective in the debate and dialogue* .* Increased leadership role by people disadvantaged by the issues of concern* /* 0esonates 'ith the maAor issues of interest to peopleDs everyday interests*

!!"

3* Lin&ed people and groups 'ith similar interests 'ho might other'ise not be in contact* hese indicators include both process and outcome indicators* In the long-term+ the purpose is to measure positive change in the issues of concern+ 'hether that is greater gender e@uality+ less GI?I%ID8+ better nutrition+ and so on* %s social change can ta&e a long time+ progress to'ard long-term social change can at times be an acceptable measure of effectiveness* BPar&s et al*+ "$$/7 !4E* >hen measured in the short-term+ sta&eholders need indicators that indicate a strong li&elihood of short-term change in the issue being addressed and are applicable across issues* In communication research intent to change has been used as a predictor of actual change BPar&s et al*+ "$$/7 !-E* % further issue is that+ as D%NID% notes+ C.D interventions are e)tremely gender sensitive* In most countries men and 'omen have une@ual access to information and freedom of e)pression+ and Cit is necessary to mainstream gender in all C.D interventions* his means that indicators on communication for development interventions should reflect the need for se) disaggregated dataD BD%NID%+ "$$/7 !,E*

Indicators and the UN C-D a

roaches

here are several challenges involved in developing indicators+ some of 'hich are discussed in the latter part of this section* In terms of generating+ and gaining consensus on+ broad or general indicators for C.D+ and approaches that support them+ 'e might usefully thin& about the main approaches the UN ta&es to C.D as described in Communication for Development: Strengthening the Effectiveness of the nited !ations B1cCall et al*+ "$!$E* 0ehaviour Change Communication %s >ebb and (lliott B"$$"7 ,4-,-E e)plain+ different types of indicators can be used at different stages of a :CC program+ including health related behaviour change* hese stages might include !* Pre-intervention+ to e)plore attitudes and self reported behaviours+ 'here baseline indicators can be usedF "* Documentation of inputs+ activities and outputs+ beneficiaries and coverage J i*e* 'hat the proAect or programme has actually done+ 'hich re@uires process indicatorsF ,* 1easures of short-term changes+ for e)ample in attitudes and behaviours+ 'hich re@uires intermediate indicators+ and can predict behaviour changeF .* Long-term indicators of change+ at individual and social levels+ in reported behaviour and social norms* :oth 81%0 and 8PIC(D indicators 'ould be appropriate to this approach+ and both @ualitative and @uantitative methods* Communication for /ocial Change In the C#8C approach+ indicators+ li&e 12( @uestions+ measures and methods should be developed 'ith those most affected and involved rather than pre-determined BPar&s et al*+ "$$/7 !E* %s 'ell as thin&ing about 'hat indicators are relevant+ 'ho should develop and use these indicators 'ill be a central consideration* he process is as important as the results* % 'ide group of sta&eholders 'ould be involved+ facilitated by inclusive processes and dialogue+ and an empo'ering approach* 8PIC(D indicators 'ould be particularly suitable to this approach+ 'hich is li&ely to dra' heavily on @ualitative and descriptive measures* 81%0 indicators and @uantitative approaches also have a role+ depending on the indicators and their purposes as designed through participatory processes* !!,

Communication for advocacy % similar approach to that used in C#8C could be adopted here+ given the focus on change in po'er relations and social relations* Communication for advocacy tends to re@uire an evaluation that measures both e)ternal and internal changes B1orariu et al*+ "$$67 /E* ()ternal changes consist of things li&e support through partnerships and development of leadersF improved a'areness and response to opponents of the desired change as 'ell as political+ social and economic environmentsF and+ progress 'ith decision ma&ers* Internal changes refer to the development of capacity to effect e)ternal changes* % learningfocussed evaluation approach to assess internal change is recommended* 8PIC(D indicators 'ould in general seem more suitable than 81%0 indicators for communication for advocacy* 8ince communication for advocacy might be 'or&ing to'ard large-scale political or environmental change+ including changes in media and communication environments+ the &inds of comprehensive indicators developed by UN(8C= Bsee belo'E might also be appropriate in some cases* Go'ever+ it is 'orth noting that advocacy evaluation+ as recommended by 1orariu et al* B"$$6E+ does not include indicators at all* he challenges presented by advocacy evaluation are the long time frame that is often re@uired before changes through advocacy campaigns are evidentF the need for advocacy 'or& to be sustained over long periods of timeF the need to pay attention to contribution+ not attribution as 'e discussed in 8ection . of this reportF and+ the need for interim measures of success due to the long-term goals of advocacy B1orariu et al*+ "$$67 !E* hey recommend an --step approach to advocacy evaluation+ 'hich includes a heory of Change step Bstep ,E to agree on the ultimate goals and interim outcomes+ follo'ed by a step to determine 'hat is to be measured+ including both e)ternal and internal changes* % learning-focused methodology is recommended+ because the ultimate goal is to enable advocates themselves to ma&e more informed decisions and provide information to decision-ma&ers* In this 'ay it is less about proving impact+ than learning from evaluation for more effective 'or& to'ards long-term goals* /trengthening an ena"ling media and communication environment UN(8C= B"$$-E produced a comprehensive set of media development indicators+ based on the obAectives of promotion of freedom of e)pression and media pluralismF development of community mediaF and human resource development Bcapacity building of media professionals and institutional capacity buildingE* here are five maAor categories for analysing the media development of a country+ and for each of these there is a set of &ey indicators* he categories are7 Category %7 % system of regulation conducive to freedom of e)pression+ pluralism and diversity of the media Category #( Plurality and diversity of media+ a level economic playing field and transparency of o'nership Category 5( 1edia as a platform for democratic discourse Category -( Professional capacity building and supporting institutions that underpins freedom of e)pression+ pluralism and diversity

!!.

Category 6( Infrastructural capacity is sufficient to support independent and pluralistic media (ach category is bro&en do'n into a set of issues+ 'hich each in turn contain a series of broad indicators* 8o+ for e)ample+ in category !+ there are four main issues identified+ each 'ith their o'n set of indicators and means of verification* 8o 'ithin the issue of CcensorshipD one indicator is identified as C he 8tate does not see& to bloc& or filter Internet content deemed sensitive or detrimentalD+ and the follo'ing set of more nuanced indicators are listed7 Internet content is not bloc&ed or filtered by the state because of its content or source Internet users are not subAect to sanctions for accessing or publishing content on the Internet deemed sensitive or detrimental Internet service providers+ 'ebsites+ blogs or Internet broadcasters are not re@uired to register 'ith or obtain permission from a public body

hese can be verified by loo&ing for7 Documented cases of Internet users being subAect to sanction for accessing or publishing content deemed sensitive or detrimental Documented cases of forced closures or threatened closures of 'ebsites Internet users subAect to sanction for accessing or posting ne's items or opinions deemed obAectionable (vidence of state activity in bloc&ing or filtering 'ebsites BUN(8C=+ "$$-7 !-E

>hile this e)ample can be seen to provide a fairly straightfor'ard set of visibly or @uantitatively measurable indicators+ another e)ample+ from category ,+ provides a different set of indicators more interested in measuring perceptions7 he issue is Clevels of public trust and confidence in the mediaD+ and one of the broad indicators is Cthe public displays high levels of trust and confidence in the mediaD* he more nuanced indicators are listed as7 Perception that the media reports on issues of real concern to people 8atisfaction 'ith the balance of local and national ne's and information Perception that Aournalists and media organisations have integrity and are not corrupt Perception that ne's reporting is fair and impartial % high level of citiHen participation in media as sho'n by7 the level of participation of audiences in tal&-bac& programmes+ space devoted to readersD comments in ne'spapers+ etc*

his can be verified as follo's7 Public opinion polls relating to trust and confidence in the media %ssessment of media by e*g* radio listening clubs Gousehold surveys and other field'or& relating to perceptions of the media Intervie's 'ith samples of listenersIvie'ersIreaders on their perceptions of the media (vidence of community involvement in evaluating community broadcasters !!/

BUN(8C=+ "$$-7 ."E

Challenges and alternatives to the use of indicators

he challenges involved in the use of indicators for C.D are numerous+ as 'ill be clear already* 8ection . of this literature revie' sets out ne' thin&ing and trends in 0+ 12( that are highly relevant to C.D* Gere 'e thin& about challenges concerning indicators in relation to some of this ne' thin&ing+ specifically in terms of comple)ity theory and participation* Aolism and com le+ity thinking Indicator setting is a comple) process+ attempting to provide indications of change in comple) conte)ts* Indicator setting can occupy a lot of time since they are Cappro)imations of comple) processes+ events or trendsD BGuiAt+ "$$$7 "$,E* Par&s et al* B"$$/7 !4E echo this statement+ going on to suggest that indicators can Cmeasure the tangible Be*g* service upta&eE+ the intangible Be*g* community empo'ermentE and the unanticipated Bi*e*+ results that 'ere not plannedE* Ideally indicators reveal changes related to a specific phenomenon that in itself represents a bigger @uestion or problemD* %t the same time+ and precisely because of the time and comple)ity involved+ Ca small number of meaningful indicators 'hich can be loo&ed at regularly and carefully is more useful than a long list 'hich is too timeconsuming to useD B8ave the Children+ @uoted in 8outer+ "$$-7 !3-E* %s e)plored in 8ection .+ an approach that encompasses comple)ity recognises the need to understand rather than measure social change processes BLacayo+ "$$3E* 8uch an approach re@uires loo&ing for different 'ays to do things+ as&ing different @uestions to get different ans'ers+ trying different strategies+ understanding the importance of conte)t+ and ho' and 'hy social change happens BLacayo+ "$$37 .-E* he re@uirement therefore is for indicators that are fle)ible and encompass comple)ity+ or+ an alternative to indicators* Creative approaches such as those using pictures or stories can be particularly useful 'ith 'or&ing 'ith community groups that include those 'ith lo' levels of literacy* Par&s et al* B"$$/7 ""E insist that C'e should be 'ary of forcing C#8C initiatives into e)isting or predetermined monitoring and evaluation frame'or&s and systemsD+ and remind us that+ as Chapman and >ameyo B"$$!E note+ frame'or&s can help us gain an overvie' of areas to loo& at and can facilitating creative thin&ing+ but the challenge Cis to remain open to unintended outcomes that fall outside the frame'or& of assessment that may have been adoptedD Bcited in Par&s et al*+ "$$/7 ""E* %s one of the ()pert Panel commented7 ;owever strong certain indicators might be& their ability to meaningfully capture comple'& dynamic processes of social change& over time and in multiple and diverse conte'ts& will always be limited( GuiAt B"$$$E provides t'o suggestions for alternatives to indicators+ Csignificant changeD and Cverifying assumptionsD* %s an e)ample of the significant change approach GuiAt cites Davies B!66-E 'ho describes a Christian Commission for Development in :angladesh BCCD:E initiative* % net'or& of credit groups+ funded by CCD:+ provided monthly reports 'hich detail the single most significant change that occurred amongst the group members* hese changes relate to people;s 'ell-being+ sustainability of people;s institutions+ people;s participation+ and one other open-ended change+

!!3

he report as&s for the ;facts; B'hat+ 'hen+ 'here+ 'ith 'homE and an e)planation of 'hy that change is the most significant one of all the changes that have occurred* his last aspect ensures a process of reflection and learning by the group members+ an aspect that is missing from most 12( systems that see& numeric data 'ithout any interpretation of the numbers* 8o+ instead of pre-determined @uestions+ CCD:;s monitoring aims to find significant e)amples related to its long-term development obAectives BDavies+ !66-+ in GuiAt+ "$$$E* he strengths and limitations of the 1ost 8ignificant Change techni@ue+ 'hich 'as later developed by Davies and Dart B"$$/E+ 'ere outlined in 8ection 4* 18C 'as previously described as Cmonitoring-'ithout-indicatorsD BDavies 2 Dart+ "$$/7 -E* In a criti@ue of indicators+ one of the ()pert Panel emphasised the benefits of this alternative approach7 ,f capacity is low& time and essential support and resources for research and evaluation are minimal3inade5uate& we have found that stories of change for e'ample can reveal far more& more accurately and in timely ways& than can indicators( And that$s before you get to the strengths and benefits of participation in the process itselfB Partici atory a roaches

%n e)ample of Cverifying assumptionsD is provided via GarnmeiAer B!666E 'ho promotes a fle)ible and creative approach to evaluation that prioritises participation* In the 8mall Dam 0ehabilitation ProAect+ implemented by C%0( international in Yimbab'e+ an in-depth revie' 'as considered more appropriate than a conventional+ broad evaluation* he evaluation team 'as made up of one e)ternal evaluator based in Yimbab'e+ and t'o Yimbab'ean consultants* he evaluation 'as designed through a 'or&shop+ 'hich included field staff* he evaluation 'as then conducted through a series of 'or&shops 'ith dam users* he evaluation team had identified ten proAect assumptions about e)pected changes+ including C-erceived impacts: Improved nutrition and income security are the main benefits perceived by users of the proAect damsD BGarnmeiAer+ !6667 "E* he idea 'as to 'or& 'ith local dam users to find evidence to support+ refine or reAect these assumptions* #indings from the evaluation led to revision of &ey assumptions found to be untenable through the evaluation* 8imply measuring impact against pre-determined indicators+ based on those assumptions+ 'ould have failed to have allo'ed the initiative to adAust its assumptions and improve its practices+ 'hich made it more li&ely to succeed in the future* %nother e)ample of an impact assessment of a rural development proAect in ?ietnam 'hich 'as underta&en 'ithout indicators is provided by Innocenti B"$$-E* his process involved Ca deep assessment of proAect history and consolidation of the information ta&en in the monitoring reports produced during the proAect life timeD and 'as based on a basic principle of participatory evaluation Cinvestigating impact through sta&eholdersD perception of changeD BInnocenti+ "$$-7 4E* Ge suggested that tools such as 18C could have strengthened the process and that one of the constraints 'as Cthe relatively comple) analysis and design of the overall methodologies usedD* Go'ever+ a strength of the process 'as that it 'as Crapid and cost effectiveD BInnocenti+ "$$-7 6E* %s Par&s et al* B"$$/7 !4E remind us+ selecting indicators is one of the most difficult steps in setting up a P12( approach as it Chighlights+ more than any other+ the different information needs and e)pectations that the different sta&eholders have of the monitoring 'or&D* !!4

Participatory approaches to C.D+ and 12(+ re@uire that the development of indicators Cfocus not Aust on 'hat is measured+ but also on ho' it is measured+ and especially on 'ho decides 'hich indicators are importantD B:ennett 2 0oche+ "$$$7 "3E* GuiAt B"$$$E provides various e)amples of participatory indicator identification and states that Cin each of these e)periences indicators 'ere identified by primary sta&eholders+ often local people 'ho live 'ith the changes being trac&edD* 8he goes on to recommend that indicators do not have to be perfect+ Conly sufficiently relevant and accurate for those 'ho are going to interpret the information to be able to do soD* (strella and Gaventa B!66-E e)plain the difference in approach to indicators bet'een conventional evaluation and participatory evaluation as predetermined indicators of success+ principally cost and production outputs+ compared to indicators of success identified by people themselves+ 'hich may or may not include cost and production outputs* % participatory approach re@uires recognition of different registers of success* Par&s et al* B"$$/7 ""E suggest that it may 'ell be that Ccombinations of locally-generated measurements and P12( processes and e)ternally derived indicators and 12( approaches are at times the most appropriate 'ay of monitoring and evaluating C#8C initiativesD* Gaving said that+ e)ternally derived indicators should only be used to stimulate discussion* Identification of indicators is best begun after dialogue about the communityDs concerns+ goals+ issues and obstacles and the vision of the change they see&* Par&s et al* B"$$/E cite 0oche B!666E 'ho proposes the follo'ing7 % multi-level approach including annual Audgments of impact by sta&eholders+ facilitated peer revie's+ independent evaluations and infre@uent long-term impact research* rac&ing and correlating change occurring at the level of individuals+ organisations+ and communities or societies as a 'hole* ()panding the possibilities for collecting evidence of 'hat is changing in peopleDs lives and 'hy for other people+ including partners+ but also from a 'ider range of people+ possibly using the Internet*

#ontalvo-GeraHo et al* B"$$4E used four se@uential activities to identify priorities and indicators for coastal management7 0emembering past problems and current solutions 8tating current problems hin&ing about desires for the future Defining indicators of change*

Conclusion
o conclude+ the literature revie'+ our survey and further consultations suggest a fe' &ey ideas concerning indicators and indicator setting in the C.D field7 Develop the types of indicators that are appropriate to your initiative+ through processes that are appropriate J lin& them very strongly to your aims and obAectives* 0elevant indicators should be developed 'ith input from relevant sta&eholders* !!-

5eep them manageable* 5eep them to a reasonable number* (nsure that they reflect the need for gender disaggregated data+ or data on other important differences* Understand that this means that indicators are not able to capture comple) realities and relationships J they are+ after all+ good 'ays of measuring change but not of capturing the reasons behind such change* Use methods that are appropriate for the type of indicator BtangibleIintangibleF processIoutcomeE* Consider indicators as Aust one part of a 0+ 12( strategy - they can allo' you to demonstrate progress to'ards defined obAectives+ but cannot tell you 'hy+ or 'hat this means to peopleDs lives* 8ome practitioners have begun loo&ing for alternatives to indicators+ such as stories of significant change* hey may be better 'ays to monitor significant and sometimes unanticipated or negative impacts associated 'ith long-term development goals*

!!6

;8 Conclusion and 'ays for'ard

Introduction and overvie'
his report has highlighted some significant trends+ challenges and approaches associated 'ith researching+ monitoring and evaluating Communication for Development 'ithin the UN and international development conte)t* >e begin this final section 'ith a summary of the &ey challenges+ tensions and issues that 'ere identified* Dra'ing on the ne' thin&ing and trends identified in this report+ 'e then present an emerging #rame'or& for 0esearch+ (valuation and 1onitoring of C.D* his frame'or& incorporates the principles for effective+ appropriate and sustainable 0+ 12( of C.D laid out in 8ection "* he frame'or& is divided into t'o parts7 !* Conceptual and theoretical frame'or& "* 1ethodological and reporting frame'or& Ne)t+ 'e list various strategies that aim to address the many challenges and issues that 'e identified in this research+ and to gradually refine and implement elements of this frame'or&* his clearly needs to be seen as a fle)ible+ long-term process of change+ given the many barriers to changing e)isting systems+ policies+ attitudes and practices that 'e have identified* >e recognise the need to be pragmatic and realistic about such a process of change+ given that effective implementation of alternative approaches is more difficult 'ithin hierarchical and bureaucratic organisational structures and cultures* >hile this report emphasises the benefits of alternative approaches to evaluating C.D+ 'e ac&no'ledge the strengths and limitations of both dominant and alternative approaches+ methodologies and methods* >e also emphasise the need for evaluation of C.D to use a fle)ible and open approach that dra's on complementary approaches and methodologies* #inally+ 'e outline plans 'hich 'ere developed during consultation meetings in Ne' <or& in December "$!$ for the further development and implementation of the 0+ 12( for C.D 0esource Pac& and related capacity development strategies+ over the ne)t five years and beyond*

2ey challenges, tensions and issues

his report has highlighted some significant challenges+ tensions+ issues and barriers in the areas of C.D+ 0+ 12( of C.D and evaluation capacity development 'ithin the UN and international development conte)t* he follo'ing summarises the &ey challenges+ tensions and issues that 'ere identified* * roaches to Communication for Develo ment

>e identified a shift Bin rhetoric at leastE from vertical+ one-'ay+ top-do'n models of C.D to horiHontal models that aim to facilitate participation+ inclusion and empo'erment* Go'ever+ many C.D approaches refer to both perspectives in contradictory 'ays+ resulting in !"$

confusion and inappropriate compromises that limit the effectiveness of C.D* >e also identified a recurring issue of decision ma&ers in development organisations not appreciating 'hat C.D actually means or its important role in the development process* Go'ever+ institutions that communities might engage 'ith through C.D are often structurally unsuited for listening to the community* #ull and direct participation 'as therefore seen as difficult to achieve 'ithin dominant organisational cultures+ practices and evaluation approaches* his is e)acerbated by the issues of po'er and inclusion that participation raises+ the politics involved+ and perceptions about the greater time and resources re@uired to implement alternative participatory approaches* * roaches to R, M3E

=ur revie' and consultations identified a lac& of funding and support for alternative 0+ 12( approaches that are often more appropriate for C.D* Policy ma&ers and managers 'ithin the UN system+ 'ho often have a hard science bac&ground+ tend to lac& an understanding and appreciation of the potential of alternative approaches and the value of participatory tools for eliciting information that is often more credible and useful to communities* hey @uestion the rigour and validity of alternative approaches+ 'hich are criticised for lac&ing CobAectivityD+ CrigourD and CvalidityD* >e also noted that participatory+ mi)ed methods approaches+ 'hich 'e have advocated in this report+ re@uire a 'ide range of s&ills and &no'ledge to use effectively* In addition+ there are particular challenges concerning rigour 'hen using this approach in some resource+ s&ill and time poor developing countries* *ssessing the im acts of C-D >e highlighted that demonstrating the impact of C.D is notably more comple) and difficult than for other types of development initiatives* his is because it involves assessing changes in ho' people thin& and respond to issues and their local conte)ts+ 'hich can be affected by many different factors* In addition+ it can be easier to isolate individual changes in the environment in other development initiatives B8outer+ "$$-E* Given the comple)ity of the change process+ social and behavioural change often needs to be assessed against a moving baseline+ 'hich is inconsistent 'ith dominant evaluation practices* %nother &ey challenge is the attribution of impact in a comple) and rapidly changing 'orld* he politics of aid means that agencies often inappropriately claim credit for impacts+ and reporting focuses more on CsuccessesD than on CfailuresD* =ur revie' and consultations indicated that 12( is often approached in a vertical rather than an integrated manner+ in 'hich it is considered right from the design and development stage+ and fully incorporated into the programme cycle* %s 'ith other research conducted for the UN C.D 0ound ables+ 'e identified issues 'ith inade@uate funding for 0+ 12(+ generally 'ea& research and evaluation capacities+ and inade@uate resources+ including time to underta&e impact assessment of C.D* Donors 'ere seen as often 'anting to see results in an unreasonably short time frame* hey 'ere also seen as reluctant to fund longitudinal studies* %s a result+ there is a lac& of strong evidence on 'hich to build C.D research and demonstrate its value* Indicators B'hich are often re@uired by donors and funding agenciesE 'ere seen as largely unable to capture comple) realities and relationships or to capture the reasons behind social change* hey are usually set 'ithout input from &ey participants+ are often @uantitative and unrealistic+ and do not fit C.D outcomes very 'ell*

!"!

Evaluation ca acity develo ment It can be difficult to develop evaluation capacities and Cbuy-inD and o'nership of 12( and C.D initiatives* his is especially difficult in pressured and resource constrained organisational conte)ts in developing countries* here can be cultural+ communication and language issues and barriers to effective (CD+ including issues of po'er and access to IC s* >e argued that managers need to act as models of learning+ and that greater funding and support is needed for long-term+ sustainable capacity development at all levels* Go'ever+ this is often difficult to achieve+ particularly for organisations based on hierarchical or bureaucratic structures and policies* >e identified numerous challenges and issues that have a particular impact on the effectiveness and sustainability of (CD in the C.D conte)t including7 )he diversity of C%D approaches+ 'hich affects capacity development and training needs* )he comple'ity inherent to assessing the impact of C%D: 1any staff 'or&ing on C.D programmes at country and field level 'ere considered not 'ell e@uipped to deal 'ell 'ith these challenges and comple)ities* Attitudes to M#E among donors+ C.D organisations and NG=s7 1any donors and mangers 'ere seen as not valuing alternative 12( approaches* he attitudes of some programme staff and management indicated a lac& of support for evaluation and a lac& of understanding of its important role in the programme design and development process* Maintaining& supporting and sustaining evaluation capacity( Gigh staff turnover and loss of change agents 'as considered to be a &ey challenge that can undermine (CD efforts in organisations developing and implementing C.D programmes* Eacilitating wide participation in M#E for C%D( %chieving a high level of participation by a 'ide range of sta&eholders in 0+ 12( of C.D can be difficult+ especially for time and resource poor organisations* Coordinating M#E with the programme content and improvement processes* he time re@uired for effective 12(+ including mi)ed methods data analysis and reporting+ may not match the needs of C.D programme developers+ 'ho often re@uire 12( findings more @uic&ly* Developing the wide range of skills re5uired( It is time consuming to develop the particular s&ills re@uired to effectively monitor+ evaluate and assess the impacts of C.D programmes+ especially 'hen using participatory+ mi)ed methods approaches* 8ack of practical and sustainable impact assessment frameworks for C%D* #rame'or&s often have to fit the narro' up'ard accountability demands of donors* his 'as seen as !""

incompatible 'ith effective impact assessment of C.D+ 'hich entails a long-term approach based on ongoing learning and programme improvement*

!rame'ork for research, monitoring and evaluation of C-D

Dra'ing on the ne' thin&ing and trends in this area that 'e have identified in this report+ 'e no' present an emerging #rame'or& for 0+ 12( of C.D that incorporates the principles for effective+ appropriate and sustainable 0+ 12( of C.D set out in 8ection "*

%8 Conce tual and theoretical frame'ork ' holistic perspecti e based on comple!ity and systems thinking he frame'or& highlights the value of ta&ing a Cbigger pictureD holistic perspective that dra's on comple)ity and systems thin&ing* 8ystems thin&ing and comple)ity theory are important to understand comple) social problems+ such as those that C.D addresses* hey provide better and more fle)ible and organic frame'or&s and strategies for understanding the dynamics of social change+ ho' and 'hy social change happens+ 'hat 'or&s and 'hat does not 'or& in different conte)ts+ and the system and conte)ts 'ithin 'hich evaluations are underta&en B:yrne+ "$$-F Lacayo+ "$$3E* =ne of the implications of adopting a holistic vie' Cis a recognition that any e)planation of a phenomenon 'ill not be able to point to single causes and effectsD BGearn et al*+ "$$67 ,3E* (akes the wider conte!t into account %dopting a holistic perspective based on systems thin&ing means that the 'ider conte)t and larger structural issues are ta&en into account* his entails gathering information and developing an in-depth understanding about the social+ economic+ political+ cultural+ communications+ technological and environmental conte)t* It involves paying attention to the particular conte)tual and cultural issues and barriers that affect a C.D initiative and are of concern and interest to beneficiaries or users of the initiative* he rapidly changing information and communications conte)t in 'hich C.D initiatives are implemented+ and different levels of access to and use of ne' communication technologies+ clearly affect the outcomes of these initiatives and their evaluation* he frame'or& highlights the value of using a range of complementary methodologies and methods to better understand these conte)tual issues* 4ocuses on gender, power and social norms he process the frame'or& advocates involves actively and e)plicitly addressing issues related to gender+ caste+ race and other differences+ and une@ual po'er and voice among participants* Giving value to diversity and difference and ta&ing an inclusive approach is seen as important to enabling a more ade@uate understanding of social problems and issues* It provides ne' insights and understanding of alternative perspectives B1organ 2 0amireH+ !6-.+ in Gearn et al*+ "$$6E* Ine@ualities in gender+ po'er and &no'ledge need to be more openly ac&no'ledged and ta&en into account in the evaluation process+ since this can have a maAor effect on the outcome of C.D programmes and their evaluation* !",

=ur literature revie' highlights the urgent need to focus attention on the importance of local social norms* %s :urns B"$$47 ,3E argues+ if interventions do not attend to local social norms+ Cmany policy initiatives 'ill fail to 'in community support+ rendering them unsustainableD* his has maAor implications for C.D programmes that aim to change harmful social and cultural practices such as child marriage and female genital mutilation and to prevent the spread of significant health problems such as GI?I%ID8* Changing harmful social norms is the area 'here there is li&ely to be the most cross-over bet'een mainstream and participatory approaches to 12(*

5olistic approach to e aluation capacity de elopment he frame'or& advocates ta&ing a long-term perspective on evaluation capacity development+ focusing on organisations as a 'hole and the development of learning organisations BGorton et al*+ "$$,E* In this approach+ staff at all levels and &ey sta&eholders Bincluding community membersE are involved in ongoing capacity development* his approach aims to address issues such as staff turnover and the sustainability of evaluation systems and capacities 'ithin organisations and sta&eholder groups* Capacity development should aim to increase understanding of the fundamentals of 0+ 12(+ and provide practical guidance and simple+ user-friendly information for managers+ field staff and community members* his re@uires the development of guides and other resources in close cooperation 'ith country-level UN staff and others 'ho 'ill use this information* E aluation as an ongoing learning and impro ement process >hen this alternative approach to evaluation is ta&en+ evaluation is seen an ongoing learning and programme improvement process that is integrated into programmes and organisations and the 'hole proAect cycle* It is also seen as an important means of strengthening capacity and improving organisational performance and relationships and understanding bet'een staff in different organisational areas and bet'een staff and &ey sta&eholders* he aim is to develop learning organisations that are constantly critically reflecting on their 'or&+ identifying 'ays that their programmes and practices can be improved+ and then putting these ne' strategies and plans into practice* his entails a greater focus on the impacts of the actual process of participating in an evaluation+ 'hich can result in positive changes to the capacity+ processes and culture of an organisation BGorton et al*+ "$$,F Patton+ !66-E* It also involves a 'ider range of staff and sta&eholders ta&ing responsibility for research and evaluation+ rather than it being mainly seen as the responsibility of 12( staff or specialists* Long#term, sustainable perspecti e %s 'e have indicated above+ this frame'or& ta&es a long-term perspective on 0+ 12( and evaluation capacity development in the C.D area* >hile a participatory+ mi)ed methods approach can re@uire more time and resources to be effective+ our revie' and consultations clearly indicate that a long-term vie' of the many benefits of these approaches is needed* he aim here is to ma&e 12( and (CD systems sustainable and effectively integrated into organisations and C.D programmes through the use of holistic+ participatory and systembased approaches* !".

#8 Methodological and re orting frame'ork -pen, fle!ible approach to designing e aluations and selecting methodologies and methods he #rame'or& responds to an identified need for greater openness+ freedom+ fle)ibility and realism in planning and designing C.D evaluations and in the selection of methodologies+ methods and indicators* %s 8outer B"$$-7 !4.E argues+ it Cis not possible to construct a single impact assessment model or frame'or& for ICD proAects *** because different methodologies are re@uired for the very different conte)ts and types of obAective involvedD* his process begins by sta&eholders agreeing on the initiativeDs obAectives and outcomes and clarifying the evaluationDs purpose and sta&eholder e)pectations of the evaluation* 1ethodologies and methods need to be appropriate and to fit the underlying aims and values of the C.D initiative* Ideally they 'ould be underpinned by the principles proposed in this report* hey also need to ta&e into account various constraints such as time+ resources and organisational challenges* Participatory+ @ualitative and mi)ed methods approaches and methods have enormous value+ if they are effectively used* It is necessary to ta&e a critical approach to selecting approaches+ methodologies and methods that is based on an a'areness of their strengths and limitations* his includes the strengths and limitations of both alternative and mainstream approaches to 0+ 12(* 6ses participatory approaches as appropriate Participatory approaches to 0+ 12( of C.D are particularly effective and appropriate in C.D programmes+ given that the &ey aims of C.D include facilitating community participation and empo'erment through t'o-'ay communication+ dialogue and information sharing* he many benefits of participatory approaches include fle)ibility+ the encouragement of ongoing critical reflection and learning+ strengthening of capacities+ and an increase in the utilisation of evaluation results* Go'ever+ 0+ 12( processes need to be meaningfully participatory and inclusive+ and 'ell facilitated+ to be most effective* he aim here is for participants to develop an o'nership of the initiative and its evaluation and to become active and e@ual partners in decision ma&ing+ 'hich is an honest and transparent process* %s 'ell as local o'nership+ participatory processes should aim to foster national o'nership of the initiative and the evaluation* Participatory 1onitoring and (valuation methodology B:yrne et al*+ "$$/F 1yers+ "$$/F Par&s et al*+ "$$/E is an effective 'ay of actively engaging &ey sta&eholders in all stages of the evaluation of C.D and strengthening evaluation capacities and o'nership of the process* 6ses a mi!ed methods approach and triangulation he evaluation of C.D needs to be based on an appropriate combination of @ualitative and @uantitative techni@ues+ complementary approaches and triangulation+ in recognition that different approaches are suitable for different issues and purposes+ and different approaches to C.D* % pragmatic+ mi)ed methods approach to research and evaluation often results in superior research and is appropriate for comple) development programmes B:amberger et al*+ "$!$F Johnson 2 =n'uegbuHie+ "$$.E* he many benefits of using a mi)ed methods approach and the triangulation of various data sources and types include that they can7 shed light on different issues+ provide much needed fle)ibility+ increase the strength and rigour of !"/

evaluation and impact assessment findings+ be more effective 'hen time and budgets are constrained+ and can contribute to developing @ualitative indicators B:amberger et al*+ "$!$E* (ffective triangulation also ensures that a diversity of voices and vie's are included in the evaluation* "mpact assessment uses contribution assessment and a dynamic theory of change and in ol es longitudinal studies It is often more important to focus on progress towards long-term social change and the contribution made by C.D+ rather than attempting to CproveD impacts of programmes in the short-term B:yrne+ "$$-E* his is a realistic measure of effectiveness that can provide practical and useful recommendations for the implementation of policies and programmes* his approach considers the short-term+ intermediate and long-term impacts of initiatives and uses methodologies such as =utcome 1apping B(arl et al*+ "$$!E* Given the comple)ity of the social change process and often rapidly changing social and communication conte)ts+ 'e identified the need to use a dynamic+ moving theory of change* (valuation of C.D also re@uires adAusting baseline information to recognise changes in the conte)t* (valuation and impact assessment data needs to be disaggregated by gender+ age+ ethnicity and other relevant differences in order to better understand impacts on particular groups+ especially the most vulnerable and disadvantaged* here is a need to fund and support longitudinal studies of C.D in order to better assess impacts and outcomes and to build a stronger evidence base* Longitudinal studies are the best 'ay to assess lasting and sustainable change* Makes more use of 2ualitati e and participatory indicators 9ualitative indicators are often the most effective and appropriate for assessing the impacts of C.D* In some C.D evaluations+ it may be more useful to use alternatives to indicators such as stories of significant change and Cverifying assumptionsD* here is a need to develop both locally and e)ternally derived indicators* Developing indicators in dialogue 'ith relevant community members and other sta&eholders provides more realistic and useful indicators of social and behavioural change* E aluation is independent and learns from )failures* %dopting a learning and improvement-based approach to evaluation means that the process is underpinned by a high level of independence+ integrity and honesty* here is a need for evaluation to be open to negative findings+ 'ea&nesses and CfailuresD+ as 'ell as Csuccess storiesD+ in order to learn from these findings* Establishes open communication and feedback systems his frame'or& includes the establishment of open communication and feedbac& systems and methods in order to share evaluation and impact assessment findings+ learnings+ outcomes and e)periences 'ith a range of staff and sta&eholders Bincluding donors and beneficiariesE* his involves openly reporting positive+ negative+ unplanned and une)pected results using a range of methods that match the needs of different groups* he frame'or& advocates the use of creative and engaging 'ays of communicating results and learnings such as digital storytelling and sharing stories of significant change 'here ever possible* !"3

Pro osed strategies to im lement the R, M3E for C-D frame'ork

Implementing both the conceptual and methodological parts of the frame'or& re@uires a clear strategy* %s this report has identified+ there are a number of challenges to be addressed in order for the 0esource Pac& for 0+ 12( of C.D that is no' being developed+ to succeed* he follo'ing elements of a strategy can be clearly identified+ not only from the revie' of literature and consultations underta&en as part of the development of the 0esource Pac&+ but also from various preparatory reports and discussions and recommendations from the UN 0ound ables for C.D over the past fe' years* *dvocating for C-D and R, M3E here is a clear need for advocacy across the UN and 'ith other organisations and donors+ to highlight the importance of C.D and 0+ 12( in development* % greater appreciation of 'hat C.D means is re@uired* It is necessary to demonstrate C.DDs contribution to development goals and results+ and the cost-effectiveness of investing in C.D* o do this+ there is a need to identify creative and innovative 0+ 12( approaches and e)amples to illustrate 'hat 'or&s best and to demonstrate the results of C.D and the rigour and effectiveness of alternative 0+ 12( approaches+ methodologies and methods* Greater understanding among mainstream evaluation specialists+ of the appropriateness and longterm benefits of participatory approaches is re@uired* his report has the potential to begin this process* Institutionalisation Connected to the need for advocacy+ creating a common understanding of C.D and its various benefits for different %gencies might facilitate collaborative research and evaluation bet'een those UN agencies that focuses on common C.D goals+ and changing social norms* hrough such collaboration it might be possible to provide sufficient budgets+ resources and time+ including for longitudinal studies* his report identifies the need to improve capacity in conceptualising+ managing and planning 0+ 12( of C.D 'ithin the UN+ and 'ith partners* Long-term capacity development for staff at all levels is re@uired+ 'ith high @uality and yet accessible training and reference resources* =n the one hand it is necessary to demonstrate the rigour+ usefulness and appropriateness of alternative and participatory approaches to 0+ 12( for C.D to mainstream evaluation specialists and managers* his re@uires a detailed and thorough presentation of current thin&ing and trends in the field+ and this report aims to deliver that* =n the other hand+ for those planning and implementing 0+ 12( of C.D on the ground+ evaluation guides+ methodologies+ methods and 12( systems need to be as practical+ userfriendly and simple as possible* his is the role of the Guide section of the 0esource Pac&+ 'hich is still in development* Given the inade@uate funding and resources available for 0+ 12( of C.D+ institutionalisation of the re@uired capacity development+ along 'ith the development of learning and support resources 'ould be highly beneficial and cost effective* 0roader vie', sharing of skills and kno'ledge !"4

his report has detailed a range of creative and innovative strategies to enhance sta&eholder engagement and participation in 0+ 12( of C.D* 8tepping bac&+ similar creative and innovative strategies might be employed to develop the frame'or&+ using participatory and collaborative methods* In addition to the need to collect and present good e)amples of highly effective 0+ 12( for C.D+ there is also benefit in underta&ing meta-evaluations of these e)amples+ to identify and frame issues+ improve evaluations and increase the utilisation of evaluation results* he open sharing of both positive and negative+ intended and une)pected findings needs to ta&e place 'ithin an environment that understands the huge benefit of learning from both success and failure* he consultations underta&en for this report recommend the establishment of a community of practice 'ith online access to e)pert advice*

!urther develo ment of the Resource Pack

%t the Ne' <or& consultation meeting+ plans 'ere developed to complete the 0esource Pac&+ and roll it out* >e have set out our recommendations for further development of the Guide section of the Pac& in the initial outline of the Guide* #rom 1ay "$!!+ e)tensive consultation of the Guide section of the Pac& 'ill ta&e place+ along 'ith testing* his 'ill involve receiving and incorporating feedbac& from programme staff in the field+ piloting the Guide in Nepal and other countries+ contingent on availability of funds and human resources+ convening a Aoint UN meeting in 5athmandu+ Nepal in mid"$!! to gain further feedbac& to develop a 'or&able version of the Guide+ to be presented at the !"th UN Inter-agency 0ound able BUN0 E on C.D in November "$!!* he version 'ill be developed around the theme C%dvancing the 0ights of %dolescent Girls through Communication for DevelopmentD+ 'hich is the theme for the !"th UN 0ound able on C.D* %fter incorporating feedbac& received at the UN0 a ClivingD 0esource Pac& 'ill be rolled out regionally from the end of "$!! until "$!/* his re@uires that the UN systemDs C.D focal points 'ill 'ish to ta&e this process for'ard and that resources are made available to do so* :et'een "$!/ and "$"$ a consolidation phase 'ill collect and incorporate feedbac&* Ne' post-1DG theories can be incorporated into a ne' version of the 0esource Pac&+ to bring it up to date*

!inal comments

his report is the first part of the 0esource Pac& to be completed* he ne)t part of the Pac& to be fully developed 'ill be the practical Guide to 0+ 12( for C.D for use across the UN* In collaboration 'ith the ()pert Panel for this proAect and other UN agencies+ UNIC(# 'ill also develop a Directory of specialists 'ho can deliver training and provide e)pert advice and support to those using the 0esource Pac&* he further development of the 0esource Pac& 'ill benefit enormously from the inter-agency approach adopted so far+ to great effect* It 'ill also depend on good communication and collaboration as 'ell as &ey CchampionsD across the different agencies to put these strategies and plans into action* !"-

References
%ndre's+ %* :* B!663E* 0ealising participant empo'erment in the evaluation of nonprofit 'omen;s services organisations* In D* #etterman+ 8* 5aftarian 2 %* >andersman B(ds*E+ Empowerment evaluation: Knowledge and tools for self/assessment # accountability* housand =a&s7 8age* %nyaegbunam+ C*+ 1efalopulos+ P* 2 1oetsabi+ * B"$$.E* Participatory rural communication appraisalF 8tarting 'ith people* % handboo&* B"nd edE* 0ome7 #ood and %griculture =rganiHation of the United Nations* http7II'''*fao*orgIsdIdimZ&n!IdocsIy/46,e$$*pdf %t&inson+ D* D*+ >ilson+ 1* 2 %vula+ D* B"$$/E* % participatory approach to building capacity of treatment programs to engage in evaluation* Evaluation and -rogram -lanning& CFB,E+ ,"6-,,.* :a all+ =* B"$$6E* he role of national+ regional and international evaluation organisations in strengthening country-led monitoring and evaluation systems In 1* 8egone B(d*E+ Country/led monitoring and evaluation systems: 4etter evidence& better policies& better development results* Bpp* !!6-!,.E* Geneva7 UNIC(#* http7IImymande*orgIV@Rvirtual :ailur+ 8* B"$$4E* he comple)ities of community participation in IC for development proAects* Paper presented at the Gth ,nternational Conference on Social ,mplications of Computers in Developing Countries* 8ao Paulo+ :raHil* :a&e'ell+ =* 2 Garbutt+ %* B"$$/E* he use and abuse of the logical frame'or& approach* 8toc&holm7 8'edish International Development Cooperation %gency* http7II''',*interscience*'iley*comIcgi-binIfullte)tI4"/$3-..IPD#8 %0 :alit+ 8* B"$!$aE* Communicating with decision makers( Glocal imes+ !.* http7II'ebHone*&,*mah*seIproAectsIgt"Ivie'article*asp)VarticleIDR!-!2issueIDR"! :alit+ 8* B"$!$bE* Communicating with decision makers 0continued1( Glocal imes+ !.* http7II'ebHone*&,*mah*seIproAectsIgt"Ivie'article*asp)VarticleIDR!-,2issueIDR"! :amberger+ 1* B"$$6E* ,nstitutionali=ing impact evaluation within the framework of a monitoring and evaluation system( >ashington DC+ U8%7 >orld :an&* http7IIsiteresources*'orldban&*orgI(O (?%C%PD(?I0esourcesI./-/34"!"/!.3!-4/.,"IinstZieZframe'or&Zme*pdf :amberger+ 1*+ 0ao+ ?* 2 >oolcoc&+ 1* B"$!$E* Using mi)ed methods in monitoring and evaluation7 ()periences from international development* 1anchester7 he >orld :an& Development 0esearch Group* http7II''''ds*'orldban&*orgIservletI>D8Content8erverI>D8PII:I"$!$I$,I",I$$$!/-,.6Z"$! $$,",!$$3"-I0enderedIPD#I>P8/"./*pdf :amberger+ 1*+ 0ugh+ J*+ 2 1abry+ L* B"$$3E* "eal world evaluation* housand =a&s7 8age* :ennett+ #* 2 0oche+ C* B"$$$E* Developing indicators7 he scope for participatory approaches+ !ew Economy& .+ ".-"-E* :hattacharya+ N* B"$$4E* Stories of significance: "edefining change( An assortment of community voices and articulations( % report based on an evaluation of the program Ccommunity driven approaches to address the feminisation of GI?I%ID8 in IndiaD by means of the C1ost 8ignificant ChangeD techni@ue Ne' Delhi7 he India GI?I%ID8 %lliance* http7II'''*aidsallianceindia*netIPublicationsI8toriesZofZsignifincanceZ0edefiningZcha nge*pdf

!"6

:oyle+ 0* B!666E* Professionalising the evaluation function7 Guman resource development and the building of evaluation capacity* In 0* :oyle 2 D* Lemaire B(ds*E+ 4uilding effective evaluation capacity( 8essons from practice( Bpp* !,/-!/!E* London7 ransaction Publishers* :runner+ I* 2 GuHman+ %* B!6-6E* Participatory evaluation7 % tool to assess proAects and empo'er people* In 0* #* Conner 2 1* Gendric&s B(ds*E+ ,nternational innovations in evaluation methodology: !ew directions for program evaluation& Bpp* 6-!4E* 8an #rancisco7 Jossey-:ass* :urns+ D* B"$$4E* Systemic action research( A strategy for whole system change( :ristol7 he Policy Press+ University of :ristol* :yrne+ %*+ Gray-#elder+ D*+ Gunt+ J* 2 Par&s+ >* B"$$/E* Measuring change: A guide to participatory monitoring and evaluation of communication for social change* Ne' Jersey7 Communication for 8ocial change Consortium* http7II'''*communicationforsocialchange*orgIpublications-resources*phpVidR"-" :yrne+ %* B"$$-E* (valuating social change and communication for social change7 Ne' perspectives* MAH,& IJ* http7II'''*communicationforsocialchange*orgImaHiarticles*phpVidR,-/ :yrne+ %* B"$$6aE* Pushing the boundaries7 Ne' thin&ing on ho' 'e evaluate* MAH,& IG http7II'''*communicationforsocialchange*orgImaHi-articles*phpVidR,66 :yrne+ %* B"$$6bE* Evaluating social change communication* :righton+ (ngland7 he Institute of Development 8tudies* http7II'''*communicationforsocialchange*orgIpdfsIevaluatingN"$socialN"$change N"$communicationN"$brightonN"$mayN"$$6N"$-N"$unaids*pdf Chambers+ 0* B!66"E* "ural appraisal: "apid& rela'ed and participatory& ,DS Discussion -aper KII( http7II'''*ids*ac*u&IgoIidspublicationIrural-appraisal-rapid-rela)ed-andparticipatory Chambers+ 0* 2 Petit+ J* B"$$.E* 8hifting po'er to ma&e a difference* In L* Groves 2 0* Ginton B(ds*E+ ,nclusive aid: Changing power and relationships in international development* London7 (arthscan* Chambers+ 0* B"$$-E* "evolutions in development in5uiry* London7 (arthscan* Chambers+ 0* B"$$6aE* 1a&ing the poor count7 Using participatory methods for impact evaluation* In G* >hite B(d*E* Designing impact evaluations: Different perspectives( >or&ing Paper .7 he International Initiative for Impact (valuation* http7II'''*,ieimpact*orgIadminIpdfsZpapersI/$*pdf Chambers+ 0* B"$$6bE* Using participatory methods to assess impacts* ()pert lecture presented at the -erspectives of ,mpact Evaluation: Approaches to Assessing Development Effectiveness ,nternational Conference* Cairo+ (gypt+ "6 1arch J " %pril "$$6* Chaplo'e+ 8* B"$$-E* 1onitoring and evaluation planning+ Monitoring and Evaluating Shortcuts: Luidelines on monitoring and evaluation planning* %merican 0ed Cross* http7IIcrsprogram@uality*orgIpubsI1(I1(shortcutZplanning*pdf Chavis+ D*+ Lee+ 5* 2 Jones+ (* B"$$!E* -rinciples for evaluating comprehensive community initiatives( http7II'''*capablecommunity*comIpubsIN#C?P$3"$$!*pdf Clar&+ 1* 2 8artorius+ 0* B"$$.E* Monitoring and evaluation: Some tools& methods and approaches* >ashington DC7 >orld :an&* http7II'''*'orldban&*orgIiegIecdImeZtoolsZandZapproaches*html

!,$

Classen+ L* 8* B"$$,E* Assessing sustainable rural development: A participatory impact assessment of the ,-CA pro6ect in ;onduras( Unpublished 1aster of %rts thesis+ University of Guelph+ Canada* Coo&e+ :* 2 5othari+ U* B(ds*E* B"$$!E* -articipation: )he new tyranny( London7 Yed :oo&s* Cooperrider+ D*+ >hitney+ D* 2 8travos+ J* B"$$,E* Appreciative in5uiry: )he handbook( 0Ist ed(1* :edford Geights+ =G7 La&eshore Communications and 8an #rancisco+ C%7 :errett-5oehlher* Cooperrider+ D* 2 >hitney+ D* B"$$/E* Appreciative in5uiry: A positive revolution in change* 8an #rancisco+ C%7 :errett - 5ohler* Corn'all+ %* B"$$$E* Making a difference: Lender and participatory development( ID8 discussion paper ,4-( :righton+ (ngland7 he Institute of Development 8tudies* http7II'''*ntd*co*u&Iidsboo&shopIdetails*aspVidR/64 Corn'all+ %* B"$$3E* Gistorical perspectives on participation in development* Commonwealth and Comparative -olitics& %%B!E+ 3"--,* Corn'all+ %* 2 Coelho*?* B"$$4E* Spaces for change: )he politics of citi=en participation in new democratic arenas* London7 Yed :oo&s* Crac&nell+ :* (* B"$$$E* Evaluating development aid: ,ssues& problems and solutions( housand =a&s7 8age* D%NID%* B"$$/E* Monitoring and indicators for communication for development* Copenhagen7 echnical %dvisory 8ervice+ 0oyal Danish 1inistry of #oreign %ffairs* http7II'ebHone*&,*mah*seIproAectsIcomdevIZcomdevZPD#ZdocIDanidaZComDevt*pdf Davies+ 0* B"$$.E* 8cale+ comple)ity and the representation of theories of change* Evaluation& IDB!E+ !$!-!"!* Davies+ 0* 2 Dart+ J* B"$$/E* )he *Most Significant Change$ 0MSC1 techni5ue( A guide to its use( http7II'''*mande*co*u&IdocsI18CGuide*pdf Deane+ J* B"$$.E* he conte)t of communication for development* Paper presented at the Gth nited !ations "oundtable on Communication for Development& 0ome+ Italy7 #%=* http7II'''*communicationforsocialchange*orgIpdfIroundtable*pdf Dearden+ P* B"$$!E* Programme and proAect cycle management BPPC1E7 Lessons from D#ID and other organisations Paper presented at the Eoundation for Advanced Studies for ,nternational Development* o&yo+ Japan* http7II'''*informa'orld*comIsmppIcontent[dbRall[contentRa4!.--,,6DiaH-Puente+ J* 1*+ <ague+ J* L* 2 %fonso+ %* B"$$-E* :uilding evaluation capacity in 8pain7 % case study of rural development and empo'erment in the (uropean Union* Evaluation "eview& KCB/E+ .4--/$3* DoHois+ (*+ Langlois+ 1*+ 2 :lanchet-Cohen+ N* B"$!$E* D( "$!7 % practitioner;s guide to developmental evaluation7 IIC0D* http7IImcconnellfoundation*caIassetsI1edia N"$LibraryIPublicationsID(N"$"$!N"$(N*pdf (arl+ 8*+ Carden+ #* 2 8mutylo+ * B"$$!E* 9utcome mapping: 4uilding learning and reflection into development programs* =tta'a7 ID0C* http7II'''*idrc*caIenIev-6,,$"$!-!-D=Z =PIC*html (arle+ L* B"$$"E* Lost in the matri)7 he logframe and the local picture* Paper presented at the ,!)"AC<s Mth Evaluation Conference: Measurement& Management and Accountability: he Netherlands* http7II'''*intrac*orgIdataIfilesIresourcesI!/.ILostin-the-1atri)- he-Logframe-and-the-Local-Picture*pdf (strella+ 1* B"$$$E* Introduction7 Learning from change* In 1* (strella et al* B(ds*E 8earning from change: ,ssues and e'periences in participatory monitoring and evaluation(

!,!

London7 Intermediate echnology Publications* http7II'''*idrc*caIopeneboo&sI-6/,IQpageZ"$! (strella+ 1*+ 2 Gaventa+ J* B!66-E* >ho counts realityV Participatory monitoring and evaluation7 % literature revie'* ,DS working paperN JD+ 4,* 0etrieved !6 July "$$4 from+ http7II'''*ntd*co*u&Iidsboo&shopIdetails*aspVidR."! #%= B"$$/E* Communication for development roundtable report: Eocus on sustainable development* 0ome+ Italy7 #ood and %griculture =rganiHation of the United Nations* http7II'''*fao*orgIdocrepI$$-Iy/6-,eIy/6-,e$$*htm #ee&+ >* 2 1orry+ C* B"$$6E* #itting the glass slipper\ Institutionalising communication for development 'ithin the UN BDiscussion documentE* Paper presented at the IIth ! inter/agency round table on communication for development* >ashington DC* U8%7 UNDP J >:* http7IIundg*orgIinde)*cfmVPR!!"/ #etterman+ D* B"$$!E* Eoundations of empowerment evaluation* housand =a&s+ C%7 8age* #etterman+ D* 2 >andersman+ %* B(ds*E* B"$$/E* Empowerment evaluation principles in practice( Ne' <or& he Guilford Press* #etterman+ D* B"$!$E* Empowerment Evaluation: Collaboration& action research and a case e'ample(http7IIpreval*orgIenIempo'erment-evaluation-collaboration-action-researchand-case-e)ample #igueroa+ 1*+ 5incaid+ D* L*+ 0ani+ 1* 2 Le'is+ G* B"$$"E* Communication for social change7 %n integrated model for measuring the process of its outcomes* In :* I* :yrd B(d*E+ Communication for social change +orking -aper Series& !o(I* Ne' <or&7 he 0oc&efeller #oundation* '''*communicationforsocialchange*orgIpdfIsocialchange*pdf #ontalvo-GeraHo+ 1* L*+ Glaser+ 1* 2 Lobato-0ibeiro+ %* B"$$4E* % method for the participatory design of an indicator system as a tool for local coastal management* 9cean and Coastal Management& MDB!$E+ 446-46/* #orss+ 5*+ 5ruse+ 8*+ aut+ 8* 2 enden+ (* B"$$3E* Chasing a ghostV %n essay on participatory evaluation and capacity development* Evaluation& ICB!E+ !"--!..* #raser+ C* 2 ?illett+ J* B!66.E* Communication: A key to human development( 0ome7 #%=* http7II'''*fao*orgIsdIcddirectIcdpubIsdrepub*htm #raser+ C* 2 0estrepo-(strada+ 8* B!66-E* Communicating for development( ;uman change for survival* London7 I*:* auris Publishers* Gergen+ 5* B"$$,E* %ction research and orders of democracy* Action "esearch+ IB!E+ ,6-/3* Gibbs+ D* %*+ Ga'&ins+ 8* 0*+ Clinton-8herrod+ %* 1* 2 Noonan+ 0* 5* B"$$6E* (mpo'ering programs 'ith evaluation technical assistance* =utcomes and lessons learned* ;ealth -romotion -ractice& IDB!E+ ,-/-../* Gosling+ L* 2 (d'ards+ 1* B(ds*E* B"$$,E* )oolkits: A practical guide to monitoring& evaluation& impact assessment* London7 8ave the Children #und* Gould+ J* B(d*E* B"$$/E* )he new conditionality: )he politics of poverty reduction strategies* London7 Yed :oo&s* Gray-#elder+ D* 2 Deane+ J* B!666E* Communication for social change7 % position paper and conference report* Ne' <or&7 he 0oc&efeller #oundation* '''* communicationforsocialchange*orgIpdfIpositionpaper*pdf Greene+ J* B"$$"E* 1i)ed-method evaluation7 % 'ay of democratically engaging 'ith difference* Evaluation Oournal of Australasia+ CB"E+ ",-"6* Greene+ J* 2 Caracelli+ ?* B"$$"E* 1a&ing paradigmatic sense of mi)ed-method practice* In %* asha&&ori 2 C* eddlie B(ds*E* ;andbook of mi'ed methods in social and behavioural research* housand =a&s7 8age* !,"

Gregory+ %* B"$$$E* Problematising participation* % critical revie' of approaches to participation in evaluation theory* Evaluation& PB"E+ !46-!66* GuiAt+ I* B"$$$E* 1ethodological issues in participatory monitoring and evaluation* In 1* (strella B(d*E+ 8earning from change: ,ssues and e'periences in participatory monitoring and evaluation* London7 Intermediate echnology Publications* http7II'''*idrc*caIopeneboo&sI-6/-,IQpageZ"$! Gumucio Dagron+ %* B"$$!E* Making waves: -articipatory communication for social change( Ne' <or&+ U8%7 0oc&efeller #oundation* Gumucio Dagron+ %* B"$$3E* he :abel challenge* MAH,+ 6* November "$$3* http7II'''*communicationforsocialchange*orgImaHi-articles*phpVidR,"/ Gumucio Dagron+ %* B"$$-E* ?ertical 1inds versus GoriHontal Cultures7 %n =vervie' of Participatory Process and ()periences* In J* 8ervaes B(d*E* Communication for development and social change* London7 8age* Gall+ :* B!66,E* Introduction* In P* Par&+ 1* :rydon-1iller+ :* Gall 2 * Jac&son B(ds*E+ Aoices of change: -articipatory research in the nited States and Canada B?ol* )i)iiE* London7 :ergin 2 Garvey* GarnmeiAer+ J* B!666E* #rom terms of reference to participatory learning7 Using an evaluation;s creative space+ -8A !otes KP* London7 II(D* Gearn+ G*+ acchi+ J*+ #oth+ 1* 2 Lennie+ J* B"$$6E* Action research and new media: Concepts& methods and cases( Cress&ill+ Ne' Jersey7 Gampton Press* Gee&s+ 0* 2 1olla+ %* B"$$6E* Compendium on impact assessment of IC -for-development proAects* Development ,nformatics +orking -aper Series* Paper No* ,3* Development Informatics Group+ Institute for Development Policy and 1anagement+ University of 1anchester* http7IIict.dblog*'ordpress*comI"$$-I!"I$,Iimpact-assessment-of-ict.d-proAectsI Golte-1c5enHie+ 1*+ #orde+ 8*+ 2 heobald+ 8* B"$$3E* Development of a participatory monitoring and evaluation strategy* Evaluation and -rogram -lanning& CG+ ,3/-,43* Gorton+ D*+ %le)a&i+ %*+ :ennett-Lartey+ 8* et al* B"$$,E* (valuating capacity development7 ()periences from research and development organiHations around the 'orld* )he ;ague: ,nternational Service for !ational Agricultural "esearch( http7II'''*idrc*caIenIev-,!//3-"$!-!-D=Z =PIC*html Inaga&i+ N* B"$$4E* Communicating the impact of communication for development* 0ecent trends in empirical research* Paper presented at the +orld congress on communication for development 0ome+ Italy7 he >orld :an&+ >or&ing Paper 8eries+ Development Communication Division* http7II''''ds*'orldban&*orgIe)ternalIdefaultI>D8Content8erverI>D8PII:I"$$4I$-I! $I$$$,!$3$4Z"$$4$-!$!",,$3I0enderedIPD#I.$/.,$Communic!-$-"!,4!34!$!P U:LIC!*pdf Innocenti+ D* B"$$-E* %ssessing impact 'ithout indicators+ case study from ?ietnam* Paper presented at the Krd MES ,nternational Evaluation Conference( +orkshop: ,mpact Evaluation: Challenges for Decision/making& Accountability and Quality* 5uala Lumpur+ 1alaysia* http7II'''*ideas-int*orgIdocumentsIDocuments*cfmVdocIDR!!! Jallov+ :* B"$$/E* %ssessing community change7 Development of a ;bare foot; impact assessment methodology* "adio Oournal: ,nternational Studies in 4roadcast # Audio Media& KB!E+ "!-,.* Jallov+ :* B"$$4aE* Impact assessment of (ast %frican community media proAect "$$$-"$$3* 0eport from 0adio 1ang;elete+ 5enya and selected communities* 8toc&holm7 :irgitte !,,

Jallov Communication Partners* http7II'ebHone*&,*mah*seIproAectsIcomdevIZcomdevZPD#ZdocIscp$-Zsem"ZImpactZ %ssessmentZ01*PD# Jallov+ :* B"$$4bE* 1ost 8ignificant Change7 % tool to document community radio impact* 1easuring change* Planning+ monitoring and evaluation in media and development cooperation* In %* 8* Jannuschm B(d*E+ Krd symposium forum on media and development( :ad Gonnef+ Germany7 Catholic 1edia Council*+ http7II'''*cameco*orgIfilesImeasuringZchangeZ!*pdf Jennings+ L* 2 Graham+ %* B!663E* ()posing discourses through action research* In =* Yuber-8&erritt B(d*E+ !ew directions in action research Bpp* !3/-!-!E* London7 he #armer Press* Johnson+ 0* 2 =n'uegbuHie+ %* B"$$.E* 1i)ed methods research7 % research paradigm 'hose time has come* Educational "esearcher& KKB4E+ !.-"3* 5eystone %ccountability B"$$6E* Developing a theory of change* % guide to developing a theory of change as a frame'or& for inclusive dialogue+ learning and accountability for social impact+ ,-A8 Luide C* London7 5eystone* http7II'''*&eystoneaccountability*orgIsitesIdefaultIfilesI"N"$DevelopingN"$a N"$theoryN"$ofN"$change*pdf 5hadar+ I* B"$$,E* 8haping evaluation practice through consultation and debate7 Implications for information programmes* ,nformation Development& IGB"E+ 6!-!$!* 5hagram+ 8* B"$$6E* Presentation on CComparative Bcase studyE evaluationD given as part of a one day Impact (valuation for Improving Development 'or&shop at the -erspectives of ,mpact Evaluation: Approaches to Assessing Development Effectiveness ,nternational Conference* Cairo+ (gypt+ "6 1arch J " %pril "$$6* 5han+ 1* B!66-E* (valuation capacity building* Evaluation& %B,E+ ,!$-,"-* 5otvoAs+ #* 2 8hrimpton+ :* B"$$4E* Contribution analysis* % ne' approach to evaluation in international development* Evaluation Oournal of Australasia+ JB!E+ "4-,/ Lacayo+ ?* B"$$3E* Approaching social change as a comple' problem in a world that treats it as a complicated oneN )he case of -untos De Encuentro& !icaragua( =hio University* 0etrieved from http7II'''*puntos*org*niIsidocIdescargasImar&etingImaterialesIenglishI%pproachingZs ocialZchangeZasZaZcomple)Zproblem*pdf Leal+ P* B"$$4E* Participation7 the ascendancy of a buHH'ord in the neo-liberal era Development in -ractice& IJB.E+ /,6-/.-* Leeu'+ #* 2 ?aessen+ J* B"$$6E* Impact evaluations and development* N=NI( guidance on impact evaluation* >ashington7 he Net'or& of Net'or&s on Impact (valuation* http7IIsiteresources*'orldban&*orgI(O =(DI0esourcesInonieZguidance*pdf Lennie+ J* B"$$/E* %n evaluation capacity-building process for sustainable community I initiatives* (mpo'ering and disempo'ering impacts* Evaluation& IIB/E+ ,6$-.!.* Lennie+ J* B"$$3E* Increasing the rigour and trust'orthiness of participatory evaluations7 Learnings from the field* Evaluation& PB!E+ "4-,/* Lennie+ J* B"$$6aE* 0eport on the -erspectives on ,mpact Evaluation: Approaches to Assessing Development Effectiveness ,nternational Conference( Creative Industries #aculty+ 9ueensland University of echnology* Lennie+ J* B"$$6bE* )roubling empowerment: A feminist criti5ue of a rural women and communication technology pro6ect( 8aarbruc&en+ Germany7 ?D1* Lennie+ J*+ 8&use+ %*+ acchi+ J* 2 >ilmore+ 1* B"$$-E* Challenges+ issues and contradictions in a participatory impact assessment proAect in Nepal* Paper !,.

presented at the Australasian Evaluation Society ,nternational Conference+ Perth+ !" 8eptember "$$-* http7II'''*aes*asn*auIconferencesI"$$-IpapersIp!.*pdf Lennie+ J*+ 8&use+ %*+ 5oirala+ :*+ 0iAal+ N* 2 >ilmore+ 1* B"$$6E* Developing a participatory impact assessment approach and action research culture 'ithin a communication for social change organisation in Nepal* Paper presented at the -erspectives on ,mpact Evaluation: Approaches to Assessing Development Effectiveness ,nternational Conference+ " %pril "$$6+ Cairo+ (gypt* http7II'''*,ieimpact*orgIcairoZconference*html Lennie+ J*+ acchi+ J* 2 >ilmore+ 1* B"$!$E* Critical reflections on the use of meta-evaluation in the C%ssessing Communication for 8ocial ChangeD proAect* Paper presented at the Australasian Evaluation Society ,nternational Conference* >ellington+ Ne' Yealand+ , 8eptember "$!$* http7II'''*aes*asn*auIconferencesI Lundgren+ G* 2 5ennedy+ 1* B"$$6E* 8upporting partner country o'nership and capacity in development evaluation* he =(CD evaluation net'or& In 1* 8egone B(d*E Country/ led monitoring and evaluation systemsN 4etter evidence& better policies& better development results Bpp* 44-6/E* Geneva7 UNIC(#* http7IImymande*orgIV@Rvirtual 1ac5ay+ 5* B"$$4E* ;ow to build M#E systems to support better government* >ashington DC7 >orld :an& Independent (valuation Group+ he >orld :an&* http7IIsiteresources*'orldban&*orgI(O (?%C%PD(?I0esourcesI./-/34"!"/!4,4,343-.IGo'ZtoZbuildZ1(Zgov*pdf 1arsden+ 0* B"$$.E* ()ploring po'er and relationships7 % perspective from Nepal* In L* Groves 2 0* Ginton B(ds*E ,nclusive Aid: Changing power and relationships in international development* Bpp* 64-!$4E* London7 (arthscan* 1artin+ 1* B!663E* Issues of po'er in the participatory research process* In 5* de 5oning 2 1* 1artin B(ds*E+ -articipatory research in health( ,ssues and e'periences Bpp* -"-6,E* London and Ne' Jersey7 Yed :oo&s* 1ayne+ J* B!666E* Addressing attribution through contribution analysis: sing performance measures sensibly( Discussion paper+ =ffice of the %uditor General of Canada+ June* 1ayou)+ L* 2 Chambers+ 0* B"$$/E* Policy arena7 0eversing the paradigm7 9uantification+ participatory methods and pro-poor impact assessment* Oournal of ,nternational Development& IJ+ "4!-"6-* 1cCall+ (* B"$!$E* Communication for development: Strengthening the effectiveness of the nited !ations: #%=+ IL=+ UN%ID8+ UNDP+ UN(8C=+ UNIC(# and >G=*
1c5ee+ N* B!66"E* Social mobili=ation and social marketing: 8essons for communicators * Penang7 8outhbound*

1c5ie+ L* B"$$,E* 0hetorical spaces7 Participation and pragmatism in the evaluation of community health 'or&* Evaluation& GB,E+ ,$4-,".* 1efalopulos+ P* B"$$/E* Communication for sustainable development7 %pplications and challenges* Bpp* ".4-"/6E* In =* Gemer 2 * ufte B(ds*E+ Media and glocal change: "ethinking communication for development* 8uecia7 Nordicom* 1ertens+ D* 2 Chilisa+ :* B"$$6E* 1i)ed methods and social transformation* >or&shop presented at the -erspectives on ,mpact Evaluation: Approaches to Assessing Development Effectiveness ,nternational Conference* Cairo+ (gypt+ "6 1arch J " %pril "$$6* http7II'''*,ieimpact*orgIcairoZconference*htmlQ'or&shop 1iller+ >* 2 Lennie+ J* B"$$/E* (mpo'erment evaluation7 % practical method for evaluating a national school brea&fast program* Evaluation Oournal of Australasia& MB"E+ !--"3*

!,/

1ongella+ G* B!66/E* 1oving beyond rhetoric* In +omen: 8ooking beyond CDDD* Ne' <or&7 United Nations Publications* 1orariu+ J*+ 0eed+ (*+ :rennan+ 5*+ 8tamp+ %*+ Parrish+ 8*+ Pan&aA+ ?* et al* B"$$6E* -athfinder evaluation edition( A practical guide to advocacy evaluation( >ashington DC7 Innovation Net'or& http7II'''*innonet*orgIclientZdocsI#ileIadvocacyIpathfinderZevaluatorZ'eb*pdf 1yers+ 1* B"$$/E* 1onitoring and evaluating information and communication for development BICDE programmes7 Guidelines* http7II'''*idrc*caIuploadsIuser8I!!/6"!$//-!icd-guidelines*pdf Naccarella+ L*+ Pir&is+ J*+ 5ohn+ #*+ 1orley+ :*+ :urgess+ P*+ 2 :lash&i+ G* B"$$4E* :uilding evaluation capacity7 Definitional and practical implications from an %ustralian case study* Evaluation and -rogram -lanning& KDB,E+ ",!-",3* Napp+ D*+ Gibbs+ D* Jolly+ D* et* al* B"$$"E* (valuation barriers and facilitators among community-based GI? prevention programs* A,DS Education and -revention& I% B8upplement %E+ ,--.-* Narayan+ D*+ Chambers+ 0*+ 5aul 8hah+ 1 2 Petesch+ P* B"$$$aE* Crying out for change( Aoices of the -oor series* =)ford7 =)ford University Press+ >orld :an&* http7II'eb*'orldban&*orgI>:8I (I(O (0N%LI =PIC8I(O P=?(0 <I$++content1D 57"$3!6"6,[isCU0L7<[menuP57,,366-[pageP57!.-6/3[piP57"!33!-[the8iteP57, ,366"[isCU0L7<+$$*html Narayan+ D*+ Patel+ 0* 8chafft+ 5*+ 0ademacher+ %* 2 5och-8chulte+8* B"$$$bE* Can anyone hear us: Aoices of the -oor Series* =)ford7 =)ford University Press+ >orld :an&* http7II'eb*'orldban&*orgI>:8I (I(O (0N%LI =PIC8I(O P=?(0 <I$++content1D 57"$3!6,$"[isCU0L7<[menuP57,,366-[pageP57!.-6/3[piP57"!33!-[the8iteP57, ,366"[isCU0L7<+$$*html Narayan+ D* 2 Petesch+ P* B"$$"E* Erom many lands( Aoices of the -oor series* =)ford7 =)ford University Press >orld :an&* http7II'eb*'orldban&*orgI>:8I (I(O (0N%LI =PIC8I(O P=?(0 <I$++content1D 57"$3!6,!/[isCU0L7<[menuP57,,366-[pageP57!.-6/3[piP57"!33!-[the8iteP57, ,366"[isCU0L7<+$$*html =;Donnell+ P*+ Lloyd+ P* 2 Dreher+ * B"$$6E* Listening+ path building and continuations7 % research agenda for the analysis of listening* Continuum& CKB.E+ .",-.,6* =leari+ 5* B"$$$E* 1a&ing your Aob easier7 Using 'hole-system approaches to involve the community in sustainable planning and development* -ublic Management+ December+ .!$* Papineau+ D* 2 5iely+ 1* B!663E* Participatory evaluation in a community organiHation7 #ostering sta&eholder empo'erment and utiliHation* Evaluation and -rogram -lanning& IGB!E+ 46-6,* Pamer+ :*+ Lennie+ J*+ 8tuart+ * 2 Claycomb+ P* B"$!!E* "eport of nited !ations inter/ agency and e'perts consultation on research& monitoring and evaluation in communication for development* 4-6 December "$!$+ UNIC(# Gead@uarters+ Ne' <or&* Par&s+ >*+ Gray-#elder+ D*+ Gunt+ J* 2 :yrne+ %* B"$$/E* +ho measures change: An introduction to participatory monitoring and evaluation of communication for social change( Ne' Jersey7 Communication for 8ocial Change Consortium* http7II'''*communicationforsocialchange*orgIpdfI'hoZmeasuresZchange*pdf Patton+ 1* B!664E* tili=ation/focussed evaluation* Ne'bury Par&7 C%7 8age* Patton+ 1* B!66-E* Discovering process use* Evaluation& %B"E+ ""/-",,* !,3

Porter+ 8* B"$$6E* % scan of the intersection bet'een evaluation and human rights* http7II'''*ideas-int*orgIdocumentsIfileZlist*cfmVDocs8ubCatIDR,. Pres&ill+ G* 2 :oyle+ 8* B"$$-E* 1ultidisciplinary model of evaluation capacity building* American Oournal of Evaluation& CGB.E+ ..,-./6* Puddephatt+ %*+ Gorse'ell+ 0* 2 1enheneott+ G* B"$$6E* Discussion paper on the monitoring and evaluation of UN-assisted communication for development programmes* 0ecommendations for best practice methodologies and indicators* Paper presented at the IIth ! inter/agency round table on communication for development( March IIth/ IKth >ashington DC7 UNDP - >orld :an&* http7II'''*communicationforsocialchange*orgIpdfsImonitoringN"$andN"$evaluation N"$ofN"$un-assistedN"$cfdN"$programmes*pdf 9uarry+ >* B"$$-E* Decision makers do want communication R but they may not want participation( Llocal )imes&!$* http7II'''*glocaltimes*&,*mah*seIvie'article*asp)V articleIDR!,-2issueIDR!3 9uarry+ >*+ 2 0amireH+ 0* B"$$6E* Communication for another development: 8istening before telling* London7 Yed :oo&s* 0aAbhandari+ :* B"$$3E* 8ustainable livelihood and rural development in south %sia* Issues+ concerns and general implications In 1* C* :ehera B(d*E+ Llobalising rural development: competing paradigms and emerging realities Bpp* "!!-".!E* Ne' Delhi7 8age* 0eed+ 1*+ #raser+ (* 2 Dougill+ %* B"$$3E* %n adaptive learning process for developing and applying sustainability indicators 'ith local communities* Ecological Economics+ /6 B.E+ .$3-.!-* 0egeer+ :*+ Goes+ %*+ 8anne+ 1*+ Caron-#linterman+ #* 2 :unders+ J* B"$$6E* 8i) guiding principles for evaluating mode-" strategies for sustainable development* American Oournal of Evaluation& KDB.E+ /!/-/,4* 0ice+ 0* 2 #oote+ D* B"$$!E* % systems-based evaluation planning model for health communication campaigns in developing countries* In 0* (* 0ice 2 C* %t&in B(ds*E+ -ublic communication campaigns 0Krd ed(1 Bpp* !.3-!/4E* housand =a&s+ C%7 8age* 0ogers+ (* B(d*E* B!643E* Communication and development: Critical perspectives* housand =a&s+ C%7 8age* 0ogers+ P* B"$$6E* 1atching impact evaluation designs to the nature of the intervention and the purpose of the evaluation* In G* >hite B(d*E* Designing impact evaluations: Different perspectives( >or&ing Paper .7 he International Initiative for Impact (valuation* http7II'''*,ieimpact*orgIadminIpdfsZpapersI/$*pdf 8an&ar+ 1* 2 >illiams+ :* B"$$4E* (ditorial J (valuation matters In :* >illiams 2 1* 8an&ar B(ds*E+ Evaluation South Asia* 5athmandu7 UNIC(#* http7IIunicef*orgIrosaI0=8%Z(valuationZJournal*pdf 8chensul+ J*+ :erg+ 1* 2 >illiamson+ 5* B"$$-E* Challenging hegemonies7 %dvancing collaboration in community-based participatory action research Collaborative Anthropologies& I+ !$"-!,4* 8chiavo-Campo+ 8* B"$$/E* :uilding country capacity for monitoring and evaluation in the public sector7 8elected lessons of international e)perience+ Evaluation Capacity Development& ECD +orking -aper Series !o( IK7 he >orld :an& =perations (valuation Department* http7IIpreval*orgIfilesI"$-3*pdf 8egone+ 1* B(d*E B"$$6E* Country/led monitoring and evaluation systems: 4etter evidence& better policies& better development results* Geneva7 UNIC(#* http7IImymande*orgIV @Rvirtual !,4

8ervaes+ J* 2 Liu+ 8* B(ds*E* B"$$4E* Moving targets: Mapping the paths between communication& technology and social change in communities( Penang7 8outhbound* 8ervaes+ J* B"$$4E* Garnessing the UN system into a common approach on communication for development* ,nternational Communication La=ette& PGB3E+ .-,-/$4* 8ervaes+ J* B(d*E* B"$$-E* Communication for development and social change* London7 8age* 8&use+ %* B"$$3E* Aoices for change: Strategic radio support for achieving the millennium development goals* London7 D#ID* http7II'''*bnnrc*netIresoucesIvoices-ofchange*pdf 8outer+ D* B"$$-E* Investigation .7 Impact assessment* In 4C9 impact assessment study( )he final report* Bpp* !/,-!6$E* :uilding Communication =pportunities %lliance* '''*bcoalliance*orgIsystemIfilesI:C=Z#inal0eport*pdf 8ullivan+ G*+ :arnes+ 1* 2 1at&a+ (* B"$$"E* :uilding collaborative capacity through Ctheories of changeD* Evaluation& FB"E+ "$/-""3* acchi+ J* BforthcomingE =pen content creation7 the issues of voice and the challenges of listening* !ew Media and Society( acchi+ J*+ 8later+ D* 2 Gearn+ G* B"$$,E* Ethnographic action research handbook* Ne' Delhi+ India7 UN(8C=* acchi+ J* #ildes+ J*+ 1artin+ 5*+ 1ulenahalli+ 5*+ :aulch+ (* 2 8&use+ %* B"$$4E* Ethnographic action research training handbook( http7IIear*findingavoice*orgI acchi+ J* 2 5iran+ 1*8* B(ds*E* B"$$-E* Einding a voice: )hemes and discussions( Ne' Delhi7 UN(8C=* http7II'''*findingavoice*orgIfilesIfindingZaZvoiceZthemesZandZdiscussions*pdf acchi+ J* B"$$6E* #inding a ?oice7 Digital storytelling as participatory development* In J* Gartley 2 5* 1c>illiam B(ds*E Story Circle: Digital storytelling around the world* Bpp* !34-!4/E* =)ford7 >iley-:lac&'ell* acchi+ J*+ Lennie+ J* 2 >ilmore+ 1* B"$!$E* Critical reflections on the use of participatory methodologies to build evaluation capacities in international development organisations* Paper presented at the )he Fth +orld Congress CDID -articipatory Action "esearch and Action 8earning* 1elbourne+ %ustralia+ 3-6 8eptember "$!$* aut+ 8* B"$$4E* 8tudying self-evaluation capacity building in a large international development organiHation* American Oournal of Evaluation& CFB!E+ ./-/6* he Listening ProAect B"$$6E* 8- Eactsheet and )imeline( http7II'''*cdainc*comIcda'''IpdfIotherIlpZfactsheetZ"$$6!!$/ZPdf*pdf* UN 0esolution /!I!4" B!664E* http7II'''*undemocracy*comI%-0(8-/!-!4"*pdf UNDP B"$$6aE* Communication for development: A glimpse at !D-<s practice* =slo Governance Centre =slo+ UNDP* http7II'''*undp*orgIoslocentreIdocs$6I#inalCfDZboo&let*pdf UNDP B"$$6bE* Communication for development: Demonstrating impact and position institutionally* 0eport of the !!th United Nations Inter-agency 0ound able on Communication for Development >ashington+ DC* U8%* http7IIportal*unesco*orgIciIfrIfilesI"-6!"I!".3.3.-44,!!ZUNZ0 ZC.DZProceedingsZv 3Z#IN%L*pdfI!!N":UNN":0 N":C.DN":ProceedingsZv3N":#IN%L*pdf UNDP B"$$6cE* ;andbook on planning& monitoring and evaluating for development results* Ne' <or&7 UNDP* http7IIundp*orgIeoIhandboo& UN(8C= B"$$4E* )owards a common ! systems approach: ;arnessing communication to achieve the Millennium Development Loals* :ac&ground paper prepared for the !$th !,-

UN Inter-%gency 0ound able on Communication for Development* %ddis %baba+ (thiopia* http7IIunesdoc*unesco*orgIimagesI$$!.I$$!.63I!.63-4e*pdf UN(8C= B"$$-E* Media development indicators: A framework for assessing media development( Paris7 UN(8C=* http7IIunesdoc*unesco*orgIimagesI$$!3I$$!3,!I!3,!$"e*pdf University of asmania B"$$,E* Developing an evaluation plan* In -ro6ect evaluation toolkit* http7II'''*utas*edu*auIpetIsectionsIdeveloping*html ?alery+ 0* 2 8ha&ir+ 8* B"$$/E* (valuation capacity building and humanitarian organiHation* Oournal of Multidisciplinary Evaluation& CB,E+ 4--!!"* Uusi&yla+ P* 2 ?irtanen+ P* B"$$$E* 1eta-evaluation as a tool for learning* % case study of the (uropean 8tructural fund evaluations in #inland* Evaluation& PB!E+ /$-3/* ?ernooy+ 0*+ 9iu+ 8* 2 Jianchu+ O* B"$$,E* ?oices for change* Participatory monitoring and evaluation in China* =tta'a7 International Development 0esearch Centre* http7II'''*idrc*caIopeneboo&sI66.-!I >aisbord+ 8* B"$$!E* #amily tree of theories+ methodologies and strategies in development communication* 8outh =range+ NJ7 Communication for 8ocial Change* http7II'''*communicationforsocialchange*orgIpdfIfamilytree*pdf >aisbord+ 8* B"$$-E* he institutional challenges of participatory communication in international aid* Social ,dentities& I%B.E+ /$/* >andersman+ %*+ 8nell-Johns+ J*+ LentH+ :* et al* B"$$/E* he principals of empo'erment evaluation* In D* #etterman 2 %* >andersman B(ds*E Empowerment evaluation( -rinciples in practice Bpp* "4-.!E* Ne' <or&7 he Guildford Press* >at&ins+ J* 2 acchi+ J* B(ds*E* B"$$-E* -articipatory content creation for development: -rinciples and practices* Ne' Delhi+ India7 UN(8C=* http7IIunesdoc*unesco*orgIimagesI$$!/I$$!/-3I!/-3".e*pdf >ebb+ D* 2 (lliott+ L* B"$$"E* Learning to live7 1onitoring and evaluating GI?I%ID8 programmes for young people* London7 8ave the Children #und* http7II'''*relief'eb*intIr'Ilib*nsfIdb6$$sidILG=N-3.L53CI ]fileI8ave heChildrenZ%ID8Z1(ZJanuaryZ"$$"*pdfVopenelement >estat+ J* B"$$"E* )he CDDC user friendly handbook for pro6ect evaluation* %rlington+ ?%7 he National 8cience #oundation* http7II'''*nsf*govIpubsI"$$"Insf$"$/4Insf$"$/4*pdf >hite+ 8* B!663E* Depoliticising development7 the uses and abuses of participation* Development in -ractice& PB!E+ 3-!/* >hite+ G* B"$$6E* Designing theory-based impact evaluations* >or&shop presented at the -erspectives on ,mpact Evaluation: Approaches to Assessing Development Effectiveness ,nternational Conference* Cairo+ (gypt+ "6 1arch J " %pril "$$6* >il&ins+ 5* B(d*E B"$$$E "edeveloping communication for social change: )heory& practice and power( :oulder+ Colorado7 0oman 2 Littlefield* >il&ins+ 5* 2 1ody+ :* B"$$!E* 0eshaping development communication7 Developing communication and communicating development* Communication )heory& IIB.E+ ,-/J ,63* >illetts+ J* 2 Cra'ford+ P* B"$$4E* he most significant lessons about the 1ost 8ignificant Change techni@ue* Development in -ractice& IJB,E+ ,34-,46* >orld Congress on Communication for Development B>CCDE 0ome Consensus B"$$3E* http7II'''*uneca*orgIafricanmediaIdocumentsI0ecommendationsZ0omeZConsensus* pdf

!,6

>rigley+ 0* B"$$3E* Learning from capacity building practice7 %dapting the C1ost 8ignificant ChangeD B18CE approach to evaluate capacity building provision by C%:UNG= in 1ala'i* -ra'is -aper IC+ IN 0%C7 =)ford* http7II'''*eldis*orgIgoIhome2idR"./"62typeRDocument

* endi+ %( .ist of E+ ert Panel mem"ers, C-D !ocal Points and other artici ants
E+ ert Panel mem"ers
SSilvia 4alit #reelance C.D consultant+ formerly 'ith #%= BretiredE 0ome+ Italy SKrishna 4elbase 8enior (valuation 8pecialist UNIC(#+ Ne' <or&+ U8% -enelope 4eynon 1onitoring and (valuation Coordinator+ Impact and Learning eam Institute of Development 8tudies+ University of 8usse)+ :righton+ (ngland SAilish 4yrne 0esearch and (valuation 8pecialist Communication for 8ocial Change Consortium Bbased in Nairobi+ 5enyaE Lary Coldevin Independent Consultant on C.D Bbased in Cebu+ PhilippinesE SOackie Davies Consultant Communication for Development BC.DE Consulting and 1edia1( London+ (ngland Alfonso Lumucio Dagron Independent Development Communication 8pecialist 1e)ico S+ill -arks Deputy 0epresentative UNIC(#+ 5athmandu+ Nepal S"afael 9bregon Director+ Communication and Development 8tudies Programme =hio University+ U8% !.$

Andrew -uddephatt Independent Consultant Global Partners+ London+ U5

S!ora Quebral Professor (meritus+ College of Development Communication+ University of the Philippines Los :anos+ Philippines Bbased in Ga'aii+ U8%E Oan Servaes Professor University of 1assachusetts+ U8% Matthew Smith Program =fficer+ Connectivity and (@uity in the %mericas International Development 0esearch Centre+ =tta'a+ Canada Silvio +aisbord Director of Graduate Programs+ 8chool of 1edia and Public %ffairs George >ashington University+ U8% SKitty +arnock 8enior %dvisor+ C.D P%N=8+ U5

1ther e+ ert contri"utor

Caroline Davies!4 #ormer %LINe ProAect 1anager Institute of Development 8tudies+ University of 8usse)+ :righton+ (ngland

C-D !ocal Points and M3E E+ erts

!*1 S"icardo del Castello Communication =fficer 0ome+ Italy I.1 "egina Monticone Chief+ Development Partnerships+ Department of Communication Geneva+ 8'itHerland Carla ;enry
!4

Caroline completed the survey for ()pert Panel members but 'as unable to participate in the Panel because she left ID8 in early July "$!$*

!.!

8enior (valuation =fficer Geneva+ 8'itHerland UNE/C1 SAenus Oennings Programme 8pecialist+ C.D Paris+ #rance UNDP 8e 8an 8e eam Leader+ UNDP ?iet Nam 12( team
Ganoi+ ?iet Nam

-aavani "eddy
Programme %ssociate+ =slo Governance Centre =slo+ Nor'ay

UN!P* !eil Eord Chief+ Communications and 1edia Ne' <or&+ U8% 4oniface Kalanda 12( 8pecialist 8ierra Leone UNICE! ^-aula Claycomb =ICI8enior %dvisor+ C.D Ne' <or&+ U8% S )eresa Stuart C.D 8pecialist Ne' <or&+ U8% 4orld 0ank Masud Mo=ammel 8enior Communications =fficer+ Department of =perational Communications >ashington DC+ U8%

1ther artici ants

In addition to those mar&ed 'ith an asteris& B^E above+ the follo'ing people provided input at the consultation meetings in Ne' <or& in December "$!$7 UNI!EM, Ne' Bork !."

:elen 8anH+ Gead+ (valuation Unit Isabel 8uareH+ (valuation %nalyst UNICE!, Ne' Bork %ntonia %ntonopolous+ Communication 8pecialist Jeffrey :ates+ C.D 8pecialist+ GealthIImmuniHation 8amuel :ic&el+ =fficer in Charge+ (valuation =ffice Ivana :Aelic+ 1IC8 8pecialist James Da'son+ C.D >eb Consultant herese Dooley+ 8enior %dvisor J Gygiene 2 8anitation %sha George+ Gealth 8pecialist 0ina Gill+ Deputy Director+ Division of Policy and Practice Neha 5apil+ C.D 8pecialist 1arina 5omarec&i+ 5no'ledge 1anagement 8pecialist Jesus LopeH 1acedo+ C.D 8pecialist+ Gealth #rancesca 1oneti+ 8enior Child Protection 8pecialist 5ent Page+ 8enior %dviser for 8trategic Communication %&i&o 8a&aedani Petrovic+ C.D 8pecialist 8halini 0oHario+ Communication Consultant ,raduate /tudent Partici ant %licya Lloyd+ University of 1assachusetts+ U8% *dministrative /u ort

%nna :urlyaeva-Norman+ %dministrative %ssistant+ UNIC(#+ Ne' <or& :arry Pamer+ 0apporteur+ Ne' <or&

!.,