You are on page 1of 21

WhatWeKnowAbout JointEvaluations

ofHumanitarianAction

LearningfromNGOExperiences
Section1of3:THEGUIDE April2011

Contents

ACKNOWLEDGEMENTS......................................................................................... 3 ABOUTTHISBOOKLET .......................................................................................... 4 THEGUIDE .............................................................................................................. 5 CHAPTER1:WHYDOAJOINTEVALUATION? .......................................................... 5


TheBenefitsofaJointEvaluation ............................................................................................... 5 TheDownsidesofaJointEvaluation ........................................................................................... 6 Conclusion ................................................................................................................................ 7

CHAPTER2:JOINTEVALUATIONWHEN,WHOANDHOW? .................................... 8
Whenwillittakeplace? ............................................................................................................. 8 Whowilltakepartinit? ............................................................................................................. 8 Isthereenoughtimeforajointevaluation? ................................................................................ 8 Howwillitbepaidfor? .............................................................................................................. 9 Howcanthejointevaluationbemostusefultovariousstakeholders? .......................................... 9

CHAPTER3:WHATTODOBEFORETHEEVALUATION ............................................ 10
Choosealeadagencyandagreeonroles .................................................................................. 10 Setupamanagementstructure................................................................................................ 10 Estimatecostsandduration ..................................................................................................... 11 Communicatewhattheevaluationisabout ............................................................................... 12 Findacompetentadministrator/manager ................................................................................. 12 Carefullypickevaluationteammembers ................................................................................... 12 Chooseafewobjectivestocover .............................................................................................. 14 Agreeonevaluationstandardsandmethods ............................................................................. 14 Writeaninceptionreport......................................................................................................... 15 Managecommunicationswithinthecollaboration ..................................................................... 15 Prepare,prepare,prepare!....................................................................................................... 15

CHAPTER4:WHATTODODURINGTHEEVALUATION ............................................ 16
Brieftheteamuponarrival ...................................................................................................... 16 Sharefindingsasyougo ........................................................................................................... 16 Ensurefindingsarereportedwithsensitivity ............................................................................. 16 FinalizingtheEvaluationReport ................................................................................................ 17

CHAPTER5:WHATTODOAFTERTHEEVALUATION............................................... 18
Developbothcollectiveandindividualrolloutplans ................................................................... 18 Emphasizepeeraccountability ................................................................................................. 18

CHAPTER6:JOINTEVALUATIONSINREALTIME .................................................... 19
Preparefortheevaluationbeforetheemergencystarts ............................................................. 19 Takeagoodenoughapproachtotheevaluation ..................................................................... 19 Callonadditionalresources ..................................................................................................... 19 Considersomeotherjointreflectionprocess ............................................................................. 20

ReferencesandFurtherReading.......................................................................... 21

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

ACKNOWLEDGEMENTS

Manypeoplehavesharedtheirvaluableexperienceandtimeinthecreationofthisbooklet.Special thanksgotoallofthem,particularly,inGuatemala,CarlaAguilarofSavetheChildrenUS,HughAprileof CatholicReliefServices,BorysChinchillaofMercyCorps,andJuanManuelGironDurinioftheECBProject; inNiger,JasmineBatesandMariannaHensleyofCatholicReliefServices,andJuliannaWhite,ofCARE;in Indonesia,AdhongRamadhanandJosephineWijiastutiofCatholicReliefServices,AgusBudiartoandEvi EsalyKabanofSavetheChildren,HariningMardjukiandAnwarHadipriyantoofCARE,andRichardusIndra GunawanandYacobusRuntuweneofWorldVisionInternational.SpecialthanksalsogotoJohnWilding, PaulineWilson,JohnTelford,MauriceHersonofALNAP,JockBakerofCAREandGuySharrockofCatholic ReliefServiceswhohavegivencriticalinputintothiswork.MalaikaWrightwastheauthorofthefirst paper. TheApril2011versionofthebookletwasupdatedbyKatyLovefromtheECBProject,LorettaIshidaof CatholicReliefServices,JockBakerofCARE,HanaCroweofSavetheChildren,andKevinSavageofWorld Vision.ThebookletwasrevisedbasedonfeedbackandreportsfromthosewhoparticipatedinECB supportedjointevaluationsin2010inIndonesia,Haiti,theHornofAfrica,andNiger.Thesepeopleserved asevaluationmanagersandcoordinators,teamleaders,teammembers,SteeringCommitteemembers, ECBfieldfacilitators,andECBaccountabilityAdvisersinjointevaluations,including:PaulOHagan,Greg Brady,YvesLaurentRegis,AngelaRouse,KatyLove,andJockBaker(Haiti);YenniSuryani,PaulineWilson, LorettaIshida,andLeAnnHager(Indonesia);KevinSavage,CheleDeGruccio,JimAshman,andWynn Flaten(HornofAfrica);andKadidaMambo(Niger).TheECBProjectthanksallwhocontributedtothis work.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

ABOUTTHISBOOKLET
Thisbookletwaswrittentoshareknowledgegainedfromtheexperiencesofpeoplethathavebeen involvedinjointevaluationsconductedbynongovernmentalorganizations(NGOs).Itmainlyprofilesthe workofNGOsinvolvedintheEmergencyCapacityBuildingProject(ECB),whichhasagoaltoimprovethe speed,qualityandeffectivenesswithwhichthehumanitariancommunitysaveslives,improvesthe welfare,andprotectstherightsofwomen,menandchildrenaffectedbyemergencies. Thisbookletalsodrawsonthelessonsofmultiagencyevaluationsthatalreadyexistwithinthe humanitariansector.Majorcontributionshavecome,inparticular,fromtheActiveLearningNetworkfor AccountabilityandPerformanceinHumanitarianAction(ALNAP). Wehopethatlearningfrompreviousexperiencescapturedherewillbeusefulforallthoseconsidering leadingtheiragenciesthroughajointevaluation.Thelearningsharedhereistargetedatevaluation practitioners,managers,andNGOscontemplatingajointevaluation.Additionally,wehopethatitwill contributetoagrowingbodyofknowledgeontheseprocessesandshowthatwhiletherearemany unansweredquestionsaboutjointevaluations,thereisalotwealreadyknow. Thisbookletiscomprisedofthreesections.Thefirstsection,TheGuide,canbereferredtoasahowto forthosecloselyinvolvedinjointevaluations.Itprovidesaframeworkforthoseapproachingan interagencyevaluation.Thesecondsection,TheStories,sharesseveralcasestudiesfromtheECB Projectsexperiences.Thethirdsection,TheTools,includesmanytemplatesandtoolsthatcanbe adaptedforevaluations,includingsampletermsofreferences,agreementdocuments,andchecklists.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

THEGUIDE
CHAPTER1:WHYDOAJOINTEVALUATION?
Whyshouldanagencyconsidertakingpartinajointevaluationofanemergencyresponseprogram?1 Afterall,jointevaluationsrequirecollaboration,collaborationmeansmoreworkandtime,andtimeisa scarcecommodityinemergencyprograms. Inrecentyears,severalNGOshavesoughttoanswerthisquestionwhiletakingpartinthejoint evaluationsprofiledinthisbook.Whiletheresultshavebeenmixedandthelearningcurveshavebeen steep,jointevaluationsconfermanybenefits.Whiletheevaluationsthemselveshaveyieldedinstructive andusefulfindings,agencieshavealsobenefitedsignificantlyfromthequalityoftheinteractionsthat tookplaceamongpeeragencies.Jointevaluationsoftenserveasforumsforongoinglearning,dialogue andevenbegincollaboration. Agenciesalsoinevitablylearnthattherearesomepitfallsintheprocessofconductingjointevaluations. Thoughajointevaluationisnotsodifferentfromasingleagencyevaluation,therearesomemajor differences,someofwhicharehighlightedbelowandaddressedthroughoutthisguide. Aboveall,likeasingleagencyevaluation,ajointevaluationprovidesanopportunitytolearnfrompast actionsoastoimprovefuturedecisionmaking. Itshouldbenotedthatthisguidesetsouttheidealprocessesandstructuresforajointevaluation.Inan emergencysetting,ofcourse,constraintsemergethatmaketheidealprocessachallengetoachieve. Evaluators,therefore,mustbeflexibleandwillingtoadapttotherealitiesonthegroundinorderto achievesomeifnotalloftheobjectivestheysetouttoachieve.2

TheBenefitsofaJointEvaluation 1. SeeingtheBigPicture
Oneevaluatorsaid,You[may]thinkyouvecoveredtheworldbutyouveonlycoveredone villageinten.Emergencyresponsestypicallyinvolveseveralhumanitarianactors.Whenthe responsesofmorethanoneactorareputsidebysideandexamined,theoverallpicture becomesclearer,revealinghowfactorssuchasgeographiccoverage,sectorspecific interventions,andcommunityinvolvementallfittogether.Jointevaluationsgofurthertowards measuringimpactbylookingatthecollectiveeffortsofseveralactorstomeetbeneficiaryneeds andtoidentifywhatgapsexist.

2. BuildingCoordinationandCollaborationtoImproveResponse
Giventhescaleofdisastersandthedisproportionateamountofsufferingtheycause,agencies workingalonearegenerallynotabletohavealargeimpact.Infact,agenciesthatcoordinate responsesandworktogetherduringemergenciesarebetterabletomeettheneedsofdisaster affectedpopulations.Bycomparingagenciesresponsessidebyside,jointevaluationsarebetter abletopointoutareaswhereNGOscouldhaveactedinacomplementaryfashionandmake

Themodelanddefinitionofjointevaluationsusedinthisbookletisanyevaluationthatlooksattheworkofmorethanoneagency. Thisusuallymeansthatinadditiontomoreactorsbeinginvolved,thereisagreaterbreadthofprogrammingbeingexamined. 2 ReaderslookingforfurtherguidanceshouldreviewShoestringEvaluation:DesigningImpactEvaluationsUnderBudget,Time,and DataConstraintsbyBamberger,Rugh,Church,andFort.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

recommendationsforhowtheycoulddosoinanticipationofthenextemergency.Evaluation reportsrepeatedlyshowthatbettercoordinationwouldhaveledtoamoreeffectiveresponse. Insomecaseswhereagenciesarealreadyworkingtogether,ajointevaluationcanbealogical conclusiontoajointactionorresponse.InIndonesiaandinNigerin2010,agenciesagreedto conductajointevaluationtoassesstheimpactoftheirjointactivities.3 Theprocessofcollaboratingontheevaluationitselfcanalsobeapowerfulwayofbuilding relationshipsamongpartneragencystaffthatendureforthelongterm.IntheECBexperience, someoftheserelationshipshaveledtoongoingactivitiesandeventheformationofanNGO coordinationforum(seeNiger).InHaiti,thejointevaluationhelpedtobuildrelationshipsamong nationalstaffandmanagers,servingasastartingpointforlongerterminteragency collaboration. 3.

WieldingWeightierConclusions,ImprovingPeerAccountabilityand Transparency
Jointevaluationscanbemoreauthoritativebecauseofthecombinedweightofthosebacking them.Assuchevaluationsareavailabletoawideraudience,thereislikelytobegreaterpressure toactupontherecommendations.Additionally,theyprovidealargerbodyofevidencefor purposesofjointadvocacy. Whenagenciesopenuptooneanotherbysharingweaknessesandstrengths,theyincrease transparencyandmakeiteasierforthemtoholdonanotheraccountableforactinguponthe recommendations.Transparencyiscriticalforagenciesinhumanitarianresponses,andsharing thefindingsofevaluationsacrossagencieshelpstobecomemoretransparent.Infact,agency peersmaypressuretheagencyactonrecommendationsfromanevaluation.

4. LearningfromandRelationshipBuildingwithPeers
Partnersinajointevaluationhavearareopportunitytolearnabouteachothersprogramming andoperations,andmaysharetechnicalknowledgethroughtheevaluationprocess,butalso throughtheongoingrelationshipsthatareoftenestablished.Onepractitionernotedthat workingwithstafffromotheragenciessometimesbringsnewperspectivesorevenchangesher thinkingaboutaparticularissue. Therelationshipbuildingthatoccursthroughajointevaluationallowsagencystafftoidentify otheragenciesstrengthsandcapacities.Therelationships,foundedontrust,thatarebuilt throughajointevaluation,mayresultinagencycooperationinthefuture.

TheDownsidesofaJointEvaluation 1. MoreComplexity
Ittakestime,skillandpatiencetogetagenciestoagreetodoajointevaluation,agreeona manageablelistofobjectives,diffuseanytensionsthatmayarise,ensurethatgroupdecision makingprocessesareclearandrespected,allwhiledealingwithhiringandsupervisingan evaluationteam,settingupinterviews,ensuringlogisticsareinplace,etc.Thisbecomeseven harderduringanemergency.

Seethe2010Indonesiajointevaluationreportformoreinformation,availableatwww.ecbproject.org/resources
April2011(v6)

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

Withoutaleadagencytotakeontheprimaryresponsibilityforthesetasks,andacommitted steeringcommitteethatcanjointlyhandlestrategicdecisionmaking,ajointevaluationcanbe frustratingandunsuccessful.

2. LessDepth
Oftenitisnotfeasibleorrelevanttogointomuchdetailonanyparticularagencysprogramsas wouldhappeninasingleagencyevaluation.Thereforemanyoftheevaluationquestionsof interesttoeachagencymaynotgetanswered.

3. MoreExpensive
Giventhenumberofactionsinvolved,jointevaluationscansometimesbemorecostlythan singleagencyevaluations.Ifagenciesagreetosharethecostsoftheevaluation,however, additionalcostsperagencywillbeminimal.

Conclusion
JointevaluationsallowNGOstolearnfrommultipleperspectivesandgiventhemamorecomplete understandingofanemergencyresponse.Theyhelpusworktogethernowandinthefutureandleadto relationshipsthatcanbeveryproductive.Forthesereasonstheycanbeenrichingexperiencesandhave aprofoundimpactonthewaywedothingsasindividualagenciesandascollectives.Itisimportantto havearealisticunderstandingofwhatcanandcannotbeaccomplishedbyajointevaluationbefore conductingone.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

CHAPTER2:JOINTEVALUATIONWHEN,WHOAND HOW?

Youmaywanttodoajointevaluationandhavegoodreasonstodoso.Butfirst,makesuretherewill beenoughtimefortheevaluation,willingpartners,andhuman,financialandotherresourcestogetit done.Thefollowingquestionsaremeanttohelpyoudeterminewhetherajointevaluationisfeasible.

Whenwillittakeplace?
Evaluationscantakeplaceatdifferentpointsofaresponse(during,immediatelyafter,orseveralmonths after).Thetimingdependsonwhattheagencieswanttogetoutoftheevaluation.Realtimeevaluations duringaresponseprovideresultsthatcanimprovetheresponsegoingforward(seeChapter6). Evaluationsconductedneartheendofanemergency,ontheotherhand,captureexperiencesand learningwhileitisstillfresh.Evaluationsconductedwellafteranemergencyendscanstillbeusefuland cancapturelongertermimpactofaresponse. Oneimportantfactorwhencreatingatimelineforjointevaluationsistorememberthatworkingwith multipleactorscanslowyoudown.Thereisrarelyaperfecttimetoconductajointevaluation,asall agenciesarebusy.Therefore,especiallyforrealtimeevaluations,itisimportanttostartplanningasearly aspossibleduringtheemergencyresponse.

Whowilltakepartinit?
Approachotheragenciesthatmayalreadybeconsideringanevaluationforthesamehumanitarian response.Consideragenciesthathavethesameoverallgoal(e.g.ensuringaffectedpopulationsareable torecoverquicklyfromthedisaster),andthathavesimilartypesofprogramsingeographicareasthatare closeenoughtogether.Identifytheappropriatepersontocontact,ideallysomeonewhoprovides strategicdirectionforthecountryoffice.Explainwhatwillbegainedfromdoingthisevaluationjointly (seeChapter1).Listentotheirviewsandnotethemdown.Dontbediscouragediftheyarenot interested.Keeptalkingtootheragencies. Whentalkingtootheragencies,findouthowtheyapproachevaluations.Dotheyconductthembecause donorsrequirethem?Howdotheyusethefindings?Whatresourcesdotheydesignateforevaluations? Takenoteofthistogetasenseofhoweachagencywillapproachtheevaluationandusethefindings. Theiranswerswillalsohelpprepareyouforpotentialareasofconflict,suchaswillingnesstocontribute stafftime.SeeTheTools,SuggestedTopicsforDiscussionwithProspectivePartners.Besuretheagencies arewillingtocommitstafftimeandresourcestosupporttheevaluation. Forajointeffort,andbecauseevaluationsmayrevealsensitiveissues,itsalsoimportanttobuildtrust amongtheagencies.Todoso,agreeonthefocusoftheevaluationtogether,ratherthanapproaching otherswithyourvisionandaskingiftheyareinterestedinjoiningin.Continuecollaboratingby communicatingclearly,beingtransparentwithinformationandintentions,andfollowingthroughwith commitments.

Isthereenoughtimeforajointevaluation?
Besuretoallocateenoughtimefortheevaluationteamtogetthejobdone.Unlessthelogisticsofgetting toandfromfieldsitesisunusuallytimeconsuming,athirtyorfortydaycontractfortheleadevaluatoris reasonable.Ensuretheevaluatorhasatleasttwodaysbeforeofficiallystartingtheevaluationtodo preparatoryworksuchastoreviewdocuments,proposemethodology,andplanlogisticswiththe

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

agencies.Notensuringenoughtimefortheadditionalworkwillcompromisethequalityoftheevaluation. Ensurethattimeisbuiltintoaccountformisseddeadlines,asmayoccurwhenmanyactorsareinvolved.

Howwillitbepaidfor?
Jointevaluationsusuallytakemoretimetoconduct,andmayrequirearelativelylargeteam.Costs, therefore,maybehigherthanforsingleagencyevaluations.Thecostscanbespreadoutamongagencies, andthisshouldbediscussedaspartoftheearliernegotiation. Havearoughideaofwhattheevaluationmaycost.Themaincostsareforhiringconsultantsandsupport staff.Comparethiswithwhatfundsmaybeavailableandwhatotheragenciesmaybewillingto contribute,includingstafftime,lodging,andvehicles.Ifinsufficientfundsareavailablefortheevaluation, considerajointpeerreviewtoreviewoneanothersprogramsandcometogethertodiscussfindings. Properandrealisticbudgetingiscritical. Donorsarelikelytobereceptivetojointevaluationsiftheybringaboutabetterunderstandingofthe contextandtheoverallhumanitarianresponseandsomedonorscommissionjointevaluations themselves.Therefore,ifabudgetfundedbyagivendonoralreadyaccountsforanevaluation,thedonor maybeopentoredirectingthatactivityfromthesingleagencysevaluationtocontributetoajoint evaluation.

Howcanthejointevaluationbemostusefultovariousstakeholders?
Evaluationstakealotofresourcesandeffortandeveryonewantsthemtobeuseful.Jointevaluations canbeusefultodifferentstakeholdersindifferentways.Inalargeemergency,agencystaffatregional andgloballevelswilllikelybeinterestedinthefindings. TheideafortheECBsupportedGuatemala Talktopeopleattheheadofficelevelinthecountrywhere evaluation camefromheadquarters.Theteamin theemergencyhappened,attheregionallevel,andat Guatemala feltthatthiswasanotherHQdriven headquarterslevel.Evenifthefindingsrefertoprograms initiative,sotheirparticipationinsteering thathaveended,cantheybeusedtoinformother committeemeetingswaslimited.Theagencieson programs,systemsandpolicieswithintheorganization? thegroundtriedtocustomizetheobjectives,butin retrospectfelttheyshouldhavestartedfrom Iftheproposalfortheevaluationcamefromheadquarters, scratch.Thisnegativelyimpactedtheevaluation dothoseinthefield,particularlycountryofficeleadership, processandthustheusageofthefindings. believethatthiswillbeausefulexerciseforthem?Ifnot, theymaynotwanttoengage,andtheevaluationwillprove Incontrast,theideafortheECBsupportedjoint hardtocarryout.Howwilltheyusethefindings?How evaluationinJogyakartaalsocamefrom committedwilltheybetotheevaluation?Theirinterest headquarters.However,theparticipatingagencies andengagementneedtobehightomakethisasuccessful onthegroundtooktheleadondefiningtheir experience. objectives,withadvicefromheadquarters.This helpedensurethepartnersweremoreincontrolof theevaluationprocess. Thereshouldbeareasonablelevelofconfidencethatthe findingswillbeusedbeforeproceedingwiththe evaluation.Ifnot,theevaluationteamwillstruggletoachievetheobjectives.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

CHAPTER3:WHATTODOBEFORETHEEVALUATION

Ifyouhavedecidedtopursueajointevaluation,herearesomethingstoconsider.

Choosealeadagencyandagreeonroles
ECBhashadthebestresultswhenoneagencyleadsthejointevaluationprocess.Thoughsomesharingof responsibilityisdesirable,agenciesshoulddesignatethemajorityofthedaytodaymanagement responsibilitiestotheleadagency.Anyoftheagenciesbeing Aleadagencythatplaysits evaluatedcouldserveasthelead;whatmattersisthattheagencyis rolewellcanmakeamajor capableofcarryingouttheresponsibilities. difference intheprocess.The head evaluator inECBsNiger Theleadagencyhiresandsupervisestheevaluationteam,coordinates evaluation found thelead travellogistics,providesteammemberswithworkspaces,organizes agencys organization ofthe meetings,andgivesleadershipregardingthedefinitionofthe evaluationprocessand objectives.Ultimately,itisthisagencythatisaccountableforensuring logisticstobethemosthelpful thattheevaluationtakesplace. thingtohimincarryingouthis work.Itwasoneissuewe Asteeringgroupmadeupofrepresentativesfromeachagencycan didnthavetothinkabout;it cometogethertoagreeontherolesoftheleadagency,theroles wassowellorganized,he assignedtootherparticipatingagencies,andsharethemwithall noted. involvedpersons,staffandevaluators.(Formore,seesectionbelow onManagementStructure.)

Setupamanagementstructure
Whensettingupamanagementstructurefortheevaluationitsimportanttorecognizethatyouare managingnotjustanevaluationbutacollaboration.Dontsuccumbtopressuretomakechoices favorabletotheleadagency,steeringcommitteemembersorhighlevelsponsorsoftheevaluation(e.g. wemusthavex,y,andzrepresented,andanyindividualswilldo).Seekoutindividualsforthesteering committeeandevaluationteamwhoarecommittedtoasuccessfuloutcome,eveniftheyarenot conventionalchoices.Wherethereisaneedforagencyrepresentation,createspacefortheseindividuals insomehighprofile,butlesscriticalfunction. Ajointevaluationmanagementstructurecouldlooksomethinglikethis: Asteeringcommittee.Thisgroupwillberesponsibleforstrategicdecisionmakingveryearlyon regardingobjectives,timing,andresourceallocation,includingstaffandfunding.Thesteering committeewillalsobeactiveinreviewinganddebatingthefindingsandactingupontheir implicationswithintheiragenciesandbeyond.Itisnormallychairedbytheleadagencyandhas representationfromeachoftheparticipatingagencies.Thecommitteewouldideallybekepttoa maximumoffive,makingoversightanddecisionmakingmorefocusedandachievablein reasonableamountsoftime.This,however,supposesthatagenciesinvolvedarewillingto delegatestafftoacommittee. Theidealsteeringcommitteememberisseniorenoughtospeakonbehalfofhis/heragencyand hastheauthoritytomakedecisions.Thisindividualmusthaveagoodknowledgeofhisorher organizationsemergencyprogramsandongoingdevelopmentwork.Inaddition,heorshe shouldbeabletothinkstrategically,andknowenoughaboutevaluationstoadviseonthe evaluationmethodstobeusedandonthefieldlocationstobecovered.Theseindividualswill alsobethosemostlikelytofollowuponrelevantrecommendations.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

10

Steeringcommitteemembersshouldagreeonanddocumentclearprocessesandstandardsof efficiency,transparencyandaccountabilityregardingrolesandresponsibilities.Theyshould agreeonhowdecisionmakingwouldwork,howtoresolvedisagreementswithinthesteering committee,andhowtoshareinformation.Steeringcommitteemembersshouldbethesame throughouttheentiretyoftheevaluation.Whenmembersrotateonandoffthecommittee, decisionsandguidancemaychange,whichwillcomplicatemattersfortheevaluationteam. Otheragreementsconcernthereportformat,theuseofagencylogos,theownershipofthe productsoftheevaluation(i.e.intellectualcopyright),howagencieswillusethefindingsandif theywillholdoneanotheraccountable,etc. Achairperson.Thispersonisbasedattheleadagencyfortheevaluationandisamemberand chairofthesteeringcommittee.Heorshehasmostofthestrategicdecisionmaking,operational andcollaborationresponsibilitiesoftheevaluation.Thisindividualusuallyassumestheroleof evaluationmanagerandisthedirectreportinglinefortheteamleader.Thechairpersonshould managethebudgetandtrackexpenses. Amanageroradministrator.Thispersonmustideallybebasedintheleadagencywithacertain percentageofhisorhertimededicatedtotheevaluation.Seethefollowingsectionformore detailsonthemanagersresponsibilities.Heorshecouldalsositonthesteeringcommitteebut withoutvotingrights. Theevaluationteam.Theteamistypicallycomposedofoneortwoindependentconsultants, andarepresentativefromeachofthepartneragencies.Thisteamisaccountabletothesteering committee,particularlythecommitteechair.

Therearevariationsonthisstructure,ofcourse.Mostevaluationsalsohavehigherlevelsponsors thatmayalsoformasuperstructure.Sectorexpertsmayalsobeneededontheevaluationteam. SeeTheToolsforaSampleAgreementsDocument.

Estimatecostsandduration
Basedonthedraftitinerary,thesteeringcommitteeshouldagreeonadraftbudgetandcostsharing arrangements.Typicallyagenciesshareconsultantcostsequallyandprovidefundingforthestaffmember theyappointtojointheevaluationteam.Thinkthroughfundingimplicationsforallaspectsoftheprocess andhowlongeachactivitywilltake.Forexample,ensurefundsforgoodqualityediting,formatting,and presentation,asthesecanmakeasignificantdifferenceinhowwidelythereportisread.Berealistic aboutthetimeitwilltaketheevaluationteamtogetthejobdone.Atleast30daysorevenfortydaysare recommendedfortheteamleader.Theconsultantwilllikelybethelargestcost,butitisessentialto budgetfor,ashis/hertaskswillinclude: Reviewdocuments,preparemethodology,andcorrespondwiththesteeringcommitteepriorto theevaluation. Conductfieldvisitstoatleastthreesitesforeachoftheagencies. Interviewagencystaff. Interviewotherstakeholders. Presentthefindingstostakeholdersincountry. Prepareadraftofthereport. Incorporateeditsandcommentsonthereportfrommultipleactors.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

11

Communicatewhattheevaluationisabout
Ajointevaluationisanewsworthyevent,butnoteveryonewillunderstanditspurpose.Makesurepeople insideandoutsidetheparticipatingagencies,includingbeneficiaries,areawareoftheevaluationsothat theywillbemorelikelytoreviewandmakeuseofthefindings.Draftaonepageinformationalsheet abouttheevaluationforwidespreadsharing,andespeciallywithcountryofficestaffwhoneedtobe awareoftheevaluation(thoughtheywouldideallybeengagedthroughouttheprocess). Itisparticularlyimportanttohavepreparatorydiscussionswithbeneficiarycommunitiestoensurethey understandthepurposeoftheevaluationandtheyagreetoparticipateintheevaluation.Theyshould understandthatevaluatorsdonothaveanyassistancetogive.4InHaiti,theevaluationteamtrained30 nationalstafffromtheparticipatingagencieswhospokeHaitianCreoletoberesponsibletoengage beneficiaries.Theydidsobyconductingfocusgroupdiscussions,askingopenendedquestionsto understandpeoplesexperiencewiththeemergencyresponse.5

Findacompetentadministrator/manager
Considerhiringsomeonewhocanspendasignificantamountoftime(50100%)focusedonthe evaluation,especiallyinthemonthortwoleadinguptoit.Thispersonmaybeanadministrator,but shouldalsobesupervisedbyaseniorpersonwhocanadviseonstrategicissues. Asuperbadministratorcanmakeamajordifferenceinthesuccessofanyevaluationbutparticularlya jointevaluation.Ideally,anationalstaffpersonshouldbehiredorsecondedfromoneoftheagencies.He orshewillberesponsibleformeetingthelogisticalandadministrativeneedsofboththesteering committeeandtheevaluationteam. Asampletasklistforthispersoncouldlooklikethis: Organizetherecruitmentfortheindependentconsultant(s). Draftandprocesscontractwithconsultant(s). Arrangeschedulesandmanagethecalendar. Arrangelogisticalarrangementsfortravelinginthefield. Coordinateinformationexchangebetweentheagenciesandtheevaluationteam,such ascollectingtherelevantbackgrounddocumentsfortheevaluationteam. Arrangemeetingsbothwiththeparticipatingagenciesandwithoutsideactors. Helpdocumentwhoisresponsibleforwhatandsharethiswithallparties. Agreeonnormsforperdiemandotherpolicies.Typically,eachagencyfollowstheir ownandthecoordinatingonehirestheexternalsandusestheirperdiems. Meetwiththeevaluationteam. Thesteeringcommitteeorchairpersoncouldappointtheevaluationadministratorormanager.Ideally, thesteeringcommitteewilldefinetheauthorityleveloftheadministrator,whoheorshewillreportto, andwhatlevelofauthorityheorshewillhavetomakedecisions.Itshouldbemadecleartheamountof timeanadministratorwillprovidetosupporttheevaluationteam.

Carefullypickevaluationteammembers
Selecttherightteam.Inadditiontothetechnicalskillstheyneedtoconducttheevaluation,team memberswillalsohavetobegoodatbalancingtheneedsofmultipleclientswithsensitivity.Whiletheir rolesshouldbemadeclearbeforetheevaluation,experiencehasshownthattheywillneedtobeflexible

SeeTool9intheGoodEnoughGuidetoImpactMeasurementandAccountabilityinEmergencies

SeetheCARESavetheChildrenjointevaluationreportatwww.ecbproject.org
April2011(v6)

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

12

oncetheevaluationbegins.Theteamshouldnotbetoolargethatitisdifficulttomanage.Threetofour teammembersisusuallysufficient.Atypicalteammaybecomposedof: Anindependentconsultant/teamleader.Thispersonknowsalotaboutevaluationandalsohas strongmanagementandleadershipskills,theabilitytostaycalmunderpressureandtobe adaptiveinthefaceoftheunexpected.Ajointevaluationteamleaderalsoneedstheabilityto dealwithmultiplelayersofmanagementandbalancevariousexpectations,andthusmusthave strongdiplomacyandcommunicationskillsbothwrittenandverbal.Theteamleadershould alsohavehadpreviousexperienceasateamleader,sincethisisitselfaspecialskill.Team leaderswiththeseskillsetsaresometimeshardtofind,anditiscriticalthatthereisabudgetline topayforthem. Thoughitsnotalwayspossibletorecruitateamleaderwhohaspreviouslyledjointevaluations, confirmthatheorshehasexperienceinimpactanalysisinemergencies,asheorshewillneedto understandhowthevarioussetsofdatacometogethertoformabiggerpicture.Notethat consultantsoftencomewiththeirownideasandmethodologies,andtheywillneedguidance andparametersfromthesteeringcommittee. Anationalconsultant.Thenationalconsultantprovidescriticalguidanceonthepolitical,social, andculturalcontextoftheemergencytotheteam,especiallytotheteamleader,whoisoftenan expatriate.Havingsuchapersononhandforajointevaluationcanhelpinnetworkingwith nationalstakeholdersandensurethatknowledgeisquicklytransferredtotheevaluationteam aboutkeyactorsandeventsandcanminimizesomeofthecomplexityofthedataandfactorsto beanalyzed. Asectorspecialist.Ajointevaluationwillchallengetheteamofevaluatorstoaddressthewide rangeofprogramareasbeingcoveredwhilealsofocusingonselectedkeyandpriorityaspects, especiallyaseachagencymayhaveuniqueinterests.Ifagenciesneedmoreindepth examinationofaparticulartypeofprogram,theyshouldconsiderbringingasectorspecialistto theteam,freeingupothermemberstofocusontheoverallpicture. Agencyteammembers.Eachagencytypicallyappointsonerepresentativetotheevaluation team.Theseindividualsarenotactingonbehalfoftheiragencybutrathermustbeimpartial evaluators.Theskillsetsofthesepeople,forexampletheirexpertiseincertainsectoralareas, languageandfacilitationskills,andevaluationexperienceareveryimportanttotheoverall successoftheteam.Ensuringthatagencyandevencountryofficestaffarerepresentedon theevaluationteamwillincreaseownershipoftheevaluationfindings. Itmaybehardforagencyteammemberstobeavailableforthefulllengthoftheevaluation,but experiencehasshownthatcontinuityisimportanttoevaluationqualityandthelearning experienceisalsogreatlyenhanced.Agencymanagersshouldthereforemakeeveryeffortto ensurefullparticipationofagencystaffontheevaluationteam. Giventheimportanceofgettingcompetentteammembers,itsimportanttostarttherecruitment processearly.Goodindependentconsultantsnationalandinternationalalikeareoftenbookedfor weeksorevenmonthsinadvance. OncethesteeringcommitteehasfinalizedtheTermsofReferenceforteammembersandtheskillsthey want,agenciesshouldconsiderrequestinghelpfromtheirheadorregionalofficesinrecruitingtheteam, suchasdoingtheinitialadvertisingandscreeningandthensendingashortlistofcandidatestothelead agency.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

13

Bythetimehiringbegins,theobjectivesfortheevaluationshouldhavebeenwellenoughdefinedthatthe steeringcommitteeisclearonwhatprofilesareneededontheteamandthus,whotohire. SeeTheToolsforSampleTermsofReferenceforEvaluationTeamMembers.

Chooseafewobjectivestocover
Theparticipatingagenciesmayhavedifferentinterestareastheydliketocoverinajointevaluation.But itisnotpracticaltoaddresstoomanyobjectivesinajointevaluationasthereisalreadymorecontentto cover.Ideally,thereshouldbenomorethanthreeorfourobjectiveswithinthetermsofreference,and thescopeshouldbeasnarrowaspossible.Forexample: Howwelldidthevariousagenciescoordinatetheirresponses? Howappropriatewastheintervention? Howtimelywastheintervention? Howwelldidtheresponseassistpeopleinrecoveringfromthedisaster? Objectivesthatconcerntheoverallimpactoftheresponseareusuallybestforajointevaluation. Objectivesofuniqueconcerntooneortwoparticipatingagencies,suchasissuesofoperationalefficiency, arenotgenerallyappropriate.Inareaswheremoredepthisneeded,hireanadditionalteammemberto focusspecificallyonaparticulartypeofprogrammingorissue. Doaskforinputonthescopefromstaffatdifferentlevelsofeachagencywhomyouexpecttousethe evaluationfindings.Atthesametime,itiswisenottoconsulttoowidely,asyouwillruntheriskofadding toomanyobjectivesandanunrealisticscopefortheevaluation. Objectivesshouldbeagreeduponbeforetheevaluationteamishired.Infact,considerbringinginan externalfacilitatortonegotiatethescopeoftheevaluationaheadoftime.Oncetheleadevaluatorjoins, heorsheshouldhavethechancetotelltheparticipatingagencieswhatisfeasibleandrealistic.Itis criticaltofindabalancebetweenwhatagencieswantandwhattheleadevaluatorbelievesispossible. SeeTheToolsforaTermsofReferenceTemplate.

Agreeonevaluationstandardsandmethods
Jointevaluationsshouldincludeadocumentreview,keyinformantinterviewsandfocusgroupdiscussions withstaffandbeneficiarygroups. Theteamleaderwillbuildanapproachtoexamineeachagencysworkwithenoughrigortoinspire confidenceinthefindings,butnotdetracttoomuchfromafocusontheoverallimpactoftheagencies response.However,thesteeringcommitteeisexpectedtoadvisethisprocessandalsocommunicatethe criteriatheywilluseforvillageandbeneficiaryselectionfortheinterviews. Inadditiontomorelocationsforfieldvisits,forjointevaluations,theremayalsoneedtobemore interviewswithotheractors,suchasUNagencies,representativesfromcivilsociety,nationalandlocal partners,andgovernmentofficials. Certainindicatorswillbenonnegotiabletobeinlinewithacceptedinternationalstandards,suchasthe OECD/DACstandardsforevaluation.6Spherestandardsareanotherkeypointofreferencewhichshould
6

http://www.alnap.org/resources/guides/evaluation/ehadac.aspx

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

14

beassessedduringanevaluation,notonlyfortechnicalsectors,butalsocorestandardsofparticipation. Onesetofstandardsshouldbeusedforconsistencyinmeasuringperformance.Beclearonwhat organizationalminimumstandardsare.ReferencetheKeyElementsofAccountabilityontheECBProject website.

Writeaninceptionreport
Theevaluationteamshoulddevelopaninceptionreportonthetermsofreferenceandwithaworkplan. Thisreport,writtenbytheteamleader,willensureexpectationsareagreedonbythesteeringcommittee andtheteamitself.Italsoallowsfortheevaluationteamleadertodialoguewiththesteeringcommittee aboutwhatisrealisticandfeasible,givenavailabilityofstaff,budget,anddeadlines.

Managecommunicationswithinthecollaboration
Agenciesconductingajointevaluationneedclearagreementsaroundcommunication.Facetoface meetingsarecriticaltomakesureunderstandingsareclearandtobuildcohesion.Decidinghowtostore keydocumentsisalsoveryimportant.Onesolutionistosetupasimplewebpagetouploaddocuments, contactlists,schedulesandotheressentialinformation. Itisalsoimportanttohaveregularopportunitiesalongthewayfortheevaluationteamtodiscussany concernswithsteeringcommitteemembers.Forexample,earlyintheprocess,theteamcangive feedbackastohowwelltheevaluationmethodsareworkingandcheckwiththesteeringcommittee whethertheseshouldbemodified.Ifthesteeringcommitteeisengaged,theevaluationwillbemuch morelikelytosucceed. Itisalsoimportanttoagreeinadvanceonprinciplesoftransparencywithevaluationresults,including communicatingresultsinatransparentwaytobeneficiarycommunities(whichcouldbeintheformofa discussionorroundtable).Tryingtocoveruporhideevaluationresultsisnotonlyagainstprinciplesof accountability,butunderminesorganizationallearningandcanoftenbackfire.

Prepare,prepare,prepare!
Ourexperiencehasshownthatagoodamountofworkcanbeundertakenevenbeforetheevaluation teamarrives.OncetheTermsofReferencefortheevaluationhasbeenestablished,alistofkeyinformant interviewscanbedetermined,meetingsestablished,focalpointsready,andpreparatorydocumentscan beemailedtotheevaluator.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

15

CHAPTER4:WHATTODODURINGTHEEVALUATION

Ifrealisticobjectives,managementstructure,andacompetentteamhavebeenchosenand
established,theevaluationshouldbeeasiertomanage.Theteamwillstillneedgoodlogisticalsupport andguidance.Hereissomeadditionalguidanceonconductingtheevaluation.

Brieftheteamuponarrival
EnsurethattheteamhasachancetodiscusstheTermsofReferencewitheachofthesteeringcommittee members.TheleadevaluatorshouldalsogoovertheTermsofReferencewiththesteeringcommitteeas agroup. Theadministrator/managerorsteeringcommitteechaircanbriefteammembersonrolesand responsibilitieswithintheevaluationstructure.Theteamwillneedtobebriefednotonlyonthelogistics andtheprocessoftheevaluation,butalsoontheresponseprogramswhicharetobeevaluated.The teamwillneedtobeclearonhowtheevaluationisrun,theroleoftheleadagencyandtheother agencies,towhomtheteamreports,wheretheywillgetlogisticalsupport,andhowtheywillmaintain independence.Whenthesearenotclear,confusionabounds,andtheteamwillstruggletoachieve objectives.Anticipatetheextraconsultationtimeneededwhenestimatinghowmuchtimetheteamwill needfortheevaluation.

Sharefindingsasyougo
Theteamleadersshouldalsoensurethatthesteeringcommitteeandthestakeholders(asmentioned below)receiveregularupdatesthroughouttheprocess.Ifthesteeringcommitteeandstakeholdersare wellbriefedabouttheprogressandinitialfindingsoftheevaluation,therewillbenosurprisesattheend. Dailydebriefsamongtheevaluationteamdrawsoutpreliminaryfindingswhichtheteamleadercanuse toprovideregularupdates.

Ensurefindingsarereportedwithsensitivity
Receivingandreviewingthefindingsofajointevaluationcanbeanexcitingtimefortheagenciesbutalso atimeofapprehension.Theleadevaluatorshouldpresentfindingsinawaythatwillnotmakeanyagency feelinferiororunfairlycomparedwithothers.Agencieswillalsoinevitablylookformentionsof themselvesandjudgewhethertheythinkthefindingsarefair.Findingsthatarecriticalinnatureshould bephrasedinaconstructiveway,supportedbyreasonableevidenceandbalancedwithpositivefeedback. Inadditiontothemainreport,theevaluatorcouldalsocreateshortindividualreportsforeachofthe agencies.Inpractice,however,thismaynotbeworththeadditionaleffortsincejointevaluationstendto bebetteratlookingattheoverallresponseandcoordinationbetweenagencies(i.e.fromabeneficiary perspective)thanlookingatindividualagencyoperationsindetail. Anotherapproachisfortheevaluationteamtodoapreliminaryanalysisthatcompilesandgroups findings.Throughaworkshopormeeting,theteamcanfacilitatestafffromtheparticipatingagencies (especiallythosewhowillusethefindings)tocollectivelydrawconclusionsandrecommendations.With thistypeofparticipation,agenciesaremorelikelytoaccepttheconclusionsandfeelresponsiblefor actingontherecommendations. Ultimately,whenjointevaluationsarewellplannedandagenciesandtheteamcommunicatethroughout theprocess,agenciesarelesslikelytotakeissuewiththeresults.Afocusonlearningmakeseventhe leastflatteringfindingsmorepalatablebecausetheycanbeinstructive.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

16

Oncetheagencieshavehadachancetodiscussanddebatethefindings,theyshoulddiscussthemwitha broadergroupofstakeholders,especiallythosethatwereconsultedduringtheevaluationprocess,such asUNagencies,beneficiaries,localNGOs,andlocalgovernment.Onewaytodothisistoholdaninter agencyvalidationworkshopwherestakeholdersaregivenanopportunitytoconfirmordisputethemajor findingsandrecommendations.

FinalizingtheEvaluationReport
Theevaluationreportshouldbeeasytoreadandrelativelyshortnomorethan30pages.Itisimportant tofocusonthatwhichhasgonewell,andgoodpracticeshouldbehighlightedinthereport. Assumingstakeholdershavebeenbriefedthroughouttheevaluation,thefindingsandrecommendations intheevaluationshouldnotcomeasasurprise.Doanticipate,however,thatstakeholderswillnotagree withallfindingsandthesteeringcommitteeshouldbepreparedtoaddressthis. Itiscriticaltosetoutaperiodtoreceivefeedbackonthedraftofthereport.Beclearandrealisticabout thetimelineforthisperiod.Itneedstobelongenoughtoallowtherightpeopletoprovidefeedback,but notsolongthatthefindingsarenolongerrelevantbythetimethereportiscompleted.Afterensuring thatallofthepeoplewhoneedtogivefeedbackareinformedofthescheduleinadvance,twotofour weeksisareasonabletimeinwhichtoallowpeopletosubmitfeedback. Theevaluationteamleaderisultimatelyresponsibletomakethedecisionsaboutwhichfeedbackis incorporatedandwhichisnot.Ifthereisenoughdisagreementaboutcertainfindingsorconclusions, thesecanbeaddressedinamanagementresponsethatisannexedtothefinalreport.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

17

CHAPTER5:WHATTODOAFTERTHEEVALUATION
Withtheevaluationworkdoneandthefindingsdetailed,onepartoftheprocesscomestoaclose.But inotherways,therealworkisjustbeginning.Hereissomeguidanceformakingthemostofthe completedevaluation.

Developbothcollectiveandindividualrolloutplans
Becausejointevaluationshaverelevancetoawiderangeofactors,agenciesshouldsharethereportwith humanitarianbodiesandsuchnetworksastheActiveLearningNetworkforAccountabilityand PerformanceinHumanitarianAction(ALNAP),inadditiontoheadquarters.Sharingthereportfromajoint evaluationreportwidelydemonstratestransparencyandacommitmenttocontributetolearningwithin thebroaderhumanitariansector. Theagenciesmaywanttodevelopsimplecollectiveandindividualcommunicationsplansincluding distributionlistsforthereportandsmallactionplanningmeetingstodiscussandpresenttheimplications ofthefindings.

Emphasizepeeraccountability
Withjointevaluations,agencieshavetheopportunitytoholdoneanotheraccountableforprogresson recommendations.Theymaychoosetoworkonsomerecommendationstogether.Theymayagree beforehandtoholdafollowupworkshopinsixmonthsorayearstime.Atsuchatime,theycouldthen discusshowthefindingswereshared,whatprogresswasmadeandwhatwastheoutcomeofanyactions taken.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

18

CHAPTER6:JOINTEVALUATIONSINREALTIME
Conductedwhiletheemergencyresponseisstillongoing,realtimeevaluations(RTEs)arevaluable toolsforrectifyingproblemsandmakingimprovementsinprogrammes.However,ajointRTEcanbe especiallychallenging.Herearesomethingstoconsideriffacingthedecisionofconductingajoint evaluationinrealtime.

Preparefortheevaluationbeforetheemergencystarts
Incasesofslowonsetemergencies,theremaybetimetoplanmonthsinadvance.Evenwithrapidonset emergencies,preparednessispossible.Youcanjointlyoutlinegenericplansfortheevaluationwhichcan beturnedintoactualplansinthefaceofadisaster.Theseplansshouldincludeasmanyoftheaspectsas possiblethatareoutlinedinthisbookletonhowtoorganiseajointevaluation,including,crucially,the designationoffocalpoints.Thesefocalpointsareonstandbyandwillhavetheresponsibilityofgettingan RTEprocessstarted,conveningthevariousactors,etc.andwhowillideallyremainaspointpersons duringtheprocess.

Takeagoodenoughapproachtotheevaluation
Youmayhavetotakesomeshortcutsanduseagoodenoughapproach.Goodenoughdoesnotmean secondbest:itmeansinanemergencyresponse,adoptingquickandsimplesolutionsmaybetheonly practicalpossibility.Whenthesituationchanges,youshouldaimtoreviewyourchosensolutionand amendyourapproachaccordingly. Forexample,youcansimplifymanagerialstructures.Withindaysofthepartiesagreeingtodoajoint evaluation,youmayagreetoestablishasmall,rapidlyorganizedmanagerialstructure(whichcan subsequentlytransitiontoamorerobustoneatalaterstage).Duringthefirstweek,forinstance,that groupcouldlookatwhatisminimallynecessary,andcreateapracticalquickanddirtytermsof reference.Themanagementcommitteecoulddelegatemuchofthedaytodaymanagementtooneor twokeyactors,andthusspendlesstimeongroupdecisionmakingandconsensusbuilding. Eachparticipatingagencywouldbetrustedtocarryoutthetasksassignedtotheminaccordancewith predeterminedplansandstandards.Oncetheprocesshasbeenstartedandtheevaluationisinmotion, agenciescanthengraduallybuildintighterqualitycontrolmechanisms,morefocusedtermsofreference, andamoreinclusiveprocess(e.g.alargermanagementgroup). Suchagoodenoughapproachisnotanidealevaluativeprocess,butlikelyrelevantforRTEsbecause agenciesareparticularlybusywiththeimplementationofaresponse.However,certainaspectsofthe jointevaluationshouldnotbesubjectedtoshortcuts.Theseincludeethicalstandards,suchasthe confidentialityandindependenceoftheevaluativeprocess.

Callonadditionalresources
Participatingagenciescouldconsidercallingonadditionalinternalsupport.Astaffmembercouldbe secondedtoacountryofficeforsomeweekstohelpwiththejointRTE.Unlikecountryofficestaff,who wouldpresumablybepreoccupiedwiththeemergencyresponse,thispersonwouldhavetimetofocuson theevaluation.Heorshecoulddoaninitialscoping,astakeholderanalysisandholdameetingcollectively orindividuallywithpartnerstogettheirviews.Heorshecouldalsoassistwithpracticalpreparationsfor theteam,includingsettingupfieldvisits.

Considersomeotherjointreflectionprocess
IfajointRTEisnotrealistic,considerotherlearningprocesseslikeajointafteractionrevieworapeer review.Agenciescandoquickassessmentsoftheirwork(seeImpactMeasurementandAccountabilityin Emergencies:TheGoodEnoughGuide)andgettogetherforashortmeeting/workshop,orinvitean experiencedexperttoprovideadviceonhowtheoperationmaybeadapted.

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

20


ReferencesandFurtherReading

JointEvaluations

GuidanceforManagingJointEvaluations.DACEvaluationSeries,OECD2006. http://www.oecd.org/dataoecd/29/28/37512030.pdf JointEvaluations:RecentExperiences,LessonsLearned,andOptionsfortheFuture.DACEvaluation NetworkWorkingPaper,OECD,2005. LessonsAboutMultiAgencyEvaluations:AsianTsunamiEvaluationCoalition.http://www.tsunami evaluation.org/NR/rdonlyres/9DBB5423E2EF43ABB6D2 2F5237342949/0/tec_lessonslearned_ver2_march06_final.pdf

GeneralEvaluations
USAIDCenterforDevelopmentInformationandEvaluation,PerformanceMonitoringandEvaluationTIPS series. http://evalweb.usaid.gov/resources/tipsseries.cfm WesternMichiganUniversity,EvaluationCenter.EvaluationChecklists http://www.wmich.edu/evalctr/checklists/checklistmenu.htm#models ShoestringEvaluation:DesigningImpactEvaluationsUnderBudget,Time,andDataConstraints.M. Bamberger,J.Rugh,M.Church,andL.Fort,TheAmericanJournalofEvaluation,2004. UtilizationFocusedEvaluationChecklist.MichaelQuinnPatton http://www.wmich.edu/evalctr/checklists/ufe.pdf

WhatWeKnowAboutJointEvaluations

TheECBProject

info@ecbproject.org

April2011(v6)

21