ROBERT WOOD JOHNSON FOUNDATION EVALUATION SERIES

An Ecological Understanding of Evaluation Use

A Case Study of the Active for Life Evaluation

2010

By Judith Ottoson, Ed.D., M.P.H. and Diane Martinez, M.P.H.

www.rwjf.org

ACKNOWLEDGMENTS
ThisreportwaspreparedfortheRobertWoodJohnsonFoundation,underthedirectionof LauraC.Leviton,Ph.D.,SpecialAdviserforEvaluation,byJudithM.Ottoson,Ed.D.,M.P.H.and DianeMartinez,M.P.H.JudithM.Ottoson(jottoson@comcast.net)isanindependentevaluation consultantandadjunctlectureratSanFranciscoStateUniversity.DianeMartinez (dmartinez@rwjf.org)isaRutgersPolicyAnalystFellowattheRobertWoodJohnsonFoundation.

Copyright2010RobertWoodJohnsonFoundation

RouteOneandCollegeRoadEast P.O.Box2316 Princeton,NJ085432319 ThispublicationisavailablefordownloadingfromtheFoundationsWebsiteat: www.rwjf.org/pr/product.jsp?id=71148

An Ecological Understanding of Evaluation Use: A Case Study of the Active for Life Evaluation

TableofContents

WhatisEvaluationUse? Introduction ADescriptionoftheProgramEvaluated:ActiveforLife ADescriptionoftheEvaluation CaseStudyMethodology Findings - TypesofEvaluationUse - ThreadsofEvaluationUse - LeveragedKnowledge FacilitatorsofEvaluationUse BarrierstoEvaluationUse Conclusions Implications

3 4 5 5 6 7 9 15 16 17 18 19 20 22 25 33 35

AppendixA:LiteratureReviewonEvaluationUse AppendixB:FullDescriptionofMethodology References Bibliography

RWJFEvaluationSeries

WhatisEvaluationUse?
Atitscore,evaluationsinvolvejudgmentsaboutthevalueofsomething,suchasaprogramor policy.Determiningprogramvaluetakesresources,timeandeffort.Theresultofthiseffortis usuallyapresentation,reportorotherproductsthatfeaturefindingsabouttheprogramprocess oractivitiesandevents,and/orprogramoutcomesthatincludetheresultsoreffectsofprogram activities.Questionsaboutevaluationusetrytogetatwhetheranyoneeverdidanythingwith thoseevaluationpresentations,reports,orotherproducts.Didanyonereadthem?Listento them?Passthemontoothers?Makeaprogramorpolicyrelateddecision?Changeaprogram activity?Chooseonecourseofactionoveranother?Shiftpriorities?Allofthesemightbe consideredkindsofevaluationuse.Tounderstandfurtherhowevaluatorshaveexplored evaluationuseovertime,seeAppendixA:LiteratureReview. So,whydoevaluatorsmakesuchafussaboutevaluationuse?Whydotheykeepstudyingit? Evaluationuseiscoretobothevaluationstandards 1 andpractice 2 .Deeperthantheseprofessional concerns,however,iscoreevaluationtheoryandbeliefthatevaluationsaresupposedtoalleviate thesocialproblemstowhichprogramsandpoliciesaredirected(Shadish,CookandLeviton,1991). Ifnooneeverdoesanythingwithevaluationsorevenknowsoftheirexistence,thenresources, timeandeffortarewastedandopportunityforsocialbettermentislost(OttosonandHawe, 2009).Evaluationuseisasocialpromise,notprofessionalnarcissism.Perhapsoneofthe respondentsinthiscasestudysumsupevaluationusebest:Theevaluationwasasimportantas actuallyimplementingtheprogram.

1 2

ProgramEvaluationStandards AmericanEvaluationAssociation

3 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

Introduction
Evaluationsaresupposedtobeuseful.Aspreviouslynoted,boththestandardsofevaluation practice 3 andevaluationtheory 4 supportthisassertion.Fordecades,studieshaveexplored whetherandhowevaluationsareused. 5 Isthereanythingnewtosaynow?Wethinkso.We presenthereacasestudyofuseofanevaluationofActiveforLife:IncreasingPhysicalActivity LevelsinAdultsAge50andOlder 6 aprogramsponsoredbytheRobertWoodJohnsonFoundation (RWJF).Thebreadthofboththeprogramandtheevaluationcreatedadditionalwaysofthinking aboutevaluationusewhatitlookslike,whereyoufinditandhowtoconnectit. WefoundthattheActiveforLifeevaluationwasused extensivelybymultiplestakeholders.Duringthecasestudy, Idothinkthereisa weconfirmedfamiliarkindsofuse,suchasconceptualor substantialamountofuse. instrumentaluse(discussedbelow),aswellaskindsofusenot Whenlookingatalltheways welldescribedpreviously,suchasvaluinguse. 7 Wealso theevaluationdatahave uncoveredsequentialpatternsofevaluationusethatwe beenusedandconsidered, calledthreads 8 andleverageduse 9 ofevaluationrelated itshearteningandabit surprisingtoseetherange knowledgeacrosstimeandcontexts.Thesefindingsledusto ofwaysinwhichActivefor thedevelopmentofTheEcologicalModelofEvaluationUse Lifemadeanimpact. (Figure1);weusethatmodeltotellthestoryofthiscase Michelle Maloney,M.B.A. study. InstructorDevelopment DirectoratHumanKinetics ThisreportseekstobehelpfulnotonlytotheRobertWood JohnsonFoundationandotherfoundations,butmorebroadly tothefieldsofevaluation,aging,physicalactivityandpublic healthinunderstandingwhatevaluationuselookslikeincontextandhowitcanbefacilitated.The communityuseofevaluationmayhaveparticularinterestforthoseengagedincommunitybased participatoryresearch;theevidencebasedprogramlinktoevaluationmayhaveparticularinterest forpolicymakersandresearchers;theecologicalunderstandingofevaluationuseaddsa complementaryapproachtofamiliarcategoriesinthestudyofevaluationuse.

TheProgramEvaluationStandardsprovideaguideforevaluatingeducationalandtrainingprograms,projects,andmaterialsinavarietyof settings.Thefourprogramevaluationstandardsareutility,feasibility,propriety,andaccuracy.Reference:AmericanEvaluationAssociation 4 Anidealevaluationtheorydescribeswhyselectedevaluationpracticesleadtoparticularkindsofresultsacrosssituationsandcontexts.Sucha theorywouldincludeissuesfundamentaltoprogramevaluationincludingsocialprogramming,knowledgeconstruction,valuing,knowledgeuse, andevaluationpractice.References:Shadish,Cook,&Leviton,1991;AmericanEvaluationAssociation 5 Studiesofevaluationuseseektoanswerabasicquestion:Didanyonedoanythingwithevaluationfindingsorprocess,suchasmakeadecisionor changeaprogram?SeeAppendixA:LiteratureReview 6 ActiveforLife 7 Valuinguseistheuseofthecoreworkofevaluationtoplacevalueonaprogramorpolicy;itistheuseofthetotalityoftheevaluationnotsolely itsprocessand/oroutcomesTable1. 8 Threadsofevaluationuse 9 Leverageduse

RWJFEvaluationSeries

ADescriptionoftheProgramEvaluated:ActiveforLife
From2003through2006,theActiveforLifeprogramsoughttoimplementtwoefficacious, evidencebased,physicalactivityinterventionsandtodeterminewhethertheseinterventions couldbetranslatedwithappropriateadaptationstobeeffectiveinrealworld,diverse,community settings.Nineleadorganizationsand12geographicallydiversesitesparticipatedinthisnational programbyimplementingeitherActiveLivingEveryDay(ALED)orActiveChoices.(Publicationsof theevidencebasedtrialsforActiveChoicesandActiveLivingEveryDayarelistedinthe Bibliography).(Pleaseclickhereforadditionalinformation: http://www.rwjf.org/pr/product.jsp?id=37128). ActiveLivingEveryDay(ALED)wasevaluatedintwodifferentformats,a20weekprogramanda 12weekprogram.Theoriginal20weekALEDprogramwasconductedinagroupsettingwith12 to20participantsaged50orolderwhowereledbyatrainedfacilitator.ActiveChoiceswasasix monthprogramthatbeganwithaninitialfacetofacemeetingfollowedbyeightoneonone telephonecounselingsessions.Overfouryears,eachinterventionsiteaimedtorecruit900adults, aged50andolderwhorepresentedavarietyofethnicitiesandincomelevels.RWJFestablished theActiveforLifenationalprogramoffice(NPO)atTexasA&MHealthScienceCenterwhereNPO leadershipprovidedtechnicalassistancetotheparticipatingorganizationsandsites,while maintainingcommunicationwithRWJFstaff,theprogramdevelopersandtheevaluators contractedbytheFoundation.(Pleaseclickhereforadditionalinformation: http://www.rwjf.org/pr/product.jsp?id=37128)

ADescriptionoftheEvaluation
RWJFcommissionedtheUniversityofSouthCarolinaPreventionResearchCentertoconductan externalprocessandoutcomeevaluationofActiveforLifefrom2003to2007. 10 Theprocess evaluationresultsrevealedthattheleadorganizationscouldimplementthetwoevidencebased interventionsforolderadultsofdiverseethnicitiesandincomelevelswhilestillmaintaining programfidelitywithagreeduponinterventionadaptations.Opencommunicationamongthe projectstaff,programdevelopersandevaluationteamwasakeyfacilitator(Pleaseclickherefor additionalinformation:ResultsfromtheActiveforLifeProcessEvaluation:ProgramDelivery FidelityandAdaptations).Theoutcomeevaluationfindingsrevealedthatsimilartotheprevious efficacystudies,theprogramparticipantsexperiencedstatisticallysignificantincreasesin moderatetovigorousphysicalactivityandtotalphysicalactivity,aswellasdecreasesin depressivesymptomsandstress,increasesinsatisfactionwithbodyappearanceandfunctionand decreasesinbodymassindex.(PublicationswiththeresultsfromtheActiveforLifeprogramare listedintheBibliography).(Pleaseclickhereforadditionalinformation: http://www.rwjf.org/pr/product.jsp?id=15205).

10

Onaparalleltracktotheprocessandoutcomeevaluations,theREAIMframeworkwasintroducedtohelpparticipatingprogramsassessreach intothecommunityandsustainability.(PleaseclickhereformoreinformationontheREAIMframework: http://www.rwjf.org/pr/product.jsp?id=15569) 5 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

CaseStudyMethodology
ThequestionforthiscasestudyfocusedonwhetherandhowtheActiveforLifeevaluationwas used.Toinformmethodology,weconductedareviewoftheliterature(AppendixA)toidentify conceptualframeworksofevaluationuseanditsfacilitation(Weiss,1979;LevitonandHughes, 1981;KingandPechman,1984;CousinsandLeithwood,1986;AlkinandTaut,2003;Tautand Alkin,2003;Johnsonetal.,2009).Followingthis review,wedevelopedanopenendedstudy protocol. Ithinkyourresultsdoshowuseof theevaluationofcourseIwouldlove Overafivemonthperiodstartingin2009,we toseemoreexamples,afterallthe interviewed23informantsidentifiedthrougha time,blood,sweatandtears(yes,a snowballsamplingtechnique.Thesehourlong few)thatwentintoit.Butyoudid recordedinterviewsexploredpotentialevaluation conductanumberofinterviewswith keyinformants,sonotsuredoing users,evaluationproductsandcommunication morewillyieldanythingmore strategies,examplesofevaluationuseornonuse, substantial. andfacilitatorsofevaluationuse.Tocapturediverse Anonymous perspectivesontheActiveforLifeevaluation,the intervieweesincludedtheRWJFstaff,National Theconclusionseemstobethatthe ProgramOffice(NPO) 11 staff,evaluators,grantees, AFLevaluationwasusefulbecause programdevelopers,consultantsandother stakeholdersmadecommentsthat stakeholders.Inaddition,documentsthatprovided indicatedusefulnessineachofthe evidenceofevaluationusewerecollected,including traditionalcategories emails,presentations,flyersandWebsites.Data Anonymous analysisinitiallyinvolvedtheapplicationof literatureidentifiedconceptualframeworksand evolvedintoTheEcologicalModelofEvaluationUse (Figure1).AfulldescriptionofthemethodologycanbefoundinAppendixB. InformantFeedback ThecasestudyfindingswerereviewedandcritiquedbytherelevantRWJFpersonnelandsentto all23keyinformants.Theywereaskedspecificallyabouttheplausibilityoftheusetypesand facilitatorsidentifiedandtheusefulnessoftherevisedconceptualframeworkproposed.Such membercheckingaddstotheveracityofcasestudyfindings(Merriam,1988).Reviewsofsome keyinformantsareincludedinthisreport.

The RWJF National Program Office Model stresses the provision of technical assistance to its national programs as a way of ensuring that these grantees are well equipped to meet the challenges and opportunities that may be present in a grant initiative.

11

RWJFEvaluationSeries

Findings
AsDetroitwasuniqueinmany WeintroduceTheEcologicalModelofEvaluationUseto ways,Ibelievethatyourfindings framecasestudyfindings.Linearmodelsjustdonottell capturethegroundednatureof ourworkwithsmall,voluntary theinteractivestorywefound.Likeotherecological organizationstoevaluatetheAFL models,thisoneproposesmultipleecosystemsor intervention.Icanseeour contextsofevaluationuseinthiscasestudy,withmulti perspectivesreflectedinyour directionalandmultilayeredinfluences. aggregated findings.Icommend yourefforts,aseachsiteworked Themodelcentersitselfonthecoreworkof withauniquesetofcircumstances evaluationvaluing,e.g.,determiningifthe andconditionsanditcanbea programisgoodorneedsimprovement. challenge,attimes,todecipherthe continuities.Ifeelthatyou Valuingembedsintheimmediateprogram,Active capturedthedynamicsinour settingaccuratelyandinan forLifeinthiscase,wheremultiplestakeholders unbiasedmanner. judgetheprogram.Theimmediateprogram Kimberly Campbell Voytal,Ph.D. includesboththeprogramprocess,e.g.,activities ProgramDirectorforDetroitALED andevents,aswellasprogramoutcomes,e.g.,the program,AssistantProfessorat resultsoreffectsofprogramactivities. WayneStateUniversity Thecommunitycontextencompassesthe geographicalsettingsoftheprogram,e.g.,Chicago orBaltimore;theorganizationswhichhousetheprograms,e.g.,alocalcommunitybased organizationorYMCA;past,currentandfutureprogramparticipants;andlocalpolicy makers,e.g.,organizationaladministratorsorlocalofficials. Thefieldencompassesprofessionalanddisciplinaryinterestsandpracticesthatdirectlyor indirectlyhaveastakeintheimmediateprogram.InthecaseofActiveforLife,thefields includedphysicalactivityandaging,andmorebroadlypublichealth.Includedinthefields arepractitioners,academics,advocatesandpolicymakers.Privatefoundationsstraddle fieldandsocietyinfluences. Societyencompassesbroadcultural,social,political,economicinfluences,trendsand policies.Incorporatedintothesearesocialneeds,valuesandprioritiesandthe negotiationsamongthemforpublicattention. Themultiplecontexts,asrepresentedbytheseconcentriccircles,influenceeachother.For example,throughanevaluationthefieldmightplacevalueontheimmediateprogramasabest practiceandthepublicationofsuchfindingsmightinfluencesociety(policymakers)toprovide additionalfundingforsuchprograms.

7 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

Cuttingacrosstheseconcentriccontextsofusearemultiplepathstounderstandingevaluation use:typesofuse;sequencesofuse(welabeledthreads);andleverageduseovertimeand context. Weidentifiedseventypesofevaluationuseinthiscasestudyincludingvaluinguse, instrumentaluse,conceptualuse,evaluationlearninguse,symbolicuse,communication useanddecisionmakinguse.ThesearediscussedbelowandhighlightedinTable1.As notedpreviously,someofthesearefamiliartypesofuse,suchasinstrumentaland conceptual;othersarenewlydescribed,suchasvaluinguse. Thethreadsofuseuncoveredinthecasestudyshowsequencesamongtypesofevaluation use.Examplesofsuchthreads(page15),arenotmeanttoimplyanyuniversalorderof evaluationuse;rather,thethreadssuggestthattypesofevaluationusearenotisolated eventsandunderstandingtheirconnectionsmayadvanceourunderstandingofhow evaluationsareused. Leverageduseincludestheevaluationrelatedlearning,toolsanddatasets,networks, relationshipsandotherbyproductsofmultipleevaluationstakeholdersthatinfluencenot onlytheimmediateprogram,butarecarriedacrosstime,settings,fields,jobsand disciplineswithinfluenceelsewhere(describedindetailonpage16).
Figure1:TheEcologicalModelofEvaluationUse (Source:OttosonandMartinez,2010)

RWJFEvaluationSeries

 TypesofEvaluationUse Table1identifies,definesandexemplifiessevengeneral typesofevaluationusefoundinthiscase,aswellas additionalsubcategories.Whilemanyofthesetypesofuse relatetotheimmediateActiveforLifeprogram,theyare notlimitedtoitandcanbefoundinothercontextsas suggestedbythemodelanddescribedbelow. Valuinguse.Stakeholdersusedtheevaluationtotest theprogram,proveefficacy,provideevidenceandoffer credibility.Severalinformantsindicatedthatwithout theabilitytotesttheinterventiontheycouldnot havechanged,disseminated,marketedor recommendedtheintervention.Whileitmaybe temptingtodismissthisassymbolicuse,e.g.,themere existenceofevaluation,thereissomethingdifferent, deepergoingonhere.Valuingisacomplexprocessthat isasmuchaboutculture,social,personal,economic andpoliticalforcesasitisaboutmethodology.Itisa processthatisdoneincontextamidstmultiple stakeholdersandpowerdifferentials.Fundamentally, theevaluationenabledvaluetobeplacedonthe wholeActiveforLifeprogram,aswellasits components,itsprocessanditspurpose.Stakeholders didnotusejusttheevaluationprocessoritsfindings; theyusedthetotalityoftheevaluationefforttotestthe program.

Ilikedthismodelmadesenseto meandhelpedtointegratethe complexityintoanicevisual model. SaraWilcox,Ph.D. Evaluator,ProfessoratUniversity ofSouthCarolina Ithinkthatwouldworknicelyit providesaframeworkand everythingisgearedtothe graphic/infoasyouhave presented.ButthenImavisual learnersoifsomeonequestions thatapproachIcouldbe convincedotherwise.Thinkwhat youhavesharedgivesclaritytoa potentiallyconfusingtopicso convoluted. DianeDowdy,Ph.D. NPODeputyDirector,Assistant ProfessoratTexasA&MHealth ScienceCenter

Reallylikedit.Likehowyou explainedfullybutnottoo lengthyprovidinglinks.Likehow youweresoopenwith processtryingtofollowprevious modelsbutthenhavingtotake anothertwistortwotocaptureall thelessons.EasytoreadandIm notanevaluator.Foundtheflow smoothinfohelpful.Likehow youemphasizedtheopen communicationandsharingatall levelsandfromallparties. DianeDowdy,Ph.D. NPODeputyDirector,Assistant ProfessoratTexasA&MHealth ScienceCenter

9 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

Table1:TypesofEvaluationUse
TYPESOFUSE VALUINGUSE n=13 DEFINITIONSOFUSE Useofthecoreworkofevaluationtoplace valueonaprogramorpolicy;useofthe totalityoftheevaluationnotsolelyits processand/oroutcomes Evaluationprocessoroutcomefindingsare directlytiedtouse,inrelativeproximityof timeandwithvisibleeffect.Decisionmaking inherentinactionstaken Actionstakentowardprogramactivitiesand processes EXAMPLESOFUSE Testtheprogram;provideevidence; proveefficacy;rigorousevaluation

INSTRUMENTAL USE

PROCESS FINDINGS Improve implementation n=6 Adaptprogram; program products n=7 Improve evaluation n=1 OUTCOME FINDINGS Recruit n=6

Trackactivities;identifyredflags; guidestakeholderimplementation discussions;adjustprogramactivities Adapt,tweak,ormodifyprogram; identifyessentialprogramelements; addcontent;improvematerial usefulness Changestoevaluationprocess, instrumentsandfinetuningthe system

Actionstakenrelativetoprogramcontent, materialsandcoreelements

Actionstakentomodifytheevaluation

Outcomefindingsusedtomeetprogram requirementsforparticipantrecruitment

Changeprogram n=>10

Recruitparticipantsandcommunity partners;promoteandmarketthe program;usesuccessstoriesand otherdata Policyandstructuralchangestoprogram(not Original20weekprogram tweaking) restructuredintoa12weekoption; medicalreleaserequirementdropped Outcomefindingsusedtosecurefunding Catalystforfunding;findingswritten intograntapplications;publications attached

ObtainFunding n=5

Note:Inthefirstcolumnthenequalsthenumberofinformantswhoprovidedanexampleofthistypeofuse.

RWJFEvaluationSeries

10

Typesofuse n=#of informants CONCEPTUAL USE EnableChoice n=5

DefinitionsofUse

ExamplesofUse

Evaluationleadstoachangein understanding,notdirectaction Processandoutcomefindingsenable choiceamongoptions

Chooseprocesses;chooseamongprogram options;chooseanotherprogram;make recommendations Provideinsight;movetheagenda,furtherthe work;serveasatippingpoint Lessonslearnedhaveotherlifeinexercise guide;infusefindinginnewprogram; supportingdocument;legacyeffect

Advance discourse n=4 Embedconcepts n=4

Evaluationfacilitatesdiscussionand thinking,especiallyinthefield Findingsfromevaluationembedded intootherforms,contextandfields

Provide reference point n=7 EVALUATION LEARNINGUSE

Findingsconfirmcommunitytested, evidencebasedprogram

Concreteexampleofatestedprogram, includingitsevaluation;sourceofdataand methods;helpbuildacasefortheprogram

Changesinevaluationstakeholders and/orparticipatingorganizationsas aresultoflearningthatoccursduring theevaluation

Organizational Learningwithintheorganizationsthat learning receivedprogramorevaluation n=6 funding;orthatparticipatedinthe evaluation Stakeholder learning n=4 Learningamongdiverseevaluation stakeholdersincludingprogram managers,developers,participants andfieldassociates Learningamongparticipating evaluators

Theevaluationhastakentheorganization downaroadofmoreindepthevaluation; understandtheimportanceofevaluation; sustainedevaluationactivities Insights;organiclearning;understand implementation;howtogetdatabackto scientificandparticipatingcommunities

Evaluator learning n=3

Managelargescaleevaluation;moreefficient; understandevaluationtoolincontext;realities ofdatacollection

11 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

Typesofuse n=#ofinformants SYMBOLICUSE n=2

DefinitionsofUse Themereexistenceofan evaluation(notitsresults)is usedtopersuadeortoconvince

ExamplesofUse Evaluationexistenceaidedparticipant recruitment;organizationaland/orpersonal credibilityforevaluationparticipation

COMMUNICATION USE Advocacy n=7 Professional dissemination n=7 Networks n=3 Community events n=5

Evaluationfindingsareusedas theymovethroughmultiple communicationchannels Evaluationfindingsusedto influencedecisionmakersand resourceallocation Findingsusedinmultiple professionaldissemination channels Interagencynetworksaround programarea,suchasagingor physicalactivity Evaluationfindingusedin communityevents

Educatelawmakers;testifytoownphysician; attendCongressionalbriefingonfindings Publishfindingsinscientificjournals;conference presentations;usedataindissertation;written reports Contributeideastoevaluationinstrument;share learningfromevaluationprocess,notjust outcomes Evaluatorsharesfindingswithparticipantsand community;participantsuccessstories;senseof pride;reinforcenewhabits;accountability moment;recruitment MultiplestakeholderWebsites;Websitehitscan leadtopersonalcontactformorefindings Pressreleasetonewspaper;findingspublishedin 5countymagazine Improveprogramprocess;recruitparticipants; structuralchangetoprogram;advocate; disseminate

Websites n=4 Media,press n=1 DECISIONMAKING USE

Electronicaccesstoevaluation findings,toolsandprocesses Evaluationfindingsusedin mediareleases Useoftheevaluationtoinform decisions

RWJFEvaluationSeries

12

Instrumentaluse.Stakeholdersdirectlyusedprocess andoutcomeevaluationfindingstotakeconcrete,near termaction.Processfindingswereusedtoimprove Ineededmoreclarityabout programimplementation,adapttheprogramorits howonegetstoastrong products,e.g.,changemanualsandtoimprovethe conclusionofusefulness.Isit evaluation,e.g.,refinetools.Stakeholdersused thebreadthofcomments?The natureofthecomments?An outcomefindingstorecruitfutureprogramparticipants, interpretationthatcomments changetheprogramstructure,e.g.,reduceprogram areevidenceofimpact? durationandtoobtainfundingforotherprograms. Anonymous Someoftheseactionsoccurredintheecologicalcontext oftheimmediateprogram,e.g.,implementation Imaybebiasedbutthinkthat changes;somechangesoccurredinthecommunity,e.g., youclearlyshowedextensiveuse participantrecruitment;somechangesoccurredinthe ofevaluationonmultiplelevels fieldofphysicalactivity,e.g.,securingfundingforother andinmultipleways.Iwas projects. surprisedastohowyou capturedsomanyideasand expressedthemsosuccinctly. Conceptualuse.TheActiveforLifeevaluationinformed DianeDowdy,Ph.D. somestakeholderunderstanding,eveniftheytookno NPO Deputy Director,Assistant directaction.Theevaluationenabledchoiceamong Professor at TexasA&MHealth alternatives,e.g.,programoptionsorprocesses;it ScienceCenter advanceddiscourseinthefieldbymovingtheagenda forwardorprovidinginsight;itembeddedfindingsin otherformsorfields,e.g.,infusedintoanewprogram orexerciseguides;itprovidedareferencepointforactionordecisions,e.g.,casebuildingfor otherprograms.Intheecologicalmodel,wefoundmanyoftheseconceptualexamplesinthe field,particularlyinphysicalactivityandaging. Evaluationlearninguse 12 .Variousstakeholdersandorganizationslearnedaboutevaluationby participatinginthetotalityoftheActiveforLifeevaluationeffort.Someorganizationsthat participatedintheActiveforLifeprogramsustainedevaluationactivitiesafterfunding subsidedorinitiatedevaluationsofotherprograms.Diversestakeholders,includingprogram managers,developersandparticipants,usedtheevaluationexperiencetogaininsightsor understandprocessesdifferently.Eventheevaluatorslearnedmoreaboutthemanagementof largescaledatabasesortherealitiesofdatacollection.Examplesofthistypeofuseemergein theecologicalmodelinthecontextsoftheimmediateprogram,communityandfield. Symbolicuse.Theconductofanevaluationtomakeaprogram,funderorstakeholderlook responsive,ratherthanberesponsivetoprogramresults,representssymbolicevaluationuse. Afewexamplesofthiskindofuseemergedinthecasestudy.Intheseexamples,respondents referencedortoutedtheActiveforLifeevaluationtogainpersonal,programmaticor

Rather than using the term evaluation process as a category of use (Patton, 1997), we have labeled a comparable kind of use as evaluation learning. The re-labeling helps avoid confusion between using process evaluation findings and learning from the totality of the evaluation effort (not just its process).

12

13 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

organizationalcredibilitybyvirtueofbeing associatedwithanevaluation.Symbolicusein YesmakesgoodsenseandIthink thiscasewasmostoftentiedtootherkindsof helpsunderstandthemultilevel evaluationuseandnotasoleexampleofuse. approachandtheopennesstousing thelessonslearned.Isntthatexactly Communicationuse.Previousstudiesof whatyouaretryingtodemonstrate withthispiece?dontbetiedto evaluationuseestablishedtheeducationof followingaformatbuthaveanopen decisionmakersasacategoryofevaluation mindtolearningwhatnewthereisto use(CousinsandLeithwood,1986).Forthis learnwhilebeingthankfulfor casestudythatcategorymorphedintothe validatingpreviouslessonslearned?I broadercategoryofcommunicationuseto likeditandtickledthatAFLcould captureadvocacy,professionaldissemination, demonstratethatsoclearly networking,communityevents,Websites, (especiallythatyoupickeduponthe media,resourcecenters,intraorganization uniquenessoflearning). reportingandeducation.Thesewerenot DianeDowdy,Ph.D. sterilecommunicationchannelsthroughwhich NPODeputyDirector,Assistant evaluationlearningandfindingspassed ProfessoratTexasA&MHealth ScienceCenter untouched.Thesewereimplementation channelsthroughwhichevaluationlearning andfindingswereadapted,adopted,engaged, discussedandused(PressmanandWildavsky, 1984).Forexample,somecommunitiesused evaluationfindingstotestifylocallyandtocelebratetheircollectiveaccomplishment;other stakeholderstooktheevaluationfindingstotestifyonCapitolHill.Inthesecommunications, stakeholdersengagedandusedtheevaluation. Decisionmakinguse.Stakeholdersusedevaluationfordecisionmakinginnearlyallthe precedingtypesofevaluationuse.Stakeholdersmadedecisionstoevaluate(valuinguseor symbolicuse);decisionstoimprovetheprogramoritsimplementation(instrumentalprocess use);decisionstorecruitorchangetheprogram(instrumentaloutcomeuse);decisionsto chooseorembedconcepts(conceptualuse);anddecisionstocommunicateaboutthe evaluation(communicationuse).Withdecisionmakinginherentinallofthesecategories,we wereloathtopullitoutasaseparatecategoryorputitonparwithactionstaken.Intheend wedecidedtodoso,however,sothattheuseofevaluationfordecisionmakingnotbe embeddedoutofsightandoutofmindinthiscase.Stakeholdersusedtheevaluationfor decisionsandaction. TheseindividualcategorieshelpilluminatewaysinwhichvariousstakeholdersusedtheActivefor Lifeevaluationandthemultiplecontextsinwhichtheydidso.Althoughvaluingiscoretothework ofevaluation,thecategoryofvaluinguseoffersanewcontributionthroughthiscasestudy,as doestheexpansionofprocesslearningtoevaluationlearningandeducationuseto communicationuse.Ashelpfulasthesecategoriesare,theytoldonlypartofthestoryon evaluationuseforthiscasestudy.
RWJFEvaluationSeries

14

ThreadsofEvaluationUse

Tomorefullytellthestoryofusewefoundthatweneededtomoveawayfromasole focusonseparatecategoriesofuse.Ourstrugglewithwhattodoaboutdecisionmaking uselinkedtoactionorotherkindsofuseledustoexplorepatternsorlinksamong categories.Welabeledtheseasthreadsofevaluationuse.Thesethreadsinvolvetracing whatprecedes,interveneswithand/orfollowstypesofevaluationuseacrosscontexts, timeandstakeholders,asshownintheecologicalmodel.Someexamplesofevaluationuse threadsfoundinthiscasestudyareillustratedbelowinquotesfrominformantswherethe individualcategoriesofusehavebeenitalicized:

Ifprogramshadjustbeenimplemented,butnotevaluated(valuinguse),theywouldnt havebeenpickedup(decisionmaking)sincethepushnowistogetontheevidence basedlist(referencepoint)byhavingpublishedpeerreviewedarticles(professional dissemination)...beingontheCMS[CentersforMedicareandMedicaidServices]listing ofevidencebasedprograms(referencepoints)madetheprogrameligiblefor reimbursement(obtainfunding). Wetooktheprogramandgrewit(changeprogram)insomanydifferentdirections (adaptprogram)becauseweknewitworkedandcouldproveitworked(valuinguse). Havingthesearticles(professionaldissemination)gaveusleveragetopresentat conferences(professionaldissemination)sincethefindingscarryalotofweight (valuinguse).canreachalotofthoughtleaders(decisionmaking)andpeoplethat canimplementtheprogramsatthecommunitylevel(improveimplementation). Wehadtheluxurytoexperiment(evaluationlearning)withashortercourse(change program)andtest(valuinguse)whethertheshorterprogramwasaseffective.

Valuinguseanchorseachofthesethreads.Throughevaluationlearninganddissemination, decisionsaremade,programchangesoccur,andfundingmaybeattracted.Thesekindsofuse crisscrossmultiplecontextsofevaluationuseandarecharacteristicofthereciprocaldeterminism ormutualinfluenceinherentinecologicalmodels(Green,PotvinandRichard,1996).Forexample, thefieldcaninfluencetheimmediateprogrambyrecommendingpromisingpracticesandthe immediateprogramcaninfluencethefieldbydemonstratingthefeasibilityofpractices.By understandingtheevaluationusecategories,itispossibletopushafurtherunderstandingofhow thecategoriesrelatetoeachother,overtimeandthroughmultiplecontexts.Intheendwedidnot findonestoryofevaluationuse,wefoundawebofinterconnectedvignettes.

15 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

LeveragedKnowledge Muchoftheevaluationuseliteraturefocusesonusein thecontextofaspecificprogram,itsorganization,orits Havetheresultsofthe evaluationbeenusedinother stakeholders.ThewidecanvasoftheActiveforLifecase areasinadditiontophysical studyalloweddemandedanexplorationofuseof activityandaging?...Ifthere evaluationlearningbeyondtheimmediateprogram. areexamplesthatcouldbe Stakeholderschangedjobs,evaluationfindingsdiffused, cited, Ithinkitwouldmakefor evaluationtoolswereadaptedtootherprogramsand astrongercaseforevidenceof insightskeptemerging.Inallofthese,theevaluation evaluationuse.[SeeREAIM experiencefromtheActiveforLifeprogramwas Example] leveragedovertime,context,programsandstakeholders. TeresaKeenan,Ph.D. Belowaresomeexamplesofothersituationsinwhich HealthResearchTeamLeader theActiveforLifeevaluationwasused: atAARP Otherprograms: Evaluationtools,formsanddatabaseservedasamodelfor evaluationofanewprogram;improveevaluationofother programs Othergrants: EvaluatorusedtheActiveforLifeevaluationprocesstoidentify coreprogramcomponentsinanewNIHgrant Othersites: EvaluationprocessandtoolsusedatexpandedALEDprogram sites;evaluationprocessinfusedintootherprograms;evaluation toolsusedinotherstates Otherpurposes: EvaluationlearningledtoREAIMexpansionfromsummativeto formativeevaluationtool Otherfields: EvaluationlearningledREAIMtomovebeyondbehavioral medicineandpublichealthtoincludethefieldsofagingand physicalactivity Otherorganizations: Networksofprofessionalsororganizationsleveragedevaluation learningtomultipleparticipatingorganizations Otherclients: Programdeveloperssoldimprovedparticipantmanualstoother clientsworkinginrelatedfields Otherparticipants: Findingsofeffectivenessopenedthedoortoworkingwithmore diversecommunities Otherwork: Reshufflingofstakeholdersovertimeledtonewuseforfindings, e.g.,localprogrammanagersworktogetheronnewgrants; programmanagerservesasanevaluationconsultant

RWJFEvaluationSeries

16

Tofocusevaluationuseonlyontheimmediate programorcontextmissesalot.Whilethethreads Didfinditinterestingthatongoing ofevaluationusetellastory,anexplorationofthe useofdatawashelpfulbutthena leveragedevaluationrelatedknowledgepaintsthe doubleedgedswordinthatittooka bigpicture. whiletogetthecleaned data/analyzed/publicationbefore FacilitatorsofEvaluationUse couldbefullyutilizedbythesites. DianeDowdy,Ph.D. TheJohnsonetal.(2009)frameworkinformed,not NPODeputyDirector,Assistant dictated,ourfindingsoffacilitatorsofevaluation ProfessoratTexasA&MHealth use. ScienceCenter Opencommunicationandstakeholder involvement.Afterinterviewing23stakeholders involvedintheActiveforLifeprogram,itwasquitetellingthateveryone,e.g.,NPOstaff, evaluators,leadorganizationstaff,RWJFstaff,wasinagreementastothepurposeofthe ActiveforLifeprogramanditsevaluation.Opencommunicationandstakeholderinvolvement establishedaclearunderstandingoftheobjectivesofprogramgoals,programprocessesand measurableevaluationoutcomes.Thegranteesfeltcomfortableinhelpingtheresearchers understandhowtheActiveforLifeprogramscouldbetranslatedintorealworldsettingsand whatadaptationsneededtobemadetoenhanceparticipantrecruitmentandcommitmentto theentirelengthoftheActiveLivingEveryDay(ALED)andActiveChoicesprograms. Competenceoftheevaluatorsandtheevaluationquality.Multipleinformantsexpressed theirappreciationthattheevaluationteamattheUniversityofSouthCarolinaconductedthe ActiveforLifeevaluationwithaflexibleapproachtolearnhowbesttocollecttheevaluation datafromtheleadorganizations.Oneinformantcommentedthattheevaluatorknewand understoodhowtoworkwithcommunitybasedorganizations.Mutualrespectbetweenthe evaluatorsandprogramstaffenabledthegranteestobecomemorereceptivetothe evaluationprocess.Thisevaluationformatultimatelyenhancedthecompletenessofthe evaluationdataandbroadenedthegranteesunderstandingofhowtheevaluationfindings couldbeusedtoimproverecruitmenteffortsandtocreatenewpartnershipsforadditional locationsofferingtheALEDorActiveChoicesprogramduringthefouryeargrantperiod. Preliminaryevaluationfindingssharedonanongoingbasisduringtheannualgrantee meetings.Thisstrategyfordisseminatingevaluationfindingsfurtherfacilitateduseand understandingoftheevaluationresultsbytheleadorganizations.Oneinformantcommented thatpreliminaryevaluationresultsduringthegrantperiodwerehighlyanticipatedbecausethe leadorganizationstaffandprogramparticipantswantedtoknowiftheevidencebased programworkedforthem.Buyinatthelocallevelwasachievedatonesiteparticularlywell sincetheyhadthefollowingapproachtotheActiveforLifeprogramandevaluation:Lets participatesothatwecanshowtheresearchers[oftheoriginalefficacystudies]howitworks anddoesntworkforus. 17 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

Articlepublicationsandconferencepresentations.Useoftheevaluationfindingsby professionalsinthefieldsofagingandphysicalactivityresultedfromarticlepublicationsand conferencepresentationsgivenbytheActiveforLifeNPOstaffandevaluators.Oneinformant fromthefieldviewedtheActiveforLifeevaluationresultsasbeingatippingpointforcertain nationalorganizationstoprovidemorefinancialsupportforevidencebasedprogrammingat thelocallevel.TheActiveforLifeevaluationwasalsoviewedastrulyalandmarkinthelevel ofevaluationwhosefullbenefitwasbeyondjustscientificpapers.Anotherinformant commentedthat:Fromthenationalperspective,if[theActiveforLifegranteeswere] successfulinimplementingtheprogramsandiftheevaluationdatawasstrong,itwouldhelp [certainnationalorganizationsto]becomemoreengagedinevidencebasedprograms.The evaluationwasasimportantasactuallyimplementingtheprogram.

BarrierstoEvaluationUse WhilethiscasestudyhaspresentedarangeofuseexamplesfortheActiveforLifeevaluation,the informantsalsoreportedbarrierstouse. Time.Oneofthebiggestimpedimentstotheuseofevaluationfindingswasthetimeittook forthegranteestoreceivethemduringthefouryeargrantperiod.Certainstakeholders viewedtheevaluationdataasbeingcrucialfortheirsustainabilityeffortsinsecuringfuture funding,withthedelayspreventingthemfromhavingdatatosupporttheobservedsuccesses thatwereoccurringontheground.Sincetheevaluatorspublishedthefinalevaluationresults in2008,twoyearsafterthegranthadformallyended,someinterviewedstakeholderswere notawareoftheirexistenceinapeerreviewedjournal. Nonuseoftheevaluation.AlthoughoneActiveChoicesgrantee,BlueShieldofCalifornia,had asuccessfulexperienceinimplementingtheprogram,acorporatedecisionwasmadeto eliminatetheentiredepartmentthatrantheActiveforLifeprogram.Despitethepositive evaluationresults,thiswasanexampleofhowacorporatedecision,nottheevaluation findings,determinedtheprogramsfuturebyendingit. TheFoundationsshiftawayfromthefieldofaging.In2005,theFoundationmovedtowards itsnewcommitmenttoreversingthechildhoodobesityepidemic.Thisshiftoccurredabout halfwaythroughtheActiveforLifeprogramandresultedinsomestakeholdersfeelinglike RWJFhadabandonedtheagingfieldoncetheActiveforLifeprogramendedin2006.The FoundationsshiftawayfromagingmayhavepreventedtheActiveforLifeevaluationsfrom beingusedifpotentialusersmaynothavevieweditasagotosourceforthisinformation. Partnershipsmayhaveendedafterthegrantperiod;sustainabilitywasnotpossiblewithlack ofresources.Oneinformantcommentedthatwhenleadersatauniversityand/orcommunity basedorganizationleave,thememoryandinstitutionalknowledgeoftheprogramssuccess fadesforthatorganization.Asaresult,themomentumtosustainaprogrammayendandthe previouspartnersandaccesstoresourcesmaynolongerbeavailable.
RWJFEvaluationSeries

18

Conclusions Atthebeginningofthisreport,westatedourconclusionthattheActiveforLifeevaluationwas (andisstillbeing)used.Bythispoint,thereadermightagreewithus.Thenumberofexamples, theirspecificity,thecorroborationsanddocumentsthatsupportthem,andthenumberof stakeholderswhoofferedthemconvincedusthatusehappened.Further,weconcludethatthere wasalotofusemultipleexamples,bymultiplestakeholders,inmultiplecontextsandthatsome ofthiswasbiguse,e.g.,changedprogramstructure,organizationallearning,advocacyand leveragedusetootherprograms.Understandinganddiscoveringcategoriesofevaluationuse enabledustohearthestoriesorthreadsofevaluationuseandseethebiggerpictureofleveraged use.Findingtheobvious,valuinguse,remindsusthatevaluationuseoccursnotjustincategories, butalsointhetotalityoftheevaluation.Thiscoreworkofevaluationvaluinganchorsalluse. Thefacilitatorsandbarrierstoevaluationusehelpedexplainwhyuseoccurredinthiscase. Further,theyconcurwiththeevaluationuseliterature.Johnsonetal.(2009)concludedthat: engagement,interactionandcommunicationbetweenevaluationclientsandevaluatorsarekey tomaximizingtheuseoftheevaluationinthelongrun(p.389).Thesekeyactionsexistedand persistedinthiscasestudy.

19 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

Implications FortheRobertWoodJohnsonFoundation.TheActiveforLifeprogramisanexemplarymodel forhowthecommunicationqualityandpartnershipsamongRWJFstaff,NPOstaff,project staff,andtheevaluatorswereessentialtouseoftheevaluationfindings.Certaininformants commentedthatbyhavingRWJFstaffandNPOstaffsetthetoneforthecomplementaryroles ofprogramimplementationandevaluationatthebeginningoftheinitiative,theprogramstaff andprogramparticipantsbetterappreciatedtheevaluationprocessandlearnedhowthe resultscouldstrengthentheirprogrammingandpotentialsustainability.RWJFprogramstaff shouldlooktotheActiveforLifeevaluationasexemplifyingtheimportanceofopen communication,strongdualleadershipfromtheNPOandevaluatorsandtheimportanceof engagingstakeholdersatthestartofprogramevaluation.(PleaseclickhereforanotherRWJF fundedevaluationproduct:AGuideforEngagingStakeholdersinDevelopingEvaluation Questions) Forfoundationsingeneral.Foundationstaff,programstaffandevaluatorsmustbein constantcommunicationandsupporteachotherinholdingbothprogramimplementationand evaluationasbeingcomplementaryandimportantforlearningandprogramimprovement. Mutualrespectamongthesegroupswillformulatestrongleadershipfromthebeginningthat willcarrythroughthegrantperiodandenhancedatacollectionanduse/understandingofthe evaluationfindings.Sincefoundationsdonotalwaysfinanciallysupportprofessionalnetworks, itwasrecommendedbysomeinformantsthatfoundationsrecognizethebenefitsof professionalnetworksandconsidersupportingthesenetworkstosustainandfurtheradvance aparticularfield.Itwasalsorecommendedthatfoundationsencouragethepublicationof evaluationfindingsbeyondpeerreviewedjournals,e.g.,trademagazines. Forevaluators.Anecologicalunderstandingofevaluationuseopenedthedoortomultiple pathstostudyevaluationuse,includingtypesofuse,threadsofuseandleverageduse.Itgave usabroadperspective.Forthevisualtypeswhetherevaluators,policymakers,program managers,orcommunitymembersthemodelprovidesawaytosee,track,connect,place, groundandenvisionevaluationuse.Wethinkthemodelisworthtestingfurther.Thisbringsus tothenamingofvaluinguseasanotherkindofusetoaddtocategoriesofevaluationuse.In thiscase,evaluationwasusednotjustforitscomponents,e.g.,processlearningoroutcomes, norforitssymbolicpresence.Thetotalityoftheevaluationtestedandvaluedtheprogram. Valuinguseanchorsallevaluationuse.Inaddition,evaluatorcompetenceandunderstanding oftheevaluationusersareimportantduringtheevaluationprocesstoensurethatfindingsare relevantandunderstoodbytheusers.TheActiveforLifeevaluationwaseffectivelyusedto sustainsomeoftheprogramssincetherewasopen,honestcommunicationamonggrantees, theNPO,theprogramdevelopersandtheevaluatorsonwhatworked,andwhatdidntwork, andhowthiscouldberemedied.

RWJFEvaluationSeries

20

Forotherstakeholders.Evaluationfindingscanbe usefulfortheprogramdevelopersandprivate marketersoftheprograminorderforthemto knowhowtheprogramistranslatedintoreal worldsettings;andformarketers,howtobetter communicateabouttheprogram.Notevery programwillworkineverysetting.Bytestingand understandingadaptationsthatcanbemade whilemaintainingprogramfidelity,thedevelopers learnwhatworksandthemarketerslearnhowto betterservetheircustomerbasebefore,during andaftertheprogramshavebeenimplemented. Professionalconferencesandnetworksalsoserve asameansforbringinganewandevolvinglifeto evaluationfindingsandtheiruse.

Inclosing,thiscasestudyofuseofanevaluationof theActiveforLifeprogramprovidednewinsights, somenewcategoriesofevaluationuseandan ecologicalapproachtothestudyofevaluationuse. Thecasestudyalsoconfirmedsomeoftherecentworkintheevaluationliterature,especiallyon thefacilitatorstoevaluationuse.Wehopeweaccuratelyreflectedtheexperienceofcasestudy participants.Mostly,wehopethiscasewillbeusefulbeyondthesepages.

Struckmeasasubstantialamountof use(whichwassonicetosee!!) SaraWilcox,Ph.D. Evaluator,ProfessoratUniversityof SouthCarolina Itwasnotasurprisingamount... givenwewerereally"living"that experience.But,itwasgreattoread thereportandberemindedofjust howmuchwasdoneandhowmuch theevaluationwasused(which makesmefeelgreataboutthe work). SarahGriffin,Ph.D. Evaluator,AssistantProfessorat ClemsonUniversity

21 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

AppendixA LiteratureReviewonEvaluationUse BeforeassessingwhetherornotuseoftheActiveforLifeevaluationoccurred,anunderstandingof thetheory,practiceandmethodologyofevaluationusewassoughtbyconductingaliterature reviewofempiricalresearchpublishedduringthelast40years.Thefollowingonlineresearch databaseswereusedtoidentifyarticlesforthisliteraturereview:JSTOR,PROQUESTandERIC.The followingkeywordsweresearchedforinthedatabases:evaluationuse,evaluationutilization,use ofevaluationandprocessevaluation.Thisliteraturereviewresultedinanexaminationof36 articlespublishedfrom1979to2009.Asummarizationoftheevaluationuseconceptsoutlined belowenabledustotakewhatwelearnedfromtheliteratureandapplyittoourevaluationuse casestudymethodology. TypesofEvaluationUse Theconceptofuseiscoretoboththefoundationsofevaluationtheory(Shadish,Cookand Leviton,1991)andprogramevaluationstandards(TheJointCommitteeonStandardsfor EducationalEvaluation,1994).Evaluationsaresupposedtobeuseful.Howevaluatorshave understooduseandwhatfacilitatesithaveevolvedoverthelastfewdecades.Oneoftheearly understandingsofuseidentifiedsevendifferentcategoriesofevaluationuse,rangingfromthe rationaleofproblemsolvingtothefodderforpoliticalnegotiations(Weiss,1979).Bytheearly 1980s,threegeneralkindsofuseofevaluationfindingswereidentified:instrumental,conceptual andsymbolic(Weiss,1980;LevitonandHughes,1981;KingandPechman,1984).Instrumentaluse occurswhenevaluationfindingshavebeenusedindecisionmakingorproblemsolving.Suchuseis morelikelytooccursoonerratherthanlateranddirectlyratherthanindirectly.Conceptualuse occurswhenapersonhasbeenenlightenedbytheevaluationfindings,butnodirectactionis taken.Symbolicuseoccurswhentheevaluationfindingsarenotused,butrathertheexistenceof theevaluationisreferencedtopersuadeorconvince.In1997,Pattonintroducedtheconceptof processuseinwhichindividualsaltertheirbehaviorsand/ororganizationschangetheirprogram structureasaresultoflearningfromtheevaluationprocess.Thisaddedlensbroadenedan understandingthatanevaluationcanbeusefulformorethanitsfindings. Insubsequentliteraturereviews,newandrefinedunderstandingsofevaluationusewere identified.CousinsandLeithwood(1986)examined65empiricalstudiespublishedbetween1971 and1986.Theseresearchershighlightedthefollowingthreeconceptsofuseofevaluation findings:(1)useasdecisionmaking,(2)useaseducation,and(3)useastheprocessingof evaluationinformation.Examplesofdecisionmakingincludeddiscretedecisionsaboutprogram funding,programoperationsand/orprogrammanagement.Withrespecttotheconceptofuseas education,evaluationfindingsmayhelpeducatedecisionmakersandprogramstaffaboutthe structureandintendedoutcomesoftheprogrambeingevaluated.Theprocessingofevaluation informationoccurswhenprogramstaffrespondstotheevaluationfindingsbyeitheractingupon therecommendationsand/orimprovingaspectsoftheprogram.Anotherliteraturereviewfound thatknowledgeforactiontheoriestransfer,translation,diffusion,implementationand knowledgeutilizationshapewhatisvalued,howitisvaluedandwhatconstitutesevaluationuse
RWJFEvaluationSeries

22

(OttosonandHawe,2009).Afurtherexpansionbeyonddirectuseofevaluationprocessesor findingswasconductedbyKirkhartwhoestablishedthetheoryofevaluationinfluence,whichshe definedasthecapacityofpersonsorthingstoshapeothersinindirectways(2000).Researchers HenryandMark(2003)havealsoexploredthisconcept,viewingthedifferingpathsandlevelsof evaluationinfluenceasameansthroughwhichevaluationsleadtotheimprovementofsocial conditions. FacilitatorsandBarrierstoEvaluationUse Theconceptofuseshapestheunderstood(andresearched)barriersandfacilitatorsofsuchuse.In 1986,CousinsandLeithwoodidentifiedthefollowingsixfactorsthataffectedevaluation implementationintheempiricalstudiestheyreviewed:evaluationquality,credibility,relevance, communicationquality,findingsandtimeliness.Theseresearchersalsohighlightedthefollowing sixfactorsrelatedtodecisionorpolicysettings:informationneeds,decisioncharacteristics, politicalclimate,competinginformation,personalcharacteristicsandcommitmentand/or receptivenesstoevaluation.Afterreviewingtheevaluationuseliterature,CousinsandLeithwood concludedthatevaluationusewasmostevidentwhenintendedusersoftheevaluationwere involvedfromthebeginning,theevaluationmethodswereappropriateandtheevaluation findingswererelevanttotheintendedusers.Nonuseanddiminishedusewerealsopotential possibilitiesiftheevaluationresultswereperceivedbytheintendedusersasathreat(Cousinsand Leithwood,1986). ResearchersAlkinandTaut(2003)andTautandAlkin(2003)furthercontributedtotheliterature throughtheirexplorationofthefollowingbarrierstoevaluationuse:humanfactors(evaluatorand usercharacteristics);programcontext(contractualobligations,fiscalconstraints,organizational characteristics,externalcommunityfactors);andevaluationcharacteristics(methodology, communicationoffindings,personalcharacteristics,andactionsoftheevaluator). Inacomprehensivereviewtoidentifyfacilitatorsofevaluationuse,Johnsonandhercolleagues (2009)examined41studiesofevaluationusefrom1986to2005,andclassifiedthemintothe followingthreecategories:(1)evaluationimplementation,(2)decisionorpolicysetting,and(3) stakeholderinvolvement.Table2illustrates22specificcharacteristicsofthesethreeoverarching categories. Afterexaminingtheliterature,Johnsonandcolleaguesfoundthatthemostcommonlyreferenced characteristicsofevaluationusewerethefollowing:stakeholderinvolvementwithcommitment orreceptivenesstoevaluation;communicationquality;andpersonalcharacteristicsofusers. Inadditiontoemphasizingtheimportanceofengagingstakeholdersinprogramevaluations,the researchersalsohighlightedtheimportanceofevaluatorcompetenceinfacilitatinguseof evaluations(Johnsonetal.,2009).

23 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

Table2:FacilitatorsofEvaluationUse EvaluationImplementation Decisionorpolicysetting Communicationquality Personalcharacteristics Timeliness Evaluatorcompetence Evaluationquality Findings Relevance Credibility Commitmentand/or receptivenesstoevaluation Politicalclimate Decisioncharacteristics Competinginformation Informationneedsorthe evaluationaudiences

StakeholderInvolvement Involvementwithcommitment orreceptivenesstoevaluation Involvementwith communicationquality Directstakeholderinvolvement Involvementwithcredibility Involvementwithfindings Involvementwithrelevance Involvementwithpersonal characteristics Involvementwithdecision characteristics Involvementofstakeholders facilitatedtheintroductionof theirinformationneeds

(Source:Johnsonetal.,2009)

RWJFEvaluationSeries

24

AppendixB FullDescriptionofMethodology ActiveforLifewasalargescaleevaluationcoveringmultiplesites,thousandsofprogram participantsandmultiplelayersofstakeholders.Thisbroadevaluationcanvasneededa methodologythatwouldallowaholisticexplorationofevaluationuse.Casestudymethodologyis selectedasthepreferredstrategywhenhoworwhyquestionsarebeingposed,whenthe investigatorhaslittlecontrolovereventsandwhenthefocusisonacontemporaryphenomenon withinsomereallifecontext(Yin,2003a,p1).Ourguidingresearchquestionsaskedhow evaluationwasused(ornot)andwhyusewasfacilitatedorhindered.Casestudyinvestigators hadnocontroloverevents.Begunin2003andendingin2007,theevaluationwassufficiently contemporarysincestakeholders,evaluationproducts,stakeholdersandlinkswerepossibleto findandremember.EvaluationofActiveforLifeprogramsatsomesitesandevaluation publicationscontinuetodaydespitecompletionofFoundationfunding. Havingsettledonthecasestudymethodology,itwasnecessarytopickamongseveralapproaches thatbestfitthisstudy(Merriam,1988;Stake,1995;Yin,2003a;Yin,2003b).Theseapproaches varybythetypesoffieldsinwhichtheyweredeveloped(e.g.,educationorpolicy),theoretical approach(e.g.,developingtheoryfromthegroundupversusstartingwithaconceptual framework)andmethodsproposed(e.g.,onlyqualitativeversusamixedmethodapproach).The approachselectedasmostappropriateforthiscasestudywasthatofYin(2003a)becauseofits multidisciplinaryapplications,thematchtothequestionsaskedandcontextgivenand investigatorfamiliaritywiththisapproach(Ottoson,Rivera,DeGroff,HackleyandClark,2007). Yin(2003a)identifiesfivecomponentsorlevelsofcasestudyresearchdesign:(1)studyquestions; (2)studypropositions(conceptualframework);(3)unitofanalysis;(4)logiclinkingdatatothe propositions(datacollection);and(5)criteriaforinterpretingfindings(analysis).Eachisreviewed belowfortheirapplicationtothiscasestudy.

25 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

StudyQuestions Theguidingstudyquestionsforthiscasestudywereasfollows: WastheActiveforLifeevaluationnotjustitsfindingsuseful? o Ifyes,whatweretheparametersofuse,e.g.,forwhom?when?how? o Ifnot,whynot? o Whatconstitutedevaluationuseincontext? HowcanfindingsfromthisandotherevaluationssponsoredbytheRobertWoodJohnson Foundationbemademoreuseful? WhatarethegeneralimplicationsofthisevaluationusecasestudyontheActiveforLife programforfoundations,leadersinthefieldsofaging,medicineandpublichealth,aswell asotherstakeholders?

Thesequestionswereusedastheentrancetotheliteraturereview,thebasisforthestudy protocol,andtheorganizationforthecaseanalysisandreporting.Thequestionswereapproved andsupportedintheirexplorationbytheFoundation. ConceptualFramework Havingcompletedaliteraturereviewonevaluationuse(BlakeandOttoson,2009),aswellas writtenandtheorizedaboutevaluationuseovertime(OttosonandHawe,2009),weagreedwith Yinabouttheimportanceofmakingtheinvestigatorsconceptualapproachtransparent.Recent literaturereviewsidentifiedseveralframeworksthathavebeenemployedtoclassifyand understandtypesofevaluationuse,aswellasfacilitatorsandbarrierstouse.Weinitiallysettledin thisstudyonthreeofthoseframeworkspreviouslydescribedintheliteraturereview.First,was theclassificationofevaluationuseasinstrumental,conceptual,orsymbolic(Weiss,1980;Kingand Pechman,1984;LevitonandHughes,1981).Second,wastheclassificationofevaluationprocess use(Patton,1997).Third,wastheclassificationofevaluationuseasdecisionmaking,education andtheprocessingofevaluationinformation(CousinsandLeithwood,1986).Workingwithinthe consistencyofeachframeworkandrecognizingthepotentialredundancyacrossframeworks,we foundnocompellingreasontochooseoneframeworkoveranotherintheinitialexplorationof evaluationuse.Wewantedasmanylensesaswereappropriatetothequestions 13 . Inexploringthefacilitatorsandbarrierstoevaluationuse,wesettledontheframeworkdeveloped byJohnsonetal.,2009.AsnotedintheliteraturereviewandinTable2,thisframeworkidentifies threebroadareasoffacilitators,i.e.,evaluationimplementation,decisionorpolicysettingand

13

We agreed with Johnson, et al, (2009, p379) on the impracticality of adding influence to this case and stayed with the concept of use.

RWJFEvaluationSeries

26

stakeholderinvolvement.Atotalof22subcategoriesoffacilitatorsfallwithinthesethreebroad categories. WhileweagreedwithYinscallforconceptualtransparency,wereservedtheanalyticaland experientialopportunitytochallengetheseframeworks.Ifthedatadidnotfitacategory, suggestedanotherone,ortookusinsomeotherdirection,wefollowedit. UnitofAnalysisandCaseStudySample TheunitofanalysisorcaseisuseoftheprocessandoutcomeevaluationsoftheActiveforLife programfundedbytheRobertWoodJohnsonFoundation.Therewereseveralotherevaluations occurringinthesamecontextandaroundthesametimeasthecasefocusedevaluation.These includedanAARPmarketingandpolicyinitiative;acomparisonoftheActiveChoicesandCommit toBeFitprograms;theprogramdevelopersevaluationofprogrammaterials;andthe introductionofREAIM 14 intotheActiveforLifeprogram.Whileitwaspossibletoseparatethe firstthreeevaluationeffortsfromthecasestudy,itwasnotpracticaltodothesamewithREAIM. AlthoughthisframeworkwasapartoftheActiveforLifeevaluationfromthebeginningasa summativetool,halfwaythroughtheevaluationitbegantobeusedasaformativetoolthataided planningandevaluation. Withinthecase,wesoughtasnowballsampleofpotentialusers,productsandtransmissionsof evaluationprocessesandfindings.Asnowballsampleisatypeofpurposivesamplinginwhich earlykeyinformantsidentifyotherswhomeetthesamecriteriaforinclusion,henceincreasingthe sizeofthesnowball(Rich,1977;LevitonandBoruch,1983;Merriam,2009).Startingwiththe Foundationsevaluationleadershipandprogrammanagement,theActiveforLifeNational ProgramOffice(NPO),andtheprocessandoutcomeevaluators,thesamplereachedoutto participatingorganizations,leadershipinthefieldsofagingandphysicalactivity,government agencies,frontlineprogrammanagers,communicationexpertsandprogramdevelopers.Wewere referredtoprogramparticipants,butunsuccessfulinreachingthem.Atotalof23stakeholders wereincludedinthefinalsample. Onenoteofexplanationneedstobemadeaboutthissample.Becauseofthepassageoftime, manyofthoseinterviewedwerereshuffledintodifferentroles,contextsandprogramsbythetime theinterviewswereconducted.Forexample,someoneformerlyinFoundationmanagementis currentlyinthefederalgovernmentandanotherFoundationemployeeisnowinprivate consulting;alocalprogramdirectorremainsinthesameorganization,butmovedtoanotherstate; andprogrammanagementhasplayedsubsequentconsultingrolesintheagingfield.Asaresult,a numberofstakeholdersbroughtmorethanoneperspectivetobearonuseoftheActiveforLife evaluation.

14

RE-AIM is not a theory or conceptual model, rather it is a framework and a set of criteria for evaluating interventions that are intended to eventually be broadly implemented or widely adopted. The acronym stands for reach, efficacy, adoption, implementation, and maintenance. (Please click here for additional information: http://re-aim.org/2003/FAQs_support.html )

27 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

DataCollection Datawereprimarilycollectedthrough20telephoneinterviewsusinganopenendedprotocol. (Twointerviewshadmorethanonerespondentonthecall).Theinterviewswereconducted betweenDecember2009andApril2010,eachlastingapproximatelyonehour.Bothinvestigators participatedinallinterviews. TheprotocolusedtoguidetheinterviewswasbasedonanextensivereviewofActiveforLife programmaterialsandpublications,aswellastheliteraturereview.Theprotocol,onthelastpage ofAppendixB,exploredissuesaroundevaluationuseinthecasestudy,aswellasidentificationof relevantevaluationproducts(e.g.,reportsorflyers),evaluationtransmissionstrategies,(e.g.,Web sitesorconferences)andotherinformantswemightincludeinthesnowballsample.Whilethe protocolwasoriginallydevelopedforprogramleadership,itturnedouttobeusefuland appropriateforalltypesofstakeholders,withonlyminorchanges.Theprotocolwassentto informantsbeforethescheduledinterviewsothattheywerefamiliarwiththequestions.Every informantdidnotanswereveryquestion. Twopointsneedtobemadeabouttheprotocol.Oneisthatwedidnotdefineevaluationuseto keyinformantsandtheotheristhatwedidnotpromptorsolicitanswersaboutanyparticular typeofevaluationuse.Sincetheliteraturedoesnotofferasingleunderstandingofevaluationuse, wedecidednottoimposesuchalimitationoninformants.Theywereabletoidentifyandexplain evaluationuseastheyunderstoodittofitwithintheircontextandthiscasestudy. Inthecourseofinterviews,additionalandrelatedquestionsweregenerated.Insomecasesthese werecallforwardquestionstobeaskedoffutureinformants.Insomecasesthesewerecall backquestionstobeclarifiedwiththosealreadyinterviewed.Mostoftenthesequestionswere aroundconfirminganunderstandingofevaluationuse. Allcallswerevoicerecordedwithpermissionoftheinformant.Bothoftheinvestigatorstook extensivenotes,withintentofcapturingrespondentsintheirownwordsasmuchaspossible. Withoutresourcestotranscribetheinterviews,weusedtheextensivedualnotestakenasthe datasource,withthevoicerecordingsasclarificationforanyuncertaintyordiscrepancybetween notes.Oneinvestigatortypedhernotesindepthshortlyaftereachcall,insomecasestranscribing extendedanswerstoquestions.Theotherinvestigatorsnotesandthevoicerecordingswereused asacheckonthemorecompletesetofnotes. Inadditiontointerviewnotes,wecollectedartifactsordocumentswherepossibletoconfirm examplesofevaluationuse.Theseincludedpublications,emailsaboutmeetingsandexchanges, andrecruitmentflyers.Somerespondentssentfollowupemailstotheinvestigatorstoaddother understandingsofevaluationuse,directedustoWebsites,oraddedfurtherclarificationtotheir interviewcomments.

RWJFEvaluationSeries

28

DataAnalysis Findingtypesofevaluationusewasthefirstdataanalysispriority.Whiletheconceptual frameworksalertedustokindsofevaluationuseandinformantsprovidedtheirselfidentified examplesofuse,thefollowingquestionwascontinuouslyexploredduringthedataanalysis process:Whatcountsasevaluationuseinthiscasestudy?First,anycategoryortypeofevaluation useidentifiedneededtobecorroboratedbymorethanonekeyinformanttobeincludedinthe case.(Thiswastrueforallcategoriesexcepttwowhichcouldbecorroboratedbyothermeans.) Second,theinformantneededtoprovideadescriptionoftheirexampleofevaluationusein context.DescriptionsthatwereabouttheActiveforLifeprogram,nottheevaluation,were eliminatedfromconsideration.Third,auseexamplewasconsideredstrongerifitwasidentifiedby stakeholdersinmorethanonerole,suchasanevaluatorandaprogrammanager.Fourth,use exampleswereconsideredstrongeriftheyweresupportedbyartifacts,suchasemail,Power Pointpresentations,orflyers.Lastly,ifinformantsidentifiedanotherkeyinformanttobackan assertionofevaluationuseandthesecondinformantfailedtodoso,theexamplewasdropped. Analysisbeganbyreviewinginterviewnotesacrosstypesofkeyinformants,suchasprogram managers,developersorevaluators.Theintentwastoallowtheunderstandingofevaluationuse tobeshapedsimultaneouslybyallstakeholders,notsequentiallybyonetypethenanother.The indepthinterviewnoteswerereadfirstandpotentialexamplesofevaluationuse,facilitatorsand barrierstouse,andrecommendationsaboutusewereidentifiedandputintoseparateExcel spreadsheets.Oneinvestigatorworkedprimarilyonthedatabasefocusedontypesofuseandthe otherworkedprimarilyonthefacilitatorsandbarrierstouse.Eachprimaryinvestigatorplayeda secondaryroleontheotherdataset.Whileworkingwiththedata,topofmindimpressionsand memoswerewrittenthroughouttheanalyticprocesstoclarifyunderstandings,shapethe meaningofcodes,andexplorelinks. Eachexampleofusewasgivenauniquecode,identifiedbysourceandcodedusingtheconceptual frameworkspreviouslyidentified.Manyexampleswereinitiallycodedinmorethanone framework,(e.g.,conceptualuseinoneframeworkanddecisionmakinginanother).Followingthe firstreview,thesecondsetofinterviewnoteswasanalyzed.Insomecases,newuseexamples wereidentifiedandotheruseexampleswereclarifiedwiththecodingadjustedaccordingly.If discrepanciesoccurred,thevoicerecordingswereconsulted.Thegrowingusedatabasewas developedbyoneoftheinvestigatorsandexchangedwiththeotherinvestigatorforcomment, critique,andchallengetotheexamplesofuseandthecodingsystem. Earlyon,issuesarosewiththecodingprocess.First,thedefinitionsofframeworkcodeswerenot obviousinthiscontext,e.g.,whatcountedasinstrumentaluseinthiscase?Second,somecodes seemedtoobroadtocapturediscreteuseexperiences,suchasconceptualuse,whileother seemedtoospecific,suchaseducation.Third,thecodesbegantofeellikeleakybucketswhen workingacrossframeworks.Forexample,oneframeworksdecisionmakingcodespilledinto anotherframeworksconceptualusecode.Wasoneofthesecodesmoreexplanatoryto understandingusethantheother?Fourth,wastheissueofwhatwecalledhigherordercoding. 29 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

Forexample,anidentifiedusemightbeaboutdecisionmaking,butarelatedexamplewouldshow thatthedecisionwasturnedintoaction,i.e.,someonedidsomething,notjustthinksomething. Theinvestigatorsgrappledwiththequestion:Howwouldthecodesaccountforthislink?Fifth, newcodeswereneededtocaptureuseexamplesthatdidntfitanyoftheexistingones.For example,datashoweduseinnewways,withadifferentaudience,beyondthecasetoother times,placesandcontexts.Sixth,somecodesstartedtobreakdownirreparably.Forexample, educationofdecisionmakersfailedasacodetocapturenoneducationactionwithawiderange ofstakeholderswhomakedecisionswithevaluationfindings,includingfutureprogram participants.Theseissuesaroundcodingoccurrednotjustwithframeworksontypesofuse,buta frameworkonfacilitatorsandbarrierstouse.Theframeworkandcodesweregettingintheway, notshowingaway. Adjustmentsweremadethroughouttheanalyticalprocess;however,athirdofthewaythrough codingtheinterviewnotes,investigatorsstoppedtoreviewthedatasets,thegrowingproblem withcodesandtheneedtoaddressallcasestudyquestions.Severalkeyanalyticaldecisionswere made.First,wedecidedtotakechargeofthecodes,ratherthanabandonthem.Wedefined, adapted,addedandrelabeledcodesasneededtomakesenseofevaluationuseinthiscasestudy. Despiteproblemswiththecodes,wefoundtheycouldbeahelpfullensandtheyprovideda connectionbetweenthiscasestudyandthebroaderevaluationuseliterature.Wedidnotseeka newlanguageofuse;wewantedaclearerunderstandingoflanguageincontext.Second,we codedtomorespecificmeaningsorhigherorderuse(action)whenachoicewasgiven.For example,ifanidentifiedusecouldbecodedasconceptualordecisionmaking,wecodedtoward themorespecificdecisionmaking.Ifhowever,decisionmakingwastiedtoaction,wecoded towardactiontaken.Third,weaddedtwonewcasestudycodes.Valuinguseemergedasa categorysincethetotalityoftheevaluationenabledvaluetobeplacedonthewholeActivefor Lifeprogram,aswellasitscomponents,itsprocesses,resultsandpurpose.Leveraged knowledgewasaddedtocapturethemovementofevaluationuseacrosstime,contextsand otherstakeholders.Fourth,thedecisionwasmadetousethisrevisedapproachwithallinterview notes,includingthoseforthcomingandalreadycompleted.Lastly,thedecisionwasmadetostop codingbarriersandfacilitatorstouseinto22differentandoverlappingcategoriesandinsteadtell thestoryoffacilitatorofuseasitemergedfromthedata.Forexample,communicationwasa leadingfacilitatormentionedbynearlyeveryinformant;nocodewasneededtoidentifyitassuch. Theanalysisofevaluationusetypescontinuedasaniterativeprocessofidentifying,questioning, checkingnotes,reclassifyingandclarifyingmeaningofcodes.Notonlydiduseexamplesbecome clearerintheprocess,butthreadsofevaluationuseemerged.Thesewereexamplesof connectionsamongtypesofuseacrosstime,contextsandcategoriesorcodes.Furthermore,the newcategoriesofvaluinguseandleveragedknowledgecrackedopenanewconceptual understandingofevaluationuseasecological.Thesenewunderstandingswereusedtotellthe storyofevaluationuseinthiscasestudyandorganizecasestudyimplications.

RWJFEvaluationSeries

30

StudyLimitations Thereareseverallimitationstothiscasestudy.First, wewereunabletotranscriberecordedinterviews, whichwouldhaveallowedagreateropportunityto workwithandreportmoreoftheinformantsown words.Second,wewereunabletopursuemorefield andcommunityinterviews.Whileexamplesofuse begantoreachredundancyamongsomestakeholders, ithadnotwithfieldandcommunityinformantswhen interviewswereterminatedtomeetreporting deadlines.Futureresearchpotentialexistswiththese stakeholders.Third,wewerenotabletopursuecall backquestionsduetolimitationsoftimeand resources.

InformantFeedback Thecasestudyfindingswerereviewedandcritiquedby therelevantRWJFpersonnelandsenttoall23 interviewedstakeholders.Suchkeyinformantswere askedspecificallyabouttheplausibilityoftheusetypes andfacilitatorsidentifiedandtheusefulnessofthe revisedconceptualframeworkproposed.Such membercheckingaddstotheveracityofcasestudyfindings(Merriam,1988).

ThisisanicereviewofAFL evaluationuseswelldocumented andcited,theoreticallybasedand supportedbythefindings. Anonymous IhavereviewedtheAFLCaseStudy andIhavenochanges. MelissaWatford,Ed.M. HealthEducatoratFirstHealth TherewereacoupleofspotswhereI havequestionsorveryminor corrections. DianeDowdy,Ph.D. NPODeputyDirector,Assistant ProfessoratTexasA&MHealth ScienceCenter CommunityHealthServices

31 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

ActiveforLifeEvaluationUseStudy:Tier1Questions 1. Whatwasyourrolein a. theActiveforLifeprogram b. theActiveforLifeevaluation 2. WhatwasthepurposeoftheActiveforLifeevaluation? 3. WhatweresomeofthekeyfindingsoftheActiveforLifeevaluationthatcouldmakea differenceifused? 4. WhatconsiderationwasgiventoActiveforLifeevaluationuse? a. Wasthereaspecificevaluationuseplan?Whendeveloped?Strategies? b. WhatwouldidealActiveforLifeuselooklike?WhatwouldprobableActiveforLife uselooklike? 5. WhowerethestakeholdersoftheActiveforLifeevaluation?(reviewpotentiallist) a. Stakeholderinvolvementintheevaluation,who?when?how? b. Whatwerefacilitators/barrierstostakeholderinvolvement? c. Ofthesestakeholders,whowereprimaryorsecondaryintendedusersofthe evaluation?Nonusers? 6. WhatweretheproductsoftheActiveforLifeevaluation?(reviewlist) a. Didtheseproductschangeovertime?Ifyes,how? 7. Whatstrategiesorprocesseswereusedtodisseminate,communicate,ordeliverActivefor Lifeevaluationproducts?(reviewlist) 8. Whichproductswereintendedforwhichusersoftheevaluation?(howtomatch?) 9. AreyouawareofhowfindingsfromActiveforLifemighthavebeenused? a. Ifyes,examplesofuse,e.g.who,when,what,how?Corroborationsource b. Notsure,whatsourcesmightweexploreaboutpotentialActiveforLifeuse 10. InadditiontoActiveforLifeoutcomes,areyouawareofanyexamplesofuseoftheActive forLifeevaluationprocess,e.g.who,when,what,how?Corroborationsource. 11. HaveyoupersonallyusedtheActiveforLifeevaluationfindingsorprocess?Exampleand corroboration 12. WhatevidencedoyouhaveaboutthefacilitatorsorbarrierstouseoftheActiveforLife evaluation? 13. HowcouldtheRobertWoodJohnsonFoundationbetterfacilitatetheuseofitsprogram evaluations? 14. Howcanthefieldof[AgingOrganizations,PublicHealthAudienceandMedicalCommunity] usetheinformationfromtheActiveforLifeevaluation? 15. WhoelsewouldyousuggestwetalkwithaboutuseoftheActiveforLifeevaluation? WhereelsemightwelookforevidenceofuseoftheActiveforLifeevaluation?

RWJFEvaluationSeries

32

References AlkinMCandTautSM.UnbundlingEvaluationUse.StudiesinEducationalEvaluation,29:112, 2003. BlakeSCandOttosonJM.KnowledgeUtilization:ImplicationsforEvaluation.InJ.M.Ottosonand PHawe,(eds).KnowledgeUtilization,Diffusion,Implementation,Transfer,andTranslation: ImplicationsforEvaluation,NewDirectionsforEvaluation,No124.JosseyBass:SanFrancisco,CA: 2134,Winter2009. CousinsJBandLeithwoodKA.CurrentEmpiricalResearchonEvaluationUtilization.Reviewof EducationalResearch,56(3):331364,1986. GreenLW,PotvinLandRichardL.EcologicalFoundationsofHealthPromotion.AmericanJournal ofHealthPromotion,10(4):270281,1996. HenryGTandMarkMM.BeyondUse:UnderstandingEvaluationsInfluenceonAttitudesand Actions.AmericanJournalofEvaluation,24(3):293314,2003. JohnsonK,GreensideLO,ToalSA,KingJA,LawrenzFandVolkovB.ResearchonEvaluationUse:A ReviewoftheEmpiricalLiteraturefrom1986to2005.AmericanJournalofEvaluation,30(3):377 410,2009. KingJAandPechmanEM.PinningtheWavetotheShore:ConceptualizingEvaluationUsein SchoolSystems.EducationalEvaluationandPolicyAnalysis,6(3):241253,1984. KirkhartKE.ReconceptualizingEvaluationUse:AnIntegratedTheoryofInfluence.New DirectionsforEvaluation,88:524,2000. LevitonLCandHughesEFX.ResearchontheUtilizationofEvaluations:AReviewandSynthesis. EvaluationReview,5(4):525548,1981. LevitonLCandBoruchRF.ContributionsofEvaluationtoEducationProgramsandPolicy. EvaluationReview,7:563598,1983. MerriamSB.CaseStudyResearchinEducation:AQualitativeApproach.SanFrancisco,CA:Jossey Bass,1988. MerriamSB.QualitativeResearch:AGuidetoDesignandImplementation.SanFrancisco,CA: JosseyBass,2009.

33 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

OttosonJM,RiveraMD,DeGroffA,HackleySandClarkC.OntheRoadtotheNationalObjectives: ACaseStudyofDiabetesPreventionandControlPrograms.JournalofPublicHealthManagement andPractice,13(3):287295,2007. OttosonJMandHaweP.KnowledgeUtilization,Diffusion,Implementation,Transfer,and Translation:ImplicationsforEvaluation,NewDirectionsforEvaluation,No124.SanFrancisco,CA: JosseyBass,2009. PattonMQ.UtilizationFocusedEvaluation.ThousandOaks,CA:Sage,1997. PressmanJLandWildavskyA.Implementation,3rded.Berkeley,CA.:UniversityofCaliforniaPress, 1984. RichRF.UsesofSocialScienceInformationbyFederalBureaucrats:KnowledgeforActionVersus KnowledgeforUnderstanding,inCHWeiss(ed.)UsingSocialResearchinPublicPolicyMaking. Lexington,MA:LexingtonBooks,1977. ShadishWR,CookTDandLevitonLD.FoundationsofProgramEvaluation:TheoriesofPractice. NewburyPark,CA:SagePublications,1991. StakeRE.TheArtofCaseStudyResearch.ThousandOaks,CA:Sage,1995. TautSMandAlkinMC.ProgramStaffPerceptionsofBarrierstoEvaluationImplementation. AmericanJournalofEvaluation,24(2):213226,2003. TheJointCommitteeonStandardsforEducationalEvaluation.TheProgramEvaluationStandards, 2nded.ThousandOaks,CA:Sage,1994. WeissCH.TheManyMeaningsofResearchUtilization.PublicAdministrationReview, September/October:426431,1979. WeissCH.KnowledgeCreepandDecisionAccretion.Knowledge:Creation,Diffusion,Utilization, 1(3):381404,1980. YinRK.CaseStudyResearch:DesignandMethods,3rded.ThousandOaks,CA: Sage,2003a. YinRK.ApplicationsofCaseStudyResearch,2nded.ThousandOaks,CA:Sage,2003b.

RWJFEvaluationSeries

34

Bibliography PublicationsofActiveChoicesandActiveLivingEveryDayEvidenceBasedTrials DunnAL,MarcusBH,KampertJB,GarciaME,KohlHWandBlairSNComparisonofLifestyleand StructuredInterventionstoIncreasesPhysicalActivityandCardiorespiratoryFitness:A RandomizedTrial.JAMA,281(4):327334,1999. KingAC,HaskellWL,TaylorCB,KraenerHCandDeBuskRF.GroupVersusHomeBasedExercise TraininginHealthyOlderMenandWomen:ACommunityBasedClinicalTrial.Journalofthe AmericanMedicalAssociation,266:15351542,1991. KingACandBrassingtonG.EnhancingPhysicalandPsychologicalFunctioninginOlderFamily Caregivers:TheRoleofRegularPhysicalActivity.AnnalsofBehavioralMedicine,19:91100,1997. KingAC,PruittLA,PhillipsW,OkaR,RodenburgAandHaskellWL.ComparativeEffectsofTwo PhysicalActivityProgramsonMeasuredandPerceivedPhysicalFunctioningandOtherHealth RelatedQualityofLifeOutcomesinOlderAdults.JournalofGerontology:MedicalSciences,55A: M74M83,2000. KingAC,BaumannK,OSullivanP,WilcoxSandCastroC.EffectsofModerateIntensityExercise onPhysiological,Behavioral,andEmotionalResponsestoFamilyCaregiving:ARandomized ControlledTrial.JournalofGerontology:MedicalSciences,57A:M26M36,2002. KingAC,FriedmanRH,MarcusB,CastroC,NapolitanoM,AhnDandBakerL.OngoingPhysical ActivityAdvicebyHumanVersusComputers:TheCommunityHealthAdvicebyTelephone(CHAT) Trial.HealthPsychology,26(6):718727,2007.

35 AnEcologicalUnderstandingofEvaluationUse:ACaseStudyoftheActiveforLifeEvaluation

PublicationswiththeResultsfromtheActiveforLifeProgram GriffinSF,WilcoxA,OryMG,LattimoreD,LevitonL,CastroC,CarpenterRAandRheaumeC. ResultsfromtheActiveforLife:ProcessEvaluation:ProgramDeliveryFidelityandAdaptations. HealthEducationResearch,25(2):325342,2010. WilcoxS,DowdaM,GriffinSF,RheaumeC,OryMG,LevitonL,KingAC,DunnA,BuchnerDM, BazzarreT,EstabrooksPA,CampbellVoytalK,BartlettPrescottJ,DowdyD,CastroCM,Carpenter, RA,DzewaltowskiDA,MockenhauptR.ResultsoftheFirstYearofActiveforLife:Translationof2 EvidenceBasedPhysicalActivityProgramsforOlderAdultsintoCommunitySettings.American JournalofPublicHealth,96(7):12011209,2006. WilcoxS,DowdaM,LevitonLC,BartlettPrescottJ,BazzarreT,CampbellVoytalK,CarpenterRA, CastroCM,DowdyD,DunnAL,GriffinSF,GuerraM,KingAC,OryMG,RheaumeC,TobnickJ, WegleyS.ActiveforLife:FinalResultsfromtheTranslationofTwoPhysicalActivityPrograms. AmericanJournalofPreventiveMedicine,35(4):340351,2008. WilcoxS,DowdaM,WegleySandOryMG.MaintenanceofChangeintheActiveforLife Initiative.AmericanJournalofPreventiveMedicine,37(6):501504,2009.

RWJFEvaluationSeries

36