• Embed Doc
  • Copy Link
  • Readcast
  • Collections
  • CommentGo Back
Download
 
Ill:r:l'liI_[dli:1:1[1_.1.1'(c1:1[.]:!:_'
(
:1
¥r
:_IIII;I
.....
III.MEASURINGCHANGEINTHEREALWORLD
:
LEARNINGFROM
ONGOINGANDPASTPROJECTS
HowRelatedFieldsUseEvaluationtoDocumentChangesinHealthBehaviors
WhatWe'veLearnedSoFar:TenObservationsfortheRealWorld
El
a
ineBr
a
#
cArkinH
ea
lthCommunic
a
tiononsult
a
nt
1.Itispossibleto
c
hangebehaviorandtomeasurethosechanges
.
However,italldependsonwhatkindofchange,withwhom,andhowoneintendst
o
intervene.Changingbehavi
o
risverycomplexandmeasuringthosechangesisverycomplicated.Very
o
ften,measuringchangetakesdecades.Whilenati
o
nwidedr
o
psinsm
o
kingprevalenceandstrokemortalityarebehavi
o
r-changesuccessst
o
ries,weha
v
ebeenworking
o
nsm
o
kingf
o
r30years,andhighbloodpressurefor25years.Behavi
o
rchangecannotbemeasuredinfiscalyears,anditisimportanttoresistpolicy
/
decisionmakerswh
o
willdeterminey
o
urprogram'ssuccessonayear-to-yearbasis.Whensmokingorhighbl
oo
dpressureratesareexaminedannually,therearefluctuati
o
ns.Onlybylo
o
kingacr
o
ssdecadesd
o
weseeadramaticdrop.Changingbeha
v
ior
--
andmeasuringthatchange
--
a
l
s
o
dependsupont
h
etypeofinterventi
o
nandthetype
o
fevaluation.Successfulinterventionsarebasedonmultiplethe
o
riesandmodels,andareveryc
o
mplicated,multidi-mensi
o
nalinterventi
o
nswithmanydifferentstrategies.Theirevaluationsalsoaremultidimensi
o
nalwithmanyevaluationmethodsandmeasures.Wecannotexpecttoseesimilarkindsofchangeswitha
o
ne-dimensionalpr
o
-gram
o
revaluationeffort.
2.Ifyoucanaffordanintervention,
th
enyoucanaffordtoevaluateit
.
Often,inthe"realworld,"wearehandedjusten
o
ughtimeandjustenoughmo
n
eytodothein
t
ervention.Makesureevaluati
o
nisbuiltint
o
thetimetableupfr
o
nt,andmakesuredecisi
o
n-makersandbudgetersunderstandwhatisrequiredt
o
dotheevaluati
o
naswellstheintervention.Noteveryevaluati
o
nhastobeelaborate.Often,therearequickandeasywayst
o
m
o
nit
o
rthepr
o
gram,sotheexcuse
o
fn
o
money
o
rtimetoevaluateisrarelyvalid.However,thereisaminimumbel
o
wwhichtheinterventionandtheevaluationwon'tw
o
rk.Insituationswherey
o
uaren
o
tsurey
o
uhavetheres
o
urcest
o
interveneorevaluate,thinkthr
o
ughwhetheryoucanmakeachangeandmeasurethatchangebeforec
o
mmittingres
o
urcestoit.Some-timesthesearequesti
o
nswedon'taskbecausewehavebeengivenanassignment.Aspr
o
fessi
o
nalsin
o
urfield,we
needtoaskthemmoreoften.
3
.
Evaluate
t
oachie
v
e
n
otjustmeasure
_
suc
c
ess.
Evaluationreallyishowsuccessisachieved,and
t
hatiswhatismos
t
importantin
t
herea
l
wor
l
d.Outcomeevaluati
o
nsareterrificforp
o
licymakersanddecisionmakers,andf
o
rpilotpr
o
gramsbeforereplication.Butifthepr
o
gramhasalimitedtimeframe,processevaluationiscriticalf
o
rmid-coursecorrecti
o
ns,becausetheremaynot
beachancetoreplicate.Thosechangesneedtobemadethefirsttimearound.
FEBRUARY2
8
,1997
4
2
 
.
,
re
.
t
:I,!-"K
I
"-III
_
[.
_
!1I
I
"-
I
=i1"[t]11;
t
-'
1
1
I
l
l
ie];!:lV
l
_
.
IIII
Y
.
_
.
_
I[I]
Pretestingandot
h
erformativeresearc
h
isvital,toensurethepr
o
gramisthebestitcanbeinitially.Itisespeciallyimp
o
rtantifthetimeframeisshortandmid-coursecorrectionsmayn
o
tbeablet
o
bemade.Butlackofpretestingcanals
o
causepr
o
blemsinlargeprogramswithveryelab
o
rateoutcomeevaluati
o
ns:flawsinthemateri-alsmayimpacttheoutc
o
meofthepr
o
gram,leadingevaluat
o
rst
o
c
o
ncludethatthepr
o
gramdidn'tw
o
rkwhenin
factthematerialsneededtoberefined.
4
.
Theevaluationdesignerandtheprogramdesignermustworktogetheronbo
t
haspectsoftheprogram.
Thesametheoriesandm
o
delsneedtobeusedf
o
rinterventi
o
nandevaluati
o
ndesign;b
o
thpartiesneedt
o
beworkingfromthesamepremise.Evaluat
o
rsneedt
o
understandwhatthepr
o
gramcand
o
,sotheycandesignappr
o
priatemeasures.With
o
utcl
o
secollab
o
rationwiththeprogramdesigner,theymaybemeasuringthingsthatareunrealistict
o
measureandmissingareaswheregreatprogressc
o
uldbesh
o
wn.Similarly,pr
o
gramdesignersneedtobuildinevaluationneedsfromthestart,t
o
ensuretheyaregettingtherightinformationintherightwayfromthefieldandfromt
h
econsumer.
5
.
Programsaredesignedtoeffectmorethanonetypeofchange.Evaluationsshouldmeasuremorethanone
ty
peofchange.
Asseveralspeakersnotedyesterday,theroleofourinterventionsisn'tsosimpleaschangingbehavior.
Rather,ourinterventi
o
nsmustchangeallofthevariables,orasmanyaswecan,thatwethinkinfluencethebehavi
o
r.Thenweh
o
pethatbehavi
o
rchangefoll
o
ws.Ifapr
o
gramistryingt
o
affectawh
o
leseriesofvariables,thenevaluationneedstomeasureintermediatefact
o
rsaswellasbehavi
o
rchange(i.e.,interpers
o
nal,environmen-
tal,accessibilityandavailabilitychanges).
6
.
Youdon'thavetorepl
i
c
at
ecostlyevalu
at
ionsifyoucanuseotherstudiesasproxymeasures
.
Noteveryprogramhastohavethem
o
stelegantevaluationdesign.Often,otherpeoplehavealreadydoneit,andtheirworkcanbeusedtoillustratethaty
o
urpr
o
gramcanhaveane
ff
ect.Oneexampleisthemeta-analysisdonebyIsobelC
o
ntentoandherteam.An
o
therisareviewshowingtheresultsofwelldesignedcampaigns.Sp
o
nsoredbytheNationalHeart,Lung,andBl
o
odInstitute,itl
o
okedathundredsofarticlesdiscussingpublichealth,media,
andcommunitycampaigns.
Onceyouhavereviewedotherstudies,c
o
ncentrateonevaluationmeasuresthatassurethequalityofyourinterventi
o
n--otherwiseitwon'thaveaneffect.
7.Th
e
elega
nc
e
o
fth
ee
valuati
o
nsho
u
l
dm
at
c
h
th
e
c
o
mpl
e
xity
o
fth
e
i
nt
e
r
ve
nti
o
n
.
Sometimest
h
eevaluationteamisstronger,inpowerorindollars,thantheinterventi
o
nteam.Iftheevalua-ti
o
nisn
o
tlinkedbacktowhattheinterventionrealisticallycanbeexpectedtod
o
,theinterventi
o
nwillbeshownupasafailurewithouteverhavingachance.Conversely,iftheinterventi
o
nisflawed,theevaluationdesigncannotoverc
o
methat
fl
aw.
8
.
The
d
esiredoutco
m
es--an
d
wha
tt
heevaluationmeasures--shoul
d
bebase
do
nrealisticexpectationsfor
thelengthandcomplexityoftheintervention.
FE
II
R
UA
R
Y
2
8
,f
l
)
l
l
?
43
 
Boththeinterventionandtheeval
u
ati
o
nneedtoberealisticbasedup
o
n
t
hestatu
s
ofthetargetaudience.F
o
rexample,at
o
nep
o
intwhentheNationalHeart,Lung,andBl
oo
dInstitutewasw
o
rking
o
nhighbl
oo
dpressureandch
o
lester
o
leducati
o
n,theyhadtw
o
verydifferenttargetaudiences.F
o
rserumch
o
lester
o
l,theAmericanpublicdidn
o
tknowwhatitwas.NHLBIworkedwithprec
o
ntemplators,tryingt
o
introducethemt
o
theissueandgettheminterestedinit.F
o
rhighbloodpressure,theaudiencewaspeoplewh
o
havehighbl
oo
dpressureandhavebeen
o
nmedicati
o
nf
o
ryears.Thesewerepe
o
pleinthemaintenancestagewhofounditveryhardt
o
c
o
mplywithtreatment
o
vertherestoftheirlives.Thepr
o
gramdesignerswerebeingveryrealisticab
o
utthekindsofchangestheywantedtoseeforthecholester
o
lprogramversusthehighbl
o
odpressurepr
o
gram.Als
o
,weneedtoredefinewhatarealistic
o
utc
o
meis.Inpublichealth,wew
o
rkwiththehardestaudiencest
o
reach,thenexpectdramaticchanges.Ifwemakesmallchanges,wefeelwehavefailed.C
o
ntrast
o
urappr
o
achtothat
o
ftheprivatesector.AlanAndreasensometimesusesChevroletasananal
o
gy
-
-thatthereisn
o
tonekind
o
fChevy,therearemany
o
fthemf
o
rdi
ff
erentaudiences.Considerthatmaybe3percent
o
fAmericanswantt
o
buyaChevy.Thensplitthat3percentintothedifferentChevym
o
dels,andl
oo
katthepercentagewh
o
willactuallybuy--perhaps1
0
percent.Thenumbersaregettingverysmall,butthemanufacturerstillsh
o
wsapr
o
fit.Whatdowedoinpublichealth?Wetakethehardestmarkettoreach,n
o
ttheeasiest.Wearealwaysw
o
rkingwithpe
o
plewh
o
areathighrisk,wh
o
haveallkindsofbarrierst
o
behaviorchange.Thenwelookf
o
radeclineby5
0
percentinthestrokerate.OrwewanttogetallAmericanst
o
quitsm
o
king.Ifwemakea5percentchange,wethinkwehavefailedandoftenwelosebudget.Weneedt
o
stepbackandthinkab
o
utwhatisrealistict
o
acc
o
mplish.
9.Evaluations,likeinterventions,needtobedesignedwithapurposeandatargetaudienceinmind
.
Marketsegmentationappliestoevaluationsaswellasinterventi
o
ns
.
Ifpolicy
/
decisi
o
nmakersaretheaudi-ence,theywantt
o
knowwhatkind
o
fchange
o
ccurredbecause
o
fyourefforts.Ify
o
uwantt
o
gobacktoaprogramtargetaudience,theymaybem
o
stinterestedinwhethertheinterventionworksforpeoplelikethem.Implementa-tionmanagerswanttoknowwhatisworking,sotheyknowwhattheyneedt
o
fix.
10
.
Somecomponentsofevaluationdesignaretransferablefromprogramtoprogram;methodsandinstrumentsshoul
d
bemorewi
d
elyshare
d
.
Itismuchm
o
recomm
o
ntoseecross-sharingofimplementationstrategies,materials,andtrainingratherthanofevaluationdesignsandinstruments.Therearemanyless
o
nst
o
belearnedfr
o
mc
o
nductingevaluationsthatshouldbeshared.Whenstartingtoevaluatesomething,remembert
o
callapeerandseeiftheyhaveanythingin
theirfilesthatwillhelpyougetstarted.
F
E
BI
fi
JA
J
_
Y28
,
1997
of 00

Leave a Comment

You must be to leave a comment.
Submit
Characters: ...
You must be to leave a comment.
Submit
Characters: ...