You are on page 1of 3

Issue Brief

TranslatingResearchtoPolicy

February2008/Issue13

AreLowerResponseRatesHazardoustoYourHealthSurvey?

Introduction
Surveysarewidelyusedinpublichealthforsurveillance,
evaluationandmonitoringofimportantpublichealth
policyissues.Themostwidelyusedmeasureofthe
qualityofsuchsurveysisresponserate.1Aworking
assumptionhasbeenthatforasurveytobeconstruedas
verygood,itmustattainahighresponserate,and
responseratesareoftenreportedtosummarizethe
qualityofthedataset.Thehighresponseratewas
seenasaproxyfortheamountofpotentialbiasdueto
thenonrespondentsbeingdifferentthanthe
respondents.Recentresearch,however,hasshownthat
responseratesareonlyweaklyassociatedwithresponse
bias,2andpublicopinionsurveyswithmuchlower
responseratescanyieldsimilarestimatestothosewith
higherresponserates.3

ALookatResponseRates
Responseratestomosttelephonesurveysarerarely
higherthan60percent4andhavebeendecreasingover
thecourseofthepastseveralyears.Forexample,recent
responseratesfortheBehaviorRiskFactorSurveillance
System(BRFSS)survey,atelephonesurveyconductedby

states,hoveraroundthe50percentrange.Ratesfor
generalpopulationtelephonesurveysinthelate1980s
weretypicallyinthevicinityof70percent.5

Inordertosecurethehighestpossibleresponserates,the
standardprocedureinmanytelephonesurveysisto
makemultiplecallattempts(e.g.,upto50calls)andto
convertpeoplewhoinitiallyrefusetoeventually
participateinthesurvey.6RecentworkbySHADAC
examinedwhetherrefusalconversionandmakingmany
callstothesamenumberalterhealthsurveyestimates.
Weuseddatafromthreelargegeneralpopulation
telephonesurveysofresidentsofMinnesotaand
Oklahomafrom20032005conductedbytheUniversity
ofMinnesotasSchoolofPublicHealth.

Figure1showstheresponseratesforthetotalsample
comparedtotheratehadinitialrefusersbeenexcluded
andtherateforthoserequiringfivetoeightandnineor
morecontactstocompletethesurvey.Resultsshowthat
extracontacteffortclearlyleadstoahigherresponserate.
Thequestion,however,istowhatextenttheefforttaken
toachievetheseresponsesimprovesdataquality.

Figure 1: Response rates for three surveys, total sample versus the rate had refusers been excluded, the rate for
completes requiring 5 to 8 contacts, and the rate for completes requiring 9 or more contacts
80%
Total sample

59%
60%

54%

51%

48%

45%
39%

36%

40%

47%
41%

37%
30%

26%
20%

Excluding at least one


refusal
Excluding 5 to 8
contacts
Excluding 9 or more
contacts

0%
Oklahoma Health Access
Survey

Minnesota Health Access


Survey

Minnesota Treatment
Needs Assessment Survey

Source: 2004 Minnesota Health Access Survey, 2004 Oklahoma Health Care Insurance and Access Survey, 2004/2005 Minnesota Treatment Needs Assessment
Survey; Note: All responses are not weighted: n= 13,802 (MN Health Access), 5,847 (OK Health Insurance), 16,891 (MN Treatment Needs Assessment). Response
rates are AAPOR RR4.

State Health Access Data Assistance Center (SHADAC) | University of Minnesota School of Public Health
612-624-4802 | fax: 612-624-1493 | www.shadac.org

DifferencesTypesofResponders
Davernandcolleaguesexamineddifferencesin
demographicsandhealthmeasuresbywhethera
respondenttookfiveormoredaystocompletethe
survey(atypicalfieldperiodforapublicopinionpoll)
orthehouseholdinitiallyrefusedtoparticipate.7

Theanalysis,showninTable1,revealedsome
differencesbetweenearlyandreluctantsurvey
respondersbydemographiccharacteristics(e.g.,people
over65tendtobeearlyrespondentsandalsotendtobe
initialrefusers).However,therewerenotdifferences
withrespecttohealthcharacteristicssuchasreported
healthinsurancecoverage,accesstohealthcare,and
generalhealthstatusaftercontrollingfordemographic
characteristicscommonlyusedtomakeweighting
adjustmentstosurveydata(age,sex,geography,race
andethnicity).Thesefindingsaresimilartothefindings
frompublicopinionsurveys.8

Inoneofthethreesurveysasignificantdifferencewas
foundinthetypeofreportedhealthinsurancecoverage
andselfreporteddruguseafterintroducingcontrol
variablesthatarecommonlyusedtomakeweighting
adjustments.Thissurveysmethodsdifferedin
importantwaysfromthoseoftheothertwosurveys
examined.Mostnotably,proxyresponseswerenot
allowedandthesurveydealtwithsensitivemental
healthanddrugabuseissues.Initialrefusersweremore
likelytobecurrentsmokersandtohaveneverused
illicitdrugsorabusedprescriptiondrugs.Thosewho
tookfivetoeightdaystobecomearespondentwere
morelikelytohaveprivateinsurancecoverage,binge
drinkwithinthepastyear,andhaveanalcoholdisorder.

Table 1: Health insurance status and outcomes of interest of respondents with an initial refusal, and by number of call attempts
Oklahoma Access Survey
At least
one
refusal

Minnesota Access Survey

9 or
More
Attempts

5-8
Attempts
**

27.7%

**

At least
one
refusal

5-8
Attempts

9 or
More
Attempts

Insurance any public

45.1%

31.1%

37.3%

27.9%

Insurance any private

68.8%

66.4%

67.4%

76.5%

76.1%

74.7%

Uninsured

13.1%

17.7%

16.8%

8.0%

7.5%

9.1%

Health Status
(excellent, very good,
good)

79.4%

82.8%

86.8%

85.7%

88.9%

**

Has a usual source of


care

86.6%

85.4%

83.4%

90.8%

87.7%

**

**

Minnesota Treatment Needs Survey

**

23.5%

At least
one
refusal
**

9 or
More
Attempts

5-8
Attempts

38.0%

23.9%

76.7%

78.2%

**

23.0%

**

7.0%

8.0%

8.6%

**

87.9%

**

79.1%

83.2%

**

84.0%

**

86.3%

**

n/a

n/a

77.7%

n/a

Source: 2004 Minnesota Health Access Survey, 2004 Oklahoma Health Care Insurance and Access Survey, 2004/2005 Minnesota Treatment Needs Assessment Survey
* p<0.01, ** p<0.001 indicates a significant difference between "at least one refusal" and "no refusal", a significant difference between "1-4" attempts and "5-8 attempts", or
between "1-4" attempts and "9 or more attempts." Note: All responses are not weighted: n= 13,802 (MN Health Access), 5,847 (OK Health Insurance), 16,891 (MN Treatment
Needs Assessment).

TheValueofIncreasedResponseRate
Theresultssuggestthatresearchersshouldcarefully
examineunderwhatcircumstancesadditionalsurvey
resourcesshouldbeexpendedtowardsachievinghigher
responserates.Makingmanycallstoonenumberand
attemptingrefusalconversionincreasesthecostof
fieldingsurveysandalsocausesmorerespondent(or
nonrespondent)burden.Theseincreasedcostsreduce
thetotalnumberofcompletedsurveysonecouldobtain
withinagivensurveybudget.9Asaresult,asurveys
statisticalpowerwillbelower.Thatis,theeffort
investedinmakingmanymoreadditionalattemptsto
reachanumbercouldhavebeenputtowardsreachinga
newnumberthathasahigherprobabilityofresponseon

thenextattemptthanonethathasbeencalledmany
timesalready.10

Effortstoincreaseresponseratescanalsocreatemore
respondentburden.Itisplausiblethatrepeatedcall
attemptscreateasituationinwhichpeoplehaveto
refusetwice.Excessivecallstoahouseholdthatscreens
telephonecallsmayalsocreateafairamountof
respondentannoyanceandburden(evenforthosewho
mayneveranswerthephone).Inthelongterm,this
burdenmayheightenthedownwardtrendofresponse
ratesthattheseeffortsweredesignedtoforestallor
eliminatealtogether.

Conclusions
Severalrecentstudiessuggestthattherespondentswe
workhardesttoobtainresponsesfrommaybe
somewhatdifferentsociodemographically,butquite
similarintheirsubstantiveresponses,totheirmore
accessibleandreceptivecounterparts.11Asurveywitha
90%responseratecanhavethesameamountofoverall
responsebiasasonewitha30%responserate.

Theextentofbiasrestsonthedifferencesbetweenthe
respondingsampleandthebalanceoftheframe.The
fieldneedstomoveawayfromanoverrelianceon
responseratesasindicatorsofresponsequality.Health
careresearchersrelyingontelephonesurveymethodsin
theirinvestigationsshouldundertakemorenonresponse
biasanalysesasamatterofcourse,insteadofsimply
relyingontheresponserateasaqualitymeasure.

Notes
1

Atrostic,B.K.,Bates,N.,Burt,G.,&Silberstein,A.(2001).NonresponseinU.S.governmenthouseholdsurveys:Consistentmeasures,
recenttrends,andnewinsights.JournalofOfficialStatistics,117,209226;Biemer,P.,&Lyberg,L.(2003).IntroductiontoSurvey
Quality.NewYork:Wiley.

Blumberg,S.,Davis,K.,Khare,M.,&Martinez,M.(2005).Theeffectofsurveyfollowuponnonresponsebias:JointCanada/United
Statessurveyofhealth,200203.PaperPresentedattheAnnualMeetingoftheAmericanAssociationforPublicOpinionResearch,Miami
FL;Groves,R.M.(2006).Nonresponseratesandnonresponsebiasinhouseholdsurveys.PublicOpinionQuarterly,70(4),646675.
Keeter,S.,Kennedy,C.,Dimock,M.,Best,J.,&Craighill,P.(2006).Gaugingtheimpactofgrowingnonresponseonestimatesfroma
nationalRDDtelephonesurvey.PublicOpinionQuarterly,70(4),125148;Keeter,S.,Kohut,A.,Miller,A.,Groves,R.,&Presser,S.
(2000).Consequencesofreducingnonresponseinalargenationaltelephonesurvey.PublicOpinionQuarterly,64(2),12548.

Brehm,J.(1993).ThePhantomRespondents:OpinionSurveysandPoliticalRepresentation.AnnArbor,MI:UniversityofMichiganPress;
CentersforDiseaseControlandPrevention.(2006).2005BehavioralRiskFactorSurveillanceSystemDataQualityReportHandbook.
AtlantaGA:CentersforDiseaseControlandPrevention.

Groves,R.M.,Fowler,F.J.,Couper,M.P.,Lepkowski,J.M.,Singer,E.,&Tourangeau,R.(2004).Surveymethodology.NewYork:
Wiley;Lavrakas,P.J.(1993).TelephoneSurveyMethods:Sampling,Selection,andSupervision.ThousandOaks,CA:SagePublications.

Frey,J.H.(1983).SurveyResearchbyTelephone.BeverlyHills,CA:SagePublications;Groves,R.M.,&Lyberg,L.E.(2001).An
overviewofnonresponseissuesintelephonesurveys.InR.M.Groves,P.P.Biemer,L.E.Lyberg,J.T.Massey,W.L.Nicholls&J.
Waksberg(Eds.),Telephonesurveymethodology(pp.191212).NewYork:Wiley.

Davern,M.,Call,K.T.,Ziegenfuss,J.,McAlpine,D.,&Beebe,T.J.(2007).Arelowresponserateshazardoustoyourhealth?Paper
presentedattheannualmeetingoftheAmericanAssociationofPublicOpinionResearch.May21,2006MontrealCanada.(This
paperisalsocurrentlyunderreviewatapeerreviewedjournal).
Keeteretal.(2006);Keeteretal.(2000).

Allison,K.R.,&Yoshida,K.K.(1989).Increasingresponseratesincommunityhealthsurveysadministeredbytelephone.Canadian
JournalofPublicHealth,80,6770;Groves,R.M.(Ed.).(1989).SurveyErrorsandSurveyCosts.NewYork:Wiley.

Groves(1989);Triplett,T.(2002).Whatisgainedfromadditionalcallattemptsandrefusalconversionandwhatarethecost
implications?Report.WashingtonDC:UrbanInstitute.

10

Holle,R.,Hochadel,M.,Reitmeir,P.,Meisinger,C.,&Wichman,H.E.(2006).Prolongedrecruitmenteffortsinhealthsurveys.
Epidemiology,17(6),639643;Keeteretal.(2006);Keeteretal.(2000).

11