American Association for Public Opinion Research

Nonresponse, Sample Size, and the Allocation of Resources

Author(s): Ivor Wayne
Source: The Public Opinion Quarterly, Vol. 39, No. 4 (Winter, 1975-1976), pp. 557-562
Published by: Oxford University Press on behalf of the American Association for Public
Opinion Research
Stable URL: http://www.jstor.org/stable/2748509
Accessed: 16-07-2016 21:45 UTC

REFERENCES
Linked references are available on JSTOR for this article:
http://www.jstor.org/stable/2748509?seq=1&cid=pdf-reference#references_tab_contents
You may need to log in to JSTOR to access the linked references.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
http://about.jstor.org/terms

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted
digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about
JSTOR, please contact support@jstor.org.

Oxford University Press, American Association for Public Opinion Research are
collaborating with JSTOR to digitize, preserve and extend access to The Public Opinion Quarterly

This content downloaded from 152.2.176.242 on Sat, 16 Jul 2016 21:45:28 UTC
All use subject to http://about.jstor.org/terms
 NONRESPONSE, SAMPLE SIZE, AND
THE ALLOCATION OF RESOURCES

BY IVOR WAYNE*

Nonresponse is here to stay-every survey researcher knows that. We

are forced to take this grim phenomenon into consideration in making
decisions on design, data collection, and analysis. Carol H. Fuller' has
made some of these decisions easier for us by showing how maximal
nonresponse bias is related (in probability samples) to observed
proportion, to sample size, and to response rate. Thus she has performed
a public service.
This note is merely a follow-up on her approach, an effort to further
increase its practical utility in the decision-making process. She illustrates
the three relationships noted above by presenting tables based on Coch-
ran's equations2 for the upper and lower limits of the 95 percent
confidence interval around the observed sample proportion.
In the original version, Cochran's equations read as follows:

Pu= W1 (Pi+2 p-q1/Nj)+ W2(1)

PL= W1(p1 - 2 P1q1/N1 + W2(O) (1)
W, = response rate

W2= nonresponse rate

We eliminate W2 by substituting (1 - Wi) for it, and simplify

graphically by dropping the subscript 1. Hence:

p= Wp + 2 W pq/N+ 1-W (2)

PL= Wp-2W 8pq/N

Since its is the width of the confidence interval, Kj5, that serves as a
measure of precision or "goodness of measurement," we subtract PL
from Pu and get:

Kg5 4 + 1-W (3)

* The author is Adjunct Professor at the National Graduate University, Washington,

D.C., 20016.
' Carol H. Fuller, "Weighting to Adjust for Survey Nonresponse," Public Opinion
Quarterly, Vol. 38, 1974, pp. 239-246.
2 William G. Cochran, Sampling Techniques, 2nd ed., New York, John Wiley, 1963, p.
357.

This content downloaded from 152.2.176.242 on Sat, 16 Jul 2016 21:45:28 UTC
All use subject to http://about.jstor.org/terms
 558 IVOR WAYNE

(Those who would rather use K99 should simply substitute 6 for 4 in this
equation.)
In designing his survey, the study director makes a number of
assumptions upon which he bases his decisions for action. He may
assume that the proportion of "yes" responses may approach a fifty/fifty
split (p = .5); he wishes to stay within 95 percent confidence limits of ?
.15 (Kg5 = .3), and plans to get responses from 900 individuals (N = 900).
What response rate must he reach? Or, obversely, how much

W-- VN(4)
nonresponse can he tolerate? In order to obtain answers to these run-of-
the-mill questions, we give equation (3) a more convenient format by
means of some simple algebra.

1= - K95 4

By substituting the above values, we get:

1- .3 .7 .7
W= = 7 2 = 7 = .75
41~. 2-5 1 __2 .933

To stay within the desired confidence interval, the study director must
attain a response rate of 75 percent; that leaves 25 percent (N2 = 300) for
nonresponse; he must obtain the 900 respondents out of 1,200 persons
selected and approached. This is the maximum tolerable nonresponse,
given the values in the above example.
It is clear that for fixed values of p and Kg5 (or Kgg) simple tables
showing the relationship between W and N can be worked out, using
equation (4) as the basis. Two such tables (with K95 = .2, .3, .4) are
presented here for the convenience of the reader: Table 1, with p = .5, is
applicable over a large middle portion of the p distribution when N is
large. (In the example where W = .75, its value would have Xbeen W =
.743, hadp = .25 been the observed sample value.) Table 2 is based upon
p = .2.-
What do the tables tell us? First, as the p value, observed for the
respondents, deviates from the .5 mark, the minimum response rate
drops very slightly when N is of good size. Thinking back to our example
where N was equal to 900, we see that Table 2 (for p = .2) prescribes a
response rate that is merely 1 percent to 2 percent smaller than that
shown for p = .5. Even a 10/90 break would decrease the required re-
sponse rate by only one percentage point or less. Since surveys are rarely
conducted with the expectation of such extreme distributions, the study
director cannot defend himself against the biasing effect of the
nonresponse rate by positing an extreme p value in his sample.
The second message from the tables spells equally bad news for the

This content downloaded from 152.2.176.242 on Sat, 16 Jul 2016 21:45:28 UTC
All use subject to http://about.jstor.org/terms
 ALLOCATION OF RESOURCES 559

TABLE 1
MINIMUM RESPONSE RATES REQUIRED FOR DESIRED
95 PERCENT CONFIDENCE INTERVAL ABOUT P = a

K5= .2 K95g .3 K95=.4

NC Wd N/ We W N/W W N/W

100 1.00 100 .88 114 .75 133

200 .93 215 .82 245 .70 286
300 .90 332 .79 379 .68 442
400 .89 450 .77 519 .67 600
600 .87 689 .76 788 .65 918
900 .86 1,046 .75 1,200 .64 1,406
1,600 .84 1,905 .74 2,162 .63 2,540
2,500 .83 3,012 .73 3,425 .62 4,000
10,000 .82 12,195 .71 14,084 .61 16,393

a p = value of proportion for respondents.

b K95 = width of 95 percent confidence interval.
C N = number of respondents.
d W = response rate.
e N/ W = total sample (i.e., number of respondents + nonrespondents).

survey researcher intent on good (precise, reliable) measurement, that is,

a narrow confidence interval around the observed p value: Once the
planned sample is large, say equal to 1,200, increasing the size of the
sample even dramatically does not alleviate nonresponse bias.
A look at the middle column of Table 1 should convince the reader
that this is so: a jump from N = 900 to N = 10,000 increases the
permissible nonresponse rate merely from 25 percent to 29 percent, with
equally good measurement. Moreover, equation (4) tells us the limit of
nonresponse tolerance for a given confidence interval. For N = c, the
equation becomes

K= 1- W (5)

TABLE 2
MINIMUM RESPONSE RATES REQUIRED FOR DESIRED
95 PERCENT CONFIDENCE INTERVAL ABOUT P = .2a

K95 =.2 K95 =.3 K95 4

W N/W W N/W W N/W

100 .95 105 .83 120 .71 140

200 .90 222 .79 253 .68 296
300 .88 340 .77 389 .66 454
400 .87 460 .76 526 .65 613
600 .86 701 .75 801 .64 935
900 .84 1,065 .74 1,216 .63 1,420
1,600 .83 1,928 .73 2,192 .63 2,560
2,500 .83 3,026 .72 3,458 .62 4,032
10,000 .81 12,300 .71 14,084 .61 16,393

a For meaning of symbols, see Table 1.

This content downloaded from 152.2.176.242 on Sat, 16 Jul 2016 21:45:28 UTC
All use subject to http://about.jstor.org/terms
 560 IVOR WAYNE

In other words, the nonresponse rate cannot exceed the width of the
confidence interval.
How can the survey researcher defend himself against the evils of
nonresponse? One solution is familiar-the nonrespondent sample sur-
vey to ascertain how nonrespondents differ from respondents, and thus
estimate how the observed values should be adjusted, given the
established bias.3
The second approach is less obvious; it is based on Cochran's
statement quoted by Fuller, "It is evidently worthwhile to devote a
substantial proportion of the resources to the reduction of non-
response."4 That this is easier to do when the study sample is small
can be demonstrated by referring to our initial example.
The study director planned a sixteen-page questionnaire survey with a
sample of 1,200, an eventual N = 900, i.e., a response rate of 75 percent.
He anticipates three rounds of data collection with the following re-
sponse rates and costs:
first round, response rate a1 = 40 percent, cost per respondent m1, -
$2.50
second round, a2 = 20 percent, m2 = $3.25
third round, a3 = 15 percent, m3 = $4.00
The total cost for N = 900 is $2,700.
As an alternative, he considers a sample of 788, with an eventual N =
600, i.e., response rate of 76 percent (the row just above in Table 1, col-
umn Kj5 = .3). By applying the same a,, mi, assumed in the preceding
plan, he finds that he can collect 591 questionnaires at a cost of $1,774.
Thus he has available $926 for the missing nine cases; he is free to spend
over $100 per case-enough for personal interviews. The argument in
favor of the smaller sample becomes even more compelling, if the study
director is less confident about reaching the 75 percent response rate by
relatively cheap means.
Even closer attention needs to be paid to the allocation of resources
between expensive and cheap ways of data collection when a stratified
sampling frame is contemplated where inferences are to be drawn from
small cell frequencies. Here the mere loss of a couple of cases may have so
powerful a biasing effect that the survey researcher may arrive at
generalizations so imprecise as to be practically meaningless. Consider
the practical significance of the statement: yes = 50 percent, ? 27.8
percent for the 95 percent confidence interval. In other words, in 19 out
of 20 instances, the value would lie between 22.2 percent and 77.8
percent! Such a finding would be derived from a cell where twenty re-
spondents coexisted with five nonrespondents.
In order to make the study director's planning for adequate precision
3 This is well discussed in Fuller's article, op. cit., p. 245.
4 Cochran, op. cit., p. 358.

This content downloaded from 152.2.176.242 on Sat, 16 Jul 2016 21:45:28 UTC
All use subject to http://about.jstor.org/terms
 ALLOCATION OF RESOURCES 561

and for efficient allocation of resources easier, we present, in Table 3,

information relevant to the sizes of subsamples apt to occur in stratified
sampling.
The table shows at a glance that, with perfect response, the sample size
cannot drop below 25 for Kj5 = .4, nor below 45 for K&5 = .3. When the
study design prescribes that generalizations with narrow confidence inter-
vals be made on proportionally small substrata, the initial sampling plan
must be modified to allow for oversampling of such substrata. This is, of
course, daily practice in survey research. But how much oversampling?
Table 3 provides the clues; for a K,5 = .3 (p = .5) the study director has
his choice among

S = 45 Nonrespondents = 0
S=50 NR = 1
S= 75 NR = 5
S= 100 NR = 11

Here again, the study director will arrive at the best cost-effective
solution on the basis of anticipated response rates (a,) and costs per

TABLE 3
EFFECT OF NONRESPONSE ON 95 PERCENT CONFIDENCE INTERVAL
IN SMALL SAMPLES (p = .5)a

K5. for
Number of Nonrespondents
Size of Increase in K95
Sample 0 1 10 per Nonrespondent
25 .400 .432 .710 .031
30 .366 .393 .632 .027
40 .316 .337 .524 .021
45 .299 .318 .486 .019
50 .283 .300 .453 .017
75 .231 .243 .348 .012
100 .200 .209 .290 .009

a For ease of calculation, underlying equation (3) has been adapted to read:
K95 = 1 - l/S(N - 2,/7V)
where S = size of sample
N = number of respondents

NOTE: With small samples, more precise confidence limits for proportions may be
obtained by using quadratic rather than linear equations. For instance, the quadratic
formulae presented by Wallis and Roberts (W. Allen Wallis and Harry V. Roberts,
Statistics: A New Approach, New York, Free Press, 1956, p. 467), when modified according
to Cochran, produce the following equation:

K195
I N -=1
( + 12 N
1+N(N2
N +4+ 4N-1 )
However, application of this equation would not materially change the figures in this table
nor affect the argument in the text.

This content downloaded from 152.2.176.242 on Sat, 16 Jul 2016 21:45:28 UTC
All use subject to http://about.jstor.org/terms
 562 IVOR WAYNE

respondent (mi) in the several phases of data collection. Once he has

estimated these values, the calculation is straightforward.

Cost in $ = S E aimi (6)

N= SZai
where S = size of sample
N = number of respondents

It must be pointed out that in the allocation of resources we have

limited our consideration to operational dollar costs arising from data
collection. Expenses linked to processing, to effects of the time element
(e.g., extra effort to meet deadline), to coordination of data from
different types of data gathering techniques-all those have been dis-
regarded, for the sake of simplicity of presentation.

This content downloaded from 152.2.176.242 on Sat, 16 Jul 2016 21:45:28 UTC
All use subject to http://about.jstor.org/terms