Series/Number 07–071
Eun Sul Lee
Division of Biostatistics, School of Public Health, University of Texas Health Science Center—Houston
Ronald N. Forthofer
SAGE PUBLICATIONS
International Educational and Professional Publisher
Thousand Oaks London New Delhi
Copyright 2006 by Sage Publications, Inc.
All rights reserved. No part of this book may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.
CONTENTS
Series Editor’s Introduction 
v 
Acknowledgments 
vii 

1 

3 
Types of Sampling 
4 
The Nature of Survey Data 
7 
A Different View of Survey Data ^{*} 
9 

11 
Adjusting for Differential Representation: The Weight 
11 
Developing the Weight by Poststratiﬁcation 
14 
Adjusting the Weight in a FollowUp Survey 
17 
Assessing the Loss or Gain in Precision: The Design Effect 
18 
The Use of Sample Weights for Survey Data Analysis ^{*} 
20 

22 
Replicated Sampling: A General Approach 
23 
Balanced Repeated Replication 
26 
Jackknife Repeated Replication 
29 
The Bootstrap Method 
35 
The Taylor Series Method (Linearization) 
36 

39 
Data Requirements for Survey Analysis 
39 
Importance of Preliminary Analysis 
41 
Choices of Method for Variance Estimation 
43 
Available Computing Resources 
44 
Creating Replicate Weights 
47 
Searching for Appropriate Models for Survey Data Analysis ^{*} 
49 

49 
A Strategy for Conducting Preliminary Analysis 
50 
Conducting Descriptive Analysis 
52 
Conducting Linear Regression Analysis 
57 
Conducting Contingency Table Analysis 
61 

Conducting Logistic Regression Analysis 
65 

Other Logistic Regression Models 
69 

DesignBased and ModelBased Analyses ^{*} 
74 

7. 
Concluding Remarks 
78 
Notes 
80 

References 
83 

Index 
88 

About the Authors 
91 
SERIES EDITOR’S INTRODUCTION
When George Gallup correctly predicted Franklin D. Roosevelt as the 1936 presidential election winner, public opinion surveys entered the age of scientiﬁc sampling, and the meth od used then was quota sampling, a type of nonprobability sampling repre sentative of the target population. The same method, however, incorrec tly predicted Thomas Dewey as the 1948 winner though Harry S. Truman actually won. The method failed because quota sampling is nonprobabilistic and because Gallup’s quota frames were based on the 1940 census, overlooking the urban migration during World War II. Today’s survey sampling has advanced much since the early days, now relying on sophisticated probabilistic sampling designs. A key feature is stra tiﬁcation: The target population is divided into subpopulations of strata, and the sample sizes in the strata are controlled by the sampler and are often pro portional to the strata population sizes. Another feature is cluster and multi stage sampling: Groups are sampled as clusters in a hierarchy of clusters selected at various stages until the ﬁnal stage, when individual elements are sampled within the ﬁnalstage clusters. The General Social Survey, for exam ple, uses a stratiﬁed multistage cluster sampling design. (Kalton [1983] gives a nice introduction to survey sampling.) When the survey design is of this complex nature, statistical analysis of the data is no longer a simple matter of running a regression (or any other model ing) analysis. Surveys today all come with sampling weights to assist with correct statistical inference. Most texts on statistical analysis, by assuming simple random sampling, do not include treatment of sampling weights, an omission that may have important implications for making inferences. Dur ing the last two to three decades, statistical methods for data analysis have also made huge strides. These must have been the reason my predecessor, Michael LewisBeck, who saw through the early stages of the editorial work in this volume, chose to have a second edition of Analyzing Complex Survey Data. Lee and Forthofer’s second edition of the book brings us up to date in uniting survey sampling designs and survey data analysis. The authors begin by reviewing common types of survey sample designs and demystify ing sampling weights by explaining what they are, and how they are devel oped and adjusted. They then carefully discuss the major issues of variance estimation and of preliminary as well as multivariate analysis of complex crosssectional survey data when sampling weights are taken into account. They focus on the designbased approach that directly engages sample designs in the analysis (although they also discuss the modelbased perspective, which
v
vi
can augment a designbased approach in some analyses), and they illustrate the approach with popular software examples. Students of survey analysis will ﬁnd the text of great use in their efforts in making samplebased statistical inferences.
—Tim Futing Liao Series Editor
vii
ACKNOWLEDGMENTS
We sadly acknowledge that Dr. Ronald J. Lorimor, who had collaborated on the writing of the ﬁrst edition of this manuscript, died in 1999. His insights into survey data analysis remain in this second edition. We are grateful to Tom W. Smith for answering questions about the sample design of the General Social Survey and to Barry L. Graubard, Lu Ann Aday, and Michael S. LewisBeck for their thoughtful suggestions for the ﬁrst edition. Thanks also are due to anonymous reviewers for their helpful comments for both editions and to Tim F. Liao for his thoughtful advice for the contents of the second edition. Special thanks go to many students in our classes at the University of Texas School of Public Health who participated in discus sions of many topics contained in this book.
Eun Sul Lee
Division of Biostatistics, School of Public Health,
University of Texas Health Science Center—Houston
Ronald N. Forthofer
1. INTRODUCTION
Survey analysis often is conducted as if all sample observations were independently selected with equal probabilities. This analysis is correct if simple random sampling (SRS) is used in data collection; however, in prac tice the sample selection is more complex than SRS. Some sample observa tions may be selected with higher probabilities than others, and some are included in the sample by virtue of their membership in a certain group (e.g., household) rather than being selected independently. Can we simply ignore these departures from SRS in the analysis of survey data? Is it appro priate to use the standard techniques in statistics books for survey data analysis? Or are there special methods and computer programs available for a more appropriate analysis of complex survey data? These questions are addressed in the following chapters. The typical social survey today reﬂects a combination of statistical theory and knowledge about social phenomena, and its evolution has been shaped by experience gained from the conduct of many different surveys during the last 70 years. Social surveys were conducted to meet the need for information to address social, political, and public health issues. Survey agencies were established within and outside the government in response to this need for information. In the early attempts to provide the required information, how ever, the survey groups were mostly concerned with the practical issues in the ﬁeldwork—such as sampling frame construction, staff training/supervision, and cost reduction—and theoretical sampling issues received only secondary emphasis (Stephan, 1948). As these practical matters were resolved, modern sampling practice had developed far beyond SRS. Complex sample designs had come to the fore, and with them, a number of analytic problems. Because the early surveys generally needed only descriptive statistics, there was little interest in analytic problems. More recently, demands for
1
2
analytic studies by social and policy scientists have increased, and a variety of current issues are being examined, using available social survey data, by researchers who were not involved with the data collection process. This tradition is known as secondary analysis (Kendall & Lazarsfeld, 1950). Often, the researcher fails to pay due attention to the development of com plex sample designs and assumes that these designs have little bearing on the analytic procedures to be used. The increased use of statistical techniques in secondary analysis and the recent use of loglinear models, logistic regression, and other multivariate techniques (Aldrich & Nelson, 1984; Goodman, 1972; Swafford, 1980) have done little to bring design and analysis into closer alignment. These techniques are predicated on the use of simple random sampling with re placement (SRSWR); however, this assumption is rarely met in social surveys that employ stratiﬁcation and clustering of observational units along with unequal probabilities of selection. As a result, the analysis of social surveys using the SRSWR assumption can lead to biased and misleading results. Kiecolt and Nathan (1985), for example, acknowledged this problem in their Sage book on secondary analysis, but they provide little guidance on how to incorporate the sample weights and other design features into the analysis. A recent review of literature in public health and epidemiology shows that the use of designbased survey analysis methods is gradually increasing but remains at a low level (Levy & Stolte, 2000). Any survey that puts restrictions on the sampling beyond those of SRSWR is complex in design and requires special analytic considerations. This book reviews the analytic issues raised by the complex sample survey, provides an introduction to analytic strategies, and presents illustrations using some of the available software. Our discussion is centered on the use of the sample weights to correct for differential representations and the effect of sample designs on estimation of sampling variance with some dis cussion of weight development and adjustment procedures. Many other important issues of dealing with nonsampling errors and handling missing data are not fully addressed in this book. The basic approach presented in this book is the traditional way of ana lyzing complex survey data. This approach is now known as designbased (or randomizationbased) analysis. A different approach to analyzing com plex survey data is the socalled modelbased analysis. As in other areas of statistics, the modelbased statistical inference has gained more attention in survey data analysis in recent years. The modeling approaches are intro duced in various steps of survey data analysis in deﬁning the parameters, deﬁning estimators, and estimating variances; however, there are no gener ally accepted rules for model selection or validating a speciﬁed model. Nevertheless, some understanding of the modelbased approach is essential
3
for survey data analysts to augment the designbased approach. In some cases, both approaches produce the same results; but different results occur in other cases. The modelbased approach may not be useful in descriptive data analysis but can be useful in inferential analysis. We will introduce the modelbased perspective where appropriate and provide references for further treatment of the topics. Proper conduct of modelbased analysis would require knowledge of general statistical models and perhaps some consultation from survey statisticians. Sections of the book relevant to this alternative approach and related topics are marked with asterisks (*). Since the publication of the ﬁrst edition of this book, the software situation for the analysis of complex survey data has improved considerably. User friendly programs are now readily available, and many commonly used sta tistical methods are now incorporated in the packages, including logistic regression and survival analysis. These programs will be introduced with illustrations in this edition. These programs are perhaps more open to misuse than other standard software. The topics and issues discussed in this book will provide some guidelines for avoiding pitfalls in survey data analysis. In our presentation, we assume some familiarity with such sampling designs as simple random sampling, systematic sampling, stratiﬁed random sampling, and simple twostage cluster sampling. A good presentation of these designs may be found in Kalton (1983) and Lohr (1999). We also assume general understanding of standard statistical methods and one of the standard statistical program packages, such as SAS or Stata.
2. SAMPLE DESIGN AND SURVEY DATA
Our consideration of survey data focuses on sample designs that satisfy two basic requirements. First, we are concerned only with probability sam pling in which each element of a population has a known (nonzero) prob ability of being included in the sample. This is the basis for applying statistical theory in the derivation of the properties of the survey estimators for a given design. Second, if a sample is to be drawn from a population, it is necessary to be able to construct a sampling frame that lists suitable sam pling units that encompass all elements of the population. If it is not feasible or is impractical to list all population elements, some clusters of elements can be used as sampling units. For example, it is impractical to construct a list of all households in the United States, but we can select the sample in several stages. In the ﬁrst stage, counties are randomly sampled; in the second stage, census tracts within the selected counties are sampled; in the third stage, street blocks are sampled within the selected tracts. Then, in the ﬁnal stage of selection, a list of households is needed only for the selected
4
blocks. This multistage design satisﬁes the requirement that all population elements have a known nonzero probability of being selected.
Types of Sampling
The simplest sample design is simple random sampling, which requires that each element have an equal probability of being included in the sample and that the list of all population elements be available. Selection of a sample element can be carried out with or without replacement. Simple random sampling with replacement (SRSWR) is of special interest because it simpliﬁes statistical inference by eliminating any relation (covariance) between the selected elements through the replacement process. In this scheme, however, an element can appear more than once in the sample. In practice, simple random sampling is carried out without replacement (SRSWOR), because there is no need to collect the information more than once from an element. Additionally, SRSWOR gives a smaller sampling variance than SRSWR. However, these two sampling methods are practi cally the same in a large survey in which a small fraction of population elements is sampled. We will use the term SRS for SRSWOR throughout this book unless otherwise speciﬁed. The SRS design is modiﬁed further to accommodate other theoretical and practical considerations. The common practical designs include sys tematic sampling, stratiﬁed random sampling, multistage cluster sampling, PPS sampling (probability proportional to size), and other controlled selec tion procedures. These more practical designs deviate from SRS in two important ways. First, the inclusion probabilities for the elements (also the joint inclusion probabilities for sets for the elements) may be unequal. Second, the sampling unit can be different from the population element of interest. These departures complicate the usual methods of estimation and variance calculation and, if proper methods of analysis are not used, can lead to a bias in estimation and statistical tests. We will consider these departures in detail, using several speciﬁc sampling designs, and examine their implications for survey analysis. Systematic sampling is commonly used as an alternative to SRS because of its simplicity. It selects every kth element after a random start (between 1 and k). Its procedural tasks are simple, and the process can easily be checked, whereas it is difﬁcult to verify SRS by examining the results. It is often used in the ﬁnal stage of multistage sampling when the ﬁeldworker is instructed to select a predetermined proportion of units from the listing of dwellings in a street block. The systematic sampling procedure assigns each element in a population the same probability of being selected. This ensures that the sample mean will be an unbiased estimate of the population mean
5
when the number of elements in the population (N) is equal to k times the number of elements in the sample (n). If N is not exactly nk, then the equal probability is not guaranteed, although this problem can be ignored when N is large. In that case, we can use the circular systematic sampling scheme. In this scheme, the random starting point is selected between 1 and N (any element can be the starting point), and every kth element is selected assum ing that the frame is circular (the end of the list is connected to the beginning of the list). Systematic sampling can give an unrealistic estimate, however, when the elements in the frame are listed in a cyclical manner with respect to survey variables and the selection interval coincides with the listing cycle. For example, if one selects every 40th patient coming to a clinic and the average daily patient load is about 40, then the resulting systematic sample would contain only those who came to the clinic at a particular time of the day. Such a sample may not be representative of the clinic patients. Moreover, even when the listing is randomly ordered, unlike SRS, differ ent sets of elements may have unequal inclusion probabilities. For example, the probability of including both the ith and the (i + k)th element is 1/k in a systematic sample, whereas the probability of including both the ith and the (i + k + 1)th is zero. This complicates the variance calculation. Another way of viewing systematic sampling is that it is equivalent to selecting one cluster from k systematically formed clusters of n elements each. The sampling variance (between clusters) cannot be estimated from the one selected cluster. Thus, variance estimation from a systematic sample requires special strategies. A modiﬁcation to overcome these problems with systematic sampling is the socalled repeated systematic sampling (Levy & Lemeshow, 1999, pp. 101–110). Instead of taking a systematic sample in one pass through the list, several smaller systematic samples are selected, going down the list several times with a new starting point in each pass. This procedure not only guards against possible periodicity in the frame but also allows variance estimation directly from the data. The variance of an estimate from all sub samples can be estimated from the variability of the separate estimates from each subsample. This idea of replicated sampling offers a strategy for esti mating variance for complex surveys, which will be discussed further in Chapter 4. Stratiﬁed random sampling classiﬁes the population elements into strata and samples separately from each stratum. It is used for several reasons:
(a) The sampling variance can be reduced if strata are internally homoge neous, (b) separate estimates can be obtained for strata, (c) administration of ﬁeldwork can be organized using strata, and (d) different sampling needs can be accommodated in separate strata. Allocation of the sample across the strata is proportionate when the sampling fraction is uniform across the
6
strata or disproportionate when, for instance, a higher sampling fraction is applied to a smaller stratum to select a sufﬁcient number of subjects for comparative studies. In general, the estimation process for a stratiﬁed random sample is more complicated than in SRS. It is generally described as a twostep process. The ﬁrst step is the calculation of the statistics—for example, the mean and its variance—separately within each stratum. These estimates are then combined based on weights reﬂecting the proportion of the population in each stratum. As will be discussed later, it also can be described as a onestep process using weighted statistics. The estimation simpliﬁes in the case of proportionate stratiﬁed sampling, but the strata must be taken into account in the variance estimation. The formulation of the strata requires that information on the stratiﬁca tion variable(s) be available in the sampling frame. When such information is not available, stratiﬁcation cannot be incorporated in the design. But stra tiﬁcation can be done after data are collected to improve the precision of the estimates. The socalled poststratiﬁcation is used to make the sample more representative of the population by adjusting the demographic composi tions of the sample to the known population compositions. Typically, such demographic variables as age, sex, race, and education are used in poststra tiﬁcation in order to take advantage of the population census data. This adjustment requires the use of weights and different strategies for variance estimation because the stratum sample size is a random variable in the poststratiﬁed design (determined after the data are collected). Cluster sampling is often a practical approach to surveys because it samples by groups (clusters) of elements rather than by individual ele ments. It simpliﬁes the task of constructing sampling frames, and it reduces the survey costs. Often, a hierarchy of geographical clusters is used, as described earlier. In multistage cluster sampling, the sampling units are groups of elements except for the last stage of sampling. When the numbers of elements in the clusters are equal, the estimation process is equivalent to SRS. However, simple random sampling of unequalsized clusters leads to the elements in the smaller clusters being more likely to be in the sample than those in the larger clusters. Additionally, the clusters are often strati ﬁed to accomplish certain survey objectives and ﬁeld procedures, for instance, the oversampling of predominantly minority population clusters. The use of disproportionate stratiﬁcation and unequalsized clusters com plicates the estimation process. One method to draw a selfweighting sample of elements in onestage cluster sampling of unequalsized clusters is to sample clusters with prob ability proportional to the size of clusters (PPS sampling). However, this requires that the true size of clusters be known. Because the true sizes usually are unknown at the time of the survey, the selection probability is
7
instead made proportional to the estimated size (PPES sampling). For example, the number of beds can be used as a measure of size in a survey of hospital discharges with hospitals as the clusters. One important consequence of PPES sampling is that the expected sample size will vary from one primary sampling unit (PSU) to another. In other words, the sample size is not ﬁxed but varies from sample to sample. Therefore, the sample size, the deno minator in the calculation of a sample mean, is a random variable, and, hence, the sample mean becomes a ratio of two random variables. This type of variable, a ratio variable, requires special strategies for variance estimation.
The Nature of Survey Data
If we are to infer from sample to population, the sample selection process is an integral part of the inference process, and the survey data must contain information on important dimensions of the selection process. Considering the departures from SRS in most social surveys, we need to view the survey data not only as records of measurements, but also as having different representation and structural arrangements. Sample weights are used to reﬂect the differing probabilities of selection of the sample elements. The development of sample weights requires keep ing track of selection probabilities separately in each stratum and at each stage of sampling. In addition, it can involve correcting for differential response rates within classes of the sample and adjusting the sample distribution by demographic variables to known population distributions (poststratiﬁcation adjustment). Moreover, different sample weights may be needed for different units of analysis. For instance, in a community survey it may be necessary to develop person weights for an analysis of individual data and household weights for an analysis of household data. We may feel secure in the exclusion of the weights when one of the following selfweighting designs is used. True PPS sampling in a onestage cluster sampling will produce a selfweighting sample of elements, as in the SRS design. The selfweighting can also be accomplished in a twostage design when true PPS sampling is used in the ﬁrst stage and a ﬁxed number of elements is selected within each selected PSU. The same result will follow if simple random sampling is used in the ﬁrst stage and a ﬁxed proportion of the elements is selected in the second stage (see Kalton, 1983, chaps. 5 and 6). In practice, however, the selfweighting feature is destroyed by nonre sponse and possible errors in the sampling frame(s). This unintended self selection process can introduce bias, but it is seldom possible to assess the bias from an examination of the sample data. Two methods employed in an attempt to reduce the bias are poststratiﬁcation and nonresponse adjustments. Poststratiﬁcation involves assigning weights to bring the sample proportion
8
in demographic subgroups into agreement with the population proportion
in the subgroups. Nonresponse adjustment inﬂates the weights for those who
participate in the survey to account for the nonrespondents with similar char
acteristics. Because of the nonresponse and poststratiﬁcation adjustments by
weighting, the use of weights is almost unavoidable even when a selfweighting
design is used.
The sample design affects the estimation of standard errors and, hence,
must also be incorporated into the analysis. A close examination of the
familiar formulas for standard errors found in statistics textbooks and incor
porated into most computer program packages shows that they are based
on the SRSWR design. These formulas are relatively simple because the
covariance between elements is zero, as a result of the assumed independent
selection of elements. It is not immediately evident how the formulas should
be modiﬁed to adjust for other complex sampling designs.
To better understand the need for adjustment to the variance formulas, let
us examine the variance formula for several sample designs. We ﬁrst con
sider variance for a sample mean from the SRSWOR design. The familiar
variance formula for a sample mean, y (selecting a sample of n elements
from a population of N elements by SRSWR where the population mean
is Y), in elementary statistics textbooks is ^{2} /n, where ^{2} = ^{P} (Y _{i} − Y) ^{2} /N .
This formula needs to be modiﬁed for the SRSWOR design because the
selection of an element is no longer independent of the selection of another
element. Because of the condition of not allowing duplicate selection, there
is a negative covariance [− ^{2} /(N − 1)] between ith and j th sample ele
ments. Incorporating n(n − 1) times the covariance, the variance of the sam
ple mean for SRSWOR is ^{} ^{2}
n
), which is smaller than that from SRSWR
by the factor of (N − n)/(N − 1). Substituting the unbiased estimator of
^{2} of [(N − 1)s ^{2} /N ], the estimator for the variance of the sample mean
from SRSWOR is
^
V (y) =
s ^{2}
n
(1 − f ),
(2:1)
where s ^{2} = ^{} (x _{i} − x) ^{2} /(n − 1) and f = n/N . Both (N − n)/(N − 1) and
(1 − f ) are called the ﬁnite population correction (FPC) factor. In a large
population, the covariance will be very small because the sampling fraction
is small. Therefore, SRSWR and SRSWOR designs will produce practically
the same variance, and these two procedures can be considered equivalent
for all practical purposes.
Stratiﬁed sampling is often presented as a more efﬁcient design because it
gives, if used appropriately, a smaller variance than that given by a comparable
SRS. Because the covariances between strata are zero, the variance of the
9
sample estimate is derived from the withinstratum variances, which are
combined based on the stratum sample sizes and the stratum weights. The
value of a stratiﬁed sample variance depends on the distribution of the strata
sample sizes. An optimal (or Neyman) allocation produces a sampling
variance less than or equal to that based on SRS except in extremely rare situa
tions. For other disproportionate allocations, the sampling variance may turn
out to be larger than that based on SRS when the ﬁnite population correction
factor (FPC) within strata cannot be ignored. Therefore, it cannot be assumed
that stratiﬁcation will always reduce sampling variance compared to SRS.
The cluster sampling design usually leads to a larger sampling variance
than that from SRS. This is because the elements within naturally formed
clusters are often similar, which then yield a positive covariance between
elements within the cluster. The homogeneity within clusters is measured
by the intraclass correlation coefﬁcient (ICC)—the correlation between all
possible pairs of elements within clusters. If clusters were randomly formed
(i.e., if each cluster were a random sample of elements), the ICC would be
zero. In many natural clusters, the ICC is positive and, hence, the sampling
variance will be larger than that for the SRS design.
It is difﬁcult to generalize regarding the relative size of the sampling
variance in a complex design because the combined effects of stratiﬁcation
and clustering, as well as that of the sample weights, must be assessed.
Therefore, all observations in survey data must be viewed as products of a
speciﬁc sample design that contains sample weights and structural arrange
ments. In addition to the sample weights, strata and cluster identiﬁcation
(at least PSUs) should be included in sample survey data. Reasons for these
requirements will become clearer later.
One complication in the variance calculation for a complex survey stems
from the use of weights. Because the sum of weights in the denominator of
any weighted estimator is not ﬁxed but varies from sample to sample, the esti
mator becomes a ratio of two random variables. In general, a ratio estimator
is biased, but the bias is negligible if the variation in the weights is relatively
small or the sample size is large (Cochran, 1977, chap. 6). Thus, the problem
of bias in the ratio estimator is not an issue in large social surveys. Because of
this bias, however, it is appropriate to use the mean square error—the sum of
the variance plus the square of the bias—rather than the variance. However,
because the bias often is negligible, we will use the term ‘‘variance’’ even if
we are referring to the mean square error in this book.
A Different View of Survey Data ^{*}
So far, the nature of survey data is described from the designbased
perspective—that is, sample data are observations sampled from a ﬁnite
10
population using a particular sample selection design. The sampling design
speciﬁes the probability of selection of each potential sample, and a proper
estimator is chosen to reﬂect the design. As mentioned in the introduction,
the modelbased perspective offers an alternative view of sample survey
data. Observations in the ﬁnite population are viewed as realizations of a
random variable generated from some model (a random variable that fol
lowed some probability distribution). The assumed probability model sup
plies the link between units in the sample and units not in the sample. In the
modelbased approach, the sample data are used to predict the unobserved
values, and thus inferences may be thought of as prediction problems (Royall,
1970, 1973).
These two points of view may not make a difference in SRS, where we can
reasonably assume that sample observations were independent and identi
cally distributed from a normal distribution with mean µ and variance .
From the model point of view, the population total is the sum of observations
in the sample and the sum of observations that are not in the sample; that is,
Y = ^{P} _{i}_{∈}_{S} y _{i} + ^{P} _{i} _{} _{∈}_{S} y _{i} . Based on the assumption of common mean, the
^
estimate of population total can be made as Y = ny + ðN − nÞy = N y , where
y is the best unbiased predictor of the unobserved observations under the
model. It turns out to be the same as the expansion estimator in the design
based approach, namely,
^
Y = (N/n) ^{P} ^{n} _{=} _{1} y _{i} = N y ; where (N/n) is the
i
sample weight (inverse of selection probability in SRS). Both approaches
lead to the same variance estimate (Lohr, 1999, sec. 2.8).
If a different model were adopted, however, the variance estimates might
differ. For example, in the case of ratio ^{1} and regression estimation under
SRS, the assumed model is Y _{i} = βx _{i} + ε _{i} , where Y _{i} is for a random variable
and x _{i} is an auxiliary variable for which the population total is known.
Under this model, the linear estimate of the population
total will be
^
^
^{Y} ^{=} ^{P} i∈S ^{y} i ^{+} ^{P} i ∈S ^{y} i ^{=} ^{n}^{y}
β ^{P} _{i} _{} _{∈}_{S} x _{i} . The ﬁrst part is from the sam
+
ple, and the second part is the prediction for the unobserved units based on
^
the assumed model. If we take β as the sample ratio of y/
x , then we have
^
Y = ny +
y x P i ∈S ^{x} i ^{=} ^{y} x ^{}
ðnx + ^{P} _{i} _{} _{∈}_{S} x _{i} Þ = ^{y} ^{} X, where X is the population
x
^
total of x _{i} . This is simply the ratio estimate of Y. If we take β as the esti
mated regression coefﬁcient, then we have a regression estimation. Although
the ratio estimate is known to be slightly biased from the designbased
viewpoint, it is unbiased from the modelbased reasoning if the model is
correct.
But the estimate of variance by the modelbased approach is slightly
different from the estimate by the designbased approach. The
designbased estimate of variance of the estimated population total
11
^ 
^ 

is 
V _{D} ð Y Þ = ð1 − 

^ 
^ 

V M ð 
Y Þ = ð1 − 
x 
^{Þ} 

X 
N ^{Þ}
n
N ^{2}
n
^{} ^{P} ½y _{i} −ðy =x Þx _{i} ^{2}
n−1
. The modelbased estimator is
X
^{2}
x
^{2}
^{} ^{P} ½{y _{i} −ðy =x Þ}=
ﬃﬃﬃﬃ
p
x
i
n−1
, where x is the sample total and X
is the population total of the auxiliary variable (see Lohr, 1999, sec. 3.4).
The ratio estimate model is valid when (a) the relation between y _{i} and
x _{i} is a straight line through the origin and (b) the variance of y _{i} about this
line is proportional to x _{i} . It is known that the ratio estimate is inferior to the
expansion estimate (without the auxiliary variable) when the correlation
between y _{i} and x _{i} is less than onehalf the ratio of coefﬁcient of variation
of x _{i} over the coefﬁcient of variation of y _{i} (Cochran, 1977, chap. 6). There
fore, the use of ratio estimation in survey analysis would require check
ing the model assumptions. In practice, when the data set includes a large
number of variables, ratio estimation would be cumbersome to select different
auxiliary variables for different estimates.
To apply the modelbased approach to a real problem, we must ﬁrst be
able to produce an adequate model. If the model is wrong, the modelbased
estimators will be biased. When using modelbased inference in sampling,
one needs to check the assumptions of the model by examining the data
carefully. Checking the assumptions may be difﬁcult in many circum
stances. The adequacy of a model is to some extent a matter of judgment,
and a model adequate for one analysis may not be adequate for another
analysis or another survey.
3. COMPLEXITY OF ANALYZING SURVEY DATA
Two essential aspects of survey data analysis are adjusting for the
differential representation of sample observations and assessing the loss or
gain in precision resulting from the complexity of the sample selection
design. This chapter introduces the concept of weight and discusses the
effect of sample selection design on variance estimation. To illustrate the
versatility of weighting in survey analysis, we present two examples of
developing and adjusting sample weights.
Adjusting for Differential Representation: The Weight
Two types of sample weights are commonly encountered in the analysis
of survey data: (a) the expansion weight, which is the reciprocal of the
selection probability, and (b) the relative weight, which is obtained by scal
ing down the expansion weight to reﬂect the sample size. This section
reviews these two types of weights in detail for several sample designs.
12
Consider the following SRS situation: A list of N = 4,000 elements in a
population is numbered from 1 to 4,000. A table of random numbers is used
to draw a ﬁxed number of elements (for example, n = 200) from the popula
tion, not allowing duplicate selection (without replacement). The selection
probability or sampling fraction is f = n/N = .05. The expansion weight
is the reciprocal of the selection probability, w _{i} = 1/f = N/n = 20
(i = 1,
...
, n), which indicates the number of elements represented by a
sample observation in the population. These weights for the n elements
selected sum to N : An estimator of the population total of variable Y based
on the sample elements is
Y = ^{} w _{i} y _{i} = (N/n) ^{} y _{i} = N y :
^
(3:1)
Equation 3.1 shows the use of the expansion weight in the weighted
sum of sample observations. Because the weight is the same for each ele
ment in SRS, the estimator can be simpliﬁed to N times of the sample
mean (the last quantity in Equation 3.1). Similarly, the estimator of the
population mean is deﬁned as
^
¯
Y = ^{} w _{i} y _{i} / ^{} w _{i} , which is the weighted
sample mean. In SRS, this simpliﬁes to (N/n) ^{} y _{i} /N = y , showing that
the sample mean is an estimator for the population mean. However, even
if the weights are not the same (in unequal probability designs), the esti
mators are still a weighted sum for the population total and a weighted
average for the population mean.
Although the expansion weight appears appropriate for the estimator of
the population total, it may play havoc with the sample mean and other sta
tistical measures. For example, using the sum of expansion weights in con
tingency tables in place of relative frequencies based on sample size may
lead to unduly large conﬁdence in the data. To deal with this, the expansion
weight can be scaled down to produce the relative weight, (rw) _{i} , which is
deﬁned to be the expansion weight divided by the mean of the expansion
weights, that is, w _{i} /w , where w = ^{P} w _{i} /n. These relative weights for all
elements in the sample add up to n: For the SRS design, (rw) _{i} is 1 for each
element. The estimator for the population total weighted by the relative
weights is
Y = w ^{} (rw) _{i} y _{i} = (N/n) ^{} y _{i} = N y :
^
(3:2)
Note in Equation 3.2 that the relative weighted sum is multiplied by the
average expansion weight, which yields the same simpliﬁed estimator for
the case of SRS as in Equation 3.1. Hence, the expansion weight is simpler
to use than the relative weight in estimating the population total. The
relative weight is appropriate in analy tic studies, but it is inappropriate in
estimating totals and computing ﬁnite population corrections.
13
Most publicuse survey data from government agencies and survey
organizations use the expansion weight, and it can easily be converted to
the relative weight. Such conversion is not necessary, however, because the
userfriendly statistical programs for survey analysis automatically perform
the conversion internally when appropriate.
Let us consider expansion weights in a stratiﬁed random sampling design.
In this design, the population of N elements is grouped into L strata based
on a certain variable with N _{1} , N _{2} ,
...
, N _{L} elements respectively, from
which n _{h} (h = l, 2,
...
, L) elements are independently selected from the hth
stratum. A stratiﬁed design retains a selfweighting quality when the
sampling fraction in each stratum is the same. If a total of 200 elements are
proportionately selected from two strata of N _{1} = 600 and N _{2} = 3,400
elements, then f = 200/4,000 = .05. A proportionate selection (a 5%
sample from each stratum) yields n _{1} = 30 and n _{2} = 170 because
f _{1} = 30/600 = .05 and f _{2} = 170/3,400 = .05. The weighting scheme is
then exactly the same as in SRS design.
The situation is slightly different with a disproportionate stratiﬁed sample
design. For example, if the total sample of 200 were split equally between the
two strata, f _{1} (¼100/600) and f _{2} (¼100/3,400) could have different values
and the expansion weights would be unequal for the elements in the two strata,
with w _{1}_{i} = 6 and w _{2}_{i} = 34. The expansion weights sum to 600 in the ﬁrst
stratum and to 3,400 in the second with their total being 4,000, the population
size. The mean expansion weight, w
= (100 × 6 + 100 × 34)/200 = 20,
and, hence, the relative weights sum to 30 in the ﬁrst stratum and to 170 in the
second. The sums of relative weights in both strata add up to the total sample
size. Note that the use of either type of weight is equivalent to weighting
stratum means (y _{h} ) using the population distribution across the strata
[i.e., ^{P} (N _{h} /N)y _{h} , the standard procedure]. Both the expansion and relative
weights in stratum 1 sum to 15% of their respective total sums, and the ﬁrst
stratum also contains 15% of the population elements.
Although we have used SRS and stratiﬁed sample designs to introduce the
sample weights, the same concept extends easily to more complex designs.
In summary, the sample weight is the inverse of the selection probability,
although it often is further modiﬁed by poststratiﬁcation and nonresponse
adjustments. The assignment of the sample weight to each sample element
facilitates a general estimation procedure for all sample designs. As a general
rule, all estimates take the form of weighted statistics in survey data analysis.
The scale of these weights does not matter in estimating parameters and stan
dard errors except when estimating totals and computing ﬁnite population
corrections.
Next, we present two examples of developing/adjusting the weight. The
ﬁrst example shows how sample weights are modiﬁed by the poststratiﬁcation
14
procedure in order to make sample compositions conform to population
compositions. The same procedure can be used to adjust for differential
response rates in demographic subgroups. The poststratiﬁcation approach
works well when only a few demographic variables are involved. The second
example demonstrates that differential attrition rates in a followup survey can
be adjusted for a large number of variables using a logistic regression model.
Developing the Weight by Poststratiﬁcation
To demonstrate the development of sample weights, we shall work with
the 1984 General Social Survey (GSS). This is a complex sample survey
conducted by the National Opinion Research Center (NORC) to obtain gen
eral social information from the civilian noninstitutionalized adult popula
tion (18 years of age and older) of the United States. A multistage selection
design was used to produce a selfweighting sample at the household level.
One adult was then randomly select ed from each sampled household
(Davis & Smith, 1985). There were 1,473 observations available for ana
lysis in the data ﬁle. For the purpose of illustration, the expansion weight
for these data at the household level could be calculated by dividing the
number of households in the United States by 1,473. The expansion
weight within the sampled household is the number of adults in the house
hold. The product of these two weights gives the expansion weight for
sample individuals.
For an analysis of the GSS data, we need to focus only on the weight
within the household, because each household has the same probability of
selection. The relative weight for the individual can be derived by dividing
the number of adults in the household by the average number of adults
(2,852/1,473 = 1.94) per household. This weight reﬂects the probability
of selection of an individual in the sample while preserving the sample size.
We further modiﬁed this weight by a poststratiﬁcation adjustment in an
attempt to make the sample composition the same as the population compo
sition. This would improve the precision of estimates and could possibly
reduce nonresponse and sample selection bias to the extent that it is related
to the demographic composition. ^{2} As shown in Table 3.1, the adjustment
factor is derived to cause the distribution of individuals in the sample to
match the 1984 U.S. population by age, race, and sex. Column 1 is the 1984
population distribution by race, sex, and age, based on the Census Bureau’s
estimates. Column 2 shows the weighted number of adults in the sampled
households by the demographic subgroups, and the proportional distribu
tion is in Column 3. The adjustment factor is the ratio of Column 1 to
Column 3. The adjusted weight is found by multiplying the adjustment
factor by the relative weight, and the distribution of adjusted weights is then
15
TABLE 3.1
Derivation of Poststratiﬁcation Adjustment Factor:
General Social Survey, 1984
Weighted 

Demographic 
Population 
Number of 
Sample 
Adjustment 

Subgroups 
Distribution 
Adults 
Distribution 
Factor 

(1) 
(2) 
(3) 
(1)/(3) 

White, male 

18–24 years 
.0719660 
211 
.0739832 
0.9727346 

25–34 
.1028236 
193 
.0676718 
1.5194460 

35–44 
.0708987 
277 
.0795933 
0.8907624 

45–54 
.0557924 
135 
.0473352 
1.1786660 

55–64 
.0544026 
144 
.0504909 
1.0774730 

65 
and over 
.0574872 
138 
.0483871 
1.1880687 
White, female 

18–24 years 
.0705058 
198 
.0694250 
1.0155668 

25–34 
.1007594 
324 
.1136045 
0.8869317 

35–44 
.0777364 
267 
.0936185 
0.8303528 

45–54 
.0582026 
196 
.0682737 
0.8469074 

55–64 
.0610057 
186 
.0652174 
0.9354210 

65 
and over 
.0823047 
216 
.0757363 
1.0867272 
Nonwhite, male 

18–24 years 
.0138044 
34 
.0119215 
1.1579480 

25–34 
.0172057 
30 
.0105189 
1.6356880 

35–44 
.0109779 
30 
.0105189 
1.0436290 

45–54 
.0077643 
37 
.0129734 
0.5984774 

55–64 
.0064683 
12 
.0042076 
1.5372900 

65 
and over 
.0062688 
18 
.0063113 
0.9932661 
Nonwhite, female 

18–24 years 
.0145081 
42 
.0145081 
.9851716 

25–34 
.0196276 
86 
.0301543 
.6509067 

35–44 
.0130655 
38 
.0133240 
.9806026 

45–54 
.0094590 
33 
.0115708 
.8174890 

55–64 
.0079636 
30 
.0105189 
.7570769 

65 
and over 
.0090016 
27 
.0094670 
.9508398 
Total 
1.0000000 
2,852 
1.0000000 
SOURCE: U.S. Bureau of the Census, Estimates of the population of the United States, by age, sex, and race, 1980 to 1985 (Current Population Reports, Series P25, No. 985), April 1986. Noninstitutional population estimates are derived from the estimated total population of 1984 (Table1), adjusted by applying the ratio of noninstitutional to total population (Table Al).
the same as the population distribution. The adjustment factors indicate that
without the adjustment, the GSS sample underrepresents males 25–34 years
of age and overrepresents nonwhite males 45–54 years of age and nonwhite
females 25–34 years of age.
16
TABLE 3.2
Comparison of Weighted and Unweighted Estimates in Two Surveys
Weighted 
Unweighted 

Surveys and Variables 
Estimates 
Estimates 
I. General Social Survey 

(Percentage approving ‘‘hitting’’) 

Overall 
60.0 
59.4 
By sex 

Male 
63.5 
63.2 
Female 
56.8 
56.8 
By education 

Some college 
68.7 
68.6 
High school 
63.3 
63.2 
Others 
46.8 
45.2 
II. Epidemiologic Catchment Areas Survey 

(Prevalence rate of mental disorders) 

Any disorders 
14.8 
8.8 
Anxiety disorders 
6.5 
18.5 
SOURCE: Data for the Epidemiologic Catchment Areas Survey are from E. S. Lee, Forthofer, and Lorimor (1986), Table 1.
The adjusted relative weights are then used in the analysis of the data—for
example, to estimate the proportion of adults responding positively to the
question, ‘‘Are there any situations that you can imagine in which you would
approve of a man punching an adult male stranger?’’ As shown in the upper
section of Table 3.2, the weighted overall proportion is 60.0%, slightly larger
than the unweighted estimate of 59.4%. The difference between the weighted
and unweighted estimates is also very small for the subgroup estimates
shown. This may be due primarily to the selfweighting feature reﬂected in
the fact that most households have two adults and, to a lesser extent, to the
fact that the ‘‘approval of hitting’’ is not correlated with the number of adults
in a household. The situation is different in the National Institute of Mental
HealthSponsored Epidemiologic Catchment Area (ECA) Survey. In this
survey, the weighted estimates of the prevalence of any disorders and of anxi
ety disorders are, respectively, 20% and 26% lower than the unweighted
estimates, as shown in Table 3.2.
Finally, the adjusted weights should be examined to see whether there
are any extremely large values. Extreme variation in the adjusted weights
may imply that the sample sizes in some poststrata are too small to be reli
able. In such a case, some small poststrata need to be collapsed, or some raking
procedure must be used to smooth out the rough edges (Little & Rubin, 1987,
pp. 59–60).
17
Adjusting the Weight in a FollowUp Survey
Followup surveys are used quite often in social science research.
Unfortunately, it is not possible to follow up with all initial respondents.
Some may have died, moved, or refused to participate in the followup
survey. These events are not randomly distributed, and the differential attri
tion may introduce selection bias in the followup survey. We could make
an adjustment in the same manner as in the poststratiﬁcation, based on a
few demographic variables, but we can take advantage of a large number of
potential predictor variables available in the initial survey for the attrition
adjustment in the followup survey. The stratiﬁcation strategy may not be
well suited for a large number of predictors. The logistic regression model,
however, provides a way to include several predictors. Based on the initial
survey data, we can develop a logistic regression model for predicting the
attrition (binary variable) by a set of wellselected predictor variables from
the initial survey, in addition to usual demographic variables. The predicted
logit based on characteristics of respondents in the original sample can then
be used to adjust the initial weight for each respondent in the followup sur
vey. This procedure, in essence, can make up for those lost to the followup
by differentially inﬂating the weights of those successfully contacted and,
hence, removing the selection bias to the extent that it is related to the vari
ables in the model. As more appropriate variables are used in the model, the
attrition adjustment can be considered more effective than the nonresponse
adjustment in the initial survey. Poststratiﬁcation used to correct for non
response resulting from ‘‘lost to followup’’ (or for other types of nonresponse)
does not guarantee an improved estimate. However, if ‘‘lost to followup’’ is
related to the variables used in the adjustment process, the adjusted estimate
should be an improvement over the unadjusted estimate.
Let us look at an example from a community mental health survey (one site
in the ECA survey). Only 74.3% of initial respondents were successfully sur
veyed in the ﬁrst annual followup survey. This magnitude of attrition is too
large to be ignored. We therefore conducted a logistic regression analysis of
attrition (1 = lost; 0 = interviewed) on selected predictors shown in Table 3.3.
The chisquare value suggests that the variables in the model are signiﬁcantly
associated with attrition. The effect coding requires that one level of each vari
able be omitted, as shown in the table. Based on the estimated beta coefﬁcients,
^
the predicted logit ( λ _{i} , i = 1, 2,
...
, n) is calculated for each respondent. It is
^
then converted to the predicted proportion of attrition by p^ _{i} = 1/ð1 + e
−
^{λ} i ).
The initial weight for a successfully interviewed respondent is inﬂated by
dividing it by
ð
1 − p^
i Þ and setting the weight for a respondent lost to follow
up to zero. As shown in Table 3.3, the adjusted weight for the retained panel
members added up to 300,172, only 59 more than the sum of the weights in the
18
TABLE 3.3
Logistic Regression Model for Attrition
Adjustment in a FollowUp Survey
Logistic Regression Model _______________
Factors 
Category Variable Beta Coefficient SurveyRelated Information 

Intercept 
 
0.737* 
Initial survey 

Age 
1824 yrs 
AGE1 
0.196 
Design: 
Multistage 

Sampling 

2534 yrs 
AGE2 
0.052 
Sample size: 
4,967 

AGE3 
0.338* 
Weighted sum: 
300,113 

3544 yrs 4554 yrs 55 and over AGE4 — 
0.016 
Followup survey 

Marital status: Sep./divorced MAR 
0.051 
Sample size: 
3,690 

(74.3%) 

Other 
 
Sum of attrition 

Gender: 
Male 
SEX 
0.084 
adjusted weights: 
300,172 

Female 
— 
Adjusted sum: 
300,113 

Race 
White 
RACE1 
0.668* 

Black 
RACE2 
0.865* 
Comparison of attritionadjusted 

and Other estimates 
 
attritionunadjusted 

Socioeconomic 
1 ^{s}^{t} quartile 
SES1 
0.534* 

status 
2 ^{n}^{d} quartile 
SES2 
0.389* 
Disorders Unadjusted ^{a} Adjusted ^{b} 

Diff. 

3 ^{r}^{d} and 4 ^{t}^{h} 
 

Family size: 
One 
SIZE1 
0.033 
Any disorder 
39.2% 
43.8% 

4.6% 

24 members 
SIZE2 
0.003 
Major depre. 
15.0 
18.1 

3.1 

5 or more 
 
Cog. Impair. 
1.8 
1.3 

0.5 

Diagnosis: 
Cog. Impair. 
DX1 
0.472* 
Phobias 
8.4 
7.6 

0.4 

Schizophrenia DX2 
0.049 
Alcohol abuse 13.9 
13.7 

4.2 

Antisocial 
DX3 
0.412* 
Schizophrenia 0.6 
0.6 

0.4 

Anorexia 
DX4 
2.283* 

No disorder 
— 
Likelihood ratio chisquare(16) = 198.0, p < 0.00001.
a. Based on the initial weights.
b. Based on attritionadjusted weights.
^{*} p < .05.
initial survey, suggesting that the procedure worked reasonably well. (If there
were a large discrepancy between the sum of the adjusted weights and the sum
of the weights in the initial survey, there would be a concern about the adjust
ment process.) The adjusted weights are readjusted to align to the initial sur
vey. To show the effect of using attritionadjusted weights, the prevalence
rates of six selected mental disorders were estimated with and without the attri
tionadjusted weights, as shown in Table 3.3. Using the adjusted weights, we
see that the prevalence of any disorders (DSMIII deﬁned disorders) is nearly
5 percentage points higher than the unadjusted prevalence.
Assessing the Loss or Gain in Precision: The Design Effect
As shown in the previous chapter, the variance of an SRSWOR sample
mean is the variance of an SRSWR sample mean times the ﬁnite population
19
correction factor (1 − f ). The ratio of the sampling variance of SRSWOR to
the sampling variance of SRSWR is then (1 − f ), which reﬂects the effect
of using SRSWOR compared to using SRSWR. This ratio comparing the
variance of some statistic from any particular design to that of SRSWR is
called the design effect for that statistic. It is used to assess the loss or gain
in precision of sample estimates from the design used, compared to a
SRSWR design. A design effect less than one indicates that fewer observa
tions are needed to achieve the same precision as SRSWR, whereas a design
effect greater than one indicates that more observations are needed to yield
the same precision. In the case of SRSWOR, the design effect is less than
one, but it is close to one when the sampling fraction is very small. Because
SRSWOR is customarily used in place of SRSWR, researchers have tended
to base the design effect calculation on SRSWOR instead of SRSWR. In
addition, in complex surveys the design effect is usually calculated based
on the variance of the weighted statistic under the SRSWOR design. We
shall do that throughout the rest of the book.
Relating this notion of the design effect to the sample size, the effective
sample size can be deﬁned to be the actual sample size divided by the design
effect. If the design effect is greater than one for a sample design, then, in
effect the sample size would be reduced for a statistical analysis, thus leading
to a larger sampling error. In other words, when the design effect is greater
than one, the effective sample size is smaller than the actual size.
Let us examine the design effect in more complex sample designs. The
properties and estimation of the sampling error for stratiﬁed random sam
pling are well known, and so are the conditions under which stratiﬁcation
will produce a smaller variance than SRS. However, stratiﬁcation often
is used in conjunction with other desig n features, such as cluster sam
pling in several stages within the str ata. As discussed e arlier, clustering
tends to increase the sampling error. The effect of stratiﬁcation can be
diluted by the effect of clustering in many practical designs. Unfortu
nately, the assessment of the design e ffect cannot be dete rmined theoreti
cally from properties of stratiﬁcation and clustering separately, but
instead must be approximated numer ically to account for their combined
effects.
The following example demonstrates the determination of the design
effect in a relatively simple situation. Consider the case of singlestage clus
ter sampling in which all clusters are of the same size. Suppose that there are
N English classes (clusters) in a high school, with M students in each class.
From the N classes, n classes are selected by SRS, and all the students in the
chosen clusters are asked to report the number of books they have read since
the beginning of the year. The number of students is NM in the popula
tion and nM in the sample. The sampling fraction is f = nM/NM = n/N.
20
Because the class sizes are equal, the average number of books read per
student (population mean) is the mean of the N class means. The n sample
classes can be viewed as a random sample of n means from a population of
N means. Therefore, the sample mean ( y ) is unbiased for the population
mean ( Y ), and its variance, applying Equation 2.1, is given by
^
V
ðy
Þ =
s
2
b
n
ð1 − f Þ,
(3:3)
where s
2
b
= ^{} ð
y − y
i
Þ ^{2} /ðn − 1Þ, the estimated variance of the cluster
means. Alternately, Equation 3.3 can be expressed in terms of estimated
ICC (ρ^ ) as follows (Cochran, 1977, chap. 9):
^
V ( ) =
y
s ^{2} [1 + (M − 1)ρ^ ]
nM
(1 − f ),
(3:4)
where s ^{2} = ^{}^{} (y _{i}_{j} − y ) ^{2} /ðnM − 1Þ, the variance of elements in the sam
ple. If this is divided by the variance of the mean from an SRSWOR sample
^
of size nM, [ V ð y Þ = _{n}_{M} ð1 − f Þ, applying Equation 2.1], then the design
s ^{2}
effect of the cluster sample is 1 + ðM − 1Þρ^ .
When ICC = 0, the design effect will be one; when ICC > 0, the design
effect will be greater than one. If the clusters were formed at random, then
ICC = 0; when all the elements within each cluster have the same value,
ICC = 1. Most clusters used in community surveys consist of houses in the
same area, and these yield positive ICCs for many survey variables. The
ICC is usually larger for socioeconomic variables than for the demographic
variables, such as age and sex.
The assessment of the design effect for a more complex sample design is
not a routine task that can be performed using the formulas in statistics text
books; rather, it requires special techniques that utilize unfamiliar strategies.
The next chapter reviews several strategies for estimating the sampling var
iance for statistics from complex surveys and examines the design effect
from several surveys.
The Use of Sample Weights for Survey Data Analysis ^{*}
As discussed above, sample weights are used when computing point
estimates. All point estimates take the form of weighted statistics. This use
of sample weights makes sense from the above discussion on development
and adjustment of sample weights, especially for a descriptive analysis.
However, the use of sample weights in analytical studies is not as clear as in
descriptive analysis. As discussed in the last section of Chapter 2, there are
two different points of view regarding survey data. From the designbased
21
position, the use of sample weights is essential in both descriptive and
analytical studies. Inferences are based on repeated sampling for the ﬁnite
population, and the probability structure used for inference is that deﬁned
by the random variables indicating inclusion in the sample. From the
modelbased perspective, however, it can be argued that the sample selec
tion scheme is irrelevant when making inferences under a speciﬁed model.
If the observations in the population really follow the model, then the
sample design should have no effect as long as the probability of selection
depends on the modelspeciﬁed dependent variable only through the inde
pendent variables included in the model. Conditions under which the sam
pling scheme is ignorable for inference have been explored extensively
(Nordberg, 1989; Sugden & Smith, 1984). Many survey statisticians have
debated these two points of view since the early 1970s (Brewer, 1999;
Brewer & Mellor, 1973; Graubard & Korn, 1996, 2002; Hansen, Madow, &
Tepping, 1983; Korn & Graubard, 1995a; Pfeffermann, 1996; Royall, 1970;
Sarndal, 1978; T. M. F. Smith, 1976, 1983).
A good analysis of survey data would require general understanding of
both points of view as well as consideration of some practical issues, espe
cially in social surveys. The modelbased approach is consistent with the
increasing use of modelbased inference in other areas of statistical analy
sis, and it provides some theoretical advantages. Modelbased estimates
can be used with relatively small samples and even with nonprobability
samples. In addition, modelbased analysis can be done using standard sta
tistical software such as SAS and SPSS without relying on survey packages
such as SUDAAN and others that are reviewed in this book. The model
based approach, however, assumes that the model correctly describes the
true state of nature. If the model is misspeciﬁed, then the analysis would be
biased and lead to misinterpretation of the data. Unfortunately, theoretically
derived models for all observations in the population seldom exist in social
survey situations. In addition, omission of relevant variables in the model
would be a real concern in secondary analysis of survey data because not all
relevant variables are available to the analysts. Thus, the major challenge to
modelbased inference is specifying a correct model for the purpose of the
analysis.
It has been recognized that a weighted analysis is heavily inﬂuenced
by observations with extremely large weights (that may often be a result
of nonresponse and poststratiﬁcation adjustments rather than the selection
probabilities). Another recognized limitation of weighting is the increase in
variance. In general, the increase is high when the variability of the weights
is large. There is something to lose in using weighted analysis when it is
actually unnecessary for bias reduction, and the weighted analysis will be
inefﬁcient as compared with the unweighted analysis. Korn and Graubard
22
(1999, chap. 4) discuss various issues dealing with weighted and
unweighted estimates of population parameters and offer a measure of the
inefﬁciency of using weighted estimates. They recommend using the
weighted analysis if the inefﬁciency is not unacceptably large, to avoid
the bias in the unweighted analysis. If the inefﬁciency is unacceptably
large, they recommend using the unweighted analysis, augmenting the
model with survey design variables, including the weight, to reduce the
bias. Incorporation of design variables into the model is often problematic,
however, because the inclusion of the design variables as additional covari
ates in the model may contradict the scientiﬁc purpose of the analysis. For
example, when the objective of the analysis is to examine associations
between health measures and risk factors, conditioning on the design vari
ables may interfere with the relational pathway.
In complex largescale surveys, it is often not possible to include in the
model all the design information, especially when the sample weights are
modiﬁed for nonresponse and poststratiﬁcation adjustments (Alexander,
1987). Another practical problem in incorporating the design variables into
the model is the lack of relevant information in the data set. Not all design
related variables are available to the analysts. Most publicuse survey data
provide only PSU (the primary sampling unit), leaving out secondary clus
ter units (census enumeration districts or telephone exchanges). Often, pro
vision of secondary units may not be possible because of conﬁdentiality
issues.
In a modelbased analysis, one must guard against possible misspeciﬁca
tion of the model and possible omission of covariates. The use of sample
weights (designbased analysis) can provide protection against model mis
speciﬁcation (DuMouchel & Duncan, 1983; Pfeffermann & Homes, 1985).
Kott (1991) points out that the sampling weights need to be used in linear
regression because the choice of covariates in survey data is limited in most
secondary analyses. The merits and demerits of using sample weights will
be further discussed in the last section of Chapter 6.
4. STRATEGIES FOR VARIANCE ESTIMATION
The estimation of the variance of a survey statistic is complicated not only
by the complexity of the sample design, as seen in the previous chapters, but
also by the form of the statistic. Even with an SRS design, the variance esti
mation of some statistics requires nonstandard estimating techniques. For
example, the variance of the median is conspicuously absent in the standard
texts, and the sampling error of a ratio estimator (refer again to Note 1) is com
plicated because both the numerator and denominator are random variables.
23
Certain varianceestimating techniques not found in standard textbooks are
sufﬁciently ﬂexible to accommodate both the complexities of the sample
design and the various forms of statistics. These general techniques for var
iance estimation, to be reviewed in this chapter, include replicated sampling,
balanced repeated replication (BRR), jackkniferepeated replication (JRR),
the bootstrap method, and the Taylor series method.
Replicated Sampling: A General Approach
The essence of this strategy is to facilitate the variance calculation by
selecting a set of replicated subsamples instead of a single sample. It requires
that each subsample be drawn independently using an identical sample selec
tion design. Then an estimate is made in each subsample by the identical
process, and the sampling variance of the overall estimate (based on all
subsamples) can be estimated from the variability of these independent sub
sample estimates. This is the same idea as the repeated systematic sampling
mentioned in Chapter 2.
The sampling variance of the mean (u)
of
t
replicate estimates u _{1} ,
u _{2} ,
...
, u _{t} of the parameter U can be estimated by the following simple
variance estimator (Kalton, 1983, p. 51):
v(u) = ^{} (u _{i} − u) ^{2} /t(t − 1) (4:1)
This estimator can be applied to any sample statistic obtained from
independent replicates of any sample design.
In applying this variance estimator, 10 replicates are recommended by
Deming (1960), and a minimum of 4 by others (Sudman, 1976) for descrip
tive statistics. An approximate estimate of the standard error can be calcu
lated by dividing the range in the replicate estimates by the number of
replicates when the number of replicates is between 3 and 13 (Kish, 1965,
p. 620). However, because this variance estimator with t replicates is based
on (t − 1) degrees of freedom for statistical inference, a larger number of
replicates may be needed for analytic studies, perhaps 20 to 30 (Kalton,
1983, p. 52).
To understand the replicated design strategy, let us consider a simple
example. Suppose we want to estimate the proportion of boys among 200
newly born babies. We will simulate this survey using the random digits
from Cochran’s book (1977, p. 19), assuming the odd numbers represent
boys. The sample is selected in 10 replicate samples of n = 20 from the
ﬁrst 10 columns of the table. The numbers of boys in the replicates are as
follows:
24
Replicate: 
9 
8 
13 
12 
14 
8 
10 
7 
10 
8 
Total = 99 
Proportion 

of Boys: 
.45 
.40 
.65 
.60 
.70 
.40 
.50 
.35 
.50 
.40 
Proportion = .495 
The overall percentage of boys is 49.5%, and its standard error is 3.54%
(¼
p ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
49:5 ^{∗} 50:5/200 ). The standard error estimated from the 10 replicate
estimates using Equation 4.1 is 3.58%. It is easy to get an approximate
estimate of 3.50% by taking one tenth of the range (70%–35%). The chief
advantage of replication is ease in estimation of the standard errors.
In practice, the fundamental principle of selecting independent replicates
is somewhat relaxed. For one thing, replicates are selected using sampling
without replacement instead of with replacement. For unequal probability
designs, the calculation of basic weights and the adjustment for nonresponse
and poststratiﬁcation usually are performed only once for the full sample,
rather than separately within each replicate. In cluster sampling, the replicates
often are formed by systematically assigning the clusters to the t replicates in
the same order that the clusters were ﬁrst selected, to take advantage of strati
ﬁcation effects. In applying Equation 4.1, the sample mean from the full
sample generally is used for the mean of the replicate means. These deviations
from fundamental principles can affect the variance estimation, but the bias is
thought to be insigniﬁcant in largescale surveys (Wolter, 1985, pp. 83–85).
The community mental health survey conducted in New Haven, Connec
ticut, in 1984 as part of the ECA Survey of the National Institute of Mental
Health (E. S. Lee, Forthofer, Holzer, & Taube, 1986) provides an example
of replicated sampling. The sampling frame for this survey was a geogra
phically ordered list of residential electric hookups. A systematic sample
was drawn by taking two housing units as a cluster, with an interval of 61
houses, using a starting point chosen at random. A string of clusters in the
sample was then sequentially allocated to 12 subsamples. These subsam
ples were created to facilitate the scheduling and interim analysis of data
during a long period of screening and interviewing. Ten of the subsamples
were used for the community survey, with the remaining two reserved for
another study. The 10 replicates are used to illustrate the variance estimation
procedure.
These subsamples did not strictly adhere to a fundamental principle of
independent replicated sampling because the starting points were systema
tically selected, except for the ﬁrst random starting point. However, the sys
tematic allocation of clusters to subsamples in this case introduced an
approximate stratiﬁcation leading to more stable variance estimation and,
25
TABLE 4.1
Estimation of Standard Errors From Replicates:
ECA Survey in New Haven, 1984 (n = 3,058)
Regression Coefﬁcients ^{a} 

Replicate 
Prevalence Rate ^{b} 
Odds Ratio ^{c} 
Intercept 
Gender 
Color 
Age 
Full Sample 
17.17 
0.990 
0.2237 
−0.0081 
0.0185 
−0.0020 
1 
12.81 
0.826 
0.2114 
0.0228 
0.0155 
−0.0020 
2 
17.37 
0.844 
0.2581 
0.0220 
0.0113 
−0.0027 
3 
17.87 
1.057 
0.2426 
−0.0005 
0.0393 
−0.0015 
4 
17.64 
0.638 
0.1894 
0.0600 
0.2842 
−0.0029 
5 
16.65 
0.728 
0.1499 
0.0448 
−0.0242 
−0.0012 
6 
18.17 
1.027 
0.2078 
−0.0024 
−0.0030 
−0.0005 
7 
14.69 
1.598 
0.3528 
−0.0487 
−0.0860 
−0.0028 
8 
17.93 
1.300 
0.3736 
−0.0333 
−0.0629 
−0.0032 
9 
17.86 
0.923 
0.2328 
−0.0038 
0.0751 
−0.0015 
10 
18.91 
1.111 
0.3008 
−0.0007 
0.0660 
−0.0043 
Range 
6.10 
0.960 
0.2237 
0.1087 
0.3702 
0.0038 
Standard error based on: 

Replicates 
0.59 
0.090 
0.0234 
0.0104 
0.0324 
0.0004 
SRS 
0.68 
0.097 
0.0228 
0.0141 
0.0263 
0.0004 
SOURCE: Adapted from ‘‘Complex Survey Data Analysis: Estimation of Standard Errors Using Pseudo Strata,’’ E. S. Lee, Forthofer, Holzer, and Taube, Journal of Economic and Social Measurement, copyright 1986 by the Journal of Economic and Social Measurement. Adapted with permission.
a. The dependent variable (coded as 1 = condition present and 0 = condition absent) is regressed on sex
(1 = male, 0 = female), color (1 = black, 0 = nonblack), and age (continuous variable). This analysis is used for demonstration only.
b. Percentage with any mental disorders during the last 6 months.
c. Sex difference in the 6month prevalence rate.
therefore, may be preferable to a random selection of a starting point for this
relatively small number of replicates. Therefore, we considered these sub
samples as replicates and applied the variance estimator with replicated
sampling, Equation 4.1.
Because one adult was randomly selected from each sampled household
using the Kish selection table (Kish, 1949), the number of adults in each
household became the sample case weight for each observation. This
weight was then adjusted for nonresponse and poststratiﬁcation. Sample
weights were developed for the full sample, not separately within each
subsample, and these were the weights used in the analysis.
Table 4.1 shows three types of statistics calculated for the full sample as
well as for each of the replicates. The estimated variance of the prevalence
26
rate, in percent (p), can be calculated from the replicate estimates (p _{i} )
using Equation 4.1:
v(p) =
^{} (p _{i} − 17:17) ^{2}
10(10 − 1)
= 0:3474,
and the standard error is
p ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
0:3474 = 0:59. The overall prevalence rate of
17.17% is slightly different from the mean of the 10 replicate estimates
because of the differences in respon se rates. Note that one tenth of the
range in the replicate estimates (0.61) approximates the standard error
obtained by Equation 4.1. Similarly, standard errors can be estimated for
the odds ratio and regression coefﬁci ents. The estimated standard errors
have approximately the same values as those calculated by assuming
simple random sampling (using appropriate formulas from textbooks).
This indicates that desig n effects are fairly small for these statistics from
this survey.
Although the replicated sampling design provides a variance estimator that
is simple to calculate, a sufﬁcient number of replicates are required to obtain
acceptable precision for statistical inference. But if there is a large number of
replicates and each replicate is relatively small, it severely limits the use of
stratiﬁcation in each replicate. Most important, it is impractical to implement
replicated sampling in complex sample designs. For these reasons, a repli
cated design is seldom used in largescale, analytic surveys. Instead, the
replicated sampling idea has been applied to estimate variance in the data
analysis stage. This attempt gave rise to pseudoreplication methods for
variance estimation. The next two techniques are based on this idea of
pseudoreplication.
Balanced Repeated Replication
The balanced repeated replication (BRR) method is based on the
application of the replicated sampling idea to a paired selection design in
which two PSUs are sampled from each stratum. The paired selection design
represents the maximum use of stratiﬁcation and yet allows the calculation
of variance. In this case, the variance between two units is one half of the
squared difference between them. To apply the replicated sampling idea, we
ﬁrst divide the sample into random groups to form pseudoreplicates. If it is a
stratiﬁed design, it requires all the strata to be represented in each pseudo
replicate. In a stratiﬁed, paired selection design, we can form only two
pseudoreplicates: one containing one of the two units from each stratum and
the other containing the remaining unit from each stratum (complement repli
cate). Each pseudoreplicate then includes approximately half of the total
27
sample. Applying Equation 4.1 with t = 2, we can estimate the sampling
variance of the mean of the two replicate estimates, u ^{} , u ^{}^{} , by
v(u) = [(u ^{} − u) ^{2} + (u ^{}^{} − u) ^{2} ]/2: (4:2)
As seen above, the mean of replicate estimates is often replaced by an
overall estimate obtained from the full sample. However, this estimator is
too unstable to have any practical value because it is based on only two
pseudoreplicates. The BRR method solves this problem by repeating the
process of forming halfsample replicates, selecting different units from dif
ferent strata. The pseudoreplicated half samples then contain some common
units, and this introduces dependence between replicates, which complicates
the estimation. One solution, which leads to unbiased estimates of variance
for linear statistics, is to balance the formation of pseudoreplicates by using
an orthogonal matrix (Plackett & Burman, 1946). The full balancing requires
that the size of the matrix be a multiple of four and the number of replicates
be greater than or equal to the number of strata. Then the sampling variance
of a sample statistic can be estimated by taking the average of variance
estimates by Equation 4.2 over t pseudoreplicates:
v(u)
= ^{} [(u _{i} ^{} − u)
^{2} + (u ^{}^{} − u)
i
^{2} ]/2t = ^{} (u _{i} ^{}^{} − u
i
) ^{2} /4t :
(4:3)
It is possible to reduce computation by dropping the complement
halfsample replicates:
v ^{} (u) = ^{} (u ^{} _{i} − u) ^{2} /t :
(4:4)
This is the estimator originally proposed by McCarthy (1966). This
balancing was shown by McCarthy to yield unbiased estimates of var
iance for linear estimators. For nonli near estimators, there is a bias in the
estimates of variance, but numerical st udies suggest that it is small. For a
large number of strata, the computation can be further simpliﬁed by using
a smaller set of partially balanced replicates (K. H. Lee, 1972; Wolter,
1985, pp. 125–130).
As in replicated sampling, BRR assumes that the PSUs are sampled with
replacement within strata, although in practice sampling without replacement
generally is used. Theoretically, this leads to an overestimation of variance
when applied to a sample selected without replacement, but the overestima
tion is negligible in practice because the chance of selecting the same unit
more than once under sampling without replacement is low when the sam
pling fraction is small. The sampling fraction in a paired selection design
(assumed in the BRR method) usually is small because only two PSUs are
selected from each stratum.
28
When used with a multistage selection design, BRR usually is applied
only to PSUs and disregards the subsampling within the PSUs. Such a prac
tice is predicated on the fact that the sampling variance can be approximated
adequately from the variation between PSU totals when the ﬁrststage sam
pling fraction is small. This is known as the ultimate cluster approximation.
As shown in Kalton (1983, chap. 5), the unbiased variance estimator for a
simple twostage selection design consists of a component from each of
the two stages, but the term for the secondstage component is multiplied by
the ﬁrststage sampling fraction. Therefore, the secondstage contribution
becomes negligible as the ﬁrststage sampling fraction decreases. This short
cut procedure based only on PSUs is especially convenient in the preparation
of complex data ﬁles for public use as well as in the analysis of such data,
because detailed information on complex design features is not required
except for the ﬁrststage sampling.
If the BRR technique is to be applied to other than the paired selection
designs, it is necessary to modify the data structure to conform to the tech
nique. In many multistage surveys, stratiﬁcation is carried out to a maximum
and only one PSU is selected from each stratum. In such case, PSUs can
be paired to form collapsed strata to apply the BRR method. This procedure
generally leads to some overestimation of the variance because some of the
betweenstrata variability is now included in the withinstratum calculation.
The problem is not serious for the case of linear statistics if the collapsing is
carried out judiciously; however, the collapsing generally is not recommended
for estimating the variance of nonlinear statistics (see Wolter, 1985, p. 48). The
Taylor series approximation method discussed later may be used for the non
linear statistics. Although it is not used widely, there is a method of construct
ing orthogonal balancing for three PSUs per stratum (Gurney & Jewett, 1975).
Now let us apply the BRR technique to the 1984 GSS. As introduced in
the previous chapter, it used a multistage selection design. The ﬁrststage
sampling consisted of selecting one PSU from each of 84 strata of counties
or county groups. The ﬁrst 16 strata were large metropolitan areas and
designated as selfrepresenting (or automatically included in the sample).
To use the BRR technique, the 84 strata are collapsed into 42 pairs of
pseudostrata. Because the numbering of non–selfrepresenting PSUs in the
data ﬁle approximately followed the geographic ordering of strata, pairing
was done sequentially, based on the PSU code. Thus, the 16 selfrepresenting
strata were collapsed into 8 pseudostrata, and the remaining 68 non–self
representing strata into 34 pseudostrata. This pairing of the selfrepresenting
strata, however, improperly includes variability among them. To exclude this
and include only the variability within each of the selfrepresenting strata, the
combined observations within each selfrepresenting pseudostratum were
randomly grouped into two pseudoPSUs.
29
To balance the halfsample repli cates to be generated from the 42
pseudostrata, an orthogonal matrix of order 44 (see Table 4.2) was used.
This matrix is ﬁlled with zeros and ones. To match with the 42 strata, the
ﬁrst two columns were dropped (i.e., 44 rows for replicates and 42 columns
for pseudostrata). A zero indicates the inclusion of the ﬁrst PSU from the
strata, and a one denotes the inclusion of the second PSU. The rows are the
replicates, and the columns represent the strata. For example, the ﬁrst repli
cate contains the second PSU from each of the 42 pseudostrata (because all
the elements in the ﬁrst row are ones). Using the rows of the orthogonal
matrix, 44 replicates and 44 complement replicates were created.
To estimate the variance of a statistic from the full sample, we needed
ﬁrst to calculate the statistic of inte rest from each of the 44 replicates and
complement replicates. In calculating the replicate estimates, the adjusted
sample weights were used. Table 4.3 shows the estimates of the propor
tion of adults approving the ‘‘hitting’’ for the 44 replicates and their
complement replicates. The overall proportion was 60.0%. The samp
ling variance of the overall proportion, estimated by Equation 4.3, is
0.000231. Comparing this with the sampling variance of the proportion
under the SRS design [pq/( n − 1) = 0.000163, ignoring FPC], we get the
design effect of 1.42 (= 0.000231/0.000163). The design effect indicates
that the variance of the estimated proportion from the GSS is 42% larger
than the variance calculated from an SRS of the same size. The variance
by Equation 4.4 also gives similar estimates.
In summary, the BRR technique uses a pseudoreplication procedure to
estimate the sampling variance and is primarily designed for a paired selec
tion design. It also can be applied to a complex survey, which selects one
PSU per stratum by pairing strata, but the pairing must be performed judi
ciously, taking into account the actual sample selection procedure. In most
applications of BRR in the available software packages, the sample weights
of the observations in the selected PSUs for a replicate are doubled to make
up for the half of PSUs not selected. There is also a variant of BRR, sug
gested by Fay (Judkins, 1990), in creating replicate weights, which uses
2 − k or k times the original weight, depending on whether the PSU is
selected or not selected based on the orthogonal matrix (0 ≤ k < 1). This
will be illustrated further in the next chapter.
Jackknife Repeated Replication
The idea of jackkniﬁng was introduced by Quenouille (1949) as a
nonparametric procedure to estimate the bias, and later Tukey (1958) sug
gested how that same procedure could be used to estimate variance. Durbin
(1959) ﬁrst used this method in his pioneering work on ratio estimation. Later,
30
TABLE 4.2
Orthogonal Matrix of Order 44
Rows
Columns (44)
1 11111111111111111111111111111111111111111111
2 10100101001110111110001011100000100011010110
3 10010010100111011111000101110000010001101011
4 11001001010011101111100010111000001000110101
5 11100100101001110111110001011100000100011010
6 10110010010100111011111000101110000010001101
7 11011001001010011101111100010111000001000110
8 10101100100101001110111110001011100000100011
9 11010110010010100111011111000101110000010001
10 11101011001001010011101111100010111000001000
11 10110101100100101001110111110001011100000100
12 10011010110010010100111011111000101110000010
13 10001101011001001010011101111100010111000001
14 11000110101100100101001110111110001011100000
15 10100011010110010010100111011111000101110000
16 10010001101011001001010011101111100010111000
17 10001000110101100100101001110111110001011100
18 10000100011010110010010100111011111000101110
19 10000010001101011001001010011101111100010111
20 11000001000110101100100101001110111110001011
21 11100000100011010110010010100111011111000101
22 11110000010001101011001001010011101111100010
23 10111000001000110101100100101001110111110001
24 11011100000100011010110010010100111011111000
25 10101110000010001101011001001010011101111100
26 10010111000001000110101100100101001110111110
27 10001011100000100011010110010010100111011111
28 11000101110000010001101011001001010011101111
29 11100010111000001000110101100100101001110111
30 11110001011100000100011010110010010100111011
31 11111000101110000010001101011001001010011101
32 11111100010111000001000110101100100101001110
33 10111110001011100000100011010110010010100111
34 11011111000101110000010001101011001001010011
35 11101111100010111000001000110101100100101001
36 11110111110001011100000100011010110010010100
37 10111011111000101110000010001101011001001010
38 10011101111100010111000001000110101100100101
39 11001110111110001011100000100011010110010010
40 10100111011111000101110000010001101011001001
41 11010011101111100010111000001000110101100100
42 10101001110111110001011100000100011010110010
43 10010100111011111000101110000010001101011001
44 11001010011101111100010111000001000110101100
SOURCE: Adapted from Wolter (1985, p. 328) with permission of the publisher.
31
TABLE 4.3
Estimated Proportions Approving One Adult Hitting Another in the
BRR Replicates: General Social Survey, 1984 (n = 1,473)
Estimate (%) 
Estimate (%) 

Replicate Number 
Replicate 
Complement 
Replicate Number 
Replicate 
Complement 

1 
60.9 
59.2 
23 
61.4 
58.6 

2 
60.1 
59.9 
24 
57.7 
62.4 

3 
62.1 
57.9 
25 
60.4 
59.6 

4 
58.5 
61.7 
26 
61.7 
58.2 

5 
59.0 
61.0 
27 
59.3 
60.6 

6 
59.8 
60.2 
28 
62.4 
57.6 

7 
58.5 
61.5 
29 
61.0 
58.9 

8 
59.0 
61.0 
30 
61.2 