Professional Documents
Culture Documents
PA RT I V N E W F RON T I E R S
21. Survey Experiments: Managing the Methodological Costs
and Benefits 483
Yanna Krupnikov and Blake Findley
22. Using Qualitative Methods in a Quantitative Survey Research
Agenda 505
Kinsey Gimbel and Jocelyn Newsome
23. Integration of Contextual Data: Opportunities and Challenges 533
Armando Razo
24. Measuring Public Opinion with Social Media Data 555
Marko Klašnja, Pablo Barberá, Nicholas Beauchamp,
Jonathan Nagler, and Joshua A. Tucker
25. Expert Surveys as a Measurement Tool: Challenges and
New Frontiers 583
Cherie D. Maestas
26. The Rise of Poll Aggregation and Election Forecasting 609
Natalie Jackson
Index 633
Contributors
Alex N. Adams is a PhD student in the Department of Political Science at the University
of New Mexico. His research interests focus on political psychology and survey
methodology.
Prakash Adhikari is an Associate Professor of Political Science at Central Michigan
University. His research and teaching interests lie at the intersection of comparative pol-
itics and international relations, with specific focus on civil war, forced migration, and
transitional justice.
R. Michael Alvarez is a Professor in the Division of Humanities and Social Sciences at
the California Institute of Technology. His primary research interests are public opinion
and voting behavior, election technology and administration, electoral politics, and sta-
tistical and computer modeling.
Stephen Ansolabehere is the Frank G. Thompson Professor of Government at Harvard
University where he studies elections, democracy, and the mass media. He is a Principal
Investigator of the Cooperative Congressional Election Study, and his principal areas
are electoral politics, representation, and public opinion.
Lonna Rae Atkeson is a Professor and Regents Lecturer in the Department of Political
Science at the University of New Mexico where she directs the Institute for Social
Research and the Center for the Study of Voting, Elections and Democracy. Her primary
interests are the areas of survey methodology, election science and administration, and
political behavior.
Pablo Barberá is an Assistant Professor of Computational Social Science in the
Methodology Department at the London School of Economics. His primary areas of re-
search include social media and politics, computational social science, and comparative
electoral behavior and political representation.
Nicholas Beauchamp is an Assistant Professor of Political Science at Northeastern
University. He specializes in U.S. politics (political behavior, campaigns, opinion, polit-
ical psychology, and social media) and political methodology (quantitative text analysis,
machine learning, Bayesian methods, agent-based models, and networks).
Lindsay J. Benstead is an Associate Professor of Political Science in the Mark O. Hatfield
School of Government and Interim Director of the Middle East Studies Center (MESC)
at Portland State University, Contributing Scholar in the Women’s Rights in the Middle
x Contributors
East Program at Rice University, and Affiliated Scholar in the Program on Governance
and Local Development (GLD) at the University of Gothenburg and Yale University. Her
research interests include survey methodology and the Middle East-North Africa region.
Justin A. Berry is an Assistant Professor in the Department of Political Science at
Kalamazoo College. His research and teaching interests include American politics, po-
litical attitudes & behavior, race & ethnic politics, public opinion, immigration policy,
education policy, social movements, and methodology & research design.
Paul Brace is the Clarence L. Carter Professor of Political Science at Rice University. His
areas of interest include state and intergovernmental politics, judicial decision making,
and the presidency.
Lisa A. Bryant is an Assistant Professor at California State University, Fresno. Her
teaching and research interests include political behavior and voter behavior, campaigns
and elections, election administration, public opinion, the media, political psychology,
state politics, gender politics, and political methodology, focusing on experimental and
survey research methods.
Youssef Chouhoud is a PhD student at the University of Southern California in Political
Science & International Relations. His research interests include comparative democ-
ratization, political tolerance, Middle East politics, and Muslim minorities in the West.
Blake Findley is a PhD student in the Department of Political Science at Stony Brook
University. He does research in political psychology, political communication, and po-
litical methodology.
Andrew Gelman is the Higgins Professor of Statistics, Professor of Political Science, and
Director of the Applied Statistics Center at Columbia University. His research spans a
wide range of topics in statistics and social sciences, survey methodology, experimental
design, statistical inference, computation, and graphics.
Jeff Gill is a Distinguished Professor, Department of Government, Professor, Department
of Mathematics and Statistics, and member of the Center for Behavioral Neuroscience at
American University. His research applies Bayesian modeling and data analysis (decision
theory, testing, model selection, and elicited priors) to questions in general social science
quantitative methodology, political behavior and institutions, and medical/health data.
Kinsey Gimbel is Director of the Customer Experience Division at Fors Marsh Group.
Her primary areas of experience are qualitative research, survey design and administra-
tion, data analysis and reporting, and program evaluation.
James G. Gimpel is a Professor of Government at the University of Maryland. His
interests lie in the areas of political behavior, political socialization, and the political ge-
ography of American politics.
D. Sunshine Hillygus is a Professor of Political Science and Director of the Initiative on
Survey Methodology at Duke University. Her research and teaching specialties include
Contributors xi
public opinion, political behavior, survey research, campaigns and elections, and infor-
mation technology and society.
Jonathan Homola is an Assistant Professor at Rice University. He is a political method-
ologist and a comparativist. His substantive research interests include party competi-
tion, representation, political behavior, gender and politics, and immigration.
Natalie Jackson is a Survey Methodologist at JUST Capital with experience running
survey research programs in academic, media, and nonprofit settings. She was in charge
of the election forecasting models and poll aggregation at The Huffington Post during
the 2014 and 2016 election cycles. She has a PhD in political science and researches how
people form attitudes and respond to surveys, as well as how the survey process can
affect reported attitudes.
William G. Jacoby is a Professor in the Department of Political Science at Michigan
State University. His main professional interests are mass political behavior (public
opinion, political attitudes, and voting behavior) and quantitative methodology (meas-
urement theory, scaling methods, statistical graphics, and modern regression).
Jane Junn is a Professor of Political Science at the University of Southern California. She
is the author of five books on political participation and public opinion in the United
States. Her research focuses on political behavior, public opinion, racial and ethnic poli-
tics, the politics of immigration, gender and politics, and political identity.
Jeffrey A. Karp is a Professor of Political Science at Brunel University in London. He
specializes in public opinion, elections, and comparative political behavior.
Marko Klašnja is an Assistant Professor of Political Science at Georgetown University,
with the joint appointment in the Government Department and the Edmund A. Walsh
School of Foreign Service. He specializes in comparative politics, political behavior, and
political economy of democratic accountability.
Yanna Krupnikov is an Associate Professor in the Department of Political Science at
Stony Brook University. Her research and teaching focus on political psychology,
political communication, political persuasion, political behavior, and empirical
methodology.
Ines Levin is an Assistant Professor in the Department of Political Science at the
University of California, Irvine. Her research focuses on quantitative research methods
with substantive applications in the areas of elections, public opinion, and political
behavior.
Cherie D. Maestas is the Marshall A. Rauch Distinguished Professor of Political Science
in the Department of Political Science and Public Administration at the University
of North Carolina at Charlotte where she also directs the Public Policy Program. She
studies political communication, political psychology, risk attitudes, and legislative
responsiveness.
xii Contributors
P OL L I N G
A N D SU RV E Y
M E T HOD S
I n t rodu c t i on to P olling
a n d Su rvey Method s
Introduction
In recent years political polling has been in a state of visible crisis. Recent “polling
misses” have been well-publicized: the Brexit election, the peace agreement refer-
endum in Colombia, and the U.S. presidential election. In the first example, the Brexit
vote in the United Kingdom was a close call that missed its mark, while in Colombia
polls regarding a referendum on a peace deal that took more than seven years to pro-
duce suggested that 66% of eligible voters supported it. However, when the votes were
counted on election day the referendum failed by a very close margin, with 50.2% of
voters rejecting it.
In the United States another important miss was the failure of polls conducted in the
competitive battleground states to predict a Donald Trump presidential win at nearly
any point in the election. A recent report from the American Association of Public
Opinion Research (AAPOR) argued that while the national polls in 2016 were quite ac-
curate, the state-by-state polling in important battleground states suffered from meth-
odological issues that appear to account for much of their inaccuracy (AAPOR 2017).
Moreover, poll aggregators such as fivethirtyeight.com and the Huffington Post pro-
vided odds that Hillary Clinton would win by very safe margins. For example, the final
election odds from fivethirtyeight.com gave Clinton a 71% chance of winning the elec-
tion, the lowest percentage of any poll aggregator, and the Huffington Post gave Clinton
a 98% chance of winning the election, the highest of any poll aggregator.
These polling misses are highly consequential. Not only have they provided pundits,
media, and the public with misleading information, but by being so seemingly unreli-
able they may even make people skeptical and distrustful of polling in general. Because
of these highly visible “misses,” political polling has an image problem, as a recent
2 Lonna Rae Atkeson and R. Michael Alvarez
U.S. poll finding shows that only 37% of the public trusts public opinion polls a great deal
or a good amount.1
Election polling is an especially unique industry and academic enterprise because it
is one of the few social sciences in which predictions can be validated against outcomes,
therefore providing the opportunity to assess issues related to survey error. Although
comparing predictions to outcomes provides a sense of when polls are off track, there
are many places in the survey design in which errors can be introduced, and thus being
attentive to innovation and best practices in all aspects of design is critical for a reliable
and valid survey.
Problems with polling usually stem from a variety of factors, including issues with the
sampling frame and nonresponse bias. Because of these issues, and because of the many
complex designs, which often involve multiple modes, panels, or oversamples, there
may be unequal probabilities of respondent selection, variation in response rates across
subgroups, or departures from distributions on key demographic or other variables
within the data, such as party identification, which may result in a variety of postsurvey
adjustment weighting strategies. Indeed, pollsters today do a great deal of postsurvey
adjustment weighting to create data sets that are representative of the population under
study. While there is certainly a science to weighting data, methodological differences
in how data are statistically weighted can lead to different results and different predicted
winners.
For example, in an experiment during the 2016 election the same raw data set was
given to four different pollsters for postsurvey adjustments; the result was four different
election predictions, from Trump up one point to Clinton up four points.2 Another dif-
ficult problem for pollsters in an election environment is identifying likely voters. Yet
other problems may have to do with nonresponse bias, which may lead some types of
voters to refuse to participate in the poll. Shy respondents may cause problems for a
survey if, for example, they are associated with a particular candidate or particular issue
position.
In developed countries, changes in survey research over the last fifteen years have
been tumultuous. The growth of the Internet, the decline in household use of landlines,
and the dramatic increase in cell phone use has made it both easier and more difficult to
conduct surveys. While the “gold standard” for survey research has traditionally been
probability based sampling, today many polls and surveys use nonprobability designs,
such as opt-in Internet panels for online surveys. Furthermore, surveys that begin with a
random sample often have such low response rates (less than 10% is now very common)
that the quality and accuracy of inferences drawn from the resulting sample may be
problematic.
For general population studies, the increase in Internet surveys has also meant
that researchers are relying today more on respondent-driven surveys than on the
interviewer-driven designs that dominated the field in previous decades. The preva-
lence of Internet surveys has also led to a greater number of panel designs and to consid-
eration of unique issues that arise with panel data. Survey researchers are also relying on
many more modes and combining them more often.
Introduction to Polling and Survey Methods 3
In the developing world, in-person surveys are still the norm, but technology is
allowing the development of innovative new methodologies, such as the use of com-
puter assisted personal interview (CAPI) systems or Global Positioning System (GPS)
devices, both of which may improve survey quality and reduce total survey error. But
other issues abound in surveys conducted in many developing areas, in particular
survey coverage and the representativeness of many survey samples.
In addition, there are many new opportunities in the field and many new data sets.
Table 0.1 presents a list of all the academically collected and freely accessible data sets
discussed in this Handbook. The number of readily accessible data sets is impressive and
affords researchers the chance to answer new and old questions in different contexts. But
using these data sets also presents some challenges, in particular understanding how
complex survey designs affect how researchers use them. In addition to the wide range
of survey data readily available today, there are also innovations in using surveys to in-
terview experts, social media as public opinion data, poll aggregation, the integration of
qualitative methods with survey designs, and the expanded use of survey experiments.
Technological advances in computing and statistics have also provided new and better
methods to assess opinion in subnational contexts and have created opportunities for
better methods to estimate and use latent constructs. In addition, the art of displaying
data has advanced significantly, allowing researchers to use graphics to inform their
decision-making process during the survey and modeling process, as well as after the
fact in how the data are communicated to consumers.
These changes present new opportunities and challenges and make this Oxford
University Press Handbook on Polling and Survey Methods timely. Polls, of course, tend
to focus on a single question, and simple analysis of a substantive single question usually
relies on simple two-variable crosstabs with demographic variables, whereas surveys
focus on the answers to many questions in which a research design is often embedded.
The goals of the Handbook are to outline current best practices and highlight the
changing nature of the field in the way social scientists conduct surveys and analyze and
present survey data. The Handbook considers four broad areas of discovery: survey de-
sign, data collection, analysis and presentation, and new frontiers. Following is a discus-
sion of the main contributions and points of interest of each chapter.
Survey Design
The first section of the Handbook focuses on general survey methodology considera
tions. Because survey methodology is the study of the sources of error in surveys,
with the intention of limiting as many of those sources of error as possible to pro-
duce an accurate measure or true value of the social or political world, it begins with
an essay by Herbert F. Weisberg that explains the total survey error and total survey
quality approach. Survey error is the difference between what the actual survey pro-
cess produces and what should be obtained from it. Total survey error considers both
4 Lonna Rae Atkeson and R. Michael Alvarez
approach extends the total survey error approach to consider additional criteria, in-
cluding providing usable and quality data to the researcher.
The next several chapters consider various survey design issues related to the method
of data collection. Survey researchers often have to ask: What is the best method to
collect the data I need for my research project? Data collection methods come in two
basic forms, interviewer-administered surveys or self-administered surveys, but data
collection efforts must also consider the nature of the survey and whether it is cross-
sectional or longitudinal. Panel surveys interview the same respondent over time to
track attitudes and behavior, thus measuring individual-level changes in attitudes
and behavior, which cross-sectional surveys cannot easily assess. Hillygus and Snell
consider the unique challenges and opportunities related to using longitudinal or
panel designs, including the tension between continuity across panel waves and in-
novation, panel attrition, and potential measurement error related to panel condi-
tioning of respondents and seam bias. Both the Atkeson and Adams chapter and the
Ansolabehere and Schaffner chapter address issues related to survey mode. The former
chapter focuses on the advantages and disadvantages associated with using mixed
mode surveys, which have become increasingly popular. Mixed mode surveys are
those that involve mixtures of different contact and response modes. They pay par-
ticular attention to how the presence or absence of an interviewer influences survey
response, especially social desirability, and item nonresponse. Thus, they compare
mail/Internet surveys to in-person/telephone surveys across a variety of dimensions
and consider best practices. Ansolabehere and Schaffner focus their attention on the
quality of surveys that use opt-in online nonprobability survey panels, the Cooperative
Congressional Election Study (CCES), and compare that to traditional probability
samples.
Gimpel’s chapter considers the geographic distribution of respondents and how
context, characterized as a respondent’s location, influences attitudes and behavior.
Traditional sampling designs, for example, focus on strategies that allow researchers
to make inferences about the population, which often limit the geographical space in
which respondents are found. This tends to create small sample sizes that have limited
utility in helping to understand a primary interest of social scientists, how spatial context
influences opinion. Because sometimes social scientists are interested in representing
places and people, they need to consider a different sampling design; Gimpel’s chapter
identifies when and how one can sample for context.
Oberski considers another important aspect of survey design, question wording.
While many survey methodology textbooks discuss the “art” of writing questions,
Oberski takes a more systematic approach, arguing that by using experiments we can
better differentiate good or reliable survey questions from the bad and unreliable. To this
end, Saris et al. (2012) over many years built up a large question data set that estimated
the reliability and common method variance or quality of those questions, coded
characteristics of those questions that related to their quality, and predicted question
quality based on a meta-analysis. They then created a free Web-based application that
allows researchers to input questions and obtain an estimate of their quality. The bulk
6 Lonna Rae Atkeson and R. Michael Alvarez
of Oberski’s chapter focuses on explaining the Survey Quality Predictor (SQP) tool and
how researchers can use it to make question design a solid science and less of an art.
Data Collection
The Handbook’s next section begins with a discussion of postelection exit polling. Exit
polls offer the first look at who is voting, how they are voting, and why they are voting
that way; they also offer valuable insights into political behavior, especially vote choice.
These types of surveys have been part of our election landscape since 1967, and as new
modes of voting have developed, especially early and mail voting, exit polls have had to
be modified to ensure they accurately reflect voters. Salvanto’s chapter provides an over-
view of the history and value of exit polls and much needed information on how exit poll
operations are managed.
Many researchers are interested in studying the attitudes and behavior of hard-to-
reach populations. These individuals can be hard to reach for many different reasons.
For example, some groups of people may be hard to identify (e.g., protestors), or they
may be hard to locate, such as the LGBT community, which is a very small group whose
members live everywhere, so that finding them in the population can be difficult and
expensive. It might be hard to persuade some populations to participate, for example,
politicians or their staff or people engaging in socially undesirable or illegal activities.
The chapters by Adkihari and Bryant and by Berry, Chouhoud, and Junn both focus
on these difficult-to-locate populations. Adkihari and Bryant consider hard-to-reach
populations in international or developing contexts, while Berry et al. focus on low-in-
cidence populations in the United States. Adkihari and Bryant build their story around a
research design in Nepal that examined citizens who either fled their homes or decided to
stay during the Maoist insurgency between 1996 and 2006. To examine important theo-
retical questions related to internally displaced people (IDP), the study first had to iden-
tify displacement patterns so that a sample of both those who decided to stay and those
who fled could be drawn. The study also had to solve problems related to difficult terrain,
lack of infrastructure, low-education populations, and other factors to develop a strong
survey design. Berry, Choudhoun, and Junn, on the other hand, focus their chapter on
the United States and on low-incidence populations, who make up a relatively small pro-
portion of the public that could be characterized as new immigrants, racial or ethnic
minorities, religious minorities, or small populations that are relatively dispersed, such as
gays or lesbians. They outline a strategy that uses a tailored or targeted approach to cap-
ture these hard-to-reach populations. They consider various attributes of these groups,
such as whether the group is geographically concentrated or dispersed or the degree
of uniformity among its members, and how these attributes help to make good design
decisions related to sampling, making contact and gaining cooperation, and analysis.
Both chapters provide best practices, useful advice, and important considerations on suc-
cessfully interviewing hard-to-reach populations.
Introduction to Polling and Survey Methods 7
Seligson and Moreno’s chapter and Benstead’s chapter focus on issues related to
the developing world. Seligson and Moreno’s chapter looks at the introduction of the
CAPI systems as a quality control measure in face-to-face surveys in Latin America.
They argue that CAPI systems improve the quality of the data collected in-person by
eliminating many sources of error and allowing the researcher much more control of
the field process. Benstead examines data collection issues in the Arab world, which is
an often difficult and sometimes inhospitable environment for survey researchers and
for social scientists more generally. Over the past several decades a variety of public
opinion surveys from the Middle Eastern and North African regions have been made
available to researchers (e.g., the Arab Barometer, Afrobarometer), opening up new
opportunities for research in these understudied nations. Many of these nations are
more accessible to researchers than they were previously, and Benstead also considers
unique challenges researchers face when working in this region, as well as best practices
for survey researchers.
The chapter by Perez on the connection between language and opinion rounds out
the section on data collection. Given that there are so many public opinion surveys,
often asking the same questions in different languages across different cultures, Perez
asks what the connection between language and opinion is and how we can isolate
its effects. In particular, Perez highlights how cognitive psychology can assist us in
building theoretical models that help explain how and when language will influence
opinion.
The next set of essays begins with a chapter by Gill and Homola, who discuss a variety
of issues related to statistical inference and hypothesis testing using survey data. They
highlight several methodological concerns regarding transparency of data, uncertainty
in the process, the margin of error, and significance testing. Levin and Sinclair examine
how including or excluding survey weights affects various matching algorithms. They
find that weights are important to make accurate causal inferences from complex survey
data. Their chapter demonstrates the need to account for characteristics of the sample to
make population-based inferences.
The next chapter, by Brace, is interested in the study of subnational public opinion.
Accurate and reliable measurement of subnational public opinion is especially valu-
able when researchers are interested in understanding how context, or the political
and social environment, influences opinion, and how opinion influences government
outcomes. One of the many problems with looking at these types of questions is that
there is very little systematic comparative analysis across states, congressional districts,
legislative districts, counties, or cities. Surveys at the subnational level are fairly unique
and are conducted by different polling organizations at different times, using different
methodologies and question wording. Brace discusses the history of this field and the
8 Lonna Rae Atkeson and R. Michael Alvarez
development of various tools and methods to disaggregate national opinion polls to the
subnational level to produce reliable estimates of subnational opinion.
Usually researchers are interested in abstract concepts such as political knowledge,
ideology, and polarization. But these are often measured with single variables that
possess a large quantity of measurement error. Chris Warshaw discusses the value of
latent constructs, the various ways latent constructs have been identified, and new
methodologies that are available for testing latent constructs. Proctor’s chapter follows
with a focus on the application of item response theory to the study of group conscious-
ness. She demonstrates how latent constructs help to clarify the role group conscious-
ness plays in understanding political behavior, using a study of the LGBT community,
and how some of the measurement assumptions underlying group consciousness are
incorrect.
Next, in their chapter Karp and Vowles examine the challenges and opportunities in-
herent in comparative cross-national survey research. Comparative cross-sectional re-
search creates opportunities for examining the role differing institutions and cultures
play in political behavior. They use the CSES as a vehicle to evaluate cross-cultural
equivalence in questionnaire design, survey mode, response rates, and case selection.
Presentation using data visualization is valuable for public opinion researchers and
consumers. Good visualization of survey and poll results can help researchers uncover
patterns that might be difficult to detect in topline and cross-tabulations and can also
help researchers more effectively present their results to survey and poll consumers.
Therefore, two chapters are devoted to graphing opinion data. The first, by Makela, Si,
and Gelman, argues that graphs are valuable at all stages of the analysis, including the
exploration of raw data, weighting, building bivariate and multivariate models, and
understanding and communicating those results to others. The second chapter, by
Schneider and Jacoby, provides specific guidelines on when a graph and what type of
graph would be most useful for displaying and communicating survey data and analyt-
ical results from survey models. Both chapters provide many useful examples and excel-
lent ideas for ways to explore and report data.
New Frontiers
The last section of the Handbook explores new frontiers in survey methodology.
It begins with an essay by Krupnikov and Findley that outlines the growth in survey
experiments and their usefulness. They argue that survey experiments provide a balance
between internal and external validity that provides needed leverage on opinion for-
mation. However, this is not without some costs, especially related to the participants
chosen, and researchers need to carefully consider their goals when identifying the best
test for their theory.
Gimbel and Newsome turn their attention to the consideration of how qualita-
tive data can both improve survey methodology and help to better understand and
Introduction to Polling and Survey Methods 9
interpret survey results. They focus on three qualitative tools—focus groups, in-depth
interviewing, and cognitive interviewing—and provide best practices for when and how
to use these tools. Qualitative research is an important part of many public opinion re-
search projects; Gimbel and Newsome provide a great deal of guidance about how to
best conduct this type of opinion research.
Razo considers the important role of context in social research. He argues that the
problem with context in social research is that it is often too vague, and that scholars
need greater guidance on collecting and analyzing contextual data. Razo’s chapter
provides insight into how scholars can better collect and use contextual data in their
analyses of individual-level opinion and behavior. Next Klašnja et al. discuss using
Twitter as a source of public opinion data. They identify three main concerns with using
Tweets as opinion, including how to measure it, assessing its representativeness, and
how to aggregate it. They consider potential solutions to these problems and outline
how social media data might be used to study public opinion and social behavior.
Many research questions involve the use of experts to identify processes, institutions,
and local environments or other information that only a knowledgeable informant
might have. The chapter by Maestas focuses on the use of expert surveys in providing
these bits of valuable information for researchers. It considers survey and questionnaire
design issues and aggregation procedures, with a focus on enhancing the validity and
reliability of experts’ estimates. Finally, the last chapter, by Jackson, focuses on polling
aggregation and election forecasting, which is interesting to both academics and applied
researchers. Her essay discusses the history of election forecasting and the use of poll
aggregation, the technical and statistical demands of poll aggregation and election
forecasting, and the controversies surrounding it.
This Handbook has brought together a unique mixture of academics and practitioners
from various backgrounds, academic disciplines, and experiences. In one sense, this is
reflective of the interdisciplinary nature of polling and survey methodology: polls and
surveys are widely used in academia, government, and the private sector. Designing,
implementing, and analyzing high-quality, accurate, and cost-effective polls and surveys
require a combination of skills and methodological perspectives. Despite the well-
publicized issues that have cropped up in recent political polling, looking back at the
significant body of research that has been conducted by the authors in this Handbook,
a great deal is known today about how to collect high-quality polling and survey data.
Over the course of the last several decades, the survey and polling industries have
experienced rapid change. We care about quality surveys and good survey data because
as social scientists we are only as good as the data we produce. Therefore, it is critical to
consider best practices, guidelines, and helping researchers assess a variety of factors so
that they can make good choices when they collect and analyze data. Equally important
10 Lonna Rae Atkeson and R. Michael Alvarez
is transmitting those results to others in a clear and accessible way. This Handbook goes
a long way toward providing a great deal of current information on the state of the field.
There is a bright future for further development of polling and survey methodology.
Unlike the situation a few decades ago, today there are many opportunities for innovative
research on how to improve polling and survey methodology. Ranging from new tools
to test survey design (e.g., Oberski in this Handbook, or tools found in Montgomery and
Cutler [2013]), to innovations in how interviews are conducted (Seligson and Moreno in
this Handbook), to the use of social media data to study individual opinion and behav
ior (Klašnja et al. in this Handbook), technology is changing the nature of survey and
polling methodology. We hope that the chapters in this Handbook help researchers and
practitioners understand these trends and participate in the development of new and better
approaches for measuring, modeling, and visualizing public opinion and social behavior.
Acknowledgments
Books, and in particular edited volumes like this one, require a great deal of help and assis-
tance. Of course we thank all of the authors of the chapters in this Handbook, especially for
their patience as we worked to produce this complicated volume. At Caltech, we thank Sabrina
De Jaegher for administrative assistance and for helping us stay organized and on track.
Brittany Ortiz from the University of New Mexico was instrumental in helping us get this proj
ect started.
And special thanks go to the team at Oxford University Press (current and past), who helped
us to launch, organize, edit, and most important, finish this Handbook. David McBride pro-
vided important guidance, and we also thank Claire Sibley, William Richards, Tithi Jana,
Anitha Alagusundaram, Emily MacKenzie and Kathleen Weaver. Finally, Alexandra Dauler
helped us formulate the basic idea for this Handbook and got us started with this project.
Notes
1. http://www.huffingtonpost.com/entry/most-americans-dont-trust-public-opinion-polls_
us_58de94ece4b0ba359594a708.
2. https://www.nytimes.com/interactive/2016/09/20/upshot/the-error-the-polling-world-
rarely-talks-about.html.
References
American Association of Public Opinion Research, Ad Hoc Committee on 2016 Election
Polling. 2017. “An Evaluation of 2016 Election Polls in the U.S.” https://www.aapor.org/
Education-Resources/Reports/An-Evaluation-of-2016-Election-Polls-in-the-U-S.aspx.
Montgomery, Jacob M., and Josh Cutler. 2013. “Computerized Adaptive Testing for Public
Opinion Surveys.” Political Analysis 21 (2): 172–192.
Saris, W. E., D. L. Oberski, M. Revilla, D. Z. Rojas, L. Lilleoja, I. Gallhofer, and T. Gruner, (2012).
Final report about the project JRA3 as part of ESS infrastructure (SQP 2002-2011). Technical
report, RECSM, Universitat Pompeu Fabra, Spain, Barcelona.
Pa rt I
SU RV E Y DE SIG N
Chapter 1
Herbert F. Weisberg
Introduction
The total survey error (TSE) approach has become a paradigm for planning and
evaluating surveys. The survey field began atheoretically in the early 1900s, when social
scientists simply began asking people questions. Gradually several separate theoretical
elements fell into place, starting with statistical sampling theory and then the social psy-
chology of attitudes (Converse 1987). By the mid-twentieth century the literature was
recognizing the existence of different types of error in surveys, particularly Hansen,
Hurwitz, and Madow (1953) and Kish (1965). Robert Groves’s (1989) Survey Errors and
Survey Costs systemized the consideration of errors in surveys in the comprehensive
TSE framework.
Groves’s book unified the field by categorizing the types of survey errors and pitting
them against the costs involved in conducting surveys. Each of the several types of
survey error can be minimized, but that takes financial resources, which are neces-
sarily finite. The TSE approach provides a systematic way of considering the trade-offs
involved in choosing where to expend resources to minimize survey error. Different
researchers may weigh these trade-offs differently, deciding to spend their resources to
minimize different potential survey errors. The TSE approach was developed when tel-
ephone interviewing was in its prime. It is still useful now that Internet surveys have be-
come prevalent, though Internet surveys raise somewhat different problems regarding
certain potential error sources. Furthermore, the different trade-offs between survey
errors and costs can vary between interviewer-driven studies (as in face-to-face and tel-
ephone interviewing) and respondent-driven studies (as in mail and Internet surveys).
Costs are not the only challenge that researchers face in conducting surveys. Time and
ethics also can impose constraints (Weisberg 2005). For example, the time constraints
raised when the news media need to gauge the immediate public reaction to a presiden-
tial speech are very different from when academic researchers have the luxury of being
able to take a month or two to survey public opinion. As to ethics, the concerns that
14 Herbert F. Weisberg
arise when interviewing on sensitive topics, such as people’s drug use, are very different
from those that exist when seeking to measure attitudes on public policies, such as gov-
ernment welfare programs. Depending on the survey organization, it is now common
for survey researchers to need prior approval from an institutional review board be-
fore going into the field, including approval of the research design and survey questions
(Singer 2008). Thus, there can be trade-offs between minimizing survey error and the
cost, time, and ethics involved in a survey.
In addition to survey constraints, Weisberg (2005) further emphasized the impor-
tance of another consideration: survey effects. These involve choices that must be
made for which there are no error-free decisions. For example, there may be question
order effects in a survey, but there is no perfect order of questions. It may be impor-
tant for survey researchers to try to estimate the magnitude of some of these survey
effects, though they cannot be eliminated regardless of how many resources are spent
on them.
While the TSE approach has become important in academic survey research,
the total survey quality (TSQ) approach has become important in government-
sponsored research. The quality movement developed in the management field
(Drucker 1973; Deming 1986), which recognized that customers choose the pro-
ducer that provides the best quality for the money. That led to management models
such as total quality management and continuous quality improvement. When
applied to the survey field (Biemer and Lyberg 2003; Lyberg 2012), the quality per-
spective leads to emphasis on such matters as the survey’s accuracy, credibility,
relevance, accessibility, and interpretability. For example, many survey clients ex-
pect high-quality deliverables, including a data set with a complete codebook and
a detailed description of the survey procedures, including sampling and weighting.
Thus, survey organizations must develop procedures to maximize the quality
of their product, but within the context of the trade-offs between survey errors
and costs.
The TSE approach focuses on a variety of possible errors in surveys. The early work on
surveys dealt with one type of error: the sampling error that occurs when one interviews
a sample of the population of interest rather than the entire population. As later work
identified other sources of errors, it became clear that sampling error was just the “tip
of the iceberg,” with several other potential sources of error also being necessary to
consider.
In preparing a survey effort, the researcher should consider the various potential
sources of error and decide how to handle each one. Typically, the researcher elects to
try to limit the amount of some types of error, such as by choosing how large a sample
Total Survey Error 15
to take. The researcher may opt to measure the magnitude of other types of error, such
as by giving random half samples different versions of a key question to see how much
question wording affects the results. Inevitably, the researcher ends up ignoring some
other types of error, partly because it is impossible to deal with every possible source of
error under a fixed monetary budget with time constraints.
Of course, survey research is not the only social science research technique that
faces potential errors. Campbell and Stanley’s (1963) distinction between “internal va-
lidity” and “external validity” in experimental research demonstrated how systemat-
ically considering the different types of error in a research approach could advance
a field.
Notice that the TSE approach deals with “potential” errors. It is not saying that these
are all serious errors in every survey project or that mistakes have been made. Instead,
it is alerting the researcher to where errors might be occurring, such as the possibility
that people who refuse to participate in a survey would have answered the questions
differently than those who responded. In some cases there will be no reason to think
that refusals would bias a study, but in other instances those who will not cooperate
might be expected to differ systematically from those who participate. If the research
topic is one likely to lead to this type of error, it might be worth trying to get as much in-
formation as possible about people who fell into the sample but were not interviewed,
so they can be compared with the actual respondents. But if nonresponse is unlikely
to bias the results, then it would be better to focus the survey budget on minimizing
other possible errors. Thus, the TSE approach makes researchers think about the likely
sources of error in their surveys before deciding what trade-offs to make.
In considering different sources of survey error, it is important to distinguish be-
tween random and systematic error. Random errors are the mistakes that occur by
chance without any particular pattern; they increase the variance of the variable but
should cancel out in large samples. Systematic errors are more serious, since they bias
the results, such as when questions are worded to give only one side of a policy question.
Furthermore, survey errors can either be uncorrelated or correlated. Uncorrelated
errors are the isolated errors, such as when a respondent says “strongly agree” and the
interviewer accidentally happens to press the key corresponding to “agree.” Correlated
errors are more serious because they increase the variance of estimates, making it
more difficult to obtain statistical significance. Cluster sampling, coders coding many
interviews, and a large number of interviews per interviewer all lead to correlated errors.
These procedures are commonly used to cut costs, but it is necessary to recognize that
they increase the variance of estimates.
Figure 1.1 depicts the various types of error covered in descriptions of TSE. Sampling
error is shown as the tip of the iceberg, with the other possible errors potentially being
as large or larger than the sampling error. Each of these types of error is described in the
following sections. Groves et al. (2009) provide an update of Groves (1989) that includes
later research on each type of error. Weisberg (2005) and McNabb (2014) further discuss
the different sources and categories of nonsampling error.
16 Herbert F. Weisberg
Sampling Error
Respondent Selection
Issues
Coverage Error
Nonresponse Error at the Unit Level
Nonresponse Error at the Item Level
Measurement Error Due to Respondents
Response
Accuracy Issues Measurement Error Due to Interviewers
Postsurvey Error
Survey Administration Mode Effects
Issues
Equivalence Error
Item-Level Nonresponse
Response accuracy can also be impaired when there is nonresponse on individual
survey questions. Such missing data occur when people refuse to answer particular
questions, skip questions accidentally, or do not have an opinion (“don’t know”). While
it is usually impossible to eliminate all missing data, motivating the respondent to an-
swer all questions can decrease the problem (Cannell, Oksenberg, and Converse 1977).
Many survey research specialists contend that the problem of missing data is lessened
Another random document with
no related content on Scribd:
DANCE ON STILTS AT THE GIRLS’ UNYAGO, NIUCHI
I see increasing reason to believe that the view formed some time
back as to the origin of the Makonde bush is the correct one. I have
no doubt that it is not a natural product, but the result of human
occupation. Those parts of the high country where man—as a very
slight amount of practice enables the eye to perceive at once—has not
yet penetrated with axe and hoe, are still occupied by a splendid
timber forest quite able to sustain a comparison with our mixed
forests in Germany. But wherever man has once built his hut or tilled
his field, this horrible bush springs up. Every phase of this process
may be seen in the course of a couple of hours’ walk along the main
road. From the bush to right or left, one hears the sound of the axe—
not from one spot only, but from several directions at once. A few
steps further on, we can see what is taking place. The brush has been
cut down and piled up in heaps to the height of a yard or more,
between which the trunks of the large trees stand up like the last
pillars of a magnificent ruined building. These, too, present a
melancholy spectacle: the destructive Makonde have ringed them—
cut a broad strip of bark all round to ensure their dying off—and also
piled up pyramids of brush round them. Father and son, mother and
son-in-law, are chopping away perseveringly in the background—too
busy, almost, to look round at the white stranger, who usually excites
so much interest. If you pass by the same place a week later, the piles
of brushwood have disappeared and a thick layer of ashes has taken
the place of the green forest. The large trees stretch their
smouldering trunks and branches in dumb accusation to heaven—if
they have not already fallen and been more or less reduced to ashes,
perhaps only showing as a white stripe on the dark ground.
This work of destruction is carried out by the Makonde alike on the
virgin forest and on the bush which has sprung up on sites already
cultivated and deserted. In the second case they are saved the trouble
of burning the large trees, these being entirely absent in the
secondary bush.
After burning this piece of forest ground and loosening it with the
hoe, the native sows his corn and plants his vegetables. All over the
country, he goes in for bed-culture, which requires, and, in fact,
receives, the most careful attention. Weeds are nowhere tolerated in
the south of German East Africa. The crops may fail on the plains,
where droughts are frequent, but never on the plateau with its
abundant rains and heavy dews. Its fortunate inhabitants even have
the satisfaction of seeing the proud Wayao and Wamakua working
for them as labourers, driven by hunger to serve where they were
accustomed to rule.
But the light, sandy soil is soon exhausted, and would yield no
harvest the second year if cultivated twice running. This fact has
been familiar to the native for ages; consequently he provides in
time, and, while his crop is growing, prepares the next plot with axe
and firebrand. Next year he plants this with his various crops and
lets the first piece lie fallow. For a short time it remains waste and
desolate; then nature steps in to repair the destruction wrought by
man; a thousand new growths spring out of the exhausted soil, and
even the old stumps put forth fresh shoots. Next year the new growth
is up to one’s knees, and in a few years more it is that terrible,
impenetrable bush, which maintains its position till the black
occupier of the land has made the round of all the available sites and
come back to his starting point.
The Makonde are, body and soul, so to speak, one with this bush.
According to my Yao informants, indeed, their name means nothing
else but “bush people.” Their own tradition says that they have been
settled up here for a very long time, but to my surprise they laid great
stress on an original immigration. Their old homes were in the
south-east, near Mikindani and the mouth of the Rovuma, whence
their peaceful forefathers were driven by the continual raids of the
Sakalavas from Madagascar and the warlike Shirazis[47] of the coast,
to take refuge on the almost inaccessible plateau. I have studied
African ethnology for twenty years, but the fact that changes of
population in this apparently quiet and peaceable corner of the earth
could have been occasioned by outside enterprises taking place on
the high seas, was completely new to me. It is, no doubt, however,
correct.
The charming tribal legend of the Makonde—besides informing us
of other interesting matters—explains why they have to live in the
thickest of the bush and a long way from the edge of the plateau,
instead of making their permanent homes beside the purling brooks
and springs of the low country.
“The place where the tribe originated is Mahuta, on the southern
side of the plateau towards the Rovuma, where of old time there was
nothing but thick bush. Out of this bush came a man who never
washed himself or shaved his head, and who ate and drank but little.
He went out and made a human figure from the wood of a tree
growing in the open country, which he took home to his abode in the
bush and there set it upright. In the night this image came to life and
was a woman. The man and woman went down together to the
Rovuma to wash themselves. Here the woman gave birth to a still-
born child. They left that place and passed over the high land into the
valley of the Mbemkuru, where the woman had another child, which
was also born dead. Then they returned to the high bush country of
Mahuta, where the third child was born, which lived and grew up. In
course of time, the couple had many more children, and called
themselves Wamatanda. These were the ancestral stock of the
Makonde, also called Wamakonde,[48] i.e., aborigines. Their
forefather, the man from the bush, gave his children the command to
bury their dead upright, in memory of the mother of their race who
was cut out of wood and awoke to life when standing upright. He also
warned them against settling in the valleys and near large streams,
for sickness and death dwelt there. They were to make it a rule to
have their huts at least an hour’s walk from the nearest watering-
place; then their children would thrive and escape illness.”
The explanation of the name Makonde given by my informants is
somewhat different from that contained in the above legend, which I
extract from a little book (small, but packed with information), by
Pater Adams, entitled Lindi und sein Hinterland. Otherwise, my
results agree exactly with the statements of the legend. Washing?
Hapana—there is no such thing. Why should they do so? As it is, the
supply of water scarcely suffices for cooking and drinking; other
people do not wash, so why should the Makonde distinguish himself
by such needless eccentricity? As for shaving the head, the short,
woolly crop scarcely needs it,[49] so the second ancestral precept is
likewise easy enough to follow. Beyond this, however, there is
nothing ridiculous in the ancestor’s advice. I have obtained from
various local artists a fairly large number of figures carved in wood,
ranging from fifteen to twenty-three inches in height, and
representing women belonging to the great group of the Mavia,
Makonde, and Matambwe tribes. The carving is remarkably well
done and renders the female type with great accuracy, especially the
keloid ornamentation, to be described later on. As to the object and
meaning of their works the sculptors either could or (more probably)
would tell me nothing, and I was forced to content myself with the
scanty information vouchsafed by one man, who said that the figures
were merely intended to represent the nembo—the artificial
deformations of pelele, ear-discs, and keloids. The legend recorded
by Pater Adams places these figures in a new light. They must surely
be more than mere dolls; and we may even venture to assume that
they are—though the majority of present-day Makonde are probably
unaware of the fact—representations of the tribal ancestress.
The references in the legend to the descent from Mahuta to the
Rovuma, and to a journey across the highlands into the Mbekuru
valley, undoubtedly indicate the previous history of the tribe, the
travels of the ancestral pair typifying the migrations of their
descendants. The descent to the neighbouring Rovuma valley, with
its extraordinary fertility and great abundance of game, is intelligible
at a glance—but the crossing of the Lukuledi depression, the ascent
to the Rondo Plateau and the descent to the Mbemkuru, also lie
within the bounds of probability, for all these districts have exactly
the same character as the extreme south. Now, however, comes a
point of especial interest for our bacteriological age. The primitive
Makonde did not enjoy their lives in the marshy river-valleys.
Disease raged among them, and many died. It was only after they
had returned to their original home near Mahuta, that the health
conditions of these people improved. We are very apt to think of the
African as a stupid person whose ignorance of nature is only equalled
by his fear of it, and who looks on all mishaps as caused by evil
spirits and malignant natural powers. It is much more correct to
assume in this case that the people very early learnt to distinguish
districts infested with malaria from those where it is absent.
This knowledge is crystallized in the
ancestral warning against settling in the
valleys and near the great waters, the
dwelling-places of disease and death. At the
same time, for security against the hostile
Mavia south of the Rovuma, it was enacted
that every settlement must be not less than a
certain distance from the southern edge of the
plateau. Such in fact is their mode of life at the
present day. It is not such a bad one, and
certainly they are both safer and more
comfortable than the Makua, the recent
intruders from the south, who have made USUAL METHOD OF
good their footing on the western edge of the CLOSING HUT-DOOR
plateau, extending over a fairly wide belt of
country. Neither Makua nor Makonde show in their dwellings
anything of the size and comeliness of the Yao houses in the plain,
especially at Masasi, Chingulungulu and Zuza’s. Jumbe Chauro, a
Makonde hamlet not far from Newala, on the road to Mahuta, is the
most important settlement of the tribe I have yet seen, and has fairly
spacious huts. But how slovenly is their construction compared with
the palatial residences of the elephant-hunters living in the plain.
The roofs are still more untidy than in the general run of huts during
the dry season, the walls show here and there the scanty beginnings
or the lamentable remains of the mud plastering, and the interior is a
veritable dog-kennel; dirt, dust and disorder everywhere. A few huts
only show any attempt at division into rooms, and this consists
merely of very roughly-made bamboo partitions. In one point alone
have I noticed any indication of progress—in the method of fastening
the door. Houses all over the south are secured in a simple but
ingenious manner. The door consists of a set of stout pieces of wood
or bamboo, tied with bark-string to two cross-pieces, and moving in
two grooves round one of the door-posts, so as to open inwards. If
the owner wishes to leave home, he takes two logs as thick as a man’s
upper arm and about a yard long. One of these is placed obliquely
against the middle of the door from the inside, so as to form an angle
of from 60° to 75° with the ground. He then places the second piece
horizontally across the first, pressing it downward with all his might.
It is kept in place by two strong posts planted in the ground a few
inches inside the door. This fastening is absolutely safe, but of course
cannot be applied to both doors at once, otherwise how could the
owner leave or enter his house? I have not yet succeeded in finding
out how the back door is fastened.