See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/351885851

Errors in Survey Research and their Threat to Validity and Reliability

Preprint · May 2021

DOI: 10.13140/RG.2.2.35901.18405

CITATIONS READS
0 11

1 author:

Faiswal Kasirye
International Islamic University Malaysia
21 PUBLICATIONS   2 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Master Degree Research Paper View project

Assessing the Relationships between Information Seeking, Motives of Using Social Media and Information Sharing on Haze and Air Pollution among IIUM Students View
project

All content following this page was uploaded by Faiswal Kasirye on 26 May 2021.

The user has requested enhancement of the downloaded file.

 Errors in Survey Research and their Threat to Validity and Reliability
Faiswal Kasirye
Department of Communication,
International Islamic University Malaysia

Introduction to Survey as a Research Method

Survey as a formal research method was established in the early 1930s by sociologist Paul
Lazarsfeld to examine the effects of the radio on political opinion formation of the United States.
This method has since become a very popular method for quantitative research in the social
sciences. Groves et al. (2005) define survey as a systematic method for gathering information from
a sample of entities for purposes of constructing quantitative descriptors of the attributes of the
larger population of which the entities are members.

More specifically, a survey is information gathered by asking people questions through

standardized and unstandardized interview questionnaires by having people read or hear questions
and recording own answers or otherwise be recorded by them. Data is collected from only a subset
of the population rather than from all members of a survey research and design.

The traditional definition of survey research is, that it is a quantitative method for collecting
information from a pool of respondents by asking them multiple questions. This research type
includes the recruitment of individuals, collection, and analyzing of data. It’s useful for researchers
who aim at communicating new features or trends to their respondents (Häder, 2010).

Generally, it’s the primary step towards obtaining quick information about mainstream research
topics and conducting more rigorous and detailed quantitative research methods like surveys/polls
or focus groups/on-call interviews and others that often use it.

The survey method can be used for descriptive, exploratory, or explanatory research. The method
is best suited for studies that have people as units of analysis. Although other units of analyses
such as groups, organizations or pairs of organizations, such as buyers and sellers, are also studied
using surveys (Liebau et al., 2019). Such studies often use a specific person from each unit as a
key informant for that unit, and data from such surveys may be subject to respondent bias if the
informant chosen does not have adequate knowledge or has a biased opinion about the
phenomenon under study.
Survey Errors, Response Bias and their Effect on Reliability

Weisberg (2005) defines survey error as mistakes made during the construction and
implementation of the survey instrument, as well as the interpretation of results. It is associated
with comprehensiveness, interpretability of the data, and the generalizability of the results that in
turn affect the reliability and validity of the research data and findings. This type of error affects
the validity of the measurement in the questionnaire, which creates a measurement error. Survey
error however can be avoided in part, by cautiously designing the study and its instrument, creating
a comprehensive set of well-crafted survey items, and properly interpreting and presenting the
results.

While doing survey research, the primary aim is to estimate the unknown and the known samples
or sample of population in terms of means, standard deviations and frequencies to determine the
population values in the study. however, sometimes researchers end up picking respondents that
are not appropriate for their studies and hence end up with survey errors because the respondents
are likely to answer questions that they do not have knowledge about.

There are various ways how a survey research can be wrong about its data. Foreample, when a
wrong survey is gotten or discovered during a research process, the results from that data are
referred to as having bias or an error which can invalidate the findings of a particular study in
question. Survey Error is normally caused when researchers make mistakes during the formation
of a questionnaire, as well as interpreting the results during the analysis.

Survey errors can be classified into two categories, non-sampling and sampling errors (Assael, &
Keon,1982). Non-sampling errors arise due to unplanned mistakes in the data collection process,
whereas sampling errors arise due to the variation from measuring only a subset of the population
(Groves, 1989). Often non-sampling errors are difficult to detect and quantify. In contrast,
sampling errors can be controlled and quantified. Indeed, many published surveys report results
only in terms of estimates and sampling errors, without assessments of non-sampling errors

Survey errors can as well be looked at in terms of response bias which refers to the ways that
respondents may be unjustifiably influenced when providing answers on a survey. Bias is an issue
that affects the accuracy of the survey data obtained and is the results of participant’s inability or
unwillingness to answer questions truthfully as advanced by Hermanni (2019). Furthermore, when
assessing the quality of information obtained from survey research, the researcher must determine
the accuracy of those results. This requires careful consideration of the research methodology
employed in relation to the various types of errors that might result.

There are several kinds of survey error and bias that are important to understand prior to developing
your survey and how they can affect or become a threat to validity and reliability of your data.
They can basically be classified into two categories, non-sampling and random sampling errors.
However, from these two types of errors, there emerges several other types that come from the
different research practices that happen during the process of doing research. The two main survey
errors are discussed below and broken down into other types of errors as were put forward by
Biemer and Lyberg, (2003).

Random Sampling Errors

A survey is generally formed on a sample, although sometimes the sample might be representative
of the population of the study. sampling error comes in during that process and is often reffered to
as random sampling error. In most cases, once the error happens or is discovered from the data, it
becomes hard to be eliminated until a large sample is studied to determine the significance or the
effect of the errors. These errors once they happen, they create a huge doubt on the generalizability
of the findings or the data, and therefore in the process becomes a threat to the validity and
reliability of the data and its findings.

Non-random Sampling Error

Here, it is also regarded as one of the major sources of survey error where if respondents cooperate
and provide their answers to accomplish a particular goal. However, two other categories may
occur here which are, non-response error and response bias. Non response error refers to a situation
where a few 100 percent response rates are registered. Whereas bias may be defined as a situation
where participants answer questions with a certain mindset of and they end up misrepresenting the
truth in their responses knowingly or unknowingly through deliberate falsification, unconscious
misrepresentation and other means.
Some of the other survey error types (Biemer & Lyberg, 2003) include;

Sampling error. This occurs when mistakes are made while selecting a sample. Sampling error
typically occurs when a specific subgroup within the population is over or under represented in
the sample. When this happens, the results obtained from the survey are not generalizable which
automatically results into being a threat to reliability of the findings of a particular study (Reiter,
2014). This is because the characteristics of respondents in the sample do not
match proportionately with those in the population. This often happens with small samples
especially when there are a number of small subgroups within the population. As a result, it is
generally a good idea to over sample when possible to maximize the likelihood of obtaining a good
representation of the general population and subgroups within that population which in turn helps
in increasing the reliability and reducing its threat to validity of the data and findings. However,
having a large sample size may also not solve the problem if the sample does not proportionally
reflect the population which may still create sampling error that in turn will still cause the results
to be flawed and hence becoming a huge threat to the reliability of the data and findings of the
study. Therefore, it is advisable to always find a way of eliminating sample error during the onset
of the research preparations.

Survey Scope error. This error is a mistake researchers make when a survey does not include
important items required to fully answer research questions of the study. Failing to ask important
questions may provide inaccurate answers to the research questions posed at the onset of the study.
There will always be some tension between making sure to include all important survey questions
and limit the length of the survey. Sometimes this error is made because the researcher has not
carefully considered what they need to know. You may only realize a mistake was made once you
have the survey results and find that you failed to ask some of the important question to the
participants. When a questionnaire or scale does not include items that cover all the important
aspects of the topic, the survey scope error occurs which reduces the validity and reliability of the
results. In other words, the error becomes a threat to reliability of the data and its findings.

Response Refusal error. This is somehow related to sampling error, because it affects the
generalizability of the results, but it in a different way. If participants fill your survey, they would
most likely form a representative sample of the population but several individuals choose not to
respond to the survey the results may not be generalizable which might turn out to be threat to the
validity of the results that would be achieved in the study. To ensure full ethical transparency, you
should always report the sample size and the response rate when writing up results.

Non-Response error on items. This is when a respondent fail to answer all the questions on a
survey. It can happen by accident or intentionally but incidences may happen where they might
fail to answer some items probably they forget to do so, or they may decide not to or else they also
not know how to answer them. This definitely affects the outcome of the results therefore, it is
important to know how such incidents may affect the generalizability of the study and hence
becomes a threat to reliability. Furthermore, non-response is unfortunate when you are
administering a questionnaire but do not necessarily invalidate the results, it depends on how many
respondents fail to answer a specific question. It may also happen when administering a scale.

Purpose Creep error. Researchers here add items to a survey that don’t directly concern with the
needs and purposes of the study. Unnecessary survey items, like extra demographics or “might be
nice to know” items, may be interesting but often they are never used. Adding unnecessary or
tangentially related items may not affect the validity of the results but can still be a problem as it
adds to participants’ fatigue and can affect their willingness to considerately answer all the items
in a lengthy survey.

Response Bias

For response bias. It is a general term that describes ways a respondent may be influenced when
filling a survey. Bias affects the accuracy and reliability of the results (Groves, Dillman, Eltinge,
& Little, 2001). There are several ways in which an individual’s response may be biased. The bias
may be deliberate on the part of the respondent, and it happens as a result of a respondent being
disproportionately influenced to respond in a certain way. Similarly, it can be caused when items
are unclear or poorly constructed but can also occur as a result of the response scale selected for
participants to use when they provide their answers. Researchers are advised to never
underestimate the propensity of individuals to misunderstand what is being asked or otherwise
provide information that is inaccurate to some degree.

Systematic error, or bias, results from mistakes or problems created by the research design or from
flaws in the execution of the sample design. Systematic error exists in the results of a sample if
those results show a consistent tendency to fluctuate in one direction from the true value of the
population. It includes all sources of error except those introduced by random sampling processes.
In fact, systematic errors are sometimes called non-sampling errors. The non-sampling errors that
can systematically influence survey answers can be categorized as sample design error and
measurement error.

Measurement Error: This is often a much more serious threat to survey validity and reliability
than is random error. Foreample, when the results of a public opinion poll are given in the media
and professional marketing research reports, an error figure is frequently reported like it can be a
plus or a minus small percent. The television viewer or the user of the research study is left with
the impression that this figure refers to total survey error. Unfortunately, this is not the case. Czaja
and Blair (1996) contend on this issue that, such a figure refers only to random sampling error, and
that it does not include sample design error and speaks in no way to the measurement error that
may exist in the research results. Measurement error occurs when there is a difference between the
information being sought and that the one that is actually obtained by the measurement process.

Population Specification Error: Here, Kulas, Klahr and Knights (2018) intimate that, results from
an incorrect definition of the sample population. For example, supposing a researcher defined the
population for a study as people over the age of 35. Later, it gets discovered that younger
individuals should have been included and that the population should have been defined as those
of 20 years of age or older. If those younger people who were excluded are significantly different
in regard to the variables of interest, then the sample results will be biased and the outcome will
definitely be questioned. Hence, the error turns into a threat to the study validity and reliability.

Lastly, let’s talk about interviewer Error or bias, it comes from the researcher’s influence over
the respondent knowingly or unknowingly to give untrue or inaccurate answers. Foreample, their
dress code, age, gender, facial expressions, body language, or tone of voice may influence the
answers given by the respondents. This type of error is caused by problems in the selection and
training of interviewers by failing to follow instructions. Interviewers must be properly trained and
supervised to appear neutral at all times. Another type of interviewer error occurs when deliberate
cheating takes place. This can be a particular problem in door-to-door interviewing, where
interviewers may be tempted to falsify interviews and get paid for work they did not actually do.
This type of error tends to bring out errors in the findings as a result of falsifying the results and
hence becomes a threat to the validity of research as well as its reliability.
CONCLUSION

In conclusion, survey errors are classified into two categories, and these are; non-
sampling and sampling errors as according to Reiter (2014). Common sources of non-sampling
error include measurement error, interview error, systematic error, nonresponse error, and response
refusal error and many others. Furthermore, their impact can be reduced by careful survey design.
Sampling errors occur because only subsets of populations are measured, and estimates vary by
sample. Sampling errors arise in every survey, but they are quantifiable. Their end result is to
reduce the validity and reliability of the findings and the data and hence, they automatically
become a threat to reliability and generalizability of the findings.

Therefore, when survey data is gathered, several statistical analyses are supposed to be done in
order to realize findings that were intended in the objectives of the study. several scholars (Groves
et al., 2004, 2005 and 2011) believe, that surveys are the widely preferred data collection method
that simplifies research in the shortest period of time and if done in the right procedure, survey
errors can be highly minimized to make sure that the results can be replicated.

Therefore, estimating the total impact of non-sampling and sampling errors is challenging. Unless
a gold standard file is available, in which non-sampling errors are known to be small, there are no
standard approaches for estimating the total survey error. There is need to develop approaches and
software that can best suitably address and detect sample errors before a researcher takes important
steps or developments in their research studies.
 REFERENCES

Assael, H., & Keon, J. (1982). Nonsampling vs. sampling errors in survey research. Journal of
Marketing, 46(2), 114-123.
Biemer, P. and Lyberg, L. (2003) Introduction to Survey Quality, John Wiley & Sons, Inc.,
Hoboken, NJ.
Czaja, R. and Blair, J. (1996) Designing Surveys, Pine Forge Press, Thousand Oaks, CA.
Groves, R. (1989) Survey Errors and Survey Costs, John Wiley & Sons, Inc., New York.
Groves, R. M., Fowler Jr, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R.
(2011). Survey methodology (Vol. 561). John Wiley & Sons.
Groves, R., Dillman, D., Eltinge, J., and Little, R.J.A. (2001) Survey Nonresponse, John Wiley &
Sons, Inc., New York.
Groves, RM, Fowle, F., Couper, MP, Singer, E., & Tourangeau, R. (2004). Survey Methodology.
Hoboken, New Jersey: Wiley Series in Survey Methodology.
https://doi.org/10.1093/poq/nfi018.
Häder, M. (2010). Empirical social research. Wiesbaden: VS publishing house for social sciences.
Hermanni, H. (2019). The total survey error model in telephone surveys. In telephone surveys
inGermany (pp. 45-80). Springer VS, Wiesbaden.
Kulas, J. T., Klahr, R., & Knights, L. (2019). Confound it! Social desirability and the “reverse-
scoring” method effect. European Journal of Psychological Assessment, 35(6), 855-872.
Liebau, M., Schunter, J., Schurath, R., & Schwarz, R. (2019). Questionnaire construction for
telephone surveys. In telephone surveys in Germany (pp. 193-239). Springer VS,
Wiesbaden.
Reiter, J. P. (2014). Survey Error. Wiley StatsRef: Statistics Reference Online,3, 1-5.
Weisberg, H.F. (2005) The Total Survey Error Approach. A Guide to the New Science of Survey
Research, The University of C

View publication stats