Rigor in Qualitative Research

MARIA LÖBLICH
Free University of Berlin, Germany

Rigor and quality criteria

This entry conceptualizes rigor as quality and addresses the criteria qualitative research
should measure up to, as well as the strategies for applying these criteria. Quality cri-
teria provide orientation for both researchers and readers. They help researchers to
decide about all the issues that are crucial in the course of a study and serve readers
(e.g., the methods teacher, the peer reviewer, or the funder) to evaluate the quality of
these decisions. Thus, they increase awareness about the implications of methodolog-
ical decisions. Quality is a rather vague concept. Some handbooks, particularly those
from the United States, assume a broad notion and include ethical issues (e.g., Lapan,
Quartaroli, & Riemer, 2012). This entry assumes a narrower notion of quality, which
focuses on methodological appropriateness and the production of dependable and valid
knowledge.
The understanding of good research depends on the particular paradigm, the rules
and standards the members of a scientific community are committed to. A paradigm
consists of several elements that reach from abstract metaphysical and epistemologi-
cal beliefs to concrete examples of scientific practice (Kuhn, 1970, pp. 182–187). Each
element relates to quality (see Figure 1): The epistemological belief that knowledge can
be produced independently of the observing subject leads to other quality criteria (e.g.,
objectivity) than the assumption that all knowledge is produced by subjective mental
constructs (objectivity is then rejected). If a study follows the purpose to reveal social
grievances, it will probably highlight its political stance and acknowledge the value of
subjectivity (Merrigan & Huston, 2009, p. 94). And the rule to accurately apply proce-
dures may count less in a methodology that assumes that empirical findings can always
be the object of a politics of interpretation. Finally, concrete examples illustrate what
excellent research practice is. Whereas the abstract elements of a paradigm are usu-
ally not spelled out, quality criteria serve as shorthand about the core values of that
paradigm (Tracy, 2010, p. 838). In Figure 1, the requirements for quality become more
concrete from the outside to the inner level.
From a sociology of science perspective (Kuhn, 1970), quality criteria define what is
accepted as good research in a scientific community. Like all elements of a paradigm,
they are valid as long as they are regarded as adequate by the scientific community.
With the institutionalization of a new paradigm, quality criteria can change. Different
paradigms usually compete with each other within an academic discipline. Quality
criteria then provide the rational justification for the truth claim of each paradigm.
Papers are judged on those standards by peer reviewers. However, science is not only
about having the better argument but is also about social forces such as reputation,
The International Encyclopedia of Communication Research Methods. Jörg Matthes (General Editor),
Christine S. Davis and Robert F. Potter (Associate Editors).
© 2017 John Wiley & Sons, Inc. Published 2017 by John Wiley & Sons, Inc.
DOI: 10.1002/9781118901731.iecrm0220
2 RIGOR IN Q U A L I TAT I V E R E S E A R C H

Epistemology

Purpose of
science Specification of quality requirements

Methodology

Classic
examples

Quality
criteria

Figure 1 Quality criteria in a paradigm.

resources, and power. Quality criteria provide the followers of a paradigm with “muni-
tions” in the competition for prestige and funding. In this regard, quality criteria also
play a role outside the scientific community. They are important to convince funding
bodies that a project is worth supporting or to produce legitimacy in the public sphere
for a less established field of research. Private and public funding bodies as well as the
media might tend to follow the quality ideas of established paradigms or have devel-
oped their own views regarding good science. In sum, social forces within and outside
an academic discipline shape the use and the development of quality criteria.

Positions on quality in qualitative methodology

Whereas quantitative methodology has developed more or less consensus around cer-
tain classical ideas of quality (validity, reliability, generalizability, and objectivity), the
situation is somewhat confusing in the qualitative methodological landscape. Consider-
ing not only communication studies but the social sciences in general, one encounters
a spectrum of distinct and conflicting concepts. There are diverse lists of criteria and
names for the same concept (for overviews see Creswell, 2007; Denzin & Lincoln, 2011).
Several dividing lines can be found in methodological writings: Should the standards of
quantitative research be applied to qualitative research or does qualitative methodology
need its own goodness criteria? Is it desirable to achieve consistent quality ideas across
paradigms within qualitative research or does every paradigm necessarily determine
its own criteria so that an overarching evaluation framework is impossible? This latter
question refers to the widely discussed problem of relativism—the belief that theory
and subjectivity determine the way knowledge is created and that ultimate standards
are impossible.
There are different reasons for the diversity of quality concepts. Qualitative research,
at least in Germany and the United States, has a history of competing with a strong
 RIGOR IN Q U A L I TAT I V E R E S E A R C H 3

quantitative social science tradition. In methodological writings of qualitative research

one encounters again and again the legitimacy problem of qualitative research. Some
books ask why students should use qualitative methods when “the so-called ‘norm’ is
quantitative” (Silverman, 2013, p. 12); others even start with the criticism of qualita-
tive research in order to then carve out its strengths (cf. Lamnek, 2010, p. 3). Since
the 1970s, debates revolving around the “proper place” for qualitative research in the
social sciences have shaped the development of qualitative methodologies (Silverman,
2013, p. 9). Even if the status of qualitative research might be less disputed today than it
was a few decades ago, this history explains the ongoing efforts invested in developing
sets of quality criteria (cf. Seale, 1999, p. 467). Moreover, according to some authors,
funders and reviewers are sometimes more familiar with quantitative approaches than
with qualitative ones. Qualitative project descriptions may therefore need to conform to
some of the standards of quantitative research (Creswell, 2007, p. 123). These structures
of expectation on the part of evaluating bodies might provide another explanation for
the diversity in quality concepts, and particularly for the fact that there are some that
have been modeled after quantitative ideas (see the classification below). Finally, the
diversity of quality concepts can be understood as a consequence of the differentiation
of paradigms and theory traditions operating and challenging each other in this field.
Quality is, however, not equally important to the different paradigms. Some place less
emphasis on goodness criteria, others more.
To sort the “creative complexity” of quality concepts (Tracy, 2010, p. 837), Denzin
and Lincoln (2011, p. 564) proposed a distinction of three positions according to
their attitude towards adopting unvarying, permanent standards (or foundation):
the foundational, the quasifoundational, and the nonfoundational positions. This
entry, by contrast, considers the strong quantitative research context within which
qualitative research has developed and interprets the “criteriology” (Seale, 1999, p. 466)
as a constraint to justify qualitative research and to differentiate it from quantitative
research. One manifestation for this differentiation of qualitative from quantitative
research is the fact that the term rigor, common in quantitative research, is rather
unusual in qualitative methodology. This entry distinguishes four positions according
to their relation to quantitative research, meaning the extent to which they tend to
either reject the quality criteria of quantitative research or work with them.

1 The first position assumes that there are universal standards to evaluate social
science research because quantitative research and qualitative research have a lot in
common. According to this position, both traditions share the same goals, are com-
mitted to the dissemination of knowledge and other values, and base their research
on theory, fallibility of knowledge, and inquiry. Universal standards are those already
existing in the quantitative tradition and can be applied to qualitative research. The
means of achieving these standards, however, may have to be different. Above all, good
research is a matter of comprehensiveness and transparency and should enable the
reader to evaluate the quality of a study. Developing new criteria would instead cause
confusion and increase skepticism about qualitative research. Researchers should focus
on the explication of their methodology, applied quality criteria, and their strategies
4 RIGOR IN Q U A L I TAT I V E R E S E A R C H

used (Poortman & Schildkamp, 2012). Critics have called this embracing of quantita-
tive criteria a defensive measure and a muddying of the distinction between qualitative
research and quantitative research.
2 Another position emphasizes the distinctness of epistemology and methodology
in the qualitative and the quantitative tradition. As stated by proponents of this view,
qualitative research assumes multiple, subjectively constructed realities instead of one
objective reality. It produces a different kind of knowledge and uses specific methods.
The dichotomy between the two research traditions justifies alternative evaluation cri-
teria that are regarded as more appropriate and better applicable to qualitative research
than the quantitative ones. Despite the perceived dichotomy, however, quantitative stan-
dards have not been rejected by this position but have served as (critically discussed)
guidelines for the development of sets of criteria. These criteria, however, are rather sim-
ilar than different to the quantitative ones, which has been considered as just relabeling
by some authors. The standard work by Lincoln and Guba (1985), for instance, replaced
internal validity, external validity, reliability, and objectivity by credibility, transferabil-
ity, dependability, and confirmability. Credibility demands that the presentation of the
researcher corresponds to participants’ views. Transferability refers to generalizability
and means that findings can be applied to other settings. Dependability means that an
“inquiry auditor” examines the accuracy of both research process and records. Con-
firmability requires researchers to reflect how their preconceptions affect their studies’
findings.
3 The third position shares with the second approach the idea that quantitative
and qualitative traditions are distinct. Unlike the second position, this approach has
detached itself from quantitative evaluation standards. It stresses that the quality of a
study can only be discussed in terms of the specific notion of truth that a paradigm
adheres to because each paradigm determines specific quality criteria (Merrigan & Hus-
ton, 2009, p. 79). While the second position aimed at consistent quality criteria for all
qualitative research, this position rejects this idea and emphasizes the plurality of the-
oretical traditions such as feminism, ethnography, cultural studies, or Marxism. This
plurality produces a variety of quality criteria such as caring and accountability (ethnog-
raphy), reflexivity and deconstruction (constructivism), and dialogue (feminism; for an
overview see Denzin & Lincoln, 2011). Critics of this position worry that being tied dog-
matically to some epistemological and methodological principle would threaten quality.
They argue that one can learn to do good research from a variety of examples no matter
under which paradigm they have been carried out.
4 The fourth position goes so far as to abandon classical quality ideas. If there
are multiple, equally valid realities, proponents argue, and if all knowledge is socially
constructed, it is impossible to use an overarching evaluation framework. From this
radical relativist perspective, the issue of quality has become obsolete, at least in the
meaning of validity and reliability. Good research requires a moral or political attitude
and seeks to achieve social change, social criticism, and empowerment. According to
this position, criteria such as validity and reliability reproduce a sort of research that
silences too many voices. Agreeing with the idea that a study can only be assessed in
terms of the particular paradigm it adheres to, proponents suggest criteria such as care,
 RIGOR IN Q U A L I TAT I V E R E S E A R C H 5

neighborliness, personal responsibility, shared governance, or emotionality (for more

details see Denzin & Lincoln, 2011).

Quality criteria and strategies

Considering the lack of consensus within the quality discussion, this entry presents
one understanding of quality drawing mainly on the first and second position and
elaborates on one set of quality criteria. This, of course, intensifies heterogeneity. Like
other concepts of quality, it is based on specific epistemological assumptions without
firmly aligning itself within a specific paradigm or theoretical tradition. The quality
criteria proposed by this entry are based on an understanding of qualitative research
that does not assume that quantitative research and qualitative research are incompat-
ible with each other (Meyen, Löblich, Pfaff-Rüdiger, & Riesmeyer, 2011). Moreover,
it aims at transferability and self-reflection, avoids social criticism, and is guided by
theory. This entry adheres to the idea that there can be no knowledge free of subjec-
tivity, values, preconceptions, and social context; at the same time, it retains the idea
of making qualitative studies accessible for evaluation. In order to solve this dilemma
(and the problem of relativism), it follows Karl Mannheim. His sociology of knowledge
assumes that knowledge is shaped by the social conditions of the thinker: by genera-
tion, profession, religion, political orientations, and gender. Despite the fact that there
are different perspectives to view reality, Mannheim retained the idea that (scientific)
knowledge can claim truth. He proposed a relational concept of objectivity that does
not need a final authority in terms of quality. His concept of objectivism implies the
ideas of understanding and translating: understanding the “differences in structure” of
the various ways of constructing an object and translating the different perspectives to
each other (Mannheim, 1997). Theory, self-reflection, and transparency help to bridge
different perspectives (and paradigms) in qualitative research. Self-reflection means
asking oneself why one poses different research questions than colleagues about the
same object, and how this relates to one’s own biography and to other social factors that
have shaped how one perceives an object and construes knowledge in one’s thinking
(Meyen et al., 2011). Transparency is directed towards the reader and means that every
step in the research process is made comprehensible by the researcher. This means that
researchers should be concerned with quality from the beginning of the research pro-
cess and not only—as some handbooks suggest—in the final stage of a research project.
Quality is an issue when wording the research question (as this determines the focus
and scope of the study), and when choosing theory, data collection methods, and sam-
pling procedure, and it is also an issue when data are analyzed and findings interpreted.
The use of theory helps the reader to understand how a particular interpretation was
achieved.
On the basis of the quality concept outlined above, four quality criteria are proposed
(Meyen et al., 2011). They have been developed within a dominant quantitative social
scientific research context. This entry retains the classical terms because alternative
terminologies have not generally been accepted in communication and media studies
6 RIGOR IN Q U A L I TAT I V E R E S E A R C H

(Bruhn Jensen, 2012, p. 295). Criteria are adapted to qualitative research and their def-
initions to some extent overlap with the set of criteria by Lincoln and Guba (1985).

• Reliability: Intersubjective comprehensiveness

• Validity: Consistency
• Transferability: Generalizability
• Freedom of value judgment: Absence of any normative assessment.

Reliability refers to the research process and takes into account the processual nature
of qualitative research. It addresses the extent to which decisions made in the course of a
study are comprehensible and made accessible for collegial scrutiny. Validity and trans-
ferability are central as they pertain to findings. A study is valid if research question,
theory, method, and findings are consistent. Theory and methods have to be adequate
in terms of the purpose of the study. Transferability implies that qualitative research
aims at providing explanations not only for the particular research object but beyond it.
This means that one draws from specific cases for more general cases, and that findings
can be transferred to other settings, texts, or the respective population under study.
Generalizability does, however, not mean that the sample represents all members of
a population of interest, thus is not to be confounded with statistical generalization. It
is certainly debatable to what extent freedom of value judgment can be implemented.
Normativity does not only come into play when findings are interpreted. Already the
decision for a particular research object and a theory imply normative selections, not
to mention the researcher’s personal values and beliefs. The underlying understanding
of research, however, aims at understanding and explaining, not at criticizing, society.
There is less discussion about quality strategies than about quality criteria. The
six strategies proposed below are common in methodological writings. The criteria
addressed by each strategy are indicated in parentheses.

• Proximity to the object under study (reliability, validity). Researchers should go

where participants usually are (office, home), let them narrate, and quote them in
the research report. Proximity implies that methods of data collection and analysis
are adequate to the research object. A typical strategy to address validity in hand-
books of qualitative research is “member validation” or “member checking.” The
idea is that lay people are knowledgeable about their lives and capable of assessing
the researcher’s interpretations of their routines and habits. Participants therefore
are approached to decide whether descriptions and explanations are true and accu-
rate (Lindlof & Taylor, 2011). This strategy is certainly a good way to live up to
ethical principles. On the other hand, what if the research process never comes to
an end because every member check adds a new interpretation or revises the old
one? At which point in time is the researcher allowed to stop this kind of valida-
tion? Another problem concerns social desirability: If the behavior of an interviewee
does not correspond to general norms, will the interviewee then “correct” it in the
research report?
• Provide evidence with contextualized quotes (validity, reliability). This strategy
could be subsumed under proximity but is mentioned separately to emphasize how
 RIGOR IN Q U A L I TAT I V E R E S E A R C H 7

important it is to support interpretations with empirical material. Contextualized

quotation means to describe the person who is mentioned: Whether the quote
is made by a man or a woman makes a difference, by a person living in a city or
in the countryside, by a teacher or a construction worker. A quote, however, is
not a “proof” like in natural science. It might be chosen arbitrarily or trimmed to
underpin a researcher’s argumentation (Meyen et al., 2011).
• Documentation of the research process (reliability). All decisions have to be
explained and justified: the selection of methods, data collection instruments,
training of interviewers and coders, the sampling procedure, data analysis. Ideally,
readers of the research report will not only learn who participated in the study
but will be able to inspect material such as interview guides, transcripts, photos,
and texts. Such “audit trails” also help the researcher to retrace all important
methodical steps when reporting how interpretations were developed. The benefit
of an “inquiry auditor” (Lincoln & Guba, 1985, p. 318) who attests the acceptability
of an inquiry is less obvious. What is the advantage of delegating the quality check
to an external auditor (whom the readers don’t know and would have to trust)
instead of letting the readers form their opinions by themselves?
• Self-reflection (reliability, validity, freedom of value judgment). This means to be
aware of the everyday life preconceptions and theoretical assumptions that have
guided the study and to reflect who the researcher is and how he or she relates to
the object under study (previous knowledge, interests).
• Reflection of the study’s development (validity, transferability). What resources
were accessible (money, time, staff)? In what intellectual environment has the
study been developed? Who has had what interests in the issue (also those of
participants, funders) and how have data been collected? What limits might the
study have due to these conditions (Meyen et al., 2011)?
• Interpretation in groups (validity, reliability). Ideally, several researchers are
involved in the interpretation of findings. All team members should be familiar
both with the theoretical foundation of the study and with the material. Validity and
reliability can be checked through “peer debriefing” (discussions with colleagues,
cf. Lincoln & Guba, 1985, p. 308) and by exposing findings to other perspectives,
for instance at conferences.

Future directions

The understanding of quality presented here is one proposal among many in qualitative
methodology. The discussion about quality criteria in qualitative social research will
continue. With new paradigms emerging or old ones being rediscovered, supplements
and modifications of existing notions of quality will be made. The unrest that is typical
for qualitative methodology is caused by different background theories, the competition
among qualitative approaches, and the strong quantitative research context. This unrest,
however, also involves creativity, which advances further development of methods.
More research into the history of qualitative methodology in communication
studies could contribute to the self-reflection of this creative unrest and provide
8 RIGOR IN Q U A L I TAT I V E R E S E A R C H

orientation for future attempts to define quality. Such historiography would find out
how early communication scholars have experimented both with qualitative and
quantitative methods, which solutions they proposed concerning quality requirements
in communication research, and what societal and academic developments influenced
the development of quality ideas. This research should not limit itself to a qualitative
methodology history but rather write a history that perceives both directions in
dependence on each other.

SEE ALSO: Ethics of Empirical Research; Interpretive Research; Member Checks;

Methodological Rigor in Quantitative Research; Objectivity; Qualitative Methodology

References

Bruhn Jensen, K. (2012). A handbook of media and communication research. Qualitative and
quantitative methodologies (2nd ed.). London: Routledge.
Creswell, J. W. (2007). Qualitative inquiry and research design: Choosing among five approaches
(2nd ed.). Thousand Oaks, CA: SAGE.
Denzin, N. K., & Lincoln, Y. S. (2011). The Sage handbook of qualitative research (4th ed.). Thou-
sand Oaks, CA: SAGE.
Kuhn, T. S. (1970). The structure of scientific revolutions. Chicago, IL: University Press.
Lamnek, S. (2010). Qualitative Sozialforschung [Qualitative social research] (5th ed.). Weinheim:
Beltz.
Lapan, S. D., Quartaroli, M. T., & Riemer, F. J. (Eds.). (2012). Qualitative research: An introduction
to methods and designs. San Francisco, CA: Jossey-Bass.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury Park, CA: SAGE.
Lindlof, T. R., & Taylor, B. C. (2011). Qualitative communication research methods (3rd ed.).
Thousand Oaks, CA: SAGE.
Mannheim, K. (1997). Collected works of Karl Mannheim: Vol. 5. Essays on the sociology of knowl-
edge. London: Routledge.
Merrigan, G., & Huston, C. L. (2009). Communication research methods (2nd ed.). Oxford:
Oxford University Press.
Meyen, M., Löblich, M., Pfaff-Rüdiger, S., & Riesmeyer, C. (2011). Qualitative Forschung in der
Kommunikationswissenschaft. Eine praxisorientierte Einführung [Qualitative research in com-
munication studies. A practice-oriented introduction]. Wiesbaden: VS.
Poortman, C. L., & Schildkamp, A. (2012). Alternative quality standards in qualitative research?
Quality & Quantity, 46, 1727–1751. doi:10.1007/s11135-011-9555-5
Seale, C. (1999). Quality in qualitative research. Qualitative Inquiry, 5, 465–478. Retrieved from
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.460.3511&rep=rep1&type=pdf
(accessed March 8, 2017).
Silverman, D. (2013). Doing qualitative research (4th ed.). Los Angeles, CA: SAGE.
Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research.
Qualitative Inquiry, 16(10), 837–851. doi:10.1177/1077800410383121

Further reading

Flick, U. (2005). Qualitative research in sociology in Germany and the US—State of

the art, differences and developments. Forum Qualitative Social Research, 6(3), Art. 23.
doi:10.1177/0268580907074539
 RIGOR IN Q U A L I TAT I V E R E S E A R C H 9

Guba, E., & Lincoln, Y. S. (2005). Paradigmatic controversies, contradictions, and emerging con-
fluences. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage handbook of qualitative research (3rd
ed., pp. 191–215). Thousand Oaks, CA: SAGE.
Löblich, M., & Musiani, F. (2014). Net neutrality and communication research: The implications
of Internet infrastructure for the public sphere. In E. L. Cohen (Ed.), Communication yearbook
38 (pp. 339–367). New York: Routledge.
Löblich, M., & Averbeck-Lietz, S. (2016). The transnational flow of ideas and histoire croisée
with attention to the cases of France and Germany. In P. Simonson & D.W. Park (Eds.), The
international history of communication study (pp. 25–46). New York: Routledge.
Reichertz, J., & Breuer, F. (2002). Standards of qualitative social research. Historical Social
Research, 27(4), 258–269.
Seale, C. (1999). The quality of qualitative research. London: Sage.

Maria Löblich (PhD) is a professor at the Institute for Media and Communication Stud-
ies, Freie Universität Berlin. Apart from qualitative methodology, her research interests
are media policy, the history of communication studies, and communication history.