Cresswell - 2015 - Revisiting Mixed Methods and Advancing Scientific PDF

Revisiting Mixed Methods and Advancing Scientific Practices
Revisiting Mixed Methods and Advancing Scientific

Practices
John W. Creswell
The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry
Edited by Sharlene Nagy Hesse-Biber and R. Burke Johnson
Print Publication Date: Aug 2015

Subject: Psychology, Psychological Methods and Measurement
Online Publication Date: Jan 2016 DOI: 10.1093/oxfordhb/9780199933624.013.39
Abstract and Keywords
This chapter reviews the developments in mixed methods during the past few years. The
chapter introduces 10 scientific developments to emerge in the mixed methods research
field. These developments relate to the characteristics of mixed methods and the terms
used within it, the value of mixed methods, incorporating philosophies and theory into a
mixed methods study, mixed methods designs and procedures, recognizing threats to va
lidity, research questions, displays of jointly presenting results, the writing structure of a
study, and the criteria to evaluate the quality of a study. The chapter concludes with rec
ommendations for the future conduct of mixed methods research based on these scientif
ic developments.
Keywords: mixed methods, research, quantitative approaches, qualitative approaches, scientific practices
Introduction
During the spring, 2013, when I served as a visiting professor of public health at
Harvard, I had the opportunity to present to students and faculty at Harvard’s In
stitute for the Social Sciences. This was more than a simple presentation—it was
an awakening for me to the needs of scholars about mixed methods research. My
topic was “Mixed Methods Research—The State of the Art.” The audience was
largely composed of social scientists from Harvard and nearby schools. Also in the
audience was my host, a professor from the School of Public Health. My presenta
tion was delivered in about 45 minutes, and I fielded some questions from individ
uals who seemed interested in mixed methods research. After the presentation
ended, my host from public health asked me an intriguing question: “Are all of you
social scientists so abstract?” With this question, he inferred that I skimmed the
surface in a way that would not be appealing to practical researchers in the health
sciences. At that point I began rethinking mixed methods research to better em
phasize that yes, it was a science, and yes, there have been many applied scientif
Page 1 of 24
PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).
Subscriber: University of Western Ontario; date: 08 July 2021

ic developments in how to conduct mixed methods research in the past decade. I

vowed to begin reporting these developments.
After this workshop experience, I now believe that conveying information to those new to
mixed methods needs to start with commentary about its scientific developments. This
moves it away from abstract ideas to practical use. It also calls for identifying the precise
scientific developments that have occurred in the field, and, most important, branding
them as “scientific developments” for academic audiences. Unquestionably, much has
evolved in this area in mixed methods in the past few years. For example, we have specif
ic designs for the procedures of research, ways to represent the complexity of quantita
tive and qualitative data together, and visualizations or models of the major features
(p. 58) of our designs. It is precisely these technological advances, the scientific methods
that have evolved, that need to be highlighted in our mixed methods presentations and
writings.
Science is a good word to use because it resonates well with a health science audience
(as well as a social and behavioral science audience). Granted not all scholars, especially
some social scientists, think in terms of science and how it develops and proceeds for
ward. My premise is based on a core principle that science advances based on the accu
mulation of knowledge. I realize that this is only one way to conceptualize “research.” But
I believe that we have entered a new era in the development of mixed methods, one char
acterized by increasing complexities and technical features. Moreover, mixed methods
might be seen as the first major social science research methodology in the 20th and now
21st century to fully utilize the digital capabilities to advance it. Mixed methods is emerg
ing through digital flowcharts, the use of computer software for analysis, and the Internet
for reaching individuals around the world who may not have access to current books, con
ference workshops, and mixed methods content specialists.
Thus as I surveyed the needs of the fields, both in social sciences and health, after my
presentation at Harvard, I began speaking about distinct scientific developments in the
field that, if used by empirical investigators, would considerably enhance their scholarly
work. In this chapter, I hope to convey 10 different scientific developments in mixed
methods. Many of these are reported in existing diverse chapters and books on mixed
methods. For each one, I define the development, talk about current practices, and con
vey how the developments are continually being shaped as mixed methods becomes more
and more refined into a science. I do not present the developments in any special order,
but the 10 I include are (1) being able to identify the essential characteristics of mixed
methods research (MMR); (2) using a distinct terminology that helps to communicate
about the field and define it; (3) discussing the value-added of mixed methods over tradi
tional quantitative and qualitative research; (4) incorporating philosophies and theories
into mixed methods studies; (5) reassessing the types of mixed methods designs to focus
on a parsimonious set of basic and advanced designs that simplify the task of choosing a
design and conducting MMR; (6) recognizing methodological issues or threats to validity
that reside in the types of mixed methods designs; (7) stating a new type of question, a
mixed methods question, that is implied by investigators conducting a mixed methods
Page 2 of 24

study; (8) using representational models, called joint displays, to convey the quantitative
and qualitative results together in a study; (9) structuring the writing of a study to mirror
the specific type of designs used in the study; and (10) applying criteria to evaluate the
quality of a mixed methods study. Although these 10 points are not meant to be exhaus
tive and have certainly appeared in different publications over the past few years (e.g.,
Creswell & Plano Clark, 2011), they have never been pulled together as a set and spoken
of, as far as I know, as the scientific advancements in MMR. Their identification and label
ing as part of the science enables the larger social science and health science communi
ties to see them as distinct developments and to incorporate them into new studies. After
I discuss each one, I then turn to specific recommendations for future work that will build
on these developments in the ever-emerging field of mixed methods. I begin by recapping
what has developed in mixed methods over the past few years as a benchmark for new
scientific features.
The Expanding Field of Mixed Methods

These scientific developments I speak of did not exist five years ago. Partly this is due to
writers presenting mixed methods in a clearer voice and a more concise way (e.g.,
Greene, 2007). It is also due to having more and more mixed methods empirical studies,
journal articles, and funded projects reported in the literature (Onwuegbuzie, 2012). In
response, writers such as myself are learning from published journal articles. We now
have journals specifically devoted to MMR, such as the Journal of Mixed Methods Re
search (http://mmr.sagepub.com/), Field Methods (http://www.qualquant.net/FM/), and In
ternational Journal of Multiple Research Approaches (http://pubs.e-
contentmanagement.com/loi/mra). As a field, it has also expanded considerably through
methodological writings in different fields and in leading journals. It is quite popular in
many fields in the health sciences (Creswell, Klassen, Plano Clark, & Smith, 2011; Ivanko
va & Kawamura, 2010). It has also expanded internationally with interests in many re
gions of the world, such as in the African countries, in Europe, and in the Southeast Asian
countries. It might be seen by some as an Anglo-American methodology, especially in view
of the international conference, first housed at Cambridge University and then at Leeds
University in the UK, and recently coming back to (p. 59) the United States at Boston Col
lege in 2014 (https://www.facebook.com/MMIRA2014). Major books are being published
on mixed methods, with a large number—estimated to be 24—coming from international
authors (Onwuegbuzie, 2012). It has also expanded through interest by private founda
tions, through workshops, and the federal government (Plano Clark, 2010) and through
websites of “best practices”: advancing how to conduct rigorous mixed methods investi
gations (Creswell et al., 2011). We would add to this the new courses on MMR emerging
on many large, distinguished campuses across the United States and in England. In the
spring of 2014, Harvard hosted a mixed methods course in its Department of Global
Health and Social Medicine in the School of Medicine. During the summer of 2013, an in
ternational, interdisciplinary professional association began—the Mixed Methods Interna
Page 3 of 24

tional Research Association (mmira.wildapricot.org/)—to draw together the world’s mixed

methods writers, investigators, and researchers.
Scientific Developments
I begin with the most basic development to emerge—the definition of mixed methods—
and then proceed to identify additional developments that are more technical in nature.
A Science of Defined Boundaries
A science needs to have distinguishing features as well as ways of separating it from oth
er approaches. In recent years it has been contested terrain as to what the boundaries of
mixed methods might be. In 2007, more than 19 scholars provided their definitions for
mixed methods (Johnson, Onwuegbuzie, & Turner, 2007). Some took a philosophical ori
entation, some a methodological approach of thinking across many phases of research,
and some a methods orientation. I have personally taken a methods orientation because I
believe that it is a concrete way to understand mixed methods, especially for those new to
the idea. Beginning from the perspective of philosophy loses those who are not familiar
with the epistemological and ontological issues of inquiry. Talking about the process of re
search that stretches from the philosophical to the implications creates many questions
for those not specifically trained in a detailed understanding of the process such as draw
ing inferences or designing research questions. What is an important element of mixed
methods—the methods—gets lost in all of the phases of research that may not be under
stood in the same way by different investigators. In many fields, people can easily enter
their research from a methods perspective. It is concrete, and it is a specific place to
think about research. For example, in the health sciences, the idea of qualitative research
often resides in conducting focus groups. Of course, there is much more to qualitative in
quiry than that, but the methods become a framework for thinking about research. Also,
when asked what kind of research they are doing, some newer researchers respond that
they are conducting a survey or doing interviews. In short, the natural tendency to think
about research often comes from viewing it as a “method.” Thus my core characteristics
of mixed methods, as I call them, builds on thinking about mixed methods as a “method”
and “mixing” the methods:
Mixed methods is an approach to research in which the investigator collects, ana

lyzes, and interprets both quantitative and qualitative data (closed- and open-end
ed information), integrates or combines the two approaches in various ways, and
frames the study within a specific type of design or procedure. Sometimes the re
searcher makes specific their philosophical assumptions, and more often than not,
they include a theory that guides the quantitative or qualitative strand of their re
search or both. Also, both strands need to be conducted using rigorous methods of
data collection and analysis.
Page 4 of 24

So my definition works from a methods orientation. This allows me to clearly think about
mixed methods as a “method” and to distinguish it from other types of methods being
used in social, behavioral, and health research. For example, mixed methods is not simply
collecting and analyzing both quantitative and qualitative data. This view, however, falls
short because it does not incorporate a central feature of mixed methods—the integration
or combining of the two types of data in which, I believe, something extra is added to
both the quantitative and qualitative strand of a study. Also, mixed methods is not adding
qualitative data to a quantitative design. Mixed methods can be employed in this way, but
we can also add quantitative data to qualitative, and the “adding” component suggests a
diminished stature for what is added. In mixed methods, both quantitative and qualitative
approaches are highly and equally valued. Mixed methods further is not simply collecting
multiple forms of qualitative data (e.g., interviews and observations) or collecting multi
ple types of quantitative data (e.g., surveys, experimental data), (p. 60) although some re
searchers will take this perspective. To me, taking this perspective diminishes the power
of mixed methods, that something unique and creative will occur out of collecting two dif
ferent types of data (e.g., the quantitative trends and the qualitative personal experi
ences). Thus I say that mixed methods involves collecting both quantitative and qualita
tive data. Historically, when investigators collected multiple forms of quantitative data,
the term multimethod was used (Campbell & Fiske, 1959), not mixed methods. More re
cently, Johnson et al. (2007) found that major players in the mixed methods field tended to
collect both qualitative and quantitative methods. Others disagree, such as Morse and
Niehaus (2009) who state that “a mixed methods design may also mix two qualitative
methods or two quantitative methods.” (p. 20). Viewing mixed methods as mixing only
qualitative methods or only quantitative methods does not seem to capture the richness
that the two differing databases provide to understand research problems or questions.
In addition, mixed methods is not simply an evaluation technique, such as formative and
summative, although in employing this technique a researcher could collect and integrate
both quantitative and qualitative data. Distinct from evaluation procedures, mixed meth
ods is a systematic method, and it involves planning and consciously combining both
quantitative and qualitative data. Mixed methods is not collecting qualitative data and an
alyzing it quantitatively. This is the approach of content analysis (Krippendorff, 2012).
The full advantage of mixed methods follows from collecting both quantitative and quali
tative data. Finally, MMR is not “mixed model” research. Mixed model research is a quan
titative approach that involves including both random and fixed effects of errors in a sta
tistical analysis (Raudenbush & Bryk, 2002).
A Language Science
All sciences have means of communication within their groups and to the world of re
search that includes people not familiar with their approach. A language has developed
around mixed methods that researchers new to the approach must learn. At the back of
most mixed methods books are glossary terms, and these terms often are similar from
one text to another. A key term is the name mixed methods itself. Today, with the estab
lishment of the Handbook of Mixed Methods in Social & Behavioral Research (Tashakkori
Page 5 of 24

& Teddlie, 2010), the Journal of Mixed Methods Research, and the Mixed Method Interna
tional Research Association, we have the term mixed methods as a common label for this
form of research. It has been called other names—such as multimethod, integrated, or
mixed research—but the label “mixed methods” seems to be popularly used in the field.
Unquestionably, the terminology for mixed methods has evolved from the use of quantita
tive or qualitative terms (e.g., triangulation, a qualitative term, found its way into mixed
methods in the early years; Tashakkori & Teddlie, 1998), to the perspective of separate
terms from either quantitative or qualitative (e.g., legitimation; Onwuegbuzie & Johnson,
2006; Tashakkori & Teddlie, 2003). We now have moved toward more descriptive and less
confusing terms for those learning mixed methods. For example, take the terms priority
and sequence advanced by Morgan in 1998. In time, these terms have become confusing.
What does it mean to give priority to the quantitative or qualitative phase of a study? Do
we count the number of published pages given to each? Are the two approaches imple
mented sequentially when they are apart by minutes rather than days? A clearer term, I
believe, is to talk about the intent of the design, such as to compare two databases or
have one explain the other. This moves the language, I believe, onto much firmer grounds
for understanding the nature of a mixed methods design. In addition, entirely new terms
have entered the lexicon of mixed methods that were not present even a decade ago, such
as joint displays, mixed methods questions, and data transformation approaches.
A Legitimate Science
New sciences must legitimate themselves and gain acceptance as a valued approach to
research. Mixed methods is no exception, and it has been slowly building in this area.
Around 2006, the president of SAGE Publishing first called to my attention the issue of
whether mixed methods actually added any additional insight—or value—to a quantitative
or qualitative approach. Unfortunately, the question of the value-added of mixed methods
is still under scrutiny by many individuals in the research community, and mixed methods
scholars have not seemed to pose any concrete solutions to the problem. Being able to ad
vance its value is central to creating a legitimate science of mixed methods. In 2010 a stu
dent in my graduate program in mixed methods at the University of Nebraska–Lincoln ac
tually put to an empirical test this question (Haines, 2010). Student participants were
randomly assigned to (p. 61) one of three groups. Each group read a passage about re
sults—quantitative, qualitative, and mixed methods—reported in the same study (the
study was written specifically for this experiment). On a follow-up survey measuring the
value of the results, students who read the mixed methods passage perceived it to be
more valuable than the students who read the quantitative or qualitative passage. In a
qualitative follow-up to this experiment, the students reported that the mixed methods ar
ticle had rigorous methods and gave readers a deep meaning of the phenomenon under
study.
An alternative way to examine value-added would be to analyze closely what authors of

empirical mixed methods studies claim is the unique contribution of their approach. Far
quhar, Ewing, and Booth (2011) provided an excellent illustration. This article included a
table that specified how mixed methods added to the study, such as teasing out important
Page 6 of 24

elements of the intervention, providing limitations of the quantitative research, and using
quantitative findings to compare with quantitative results. Finally, we can discuss value-
added by reviewing the rationales for the use of MMR that have been stated by authors in
the mixed methods literature over the years. The benefit might be that qualitative data
helps explain the quantitative results, best explores the types of questions that need to be
asked, helps to shape a program or a set of intervention activities that might actually
work, or yields new variables that may not have occurred to researchers before the study
began or variables that were apparent in the literature (Bryman, 2006; Creswell & Plano
Clark, 2011).
In sum, we are still working on thinking and writing about the value-added of MMR, and
the question can be approached through multiple perspectives. We do know that a mixed
methods study typically involves collecting multiple forms of data that may be more than
either gathered in only quantitative or qualitative research. We also know that its use has
increased substantially in recent years (Onwuegbuzie, 2012), providing a subtext testimo
ny to its value.
A Science with Philosophical and Theoretical Foundations
Science is often developed first by philosophers who seek to establish the foundation for
the inquiry. Later the methodologists enter the picture and add the techniques for con
ducting the research. During the 1990s, the “paradigm debate” occupied the attention of
mixed methods writers (Howe, 2004; Reichardt & Rallis, 1994). Now, I believe, the de
bate has subsided, and thoughtful scholars are discussing useful philosophical approach
es that provide the foundation for mixed methods. This discussion has proceeded not
based on establishing the one true philosophy but on allowing researchers to adopt the
philosophy with which they are the most comfortable.
For example, in 2003 Tashakkori and Teddlie suggested that pragmatism formed the cen
tral philosophy advanced by 13 mixed methods scholars. More recently, Maxwell (2012)
discussed the critical realist approach that equally applied to qualitative and MMR. John
son (2012) suggested that dialectic pluralism is a good philosophy for MMR because it ad
vances an approach for incorporating different paradigms, theories, and stakeholder
views. New philosophies are emerging all the time, and a key question for a mixed meth
ods investigator is whether they make explicit in their studies their philosophical assump
tion. In the health sciences, it seems that there is neither room in publications nor inter
est in making explicit the philosophy undergirding mixed methods. In the social sciences,
the reverse is often true.
Theory, on the other hand, has found a central place in mixed methods for social, behav
ioral, and health scientists. I think about theory use in mixed methods as one of two
forms: social/behavioral theories (e.g., theory of adaptation, theory of change, and so
forth) or a theory that frames a social justice orientation or a transformative perspective
(Mertens, 2009). In the health sciences, the use of behavioral/social science theory popu
larly frames mixed methods studies. Examples would be quality-of-life theories, behav
Page 7 of 24

ioral change theories, and adoption and adaptation theories. For example, to study care
givers for family members affected by HIV and AIDS in South Africa, Petros (2011) used
social exchange theory; Püschel and Thompson (2011) used the PRECEDE model of pre
disposing factors, enabling factors, and reinforcing change factors; Ivankova and Stick
(2006) used a theory of student attrition in their investigation of study of student persis
tence in a distance education graduate program.
The social justice theoretical orientation may be one of transformation in which the intent
is to address injustices in the world and help bring about an improved society for under
represented groups (see Mertens, 2009, for the transformative perspective applied to
MMR). For example, Sweetman, Badiee, and Creswell (2010) reported on 39 mixed
(p. 62) methods studies that utilized a feminist lens, a racial orientation, and other frame
works. One theoretical transformative orientation helpful to stakeholders in communities

is community-based participatory research. In community health, especially, researchers
are incorporating mixed methods procedures using community-based participatory re
search. A good illustration of this approach would be the study of the transition of home
less individuals from a hospital to a shelter by Greysen, Allen, Lucas, Wang, and Rosen
thal (2012). In this study, the authors not only gathered quantitative and qualitative data
—they also involved the key stakeholders in both the shelters and the hospital in many
phases of the research. In the article, the authors make explicit how these stakeholders
participated in the phases of the research from the formation of the study and the impli
cations of its results.
We are beginning to learn how to write a theory—social, behavioral, or transformative—

into a mixed methods study (Mertens, 2003). This has resulted from a close inspection of
many studies available, such as the feminist mixed methods project looking at gender eq
uity by Hodgkin (2008). It might appear at the beginning as an overall framework for the
study or it might be threaded throughout the study and inform many parts of the project.
It might inform both the quantitative and qualitative strands of a mixed methods study,
such as its use by Betancourt et al. (2011) in their study of family strengthening in Rwan
da. In my writings with Plano Clark (Creswell & Plano Clark, 2011) we have discussed
how a transformative or advocacy theory might be threaded throughout a mixed methods
study, such as at the beginning in the literature review, in the advocacy/participatory
wording of the research question, in the sensitivity to the population in data collection
process, in the themes in the findings or results, and in the call to action at the end of a
study.
Scientific Basic and Advanced Designs
Throughout the development of research methodologies, writers have focused on the

types of procedures useful in conducting investigations. During the 1960s, Campbell and
Stanley (1966) identified the types of experimental and quasi-experimental designs that
are still prevalent today. In 1987, Jacob identified the primary types of designs available
to qualitative researchers. Thus, following in this history of the evolution of different
types of designs for procedures in conducting research, MMR has also proliferated into
Page 8 of 24

multiple designs. In fact, no topic has been more extensively discussed in the literature of
mixed methods than research designs (Creswell & Plano Clark, 2011). Many types of de
signs have been introduced over the years, and they come with different names, different
procedures, and diverse levels of complexity. We recorded 13 different classifications of
designs in 2011 (Creswell & Plano Clark, 2011). With this proliferation has also come in
creasing complexity, and I feel that now the beginning inquirer ends up with a baffling ar
ray of designs that limit his or her ability to choose one or understand how to conduct the
design. Researchers present too many mixed methods designs with a confusing array of
arrows and boxes that inhibit their replication or use by another person. My approach has
always been to think in terms of a parsimonious set of designs, to focus only on those de
signs popularly used, and to encourage researchers to seek out variants of them when
needed. The use of simple, elegant, straightforward designs has another benefit: a design
becomes an anchor in research for thinking about other aspects of the research process,
such as how to phrase the title, how to think about combining or integrating quantitative
and qualitative data, and how to consider issues such as validity and ethics.
The science of mixed methods can proceed by clarifying our designs and presenting them
in an understandable way. Thus I suggest that at the heart of every mixed methods study
resides a basic design, and figuring out that design is the first step for an investigator.
Further, these basic designs are only three types, and they form the core of the intent of
the mixed methods study. Sometimes we add on to these basic designs certain features
that encase them in more complex features. I call these advanced designs.
Three basic designs and three advanced designs are illustrated in Figures 4.1 and 4.2.
Parts of these designs are discussed elsewhere by different authors (e.g., Creswell,
2014a, 2014b; Morse & Niehaus, 2009; Teddlie & Tashakkori, 2009). They do not exhaust
the possibilities of designs, but they provide core models of designs within which investi
gators can innovate.
Fig. 4.1 Basic mixed methods designs.
Creswell, 2014a. Used with permission from SAGE

Publications.
Page 9 of 24

Fig. 4.2 Advanced mixed methods designs.
Creswell, 2014a. Used with permission from SAGE

Publications.
There are three basic designs, as shown in Figure 4.1: the convergent design, the ex
planatory sequential design, and the exploratory sequential design. The procedures of a
convergent design are to collect both quantitative and qualitative data, analyze both sets
separately, and then compare the results. Thus the intent is to compare two different
(p. 63) (p. 64) perspectives on a topic and see if the databases converge or are similar. The
procedures for an explanatory sequential design are to start by collecting and analyzing
quantitative data, look closely at the results, and then follow up on the results with quali
tative data collection and analysis. The intent is to help explain the quantitative results
with qualitative data. An exploratory sequential design takes just the opposite approach.
The researcher begins by collecting and analyzing qualitative data. From this analysis,
the researcher uses the results to develop something quantitative—such as an instrument
or an intervention—and then in the final phase of the research the investigator applies or
tests the quantitative instrument or intervention. The intent of this design is to explore
first because there are no existing instruments or intervention activities to study a popu
lation, or the existing concepts or variables cannot be adapted to a particular setting.
Most mixed methods studies fall into one of these three basic types or some variation of
them.
Advanced designs, as shown in Figure 4.2, include a basic design plus something more
added to the design. For example, an experimental framework, a social justice perspec
tive, or a program evaluation dimension might be added to a basic design. The intent of
adding qualitative data into an experimental trial is to first explore before the trial be
gins, to gather qualitative data about personal experiences during the trial, to follow up
after the trials with qualitative data to better understand the outcome results, or to plan
the next experiment (Creswell, Fetters, Plano Clark, & Morales, 2009). Alternatively,
added into a basic design might be a social justice framework that surrounds the mixed
methods procedures. The intent with this design is to advance a social justice agenda and
include the theoretical orientation of a lens that informs many of the phases of the re
search. In the third advanced design, the program evaluation design, the investigator pro
vides a longitudinal study involving many substudies, all connected to fulfill a common
Page 10 of 24

objective. These substudies may be a qualitative project, a quantitative project, or a

mixed methods project involving both forms of data, and with one study building on the
other. For example, in evaluating a program, the researcher might conduct a qualitative
needs assessment, design a program based on theory, implement the program, and quan
titatively test whether it works successfully. The intent is to use a basic design within the
framework of a long-term program evaluation objective. This design is easily seen in pro
gram evaluation mixed methods studies, such as the one by Nastasi et al. (2007) in their
multiyear study of a program implemented for youth in Sri Lanka.
It is difficult to say which designs are most popular, but in our analysis of mixed methods
projects in the mental health trauma area (see Creswell & Zhang, 2009), many examples
of the explanatory sequential design were found because the design starts with a quanti
tative phase, a phase easily accepted within the quantitatively oriented health sciences.
In the health sciences, popular designs include the intervention design (because of the
gold standard of randomized controlled trials), the community-based participatory re
search design (because of the need to provide health services to communities that they
will accept), and the program evaluation design, in which new health services are tested
out over time with patient and hospital populations. In the social and behavioral sciences,
the popular designs will differ and likely reflect the scholarly communities’ acceptance of
qualitative research or quantitative research. This acceptance will direct, in turn, the de
sign and what starts the introduction to the study and forms the basic feature of the de
sign.
What is also interesting about the emergence of designs is that we have good diagrams of
the procedures that researchers might use in their presentations or in their papers. These
diagrams might be presented as a simple model of a few boxes or elaborated with boxes
indicating the major elements, detailed procedures identified for each step, and specific
products to be developed at each step. Moreover, we can identify a flow of activities for
these designs, such as found in a useful book on visualizing social science research by
Wheeldon and Ahlberg (2012). Researchers can also add into these diagrams notation
that was first established by Morse in 1991 and has now been elaborated by others (see
Creswell & Plano Clark, 2011).
Scientific Validity Threats
When Campbell and Stanley wrote their classic book on experimental and quasi-experi
mental research in 1966, they not only assessed the types of experimental designs that
were being used; they also identified the threats to validity for the designs. I believe that
we are now taking a cue from their work in the mixed methods field as we begin to identi
fy the types of designs and pinpoint threats (or they might be called methodological is
sues) that arise when conducting a mixed methods design. Moreover, because we have
some sense (p. 65) of the types of designs, we also can now look at them closely, inspect
empirical investigations that use them, and make statements about the potential threats
that researchers need to anticipate when conducting the designs.
Page 11 of 24

Table 4.1 Threats to Validity in Six Mixed Methods Designs
Type of Design Potential Threats
Convergent De • Use parallel quantitative and qualitative

sign questions
• Decide on sample size
• Determine how to merge the data (side by
side, data transformation, joint display)
• Explain divergent results
Exploratory Se • Determine how to go from qualitative re

quential Design sults to quantitative instrument, variables,
etc.
• If developing an instrument, use good psy
chometric procedures
Explanatory Se • Assess array of possible quantitative re

quential Design sults to follow up
• Determine who can best provide qualita
tive follow-up
• Make sure that qualitative explains the
quantitative results
Experimental, In • Decide where/why to put qualitative data

tervention Design into experiment
• Make sure that qualitative does not bias
outcomes
• Consider challenges associated with basic
designs
Social Justice/ • Decide how social justice/community in

Participatory De volvement will flow into study and how to
sign negotiate this involvement
designs
Page 12 of 24

Program Evalua • Keep the focus on the long-term objective

tion Design
designs
What are some of these threats? As shown in Table 4.1, they differ for types of mixed
methods designs. These differences can be seen in sampling and sample size, time and re
sources, skills of researchers, and combining or connecting data. For example, in a con
vergent design, the potential threats lie in sample size and not being cognizant of
whether the sample sizes should be equal or unequal between the quantitative and quali
tative strands and at the integration stage of not carefully weighing alternative approach
es to integration. In an explanatory design, the issues reside in not clearly following up on
the quantitative results to explain them further in the qualitative data collection. Further,
this follow-up needs to consider both what types of questions to ask and which partici
pants can best answer the questions. In an exploratory design, a challenge lies in how to
translate the qualitative results into an instrument or into an intervention. Also, when in
struments are developed, a rigorous procedure of scale development must occur, such as
found in DeVellis (2006). In an intervention design, investigators need to not only identify
a rationale for inclusion of qualitative data in certain stages in the experiment but also re
alize how this qualitative data collection might bias the outcome results. We can place
these threats into designs and build on visual models of mixed methods designs such as
Wheeldon and Ahlberg (2012). Using the convergent design as an illustration, as shown
in Figure 4.3, I have placed the validity threats into a visual model of the procedures in
the mixed methods design. These threats relate to not creating parallel questions, not ad
dressing equal or unequal sample sizes, not treating results separately, not conveying the
approach to merging the data, and not explaining discrepancies between the quantitative
and qualitative data when they occur. In this way, by recognizing threats to validity, inves
tigators can see where they may occur, anticipate them, and present strategies for ad
dressing them.
Science of Linking Designs to Questions
Fig. 4.3 Validity threats comparing quantitative and

qualitative results (convergent design).
Page 13 of 24

Researchers know that the methods are intended to answer specific research questions
or hypotheses. It is clear in quantitative research that the research questions focus on
such factors as determining if a treatment works or what variables might influence an
outcome. In qualitative research, the questions often have to do with exploring the mean
ing of a phenomenon, understanding how (p. 66) people experience something, or what
theory can be generated that explains a broad process. The types of questions posed in
quantitative and qualitative research can be easily identified. What types of questions are
best answered with mixed methods? This question is typically not squarely addressed in
the mixed methods literature. It was this thinking that led mixed methods researchers to
suggest that in mixed methods, we have a new type of research question, one that does
not appear in our research methods books: a mixed methods question. Discussions first
appeared in the Journal of Mixed Methods Research through an editorial (Tashakkori &
Creswell, 2007), and now we are beginning to see them in published studies and dis
cussed in texts (e.g., Creswell & Plano Clark, 2011). The key to identifying this mixed
methods question lies in relating it to a type of design. Thus there are different ways to
phrase this question, such as writing it from a more methods orientation to a content fo
cus or some combination (called “hybrids” in Tashakkori & Creswell, 2007). For example,
Classen et al. (2007) provide this example: “From a qualitative meta-synthesis, how do
the stakeholder perspectives, needs, and goals for safe and unsafe driving outcomes sup
port or inform the salient factors found in the FARS dataset?” (pp. 679–680). In this exam
ple, the qualitative component is represented by the comment about stakeholder perspec
tives, and the outcomes represent the quantitative component. We know that a conver
gent design is being used because it is reflected in the words “support” or “inform,”
which suggests that the researchers will compare two different databases. For an ex
planatory design, the question may be: How do the qualitative results help to explain the
quantitative findings? For an exploratory design, the mixed methods question might be:
How do the qualitative findings help to design an improved quantitative instrument (or in
tervention)? Or how are the qualitative findings from a small number of people general
ized to a larger sample from a population? Of course there are variations for each design,
but the central thought here is that the mixed methods question fits the design being
used in a study, and it is presented to audiences in a way they can understand.
The Science of Representation
More and more attention today is being given to how to analyze both the qualitative and
the quantitative data in tandem. This attention focuses on the question of integration
(Bryman, 2006; Plano Clark, Garrett, & Leslie-Pelecky, 2009). How do we merge, for ex
ample, text data from qualitative research with numeric data from quantitative data? Am
biguity exists as to how to accomplish this integration, what forms it might take, and
whether it (p. 67) relates to the type of design, the methods of data, or the interpretation
(Fetters, Curry, & Creswell, 2014). One way to think about integration from a data stand
point is to consider joint displays, a procedure of arraying the qualitative results (e.g.,
themes) against the quantitative results (e.g., scores) in a table, graphs, or a discussion.
The idea is that the two sources of data can be more easily compared if they reside in one
Page 14 of 24

place or side by side. They may more easily be seen as one database extending or explain
ing the other database. Li, Marquart, and Zercher (2000) provide a good visual of array
ing quantitative and qualitative results together in which they identified common themes
(or questions) and then positioned the quantitative and qualitative results in columns be
side each other. What they might have done is added an additional column to illustrate
points of divergence or convergence between the two databases for each of the themes.
Such a joint display works well for a convergent design, but it can also be presented for
other types of designs. In Table 4.2, I present a joint display that I developed for a hypo
thetical mixed methods study using an explanatory sequential design in an intervention
trial in which the qualitative data helps to explain the quantitative data. In this table, I
present data with results from the quantitative analysis and a column for the qualitative
data that helps to explain the quantitative results. This table illustrates that the joint dis
plays may not be complicated and difficult to understand. In a graph of geo-coded infor
mation the graph can illustrate both the quantitative density of a problem as well as pull-
out quotes from individuals in each density area. Computer software has aided us in cre
ating these joint displays. One qualitative software product, MAXQDA (Verbi Software),
for example, now has a pull-down menu for mixed methods, which can assist the re
searcher in creating joint displays of the data.
Table 4.2 Joint Display in an Explanatory Sequential Intervention De

sign
Quantitative Qualitative Follow-Up How Qualitative

Results Interviews Explaining Findings Helped to
Quantitative Results Explain Quantita
tive Results
No significant Follow-up interviews re The qualitative inter

difference be vealed that half of the views provided some
tween the in caregivers felt that the positive impacts of
tervention and intervention enhanced the intervention, im
the control their competence, confi pacts that would not
conditions on dence, coping, and abili have been gained if
the five stan ty to handle caregiving only the statistical re
dardized mea responsibilities. sults were considered.
sures
Caregivers also high
lighted the importance of
their interactions with
the peer helpers.
Page 15 of 24

Scientific Writing Structures
Since many published empirical mixed methods studies are now available in the litera
ture, we have good models for how to write and compose mixed methods studies and
what mixed methods components to put into them. I often talk about the “architecture” of
a research study (Creswell, 2014c). The science of mixed methods proceeds through
learning the structural properties of writing studies for publication. For example, we now
pay attention to the creation of good mixed methods titles, complete purpose statements,
multiple research questions (quantitative, qualitative, and mixed methods), detailed dis
cussions about mixed methods procedures, including the types of quantitative and quali
tative data and how they are integrated, and mixed methods references. There have been
some recommendations for how to publish mixed methods studies (Stange, Crabtree, &
Miller, 2006), especially when the journal reviewers call for short articles—around 3,000
to 6,000 words). When we established the Journal of Mixed Methods Research in 2007,
we allowed manuscripts of empirical investigations of 10,000 to 12,000 words, a luxury in
today’s journal environment. However, some ideas are developing as to how to write a
short mixed methods article, despite the multiple forms of data being collected and the
multiple analysis strategies. One approach is to review studies that use one type of design
—say the convergent design—and see how the authors wrote the articles for publication.
Another approach is to consider a mixed methods study as having (p. 68) three parts—the
quantitative study, the qualitative study, and the mixed methods study. Each could be pub
lished separately with cross-referencing from one article to the other so readers under
stand that all three articles are crafted from one mixed methods project. A good example
comes from an international publication on the use of mammogram screening in Chile
(Püschel & Thompson, 2011). This overall project was divided into three papers, with the
quantitative paper published in one journal (Püschel, Coronado, et al., 2010), the qualita
tive paper printed in another one (Püschel, Thompson, et al., 2010), and the third mixed
methods paper published in a third journal (Püschel & Thompson, 2011). By the dates of
publication, we can see that the quantitative and qualitative articles were probably
placed first in journals followed by the mixed methods article. Also, by inspecting these
three articles closely, we can begin to understand how an overall mixed methods study
might be abbreviated to fit a shorter word length. This was accomplished by Püschel and
Thompson by abbreviating the methods discussion, not reporting both the quantitative
and qualitative results but combining them, and casting both the quantitative and qualita
tive findings within one central theory.
Scientific Evaluation Criteria
Finally, standards for evaluating the quality of a mixed methods study are now being pro
duced. There are certainly advantages and disadvantages to using guidelines for stan
dards. They help beginning researchers navigate the terrain of mixed methods, whereas
more experienced researchers may see them as creating boundaries on how they view
and conduct mixed methods. Standards should not be seen as rigid templates but as gen
eral guidelines for use. In the mixed methods field, several authors have created useful
guidelines (see O’Cathain, Murphy, & Nicholl, 2008), and, more recently, the federal gov
Page 16 of 24

ernment has issued some quality criteria. For example, the National Science Foundation
has a document that provides guidelines for mixed methods evaluation research
(Frechtling, 2002), and the medical community has the CONSORT guidelines (Ioannidis et
al., 2004) for guidance for composing a randomized trial. The National Institutes of
Health (Office of Behavioral and Social Science Research; Creswell et al., 2011) has pro
vided recommendations on a website for the “best practices” of mixed methods in the
health sciences ((http://obssr.od.nih.gov/mixed_methods_research/pdf/
Best_Practices_for_Mixed_Methods_Research.pdf). A checklist in the best practices rec
ommendations addresses several scientific aspects of mixed methods that I have dis
cussed, such as integration, a rationale of the use of mixed methods, and how it adds val
ue to a study beyond that gained from either quantitative or qualitative research. Of spe
cial note in best practices would be the table of review criteria, useful for individuals
preparing an application for the National Institutes of Health or individuals who are eval
uating such applications. How rigidly we need to specify how mixed methods works is, of
course, open to debate, and it seems to vary depending on whether the field is in the
health sciences or the social sciences. However, I do find that beginning mixed methods
researchers appreciate some quality guidelines as they develop graduate student propos
als for theses and dissertations, conference presentations, journal article submissions,
and applications for private and public funds. Application developers and reviewers on
study panels also stand to profit from quality guidelines.
Discussion and Recommendations

It is helpful to talk about the “science” of mixed methods and to discuss how our knowl
edge has accumulated about the scientific developments to emerge over the past few
years. Some will disagree with me that we need a science of mixed methods, or even that
it is a scientific approach. For the health sciences in which I have been working, as my ex
ample of my presentation and comments that I received at Harvard illustrates, it makes
good sense, and it enhances the credibility and legitimacy of the approach as well as
highlights procedures for using the approach in a practical way (Stewart, Makwarimba,
Barnfather, Letourneau, & Neufeld, 2008). Building on the 10 points of the science of
mixed methods, my recommendations are as follows:
1. For individuals new to mixed methods, such as those flowing into the field from
the health sciences today, we need a commonly accepted definition of mixed methods
and one that clearly sets the boundaries between what is and what is not this form of
research.
2. The terms in mixed methods continue to improve, and we need to clarify some
misconceptions that arose during the early development of the methodology. Glos
saries and terms used should reflect a language unique from quantitative or qualita
tive words and embrace new terms that are coming into frequent use such as a “joint
display” or a “mixed methods question.”
(p. 69) 3. We need more empirical tests as well as clearer rationales presented in the
empirical articles about how the mixed methods procedure actually added compo
Page 17 of 24

Cresswell - 2015 - Revisiting Mixed Methods and Advancing Scientific PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cresswell - 2015 - Revisiting Mixed Methods and Advancing Scientific PDF

Uploaded by

Copyright:

Available Formats

Revisiting Mixed Methods and Advancing Scientific Practices

Revisiting Mixed Methods and Advancing Scientific

Print Publication Date: Aug 2015

Abstract and Keywords

Subscriber: University of Western Ontario; date: 08 July 2021

ic developments in how to conduct mixed methods research in the past decade. I

Subscriber: University of Western Ontario; date: 08 July 2021

The Expanding Field of Mixed Methods

Subscriber: University of Western Ontario; date: 08 July 2021

tional Research Association (mmira.wildapricot.org/)—to draw together the world’s mixed

A Science of Defined Boundaries

Mixed methods is an approach to research in which the investigator collects, ana­

Subscriber: University of Western Ontario; date: 08 July 2021

Subscriber: University of Western Ontario; date: 08 July 2021

An alternative way to examine value-added would be to analyze closely what authors of

Subscriber: University of Western Ontario; date: 08 July 2021

A Science with Philosophical and Theoretical Foundations

Subscriber: University of Western Ontario; date: 08 July 2021

works. One theoretical transformative orientation helpful to stakeholders in communities

We are beginning to learn how to write a theory—social, behavioral, or transformative—

Scientific Basic and Advanced Designs

Throughout the development of research methodologies, writers have focused on the

Subscriber: University of Western Ontario; date: 08 July 2021

Fig. 4.1 Basic mixed methods designs.

Creswell, 2014a. Used with permission from SAGE

Subscriber: University of Western Ontario; date: 08 July 2021

Fig. 4.2 Advanced mixed methods designs.

Creswell, 2014a. Used with permission from SAGE

Subscriber: University of Western Ontario; date: 08 July 2021

objective. These substudies may be a qualitative project, a quantitative project, or a

Scientific Validity Threats

Subscriber: University of Western Ontario; date: 08 July 2021

Table 4.1 Threats to Validity in Six Mixed Methods Designs

Type of Design Potential Threats

Convergent De­ • Use parallel quantitative and qualitative

Exploratory Se­ • Determine how to go from qualitative re­

Explanatory Se­ • Assess array of possible quantitative re­

Experimental, In­ • Decide where/why to put qualitative data

Social Justice/ • Decide how social justice/community in­

Subscriber: University of Western Ontario; date: 08 July 2021

Program Evalua­ • Keep the focus on the long-term objective

Science of Linking Designs to Questions

Fig. 4.3 Validity threats comparing quantitative and

Subscriber: University of Western Ontario; date: 08 July 2021

The Science of Representation

Subscriber: University of Western Ontario; date: 08 July 2021

Table 4.2 Joint Display in an Explanatory Sequential Intervention De­

Quantitative Qualitative Follow-Up How Qualitative

No significant Follow-up interviews re­ The qualitative inter­

Subscriber: University of Western Ontario; date: 08 July 2021

Scientific Writing Structures

Scientific Evaluation Criteria

Subscriber: University of Western Ontario; date: 08 July 2021

Discussion and Recommendations

Subscriber: University of Western Ontario; date: 08 July 2021

You might also like

Mixed methods is an approach to research in which the investigator collects, ana

Convergent De • Use parallel quantitative and qualitative

Exploratory Se • Determine how to go from qualitative re

Explanatory Se • Assess array of possible quantitative re

Experimental, In • Decide where/why to put qualitative data

Social Justice/ • Decide how social justice/community in

Program Evalua • Keep the focus on the long-term objective

Table 4.2 Joint Display in an Explanatory Sequential Intervention De

No significant Follow-up interviews re The qualitative inter