Advancing Mixed Methods in Psychological Research

See discussions, stats, and author profiles for this publication at: https://www.researchgate.
net/publication/344711955
Advancing Mixed Methods in Psychological Research
Article in Methods in Psychology · October 2020

DOI: 10.1016/j.metip.2020.100035
CITATIONS READS
0 71
2 authors:
Elizabeth G. Creamer David Reeping

Virginia Polytechnic Institute and State University Virginia Polytechnic Institute and State University
123 PUBLICATIONS 896 CITATIONS 35 PUBLICATIONS 71 CITATIONS
SEE PROFILE SEE PROFILE
Some of the authors of this publication are also working on these related projects:
Introduction to Fully Integrated Mixed Methods Research View project
First-Year Programming View project
All content following this page was uploaded by Elizabeth G. Creamer on 17 October 2020.
The user has requested enhancement of the downloaded file.

Methods in Psychology
--Manuscript Draft--
Manuscript Number: METIP-D-20-00008R2
Full Title: Advancing Mixed Methods in Psychological Research
Short Title: Advancing Mixed Methods in Psychological Research
Article Type: Research Article
Section/Category: Mixed Methods
Keywords: Mixed Methods; Psychology; multi-method research
Corresponding Author: Elizabeth Greene Creamer, EdD

Virginia Polytechnic Institute and State University
Blacksburg, VA UNITED STATES
Corresponding Author Secondary

Information:
Corresponding Author's Institution: Virginia Polytechnic Institute and State University
Corresponding Author's Secondary

Institution:
First Author: Elizabeth Greene Creamer, EdD
First Author Secondary Information:
Order of Authors: Elizabeth Greene Creamer, EdD
David Reeping, PHD
Order of Authors Secondary Information:
Abstract: The aim of this methodological commentary with a research component is to promote
informed uses of mixed methods in psychological research. We used a concurrent
mixed methods research design to compare the insights gained about the use of mixed
methods in psychological research from two quantitative prevalence studies with
information gleaned from a qualitative, meta-synthesis of data supplied in 11 articles
and chapters that reviewed its uses in psychological research. It is likely that the
dispersion of the examples across diverse publishing venues obscures any sense that
it is a mainstream methodology in psychology.
Suggested Reviewers: Cheryl Poth, PHD

Professor
cpoth@ualberta.co
Poth is on the editorial board; her advanced degree is in educational psychology; she
is very well versed in mixed methods and qualitative oriented mixed methods
John Hitchcock, PHD

Independent contractor
jhhitchc@gmail.com
John recently left a tenured position at the University of Indiana. He now works as an
independent contractor. He is co-editor of the International Journal of Multiple
Research Approaches and has a wide publication record. His expertise is quantitative
and mixed methods.
Hisako Kakai, PHD

Professor, Aoyama Gakuin University, Tokyo, JAPAN
hisako.aloha@gmail.com
Dr. Kakai is professor in Japan and a senior scholar who knows a good bit about mixed
methods and grounded theory. She translated one of John Creswell's book into
Japanese.
Tim guetterman, PHD

Associate Professor, Creighton University
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
tcguetterman@gmail.com
Tim just transitioned from a faculty position at the University of Michigan College of
Medicine to the Creighton University in Nebraska. He is very well regarded in the
mixed methods community and a very strong researcher and methodologist.
Cherie Edwards, PHD

Research Methodologist, Virginia Commonwealth University Medical Center
cdedward1@gmail.com
Cherie is an early career researcher with a specialization in mixed methods
methodology. She works at as a research methodologist at the medical college at
VCU. She writes extensively about visual methods.
Opposed Reviewers:
Response to Reviewers:
Additional Information:
Question Response
Please enter the Word Count of your 10536

manuscript
Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation
Response to reviewer comments
Methods in Psychology Summary of Revisions
Reviewer 2. Asked that we add to our statement about positionality what was gained by our two
different paradigmatic orientations. This statement was added on page 4:
The overlap in our expertise kept the logic of mixed methods at the forefront. The
differences in our expertise put us in a better condition to exploit the dissonance we encountered
and to weigh the evidence impartially as possible.
Reviewer 2 asked that we clarify our position about whether multiple QUALS can be
mixed methods. We added this language:
We take the position that the distinction between mixed methods research and
multimethod research is the purposeful integration of the research procedures and support
the logic of the argument that as long as they are integrated in meaningful ways, mixed
methods can extend to multiple qualitative approaches
Reviewer 2 asked us to make a statement about the prevalence of multimethod research

in psychological fields. We added this language:
Given the prominence of experimental research in many fields in psychology, as well as

the priority given to instrument development, it is likely that there is more multimethod
than mixed methods research in psychological fields.
Reviewer 2 asked for clarification of the language “partially mixed.” We revised some
wording and added this sentence:
Powell et al. applied the language of “partially mixed” to articles that provided no
evidence of procedures to integrate qualitative and quantitative strands during analysis.
They considered the majority
Both Reviewer 2 and Reviewer 3 questioned, correctly so, if we had really carried
through with the promise to talk about innovative examples of the use of mixed methods
in psychological research. Both reviewers suggested that word be removed from the
abstract and purpose.
We struck the work from the abstract and purpose statement.
Reviewer 3 questioned a statement in the discussion section about the role of editors and
reviewers as gate keepers because only one reference was given and it is over ten years
old.
Uneasiness about what is perceived by some as the incompatibility between the
paradigmatic grounding of qualitative and quantitative approaches is probably one reason
that underlies wariness about the receptivity of editors and reviewers in some journals to
mixed methods manuscripts (Alise & Teddlie, 2010; Pearce, 2012; Powell et al., 2008). T
Reviewer 3 asked the we add a footnote to Table 3 to explain the symbols.

Reviewer 3 questioned the section about triangulating that was added to the latest version
of the manuscript but that was not in the first version. He/she commented that this
section.
is now out-of-date and that it could be misleading because triangulation is no longer the
principal reason for using mixed methods.
Upon reflection, we decided we agree with the reviewer and removed the section.
Reviewer 3 asked that we tighten up the introduction, so that the purpose be cleared
sooner.
In response, we moved a section out of the introduction that was about historical
context and added it to the section where we defined mixed methods.
Title Page with author information
Corresponding Author:
Dr. Elizabeth G. Creamer, Professor Emerita
Educational Research Program, School of Education
Blacksburg, VA 24060
creamere@vt.edu
Dr. David Reeping, Postdoctoral Associate
Bradley Department of Electrical and Computer Engineering
College of Engineering | Virginia Tech
Blacksburg, VA 24060
dreeping@vt.edu
Blinded Manuscript without author info Click here to view linked References
Mixed Methods in Psychological Research
More than one journal in psychology welcomes submissions that explore the implications
of methodologically innovative applications of mixed and multiple methods – including
Frontiers in Psychology and the new journal published by the U.K. publisher Elsevier, Methods
in Psychology. Rather than endorsing the idea that mixed methods approaches are novel to
psychological research, these invitations reflect an understanding that the logic of mixed
methods is neither absent nor new to the discipline. It is possible that, like the situation in
sociology, the language of “mixed methods” had been downplayed in the formal discourse
because there is concern that the terminology is controversial (Pearce, 2012). Wariness about the
receptivity of journal editors and reviewers to mixed methods (Alise & Teddlie, 2010) may not
deter researchers from using the methodology. Still, it may discourage them from submitting a
manuscript described as mixed methods to some journals for fear of irking skeptical editors and
reviewers.
A commentary from editors of the mixed methods research topic in the journal of
Frontiers in Psychology suggests that methodological priorities in the discipline are shifting. The
authors take a stance that departs from the common language concerning methodology in
psychology. They argue it has become “obligatory to use mixed methods for research, not only
in psychology but in practically all branches of the social sciences” (Anguera et al., retrieved
October 23, 2019). It is possible that the perception that mixed methods is “obligatory” may
reflect skepticism that enthusiasm for seeming novelty in research design may be overshadowing
judgment about the design’s alignment with the purpose of the research (e.g., Chamberlain, Cain,
Sheridan, & Dupuis, 2011; Frost & Shaw, 2015). The commentary from the editors from
Frontiers includes a call for more “order,” “guidelines,” and “protocols.” They seem to be
1
joining Plano Clark (2010) in expressing uneasiness about the lack of consensus that exists
among mixed method practitioners about such fundamental issues as those related to its
philosophical foundations.
1.1.Purpose and Contribution
This critical methodological commentary aims to promote informed and innovative uses
of mixed methods in psychological research and to disrupt the perception that mixed method
approaches are rarely used in psychological research. The audience for the article includes
potential authors who have set out to develop expertise in mixed methods research, but also
instructors committed to exposing graduate students to a pluralistic array of research approaches.
Researchers may find that by reading this article that the research they have been doing or plan
to do could be characterized as mixed methods.
The American Psychological Association (APA) defines psychology as the study of
mind, how it works, and how it affects behavior. Psychological research examines psychological
constructs as dependent and independent variables and uses implicit or explicit conceptual
frameworks (Waszak & Sine, 2003). The field’s approach to producing knowledge about the
concepts it studies is frequently considered to be homogenous, however. Academic psychology
in the United States consistently has been characterized as a quantitative discipline firmly linked
to post-positivism that has long favored experimental and quasi-experimental methods (Alise &
Teddlie, 2010; McCrudden, Marchan, & Schutz, 2019; Waszak & Sines, 2003). The affinity for
such approaches is strong enough that might seem to be inhospitable to both qualitative and
mixed methods approaches.
We situate the uses of mixed methods in psychological research within a broader context
about how the methodology has been advanced by researchers in both the USA and the wider
2
global context. Our aim in this critical methodological commentary is compare the insights
drawn about the use of mixed methods from quantitative prevalence studies with information
gleaned from qualitatively analyzing articles or chapters that review examples of how mixed
methods were used in psychological research (i.e., Bartholomew & Brown, 2012; Breen &
Darlaston-Jones, 2010; Doucerain, Vargas, & Ryder, 2016; Dures, Rumsey, & Gleeson, 2010;
Hanson, Plano Clark, Petska, Creswell, & Creswell, 2005; Harkness et al., 2006; McCrudden, et
al., 2019; Powell, Mihalas, Onweugbuzie, Suldo, & Daley, 2008; Tashakkori, Teddlie, & Sines,
2011; Waszak & Sines, 2003; Yoshikawa, Kalil, Weisner, & Way, 2013). In the third phase of
the research design, we conducted a type of ‘review of reviews’ (see Smith, Devane, Begley, &
Clarke, 2011). As part of our approach, we integrate the qualitative and quantitative findings by
identifying topics where findings drawn from quantitative and qualitative methods are in accord,
as well as where they are contradictory.
The article is organized into four parts. In the first part, we review some key elements of
the definition of mixed methods research and what distinguishes it from multimethod research.
In the second part, we consider the insight afforded by two quantitative studies that reported on
the prevalence of mixed methods in two areas on psychology and what these suggest about its
use. In the third section of the article, we return to a qualitative orientation where we synthesize
findings from 11 articles and chapters that includes 108 examples of the use of mixed methods in
subdisciplines in psychology. In the fourth part, we summarize where the findings from the two
methods overlap, where they differ, and where they are contradictory.
1.2. Authors’ Positionality
The authors bring distinct, but overlapping, expertise and paradigmatic grounding to the
commentary. Both are well versed in the methodological literature about mixed methods. Both
3
authors identify as realists in their belief that there is an external reality, even as it is perceived
differently. The first author is qualitatively oriented and committed to the exploration of diverse
perspectives associated with the philosophical assumptions of dialectical pluralism (c.f., Johnson,
2012). The second author recently completed a dissertation using a fully integrated mixed
methods design (author2, 2019b). The second author is more quantitatively oriented and
identified most strongly with a pragmatic philosophical paradigm (c.f., Morgan, 2016) in
executing the dissertation. The overlap in our expertise kept the logic of mixed methods at the
forefront. The differences in our expertise put us in a better condition to exploit the dissonance
we encountered and to weigh the evidence impartially as possible.
2. Defining Mixed Methods Research and Evidence of its Emergence in Psychological
Fields
Definitions of mixed methods research share in the characterization of it as a pluralistic
methodology that integrates quantitative and qualitative approaches. This conceptualization
embraces two different analytical logics: an exploratory/hypotheses generating one and a
confirmatory/hypotheses confirmation one. Jennifer C. Greene (2007), an influential voice in
shaping the discourse, argued that a mixed methods approach is not simply the combination of
methods and types of data but a different way of knowing and making sense of the world – “a
mixed method way of thinking” (p. 208). In fact, the ability to look at data from different
perspectives is said to be a strategy that can offset bias (Fusch, Fusch, & Ness, 2018). Other
voices in the methodological literature, such as Small (2011), use the phrasing “logic of inquiry.”
Small (2011) uses this language to prevent simplifying mixed methods to analytical approaches
alone. Greene (2007) contends that mixed methods research is associated with a dialectical
mindset that assumes that the different paradigmatic assumptions associated with qualitative and
4
quantitative approaches necessarily produce different results. Beyond extensive efforts to codify
mixed methods designs, it is not a methodology driven by a well-established research protocol.
There is agreement among conviction among the proponents of mixed methods
that the integration of findings from different sources of data can be synergistic and that it
produces findings that are greater than what can be produced separately by its parts
(Fetters & Molina-Azorin, 2017). In integrated approaches, the value-added is not simply
additive but exponential. Bazeley (2018) defines integration as “purposeful
interdependence between different sources, methods, or approaches” (italics hers, p. 7).
Creswell and Plano Clark (2018) highlight integration as “the centerpiece of mixed
methods research” (p. 220). Accordingly, we define mixed methods research as a
systematic approach to data collection and analysis that integrates different sources of
data and quantitative and qualitative analytical procedures with the intention to engage
multiple perspectives in order to more fully understand complex social phenomenon.
2.1. Historical Context for the Emergence of Mixed Methods in Psychological Research
Some historical context is helpful to support the argument that mixed method research
has been an actor in psychological research both in the U.S. and the wider international context
for quite some time. Language and terminology associated with mixed methods as a distinct
methodology emerged initially in the mid-to-late 1980s (Creswell, 2015). In psychology, this
followed a “revival” of interest in qualitative methods (Leavitt et al., 2018). The terms “mixing”
or “mixed methods” appeared in the titles of the second generation of books published at about
the same time in both the United States and Britain (i.e., Brannen, 1992; Caracelli & Greene,
1989). The timing of the shift in language to recognizing “mixed methods” was similar for
psychology. In 2004 a title from a publisher in the U.K. explicitly referred to “mixed methods in
5
psychology” (i.e., Todd, Nerlich, McKeown, & Clark, 2004). References to “mixed methods” in
the title or abstract of articles began to occur with increasing frequency in all fields after 2006
(Timans, Wouters, & Heilbron, 2019).
2.2.Integrating in Mixed Methods
Some methodologists writing about mixed methods envision a research design with on-
going dialectical engagement between the qualitative and quantitative strands and purposeful
integration, particularly during analysis (author1, 2018; Bazeley, 2018). This conceptualization
applies to its role in theory development as well (author1). Integration is possible at all stages of
a research design. This could be the case, for example, by connecting sampling designs
(Onwuegbuzie & Collins, 2007), embracing a mixed perspective in the planning phase through
mixed research questions (Onwuegbuzie & Leech, 2006; Plano Clark & Baidee, 2010), and in
the researcher’s approach to interpretation.
Researchers can facilitate integration by switching between types of inquiry logics. In
their discussion of theory development in cross-cultural psychology, Frost and Shaw (2015) pick
up on an idea from Hiles (2012) about how a cyclical exchange involves moving back and forth
between inductive, hypothesis-generating logic and deductive, hypothesis-testing logic. The
earliest pairs of collaborators to publish information about mixed methods approaches in
psychology, Waszak and Sines (2003) observed: “The development of psychological theory
requires cycling between approaches” (p. 574).
Integration can also include a third type of logic referred to as abduction. This is where
multiple hypotheses are tested and systematically eliminated to explain why findings from
different data sources may not mesh (Reicherz, 2007). Bazeley (2018) asserted that an abductive
kind of give and take is characteristic of all research, not just mixed methods. “The development
6
of adequate theory of the mind requires cycling between approaches which, striving for
integration, avoid dichotomous (either or reductionist or relativistic) and therefore partial
accounts of phenomena,” Gelo and colleagues from Austria maintained (Gelo, Braakmann, &
Benetka, 2008, p. 287).
Pluralistic psychological research can tap into multiple sources of data that extend
beyond conventional, one-shot interviews (Chamberlain et al., 2011). Data can be collected in
naturalistic settings, including by using ethnographic methods that involve fieldwork and direct
observation. Integration can incorporate naturally occurring sources of data, like clinician’s
notes, meeting notes, and archival documents (author2, 2019a). Recent research has highlighted
ways that visual data can be incorporated productively (author1; Chamberlain et al., 2011),
including in mixed methods through video recordings or participant artwork (Shannon Baker &
Edwards, 2018) or extracting structure from a single source of data to create integrated figures
connecting different observations together to explore internal causality (author2, 2019b; author2,
2020). Visual data can also be used in combination with a case study, grounded theory, discourse
analysis, and other qualitative approaches (Frost et al., 2010). Taking psychological research
outside of the laboratory setting introduces the possibility of uncontrollable factors that are not
well-understood by the researcher ahead of time (Waszak & Sines, 2003). Creative approaches to
data collection can also allow for a more nuanced explanation of the complexities of human
behavior by allowing for the emergence of gaps, incongruities, and contradictions (Frost &
Shaw, 2015).
2.3.Multimethod Research
There are diverse views about what constitutes multimethod research and what
distinguishes it from mixed methods research (Anguera, Blanco-Villasenor, Losada, Sanchez-
7
Algarra, & Onwuegbuzie, 2018). Given the prominence of experimental research in many fields
in psychology, as well as the priority given to instrument development, it is likely that there is
more multimethod than mixed methods research published in psychological fields. Quantitative
researchers and experimentalists are more likely to be influenced by early writings about
multimethods than the literature about mixed methods (Mark, 2015). A mixed methods research
design will have tangible points of integration, while multimethod research designs will not
expose these. In other words, the inferences drawn from the methods employed in a multimethod
study are independent of one another.
A few methodologists writing about mixed methods take the position that the mixed
methods label extends to using multiple qualitative approaches (e.g. Frost & Shaw, 2015; Morse
& Niehaus, 2009), while the predominant view is that multimethod label is more appropriate. We
take the position that the distinction between mixed methods research and multimethod research
is the purposeful integration of the research procedures and support the logic of the argument
that as long as they are integrated in meaningful ways, mixed methods can extend to multiple
qualitative approaches.
2.4. Estimating the Prevalence of Mixed Methods Research, Including in Pyschological
Research
Research about the prevalence of mixed methods in a field or discipline, as well as more
qualitatively oriented articles and chapters that explore examples, are cornerstones of the
methodological literature about the use of mixed methods in many disciplines, including
psychology. Prevalence studies use a quantitative approach. They employ systematic search
procedures to identify trends in the literature by reporting on the percentage of empirical articles
in selected journals that can be categorized as using mixed, qualitative, or quantitative methods.
8
The large number of this type of publication in the literature makes it possible to track trends in
the use of different methodologies and to weigh the presence of it in psychology relative to other
fields. Attempts to calibrate the prevalence of mixed methods research in any discipline is
confounded by the practice of disaggregating the studies into component parts and reporting on
them separately (Archibald, Radil, Zhang, & Hanson, 2015).
Prevalence studies provide some indication of the prevalence of research that is
recognizable as mixed methods across academic disciplines, as well as within psychology.
Timans, Wouters, and Heilbron (2019) offer a fairly contemporary assessment of the prevalence
of research in all disciplines that can be characterized as mixed methods. They reported on a
comprehensive bibliometric study of journals indexed by Thompson Reuters Web of Science
between 1992 and 2016. Their research extended across 95 different journals and 241,521
articles. They place the current percentage of empirical mixed methods publications in all fields
as slightly more than 18%. Across other fields, however, a steady increase in the frequency of
language associated with mixed methods in titles and abstracts between 1992 and 2016 led
Timans and his colleagues to conclude that mixed methods “have achieved a significant degree
of institutionalization within the scientific field” (p. 196).
Two studies that reported on the prevalence of mixed methods in psychological research
both placed the percentage of articles using mixed methods as below the 18% rate reported by
Timans et al. (2019). A study published in 2010 reported that 7% of the empirical articles in top-
tier APA journals reporting on basic research integrated qualitative and quantitative methods
(Alise & Teddlie, 2010). Mixed methods are used nearly twice as frequently in journals with an
applied focus than those that prioritize experimental and theoretical work (Alise & Teddlie,
2010).
9
A second prevalence study published by Powell et al. (2008) indicates the percentage of
publications identifiable as mixed methods is higher in applied fields in psychology than
theoretical ones. They reported that 13.7% of the articles in counseling psychology were
identifiable as mixed methods.
2.5. Challenges Identified for the Use of Mixed Methods in Psychological Research
Some of the challenges presented by using a mixed methods approach are specific to
psychology, while others are portable across disciplines. The addition of qualitative methods to
the curriculum has been slower in psychology than in other fields in the social-behavioral
sciences (Tashakkori, Teddlie, & Sines, 2011). Others point to constraints that are not unique to
psychology. These include the persistent concerns about paradigm incompatibility (Hanson et al.,
2005), that extra time and resources are required to complete a mixed methods project
(McCrudden et al., 2019), a lack of transparency in reporting about how integration occurred
(Collins, 2015), and the view that mixed methods are overly prescriptive and burdened with a lot
of technical jargon (Frost & Shaw, 2015).
Povee and Roberts (2015) conducted a qualitative study about attitudes toward mixed
methods research among a small group of students and faculty in Australia. Although a healthy
number of participants expressed an interest in mixed methods, others expressed skepticism
about the motivations for its use. “Many” participants expressed reservations about the tokenism
of the qualitative component. Their concerns included cynicism about qualitative phases that
might be added to compensate for disappointing quantitative results.
A frequent limitation of the reporting about mixed methods in fields where qualitative
traditions have only gained momentum recently, is minimizing the investment of resources
devoted to the qualitative phase. The qualitative component often serves as an accessory rather
10
than making a substantive contribution to an explanatory framework (Frost & Shaw, 2015;
O’Cathain, Murphy, & Nicholl, 2008). One way that qualitative approaches are minimized is
through the frequent practice of converting narrative data to numeric values (Tashakkori et al.,
2011). A quantitative drive is maintained in other research designs as well, such as when a
deductive or hypothesis testing approach is applied to qualitative textual or observational data or
when findings rest primarily on reporting the frequency of themes or constructs (Hesse-Biber,
2010). It is also apparent in the common practice of using quotes from interviews to illustrate
quantitative results (Mark, 2015) without separate attention to their analysis. Some have argued
that mixed methods may aggravate the tendency to essentialize qualitative research (Archibald et
al., 2015). This kind of token treatment of the methodological and theoretical basis of different
qualitative research traditions underplays the potential for new theoretical insight when there is
an active and iterative interplay between findings emerging from the distinct traditions.
3. Research Methods
3.1. Data Collection
This research was conducted in three phases with two strands of data: (1) a content
analysis strand analyzed using a qualitative approach and (2) prevalence studies strand analyzed
using a quantitative approach. The first phase was used for data collection. During that phase, we
searched the literature to find every available commentary about the use of mixed methods in
psychological research, including in prevalence studies. We had two criteria for inclusion: the
reference to psychology was made in the title, abstract, and/or in the purpose statement of the
article. The second criterion was that there was a commentary about one or more examples of
mixed methods publications in the manuscript.
11
We located two prevalence studies (i.e., Alise & Teddlie, 2010; Powell et al., 2008) and
11 articles or chapters that included a commentary about exemplars in their discussion of the use
of mixed methods in psychological research. We define an ‘exemplar’ in this context to be a
publication chosen by the article and chapter authors to serve as a quality model for their
discussion about mixed methods. We considered additional publications about trends in the use
of mixed methids in fields in psychology in the development of our initial set of hypothesis (i.e.,
Breen & Darlaston-Jones, 2010; Frost & Shaw, 2015; Haverkamp, Morrow, & Ponterotto, 2005;
Hiles, 2012; Karasz & Singelis, 2009; Wiggins, 2011), but excluded them from the last phase of
our analysis because they did not feature any exemplars
3.2. Data Analysis
In phase two of the design, we synthesized information from the content analysis strand,
consisting of 11 commentaries, to explore historical trends in the use of mixed methods and to
identify the fields where its use has been reported. Second, to explore the estimated prevalence
of mixed methods research in psychology across different time spans as a historical comparison,
we summarized the two available systematic reviews drawing from nine mainstream journals in
the prevalence studies data strand. In our summaries, we first weighed the evidence of mixed
methods use in psychological fields relative to other disciplines in the social and behavioral
sciences by summarizing data about the institutional affiliation of the lead author. We further
gauged whether each set of authors devoted attention to classifying the exemplars by a
conventional set of mixed methods designs and/or gave weight to the matter of the compatibility
between the philosophical paradigms associated with qualitative and quantitative approaches.
We juxtaposed the summaries from the two strands to develop a preliminary set of themes to
direct the analysis in the third and final stage of analysis.
12
We used a meta-analytic approach in the third and final phase of the analysis. We
extracted information from 102 unique exemplars highlighted in the 11 commentaries. We
ultimately aggregated data from 108 articles because we discovered that 6 of the exemplars
published their results in more than one venue. We extracted the exemplars from sections of the
reviews signposted with phrases alluding to providing a tangible example of a mixed methods
design like “for example” or “Authors (YEAR) demonstrate.” If the review consolidated their
examples into a table, the table was used as the set of examples to extract.
The descriptions of the research design in each exemplar was classified using the review
authors’ description of the design or the review authors’ explicit classification. For example,
McCrudden et al.’s (2019) categorization was based on the core mixed methods designs from
Creswell & Plano Clark (2018). The authors, title, and journal were then recorded from the
reference list. We used the most recent revised typology provided by Creswell and Plano Clark
(2018): convergent, explanatory sequential, and exploratory sequential. These core designs are
then the ‘atoms’ that form more advanced designs – ‘compounds’ of two or more core designs.
We use the three core design typology in describing our results.
Each article cited in one of the commentaries by one of the authors was recorded in a
spreadsheet along with the following information: authors, country or countries of authors, title,
publication year, publication venue, mixed methods design type, whether the design was
experimental or quasi-experimental, and the research approach.
Once we extracted the publication information for the exemplars, we synthesized the
patterns using joint display analysis (Fetters, 2020). We consolidated the data we extracted from
the content analysis strand and juxtaposed the frequencies of the recorded constructs to create a
figure highlighting purposefully sampled examples to illustrate a particular theme. During this
13
stage of analysis, the initial themes from the summaries of the two data strands were merged, i.e.,
the two data strands were consolidated for the purpose of analysis (Fetters, Curry, & Creswell,
2013). We used the consolidated data to explore for convergence and divergence in the themes
from the available prevalence studies and commentaries. This resulted in a concordance table
summarizing our main points of the methodological commentary.
We summarize our research design in Figure 1 with a method flowchart (or procedural
diagram). The figure highlights integration. The shading at the center of the figure highlights the
ways that integration of the different sources of data and analytical approaches occurred.
---------------------------------
Insert Figure 1 About Here
--------------------------------
3.3 Integration in Our Approach
The fluidity of the arrows in the figure underscores the iterative nature of the research, and the
dynamic interplay between the qualitative and quantitative approaches both within and across the
phases of the research. Integration occurred in three points by: (1) generating themes by
intersecting the reviews of exemplars from the content analysis strand with the prevalence
studies [called blended themes (author1)], (2) extracting the exemplars from the reviews to
tabulate and characterize which publications are being presented as “exemplars,” [called
connecting (Fetters, Curry, & Creswell, 2013) or converting, and (3) comparing and contrasting
the available data sources and inferences drawn from them [called merging (Fetters, Curry, &
Creswell, 2013)].
14
4. Using Quantitative Studies to Build Themes to Explore Qualitatively
We turned to quantitative methods in the first phase of the analysis where we synthesized
trends in the use of mixed method approaches in fields in psychology from two prevalence
studies (i.e., Alise & Teddlie, 2010; Powell et al., 2008). We extracted data from these two
articles to add to a set of themes or propositions to explore in the next phase that involved a
qualitative content analysis. Both assessements left little room for doubt about the prevalence of
quantitatively dominant mixed method designs in psychological research. Both struggled to
develop a strategy to distinguish research that reported on using both qualitative and quantitative
methods from those that actually integrated the two in some meaningful way.
4.1. Phase 1 Finding: Mixed Method Research Reflects a Quantitative Priority
Findings from prevalence studies support the conclusion that there is a strong quantitative
priority in the empirical research that is identifiable as mixed methods in psychology. For
example, of the 600 articles analyzed by Alise and Teddlie (2010), 93% of the empirical articles
were classified as quantitative, 7% as mixed methods, and none as qualitative (Alise & Teddlie,
2010).
The results reported by Alise and Teddlie (2010) are supported by Powell et al.’s (2008)
analysis of mixed method articles appearing in four school psychology journals between 2001
and 2005. An onverwhelming majority of them reflected a quantitative dominance. These
authors coded 873 articles for both latent and manifest content, finding that 96.65% (44 of 46) of
the reviewed mixed methods studies demonstrated a quantitative priority. Powell et al. applied
the language of “partially mixed” to articles that provided no evidence of procedures to integrate
qualitative and quantitative strands during analysis. They considered the majority (56.67%) of
the articles reviewed to be partially mixed.
15
4.2. Phase 1 Finding: A Lack of Explicit Labelling
Authors of prevalence studies developed strategies to overcome the limitation that until
recently, most empirical research that combines qualitative and quantitative approaches was
rarely explicitly labeled as either “mixed methods” or “multi-method.”
For example, Alise and Teddlie (2010) used a unique, albeit time-intensive strategy, to
compensate for the problem that during the time span they analyzed (2005 and immediately
before) most mixed methods research was rarely flagged by that label. Rather than the more
conventional strategy of categorizing an article as mixed methods if the words “qualitative” and
“quantitative” appeared in the abstract, these authors used a conservative measure that required
an article to demonstrate “a high degree of integration of QUAL and QUANT methods” (p.
111). Each of 300 randomly selected articles was manually coded to determine which articles
combined qualitative and quantitative research procedures at any one phase. For example, a
study that combined probability and purposive sampling strategies was considered mixed
methods, as was one that combined data both in the form of words and numbers.
Finding that of the 873 articles that they classified as mixed methods that in “no case did
researchers explicitly label their study as representing mixed method research” (p. 305), Powell
et al. (2008) suggested that it is likely that the prevalence rate they reported under-estimated the
use of mixed methods in that subdiscipline. They observed that researchers may be turning
elsewhere, including outside of the discipline, to publish mixed methods studies.
16
4.3.Summarizing Themes From the First Phase of Analysis
Reviewing these prevalence studies and a wider body of publications about the use of
mixed methods approaches in psychology provided us with the following claims to explore with
a content analysis of reviews in psychology.
1. Qualitative and quantitative methods have been combined in psychology for decades.
2. Most research combining qualitative and quantitative methods is not explicitly labelled as
“mixed methods.”
3. Mixed methods have been reported more widely in applied fields in psychology.
4. Concern about the compatibility between qualitative and quantitative approaches remains
a salient issue in psychology.
5. Most mixed methods research is not appearing in the top-tier journals APA in
psychology.
6. Most studies in psychology that use mixed methods give priority to the quantitative
methods.
5. Weighing the Insight Afforded about Mixed Methods Approaches in Psychological
Research through Published Reviews of Exemplars
We extend our argument that mixed methods approaches are neither absent nor new in
psychological research by shifting the attention from the quantitative methods used in prevalence
studies to a qualitative approach that uses textual analysis. We conducted a type of meta-content
analysis by synthesizing data from 11 articles and book chapters published between 2003 and
2019 that considered 108 examples using mixed methods in psychology. We then quantitized
(Sandelowski, Voils, & Knafl, 2009) the findings to compare to the prevalence studies. Eight
findings emerged from this analysis.
17
Three of 11 articles appeared as introductions to special issues. The first special issue
about multiple research approaches, including mixed methods, appeared in the Journal of
Counseling Psychology in 2005 (i.e., Haverkamp, Morrow, & Ponterotte, 2005). A second
followed in 2009 in the Journal of Cross-Cultural Psychology (i.e. Karasz & Singelis, 2009).
Nearly ten years later, McCrudden and his colleagues produced an additional special issue
featuring research using mixed methods in Contemporary Educational Psychology (i.e.,
McCrudden et al., 2019).
Table 1 lists 11 publications that review the use of mixed methods fields in psychology.
The table is sorted by year, beginning in 2003. The table provides descriptive information about
each publication, including the authors, the geographic location of the first author, year of
publication, publication source, and subdiscipline. We also indicate in the table if attention is
devoted to addressing concerns about the paradigmatic incompatibility of qualitative and
quantitative and qualitative approaches or if an attempt was made to classify the examples using
conventional design typologies.
---------------------------------
Insert Table 1 About Here
------------------------------------
Authors of the content analyses listed in Table 1 provided a total of 102 examples of the
use of mixed methods in psychological research. They rarely explained how or why the
examples they featured were selected, but it is unlikely that the selections had anything to do
with the prestige of the publication venue. Only one of the articles is based on a systematic
review of the literature (i.e., Bartholomew & Brown, 2012). There is no doubt considerable
variability in the quality of the research methods used across the articles.
18
The review authors employed a wide range of strategies in discussing articles. For
example, Dures et al. (2010) used their own study as an exemplar to describe mixing methods
while Yoshikawa et al. (2012) drew from a variety of journals, conferences, and books to
illustrate different mixing strategies. McCrudden et al. (2019), Bartholomew and Brown (2012),
Hanson et al. (2005) discussed the exemplars at the design level. That is they provided they
identified examples by their specific design (i.e., explanatory sequential, exploratory sequential,
etc). On the other hand, Bishop (2014), Yoshikawa et al. (2012), Waszak and Sines (2003) and
Harkness et al. (2008) framed the examples from the perspective of purpose. For example, they
identified an example as ‘using mixed methods to develop and test a new questionnaire’ (Bishop,
2012, p. 12) rather than focusing on the design terminology or typologies.
We first examined these publications for additional evidence of the historical trajectory of
their appearance relative to other disciplines and the relative represention of authors from the
USA and abroad.
5.1. Qualitative and Quantitative Approaches Have Been Used Together for Decades in
Applied Sub-Disciplines in Psychology
Table 1 indicates that publications about the use of mixed methods have appeared
regularly, beginning just prior to the publication of the first book in psychology about mixed
methods in 2004 (i.e., Todd et al., 2004). This is precisely the point in time when mixed methods
was beginning to be formalized as a distinct methodology and other books were appearing by
authors in both the USA and the United Kingdom.
As supported by Alise and Teddlie’s (2010) analysis, the reviews largely involve applied
fields in psychology. They include counseling, clinical, counseling, educational, developmental,
health, and school psychology, with the most emerging on the topic of cross-cultural psychology.
19
Only one claims to address the discipline of psychology as a whole. Although 8 of 11 reviews
have a first author from the USA, the geographical dispersion of the lead author of these
publications reflects how mixed methods emerged on multiple fronts in a global context (c.f.
Fielding, 2010) that includes the United Kingdom, Canada, and Australia.
In Figure 2, we sought to explore the trajectory of mixed methods publications in
psychology by plotting the number of exemplars per year. We contrasted the timeline with the
range of years the review authors covered. We appended the exemplars that had particularly
notable qualitative approaches to show that nontokenized qualitative research is more prevalent
than prior research led us to believe. The reviews by Waszak and Sines (2003), Yoshikowa et al.
(2013), and Hanson et al. (2005) highlight that mixed methods research is not necessarily new to
the subfields of psychology, with most of the exemplars being published before the turn of the
millennium and as far back as 1956.
---------------------------------
Insert Figure 2 About Here
---------------------------------
There were considerable gaps in the number of exemplars per review among authors. At
the extremes, Powell et al. (2008) provided only one explicit exemplar and Hanson et al. (2005)
reviewed the most mixed methods studies – 22 articles. While Powell et al. (2008) only
explicitly critiqued one article, they tabulated the priority in the designs of 372 articles across
four journals in school psychology from 2001 to 2005. Given the differences in the number of
articles used by the review authors and their purposes in choosing the articles, it is reasonable to
assume variance in the range of years to which the exemplars belong as well. Waszak and Sines
20
(2003) covered the widest range of 1956-2000 while McCrudden et al. (2019) only reviewed
articles from a single year. None of the reviews we located used articles from 2015 to 2018 with
all the reviews published after 2010 only reviewing articles published after 2005, except
Yoshikawa et al. (2013).
5.2. Not all Exemplars Were Experiments: Not all Were Quantitatively Dominant
In our categorization of designs based on the review author descriptions, we found the
majority of the exemplars to be largely non-experimental – 74 observations. Our content analysis
of the research designs found that these exemplars often featured questionnaire-based designs
with interviews as a second form of data collection – observational methods were also appended
in select designs. This provided us with a finding to add to the six we identified through our
review of the prevalence studies.
7. Qualitative methods and non- experimental approaches are often used in mixed methods
studies in psychology reported in reviews of the literature.
Our joint display in Figure 2 shows that in constrast to the preliminary conclusions drawn
from the prevalence studies, several examples of mixed methods research featured in the reviews
did indeed have well-constructed qualitative approaches. Some exemplars intersected fully
developed qualitative methodologies with their quantitative approach rather than ad hoc or
thematic analyses indicative of an unequal priority design favoring the quantitative analyses.
Ethnographic follow-ups to experiments (e.g., Duncan, Huston, & Weisner, 2006) and the use of
grounded theory to conceptualize a phenomenon (e.g., Bishop, Massey, Yardley, & Lewith,
2011) were of particular note., with multiple uses of ethnographic methods, but these would go
unnoticed because of their scope to leading journals in psychology – especially considering their
range was only 2001-2005.
21
There were no striking patterns in design type between reviews or by year as most
attempted to balance their exemplars across different types of designs (e.g., Waszak & Sines,
2003). However, sequential designs were the most prevalent exemplars – with 46 observations.
Explanatory sequential designs were slightly more popular, with the quantitative phase leading in
25 of the 46 of the examples. When sequential designs are disaggregated, concurrent/convergent
designs were the most common – with 33 observations. This result shows that there exist several
examples of designs in psychology with qualitative dominance and equal priority – not phases
“tacked on” to compensate for lackluster experimental findings.
5.3. Mixed Methods Research in Psychology is Widely Dispersed in a Plethora of
Publishing Venues
Although conventional wisdom suggests that mixed methods research is difficult to
publish as a single unit, the exemplars identified in the reviews were often published as complete
products in a wide variety of journals. Only six projects explicitly split their results into multiple
publications, one of which split their results by nationality in their sample (see Harkness et al.,
2006). The exemplars represented 54 unique journals, with only 4 of the 108 appearing in the
methodologically oriented Journal of Mixed Methods Research. The most frequent journal was
the Journal of Counseling Psychology with 12 references, but this result is skewed because the
review by Hanson et al. (2005) drew heavily from this journal with 9 of his exemplars being
published in the Journal of Counseling Psychology.
While the reviewers tended to draw examples from their own countries, most reviews
included examples from authors in countries other than their own. Providing evidence of the
global attraction of mixed methods, authors from fifteen unique countries were represented.
Table 2 summarizes the countries each review represented in the choice of their exemplars.
22
---------------------------------
---------------------------------
This provides us with our last finding.
8. Examples of the use of mixed methods in psychology are disseminated in a wide range of
publication venues, including international ones.
6. Integrating Findings about the Uses of Mixed Methods in Psychological Research from
Quantitative and Qualitative Methods
We integrated findings from the qualitative and quantitative phases of our study to
construct an overview of the ways that mixed methods have been used with psychological
research. Table 3 is a joint display integrates the 8 findings that emerged during the first two
phases of analyses. The left column shows results from the quantitative prevalence studies, the
right column shows inferences from the qualitative meta-syntheses of articles and chapters, and
the center column shows their relationship. Only findings that link both approaches are listed.
We used both shading and a set of symbols to show how the findings relate to one another.
Shading indicates confirmatory results. The circular symbol designates five topics where there is
an agreement between the findings from two analytical approaches. The two-headed arrow links
one inference from the quantitative analysis and one from the qualitative that appear
contradictory. The third symbol in the last row of the table links two findings (7 & 8) that offer
complementary insight.
-------------------------------
------------------------------
23
The symbols used in Table 3 represent three of the five principal reasons for integrating
qualitative and quantitative data (Rossman & Wilson, 1985; Greene, Caracelli, & Graham,
1989). These are: (a) triangulation or confirmation, (b) complementarity, and (c) initiation. In 5
of 8 cases, findings from the qualitative and quantitative strands of the study confirmed each
other. These are:
1. Qualitative and quantitative methods have been combined in psychology for decades.
2. Most research in psychology using qualitative and quantitative methods is not explicitly
labeled as “mixed methods.”
3. Mixed method approaches have been reported most widely in applied fields in
psychology, like counseling psychology.
4. Concern about the compatibility between qualitative and quantitative approaches
remains a salient issue in psychology.
5. Most mixed methods research in psychology is not appearing in top-tier journals.
Two of the 8 inferences listed in Table 3 are complementary in that they create a more
nuanced understanding of the ways mixed methods have been reported in psychological research
when linked. We linked finding 7, that chapter authors were more likely to showcase exemplars
of mixed methods research that prioritized qualitative approaches, and finding 8, that the
research is disseminated in a wide range of non-mainstream publishing venues – both within and
outside of the USA. The link between findings 7 and 8 leads us to conclude that this type of
research is probably relatively invisible in the world of psychological research.
There was a single topic where inferences from the two analytical approaches led to
disparate conclusions. In the framing of purposes for using mixed methods, Greene et al. (1989)
24
referred to this as the initiation rationale because of the likelihood it would initiate additional
analyses.
6.1.Interpreting Divergent Findings
Findings from the qualitative and quantitative strands conflicted about genuinely reflected
qualitative analytical procedures (see Table 3). Results from a prevalence study that categorized
different design types documented that mixed methods were used most frequently with
experimental research and that the quantitative phase was awarded priority in the overwhelming
majority of cases. The meta-synthesis of the articles and book chapters reviewing exemplars of
the use of mixed methods in psychology pointed to a different priority. Despite the fact the
majority were launched with an initial quantitative phase, qualitative approaches were more
evident in the exemplars featured in the reviews.
Several factors might explain these seemingly divergent findings. One of these is a
difference in the purpose that motivated the research; the second is related to the visibility of the
research. Regarding purpose, prevalence studies consider trends in the literature by calculating
how frequently mixed methods were reported in journal articles in a discipline compared to
either quantitative or qualitative methods. On the other hand, the book chapters and articles we
considered in the meta-synthesis aimed to influence practice and promote the use of mixed
methods by highlighting unusual or innovative examples.
Differences between the findings from two analytical approaches are also evident in the
recognition and visibility of the publishing outlets. The two prevalence studies considered
evidence of the use of mixed methods in nine mainstream, content-oriented journals. The 108
exemplars were dispersed across 57 different journals and other publication venues and diverse
disciplinary areas and fields. Only four were published in a methodologically oriented journal.
25
The wide dispersion across so many publishing venues, including those outside of the USA, is
very likely to diminish the visibility of qualitatively oriented mixed methods research. The
exemplars drawn by the review authors show that nontrivial uses of qualitative approaches do
exist in psychology and did at the time of the prevalence study, but are so widely dispersed that it
makes them mostly invisible when the systematic review is restricted to publications in leading
psychology journals.
6.2. Philosophical Paradigm Raises its Hoary Head Once Again
There is another explanation for conflicting impressions about the presence of mixed
methods and qualitative methods in psychological research. Alise and Teddlie (2010) interpreted
the difference between the prevalence of quantitative, mixed, and qualitative methods in top-tier
journals in applied and basic fields as evidence of the continuing influence of the paradigm wars
that position qualitative and quantitative approaches as incompatible. They concluded:
“Quantitative methods and the underlying post-positivistic paradigm are prevalent in the articles
from ‘elite’ journals from pure disciplines, especially psychology” (p. 103). This same
conclusion is evident in the result that no studies using qualitative methods appeared in the top-
tier, APA journals Alise and Teddlie analyzed. This could reflect ways that in psychology the
status of mixed methods and qualitative research are intertwined. This statement is in line with
the opinion voiced by Rabinowitz and Weseen that the “heart” of the incompatibility or
paradigm question in psychology is “whether qualitative methods are scientific or valid” (2001,
p. 17). Given the critique from more than a decade ago that mixed methods is “a Trojan horse for
positivism” (Giddings & Grant, 2007, p. 52), it is possible that mixed methods proved a slightly
less controversial umbrella than qualitative methods to editors and reviewers in these journals.
26
Table 1 counters the claim provided by early leaders in the mixed methods movement
that the “paradigm wars are over” (Tashakkori & Teddlie, 1998, p. 5), particularly in
psychology. It provides data to gauge the persistence of concerns about the incompatibility of the
philosophical paradigms that underlie quantitative and qualitative approaches in the
methodological literature in psychology. Onwuegbuzie (2012) offered the explanation that the
concern about paradigm incompatibility has diminished over time because constructivist and
qualitative approaches have gained more prominence. It can be argued that academics whose
research emanates from the philosophical position of either post-positivism or constructivism are
likely to accept the idea that, as Cook and Reichardt (1979) proposed in early discussions about
mixed methods, that all research involves human judgment and is, therefore, subjective.
We found little evidence to support the assertion by Onwuegbuzie (2012) that discussion
about paradigm incompatibility has decreased over time. Six of 11 sets of collaborators
addressed the issue. Authors affiliated with institutions outside of the USA, where qualitative
approaches may be more mainstream, were no less likely to attend to the topic than those within
it.
7. Discussion
We set out in this article with the aim of advancing informed uses of mixed methods
approaches in psychological research. We integrated the insights gained from two quantitative
prevalence studies and a qualitative meta-synthesis of 11 reviews that reported on 108 examples.
The procedures we used confirmed that mixed methods has a longer history of use in psychology
than one might expect for a discipline so closely associated with quantitative methods. While we
contest the idea it is a short-lived fad, we found little evidence to support the assertion that a
mixed methods approach is now “obligatory” or pervasive in psychological research. The latter
27
claim is likely a reference to a growing expectation to add a qualitative component to
experimental research in psychology, as it has been done in randomized controlled trials in
health and medical fields. It is also possible that the 2019 reference to mixed methods as
“obligatory” may have been a reaction to the implementation of new standards of reporting by
APA in 2018 that included, for the first time, a section about transparency in reporting mixed
methods research (i.e., Levitt et al., 2018).
Insight from the two analytical procedures provides some preliminary indications of the
prevalence and uses of mixed methods relative to the wider community of social and human
sciences. While decidedly below the rates reported for applied fields in education and nursing,
two measures of mixed method’s prevalence pegging it at 7% for journals with a theoretical,
quantitative focus and 13.4% for journals with an applied focus in school psychology indicate
that it is not as absent as one might expect for a discipline with such an emphasis on
experimental research. The 7% rate is higher than might be expected given that the journals
sampled were top-tier, APA journals. Alise and Teddlie (2010) acknowledged that the journals
they sampled and “tend to be dominated by a more traditional approach in that field” (p. p. 121).
The integration of inferences drawn from the two analytical approaches extended our
understanding as well. Our “review of reviews” approach to extracting data from the examples
identified and the finding that no more than one or two appeared in the same publication venue
yielded new insight about its likely visibility within the discipline. Although mixed methods
research is not absent in high impact psychological journals, the reporting of mixed methods in a
diverse set of content area journals, including in health fields, no doubt diminishes the visibility
of mixed methods in psychology. This appears to be the case, especially for mixed methods
research with a qualitative emphasis. The absence of explicit labeling further diffuses any sense
28
of its pervasiveness in psychological research, which may contribute to the perception that there
are few outlets to publish mixed methods research in psychology.
Uneasiness about what is perceived by some as the incompatibility between the
paradigmatic grounding of qualitative and quantitative approaches is probably one reason that
underlies wariness about the receptivity of editors and reviewers in some journals to mixed
methods manuscripts (Alise & Teddlie, 2010; Pearce, 2012; Powell et al., 2008). This same
resistance can extend to reviewers and editors engaged in the mixed methods community who
prioritize different approaches to the methodology. Data supplied by Powell et al. (2008) about
variability in the prevalence rates of mixed methods articles across journals in a single discipline
(counseling psychology) supports the argument that this resistance is not necessarily disciplinary.
It confirms the perception that even in the same field, some journals may be considerably more
receptive to mixed method (and qualitative) research than others.
7.1. Limitations
The findings of the analyses reported here need to be weighed in light of several
limitations. The two prevalence studies about psychological fields we were able to locate both
used data that is now more than fifteen years old and is limited to articles appearing in nine
journals. The analyses overlook edited volumes showcasing mixed methods research by
researchers in psychology, such as the one produced by Weisner in 2005. The meta-syntheses we
conducted rests on articles where the review authors featured exemplars. Authors or chapters and
articles rarely provided a rationale for their choice of exemplars. An implication of this is that
there is, no doubt, considerable variability in the quality of the research methods.
29
7.2.Future Research
In addition to the challenge of establishing search terms that are effective in locating
research that used mixed methods with integrity, results from prevalence studies are like an
excavation in an archeological dig. They are time-sensitive in that they are always a backward
look at the use of mixed methods in a discipline or topic area at a defined time period.
Researchers interested in further fleshing out the landscape of the ways mixed methods
approaches have been used in psychological research could benefit by turning their attention to
doctoral dissertations. Now readily accessible through online repositories, doctoral dissertations
are likely the best prognosticator of how mixed methods will be used in the future in
psychological research.
8.Conclusions
There are several implications of the consistent finding across data sources that few
studies that combine qualitative and quantitative approaches explicitly refer to mixed methods as
the methodology. Creswell and Plano Clark (2006) have long advocated for using the expression
“mixed methods” in the title and abstract of a publication when appropriate. Interdisciplinary
research and an integration of a range of methods often go together (Weisner, 2016). Explicit
references to mixed methods as the methodological orientation is a “way to break down
disciplinary barriers and facilitate broader conversations” (Hay, 2016, p. xv). This is the type of
exchange promoted by the new journal, Methods in Psychology (authors blinded). While some
authors may be trying to be strategic by avoiding the use of the label, explicitly acknowledging
mixed methods as the methodology is a valuable discursive strategy that extends the audience
and visibility of the research beyond the narrow group of like-minded researchers who share
30
expertise in a content area. It is likely that the scarcity of this kind of labeling in psychological
research feeds an inaccurate conviction that mixed methods are “new.”
To close this methodological commentary, we return to the argument that explorations
using mixed methods are not absent from research in psychology. Such endeavors are hardly
new to the discipline, but there is significant variability across journals in the rate they are
reported. The visibility of mixed methods research in the discipline may be limited, particularly
for qualitatively oriented mixed methods research. This may be because researchers have turned,
as Powell et al. (2008) proposed, outside of the discipline to find publication venues that are
receptive to diverse methodologies. It may also be attributable to the practice of dividing up
qualitative and quantitative findings in separate publications in ways that downplay integration
(Archibald et al., 2015). Researchers in psychology who are anxious to find publishing venues
that are open to mixed methods do well to scour the aims statement of journals to find those that
invite innovation, encourage interdisciplinary scholarship, and acknowledge an interest not only
in quantitative approaches, but in qualitative and mixed methods research as well.
31
9.References
Author, 2016
Author, 2018a
Author, 2018b
Anguera, M. T., Blanco-Vilasenor, A., Jonsson, G. K., Losado, L., & Portell, M. (n.d.). Best
practice approaches for mixed methods research in psychological science. Retrieved
October 23, 2019 from: https://www.frontiersin.org/research-topics/8170/best-practice-
approaches-for-mixed-methods-research-in-psychological-science
Anuguera, M. T., Blanco-Villaseñor, A., Losada, J. L., Sánchez-Algarra, P., & Onwuegbuzie,
A. J. (2018). Revisiting the difference between mixed methods and multimethods : Is it
all in the name? Quality and Quantity: International Journal of Methodology, 52(6),
2757-2770.
Alise, M. A., & Teddlie, C. (2010). A continuation of the paradigm wars? Prevalence rates of
methodological approaches across the social/behavioral sciences. Journal of Mixed
Methods Research, 4 (2), 103-126.
Archibald, M. M., Radil, A., Zhang, X., & Hanson, W. E. (2015). Current mixed methods
practice in qualitative research: A content analysis of leading journals. International
Journal of Qualitative Methods, 14(2). DOI:
https://journals.sagepub.com/doi/pdf/10.1177/160940691501400205
Bartholomew, T., & Brown, J. R. (2012). Mixed methods, culture, and psychology: A review of
mixed methods in culture-specific research. International perspectives in psychology:
Research, practice, and consultation, 1(3), 177-190.
32
Bazeley, P. (2018). Integrating analyses in mixed methods research. Thousand Oaks, CA: Sage.
Bishop, F. (2014). Using mixed methods research designs in health psychology: An illustrated
discussion from a pragmatist perspective. British Journal of Health Psychology, 20, 5-20.
Bishop, F., Massey, Y., Yardley, L., & Lewith, G. T. (2011). How patients chose acupuncturists:
A mixed methods project. Journal of Alternative Complementary Medicine, 17(1), 19-25.
Brannen, J. (1992). Mixing methods: Qualitative and quantitative research. Routledge.
Breen, L., & Darlaston-Jones, D. (2010). Moving beyond the enduring dominance of positivism
in psychological research: Implications for psychology in Australia. Australian
Psychologist, 45 (1), 67-76.
Caracelli, V., & Greene, J. C. (Eds.). (1989). Advances in mixed-method evaluation: The
challenges and benefits of integrating diverse paradigms. San Francisco, CA: Jossey-
Bass.
Chamberlain, K., Cain, T., Sheridan, J., & Dupuis, A. (2011). Pluralism in qualitative research:
From multiple methods to integrated methods. Qualitative Research in Psychology, 8,
151-169.
Authors blinded
Collins, K. M. T. (2015). Validity in mixed method and mixed research. S. Hesse-Biber and R.
B. Johnson (Eds.), The Oxford handbook of multimethod and mixed methods research
inquiry (pp. 240-256). Oxford, UK: Oxford University Press.2409
Cook, T. D., & Reichardt, C. S. (Eds.). (1979). Quantitative and qualitative methods in
evaluation research. Beverly Hills, CA: Sage.
Creswell, J. W. (2015). A concise introduction to mixed methods research. Thousand Oaks, CA:
Sage.
33
Creswell, J. W., & Plano, C. V. (2006, 1st edition). Designing and conducting mixed methods
research. Thousand Oaks, CA. SAGE Publications.
Creswell, J. W., & Plano, C. V. (2018, 5tht edition). Designing and conducting mixed methods
research. Thousand Oaks, CA. SAGE Publications.
Doucerain, M., Vargas, S., & Ryder, A. (2016). Mixed-methods research: Integrating qualitative
and quantitative approaches to the psychological study of culture. In N. Zane, G. Bernal,
F. Leong (Eds.), Evidence-based psychological practice with ethnic minorities:
Culturally informed research and clinical strategies. Washington, DC, US: American
Psychological Association.
Duncan, G. J., Hustton, A. C., & Weisner, T. S. (2006). Higher ground: New hope for working
poor and their children. New York: Russell Sage Foundation.
Dures, E., Rumsey, N., & Gleeson, M. K. (2010). Mixed methods in health psychology:
Theoretical and practical considerations of the third paradigm. Journal of Health
Psychology, 16(2), 332-341.
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods
designs—principles and practices. Health services research, 48(6pt2), 2134-2156.
Fetters, M. D., & Molina-Azorin, J. (2017). The Journal of Mixed Methods Research starts a new
decade: The mixed methods research integration trilogy and its dimensions. Journal of
Mixed Methods Research, 11 (3), 291-307.
Fetters, M. D. (2020). The Mixed Methods Research Workbook: Activities for Designing,
Implementing, and Publishing Projects. Thousand Oaks, CA: Sage.
Fielding, N. (2010). Mixed methods in the real world. International Journal of Social Research
Methodology, 13 (2), 127-138.
34
Frost, N. A., & Shaw, R. L. (2015). Evolving mixed and multimethod approaches in psychology.
S. Hesse-Biber and R. B. Johnson (Eds.), The Oxford handbook of multimethod and
mixed methods research inquiry (pp. 375-392). Oxford, UK: Oxford University Press.
Frost, N. A., Nolas, S. M., Brooks-Gordon, B., Esin, C., Holt, A., Mehdizadeh, L., &
Shinebourne, P. (2010). Pluralism in qualitative research: The impact of different
researchers and qualitative approaches on the analysis of qualitative data. Qualitative
Research, 10(4), 441-460.
Fusch, P., Fusch, G. E., & Ness, L. R. (2018). Denzin’s paradigm shift: Revisiting triangulation
in qualitative research. Journal of Social Change, 10(1), 19-32.
Giddings, L. S., & Grant, B. M. (2007). A Trojan horse for positivism? A critique of mixed
methods research. Advances in Nursing Science, 30 (1), 52-60.
Gelo, O., Braakmann, D., & Benetka, G. (2008). Quantitative and qualitative research: Beyond
the debate. Integrative Psychological Behavior, 42, 266-290.
Greene, J. C. (2007). Mixed methods in social inquiry. San Francisco, CA.: Wiley Publishers.
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for
mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11 (3),
255-274.
Hanson, W. E., Plano Clark, V., Petska, K., Creswell, J. W., & Creswell, J. D. (2005). Mixed
methods research designs in counseling psychology. Journal of Counseling Psychology,
52 (2), 224-235.
Harkness, S., Moscardino, U., Bermudez, M. R., Zylicz, P. O., Welles-Nystrom, B., Blom, M.,
Parmar, P., Axia, G., & Super, C. (2006). Mixed methods in international collaborative
35
research: The experiences of the international study of parents, children, and schools.
Cross-cultural research, 40 (1), 65-82.
Haverkamp, B. E., Morrow, S. L., & Ponterotto, J. G. (2005). A time and place for qualitative
and mixed methods in counseling psychology research. Journal of Counseling
Psychology, 52 (2), 123-125.
Hay, M. C. (2016). (Ed.). Methods that matter: Integrating mixed methods for more effective
social science research. Chicago: The University of Chicago Press.
Hesse-Biber, S.N. (2010). Qualitative approaches to mixed methods practice. Qualitative
Inquiry 16(6), 455-468.
Hiles, D. (2012). Mixed methods and the problem of the logic of inquiry. Draft paper presented at
the QMiP symposium on “Mixed Methods in Psychology.” Annual BPS Conference,
London, April 18-20, 2012.
Johnson, R. B. (2012). Guest editor’s editorial: Dialectical pluralism: A metaparadigm whose
time has come. American Behavioral Scientist, 56, 751-754.
Karasz, A., & Singelis, T. M. (2009). Qualitative and mixed methods research in cross-cultural
psychology: Introduction to the special issue. Journal of Cross-Cultural Psychology, 40
(6), 909-916.
Levitt, H., Creswell, J. W., Josselson, R., Bamberg, M., Frost, D. M., & Suarez-Orozco, C.
(2018). Standards for qualitative primary, qualitative meta-analytical, and mixed methods
research in psychology: The APA publications and communications task force report.
American Psychologist, 73(1), 26-46.
Mark, M. M. (2015). Mixed and multimethods in predominantly quantitative studies, especially
experiments and quasi-experiments. In S. Hesse-Biber & R. B. Johnson (Eds.), The
36
Oxford Handbook of Multi-Methods and Mixed Methods Research Inquiry (pp. 21-41).
NY: Oxford University Press.
McCrudden, M., Marchan, G., & Schutz, P. (2019, April). Mixed methods in educational
psychology inquiry. Contemporary Educational Psychology, 57, 1-8.
Morgan, D. (2016). Paradigms lost and pragmatism regained: Methodological implications of
combining quantitative and qualitative methods. Journal of Mixed Methods Research,
1(1), 48-76.
Morse, J., & Niehaus, L. (2009). Mixed method design: Principles and procedures. Walnut
Creek, CA: Left Coast Press, Inc.
O’Cathain, A., Murphy, E., & Nichol, J. (2008). The quality of mixed methods studies in health
services research. Journal of Health Services Research and Policy, 13(2), 92-98.
O’Cathain, A., Murphy, E., & Nichol, J. (2010). Three techniques for integrating data in mixed
methods studies. BMJ: British Medical Journal, 341, 1147-1150.
Onwuegbuzie, A. J., & Leech, N. L. (2006). Linking research questions to mixed methods data
analysis procedures. The qualitative report, 11(3), 474-498.
Onwuegbuzie, A. J., & Collins, K. M. (2007). A typology of mixed methods sampling designs in
social science research. Qualitative Report, 12(2), 281-316.
Onwuegbuzie, A. J. (2012). Introduction: Putting the MIXED back into quantitative and
qualitative research in educational research and beyond: Moving toward the radical
middle. International Journal of Multiple Research Approaches, 6 (3), 192-219.
Pearce, L. (2012). Mixed methods inquiry in sociology. American Behavioral Scientist, 56(6),
829-848.
37
Pelto, P. J. (2015). What is so new about mixed methods? Qualitative Health Research, 25 (6),
734-745.
Plano Clark, V. L. (2010). The adoption and practice of mixed methods: U.S. trends in federally
funded health-related research. Qualitative Inquiry, 16(6), 428-440.
Plano Clark, V.L. & Badiee, M. (2010). Research questions in mixed methods research. SAGE
Handbook of mixed methods in social & behavioral research, 275-304.
Povee, K., & Roberts, L. D. (2015). Attitudes toward mixed methods research in psychology: the
best of both worlds? International Journal of Social Research Methodology, 18 (1), 41-
57.
Powell, H., Mihalas, S., Onwuegbuzie, A., Suldo, S., & Daley, C. (2008). Mixed methods in
school psychology: A mixed methods investigation of the literature. Psychology in the
Schools, 45 (4), 291-309.
Rabinowitz, V. C. & Weseen, S. (2001). Power, politics, and the qualitative/quantitative debates
in psychology. In D. L. Tolman & M. B. Miller (Eds.), From subjects to subjectivities: A
handbook of interpretive and participatory methods (pp. 12-28). New York University
Press.
Reicherz, J. (2007). Abduction: the logic of discovery. In A. Bryant & K. Charmaz (Eds.), The
SAGE handbook of grounded theory (pp. 214-228). Thousand Oaks, CA: Sage.
Sandelowski, M., Voils, C. I., & Knafl, G. (2009). On quantitizing. Journal of mixed methods
research, 3(3), 208-222.
Sandelowski, M., Voils, C., Leeman, J., & Crandall, J. (2012). Mapping the mixed methods-
mixed research synthesis terrain. Journal of Mixed Methods Research, 6(4), 317-331.
38
Shannon-Baker, P., & Edwards, C. D. (2018). The affordances and challenges of incorporating
visual methods in mixed methods research. American Behavioral Scientists, 62 (7), 935-
955.
Small, M. L. (2011). How to conduct a mixed methods study: Recent trends in a rapidly growing
literature. Annual review of sociology, 37, 57-86.
Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011). Methodology in conducting a
systematic review of systematic reviews of healthcare interventions. BMC medical
research methodology, 11(1), 15.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and
quantitative approaches. Applied Social Research Methods Series, vol 46. Thousand
Oaks. Sage.
Tashakkori, A., Teddlie, C., & Sines, M. C. (2011). Utilizing mixed methods in psychological
researcher. In J.A. Schinka, W. F. Velicer, & I. B. Weinder (Eds.), Handbook of
psychology: Research Methods in Psychology (2nd edition), (pp. 428-450). John Wiley
and Sons.
Timans, R., Wouters, P., & Heilbron, J. (2019). Mixed method research: What it is and what it
could be. Theory and Society, 48:193-216.
Todd, Z., Nerlich, B., McKeown, S., & Clarke, D. D. (Eds.). (2004). Mixing methods in
psychology: The integration of quantitative and qualitative methods in theory and
practice. Hove, East Sussex: Psychology Press.
Waszak, C., & Sines, M. C. (2003). Mixed methods in psychological research. In A. Tashakkori
and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research
(pp. 557-576). Thousand Oaks, CA: Sage.
39
Weisner, T. S. (Ed.). (2005). Discovering successful pathways in children’s development: Mixed
methods in the study of childhood and family life. Chicago, IL: University of Chicago
Press.
Weisner, T. S. (2016). Findings that matter: A commentary. In M. C. Hay (Ed.). Methods that
matter: Integrating mixed methods for more effective social science research (pp.393-
408). Chicago: The University of Chicago Press.
Wiggins, B. J. (2011). Confronting the dilemma of mixed methods. Journal of Theoretical and
Philosophical Psychology, 31 (1), 44-60.
Yoshikawa, H., Kalil, A., Weisner, T. S., & Way, N. (2013). Mixed qualitative and quantitative
research in development science: Uses and methodological choices. Qualitative
Psychology, 1(S), 3-18.
40
Figure 1. Method flowchart of our approach.
41
Table 1
Summary of Content Analyses about the Use of Mixed Methods in Fields in Psychology, Sorted
by Year (N=11)
Year First Author / Publication Field # of Design- Attention
Country Venue Examples Centered To
Paradigms
2003 Waszak Handbook Clinical 13 Yes No
US psychology,
reproductive and
family health
2005 Hanson Journal Counseling 22 Yes Yes
US psychology
2008 Harkness Journal Cross-cultural 7 No No
International research
2008 Powell Journal School 1 Yes No
US Psychology
2010 Dures Journal Health 1 No Yes
UK Psychology
2012 Bartholemew Book Culture-specific 12 Yes No
US psychological
research
2012 Tashakkori & Handbook Generic 12 Yes Yes
Teddlie
US
42
2013 Yoshikawa Journal Developmental 16 No No
US Sciences
2014 Bishop Journal Health 5 Yes Yes
UK Psychology
2016 Doucerain Book Culture-specific 8 No Yes
Canada psychological
research
2019 McCrudden Journal Educational 5 Yes Yes
US psychology
Total 102
43
Table 2
Representation of Countries in the Reviews
Review (First Author) Country of Authors Exemplar Countries
Bartholomew US US, UK, Thailand,
Tanzania
Bishop UK UK
Doucerain Canada US, Canada, Taiwan
Dures UK UK
Hanson US US, Kenya, Canada,
Uganda, UK
Harkness US, Italy, Poland, US, Italy, Poland, Spain,
Spain, Sweden Sweden
McCrudden US US
Powell US UK
Tasakkori US US, Sri Lanka, Germany,
UK
45
Table 3
Integrating Findings about the Uses of Mixed Methods in Psychological Research Using
Quantitative and Qualitative Approaches*
Finding Phase 1 Relationship between Phase 2
Quantitative the Findings Qualitative
Prevalence Meta-Syntheses
Studies
1.Qualitative and quantitative
methods have been combined in X Confirmatory X
psychology for decades.
2.Most research combining
qualitative and quantitative
methods is not explicitly labelled X X
as “mixed methods”.
3.Mixed methods have been
reported more widely in applied X X
fields in psychology.
4.Concern about the
compatibility between qualitative X X
and quantitative approaches
remains a salient issue in
psychology.
5.Most mixed methods research
is not appearing in the top-tier X X
journals APA in psychology.
46
6.Most studies in psychology that
use mixed methods give priority X

Contradictory
to the quantitative methods.
7.Qualitative methods and non-
experimental approaches are X
often used in mixed methods
studies in psychology.
8.Examples of the use of mixed Complementary

X
methods in psychology are
disseminated in a wide range of
publication venues, including
international ones.
*The circular symbol designates five topics where there is an agreement between the findings from two analytical
approaches. The two-headed arrow links one inference from the quantitative analysis and one from the qualitative
that appear contradictory. The third symbol in the last row of the table links two findings (7 & 8) that offer
complementary insight.
47
Figure 2 Click here to access/download;Figure;Figure 2 Timeline.docx
Figure 2. Timeline of reviews displaying the distribution of papers on the left and the span of the
review on the right, publication date of reviews highlighted with dashed lines and first author
Conflict of Interest
The authors declare no conflict of interest.

Credit Author Statement
CREDIT STATEMENT
Elizabeth G. Creamer is the lead author of this paper. She conceived of the paper, identified the
literature, and developed a set of hypotheses to confirm through extracting data from the 108
examples. She oversaw the drafting of the manuscript. She developed two tables, including one
from the preliminary round of data analysis and a second that integrated the findings from the
two methods.
David Reeping developed the database and conducted the analysis of the data extracted from the
examples. He developed two figures and one table, wrote the initial draft of methods section of
the manuscript, and part of the analysis. He provided input on all versions of the manuscript.
View publication stats

Advancing Mixed Methods in Psychological Research

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Advancing Mixed Methods in Psychological Research

Uploaded by

Copyright:

Available Formats

See discussions, stats, and author profiles for this publication at: https://www.researchgate.