DESIGNINGANDCONDUCTING

MIXEDMETHODSSTUDIES
BethAngellandLisaTownsend
Workshopforthe2011SocietyforSocialWorkand
Researchannualmeeting

OverviewofWorkshop

DefinitionsandterminologyofMMResearch
PhilosophicalAssumptions
MixedMethods:NutsandBolts
Break
Planningyourmixedmethodsstudy

Sampling
DataCollection

DataAnalysis
EvaluatingMixedMethodsStudies
RepresentingMixedMethodsResearch
Examples
QandAandTechnicalAssistance

Resources

Creswell&PlanoClark(2011)Designingand
conductingmixedmethodsresearch.Thousand
Oaks,CA:SagePublications,Inc.

Teddlie&Tashakkori(2009)FoundationsofMixed
MethodsResearch:IntegratingQuantitativeand
QualitativeApproachesintheSocialand
BehavioralSciences.LosAngeles:Sage
Publications,Inc.

ConsensusDefinitionofMMResearch
Mixedmethodsresearchisthetypeofresearchin
whicharesearcherorteamofresearchers
combineselementsofqualitativeandquantitative
approaches(e.g.,useofqualitativeand
quantitativeviewpoints,datacollection,analysis,
inferencetechniques)forthepurposeofbreadth
anddepthofunderstandingandcorroboration
Johnsonetal.(2007).

Keyterminology
Qualitative(QUAL)componentandQuantitative
(QUAN)componentareoftenreferredtoas
strands

PhilosophicalIssues

QuantitativeTradition

Philosophicalunderpinnings:positivism/post
positivism
Deductivelogic
Dataarerepresentednumerically
Associatedterms:surveyresearch,probability
sampling,experimentalandquasiexperimental
designs,descriptiveandinferentialstatistics

QualitativeTradition

Philosophicalunderpinnings:constructivism
Inductivelogic
Dataarerepresentedtextuallyorpictorially
Associatedterms:groundedtheory,ethnography,
casestudies,purposivesampling,categoricalvs.
contextualizingstrategies,trustworthiness,
credibility

MixedMethodsTradition

Philosophicalunderpinnings:pragmatism
Bothdeductiveandinductive
Dataarerepresentedbothnumericallyand
textually/pictorially
Associatedterms:concurrent(parallel)and
sequentialmixeddesigns,triangulation,data
conversion,inferencequality

Philosophicaldebatesaboutmixed
methods

Incompatibilitythesis fundamentaldifferences
betweenQUANandQUALapproachesaresogreat
thatmethodscannotbemixed
Pragmatism whatisthebestwaytoanswera
researchquestion;bothmethodsofferdifferent
waysofansweringresearchquestions

MixedMethods:NutsandBolts

CharacteristicsofMMStudies

Involvescollectionandanalysisofqualitativeand
quantitativedatainwaysthatare:

Themethodsaremixedby

rigorous
framedepistemologically/theoretically
orderingthemsequentially
mergingthem
embeddingonestrandwithintheother

Combinesthedatawithinthecontextofasinglestudyor
researchprogram
Encapsulatesthestrandswithinanoverallresearchdesign
thatguidesthestudyasawhole

ResearchQuestionsthatCallforMM

Exploringthemeaningofaconstructor
phenomenonfrommorethanoneperspective
Explanationofanomalousfindingsorgetting
behindthemechanismofactionofaneffect
Theorydevelopmentfollowedby
testing/extension
Measuredevelopmentusinggroundedconcepts
Augmentingevaluationstudieswithbetter
understandingofinterventionimplementation

WaysThatDesignsVary

Levelofinteractionbetweenstrands
Relativepriorityofstrands
Timingorpacingofeachstrand
Pointofinterface(atwhichpointintheresearch
processarethestrandsmixed?):during
interpretation,dataanalysis,datacollection?
Researchstance,epistemology

TypologyofMixedMethodsDesigns

Convergentparallel
Explanatorysequential
Exploratorysequential
Embedded
Caveat:evolvingfieldwithevolvinglanguage

AdaptedfromCreswell&PlanoClark(2011)

ConvergentDesigns

QUANandQUALstrandsareconductedseparately
yetconcurrentlyandmergedatthepointof
interpretation
Equalprioritygiventoeachstrand
Usedtoformamorecompleteunderstandingofa
topic,ortovalidateorcorroboratequantitative
scales

ConvergentParallelDesign
QUAL

QUAL

DataCollection

DataAnalysis

Compare
orRelate

QUAN

QUAN

DataCollection

DataAnalysis

Interpretation/
Meta
Inference

Convergent Parallel Design Example: Conceptual Adequacy of the Drug Attitude

Inventory for Youth
Design Decisions
Choice of instrument
Questionnaire
Rating scale
Sampling
Convenience
Representative

Design Decisions

QUAN
Data
Collection

Demographics
Youth DAI
Parent DAI
Adherence Ratings
Clinical Scales
(CDRS,YMRS,CBCL)

Choice of method
Interview
Ethnography
Focus group
Sampling
Purposive
Convenience
Setting

QUAL
Data
Collection

Semi-structured
interviews
Youth SEMI
Parent SEMI

QUAL
Data
Analysis

QUAN
Data
Analysis
Compare

Contrast

Metainference
Mixed Methods Question: Can prediction of youth attitudes
toward psychotropic treatment be improved by knowledge about the
factor structure of the DAI in youth and their subjective experiences
experiences
of treatment?
z

Townsend,Floersch,&Findling,2010

ExplanatorySequentialDesign

Methodsareimplementedsequentially,(QUAN
QUAL)
Usedwhenresearcherwishestousequalitative
findingstohelpinterpretorcontextualize
quantitativeresults

QUANData
Collection
and
Analysis

Followup
with

QUALData
Collection
andAnalysis

Interpretation/
Meta
Inference

ExplanatorySequentialExample:ACT
SocialNetworkStudy
QUANData
Collection
and
Analysis

ACT
Randomized
Trial:NoSocial
NetworkEffects

QUALand
QUANData
Collection
andAnalysis

QUANAnalysisof
SocialNetwork
Predictors

Angell&Test,2002;Angell,2003

QUALstudyof
RCTSubsample

Interpretation
/Meta
Inference

Interpretation
/Meta
Inference

ExploratorySequentialDesign

Methodsareimplementedsequentially,(QUAL
QUAN)
TheQUALstrandisconsideredexploratory,tobe
followedbyfurthertestingandverificationduring
theQUANphase

Qualitative
DataCollection
andAnalysis

Buildsto

Quantitative
DataCollection
andAnalysis

Interpretation

ExploratorySequentialDesignExample:Measuring
ProceduralJustice(PJ)inPoliceEncounters
Reviewof
existing
QUALstrand:
instruments
Consumer
andliterature
interviews
ledtoresearch
Analysisof
question:Do
discrete
existingPJ
encounters
instruments
usinggrounded
capture
dimensional
featuresof
analysis
contacts
betweenpolice
andcitizens
Interpretation: PJexperiences
withmental
area)contextualizedby
illness?
negativeexpectationsandb)
sensitivetosmallgesturesof
humanity

Watson,Angell,Vidalon,&Davis(2010)

QUANstrand
Instrument
development
Cognitive
interviewing
Expertreview
Surveyof
consumersusing
finalinstrument
(PCES)
EFAandRasch
Analysis
Interpretation: PCESpredicted
reactionstopoliceencounter
(resistance,cooperation)

EmbeddedDesign

ResearcherconductingeitheraQUALorQUAN
studyembedsasmallerstrandoftheother
method,asanenhancement
Secondarystrandcanbeconcurrentorsequential

QualitativeorQuantitativeDesign
QUALorQUANDataCollectionandAnalysis
QUALorQUANDataCollectionand
Analysis(before,during,orafter)

Interpretation/
MetaInference

EmbeddedDesignExample:CTI
EvaluationandFidelityStudy
RCTofCriticalTimeIntervention(CTI)forMenLeaving
Prison.QUANDataCollection,n=220
Fidelity/ProcessStudy
QUALdatacollection,n=24

MMResearchQuestions:Inwhatways
isCriticalTimeInterventionmodifiedor
adaptedwhenusedwithapopulationof
recentlyreleasedprisoners?What
processescontributetotheadapted
programslevelofeffectiveness?
Draine,Angell,Barrenger,&Kriegel(inprogress)

DataCollection
Decisions:
Interviews
FocusGroups
Fieldnotes
Recordabstraction
Samplingcriteria
DataAnalysis
Decisions:
Coding(open,
selective,axial)
ofinterviewsand
documents/
Narrative
analysis?

VariationsontheMMDesigns

Multiphaseformat
Multilevelformat
MonostrandConversion(nottrulymixed
methods):conversionofQUALdatatoQUANor
QUANdatatoQUAL,withoutadditionalstrands
Transformativestance

Questions?

Break

PlanningYourMixedMethodsStudy

Sampling:GeneralConsiderations

Strategychosenshouldbeappropriatetoeach
respectivestrand
Balancebetweensaturationofphenomenonor
theory(qualitativegoal)andrepresentativeness
(quantitativegoal)

SamplingStrategies
(TeddlieandTashakkori,2010)

Parallelmixedmethodssampling(paralleluseof
probabilityandpurposivestrategies,either
concurrentlyorwithatimelapse).
Onesamplemaybeasubsetoftheother
Bothstudiesmayusesametotalsample

Sequentialmixedmethodssampling(informationfrom
thefirstsampleisusedtodrawthesecond)
Multilevelmixedmethodssampling:usingprobability
andpurposivesamplingtechniquesatdifferentlevels
ofanalysis(e.g.,cliniciansandclients)

DataCollection
(Teddlie&Tashakkori,2009)

SelfReportTechniques

ObservationalMethods

Socialnetworkanalysis

SecondaryDataAnalysis

Participantobservation,nonparticipantobservation

Sociometry

Interviews
Questionnaires
AttitudeScales
Personalityinventories
Projectiveinstruments

Archivalanalysis
Metaanalysis

MultipleModesofDataCollection(Tashakkori&Teddlie,

1998)

DataAnalysis

QuantitativeDataAnalysis

Descriptive

summarizingdata,lookingfortrendsandpatterns;
means,frequencies,measuresofvariability

Inferential

hypothesistesting,inferencesaboutapopulation
characteristic;significancetests(2,t,F),multiple
regression,ANOVA,MANOVA,MANCOVA,
hierarchicallinearmodeling,timeseries,eventhistory

QualitativeDataAnalysis

Oftenongoingduringdatacollection(e.g.,purposive
sampling,modificationofinterviewquestions,etc.)

FRACTURINGVS.CONTEXTUALIZING

Groundedtheory
Thematicanalysis
Negativecaseanalysis

Categoricalstrategies:producecategoriesthatfacilitatecomparisons;e.g.,
constantcomparativemethod
Contextualizingstrategies:interpretnarrativedatainthecontextofthe
wholetext,focusingoninterconnectionsbetweenstatements,events,etc.;
e.g.,phenomenology

SIMILARITYVS.CONTRAST

MixedMethodsDataAnalysis
(CreswellandPlanoClark,2011)

QUAN+QUAL=convergeresultsCONVERGENTDESIGN
QUAN qual=explainresultsSEQUENTIALEXPLANATORY
DESIGN
QUAL quan=generalizefindingsSEQUENTIALEXPLORATORY
DESIGN
QUAN(+qual)=enhanceexperimentEMBEDDEDDESIGN
TRANSFORMATIVEDESIGN usesatransformativetheoretical
perspectivetoadvocateforsocialchange,addresssocialinjustice,
orgivevoicetomarginalized/underrepresentedgroup.
MULTIPHASEDESIGN aprogramofresearchthatinvolvesseveral
studies;canhavecombinationsofsequentialandconcurrent
designs

MixedMethodsDataAnalysis
(Creswell&PlanoClark,2011)

Convergentparallel:mergeddataanalysisforpurposesof
comparingresults

CollectandanalyzeQUALandQUANdata
Strandsareanalyzedindependently(couldbequalitizing/quantitizing
strategiesalso)
Howwillthetwostrandsbecompared?
Howwilltheyberepresented?

Explanatory:connecteddataanalysistoexplainfindings

Collectandanalyzequantitativedata;derivesecondresearchquestion
Designandconductqualitativeresearch
Analyzequalitativedataforanswerstosecondaryresearchquestion
Linkresultsfrombothstrands howdoqualitativeresultsexplain
quantitativefindings?

Convergent Parallel Design: Data Analysis of the Drug Attitude Inventory

QUAN
Data
Collection

Does the factor structure

of the DAI in adults fit the
youth data?
If not, what is the factor
structure of the DAI in
youth?

QUAL
Data
Collection

Univariate descriptives
Bivariate correlations
Structural Equation
Modeling
Factor analysis
Parallel Analysis
(SPSS, LISREL)

QUAN
Data
Analysis

How well do DAI items

correlate with one
another?
Do they measure a single
construct or multiple
constructs?

Compare

Are there elements of

youth medication
experience that the
DAI does not capture?

QUAL
Data
Analysis

Converge

Can prediction of youth attitudes toward psychotropic treatment be

improved by knowledge about the factor structure of the DAI in
youth and their subjective experiences of treatment?

In vivo codes
Intermediate codes
Superordinate codes
Peer review
Constant
comparative
approach
(Atlas TI)

MixedMethodsDataAnalysis
(Creswell&PlanoClark,2011)

Exploratory:connecteddataanalysistogeneralize
findings

Collectandanalyzequalitativedata;usequalitativedatatodesign
quantitativecomponent
Collectandanalyzequantitativedata
Linkresultsfrombothstrands:howdoquantitativeresultsextend
qualitativefindings?

Embeddeddesign:merged(concurrentdesign)or
connected(sequentialdesign)analysis

Collectandanalyzeprimarydataset;decidehowembeddeddatawillbe
usedandwheretheyshouldbeincorporatedintotheprimaryanalysis
Analyzesecondarydatasetdictatedbywhereitisembeddedinthelarger
design
Howdotheembeddedfindingsintegratewiththeprimarystudyfindings?

EvaluatingMixedMethodsStudies

Mixed Methods Validation Framework (VF)

(Dellinger & Leech, 2007; Leech, Dellinger, Brannagan, & Tanaka, 2010)

Five elements:

Foundational element

Construct validation

Consistency of links between various strands of the study (see

table on following slide)

Utilization/historical element

Validity of QUAN, QUAL, and mixed elements

Inferential consistency

Quality of literature review and theory base

Whether and how the studys findings went on to be used in

future work

Consequential element

Social acceptability and consequences of study findings

Construct Validation
(Dellinger & Leech, 2007; Leech, Dellinger, Brannagan, & Tanaka, 2010)

RepresentingMixedMethodsData
(Creswell&PlanoClark,2011)

Sidebysidecomparison
Jointcomparison
Mergedcategory/themedisplay

Writing the article

(Creswell, 2003)

Introduction explicit integration of both paradigms from the outset

Literature review integration of inductive/deductive reasoning, why

the literature needs this type of study

Posing the research question what are the questions and why do
they call for two paradigms?

Methods present both methodologies, in their respective

languages, integrated under the umbrella of the research question

Results present results of both modes of data collection

Discussion role of meta-inference

ExampleI
FindingsfromDAIStudy
(Townsend,Floersch,&Findling,2010)

StudyFlowchart
QUAN
Data
Collection

QUAN
Data
Analysis

Demographics
Youth DAI
Parent DAI
Decision-Making Scales
Adherence Ratings
Clinical Scales (CDRS,YMRS,CBCL)

Univariate descriptives
Bivariate correlations
Structural Equation
Modeling
Factor analysis
Parallel Analysis
(SPSS, LISREL)

Compare

QUAL
Data
Collection

QUAL
Data
Analysis

Semi-structured
interviews
Youth SEMI
Parent SEMI
Brief Parent SEMI

In vivo codes
Intermediate codes
Superordinate codes
Peer review
Constant comparative
approach
(Atlas TI)

Converge

Can prediction of youth attitudes toward psychotropic treatment be

improved by knowledge about the factor structure of the DAI in
youth and their subjective experiences of treatment?

ParallelDataReductionStrategies
@stigma {0-56}

&Desire_for_Normality {0-0}

&Crazy_Identity {0-0}

&Educating_Others {0-0}

+want_to_live_normal_life_withou
t_meds {1-0}

+for_crazy_people {2-0}

+did_bipolar_slide_show_at_schoo
l {1-0}

+nobody_else_takes_meds {2-0}

+Labeled_psycho {2-0}

+I_like_explaining_meds_to_peop
le {1-0}

StructuralEquationModelOne
DAIOriginalFactorStructure

RMSEA .061
[ideal = <.05 (Kaplan, 2000)]

CFI .925
NNFI .913
[ideal = >.95 (Kaplan, 2000)]

df 258
420.38

X2

ExploratoryFactorAnalysis

FactorSelectionCriteria:

EFAOne

Maximumlikelihoodestimation
Eigenvalue> 1.0
Minimumitemloading> .30
Retained4+items
>4itemsbutdifferentiatedwellfromotherfactors
Qualitativedataindicateretentionofitems/factorsisjustified

Twofactorswerenotinterpretedfurtherbecausetheywereeachcomprisedofonlyoneitem
(items8and13)andhadvalues>1.0.(Heywoodcases)

EFATwo28items(SupplementedbyParallelAnalysis)

Twofactorsretained,accountingfor36.61%ofthevarianceinDAIscore.
Factorlabels:

PositiveFeelingstowardMedication
NegativeFeelingstowardMedication

Fouritemsdidnotloadonanycomponent(10R,11R,20R,and30)
Cronbachsalpha=.889
YouthDAIcorrelatedpositivelywithyouthselfreportedadherence(r=.205,p<.05)
DAIshowednosignificantcorrelationswithclinicaloutcomemeasures(CBCLCompetenceand
SymptomScales).

QualitativeAnalysisMethods
Constantcomparativeapproach analysisofinvivocodes
followedbyintermediatelevelcodingandsynthesisinto
higherordersuperordinateconcepts.
@stigma {0-56}

&Desire_for_Normality {0-0}

+want_to_live_normal_life_withou
t_meds {1-0}
+nobody_else_takes_meds {2-0}

&Crazy_Identity {0-0}

+for_crazy_people {2-0}
+Labeled_psycho {2-0}

&Educating_Others {0-0}

+did_bipolar_slide_show_at_schoo
l {1-0}
+I_like_explaining_meds_to_peop
le {1-0}

SummaryofQualitativeThemes
DAIrelatedconcepts

Positivefeelings
Negativefeelings
Health/IllnessModel
InternalLOC
ExternalLOC
RelapsePrevention

Conceptsnot
representedintheDAI

Balancedresponsibility
Ambivalence
ChangeoverTime
Adherenceifeffective
Expectations
Inclusionintreatment
decisions
Autonomy
Stigma

SidebySideComparisonExample:
ConclusionsSection
QuantitativeFindings

QualitativeFindings

Insummarizingthefindingsoftheadolescent
qualitativeandquantitativeanalyses,itis
apparentthatthesevendimensionsfoundin
theoriginalDAIemployedwithadultsare
presentinteens conceptualizationsoftheir
viewsofmedication.However,adolescents
maythinkaboutthosedimensions,suchas
sideeffectsandrelapseprevention,
differentlythanadultsdo.Thesefindingsare
supportedbyboththeEFAandqualitative
results.EFAfindingsindicatethatsideeffects
differentiatedintospecificcognitiveeffectsas
wellasotherharmsforyouth.Adolescents
didnotappeartoviewrelapsepreventionas
theadultsdid,insteadevaluatingtheir
perceivedneedformedicationinrelationto
theirsymptomstabilityinformingtheir
attitudestowardmedication.

Thequalitativedatahighlightedthepresence
ofeightadditionalthemesnotreflectedinthe
originalfactorstructureoftheDAI,including
importantthemessuchaspersonalautonomy,
inclusioninmedicationdecisionmaking,the
experienceofstigma,andlevelofpersonal
responsibilityvs.externalcontrolover
behavior.Thesethemeshighlightthat
medicationexperiencesandtheformationof
attitudestowardpharmacologicaltreatment
foryoutharecomplexandmultiply
determined,renderingitdifficulttorepresent
adolescentattitudestowardmedicationfully
withasingleinstrument.Thesefindingspoint
tothepotentialvalueofcreating
individualized,qualitativeassessmenttoolsto
captureyouthexperienceswithmedication
ratherthanrelyingonasinglequantitative
measureorsetofsubscales.

Discussionsectionhighlights
divergentresults

JointDataDisplayExample
HighlightsConvergenceofFactorAnalyticandThematicAnalysisResults
PositiveFeelings

FactorLoading

2 goodoutweighsbad

.474

21 thoughtsareclearer

.789

26 happieronmeds

.487

29 inbettercontrol

.598

QualitativeThemes

Quotations

Emotional

Icaneithernottakeitandbeagrouch,ortakeitandbehappy.SoIwouldmuchrathertakeitandbehappy.

Cognitive

Sittingdown,payingattentiontotheteacher,nottalking.JustpayingattentionanddoingwhatIneedtodo.

Physical

Igainenergytowanttodomyworkandlisteningandfocusandtrytodomybestandgive110%.

NegativeFeelings

FactorLoadings

5R takeb/cofpressurefrom
others

.315

14R medicationisslowacting
poison

.714

19R Idratherbesickthan
takingmedications

.658

QualitativeThemes

Quotations

Emotional

IdontlikesometimesIfeellikeoutofmybodyorjustlikenotmyself,moreanxioussometimes.

Cognitive

WellIhadittoldmetheTopamaxorwhatevercausesshorttermmemoryloss,soIassumeitstheTopamaxorwhatever.

Physical

Makemesick.Makemegetthebubblegutsandstuff.AndIwishthattherewasamedicinethatdidnthavesideeffects.

MergedDataDisplayExample
Countsofquotationsrepresentedgraphicallytodemonstrateconvergence
withfactoranalyses
positive andnegative factorssupportedbyallformsofQUANand
QUALevaluation
Distribution of DAI-Related Quotations across Respondents
40
35
Number of Quotations

30

Positive
Negative

25

Illness Model
20

ILOC
ELOC

15

Harm
Relapse Prev.

10
5
0
4

10

12

13

14

16

17

18

20

22

23

24

25

27

29

30

32

38

41

Respondent

Number of adolescent quotations linked to original DAI factors

Positive
Subjective
Feeling

Negative
Subjective
Feeling

Health/Illness
Model

External
Locus of
Control

Internal
Locus of
Control

Harm/
Toxicity

Relapse
Prevention

103

31

21

20

10

48

34

ExampleII

ForensicAssertiveCommunityTreatment(FACT)
Evaluation(Angell&Watson,inprogress)
Keymixedmethodquestions:
HowisACTmodifiedinthecontextofprisonreentry?
Howdoestheagencysrecoveryorientedmissionshape
thetranslationofACTtoFACT?
Howdoesengagementofconsumersoccurandtowhat
extentisleverageinvolved?
Whataretheuniquefeaturesofclientprovider
relationshipsinFACT?
Howdoesengagementrelatetoratesofsuccess
(avoidanceofreincarceration)?

OverallDesign:ConvergentParallel
Sample=21adultswithmentalillnessleavingprisonandentering FACTprogram
PanelDesign

Client
reported
outcomes
assessedat
BL,1,3,6,
9,18mos
staff
reported
outcomes
monthly

QUAN
DataCollection

QUAL
DataCollection

QUAN
DataAnalysis

QUAL
DataAnalysis
WithinSubject
CrossData
Comparisons

Interpretation/
MetaInference

Ethnographicapproach
Participantand
passiveobservation
Interviewswith
consumersandstaff
Recordabstractions

AnalysisStrategies
Sample=21adultswithmentalillnessleavingprisonandentering FACTprogram
QUAL
Descriptive
PrePost
Comparisons
Outcome
differencesby
risklevel

QUAN
DataCollection

QUAL
DataCollection

QUAN
DataAnalysis

QUAL
DataAnalysis
WithinSubject
CrossData
Comparisons

Interpretation/
MetaInference

QUAL
openandfocusedcoding
creationofsynthetic
participantnarratives

MixedAnalysis:
Comparingphenomena
Acrossdatatype
Acrosstime
Acrosssubjects

QandA/TechnicalAssistance