RESEARCH METHODS & STATISTICS

Interview-based Qualitative Research in

Emergency Care Part II: Data Collection,
Analysis and Results Reporting
Megan L. Ranney, MD, MPH, Zachary F. Meisel, MD, MPH, MSc, Esther K. Choo, MD, MPH,
Aris C. Garro, MD, MPH, Comilla Sasson, MD, PhD, and Kate Morrow Guthrie, PhD

Abstract
Qualitative methods are increasingly being used in emergency care research. Rigorous qualitative
methods can play a critical role in advancing the emergency care research agenda by allowing
investigators to generate hypotheses, gain an in-depth understanding of health problems or specific
populations, create expert consensus, and develop new intervention and dissemination strategies. In Part
I of this two-article series, we provided an introduction to general principles of applied qualitative health
research and examples of its common use in emergency care research, describing study designs and
data collection methods most relevant to our field (observation, individual interviews, and focus groups).
Here in Part II of this series, we outline the specific steps necessary to conduct a valid and reliable
qualitative research project, with a focus on interview-based studies. These elements include building the
research team, preparing data collection guides, defining and obtaining an adequate sample, collecting
and organizing qualitative data, and coding and analyzing the data. We also discuss potential ethical
considerations unique to qualitative research as it relates to emergency care research.
ACADEMIC EMERGENCY MEDICINE 2015;22:1103–1112 © 2015 by the Society for Academic
Emergency Medicine

ria for REporting Qualitative research (COREQ).1 For

I
n Part I of this series, we discussed the role of quali-
tative methods in emergency care research and
introduced types of qualitative studies and their
applications (article immediately preceding in this issue).
Here in Part II, we provide an outline of rigorous quali-
tative methods, define basic qualitative terminology, and
provide resources for researchers interested in under-
taking their own qualitative work. This guide should not
serve as a substitute for consultation with an experi-
enced qualitative researcher, but should serve as an
overview for reviewers of qualitative studies and may
help with early phases of study design. For a high-qual-
ity checklist of important components of a qualitative
manuscript, we refer readers to the COnsolidated crite-
reviewers, we also recommend the Critical Appraisal
Skills Programme’s guide to qualitative research.2
DATA COLLECTION
Rigorous data collection is an integral part of qualitative
research. The majority of effort in a survey-based quan-
titative study is typically spent designing a valid and
reliable survey instrument and obtaining a nonbiased,
generalizable sample.3 In a qualitative study, the major-
ity of effort in the design phase should be spent devel-
oping a systematic, well-developed, data collection
protocol.

From the Department of Emergency Medicine, Rhode Island Hospital/Alpert Medical School, Brown University (MLR, EKC,
ACG), Providence, RI; the Department of Emergency Medicine, the University of Pennsylvania (ZFM), Philadelphia, PA; the
Department of Emergency Medicine, University of Colorado (CS), Aurora, CO; the Colorado School of Public Health (CS),
Aurora, CO; American Heart Association (CS), Dallas, TX; and the Department of Psychiatry & Human Behavior, The Miriam
Hospital/Alpert Medical School, Brown University (KMG), Providence, RI.
Received July 3, 2014; revision received March 16 and April 10, 2015; accepted April 12, 2015.
Dr. Choo is supported by NIDA grant K23DA031881; Dr. Ranney is supported by NIMH grant K23 MH095866; Dr. Morrow is
supported by NICHD grant K24HD062645; Dr. Garro is supported by American Lung Association ACP Grants Program (ACP-
231928-N). Drs. Choo and Meisel, both associate editors at this journal, had no role in the peer-review process or publication deci-
sion for this paper.
Supervising Editor: Craig Newgard, MD, MPH.
Address for correspondence and reprints: Megan L. Ranney, MD, MPH; e-mail: mranney@lifespan.org.

© 2015 by the Society for Academic Emergency Medicine ISSN 1069-6563 1103
doi: 10.1111/acem.12735 PII ISSN 1069-6563583 1103
1104
There are three essential components to a qualitative
data collection protocol: 1) developing a clear collection
strategy, 2) appropriately identifying and sampling the
population of interest, and 3) obtaining data (whether by
observation, interview, or focus group) in a reproducible
and “rich” (detailed and complete) manner. As described
in Part I of this series, numerous qualitative data collec-
tion strategies exist, ranging from observational to focus
groups to analysis of existing data sources.
Below we discuss each component in detail, focusing
on strategies for interview-based qualitative inquiry.
For further information on both interview-based and
other qualitative methods, we refer readers to qualita-
tive methods textbooks such as Denzin and Lincoln4 or
Crabtree and Miller5 or excellent online resources such
as www.qualres.org (produced by the Robert Wood
Johnson Foundation) and Family Health International’s
“Qualitative Research Methods” (www.fhi360.org/re-
source/qualitative-research-methods-data-collectors-
field-guide).
Developing Data Collection Strategies
Qualitative studies usually use written, templated data
collection guides that include topic headings, open-
ended questions within each topic, and probes that can
be used to follow up on critical questions. The data col-
lection guide should be created for each individual
study to specifically address that study’s research ques-
tions. Data collection guides should be developed in
conjunction with topical and methodologic experts to
ensure accurate content, clarity, validity, appropriate
language, and length. A hastily assembled data collec-
tion guide and inconsistent data collection (whether via
interviews, focus groups, or observation) will result in
flawed data, with poor credibility and confirmability.5,6
Regardless of the data collection strategy—observa-
tional, interview, or analysis of preexisting text—a rig-
orous qualitative manuscript should explicitly discuss
how its data collection strategy was designed.1 As many
journals request to publish the data collection guide as
an appendix, the guide should also be interpretable to a
general audience.
Observational Studies. As discussed in Part I, obser-
vation may occur in person or remotely (e.g., through
video or audio taping, via photographs, or by observing
environmental patterns). Other researchers will collect
preexisting text (for instance, examining free text in dis-
charge papers or electronic health records). In observa-
tional qualitative research, data collection guides may
be applied in real time in the field, or may be used dis-
tally, to extract data in remote studies or from existing
data sources. Even when using unstructured “field
notes” instead of a formal data collection protocol, note-
taking should be steered by predetermined guidelines
regarding what deserves the researcher’s attention. For
a detailed discussion of the development of observa-
tional data collection strategies, we refer readers to the
resources already mentioned or to observation-specific
textbooks (e.g., Schensul et al.7 or Kawulich8).
Interviews and Focus Groups. Many emergency care
qualitative researchers use interviews or focus groups
Ranney et al. • QUALITATIVE METHODS PART II
for data collection. An ideal qualitative interview will
feel like an extended conversation for the participant—
yet will yield data on each topic area outlined in the
interview guide.9 Most interview or focus group guides
will contain some information that serve as guidelines
for the interviewer.
There are many approaches to the organization of
interviews and focus groups, but one method that we
have found to be useful especially for novice researchers
is to use an outline format. It begins with a spoken intro-
duction that reviews the overarching goals of the study,
rules of the interview/focus group, a discussion of the
flow of the session, and a confidentiality statement. This
introduction is often read verbatim, but can be para-
phrased as long as all pertinent points are discussed. An
interview or focus group then often proceeds to an “ice-
breaker” question—one that is easy and noncontrover-
sial and makes the participant feel comfortable sharing
information. The introduction/icebreaker question is
usually followed by topic headings, each of which will
contain an opening question, main questions, follow-up
questions, and probes (see Table 1).
Questions are usually divided according to overarch-
ing theme. Each theme (and, sometimes, each question)
Table 1
Outline of an Interview Guide for Focus Groups or Semistruc-
tured Interviews
Section Purpose
Introduction Introduce yourself, establish the
ground rules for the discussion (no
interruptions, respect each other),
reiterate privacy and confidentiality,
allow questions.
Main topics
Opening questions Simple, straightforward, nonsensitive
question to get participants talking
and overcome awkwardness and to
help them adjust to the “narrative”
(vs. survey) style of inquiry.
Main (“grand tour”) Introduce the major topics of the
questions research project. Often more
controversial topics come later so
as not to induce early participant
discomfort.
Follow-up questions Moves conversation from the main
(“prompts”) question to a deeper level, asking
for more detail, usually within
preplanned areas of inquiry.
Probes Conversational tools that invite
further discussion or elaboration.
Probes may steer participants to
specific areas, ask for more depth
or context, or clarify earlier
statements.
Summary Closing statement summarizing
some of the key points and allowing
an opportunity for participants to
clarify these key points or add
additional pertinent data.
Summaries can occur at the end of
brief interviews or after each major
topic.
Debrief Research staff records impressions
of session, clarifies ambiguous data,
and records data not captured in
notes/recordings
ACADEMIC EMERGENCY MEDICINE • September 2015, Vol. 22, No. 9 • www.aemj.org
will be accompanied by an “intent statement,” written for
the research staff. The intent statement is meant to clarify
the goals of the question, maximize reproducibility of
methods, and ensure that all needed information is gath-
ered (K. Morrow Guthrie and R. Rosen, unpublished
data, 2014). Major thematic questions are used to begin
discussion of each new major topic or subtopic and are
generally open-ended to avoid brief or yes/no responses.
Follow-up questions and “probes” typically follow the
major thematic questions. Follow-up questions ask for
more detail within preplanned areas of inquiry. Probes
are short questions that allow an interviewer to delve
deeper into what the study participant is saying (e.g.,
“Who did this? How did you feel?”). A general rule is
that more sensitive or difficult questions should be used
later in the interview, once the participants have devel-
oped comfort and rapport with the interviewer, or
should be approached following the participant’s lead.
Although an interviewer is expected to address each
major thematic question (even if not asking the question
verbatim), the probes may or may not be asked, depend-
ing on whether or not a participant spontaneously dis-
cusses the information requested by the probes. Indeed,
although the interview guide provides an overview of
the desired content, it is not meant to be used verbatim.
A skilled moderator will almost always alter the order
and specific wording of questions, prompts, and probes,
departing from the guide as needed to follow interesting
and relevant lines of inquiry.
Interview and focus group sessions close with the
moderator providing a brief summary of the conversa-
tion, allowing the participants to clarify or add informa-
tion. It is often beneficial to have a debriefing after the
participants leave, but before the research staff departs
a session. This allows the moderator and note taker to
record initial impressions while the verbal and nonver-
bal data are fresh in memory, clarify any ambiguous
data that were collected, and record any pertinent
details of the session that may not be captured in the
notes or recordings.
The length and exact content of an interview guide
will depend on the type of study, the goals of the study,
and the amount of time that is available for the inter-
view session. Most interviews and focus groups should
last no more than 90 minutes, as longer sessions can
become stressful for participants and may affect the
later data quality.
Selecting and Training the Data Collectors
Because the data collectors themselves become part of,
and influence, the process, it is critical to select data col-
lectors carefully. For some studies, it may be desirable
for the data collector to have similarities with the partic-
ipants. For instance, physicians may be more likely to
disclose personal attitudes and beliefs in an interview
with another physician than with, for example, a stu-
dent. For other study topics, using a data collector from
a different demographic group or work environment
than the participants is helpful.10 For example, an
observer who is external to a hospital may be able to
observe the hospital employees’ workflow in a more
objective, nonpartisan manner and with less effect on
“normal” lived experience. Potential authority differen-
1105
tials should also be recognized a priori. For example, a
residency director would not be a good choice as a
qualitative interviewer to ask residents to discuss work
hour issues. Finally, the sex of the data collector may be
important. For instance, women may be less open about
a history of partner violence with a male interviewer.
Interviewers with a clinical background must avoid
inserting their clinical perspective into the data collec-
tion. Some interviewers, in fact, prefer to not describe
themselves as an MD or PhD when interviewing, to
minimize bias. Referring to oneself as a “researcher” in
this situation is often useful.
Data collectors must be aware of their background
and biases and seek to minimize their influence on the
data collected.11 Qualitative manuscripts should there-
fore always describe the training of its data collectors
and acknowledge the ways in which the collectors’ own
biases may have influenced data collection, in the manu-
script.1
In interview- or focus group–based studies, many pit-
falls for interviewers/moderators can be avoided (see
Table 2) to limit the introduction of the moderator’s
own viewpoints and opinions into the data. Some rec-
ommended techniques include asking open-ended ques-
tions, allowing pauses in the conversation, and using
reflective statements.5 Closed-ended questions (e.g.,
“yes/no” questions) lead to conversational dead ends
and do not encourage the participant(s) to explain what
they mean. Double-barreled questions (i.e., two ques-
tions in one) can be confusing to the participant(s), and
often only one of the questions is answered. Leading
questions are phrased in a way that implies a certain
answer is expected and may result in false statements
due to perceived social desirability.9 Not allowing for
silence may shut down participant’s thought processes
and may result in a more superficial data set. Reflective
techniques allow for participants to confirm a correct
understanding, enhance rapport with the moderator,
and encourage elaboration on critical themes.
Prior to conducting an interview or focus group, the
interviewer should practice the process multiple times
with the principal investigator and with researchers or
colleagues who are not involved in the study. Practice
interviews will help assess clarity and usefulness of
questions further; identify areas of potential misunder-
standing, awkwardness, or problems with tone or lan-
guage; and allow the interviewer to learn the cadence
and flow of the interview. Practice also allows the whole
team to role-play in preparation for difficult interviews,
such as those that involve participants who are exces-
sively quiet, excessively verbose, have difficulty staying
on topic, or are adversarial. Strategies for engaging
such participants or minimizing disruption to the data
are amply outlined elsewhere.12
Sampling Strategies
A qualitative manuscript must describe its sampling pro-
cess in detail and should justify its choice of sampling
strategies. Sampling strategies for qualitative research
include random, convenience, or purposeful samples of
participants. Purposeful sampling is unique to qualitative
research. It is often used when a researcher is interested
in a specific participant characteristic (e.g., ED patients
1106 Ranney et al. • QUALITATIVE METHODS PART II

Table 2
Qualitative Questions

Goal “Bad” Question Explanation “Better” Question Explanation

To elicit participants’ “Let’s talk about how “Tell me about your
conception of your child’s The question is leading
prescribed medications have and is likely to result in
medicines helped her asthma.” the participant
providing a biased and
uninformative
response. As qualitative
data is made up of a
participant’s
spontaneous answers,
this question will likely
lead to uncodeable, and
therefore unusable,
data.
To hear a variety of “Why do you think “OK, here are some
opinions about the this program is a Leading question.
design of an good idea?” Too general; fails to
intervention obtain useable data
to inform
intervention design.
To elicit participants’ “You’re not taking “Let’s talk about the last
perceived barriers your meds, huh? Are This is a leading
to medication use they too question.
expensive?” The question is
judgment-laden.
This question is
closed-ended and
invites a “yes/no”
answer.
child’s asthma These questions are
medications?” [[[pause open-ended and do not
for response, then impose the inter
probe:]]] “What, if viewer’s viewpoint on
anything, has changed the participant.
since she started the
They also use
medications?” [[[Pause
for response, then:]]] “reflective statements,”
OK. I hear you saying in which the interviewer
that even though she paraphrases the
has fewer symptoms participant’s statements
now, you worry that to ensure that they are
your kid will get understood correctly
dependent on her meds and to prompt further
if she takes them every data collection.
day? Is this right? Tell
me more!” [[[[Pause]]]
screenshots of the Elicits specific feedback
intervention. Let’s start about intervention
with the first one: tell design components.
me what this means to Allows participants to
you?” describe their thought
process about the
design in their own
words.
Allows interviewers to
observe the logic of
participants’ use of a
resource, as well as
their verbal
responses.
day when you didn’t Allows participant to
take your medicines. describe barriers in
Can you tell me about their own words, even
what was going on that when unaware that
day?” something is a barrier.
Nonjudgmental.

with low medical literacy would be purposely sampled
for a study of discharge instruction comprehension) or to
be sure to capture equal numbers of participants from
specific strata (e.g., sex, for a study on chest pain dis-
charge instructions).13 Numerous, well-validated tech-
niques for purposeful sampling exist, such as snowball
techniques and critical case sampling.13,14
Unlike quantitative studies, sample frames do not
need to mimic general populations to be valid. Although
large sample sizes are usually not necessary, and may
even harm the quality of the data analysis, obtaining an
adequate and diverse sample is still crucial for data
credibility.5
Also unlike survey or other quantitative studies, it is
inadvisable to determine a priori an exact sample size
for a qualitative study. To facilitate budgeting and grant
applications, researchers often present an expected
range of participants based on the study questions,
researcher expertise with a specific line of inquiry or
population, and published rules-of-thumb. During the
conduct of the study itself, however, researchers should
determine appropriate sample size by identifying a
point at which they will consider the data “saturated.”
In general, saturation is defined as the point during
data collection where additional data do not provide
new information to the researchers.14 Although the pre-
cise definition of “saturation” depends on the research
project and the qualitative discipline,5,15 the concept
implies that the research team is conducting at least a
preliminary analysis in real time, to determine when the
point of “no new data” is reached (see below “Data
Management and Analysis” for further discussion of the
analytic process). This expectation for an ongoing quali-
tative analytic process—which may, in turn, modify both
sample size and data collection processes—starkly con-
trasts with guidelines for quantitative research, in which
any interim analyses must be prespecified and must be
independent of the data collection process.
Unlike quantitative research, in which outliers compli-
cate data analysis, in qualitative research outliers (par-
ticipants with unique stories) are informative. They can
be used to challenge themes that have been identified,
ACADEMIC EMERGENCY MEDICINE • September 2015, Vol. 22, No. 9 • www.aemj.org
providing a context for when these themes may not
hold true.
Example: A qualitative study exploring parents’
adherence to their children’s asthma medication
plans identified that most parents adhere to their
physicians’ recommendations about duration of
therapy, because they “wanted to do what was
best for their kids” and they “always follow the
doctor’s advice.” Two families, however, discussed
that they had developed their own plans for
administering their kids’ medications. These plans
were explicitly different from those discussed with
their physician. On further examination of these
outliers, researchers realized that one of these fam-
ilies had a mother who worked in health care; dur-
ing the interview, this family explained that they
had modified their medication plans based on the
mother’s perception that she had a greater under-
standing of her child’s asthma than the doctor. In
the other family, a parent had severe asthma; this
parent explained that he had prior “bad outcomes”
from certain asthma medications, which he wanted
to avoid for his kid. These outlier viewpoints cor-
roborate the greater theme, but also illustrate
unique situations in which this theme may need to
be modified.
Qualitative Study Logistics
The logistics of a qualitative study differ slightly
between observation studies, interviews, and focus
groups. A few logistic elements are, however, universal
(see Table 3). For instance, we recommend using at
least two recording devices (whether audio or video). If
participants are invited into the research setting (e.g.,
for an interview or focus group), the room(s) used
should be vetted prior to the actual data collection, to
ensure adequate privacy, control of excessive noise (for
instance, from an air conditioning unit), temperature, or
any other factors that may preclude good data collec-
tion. The location in which an interview or focus group
is conducted may influence the quality and type of infor-
mation provided. Conducting interviews by phone or in
community locations may be necessary for practical
reasons—as when research subjects have difficulty
obtaining transportation to interviews—but should be
performed with caution: the quality of a cell phone
Table 3
Interview/Focus Groups Logistics Checklist
 Use at least two recording devices
 Check the room ahead of time (noise, temperature, set-up)
 Bring facial tissues
 Refreshments (nothing with a noisy wrapper, though!)
 Paper and pencils for participants to write down thoughts
(as applicable)
 List of resources (as applicable, according to the interview
topic)
 Assure privacy of the room; post a sign on the interview
door or consider using a white noise machine to obscure
the discussion
1107
connection may be poor or an interview in a library or
coffee shop may preclude disclosure of sensitive topics.
In some settings and for some topics, it may be advis-
able to use a white noise machine during the interview.
Providing food (but not alcohol) is a good way to ease
participants into the experience and provide a natural
“ice-breaker.” Depending on the topic under discussion,
investigators may wish to have a box of tissues and/or
mental health crisis services available in case partici-
pants become emotional during the study.
If there is an observational component to the study,
the observations should be conducted in a way that
minimizes interference with the “real” experience.
Example: A research team wishes to learn more
about electronic medical records’ effect on physi-
cian decisions about asthma treatment. They
arrange to conduct observations of physician docu-
mentation in real time. They wear research badges,
but dress in scrubs and sit inconspicuously in the
corner of the physician work room to allow obser-
vation of behavior as it would naturally occur with-
out their presence.
An acknowledgement of these logistic components
(e.g., in what setting the data was collected) is an impor-
tant part of a qualitative manuscript.
Complementing Audio/Visual Data With Written
Data
Most qualitative data collection includes some form of
note taking in addition to audio or video recording.16 In
one-on-one interviews, the note taker is usually the
interviewer. For focus groups, a dedicated note taker is
usually present (in addition to the focus group leader).
This note taker can also serve a dual role of assisting
with focus group logistics on the day of the session,
such as directing lost participants by telephone, assist-
ing with consent processes, and greeting late arrivals.
For observational studies, a data collector may take
notes in real time (“field notes”) or after leaving the
study site.
There are three primary reasons to take notes during
and after data collection. The first is that, even with two
recording devices, data collection sometimes fails. Good
notes may be the only way to keep a specific data set
from being completely lost. The second reason is that,
particularly with focus groups and ethnographic stud-
ies, the audio recording may not do justice to the full
complexities of a discussion. For instance, audio cannot
capture hand gestures or facial expressions, may not
capture who is speaking at a particular time, and may
miss softer-speaking participants. By complementing a
good note taker’s written observations with an audio
transcript, additional complexity of the data may be
captured.5 The third reason is that note taking can serve
as an initial form of analysis. For instance, in many
qualitative studies a summary document is created
immediately at the conclusion of the session. Indeed, a
good practice is to have the researchers “debrief” with
one another in person or in written form after an epi-
sode of data collection is completed. This debriefing
allows corrections to the process if problems are
observed with the data collection guide and/or the
1108
logistics, helps with iterative refinement of the inter-
view/data collection guide as new data is acquired, and
serves as an interim form of analysis, which is particu-
larly important if researchers are trying to determine
data saturation in a short period of time.1 Qualitative
manuscripts should describe the process of note taking
and its role in informing both changes in data collection
strategies and in analysis.1
DATA MANAGEMENT AND ANALYSIS
Rigor in qualitative research is ensured through trans-
parency in the coding and analysis process. By
acknowledging the background of the researchers per-
forming coding and analysis, and adequately describing
the coding and analytic process that was used,
researchers of similar backgrounds, using similar meth-
ods, should be able to come to the same conclusions if
provided the data set. The following guidelines facilitate
this level of rigor in qualitative coding and analysis.
Managing the Data
Data from recordings should be transcribed and dei-
dentified. It can be transcribed by the researchers them-
selves, although so doing is time-consuming, or by a
HIPAA-compliant professional transcription service.
Some qualitative researchers believe that the analysis
process starts in the process of transcribing. Determin-
ing how to transcribe and code pauses, verbal inflec-
tions, etc., is actually a decision about data analysis.
Early-stage qualitative researchers who are learning the
methodology will benefit from transcribing one or more
sessions initially, as it provides insight into the qualita-
tive process and logistics of recording data and allows
data collectors to be self-reflective about their data col-
lection techniques.
Transcripts should be formatted in a way that allows
easy coding and that matches the standards of the
intended software used to support the data analysis. An
example transcript format (with codes applied) is avail-
able in Data Supplement S1 (available as supporting
information in the online version of this paper). It is
often easiest to put line numbers on the transcript for
coding purposes, although these should be removed
before the transcript is uploaded to data management
software.17 Many researchers prefer to use data man-
agement software to organize transcripts, notes, mem-
os, and coding. Such software is discussed further
below.
Coding
What Are Codes, Anyhow? “Codes” are labels that the
research team applies to the data.10 Coding is designed
specifically to categorize the data in a way that can be
analyzed further. Each code corresponds to an idea or
concept. Codes differ from the themes that will emerge
after the analysis is complete. Some codes will need to
be applied to large amounts of data; others may be
more appropriately applied to specific words or
phrases. Although one may code the researcher’s ques-
tions to provide context during the analysis process,
ultimately only the participants’ words or actions serve
as analyzable data.
Ranney et al. • QUALITATIVE METHODS PART II
The Coding Structure. Creating the coding structure
is perhaps the most challenging part of qualitative
research and is what defines the qualitative approach.18
After reading the transcripts in part or in whole, the
researchers will work together to create and then refine
a codebook. In the process of coding and recoding early
transcripts, coding should be done as a group, so that
the codes themselves (and their definitions) can be
clearly delineated and agreed upon. The eventual code-
book will include an organized list of the “codes” as well
as specific, well-articulated definitions and examples for
each code. The codebook will serve to guide the staff
tasked with coding the entire set of transcripts in a con-
sistent and reliable fashion.19 A sample coding structure
is included with Data Supplement S1.
To develop a coding scheme, investigators can use a
variety of approaches. A common method in emergency
care research is “deductive” or informed by preexisting
hypotheses and preexisting research, such as in “frame-
work” or “content” analysis.20 In this coding and ana-
lytic approach, the researcher creates an initial coding
structure based off of the interview guide and then
refines the structure (iteratively) as interviews are com-
pleted. Some qualitative research, such as that using
large preexisting data sources, will also quantitatively
code (e.g., word counts).
In contrast, in “inductive” analytic approaches, such as
those informed by phenomenology or grounded theory,
investigators do not a priori impose a theoretical con-
struct or framework on the data, but are entirely open to
the ideas that arise from the participants.21 This inductive
approach allows for codes, themes, and ideas to arise
from the narrative—but nevertheless starts with some a
priori hypotheses and perspectives. This approach
requires having a greater number of data points prior to
development of a coding structure. In a more inductive
approach to coding and analysis, a researcher or team of
researchers will read multiple transcripts and begin to
create lists of themes, and memos about these themes,
that will then coalesce into a coding structure. This, too,
will be iteratively revised as coding progresses. Inductive
analytic approaches are generally more difficult and
time-consuming than deductive approaches, but are
extremely valuable to the field.
A strong coding structure will inform the researcher’s
analysis but is also, necessarily, informed by the analytic
approach. For instance, researchers who are using a
deductive approach will decide on many codes prior to
reading their transcripts, whereas those using an induc-
tive approach will read a number of transcripts prior to
identifying any potential codes. Similarly, qualitative
studies that reflect a certain theoretical background
should be sure to include codes for all of the elements
of this theory; a study that is informed by the Transthe-
oretical Model, for instance, should include codes for
specific “stages of change,” “decisional balance,” “self-
efficacy,” etc. The analytic approach should therefore be
defined a priori (see “Analysis” below for further details
about the analytic approach). Many researchers find
that their earlier codes are more descriptive (who, what,
where), whereas codes developed later in the process
are more analytic (how this is important, how it relates
to theory).
ACADEMIC EMERGENCY MEDICINE • September 2015, Vol. 22, No. 9 • www.aemj.org
Regardless of the methods used to create the initial
codebook, qualitative researchers should maintain flexi-
bility to revise the codebook: adding, subtracting, modi-
fying, and clarifying coding categories as the work
progresses. This process involves a “constant compar-
ison” of new data in the context of existing coding cate-
gories. It is important for the credibility of the work to
keep an “audit trail” of changes in the codebook as the
coding and analysis progresses.
The Coding Process. As the codebook is created, the
researchers will apply it to their data. The process of
applying codes to the content of the transcripts is often
stylistic; some researchers prefer to “chunk” larger pas-
sages with numerous codes, while others prefer to
“split” passages into small sections, each of which is
assigned only a few or single codes. “Chunking” saves
time during the coding process and maintains more
context, while “splitting” allows for easier identification
of coded statements without having to sort through
large sections of text and may save time during the
analysis phase (see Data Supplement S1 to see an exam-
ple of how codes have been applied to a transcript).
Much of the choice about coding style depends on
the research questions and analytic goals. For instance,
a researcher may choose to “chunk” broadly when
using framework analysis to improve intervention
development, or may “split,” coding line by line or even
word by word, when using grounded theory to develop
a theoretical explanation for why some caregivers have
negative experiences during their loved ones’ ED visit.
Many studies use both styles.
Most projects increase rigor by using double-coding,
with more than one researcher independently assigning
prespecified codes to the data. Once the coding struc-
ture becomes well defined, some research projects will
proceed using a single coder for remaining transcripts,
being sure to check in to avoid coder drift throughout
the coding process. This saves time. Other studies will
double-code ALL transcripts. The former approach is
more appropriate for large data sets or for coding
structures that are minimally interpretative. The latter
approach may be more appropriate for more inductive
work and/or small data sets. Some studies also use
advanced data management software to apply prelimi-
nary or structural codes to data, for instance, based on
word counts or question order.
Some journals require tracking of concordance
between the multiple coders, although this concept is
controversial with qualitative researchers.22 Some jour-
nals will also request a report of what strategies were
applied when discordance occurred to preserve the
integrity of the analysis. Concordance is generally calcu-
lated as a kappa or inter-rater reliability scale, in which
the number of agreed-upon codes is divided by the total
number of codes. It is possible to calculate concordance
by hand or using software (although we urge research-
ers to be aware of the limitations of their data manage-
ment software, if so doing).
In summary, during coding, the investigators will be
engaged in three tasks: 1) applying the predetermined
codes; 2) iteratively expanding on, and refining, the cod-
ing structure; and 3) taking notes about the data during
1109
the coding process. The notes can be attached to pieces
of data (such as a quotation), to explain what conclu-
sions the researchers drew, or they may be written as a
reflection on an entire code. These notes should be
saved and indexed, to inform the broader analysis.
Data Analysis
Software. There are a variety of data management
software packages for qualitative analysis. It is impor-
tant to note that these programs facilitate data analysis,
but do not perform any analytical techniques them-
selves. Commonly used software packages include
NVivo (formerly NUDIST), Atlas.ti, Microsoft Excel (for
simpler data sets), Anthropac, and others. Advantages
to using data management software include the ability
to quickly identify and categorize data into major cate-
gories and themes and the ability to assign demo-
graphic and clinical characteristics to the study
participants. Advanced software programs allow for
integration of data across categories, can provide a
basis for developing research frameworks for a particu-
lar topic (see “Coding” section above), and can even
autocode some information (e.g., identity of speaker,
sections of an interview) if a transcript is formatted cor-
rectly prior to upload.23 Excellent written and online
training guides exist for all software types; experienced
qualitative researchers can also be a good resource for
readers interested in further guidance.
The Analytic Process. The analysis of qualitative
research is designed to understand the data in context.
As described above, it begins with debriefs or field
notes, extends into the transcription process and devel-
opment of coding, and culminates in a consensus-based
development of themes.
Theme development is the process of identifying com-
mon threads between the pieces of the data, which have
been artificially divided and categorized by codes. Like
the coding process, it is often driven by specific frame-
works including inductive or deductive theories, as
described above and elsewhere.24 The process of theme
development usually occurs in the setting of team meet-
ings during which investigators consider the codes,
reread portions of the transcripts and describe, charac-
terize, or interpret the themes present in the text. Just
like codebook development, theme development repre-
sents the collective opinion of the qualitative research-
ers. Critical, theory-informed immersion in the coding
categories is a necessary first step toward developing
themes.
By comparing data between coding categories, look-
ing for differences in opinions between various types of
participants, applying the researchers’ own lived experi-
ences and theoretical perspectives, and collating and
refining themes in an iterative manner, a valid and reli-
able analysis can be achieved. Some researchers explic-
itly operationalize this process by: 1) starting with a
review of the text within a coding category; 2) using
data management software to compare the codes for
different types of participants (for instance, using Boo-
lean search terms, matrices of participant characteris-
tics, or network diagramming); 3) developing iterative,
evolving lists of emerging themes, and revising the
1110
codes accordingly; and 4) collaborating with other
researchers to compare and contrast emerging themes,
finally achieving consensus regarding overarching theo-
retical constructs. Final themes or theories can be con-
nected to preexisting hypotheses, theory, or some other
external framework, depending on the analytic para-
digm used (e.g., framework analysis).
A number of well-described techniques exist to
ensure the trustworthiness of analysis, such as peer
review, member checking, triangulation, and explicit
consideration of researcher bias. A good overview of
these techniques can be found in all of the textbooks
referenced, as well as in Malterud.18 These techniques
should be described in the research manuscript.
Example: Researchers were interested in how
emergency physicians (EPs) defined, used, and per-
ceived asthma management guidelines. They inter-
viewed EPs from a variety of practice and
geographic settings, recorded, transcribed, and
coded the narrative data. Using inductive tech-
niques, they categorized the main themes and
mapped them back to the central domains they
were investigating. For definition/awareness, they
identified the themes of engagement and relation-
ships between local and national recommendations.
For use, they identified communication and expec-
tation management. For attitudes, they identified
physician autonomy, standardization, and unin-
tended consequences. After identifying the themes,
they then presented the themes back to a group of
EPs to ensure that they were adequately represent-
ing their beliefs.
Unlike quantitative analysis, qualitative analysis
occurs contemporaneously with data collection. The two
forms of analysis also contrast in their explicit acknowl-
edgement of bias. Quantitative methods approach fac-
tors external to the hypothesis being tested as “noise”
that needs to be “controlled” for, using various method-
ologic and statistical approaches. In contrast, qualitative
methods embrace these external factors without
attempting to eliminate them, and the analysis acknowl-
edges these contextual factors and their effect on the
topic being studied. While quantitative analysis often
focuses on correlations and sorting of data independent
of context, qualitative analysis identifies connections
between participant statements and the observed events
to describe a coherent and whole phenomenon. Qualita-
tive analyses may produce ideas that can lead to testa-
ble hypotheses, but rarely test specific hypotheses on
their own.
Presenting the Data
There are many ways to present qualitative data, includ-
ing visual, audio, or written descriptions. Many qualita-
tive studies will divide the results section of a paper
according to the themes that emerged from the coding
analysis. Specific quotations, representing archetypical
or particularly rich descriptions of the themes that have
emerged, should be provided. Tables can be used to
present quotations or higher-level themes and ideas.
Some studies will also provide figures showing the net-
work of their constructs and themes. COREQ guidelines
Ranney et al. • QUALITATIVE METHODS PART II
encourage manuscripts to include quotations from a
wide variety of study participants, to allow independent
verification of the credibility of the data.1
It is generally not appropriate to present qualitative
findings using classic descriptive statistics, as this
implies psychometric and statistical validity that cannot
be claimed by qualitative methods.25 Instead, the
description of the data should highlight where there is
consensus among study participants (“many,” “most”)
and where there are important outlier opinions or con-
cepts (“with the exception,” “a few,” “one participant
stated”).
Other forms of data presentation include word trees,
tree maps, and other rapidly emerging strategies. A
classic overview of qualitative data visualization meth-
ods is Slone;26 the various types of data management
software are also frequently updating their abilities to
provide new data visualization techniques.
CHALLENGES AND BARRIERS
Potential Threats to Validity of a Qualitative Study
Just as with quantitative research, there are expected
threats to the validity of qualitative research that can
occur at any stage of the study.18 During study design,
if researchers create an overly focused interview guide,
they may unwittingly limit the data to meet their expec-
tations and obviate the possibility of considering alter-
native concepts. This is why some researchers choose
to use an inductive method to create their interview
guide, starting with open-ended questions to allow the
data to guide development of topics and questions. Dur-
ing data collection itself, interviewers sometimes use
leading questions, insert their own opinions into the
discussion, explicitly affirm or negate participants’ state-
ments, or otherwise influence what participants dis-
close; these errors may lead to answers that are
one-sided, socially desirable, or otherwise “thin.” Dur-
ing data analysis, anecdotalism represents a threat to
validity: this occurs when researchers may erroneously
attribute increased importance to interesting but poten-
tially nonrepresentative statements. Methods to avoid
anecdotalism include using multiple coders, “triangulat-
ing” data (when one’s own data or theory is compared
with other data sources/theories to corroborate find-
ings), and verifying data by seeking out statements that
might contradict or disprove the findings.5 Qualitative
studies may not be generalizable to other settings,
depending on the researchers’ sampling and analytic
methods.
Finally, the sheer amount of data available from a
qualitative study (“data overload”) may preclude ade-
quate analysis. Rigorous, credible qualitative analysis
requires that researchers fully immerse themselves in
the data. Cutting short the analytic process can result in
superficial or inaccurate conclusions.
Human Subjects Concerns
Often the greatest risk for subjects in a qualitative
research study is confidentiality.27 A participant’s voice
or face on a recording is considered protected health
information. Maintaining confidentiality of qualitative
recordings is therefore critical. Additionally, during
ACADEMIC EMERGENCY MEDICINE • September 2015, Vol. 22, No. 9 • www.aemj.org
data collection, participants often reveal personal, iden-
tifiable information (for instance, regarding their school,
their work, or their treatment history), which must be
carefully redacted from the transcript. It is particularly
important to be aware of potential violations of “third-
party” confidentiality (the confidentiality of a partici-
pant’s friends and family) whose names and stories may
be mentioned without their consent.
Focus groups offer an additional threat to confiden-
tiality in that study participants may have access to
other participants’ data (because it is shared in a group
setting). Therefore, focus group leaders must remind all
participants that information discussed in the group
must remain confidential. One can also caution focus
group participants to not disclose particularly private
information if they are concerned about confidentiality,
as other group members are not strictly bound to confi-
dentiality rules. It is helpful to provide, in the protocol,
an avenue for focus group participants to provide such
data privately in the course of their research participa-
tion.
Institutional review board (IRB) approval is always
needed for original qualitative research, even for
research with peers (e.g., with other physicians).28 Some
forms of secondary data analysis (e.g., analysis of preex-
isting social media) are often exempt, but researchers
should confirm with their local review board. Most IRBs
are comfortable with the fluidity of qualitative interview
guides, recognizing that interview questions will evolve
over the course of a study. Some researchers include
photo and voice release forms in their consent docu-
ments, depending on the eventual purpose of the data
and based on individual institutional and state policies
and laws. If researchers do wish to deidentify a record-
ing, to use it in future presentations or products, then
video recordings should be pixilated and audio software
should be used to alter voices. Regardless, all research-
ers should specify when the data will be destroyed.
CONCLUSIONS
Qualitative research offers opportunities for emergency
care researchers to explore novel topics, define new
survey or assessment methods, generate hypotheses,
and develop interventions. As with all forms of
research, it must be conducted with rigor at every stage
of the process, from initial interview development to
participant sampling, creation of a coding structure,
applying codes to transcripts, and identifying key
themes. With appropriate training and practice, emer-
gency physicians can conduct high-quality qualitative
studies and become critical evaluators of and contribu-
tors to the emerging body of qualitative emergency care
literature.
References
1. Tong A, Sainsbury P, Craig J. Consolidated criteria
for reporting qualitative research (COREQ): a 32-
item checklist for interviews and focus groups. Int J
Qual Health Care 2007;19:349–57.
2. Critical Appraisal Skills Programme. 10 Questions to
Help You Make Sense of Qualitative Research.
1111
Available at: http://media.wix.com/ugd/dded87_29c5
b002d99342f788c6ac670e49f274.pdf. Accessed Jul 20,
2015.
3. Mello MJ, Merchant RC, Clark MA. Surveying
emergency medicine. Acad Emerg Med 2013;20:409–
12.
4. Denzin N, Lincoln Y. Handbook of Qualitative
Research. 2nd ed. Thousand Oaks, CA: Sage Publi-
cations Inc, 2000.
5. Crabtree BF, Miller WL. Doing Qualitative Research.
2nd ed. Thousand Oaks, CA: Sage Publications Inc,
1999.
6. Lincoln YS, Guba EG. Naturalistic Inquiry. Thousand
Oaks, CA: Sage Publications Inc, 1985.
7. Schensul SL, Schensul JJ, LeCompte MD. Essential
Ethnographic Methods: Observations, Interviews,
and Questionnaires (Book 2 in Ethnographer’s
Toolkit). Walnut Creek, CA: AltaMira Press, 1999.
8. Kawulich BB. Participant observation as a data col-
lection method. Forum Quar Social Res 2005;6:arti-
cle 43.
9. Roulston K, deMarrais K, Lewis JB. Learning to
interview in the social sciences. Qual Inquiry
2003;9:643–68.
10. Miles MB, Huberman AM. Qualitative Data Analy-
sis: An Expanded Sourcebook. 2nd ed. Thousand
Oaks, CA: Sage Publications Inc., 1994.
11. Richards H, Emslie C. The ‘doctor’ or the ‘girl from
the University’? Considering the influence of profes-
sional roles on qualitative interviewing. Fam Pract
2000;17:71–5.
12. Ulin PR, Robinson ET, Tolley EE. Qualitative
Methods in Public Health: A Field Guide for Applied
Research. San Francisco, CA: Jossey-Bass, 2005.
13. Curtis S, Gesler W, Smith G, Washburn S. Ap-
proaches to sampling and case selection in qualita-
tive research: examples in the geography of health.
Soc Sci Med 2000;50:1001–14.
14. Morse JM. Strategies for sampling. In: Morse J, edi-
tor. Qualitative Nursing Research: A Contemporary
Dialogue (Rev Ed.). Thousand Oaks, CA: Sage, 1991,
pp. 127–31.
15. Guest G, Bunce A, Johnson L. How many interviews
are enough? An experiment with data saturation and
variability. Field Methods 2006;18:59–82.
16. Maloney RS, Paolisso M. What can digital audio
data do for you? Field Methods 2001;13:88–96.
17. McLellan E, MacQueen KM, Neidig JL. Beyond the
qualitative interview: data preparation and tran-
scription. Field Methods 2003;15:63–84.
18. Malterud K. Qualitative research: standards, chal-
lenges, and guidelines. Lancet 2001;358:483–8.
19. Carey JW, Morgan M, Oxtoby MJ. Intercoder
agreement in analysis of responses to open-ended
interview questions: examples from tuberculosis
research. Field Methods 1996;8:1–5.
20. Krippendorff K. Content Analysis: An Introduction
to Its Methodology. 3rd ed. Thousand Oaks, CA:
Sage Publications, 2012.
21. Starks H, Brown Trinidad S. Choose your method: a
comparison of phenomenology, discourse analysis,
and grounded theory. Qual Health Res 2007;17:
1372–80.
1112
22. Barbour RS. Checklists for improving rigour in
qualitative research: a case of the tail wagging the
dog? BMJ. 2001;322:1115–7.
23. Fielding N, Lee R. Computer Analysis and Qualita-
tive Research. Thousand Oaks, CA: Sage Publica-
tions, 1998.
24. Sandelowski M. Theory unmasked: the uses and
guises of theory in qualitative research. Res Nurs
Health 1993;16:213–8.
25. Sandelowski M, Barroso J. Classifying the findings
in qualitative studies. Qual Health Res 2003;13:905–
23.
26. Slone DJ. Visualizing qualitative information. Quali-
tative Rep 2009;14:488–97.
Ranney et al. • QUALITATIVE METHODS PART II
27. Richards HM, Schwartz LJ. Ethics of qualitative
research: are there special issues for health services
research? Fam Pract 2002;19:135–9.
28. Lincoln YS, Tierney WG. Qualitative research and
institutional review boards. Qual Inquiry
2004;10:219–34.
Supporting Information
The following supporting information is available in the
online version of this paper:
Data Supplement S1. Excerpt from transcript about
a newly prescribed asthma medication from the emer-
gency department, with coding.