Sampling methods and sample populations in quantitative mass communication research

studies: A 15-year census of six journals

Codebook

Inclusion criteria

1. Mass communication study

Code 1 if it is a mass communication study; otherwise, code 0.

A study is considered a mass communication study if it deals with any element of a

message (e.g., sender, content, technology/channel, effect) that may be disseminated to
more than one person.
Mass communication studies also include the effects of media messages on
participants’ perceptions towards a particular topic or even on participants’ behavior.
Research that uses participants working in media or mass communication industries is
also considered a mass communication study.
Other types of mass communication studies can include research investigating tactics
to better communicate to mass audiences (e.g., aspects of crisis communication), origins
of media practices/law, as well as any topic that enhances our understanding of the role of
media in our lives.
As a rule of thumb, assume that the study is a mass communication one unless
nothing in the title and/or abstract hint at any media-related element mentioned above. In
some cases, it may be necessary to look at the methods section to see what the
researchers actually did.

Examples of mass communication studies:

- Surveys of professionals who create or disseminate mass media messages (e.g.,
journalists).
- Surveys about the reasons why people use certain media.
- Experiments testing the effects of messages (may be visual, text-based, video-based,
etc.) that can be used in political or health campaigns.
- Interviews with audience members regarding their media diet and/or thoughts about
media representations.
- Focus groups discussing ads.
- Interviews with crisis communication managers.

Examples of non-mass communication studies:

- Studies using dyads (e.g., a two-person conversation), even when mediated via a
computer, phone, etc.
- Studies focused on interpersonal communication, such as how parents communicate
with their children.
2. One participant or more

Code 1 if the study has one or more participants; otherwise, code 0.

A participant is an individual who is included in a study. The people who perform

research (i.e., coders, interviewers) are not participants.
A content or textual analysis does not have any participants, unless it also includes
data from interviews, surveys, or experiments.
A study that uses data not collected by the researchers who authored the study (i.e.,
secondary data analysis; online video game logs) may or may not qualify as having one
or more participant. This depends on the unit or level of analysis.
If the research focuses on the characteristics or responses of individuals, and analyzes
these, it qualifies as having one or more participant.
If the research analyzes an aggregate of study results and uses the study as the unit of
analysis, for example, it does not qualify as having one or more participants. Similarly, a
metanalysis (a study of several published studies) does not have any participants.

3. Quantitative data

Code 1 if the primary research method in a study is a survey or experiment. Code 0 if the
primary research method in a study is not a survey or experiment.

4. Data collected only in the U.S.

Code 1:
If a study does not specifically identify its participants as being from the U.S. (e.g., a
sample of students from a college located in the Midwest), and researchers are based at a
U.S. university, assume that the participants are from the U.S. and code as 1.
If the researchers state that they used students or recruited participants face-to-face,
look at the researchers’ institution/location and assume that the participants were
recruited from there. If the institution is in the United States, code as 1.
If a study includes participants who are not U.S. citizens but are living in the U.S.
(e.g., a study about how recent immigrants to the U.S. use media to adapt to their new
country), these participants would be conceptualized as U.S. participants because they
now live in the U.S. and would be accounted for in the U.S. Census. Code as 1.

Code 0:
If the researchers state that they recruited participants online or using some online
database accessible beyond the United States, assume that some participants are not from
the U.S., and code as 0.
If researchers are not U.S.-based, examine the methods section to determine if
participants were recruited in the U.S. or not. If participants were recruited in the U.S.
and other countries, code as 0.
 Check the methods and results (including tables) to make sure that all the participants
were recruited in the U.S.

Study variables

1. Data collection method

Options:
- Survey
- Experiment

An experiment will include the assignment of participants into different conditions.

“Survey” vs “Experiment”: The use of a questionnaire does not necessarily mean that
the method is a survey. For example, most experiments include a questionnaire to test the
effects of the experimental design/manipulation. So, if participants take part in an
experiment and then answer a questionnaire, the study should be coded as experiment.

2. Sampling method

Options:
- Non-probability
- Probability

From the Internet glossary of statistical terms: “a random [probability] sample is a set of
items that have been drawn from a population in such a way that each time an item was
selected, every item in the population had an equal opportunity to appear in the sample.
In practical terms, it is not so easy to draw a random sample.”
This means that if researchers have access to all members of their sample population
of study and recruit participants from it, it would be a probability sample (for example, if
researchers are interested in how people in Lawrence feel about a topic and partner with
the mayor’s office to gain access to all residents, it would be a probability sample).
Note that a representative sample is not necessarily a probability sample.
If researchers use a probability or random sample, they will usually emphasize it, as it
is a ‘gold standard’ when conducting surveys.
A non-probability sample is a sample in which participants have been recruited using
convenience techniques, such as asking students in specific classes, asking participants to
recommend other participants, etc.
Participants recruited from online platforms or purchased panels are usually not
probability samples because only a limited number of people can be reached via these
means.

3. Sample population

Options:
- College student population
 o All participants need to be college students for “College student
population” to be coded as such. College students can be undergraduate
and/or graduate students.
- Not college student population
o Any other population, or including those that mix college students and
non-students.

4. Research funding

Code 1 if the study was supported with a grant. Otherwise, code 0.

If grant support is not specified (usually in the methods section or at the end of the
article), code as 0.