Once you have chosen a topic to investigate, you need to decide which type of method is best to

study it. This is one of the most important choices you will make on your research journey.
Understanding the value of each of the methods described in this textbook to answer different
questions allows you to be able to plan your own studies with more confidence, critique the studies
others have done, and provide advice to your colleagues and friends on what type of research they
should do to answer questions they have. After briefly reviewing quantitative research assumptions,
this chapter is organized in three parts or sections. These parts can also be used as a checklist
when working through the steps of your study. Specifically, part 1 focuses on planning a
quantitative study (collecting data), part two explains the steps involved in doing a quantitative
study, and part three discusses how to make sense of your results (organizing and analyzing data).

Quantitative Worldview Assumptions: A Review

In chapter 2, you were introduced to the unique assumptions quantitative research holds about
knowledge and how it is created, or what the authors referred to in chapter one as "epistemology."
Understanding these assumptions can help you better determine whether you need to use
quantitative methods for a particular research study in which you are interested.
Quantitative researchers believe there is an objective reality, which can be measured. "Objective"
here means that the researcher is not relying on their own perceptions of an event. S/he is
attempting to gather "facts" which may be separate from people's feeling or perceptions about the
facts. These facts are often conceptualized as "causes" and "effects." When you ask research
questions or pose hypotheses with words in them such as "cause," "effect," "difference between,"
and "predicts," you are operating under assumptions consistent with quantitative methods. The
overall goal of quantitative research is to develop generalizations that enable the researcher to
better predict, explain, and understand some phenomenon.
Because of trying to prove cause-effect relationships that can be generalized to the population at
large, the research process and related procedures are very important for quantitative methods.
Research should be consistently and objectively conducted, without bias or error, in order to be
considered to be valid (accurate) and reliable (consistent). Perhaps this emphasis on accurate and
standardized methods is because the roots of quantitative research are in the natural and physical
sciences, both of which have at their base the need to prove hypotheses and theories in order to
better understand the world in which we live. When a person goes to a doctor and is prescribed
some medicine to treat an illness, that person is glad such research has been done to know what
the effects of taking this medicine is on others' bodies, so s/he can trust the doctor's judgment and
take the medicines.
As covered in chapters 1 and 2, the questions you are asking should lead you to a certain research
method choice. Students sometimes want to avoid doing quantitative research because of fear of
math/statistics, but if their questions call for that type of research, they should forge ahead and use
it anyway. If a student really wants to understand what the causes or effects are for a particular
phenomenon, they need to do quantitative research. If a student is interested in what sorts of things
might predict a person's behavior, they need to do quantitative research. If they want to confirm the
finding of another researcher, most likely they will need to do quantitative research. If a student
wishes to generalize beyond their participant sample to a larger population, they need to be
conducting quantitative research.
So, ultimately, your choice of methods really depends on what your research goal is. What do you
really want to find out? Do you want to compare two or more groups, look for relationships between
certain variables, predict how someone will act or react, or confirm some findings from another
study? If so, you want to use quantitative methods.
Self-Quiz
A topic such as self-esteem can be studied in many ways. Listed below are some example RQs
about self-esteem. Which of the following research questions should be answered with quantitative
methods?
1. Is there a difference between men's and women's level of self- esteem?
2. How do college-aged women describe their ups and downs with self-esteem?
3. How has "self-esteem" been constructed in popular self-help books over time?
4. Is there a relationship between self-esteem levels and communication apprehension?
What are the advantages of approaching a topic like self-esteem using quantitative methods? What
are the disadvantages?
For more information, see the following website:
Analyse This!!! Learning to analyse quantitative data(link is external)
Answers: 1 & 4

Quantitative Methods Part One: Planning Your Study

Planning your study is one of the most important steps in the research process when doing
quantitative research. As seen in the diagram below, it involves choosing a topic, writing research
questions/hypotheses, and designing your study. Each of these topics will be covered in detail in
this section of the chapter.
Topic Choice
Decide on Topic
How do you go about choosing a topic for a research project? One of the best ways to do this is to
research something about which you would like to know more. Your communication professors will
probably also want you to select something that is related to communication and things you are
learning about in other communication classes.
When the authors of this textbook select research topics to study, they choose things that pique
their interest for a variety of reasons, sometimes personal and sometimes because they see a need
for more research in a particular area. For example, April Chatham-Carpenter studies adoption
return trips to China because she has two adopted daughters from China and because there is very
little research on this topic for Chinese adoptees and their families; she studied home vs. public
schooling because her sister home schools, and at the time she started the study very few
researchers had considered the social network implications for home schoolers
(cf. http://www.uni.edu/chatham/homeschool.html).
When you are asked in this class and other classes to select a topic to research, think about topics
that you have wondered about, that affect you personally, or that know have gaps in the research.
Then start writing down questions you would like to know about this topic. These questions will help
you decide whether the goal of your study is to understand something better, explain causes and
effects of something, gather the perspectives of others on a topic, or look at how language
constructs a certain view of reality.

Review Previous Research

In quantitative research, you do not rely on your conclusions to emerge from the data you collect.
Rather, you start out looking for certain things based on what the past research has found. This is
consistent with what was called in chapter 2 as a deductive approach (Keyton, 2011), which also
leads a quantitative researcher to develop a research question or research problem from reviewing
a body of literature, with the previous research framing the study that is being done. So, reviewing
previous research done on your topic is an important part of the planning of your study. As seen in
chapter 3 and the Appendix, to do an adequate literature review, you need to identify portions of
your topic that could have been researched in the past. To do that, you select key terms of
concepts related to your topic.
Some people use concept maps to help them identify useful search terms for a literature review. For
example, see the following website:
Concept Mapping: How to Start Your Term Paper Research(link is external).
Narrow Topic to Researchable Area
Once you have selected your topic area and reviewed relevant literature related to your topic, you
need to narrow your topic to something that can be researched practically and that will take the
research on this topic further. You don't want your research topic to be so broad or large that you
are unable to research it. Plus, you want to explain some phenomenon better than has been done
before, adding to the literature and theory on a topic. You may want to test out what someone else
has found, replicating their study, and therefore building to the body of knowledge already created.
To see how a literature review can be helpful in narrowing your topic, see the following
sources. Narrowing or Broadening Your Research Topic(link is external) and How
to Conduct a Literature Review in Social Science(link is external)
Research Questions & Hypotheses
Write Your Research Questions (RQs) and/or Hypotheses (Hs)
Once you have narrowed your topic based on what you learned from doing your review of literature,
you need to formalize your topic area into one or more research questions or hypotheses. If the
area you are researching is a relatively new area, and no existing literature or theory can lead you
to predict what you might find, then you should write a research question. Take a topic related to
social media, for example, which is a relatively new area of study. You might write a research
question that asks:

"Is there a difference between how 1st year and 4th year college students use Facebook to
communicate with their friends?"
If, however, you are testing out something you think you might find based on the findings of a large
amount of previous literature or a well-developed theory, you can write a hypothesis. Researchers
often distinguish between null and alternative hypotheses. The alternative hypothesis is what you
are trying to test or prove is true, while the null hypothesis assumes that the alternative hypothesis
is not true. For example, if the use of Facebook had been studied a great deal, and there were
theories that had been developed on the use of it, then you might develop an alternative
hypothesis, such as: "First-year students spend more time on using Facebook to communicate with
their friends than fourth-year students do." Your null hypothesis, on the other hand, would be: "First-
year students do not spend any more time using Facebook to communication with their friends than
fourth-year students do." Researchers, however, only state the alternative hypothesis in their
studies, and actually call it "hypothesis" rather than "alternative hypothesis."
Process of Writing a Research Question/Hypothesis.
Once you have decided to write a research question (RQ) or hypothesis (H) for your topic, you
should go through the following steps to create your RQ or H.
1. Name the concepts from your overall research topic that you are interested in studying.
RQs and Hs have variables, or concepts that you are interested in studying. Variables can
take on different values. For example, in the RQ above, there are at least two variables – year
in college and use of Facebook (FB) to communicate. Both of them have a variety of levels
within them.
When you look at the concepts you identified, are there any concepts which seem to be
related to each other? For example, in our RQ, we are interested in knowing if there is a
difference between first-year students and fourth-year students in their use of FB, meaning
that we believe there is some connection between our two variables.
2. If you think the variables you have identified are somehow related to each other, do the
following. If not, start your RQ with a word such as "what," "how," "why," or "to what extent,"
since you are most likely interested in describing something rather than seeing what the
connection is between two or more variables or concepts.
1. Decide what type of a relationship you would like to study between the variables. Do you
think one causes the other? Does a difference in one create a difference in the other? As
the value of one changes, does the value of the other change?
2. Identify which one of these concepts is the independent (or predictor) variable, or the
concept that is perceived to be the cause of change in the other variable? Which one is
the dependent (criterion) variable, or the one that is affected by changes in the
independent variable? In the above example RQ, year in school is the independent
variable, and amount of time spent on Facebook communicating with friends is the
dependent variable. The amount of time spent on Facebook depends on a person's year
in school.
If you're still confused about independent and dependent variables, check out the
following site:
Independent & Dependent Variables(link is external).
3. Express the relationship between the concepts as a single sentence – in either a
hypothesis or a research question.
For example, "is there a difference between international and American students on their
perceptions of the basic communication course," where cultural background and
perceptions of the course are your two variables. Cultural background would be the
independent variable, and perceptions of the course would be your dependent variable.
More examples of RQs and Hs are provided in the next section.
 APPLICATION: Try the above steps with your topic now. Check with your instructor to
see if s/he would like you to send your topic and RQ/H to him/her via e-mail.

Types of Research Questions/Hypotheses

Once you have written your RQ/H, you need to determine what type of research question or
hypothesis it is. This will help you later decide what types of statistics you will need to run to answer
your question or test your hypothesis. There are three possible types of questions you might ask,
and two possible types of hypotheses. The first type of question cannot be written as a hypothesis,
but the second and third types can.

Descriptive Question.
The first type of question is a descriptive question. If you have only one variable or concept you are
studying, OR if you are not interested in how the variables you are studying are connected or
related to each other, then your question is most likely a descriptive question.
This type of question is the closest to looking like a qualitative question, and often starts with a
"what" or "how" or "why" or "to what extent" type of wording. What makes it different from a
qualitative research question is that the question will be answered using numbers rather than
qualitative analysis. Some examples of a descriptive question, using the topic of social media,
include the following.

"To what extent are college-aged students using Facebook to communicate with their friends?"
"Why do college-aged students use Facebook to communicate with their friends?"

Notice that neither of these questions has a clear independent or dependent variable, as there is no
clear cause or effect being assumed by the question. The question is merely descriptive in nature. It
can be answered by summarizing the numbers obtained for each category, such as by providing
percentages, averages, or just the raw totals for each type of strategy or organization. This is true
also of the following research questions found in a study of online public relations strategies:

"What online public relations strategies are organizations implementing to combat phishing" (Baker,
Baker, & Tedesco, 2007, p. 330), and
"Which organizations are doing most and least, according to recommendations from anti- phishing
advocacy recommendations, to combat phishing" (Baker, Baker, & Tedesco, 2007, p. 330).

The researchers in this study reported statistics in their results or findings section, making it clearly
a quantitative study, but without an independent or dependent variable; therefore, these research
questions illustrate the first type of RQ, the descriptive question.

Difference Question/Hypothesis.
The second type of question is a question/hypothesis of difference, and will often have the word
"difference" as part of the question. The very first research question in this section, asking if there is
a difference between 1st year and 4th year college students' use of Facebook, is an example of this
type of question. In this type of question, the independent variable is some type of grouping or
categories, such as age. Another example of a question of difference is one April asked in her
research on home schooling: "Is there a difference between home vs. public schoolers on the size
of their social networks?" In this example, the independent variable is home vs. public schooling (a
group being compared), and the dependent variable is size of social networks. Hypotheses can also
be difference hypotheses, as the following example on the same topic illustrates: "Public schoolers
have a larger social network than home schoolers do."

Relationship/Association Question/Hypothesis.
The third type of question is a relationship/association question or hypothesis, and will often have
the word "relate" or "relationship" in it, as the following example does: "There is a relationship
between number of television ads for a political candidate and how successful that political
candidate is in getting elected." Here the independent (or predictor) variable is number of TV ads,
and the dependent (or criterion) variable is the success at getting elected. In this type of question,
there is no grouping being compared, but rather the independent variable is continuous (ranges
from zero to a certain number) in nature. This type of question can be worded as either a
hypothesis or as a research question, as stated earlier.
Self-Quiz:
Test out your knowledge of the above information, by answering the following questions about the
RQ/H listed below. (Remember, for a descriptive question there are no clear independent &
dependent variables.)
 What is the independent variable (IV)?
 What is the dependent variable (DV)?
 What type of research question/hypothesis is it? (descriptive, difference,
relationship/association)
1. "Is there a difference on relational satisfaction between those who met their current partner
through online dating and those who met their current partner face-to-face?"
2. "How do Fortune 500 firms use focus groups to market new products?"
3. "There is a relationship between age and amount of time spent online using social media."
Answers:
RQ1 is a difference question, with type of dating being the IV and relational satisfaction being the
DV.
RQ2 is a descriptive question with no IV or DV.
RQ3 is a relationship hypothesis with age as the IV and amount of time spent online as the DV.

Design Your Study

The third step in planning your research project, after you have decided on your topic/goal and
written your research questions/hypotheses, is to design your study which means to decide how to
proceed in gathering data to answer your research question or to test your hypothesis. This step
includes six things to do. [NOTE: The terms used in this section will be defined as they are used.]
1. Decide type of study design: Experimental, quasi-experimental, non-experimental.
2. Decide kind of data to collect: Survey/interview, observation, already existing data.
3. Operationalize variables into measurable concepts.
4. Determine type of sample: Probability or non-probability.
5. Decide how you will collect your data: face-to-face, via e-mail, an online survey, library
research, etc.
6. Pilot test your methods.
Types of Study Designs
With quantitative research being rooted in the scientific method, traditional research is structured in
an experimental fashion. This is especially true in the natural sciences, where they try to prove
causes and effects on topics such as successful treatments for cancer. For example, the University
of Iowa Hospitals and Clinics regularly conduct clinical trials to test for the effectiveness of certain
treatments for medical conditions (University of Iowa Hospitals & Clinics: Clinical
Trials(link is external)). They use human participants to conduct such research, regularly
recruiting volunteers. However, in communication, true experiments with treatments the researcher
controls are less necessary and thus less common. It is important for the researcher to understand
which type of study s/he wishes to do, in order to accurately communicate his/her methods to the
public when describing the study.
There are three possible types of studies you may choose to do, when embarking on quantitative
research: (a) True experiments, (b) quasi-experiments, and (c) non-experiments.
For more information to read on these types of designs, take a look at the following website and
related links in it:
Types of Designs(link is external).
The following flowchart should help you distinguish between the three types of study designs
described below.
True Experiments.
The first two types of study designs use difference questions/hypotheses, as the independent
variable for true and quasi-experiments is nominal or categorical (based on categories or
groupings), as you have groups that are being compared. As seen in the flowchart above, what
distinguishes a true experiment from the other two designs is a concept called "random
assignment." Random assignment means that the researcher controls to which group the
participants are assigned. April's study of home vs. public schooling was NOT a true experiment,
because she could not control which participants were home schooled and which ones were public
schooled, and instead relied on already existing groups.
An example of a true experiment reported in a communication journal is a study investigating the
effects of using interest-based contemporary examples in a lecture on the history of public relations,
in which the researchers had the following two hypotheses: "Lectures utilizing interest- based
examples should result in more interested participants" and "Lectures utilizing interest- based
examples should result in participants with higher scores on subsequent tests of cognitive recall"
(Weber, Corrigan, Fornash, & Neupauer, 2003, p. 118). In this study, the 122 college student
participants were randomly assigned by the researchers to one of two lecture video viewing groups:
a video lecture with traditional examples and a video with contemporary examples. (To see the
results of the study, look it up using your school's library databases).

Weber, K., Corrigan, M., Fornash, B., & Neupauer, N. C. (2003). The effect of interest on recall: An
experiment. Communication Research Reports, 20(2), 116-123.

A second example of a true experiment in communication is a study of the effects of viewing either
a dramatic narrative television show vs. a nonnarrative television show about the consequences of
an unexpected teen pregnancy. The researchers randomly assigned their 367 undergraduate
participants to view one of the two types of shows.

Moyer-Gusé, E., & Nabi, R. L. (2010). Explaining the effects of narrative in an entertainment
television program: Overcoming resistance to persuasion. Human Communication Research, 36, 26-
52.

A third example of a true experiment done in the field of communication can be found in the
following study.

Jensen, J. D. (2008). Scientific uncertainty in news coverage of cancer research: Effects of hedging
on scientists' and journalists' credibility. Human Communication Research, 34, 347-369.

In this study, Jakob Jensen had three independent variables. He randomly assigned his 601
participants to 1 of 20 possible conditions, between his three independent variables, which were (a)
a hedged vs. not hedged message, (b) the source of the hedging message (research attributed to
primary vs. unaffiliated scientists), and (c) specific news story employed (of which he had five
randomly selected news stories about cancer research to choose from). Although this study was
pretty complex, it does illustrate the true experiment in our field since the participants were
randomly assigned to read a particular news story, with certain characteristics.

Quasi-Experiments.
If the researcher is not able to randomly assign participants to one of the treatment groups (or
independent variable), but the participants already belong to one of them (e.g., age; home vs. public
schooling), then the design is called a quasi-experiment. Here you still have an independent
variable with groups, but the participants already belong to a group before the study starts, and the
researcher has no control over which group they belong to.
An example of a hypothesis found in a communication study is the following: "Individuals high in
trait aggression will enjoy violent content more than nonviolent content, whereas those low in trait
aggression will enjoy violent content less than nonviolent content" (Weaver & Wilson, 2009, p. 448).
In this study, the researchers could not assign the participants to a high or low trait aggression
group since this is a personality characteristic, so this is a quasi-experiment. It does not have any
random assignment of participants to the independent variable groups. Read their study, if you
would like to, at the following location.

Weaver, A. J., & Wilson, B. J. (2009). The role of graphic and sanitized violence in the enjoyment of
television dramas. Human Communication Research, 35 (3), 442-463.

Benoit and Hansen (2004) did not choose to randomly assign participants to groups either, in their
study of a national presidential election survey, in which they were looking at differences between
debate and non-debate viewers, in terms of several dependent variables, such as which candidate
viewers supported. If you are interested in discovering the results of this study, take a look at the
following article.

Benoit, W. L., & Hansen, G. J. (2004). Presidential debate watching, issue knowledge, character
evaluation, and vote choice. Human Communication Research, 30 (1), 121-144.

Non-Experiments.
The third type of design is the non-experiment. Non-experiments are sometimes called survey
designs, because their primary way of collecting data is through surveys. This is not enough to
distinguish them from true experiments and quasi-experiments, however, as both of those types of
designs may use surveys as well.
What makes a study a non-experiment is that the independent variable is not a grouping or
categorical variable. Researchers observe or survey participants in order to describe them as they
naturally exist without any experimental intervention. Researchers do not give treatments or
observe the effects of a potential natural grouping variable such as age. Descriptive and
relationship/association questions are most often used in non-experiments.
Some examples of this type of commonly used design for communication researchers include the
following studies.

 Serota, Levine, and Boster (2010) used a national survey of 1,000 adults to determine the
prevalence of lying in America (see Human Communication Research, 36, pp. 2-25).
 Nabi (2009) surveyed 170 young adults on their perceptions of reality television on cosmetic
surgery effects, looking at several things: for example, does viewing cosmetic surgery
makeover programs relate to body satisfaction (p. 6), finding no significant relationship
between those two variables (see Human Communication Research, 35, pp. 1-27).
  Derlega, Winstead, Mathews, and Braitman (2008) collected stories from 238 college students
on reasons why they would disclose or not disclose personal information within close
relationships (see Communication Research Reports, 25, pp. 115-130). They coded the
participants' answers into categories so they could count how often specific reasons were
mentioned, using a method called content analysis, to answer the following research questions:
RQ1: What are research participants' attributions for the disclosure and nondisclosure of highly
personal information?
RQ2: Do attributions reflect concerns about rewards and costs of disclosure or the tension between
openness with another and privacy?
RQ3: How often are particular attributions for disclosure/nondisclosure used in various types of
relationships? (p. 117)

All of these non-experimental studies have in common no researcher manipulation of an

independent variable or even having an independent variable that has natural groups that are being
compared.

Self-Quiz:
Identify which design discussed above should be used for each of the following research questions.
1. Is there a difference between generations on how much they use MySpace?
2. Is there a relationship between age when a person first started using Facebook and the
amount of time they currently spend on Facebook daily?
3. Is there a difference between potential customers' perceptions of an organization who are
shown an organization's Facebook page and those who are not shown an organization's
Facebook page?
[HINT: Try to identify the independent and dependent variable in each question above first, before
determining what type of design you would use. Also, try to determine what type of question it is –
descriptive, difference, or relationship/association.]
Answers:
1. Quasi-experiment
2. Non-experiment
3. True Experiment

Data Collection Methods

Once you decide the type of quantitative research design you will be using, you will need to
determine which of the following types of data you will collect: (a) survey data, (b) observational
data, and/or (c) already existing data, as in library research.

Surveys.
Using the survey data collection method means you will talk to people or survey them about their
behaviors, attitudes, perceptions, and demographic characteristics (e.g., biological sex, socio-
economic status, race). This type of data usually consists of a series of questions related to the
concepts you want to study (i.e., your independent and dependent variables). Both of April's studies
on home schooling and on taking adopted children on a return trip back to China used survey data.
On a survey, you can have both closed-ended and open-ended questions. Closed-ended questions,
can be written in a variety of forms. Some of the most common response options include the
following.
Likert responses – for example: for the following statement, ______ do you
strongly agree agree neutral disagree strongly disagree
Semantic differential – for example: does the following ______ make you
Happy ..................................... Sad
Yes-no answers for example: I use social media daily.
Yes / No.
One site to check out for possible response options
is http://www.360degreefeedback.net/media/ResponseScales.pdf(link is
external).
Researchers often follow up some of their closed-ended questions with an "other" category, in
which they ask their participants to "please specify," their response if none of the ones provided are
applicable. They may also ask open-ended questions on "why" a participant chose a particular
answer or ask participants for more information about a particular topic. If the researcher wants to
use the open-ended question responses as part of his/her quantitative study, the answers are
usually coded into categories and counted, in terms of the frequency of a certain answer, using a
method called content analysis, which will be discussed when we talk about already-existing artifacts
as a source of data.
Surveys can be done face-to-face, by telephone, mail, or online. Each of these methods has its own
advantages and disadvantages, primarily in the form of the cost in time and money to do the
survey. For example, if you want to survey many people, then online survey tools such as
surveygizmo.com and surveymonkey.com are very efficient, but not everyone has access to taking
a survey on the computer, so you may not get an adequate sample of the population by doing so.
Plus you have to decide how you will recruit people to take your online survey, which can be
challenging. There are trade-offs with every method.
For more information on things to consider when selecting your survey method, check out the
following website:

Selecting the Survey Method(link is external).

There are also many good sources for developing a good survey, such as the following websites.
Constructing the Survey(link is external)
Survey Methods(link is external)
Designing Surveys(link is external)
Observation.
A second type of data collection method is observation. In this data collection method, you make
observations of the phenomenon you are studying and then code your observations, so that you
can count what you are studying. This type of data collection method is often called interaction
analysis, if you collect data by observing people's behavior. For example, if you want to study the
phenomenon of mall-walking, you could go to a mall and count characteristics of mall-walkers. A
researcher in the area of health communication could study the occurrence of humor in an
operating room, for example, by coding and counting the use of humor in such a setting.
One extended research study using observational data collection methods, which is cited often in
interpersonal communication classes, is John Gottman's research, which started out in what is now
called "The Love Lab." In this lab, researchers observe interactions between couples, including
physiological symptoms, using coders who look for certain items found to predict relationship
problems and success.
Take a look at the YouTube video about "The Love Lab" at the following site to learn more about
the potential of using observation in collecting data for a research study: The "Love" Lab(link is
external).
Already-Existing Artifacts.
The third method of quantitative data collection is the use of already-existing artifacts. With this
method, you choose certain artifacts (e.g., newspaper or magazine articles; television programs;
webpages) and code their content, resulting in a count of whatever you are studying. With this data
collection method, researchers most often use what is called quantitative content analysis. Basically,
the researcher counts frequencies of something that occurs in an artifact of study, such as the
frequency of times something is mentioned on a webpage. Content analysis can also be used in
qualitative research, where a researcher identifies and creates text-based themes but does not do a
count of the occurrences of these themes. Content analysis can also be used to take open-ended
questions from a survey method, and identify countable themes within the questions.
Content analysis is a very common method used in media studies, given researchers are interested
in studying already-existing media artifacts. There are many good sources to illustrate how to do
content analysis such as are seen in the box below.
See the following sources for more information on content analysis.
Writing Guide: Content Analysis(link is external)
A Flowchart for the Typical Process of Content Analysis Research(link is
external)
What is Content Analysis?(link is external)
With content analysis and any method that you use to code something into categories, one key
concept you need to remember is inter-coder or inter-rater reliability, in which there are multiple
coders (at least two) trained to code the observations into categories. This check on coding is
important because you need to check to make sure that the way you are coding your observations
on the open-ended answers is the same way that others would code a particular item. To establish
this kind of inter-coder or inter-rater reliability, researchers prepare codebooks (to train their coders
on how to code the materials) and coding forms for their coders to use.
To see some examples of actual codebooks used in research, see the following website: Human
Coding--Sample Materials(link is external).
There are also online inter-coder reliability calculators some researchers use, such as the
following: ReCal: reliability calculation for the masses(link is external).
Regardless of which method of data collection you choose, you need to decide even more
specifically how you will measure the variables in your study, which leads us to the next planning
step in the design of a study.

Operationalization of Variables into Measurable Concepts

When you look at your research question/s and/or hypotheses, you should know already what your
independent and dependent variables are. Both of these need to be measured in some way. We
call that way of measuring operationalizing a variable. One way to think of it is writing a step by step
recipe for how you plan to obtain data on this topic. How you choose to operationalize your variable
(or write the recipe) is one all-important decision you have to make, which will make or break your
study. In quantitative research, you have to measure your variables in a valid (accurate) and
reliable (consistent) manner, which we discuss in this section. You also need to determine the level
of measurement you will use for your variables, which will help you later decide what statistical tests
you need to run to answer your research question/s or test your hypotheses. We will start with the
last topic first.
Level of Measurement
Level of measurement has to do with whether you measure your variables using categories or
groupings OR whether you measure your variables using a continuous level of measurement
(range of numbers). The level of measurement that is considered to be categorical in nature is
called nominal, while the levels of measurement considered to be continuous in nature are ordinal,
interval, and ratio. The only ones you really need to know are nominal, ordinal, and interval/ratio.

Nominal variables are categories that do not have meaningful numbers attached to them but are
broader categories, such as male and female, home schooled and public schooled, Caucasian and
African-American. Ordinal variables do have numbers attached to them, in that the numbers are in a
certain order, but there are not equal intervals between the numbers (e.g., such as when you rank a
group of 5 items from most to least preferred, where 3 might be highly preferred, and 2
hated). Interval/ratio variables have equal intervals between the numbers (e.g., weight, age).
For more information about these levels of measurement, check out one of the following websites.
Levels of Measurement(link is external)
Measurement Scales in Social Science Research(link is external)
What is the difference between ordinal, interval and ratio variables? Why
should I care?(link is external)
Validity and Reliability
When developing a scale/measure or survey, you need to be concerned about validity and
reliability. Readers of quantitative research expect to see researchers justify their research
measures using these two terms in the methods section of an article or paper.
Validity. Validity is the extent to which your scale/measure or survey adequately reflects the full
meaning of the concept you are measuring. Does it measure what you say it measures? For
example, if researchers wanted to develop a scale to measure "servant leadership," the researchers
would have to determine what dimensions of servant leadership they wanted to measure, and then
create items which would be valid or accurate measures of these dimensions. If they included items
related to a different type of leadership, those items would not be a valid measure of servant
leadership. When doing so, the researchers are trying to prove their measure has internal validity.
Researchers may also be interested in external validity, but that has to do with how generalizable
their study is to a larger population (a topic related to sampling, which we will consider in the next
section), and has less to do with the validity of the instrument itself.
There are several types of validity you may read about, including face validity, content validity,
criterion-related validity, and construct validity. To learn more about these types of validity, read the
information at the following link:
Validity(link is external).
To improve the validity of an instrument, researchers need to fully understand the concept they are
trying to measure. This means they know the academic literature surrounding that concept well and
write several survey questions on each dimension measured, to make sure the full idea of the
concept is being measured. For example, Page and Wong (n.d.) identified four dimensions of
servant leadership: character, people-orientation, task-orientation, and process-orientation ( A
Conceptual Framework for Measuring Servant-Leadership(link is external)). All of
these dimensions (and any others identified by other researchers) would need multiple survey items
developed if a researcher wanted to create a new scale on servant leadership.
Before you create a new survey, it can be useful to see if one already exists with established
validity and reliability. Such measures can be found by seeing what other respected studies have
used to measure a concept and then doing a library search to find the scale/measure itself
(sometimes found in the reference area of a library in books like those listed below).

Fischer, J., & Corcoran, K. (2007). Measures for clinical practice and research: A sourcebook (volumes 1
& 2). New York: Oxford University Press.
Maddy, T. (2008). Tests: A comprehensive reference for assessments in psychology, education, and
business. Austin, TX: Pro-Ed.
Rubin. R. B., Rubin, A. M., Graham, E., Perse, E. M., & Seibold, D. (2009). Communication research
measures II: A sourcebook. New York: Routledge.

Reliability. Reliability is the second criterion you will need to address if you choose to develop your
own scale or measure. Reliability is concerned with whether a measurement is consistent and
reproducible. If you have ever wondered why, when taking a survey, that a question is asked more
than once or very similar questions are asked multiple times, it is because the researchers one
concerned with proving their study has reliability. Are you, for example, answering all of the similar
questions similarly? If so, the measure/scale may have good reliability or consistency over time.
Researchers can use a variety of ways to show their measure/scale is reliable. See the following
websites for explanations of some of these ways, which include methods such as the test-retest
method, the split-half method, and inter-coder/rater reliability.
Types of Reliability(link is external)
Reliability(link is external)
To understand the relationship between validity and reliability, a nice visual provided below is
explained at the following website (Trochim, 2006, para. 2).
Reliability & Validity(link is external)

Self-Quiz/Discussion:
Take a look at one of the surveys found at the following poll reporting sites on a topic which
interests you. Critique one of these surveys, using what you have learned about creating surveys so
far.
http://www.pewinternet.org/(link is external)
http://pewresearch.org/(link is external)
http://www.gallup.com/Home.aspx(link is external)
http://www.kff.org/(link is external)

Sampling
One of the things you might have critiqued in the previous self-quiz/discussion may have had less
to do with the actual survey itself, but rather with how the researchers got their participants or
sample. How participants are recruited is just as important to doing a good study as how valid and
reliable a survey is.
Imagine that in the article you chose for the last "self-quiz/discussion" you read the following quote
from the Pew Research Center's Internet and American Life Project: "One in three teens sends
more than 100 text messages a day, or 3000 texts a month" (Lenhart, 2010, para.5). How would
you know whether you could trust this finding to be true? Would you compare it to what you know
about texting from your own and your friends' experiences? Would you want to know what types of
questions people were asked to determine this statistic, or whether the survey the statistic is based
on is valid and reliable? Would you want to know what type of people were surveyed for the study?
As a critical consumer of research, you should ask all of these types of questions, rather than just
accepting such a statement as undisputable fact. For example, if only people shopping at an Apple
Store were surveyed, the results might be skewed high.
In particular, related to the topic of this section, you should ask about the sampling method the
researchers did. Often, the researchers will provide information related to the sample, stating how
many participants were surveyed (in this case 800 teens, aged 12-17, who were a nationally
representative sample of the population) and how much the "margin of error" is (in this case +/-
3.8%). Why do they state such things? It is because they know the importance of a sample in
making the case for their findings being legitimate and credible. Margin of error is how much we are
confident that our findings represent the population at large. The larger the margin of error, the less
likely it is that the poll or survey is accurate. Margin of error assumes a 95% confidence level that
what we found from our study represents the population at large.
For more information on margin of error, see one of the following websites.
Answers.com Margin of Error(link is external)
Stats.org Margin of Error(link is external)
Americanresearchgroup.com Margin of Error(link is external)
[this last site is a margin of error calculator, which shows that margin of error is directly tied to the
size of your sample, in relationship to the size of the population, two concepts we will talk about in
the next few paragraphs]
In particular, this section focused on sampling will talk about the following topics: (a) the difference
between a population vs. a sample; (b) concepts of error and bias, or "it's all about significance"; (c)
probability vs. non-probability sampling; and (d) sample size issues.

Population vs. Sample

When doing quantitative studies, such as the study of cell phone usage among teens, you are
never able to survey the entire population of teenagers, so you survey a portion of the population. If
you study every member of a population, then you are conducting a census such as the United
States Government does every 10 years. When, however, this is not possible (because you do not
have the money the U.S. government has!), you attempt to get as good a sample as possible.
Characteristics of a population are summarized in numerical form, and technically these numbers
are called parameters. However, numbers which summarize the characteristics of a sample are
called statistics.
Error and Bias
If a sample is not done well, then you may not have confidence in how the study's results can be
generalized to the population from which the sample was taken. Your confidence level is often
stated as the margin of error of the survey. As noted earlier, a study's margin of error refers to the
degree to which a sample differs from the total population you are studying. In the Pew survey, they
had a margin of error of +/- 3.8%. So, for example, when the Pew survey said 33% of teens send
more than 100 texts a day, the margin of error means they were 95% sure that 29.2% - 36.8% of
teens send this many texts a day.
Margin of error is tied to sampling error, which is how much difference there is between your
sample's results and what would have been obtained if you had surveyed the whole population.
Sample error is linked to a very important concept for quantitative researchers, which is the notion
of significance. Here, significance does not refer to whether some finding is morally or practically
significant, it refers to whether a finding is statistically significant, meaning the findings are not due
to chance but actually represent something that is found in the population. Statistical significance is
about how much you, as the researcher, are willing to risk saying you found something important
and be wrong.
For the difference between statistical significance and practical significance, see the following
YouTube video: Statistical and Practical Significance(link is external).
Scientists set certain arbitrary standards based on the probability they could be wrong in reporting
their findings. These are called significance levels and are commonly reported in the literature as p
<.05 or p <.01 or some other probability (or p) level.
If an article says a statistical test reported that p < .05, it simply means that they are most likely
correct in what they are saying, but there is a 5% chance they could be wrong and not find the
same results in the population. If p < .01, then there would be only a 1% chance they were wrong
and would not find the same results in the population. The lower the probability level, the more
certain the results.

When researchers are wrong, or make that kind of decision error, it often implies that either (a) their
sample was biased and was not representative of the true population in some way, or (b) that
something they did in collecting the data biased the results. There are actually two kinds of
sampling error talked about in quantitative research: Type I and Type II error. Type 1 error is what
happens when you think you found something statistically significant and claim there is a significant
difference or relationship, when there really is not in the actual population. So there is something
about your sample that made you find something that is not in the actual population. (Type I error is
the same as the probability level, or .05, if using the traditional p-level accepted by most
researchers.) Type II error happens when you don't find a statistically significant difference or
relationship, yet there actually is one in the population at large, so once again, your sample is not
representative of the population.
For more information on these two types of error, check out the following websites.
Hypothesis Testing: Type I Error, Type II Error(link is external)
Type I and Type II Errors - Making Mistakes in the Justice System(link is
external)
Researchers want to select a sample that is representative of the population in order to reduce the
likelihood of having a sample that is biased. There are two types of bias particularly troublesome for
researchers, in terms of sampling error. The first type is selection bias, in which each person in the
population does not have an equal chance to be chosen for the sample, which happens frequently
in communication studies, because we often rely on convenience samples (whoever we can get to
complete our surveys). The second type of bias is response bias, in which those who volunteer for a
study have different characteristics than those who did not volunteer for the study, another common
challenge for communication researchers. Volunteers for a study may very well be different from
persons who choose not to volunteer for a study, so that you have a biased sample by relying just
on volunteers, which is not representative of the population from which you are trying to sample.
Probability vs. Non-Probability Sampling
One of the best ways to lower your sampling error and reduce the possibility of bias is to do
probability or random sampling. This means that every person in the population has an equal
chance of being selected to be in your sample. Another way of looking at this is to attempt to get
a representative sample, so that the characteristics of your sample closely approximate those of the
population. A sample needs to contain essentially the same variations that exist in the population, if
possible, especially on the variables or elements that are most important to you (e.g., age,
biological sex, race, level of education, socio-economic class).
There are many different ways to draw a probability/random sample from the population. Some of
the most common are a simple random sample, where you use a random numbers table or random
number generator to select your sample from the population.
There are several examples of random number generators available online. See the following
example of an online random number generator: http://www.randomizer.org/(link is
external).
A systematic random sample takes every n-th number from the population, depending on how many
people you would like to have in your sample. A stratified random sample does random sampling
within groups, and a multi-stage or cluster sample is used when there are multiple groups within a
large area and a large population, and the researcher does random sampling in stages.
If you are interested in understanding more about these types of probability/random samples, take a
look at the following website:
Probability Sampling(link is external).
However, many times communication researchers use whoever they can find to participate in their
study, such as college students in their classes since these people are easily accessible. Many of
the studies in interpersonal communication and relationship development, for example, used this
type of sample. This is called a convenience sample. In doing so, they are using a non- probability
or non-random sample. In these types of samples, each member of the population does not have
an equal opportunity to be selected. For example, if you decide to ask your facebook friends to
participate in an online survey you created about how college students in the U.S. use cell phones
to text, you are using a non-random type of sample. You are unable to randomly sample the whole
population in the U.S. of college students who text, so you attempt to find participants more
conveniently. Some common non-random or non-probability samples are:
 accidental/convenience samples, such as the facebook example illustrates
 quota samples, in which you do convenience samples within subgroups of the population,
such as biological sex, looking for a certain number of participants in each group being
compared
 snowball or network sampling, where you ask current participants to send your survey onto
their friends.
For more information on non-probability sampling, see the following website:
Nonprobability Sampling(link is external).
Researchers, such as communication scholars, often use these types of samples because of the
nature of their research. Most research designs used in communication are not true experiments,
such as would be required in the medical field where they are trying to prove some cause-effect
relationship to cure or alleviate symptoms of a disease. Most communication scholars recognize
that human behavior in communication situations is much less predictable, so they do not adhere to
the strictest possible worldview related to quantitative methods and are less concerned with having
to use probability sampling.
They do recognize, however, that with either probability or non-probability sampling, there is still the
possibility of bias and error, although much less with probability sampling. That is why all
quantitative researchers, regardless of field, will report statistical significance levels if they are
interested in generalizing from their sample to the population at large, to let the readers of their
work know how confident they are in their results.

Size of Sample
The larger the sample, the more likely the sample is going to be representative of the population. If
there is a lot of variability in the population (e.g., lots of different ethnic groups in the population), a
researcher will need a larger sample. If you are interested in detecting small possible differences
(e.g., in a close political race), you need a larger sample. However, the bigger your population, the
less you have to increase the size of your sample in order to have an adequate sample, as is
illustrated by an example sample size calculator such as can be found
at http://www.raosoft.com/samplesize.html(link is external).
Using the example sample size calculator, see how you might determine how large of a sample you
might need in order to study how college students in the U.S. use texting on their cell phones. You
would have to first determine approximately how many college students are in the U.S. According to
ANEKI, there are a little over 14,000,000 college students in the U.S. ( Countries with the
Most University Students(link is external)). When inputting that figure into the sample size
calculator below (using no commas for the population size), you would need a sample size of
approximately 385 students. If the population size was 20,000, you would need a sample of 377
students. If the population was only 2,000, you would need a sample of 323. For a population of
500, you would need a sample of 218.
It is not enough, however, to just have an adequate or large sample. If there is bias in the sampling,
you can have a very bad large sample, one that also does not represent the population at large. So,
having an unbiased sample is even more important than having a large sample.
So, what do you do, if you cannot reasonably conduct a probability or random sample? You run
statistics which report significance levels, and you report the limitations of your sample in the
discussion section of your paper/article.

Pilot Testing Methods

Now that we have talked about the different elements of your study design, you should try out your
methods by doing a pilot test of some kind. This means that you try out your procedures with
someone to try to catch any mistakes in your design before you start collecting data from actual
participants in your study. This will save you time and money in the long run, along with unneeded
angst over mistakes you made in your design during data collection. There are several ways you
might do this.
You might ask an expert who knows about this topic (such as a faculty member) to try out your
experiment or survey and provide feedback on what they think of your design. You might ask some
participants who are like your potential sample to take your survey or be a part of your pilot test;
then you could ask them which parts were confusing or needed revising. You might have potential
participants explain to you what they think your questions mean, to see if they are interpreting them
like you intended, or if you need to make some questions clearer.
The main thing is that you do not just assume your methods will work or are the best type of
methods to use until you try them out with someone. As you write up your study, in your methods
section of your paper, you can then talk about what you did to change your study based on the pilot
study you did.