721952

research-article2017
TSOXXX10.1177/0092055X17721952Teaching SociologyWhitley and Dietz

Teaching Note

Teaching Sociology

Turking Statistics: Student-

2018, Vol. 46(1) 44­–53
© American Sociological Association 2017
DOI: 10.1177/0092055X17721952
https://doi.org/10.1177/0092055X17721952

generated Surveys Increase ts.sagepub.com

Student Engagement and

Performance

Cameron T. Whitley1 and Thomas Dietz1

Abstract
Thirty years ago, Hubert M. Blalock Jr. published an article in Teaching Sociology about the importance of
teaching statistics. We honor Blalock’s legacy by assessing how using Amazon Mechanical Turk (MTurk)
in statistics classes can enhance student learning and increase statistical literacy among social science
gradaute students. In addition, we assess whether using MTurk has an impact on student ability to make
professional progress. We find that, compared to traditional teaching methods, using MTurk increased
student performance, perceptions, and outcomes. In addition, using MTurk resulted in a measurable
increase in statistical literacy. We recommend that instructors teaching statistics consider how MTurk or
similar technologies can be used in their classrooms.

Keywords
teaching statistics, Mechanical Turk, learning statistics, statistical literacy

Thirty years ago, Hubert M. Blalock Jr. (1987)
brought our attention to the art of teaching statistics
in an article published in Teaching Sociology. He
articulated five goals that all statistics courses
should aspire to: (1) overcoming fears, resistances,
and tendencies to overmemorize; (2) the impor-
tance of intellectual honesty and integrity; (3)
understanding the relationship between deductive
and inductive inferences; (4) learning to play the
role of reasonable critic; and (5) learning to handle
complexities in a systematic fashion. Three decades
later, these five goals continue to highlight and
extend what we traditionally think of as statistical
literacy, which “is the ability to understand and
critically evaluate statistical results that permeate
our daily lives-coupled with the ability to appreci-
ate the contributions that statistical thinking can
make in public and private, professional and per-
sonal decisions” (Wallman 1993: 1).
Toward these ends, most graduate programs in
sociology require at least one statistics course. Not
only should these classes teach graduate students
how to use statistics in their research, but they
should be foundational courses in critical thinking
and in understanding and assessing data. For grad-
uate students who will go on to careers outside the
professorate, statistics courses can help develop
tangible and transferable skills (Williams, Payne,
and Sloan 2015).
Background
Nearly all graduate statistics courses make some
use of exercises employing statistical software.
While we know of no data on the prevalence of
term papers in graduate statistics courses, anec-
dotal evidence suggests that many instructors feel
it is useful to require a term paper based on data
1
Michigan State University, East Lansing, MI, USA
Corresponding Author:
Thomas Dietz, Department of Sociology, Michigan State
University, 316 Berkey Hall, East Lansing, MI 48828, USA.
Email: tdietz@msu.edu
Whitley and Dietz
analysis in addition to or instead of exercises and
exams. The term paper has several advantages.
Writing a research paper based on statistical analy-
sis clarifies the relationship between theory and
method. It gives students “tacit knowledge” of how
to operationalize theoretical concepts, clean and
manage data, run diagnostic tests, and make opera-
tional decisions about missing data, recoding, and
many other practical matters. In writing up results,
students can gain a deeper understanding of what
the data do and do not reveal about the theoretical
arguments being tested. And if the paper is of suf-
ficient quality, it may lead to presentation at a
meeting or publication or otherwise advance a stu-
dent professionally.
However, if students are going to write research
papers in the statistics course, they must have
access to data. There are three typical ways to
deliver this access. One is to provide all students
with access to a single secondary general purpose
data set, such as the General Social Survey (GSS).
Another is to ask each student to find her or his
own data from archives or other sources. Sometimes
such data can be provided by the research advisor.
A third approach is to collect data for the class that
allow each student to utilize a set of questions spe-
cific to his or her interests. In this article, we report
the results of an experiment comparing the first
approach, a secondary data set, with the third, a
data set developed for the class. We argue, as has
been shown in other teaching applications, that giv-
ing students a less passive experience may evoke
greater learning (see e.g., Whitley 2013). In our
discussion, we return to a general comparison of all
three approaches. Note that we are focusing on sur-
vey data sets using individual respondents as the
unit of analysis. Of course, one might use data sets
on aggregate units, such as counties, states, cities,
or nations, but since this seems to be much less
common, we will not discuss it here (but see Dietz
and Kalof 2009 for a discussion and examples of
macrolevel data).
The use of secondary data has several advan-
tages. These data sets are usually readily available
for free, they reduce the time needed to do data col-
lection and cleaning, and they are seemingly easier
for students to manage. It is possible to select a sec-
ondary data set, such as the GSS, that is widely
accepted as of high quality, making it easier for
students to publish the results from their class proj-
ects when that is appropriate. But for many stu-
dents, existing data sets have not incorporated
measures that allow work on topics that are central
45
to their interests, so students may be less motivated
to develop statistical and data analysis expertise.
Using Amazon’s
Mechanical Turk (MTurk)
for Data Collection
An alternative to giving students secondary data
for analysis is to allow students to collect their own
data. However, collecting enough data to be useful
for students in a graduate statistics course can be
expensive and time consuming. Amazon’s MTurk
provides a quick and reasonably inexpensive means
of collecting data. MTurk is a crowdsourcing Internet
application that allows individuals to use human
intelligence to perform specified tasks, including
completing online surveys. Using MTurk to collect
survey data has become very popular in the social
sciences, always with the caveat that such data are not
a representative sample. Once a survey is designed
and uploaded to the platform, the researcher can get
several hundred responses within hours. In our expe-
rience, as long as standard protocols are in place to
preserve anonymity and make clear that participa-
tion is voluntary, and are described in the introduc-
tion to the survey, human subjects approval has
been straightforward for MTurk surveys because
subjects are volunteers who are compensated for
their time.
MTurk samples are more socioeconomically
and ethnically diverse than social media postings,
like Twitter, Facebook, and Reddit (Casler, Bickel,
and Hackett 2013), and more representative of the
U.S. population than college student respondents
(Buhrmester, Kwang, and Gosling 2011; Ross et al.
2010; Stewart et al. 2015; Weinberg, Freese, and
McElhattan 2014). MTurk data sets are relatively
representative of the population of U.S. Internet
users (Ipeirotis 2010; Ross et al. 2010) but tend to
be more female, be younger (average age of 36
with a typical range 18 to 80+), have more educa-
tion, and have lower income than the general U.S.
population (Levay, Freese, and Druckman 2016;
Paolacci and Chandler 2014; Paolacci, Chandler,
and Ipeirotis 2010). Overall, scholars in a variety of
fields consider MTurk to be a useful data collection
tool that can produce reasonable and consistent
results (Buhrmester et al. 2011; Goodman, Cryder,
and Cheema 2013). To date, MTurk has been used
to produce a wide variety of research studies in
sociology and kindred disciplines, but it has not
been discussed as a tool to teach statistics.
46
Methods
Introductory graduate-level statistics courses were
taught in the fall of 2012 (“GSS class” hereafter)
and fall of 2013 (“MTurk class” hereafter).
Although both classes were taught by a sociology
department, enrollment was open to nonsociolo-
gists. Both classes were taught using the same syl-
labus, same book (Dietz and Kalof 2009), and same
lecture materials; had the same four homework
assignments; and required a term paper and final
presentation. Both classes met once a week in the
same room at the same time for 220 minutes per
week, divided between lab and lecture/recitation.
The only major differences between the courses
was in the data used for the term paper. Of course,
students were not assigned at random to the two
course offerings, so it is possible that some selec-
tion bias could influence our results.
In the GSS class, the instructors provided data
from the year 2010 GSS (NORC 2016). Students
were given a lecture about survey data collection
but did not specifically engage in the process of
designing, collecting, or cleaning raw data. The
advantage to this method was that data were readily
available and comprised a representative sample of
the United States. However, the disadvantage was
that students had limited survey items to work
from, so they were not always able to address
research questions that matched their interests or
contributed to developing expertise in their fields
of specialization. For instance, one student was
interested in public support for biodiversity conser-
vation efforts. The closest she could come to this
topic was the 2010 GSS item “Hunting is more
likely than climate change to make polar bears
become extinct. Is that true or false?” Another stu-
dent interested in food deserts wanted to assess
who was more likely to overcome restricted food
access. The only question mentioning food in the
2010 GSS is a general question about global warm-
ing that mentions food production as being
impacted. Overall, a great deal of effort was
required by the students, professor, and teaching
assistant (TA) in finding 2010 GSS question items
that even loosely aligned with student research
interests.
In the MTurk class, each student designed sur-
vey questions that matched his or her interests.
Data were collected via MTurk. Before beginning
data collection, all students had to complete institu-
tional review board training, which gives instruc-
tors an opportunity to discuss and promote
intellectual honesty and integrity, thus meeting
Teaching Sociology 46(1)
Blalock’s (1987) second goal. The custom-
designed surveys were able to accommodate
diverse research interests. Among the questions
addressed by students were as follows: What per-
sonal and social characteristics predict support for
anti–sexual assault campaigns? What aspects of
employment influence job satisfaction? What fac-
tors drive public support for protecting natural
space? How do values motivate the choice of
locally grown foods, and what factors mitigate (or
overshadow) these values? What motivates people
to complete MTurk tasks? What drives views about
how animal–vehicle collisions and roadkill should
be managed?
Students were taught how to identify a research
question, construct survey questions, and collect
data. Specifically, students were first asked to
select a research question they were interested in
that related to their research trajectory. Next, stu-
dents were asked to find in the literature or develop
5 to 10 survey items, based on theory, that would
help them answer their research question. This set
of tasks engaged the class and instructors in what
was essentially a workshop in survey instrument
design and involved discussions of the detailed
mechanics of survey instruments, including options
for response scales, issues of question ordering
within the instrument, appropriate handling of
missing data categories, redirects and skips, and
other design topics. The instrument was designed
in Survey Monkey so students could easily imple-
ment different approaches to designing items and
blocks of items, discuss them, and change them.
We also discussed the strengths and weaknesses of
MTurk samples, including the need for attention
checks, screening data for respondents who give
the same answer to every question in a block, and
related issues. These tasks addressed Blalock’s
(1987) third and fourth goals. We then divided the
class into three groups to run three surveys to avoid
an overly long instrument. Each survey consisted
of three to five questions from each student,
amounting to 30 to 50 questions plus standard
demographic and social structural items. Many of
the questions were presented in Likert-type scale
question blocks, so the overall number of questions
was much smaller. The surveys were launched at
the end of the first month of class. A moderate sam-
ple size (~300) was obtained within 24 hours for
each of the three samples. Since the surveys took
less than 10 minutes to complete, we offered 25
cents compensation to each respondent, for a total
cost of $225. This was the standard practice at the
time. Now, it is recommended that respondents be
Whitley and Dietz
paid the equivalent of minimum wage. In hind-
sight, we would now offer $1.20 for survey com-
pletion. Once the data were available, the class
discussed how to organize and clean their data sets,
which connected to Blalock’s (1987) fifth goal.
The advantage of the MTurk process was that the
students learned firsthand about survey design, they
developed questions that interested them, and they
had a hands-on experience. The disadvantage was
that students struggled with the process, the sample
was not representative of the United States, and it cost
a few hundred dollars to collect. Students’ final proj-
ect in both classes was to use data (either GSS or
MTurk) and the methods they learned in the course,
primarily multiple regression, to answer an original
research question. In both classes, students had to pro-
duce a journal-length research paper and present the
results to the class.
We emphasize that in comparing between
classes, because students were not randomly assigned
to the year they took the course, differences
between the classes might be due to selection bias
rather than differences between using MTurk and
the GSS. It also possible that subtle biases on the
part of the instructors influenced student experi-
ences. For the measures where we have pre- and
postclass data, we examine differences between the
two classes in the change in those measures. This is
equivalent to using fixed-effects panel regression
and controls for characteristics of the individual
students that were constant over time but not for
those that change over time. Thus, any causal inter-
pretation of our results must be seen as suggestive
rather than definitive. We offer p values based on
ordinary least squares regression two-tailed t tests
using class as a dummy variable to predict scores
or differences in scores. Identical substantive
results and nearly identical p values are obtained
with t tests using the assumption of unequal vari-
ances and the Welch correction for degrees of free-
dom. Again, given the small sample size and the
lack of random assignment, p values should be
seen as approximate.
Data
We assess the impacts of using MTurk using both
pre- and postclass measures of students’ knowl-
edge and familiarity with statistical concepts,
course evaluations, and qualitative follow-up inter-
views. Our pre- and postclass measures are based
on students completing three tasks on the first and
on the final day of the class. All completed tasks
were anonymous and no points toward grades were
47
associated with their completion. First, students
were given a diagnostic exam that consisted of 30
questions, which ranged from defining basic con-
cepts to interpreting a regression table. (Content of
each of these batteries of items is available from
the authors.) Second, students were presented with
31 single-word items related to statistics and asked
to rank how familiar they were with each item on a
scale of 1 to 5 ranging from not familiar at all to
very familiar. Items included statistical concepts,
such as “confidence interval,” “F test,” and “sam-
pling distribution.” Finally, in the end-of-class
assessment, students were asked to evaluate and
rank their professor, TA (10 Likert-like items each),
and the class (15 Likert-like items). In addition, we
conducted follow-up qualitative interviews with
students in 2015 and 2016, three years after they
completed either the GSS or the MTurk course.
These interviews were used to determine whether
the statistics course assisted them in getting through
any of their graduate school milestones (compre-
hensive exams, dissertation proposal, and disserta-
tion) and whether they presented or published their
final paper projects.
A total of 10 students took the GSS class, 4 (40
percent) were sociology students, and 6 (60 per-
cent) were women. There were 13 students in the
MTurk class, 8 (61 percent) were sociology stu-
dents, and 9 (70 percent) were women. The same
professor and TA taught both courses. The profes-
sor has taught graduate statistics courses in sociol-
ogy with a similar, albeit evolving, syllabus for
over 30 years.
Results
Student Perceptions
Increasing student familiarity with statistical con-
cepts is intended to reduce fear of quantitative
methods, Blalock’s (1987) first goal. Thus, stu-
dents’ perceptions of their understanding of statisti-
cal concepts is important not just per se but also in
assessing general comfort with statistics more
broadly. On average, students in both classes
reported an increase in familiarity with concepts
between the first and final days of the course. At
the beginning of the course, students in the MTurk
and GSS class scored 2.2 out of 5 and 2.1 out of 5,
respectively, for concept familiarity (see Table 1).
These scores indicate that students had “heard of
the concepts, but [were] not familiar with them.”
The difference between the two classes on day 1
was not significant (see Table 2).
48 Teaching Sociology 46(1)

Table 1. Comparison of Familiarity, Diagnostics, and Overall Rating between the Two Courses.

Class

Variable GSS MTurk Scale

n 10 13
Precourse familiarity with concepts 2.3 2.1 Out of 5
Postcourse familiarity with concepts 3.9 4.6 Out of 5
Precourse diagnostic exam score 19.7 19.6 Out of 100
Postcourse diagnostic exam score 78.4 88.4 Out of 100
Rating of course by students 4.6 4.9 Out of 5

Note: GSS = General Social Survey; MTurk = Mechanical Turk.

Table 2. Comparison of GSS and MTurk Classes.

Comparison Difference Test Statistic p

Difference in familiarity with statistical Difference in means = −0.116 t = −0.54 p = .597
concepts at start of class
Difference in change over the course Difference in differences = 0.737 t = 3.75** p = .01
in familiarity with statistical concepts
Difference in diagnostic test at start Difference in means = −0.026 t = −0.01 p = .996
of class (mean percentage correct)
Change over the course in diagnostic Difference in means = 58.7 t = 12.23** p < .001
test score for GSS class
Change over the course in diagnostic Difference in means = 68.8 t = 27.85** p < .001
test score for MTurk class
Difference in changes in diagnostic test Difference in differences = 10.1 t = 2.01+ p = .057
between GSS and MTurk class

Note: GSS = General Social Survey; MTurk = Mechanical Turk.

+p < .10. **p < .01.

Analysis of the change from the start of class to
the end of class in each course reveals that familiarity
increased more in the class using MTurk (see Table
2). On the last day of the course, students in the GSS
class reported an average score of 3.9 out of 5 for
familiarity and comfort with statistical concepts, indi-
cating they had “a basic command of the concept.” In
contrast, students in the MTurk class reported an aver-
age of 4.6 out of 5, indicating they were somewhere
between having “a basic command of the concept”
and having “a strong command of the concept.”
In addition to quantitative data, we also obtained
open-ended statements from the students about
their familiarity and comfort with statistical con-
cepts in the course. When asked if they felt more
comfortable with statistics after taking the course,
students in the GSS course generally responded
that they enjoyed the course but still felt uneasy
with statistics and unsure about using a software
package. For instance, one GSS course student
responded, “I feel I gained a good amount of
knowledge on how to analyze social data, but I still
have trouble understanding some parts,” and
another GSS course student noted, “I really wanted
to learn statistics, but I feel like I just learned how
to write code for STATA.” In contrast, students in
the MTurk course seemed to feel more confident in
their skills and their ability to apply what they had
learned to their research. For instance, one MTurk
course student wrote, “I think I have learned the
necessary skills to be successful.” Another MTurk
course student asserted, “I am comfortable with
everything we learned in the course,” and another
stated, “I now know what I need to.” Student per-
ceptions are important, but additionally we needed
to know if perceptions translated into actual perfor-
mance differences between the two classes.
Student Performance
Students’ self-reported familiarity with statistical
concepts may not match actual understanding.
Whitley and Dietz
Students were given a diagnostic quiz on the first
and the final day of the course. This diagnostic tool
allowed us to assess if students could distinguish
between inductive and deductive reasoning, be a
critical consumer of statistics, and work with statis-
tical complexity—the third, fourth, and fifth goals
outlined by Blalock (1987). (The diagnostic quiz
and familiarity were correlated 0.27 on the first day
of class and 0.56 on the last day.) There were no
initial differences in the diagnostic test between the
two classes (see Table 2). We were happy to see
that students in both classes demonstrated a signifi-
cant increase in learning from the first to the final
day of the course. Students in the GSS class
increased their average diagnostic score by 58.7
percentage points, from 19.7 percent to 78.3 per-
cent (see Table 2). Students in the MTurk class
increased their average score by 68.8 percentage
points, from 19.6 percent to 88.5 percent (see Table
2). These findings indicate that students in the
MTurk class increased their knowledge by an aver-
age of 10 percentage points more than did students
in the first class. The difference in learning (the dif-
ference in differences) approaches but does not
quite achieve statistical significance at the .05 level
but is significant at the .10 level (see Table 2). This
is consistent with the argument that students
learned more in the course using MTurk.
Student Achievement after Class
We wanted to determine if taking either of these
courses had an impact on the completion of gradu-
ate school milestones. We conducted interviews
with students in both classes three years after com-
pleting the course. We could connect follow-up
interviews with 8 of the 10 students from the GSS
class and with 10 of the 13 students in the MTurk
class. In both cases, the missing students left their
graduate programs without graduating and could
not be reached.
We focused our interviews on two open-ended
questions. First, we asked if participating in the sta-
tistics course helped them achieve graduate school
milestones and, if so, how. Second, we asked stu-
dents what came of their final course papers.
Specifically, we asked if they were they able to
present, publish, or defend papers that were ini-
tially started in their statistics course. All students
interviewed from both classes felt that the statistics
course contributed to their achieving graduate
school milestones. In addition to the usual things
mentioned, like comprehensive exams, proposals,
and dissertations, two students from the MTurk
49
course also mentioned that taking the course
changed their minds about statistics and they have
both gone on to take more advanced courses.
Nearly every student from the MTurk class men-
tioned that the process helped them in forming and
thinking about their proposal or dissertation, even
if the topic for the statistics paper was not their dis-
sertation topic. For instance, one MTurk class stu-
dent said, “I didn’t publish it . . . but this process
helped me create a plan . . . like I knew what to
expect and do and that was helpful.” In contrast, a
student from the GSS course said, “I think I learned
about statistics, but I won’t really know what I
learned until I apply it to my own research. . . . I
mean, I’m sure I learned stuff, but it is hard to pin-
point what I learned.”
At the time we submitted this article for publi-
cation, one student from the GSS course published
his paper, a second student reported that she pre-
sented the paper at a conference, and a third student
had a revise-and-resubmit (R&R) but elected to not
resubmit the article to focus on other things. The
student with the R&R informed us,
I didn’t end up publishing that paper or
using it towards my dissertation. I submitted
it and it got an R&R but I decided to move
on to other projects and not pursue
publication. . . . I picked the topic because of
limited options with the data. It was
something to do, but it doesn’t reflect what I
am interested in.
So, the student who got an R&R decided not to
resubmit because she was not interested in working
on the project and did not see it contributing to her
research trajectory. This story is perhaps the most
representative of the issues inherent in secondary
data: students cannot always pursue their specific
research interests, and this lack of interest can
mean that they abandon projects prematurely.
Overall, the students interviewed from the GSS
class said that learning the course concepts contrib-
uted to the completion of their milestones, but only
one student built on his paper in his dissertation
work and got a publication based on work in the
class.
In interviewing students from the MTurk class,
we found that two students had published their
final papers, three students had done conference
presentations, and an additional four students had
submitted their papers to journals for review. One
student had won a graduate student paper award
after submitting his final paper to a competition.
50
An MTurk class student who has not published her
paper, but had submitted it for publication after
additional work, noted, “I haven’t published it, but
it was really useful to my dissertation. . . . I mean I
used the same methods . . . and the learning process
was good. I still hope it gets published, but it was
helpful and I learned how to conduct a real survey.”
However, not everyone was excited about the idea
of using MTurk data. As we mentioned before, a
leading concern is that data generated from MTurk,
although more diverse than a sample of college stu-
dents, are not representative of the U.S. population.
For example, the advisor of one MTurk class stu-
dent was skeptical about her being able to publish
data collected from MTurk, so he recommended
against trying it. (The advisor was not a sociolo-
gist.) Regarding trying to publish her final paper,
she noted, “I didn’t try to publish my paper. I had
considered it, but my advisor was very skeptical of
the data collection procedure and didn’t think it
would pass the peer review process and told me not
to try.”
Another student suggested that although her
final paper has not been published, the process was
helpful in thinking about and piloting her disserta-
tion research:
It allowed me to pilot survey questions that
were later used in telephone interviews and
mail-back surveys. It was nice to see what
the data from the questions actually looked
like, how I would be able to analyze it, and
make the appropriate changes needed for my
actual data collection. It was really helpful
because I knew the data set, was invested in
it, and it directly related to what I am doing
for my research. It was also a manageable
size for learning statistics, something my
actual data sets are not.
Discussion
Since Blalock (1987) published his article on
teaching statistics 30 years ago, the use of statistics
in the social sciences has grown substantially.
Many graduate statistics courses place substantial
emphasis on completing a paper based on data
analysis. The advantages to requiring a paper in
addition to exercises are that (1) it gives students an
opportunity to understand the many subtle and
complex decisions going from a research question
and raw data to drawing conclusions from an anal-
ysis; (2) it gives students practical experience in
conducting an analysis, including skills in data
Teaching Sociology 46(1)
management and analysis (Long 2009); (3) it trains
students to do exactly what is required in quantita-
tive research; and (4) it gives the students an oppor-
tunity to conduct research that can contribute to
their publication record and perhaps to their
dissertation.
However, a paper assignment faces the instruc-
tor with a decision to make about what data will be
used in those papers. There are three basic choices:
(1) all students can be required to use a general pur-
pose data set, such as the GSS; (2) each student can
be required to find data for the paper; and (3) an
MTurk survey or other method can be used to gen-
erate data for questions proposed by the students to
match their research interests. In our analyses, we
compare classes that used approaches 1 and 3, but
in this discussion it will be useful to consider all
three options.
Table 3 provides a rough summary of the advan-
tages and disadvantages of each method. Quality of
sample refers to the degree to which the data are
from a representative sample and were collected
with methodological care. Here a preexisting data
source, such as the GSS, performs best. MTurk
data are a convenience sample and thus viewed
with skepticism by some subfields. Student-
provided data vary in quality. Sometimes such data
come from an ongoing project of the student’s
advisor or is a high-quality data set from a data
archive, but sometimes students find data that,
while matching their interests, are of limited
quality.
By hidden problems, we mean discovering
unexpectedly large amounts of missing data, find-
ing that questions of interest were asked of differ-
ent subsamples and so cannot be used in the same
analysis, learning that the data are not as docu-
mented, and other issues that arise only after the
student has invested considerable time and effort in
moving toward analyzing the data. While one
might expect this is not a problem with high-qual-
ity secondary data sets, that is not our experience.
For example, the GSS has a complex set of sub-
samples and the documentation is difficult to inter-
pret, so we have seen students discover serious
problems for their analysis only after investing
considerable effort. Since the class is designing the
MTurk study, there are no such problems. The
degree to which this issue arises with student-
provided data varies.
The effort to create a data set ready for analysis
also varies across approaches. For preexisting data,
instructors can do much of this work in advance,
ensuring a usable working sample, creating recodes
Whitley and Dietz
Table 3. Issues in Using Secondary, MTurk, and Student-provided Data.
Issue Secondary Data Set
Quality of sample Very high,
representative sample
Chance of hidden Moderate to low
problems
Effort required to create Moderate to low
working data set
Match to student Moderate to low
interest
Chances of contribution Moderate to low
to dissertation or
other future research
Monetary costs None
Note: MTurk = Mechanical Turk.
of commonly used variables, and so on. This leaves
only recoding and related tasks for the data specific
to each student’s project. But as noted, in the case
of some secondary data sets, there can be subtle
problems that are hard to detect until analysis
begins. Since the MTurk data sets are generated for
the course, there should be no problems with docu-
mentation, and the process of creating a data set for
analysis is straightforward. One of the major obsta-
cles to using data provided by students is that each
student will be doing data preparation with a differ-
ent data set, and in some cases, the amount of work
required is very substantial, effectively making the
professor and TA data analysis consultants to each
member of the class.
Our experience is that secondary data sets do not
provide a good match to many student’s interests.
MTurk data will be a good match since each student
can develop his or her own questions to match his or
her interests. Whether the data analysis will advance
the students’ careers beyond honing their statistical
skills is contingent on the other issues. If students
can find questions that match their research inter-
ests in a high-quality secondary data set, then they
often can produce results of publishable quality. But
often the match between student interest and what is
available is poor. MTurk data will be a good match
to student interests since the students develop the
questions. In some subareas of the social sciences,
convenience samples are acceptable for publication
of exploratory work; in others, they are not. Even
when they are not, the MTurk data give a student an
opportunity for pilot testing, which can then be used
in a proposal or for a research grant. For data the
student provides, we have found that usually stu-
dents can find data of reasonable quality that are a
51
Data Provided by
MTurk Data Students
Diverse convenience Varies but often high
sample quality
Low Moderate to high
Low Moderate to high
High Moderate to high
High Moderate to high
Moderate None
good match to their interests. However, in some
cases, those data turn out not to be as documented or
they have serious flaws not obvious until analysis is
under way. One challenge for this approach arises
because most departments prefer students to take
the first statistics course at or near the start of the
graduate program. First or second semester students
are less likely to have learned about data sets suit-
able for this approach.
For universities and colleges that are members
of data archives consortia, such as the Inter-
University Consortium for Political and Social
Research, access to many secondary data sources
for use in class is free. Data provided by the indi-
vidual student are also typically free for the pur-
poses of the course. Collecting data via MTurk
does require a source of funds. The cost structure of
MTurk has changed somewhat since we used it in
class. The prevailing norm is that MTurk workers
should be paid for their time at something like the
minimum wage rate. So, for a survey taking eight
minutes to complete at a $10 wage rate, the cost
with Amazon current fees for an MTurk task is
about $2 per completed response, or about $600 for
300 respondents. We appreciate that for courses
without a monetary budget, this could be a substan-
tial obstacle. However, the online environment for
survey research is changing rapidly, and there may
be lower-cost options. We also note that some ser-
vices, like Qualtrics, offer quota samples that
approximate a nationally representative sample for
slightly more than $5 per completed response for a
10-minute instrument. This might allow an approxi-
mately representative sample albeit for somewhat
higher cost. We also note that in many disciplines,
courses involving “hands-on” work in collecting and
52
analyzing data have a lab budget. Sociology depart-
ments might consider moving in this direction.
It should be obvious from Table 3 and our discus-
sion of it that there is no single “best” approach. Of
the three methods we have tried, we have been least
satisfied with the use of secondary data sets and most
satisfied with MTurk. Our analysis of our experience
indicates why we prefer MTurk to use of a general-
purpose secondary data set. The disadvantage of
MTurk is that there is an extra financial cost and that,
in a one-semester course, students and instructors
must move quickly to design and implement the sur-
vey early in the semester. Further, MTurk is a source
of publishable data in some fields but not others. We
have not done a comparable experiment on the
approach of each student finding his or her own data,
although one author has used that approach several
times. But we suspect this approach produces more
variation within the class than the other two
approaches. Some students use a data set provided by
their advisor and on which they are already working.
But others, and often those with the most anxiety
about the course, are asked to find a data set at a point
in the academic career when they have very little
familiarity with quantitative data analysis. This can
produce a situation where the students who are
already most experienced at quantitative methods
gain most, and those who are least experienced gain
least, a parallel to the “Matthew effect” (Merton
1968).
Traditionally, exercises in a graduate-level statis-
tics course use secondary data on which students per-
form analyses. This allows use of a high-quality data
set that instructors can prepare so that students can
focus on the goals of the exercise. However, when
students are assigned a term paper, there are draw-
backs to secondary data. The greatest of these, in our
view, is that the data may not relate to the students’
area of interest or research trajectory. As a result, their
interest level in analyzing the data may be minimal.
For students who already lack confidence or who do
not see statistics as salient to their interests, this may
be an impediment to achieving Blalock’s (1987) goals
for the statistics course. We note that data from MTurk
surveys or other sources tailored to student interests
could also be used for exercises.
Considering why MTurk led to better outcomes,
we submit that the approach adhered to the key
goals that Blalock (1987) outlined. By collecting,
cleaning, and analyzing data made possible by
using Amazon’s MTurk technology, students were
able to overcome fears, discuss intellectual honesty
and integrity, explore the relationship between
deductive and inductive reasoning, be a reasonable
critic, and learn to work with complexity and
Teaching Sociology 46(1)
uncertainty. While these goals can be achieved
with other forms of data, the engagement students
feel with data that reflect their research interests is
a major help in moving students forward.
Our study has a number of limitations. First, since
we did not have random assignment to our two
classes, it is possible that some selection bias in who
enrolled in which class, that instructors being espe-
cially enthusiastic about the use of MTurk, or that
some other difference between the two classes other
than the use of the GSS versus MTurk generated the
differences we observed. Using individual differences
lends credibility to that claim that our results were not
due to selection bias. We also examined the differ-
ences in student evaluations of the professor and TA
and found no significant differences (for professor, p
= .613; for TA, p = .733), which also supports the
argument that the differences we observed were the
result of using the GSS versus MTurk. But that claim
must be viewed as tentative. Second, we are limited
by the size of our classes. Although we did find a sta-
tistical difference in performance and perceptions
between the two courses, we had only 10 students in
the GSS course and 13 students in the MTurk course,
so our ability to detect significant differences is
limited.
Instructors interested in using MTurk data in
statistics classes will of course have to develop
some familiarity with the system. We find the easi-
est way to implement the survey instrument is via a
survey web tool, such as Survey Monkey or
Qualtrics. These are relatively easy to use and
allow students to preview alternative ways of struc-
turing responses and organizing a survey instru-
ment. Using MTurk requires thinking through
some issues that are much the same as in any sur-
vey, such as making sure questions are clear and
meaningful. Other issues are specific to any survey
mechanism where respondents are paid to partici-
pate, including respondents who “straightline” by
giving the same response to every question and
respondents who do not read questions. Attention-
check questions and checks on time to complete
can help filter such poor-quality respondents.
Finally, we note that we do not believe the benefits
we obtained in our MTurk class were specific to the
use of MTurk. Rather, we believe the results came
from providing a mechanism by which students were
able to work on a research project that matched their
own professional interests. The assignment of a term
paper requiring original data analysis is of course a
form of inquiry-based learning, an approach that oth-
ers have noted is beneficial in teaching social science
logic and methods (McCright 2012). In the graduate
statistics course, many students have well-formed
Whitley and Dietz
research interests and others have ideas they wish to
explore. The enthusiasm that can come from pursuing
these interests can help counterbalance statistical anx-
iety and demonstrate to students how statistical meth-
odology and, in particular, the logic of statistical
reasoning can support their research interests. So, any
mechanism that would allow students to pursue their
interests could provide those benefits; at this stage of
the evolution of research methods, surveys via MTurk
provide an effective vehicle for this approach.
Editor’s Note
Reviewers for this manuscript were, in alphabetical order,
Susan Caufield and Pamela Paxton.
Authors’ Note
We thank the students in our graduate statistics course for
their engagement in the class and afterward as well as the
Teaching Sociology reviewers for helpful comments.
References
Blalock, Hubert M., Jr. 1987. “Some General Goals in
Teaching Statistics.” Teaching Sociology 15:164–72.
Buhrmester, M., T. Kwang, and S. D. Gosling. 2011.
“Amazon’s Mechanical Turk: A New Source of
Inexpensive, Yet High-quality, Data?” Perspectives
on Psychological Science 6(1):3–5.
Casler, Krista, Lydia Bickel, and Elizabeth Hackett. 2013.
“Separate but Equal? A Comparison of Participants
and Data Gathered via Amazon’s MTurk, Social
Media, and Face-to-face Behavioral Testing.”
Computers in Human Behavior 29(6):2156–60.
Dietz, Thomas, and Linda Kalof. 2009. Introduction to
Social Statistics: The Logic of Statistical Reasoning.
New York: Wiley-Blackwell.
Goodman, Joseph K., Cynthia E. Cryder, and Amar
Cheema. 2013. “Data Collection in a Flat World:
The Strengths and Weaknesses of Mechanical Turk
Samples.” Journal of Behavioral Decision Making
26(3):213–24.
Ipeirotis, Panos. 2010. “The New Demograohics of
Mechanical Turk.” Retrieved May 17, 2017 (http://
www.behind-the-enemy-lines.com/2010/03/new-
demographics-of-mechanical-turk.html).
Levay, Kevin E., Jeremy Freese, and James N. Druckman.
2016. “The Demographic and Political Composition of
Mechanical Turk Samples.” SAGE Open 6(1):1–17.
Long, J. Scott. 2009. The Workflow of Data Analysis
Using Stata. College Station, TX: Stata Press.
McCright, Aaron M. 2012. “Enhancing Students’
Scientific and Quantitative Literacies through an
Inquiry-Based Learning Project on Climate Change.”
Journal of the Scholarship of Teaching and Learning
12(4):86–101.
Merton, Robert K. 1968. “The Matthew Effect in
Science.” Science 159(3810):56–63.
53
NORC. 2016. General Social Survey. Retrieved May 17,
2017 (http://gss.norc.org/).
Paolacci, Gabriele, and Jesse Chandler. 2014. “Inside the
Turk: Understanding Mechanical Turk as a Participant
Pool.” Current Directions in Psychological Science
23(3):184–88.
Paolacci, Gabriele, Jesse Chandler, and Panagiotis
Ipeirotis. 2010. “Running Experiments on Amazon
Mechanical Turk.” Judgment and Decision Making
5(5):411–19.
Ross, Joel, Lilly Irani, M. Silberman, Andrew Zaldivar, and
Bill Tomlinson. 2010. “Who Are the Crowdworkers?
Shifting Demographics in Mechanical Turk.” Pp.
2863–72 in CHI’10 Extended Abstracts on Human
Factors in Computing Systems. New York: ACM.
Stewart, Neil, Christoph Ungemach, Adam J. L. Harris,
Daniel M. Bartels, Ben R. Newell, Gabriele Paolacci,
and Jesse Chandler. 2015. “The Average Laboratory
Samples a Population of 7,300 Amazon Mechanical
Turk Workers.” Judgment and Decision Making
10(5):479–91.
Wallman, Katherine K. 1993. “Enhancing Statistical
Literacy: Enriching Our Society.” Journal of the
American Statistical Association 88(421):1–8.
Weinberg, Jill D., Jeremy Freese, and David McElhattan.
2014. “Comparing Data Characteristics and Results
of an Online Factorial Survey between a Population-
based and a Crowdsource-recruited Sample.” Socio-
logical Science 1:292–310.
Whitley, Cameron T. 2013. “A Picture Is Worth a
Thousand Words: Applying Image-based Learning to
Course Design.” Teaching Sociology 41(2):188–98.
Williams, Malcolm, Geoff Payne, and Luke Sloan. 2015.
“Making Sociology Count: Some Evidence and
Context in the Teaching of Quantitative Methods
in the UK.” Pp. 171–86 in An End to the Crisis of
Empirical Sociology? Trends and Challenges in
Social Research, edited by L. McKie and L. Ryan.
London: Routledge.
Author Biographies
Cameron T. Whitley is a 2017 sociology PhD graduate
from the Department of Sociology at Michigan State
University. His research addresses altruism and the percep-
tion of others, with a focus on environmental decision mak-
ing and, in particular, energy developments. His interest in
teaching as research has led to publications in Teaching
Sociology; Education, Citizenship and Social Justice;
Environmental Education Research; and the International
Journal of Sustainability in Higher Education.
Thomas Dietz is professor of sociology and environmen-
tal science and policy at Michigan State University. He
has taught undergraduate and graduate statistics for over
30 years and has written texts in both social statistics and
social research methodology. His research focuses on
altruism in environmental decision making, on the human
driving forces of environmental change, and on the inter-
play of science and democracy.