806282
2018
ASRXXX10.1177/0003122418806282American Sociological ReviewLizardo et al.
Editors’ Comment:
A Few Guidelines for
Quantitative Submissions
Sarah A. Mustillo,a Omar A. Lizardo,b
and Rory M. McVeigha
In the three years we have been editing ASR,
we have been impressed with the method-
ological breadth and depth of the submissions
to the journal. Among the subset of papers
that use primarily quantitative analytic strate-
gies, an equally impressive range of methods
and techniques is on display. The field has
come a long way since any of the three of us
were in graduate school and, indeed, many of
the articles we have published in our role as
editors represent the forefront of sophistica-
tion in techniques as varied as fixed and ran-
dom effects on the one end to web scraping
and text analysis on the other end. In this
editorial, we would like to focus on a set of
issues that seem to come up repeatedly in the
thousands of papers we have read. These are
not errors per se, but fall in the category of
gaps or lags between previously accepted
practices among quantitative scholars in soci-
ology and the state of the art consensus
among quantitative methodologists. These
issues happen with such frequency that we
feel compelled to offer some recommenda-
tions for future ASR submissions.
P-Values and One versus
Two-Tailed Tests
Debates about the utility of p-values abound
in the scientific literature. On the one hand,
those concerned about replicability and stan-
dards for new discovery argue that the thresh-
old for statistical significance should be
reduced below .05 (Benjamin et al. 2018). On
the other hand, some argue that we should do
away with p values and null hypothesis sig-
nificance testing altogether (McShane et al.
2017).
American Sociological Review
2018, Vol. 83(6) 1281­–1283
© American Sociological
Association 2018
DOI: 10.1177/0003122418806282
https://doi.org/10.1177/0003122418806282
journals.sagepub.com/home/asr
We will not take a stand in this debate
except to say that, in general, p < .10 and one-
tailed tests should only be used in rare, excep-
tional circumstances with proper justification.
Many papers attempt to justify use of p < .10
standards by pointing to “directionality” in
their verbally stated hypotheses. Others use
vague language of p < .10 indicating “border-
line” or “suggestive” findings. We do not find
the first rationale compelling. In terms of the
second practice, ASR is our discipline’s top
journal. We need to be publishing strong evi-
dence rather than “suggestive” findings.
Testing Mediation
We get many submissions to the journal attempt-
ing to test mediation with a stripped down ver-
sion of the Baron and Kenny (1986) steps.
Authors usually proceed like this—they run one
model with their key predictor plus controls and
then a second model adding the mediator. If the
coefficient of the key predictor is reduced or
rendered nonsignificant, the authors conclude
that the main effect has been mediated.
There are several problems with this
approach. Most commonly, authors fail to run a
significance test for the difference in magnitude
between coefficients. This step is necessary to
determine whether mediation has occurred. The
coefficient of the key predictor can be reduced
or even rendered nonsignificant yet still be in
the window of what could be considered to have
occurred by chance alone. As Gelman and Stern
(2006) note, changes in statistical significance
may not themselves be significant.
a
University of Notre Dame
b
University of California-Los Angeles
1282 American Sociological Review 83(6)
Occasionally, authors run a significance
test like a Sobel (1986) test to see if the reduc-
tion is statistically significant, which is a step
in the right direction. More recent tests and
procedures have been developed by various
authors in various traditions that can be imple-
mented in different software packages that
improve upon the Sobel test. Some of these
deal with the next issue we will mention,
which is that mediation in nonlinear models
such as logit and probit cannot be determined
by looking at changes in coefficient magni-
tude across models. See work by MacKinnon
(2008), Imai, Keele, and Tingley (2010), Karl-
son, Holm and Breen (2012), and Vander-
weele (2015, 2016) for more detail. We
recommend that authors who aim to test medi-
ation in future ASR publications implement
more sophisticated strategies as appropriate.
Interactions in Models
with Categorical
Dependent Variables
Various problems have been raised with using
z-statistics (and associated p-values) of the
coefficient of a multiplicative term to test for a
statistical interaction in nonlinear models with
categorical dependent variables. Allison (1999),
Williams (2009), and others focus on one type
of problem (e.g., differences in residual varia-
tion among groups), others such as Mood
(2010), Breen and Karlson (2013), and Long
and Mustillo (forthcoming) focus on a litany of
issues. The case is closed: don’t use the coeffi-
cient of the interaction term to draw conclu-
sions about statistical interaction in categorical
models such as logit, probit, Poisson, and so on.
Each scholar recommends a different approach
for testing interactions. We recommend that
future authors use the appropriate technique
depending on the particular application.
“Multivariate” versus
“Multivariable”
Scholars from many disciplines use these
terms interchangeably to describe their mod-
els, but they do not mean the same thing.
Sociologists are not alone in this common
practice. Using them interchangeably can
cause confusion as to what kind of model is
actually being estimated.
Simple regression refers to a model with
one independent variable and one dependent
variable. Multiple regression refers to a model
with multiple independent variables and one
dependent variable. Another term for multiple
regression is multivariable regression.
A multivariate model is an entirely different
model from those mentioned above: a multi-
variate model is a model with multiple depend-
ent variables, such as factor analysis, a
structural equation model, or a latent growth
curve model. Given how often these terms are
confused in published work, many argue that
the distinction has become arbitrary or seman-
tic, but we think it is important to maintain the
distinction given that multivariate statistics is a
well-developed branch of statistics in its own
right, often the subject of entire courses, and it
is consistent with usage in other fields. This is
important, since ASR papers are read widely
across the social sciences, not just sociology.
Measurement
Many authors need to take measurement
issues more seriously than they currently do.
We often receive submissions in which key
variables, both independent and dependent,
are ad hoc creations of the authors in which
the rigors of measurement science have not
been properly engaged. This strategy may be
acceptable for simple, straightforward con-
cepts, but for anything more complex, vali-
dated measures or measures with a body of
literature behind them would be preferred
when possible, appropriate, or relevant.
A less serious concern, but still a concern,
is authors who use validated scales in a man-
ner that is inconsistent with published validity
work on those scales. If a validated scale has
12 items that were coded 1 to 4 and added
together in the published validity work, then
the scale should be treated as such unless jus-
tification is provided for treating it differently.
Often we receive submissions in which items
are cherry-picked from the scale, or the coding
Mustillo et al. 1283
of such items is changed, or the summing
scheme is altered. Any of these changes may
undermine the validity of the scale.
Methods Sections
Better organization of methods sections and
more detail about procedures is one final area
that could use attention. Many submissions
fail to provide enough detail on data collec-
tion procedures, sample size, missing data,
decisions about who is included or excluded
from the sample, what types of models are
estimated and why, response rate of a survey,
selection effects, measurement of the vari-
ables, and so forth. At times, all of this infor-
mation is provided, but instead of being
neatly packaged in the methods section, it is
doled out piecemeal across the methods and
results or relegated to obscure sections in an
Appendix. The easiest papers for readers to
follow are the ones in which the methods are
tightly linked to the front-end and in which
the methods proceed in a unified, organized
fashion—data, variables, models.
In summary, sociology has made substan-
tial progress in terms of the range and sophis-
tication of quantitative methods used to
address important questions in our field. We
offer these recommendations to assist schol-
ars in making decisions about which methods
to use and how to best present results in areas
that are currently lacking clear guidelines.
References
Allison, Paul D. 1999. “Comparing Logit and Probit
Coefficients across Groups.” Sociological Methods &
Research 28(2):186–208.
Baron, Reuben M., and David A. Kenny. 1986. “Mod-
erator-Mediator Variables Distinction in Social Psy-
chological Research: Conceptual, Strategic, and
Statistical Considerations.” Journal of Personality
and Social Psychology 51(6):1173–82.
Benjamin, Daniel J., James O. Berger, Magnus Johan-
nesson, Brian A. Nosek, E.-J. Wagenmakers, Rich-
ard Berk, Kenneth A. Bollen, et al. 2018. “Redefine
Statistical Significance.” Nature Human Behaviour
2(1):6–10.
Breen, Richard, and Kristian Bernt Karlson. 2013.
“Counterfactual Causal Analysis and Nonlinear Prob-
ability Models.” Pp. 167–87 in Handbook of Causal
Analysis for Social Research. Dordrecht: Springer.
Imai, Kosuke, Luke Keele, and Dustin Tingley. 2010. “A
General Approach to Causal Mediation Analysis.”
Psychological Methods 15(4):309–334.
Gelman, Andrew, and Hal Stern. 2006. “The Difference
between ‘Significant’ and ‘Not Significant’ Is Not
Itself Statistically Significant.” The American Statis-
tician 60(4):328–31.
Karlson, Kristian B., Anders Holm, and Richard Breen.
2012. “Comparing Regression Coefficients between
Same-Sample Nested Models using Logit and Pro-
bit: A New Method.” Sociological Methodology
42(1):286–313.
Long, J. Scott, and Sarah A. Mustillo. Forthcoming.
“Comparing Groups in Binary Regression Mod-
els Using Predictions.” Sociological Methods and
Research.
MacKinnon, David P. 2008. Introduction to Statistical
Mediation Analysis. New York: Routledge.
McShane, Blakeley B., David Gal, Andrew Gelman,
Christian Robert, and Jennifer L. Tackett. 2017.
“Abandon Statistical Significance.” Unpublished
Manuscript. Retrieved June 21, 2018 (http://www
.stat.columbia.edu/~gelman/research/unpublished/
abandon.pdf).
Mood, Carina. 2010. “Logistic Regression: Why We
Cannot Do What We Think We Can Do, and What
We Can Do About It.” European Sociological Review
26(1):67–82.
Sobel, Michael E. 1986. “Some New Results on Indi-
rect Effects and Their Standard Errors in Covari-
ance Structure Models.” Sociological Methodology
16:159–86.
VanderWeele, Tyler J. 2015. Explanation in Causal
Inference: Methods for Mediation and Interaction.
Oxford, UK: Oxford University Press.
VanderWeele, Tyler J. 2016. “Mediation Analysis: A
Practitioner’s Guide.” Annual Review of Public
Health 37:17–32.
Williams, Richard. 2009. “Using Heterogeneous Choice
Models to Compare Logit and Probit Coefficients
across Groups.” Sociological Methods & Research
37(4):531–59.