Journal of Memory and Language 68 (2013) 255–278

Contents lists available at SciVerse ScienceDirect

Journal of Memory and Language

journal homepage: www.elsevier.com/locate/jml

Random effects structure for confirmatory hypothesis testing:

Keep it maximal
Dale J. Barr a,⇑, Roger Levy b, Christoph Scheepers a, Harry J. Tily c
a
Institute of Neuroscience and Psychology, University of Glasgow, 58 Hillhead St., Glasgow G12 8QB, United Kingdom
b
Department of Linguistics, University of California at San Diego, La Jolla, CA 92093-0108, USA
c
Department of Brain and Cognitive Sciences, Massachussetts Institute of Technology, Cambridge, MA 02139, USA

a r t i c l e i n f o a b s t r a c t

Article history: Linear mixed-effects models (LMEMs) have become increasingly prominent in psycholin-
Received 5 August 2011 guistics and related areas. However, many researchers do not seem to appreciate how ran-
revision received 30 October 2012 dom effects structures affect the generalizability of an analysis. Here, we argue that
researchers using LMEMs for confirmatory hypothesis testing should minimally adhere
to the standards that have been in place for many decades. Through theoretical arguments
Keywords: and Monte Carlo simulation, we show that LMEMs generalize best when they include the
Linear mixed-effects models
maximal random effects structure justified by the design. The generalization performance of
Generalization
Statistics
LMEMs including data-driven random effects structures strongly depends upon modeling
Monte Carlo simulation criteria and sample size, yielding reasonable results on moderately-sized samples when
conservative criteria are used, but with little or no power advantage over maximal models.
Finally, random-intercepts-only LMEMs used on within-subjects and/or within-items data
from populations where subjects and/or items vary in their sensitivity to experimental
manipulations always generalize worse than separate F1 and F2 tests, and in many cases,
even worse than F1 alone. Maximal LMEMs should be the ‘gold standard’ for confirmatory
hypothesis testing in psycholinguistics and beyond.
Ó 2012 Elsevier Inc. All rights reserved.

it does in everyday reasoning. Consider a pet-food manu-

‘‘I see no real alternative, in most confirmatory studies, to
having a single main question—in which a question is
specified by ALL of design, collection, monitoring, AND
ANALYSIS.’’
Tukey (1980), ‘‘We Need Both Exploratory and Confir-
matory’’ (p. 24, emphasis in original).
Introduction
The notion of independent evidence plays no less impor-
tant a role in the assessment of scientific hypotheses than
⇑ Corresponding author. Fax: +44 (0)141 330 4606.
E-mail addresses: dale.barr@glasgow.ac.ukss (D.J. Barr), rlevy@ucsd.
edu (R. Levy), christoph.scheepers@glasgow.ac.uk (C. Scheepers), hjt@mit.
edu (H.J. Tily).
facturer determining which of two new gourmet cat-food
recipes to bring to market. The manufacturer has every
interest in choosing the recipe that the average cat will
eat the most of. Thus every day for a month (28 days) their
expert, Dr. Nyan, feeds one recipe to a cat in the morning
and the other recipe to a cat in the evening, counterbalanc-
ing which recipe is fed when and carefully measuring how
much was eaten at each meal. At the end of the month Dr.
Nyan calculates that recipes 1 and 2 were consumed to the
tune of 92.9 ± 5.6 and 107.2 ± 6.1 (means ± SDs) grams per
meal respectively. How confident can we be that recipe 2 is
the better choice to bring to market? Without further
information you might hazard the guess ‘‘somewhat confi-
dent’’, considering that one of the first statistical hypothe-
sis tests typically taught, the unpaired t-test, gives p = 0.09
against the null hypothesis that choice of recipe does not
matter. But now we tell you that only seven cats partici-

0749-596X/$ - see front matter Ó 2012 Elsevier Inc. All rights reserved.
http://dx.doi.org/10.1016/j.jml.2012.11.001
256 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
pated in this test, one for each day of the week. How does
this change your confidence in the superiority of recipe 2?
Let us first take a moment to consider precisely what it
is about this new information that might drive us to
change our analysis. The unpaired t-test is based on the
assumption that all observations are conditionally indepen-
dent of one another given the true underlying means of the
two populations—here, the average amount a cat would
consume of each recipe in a single meal. Since no two cats
are likely to have identical dietary proclivities, multiple
measurements from the same cat would violate this
assumption. The correct characterization becomes that all
observations are conditionally independent of one another
given (a) the true palatability effect of recipe 1 versus rec-
ipe 2, together with (b) the dietary proclivities of each cat.
This weaker conditional independence is a double-edged
sword. On the one hand, it means that we have tested
effectively fewer individuals than our 56 raw data points
suggest, and this should weaken our confidence in general-
izing the superiority of recipe 2 to the entire cat popula-
tion. On the other hand, the fact that we have made
multiple measurements for each cat holds out the prospect
of factoring out each cat’s idiosyncratic dietary proclivities
as part of the analysis, and thereby improving the signal-
to-noise ratio for inferences regarding each recipe’s overall
appeal. How we specify these idiosyncrasies can dramati-
cally affect our conclusions. For example, we know that
some cats have higher metabolisms and will tend to eat
more at every meal than other cats. But we also know that
each creature has its own palate, and even if the recipes
were of similar overall quality, a given cat might happen
to like one recipe more than the other. Indeed, accounting
for idiosyncratic recipe preferences for each cat might lead
to even weaker evidence for the superiority of recipe 2.
Situations such as these, where individual observations
cluster together via association with a smaller set of enti-
ties, are ubiquitous in psycholinguistics and related
fields—where the clusters are typically human participants
and stimulus materials (i.e., items). Similar clustered-
observation situations arise in other sciences, such as agri-
culture (plots in a field) and sociology (students in class-
rooms in schools in school-districts); hence accounting
for the RANDOM EFFECTS of these entities has been an impor-
tant part of the workhorse statistical analysis technique,
the ANALYSIS OF VARIANCE, under the name MIXED-MODEL ANOVA,
since the first half of the 20th century (Fisher, 1925; Sche-
ffe, 1959). In experimental psychology, the prevailing stan-
dard for a long time has been to assume that individual
participants may have idiosyncratic sensitivities to any
experimental manipulation that may have an overall ef-
fect, so detecting a ‘‘fixed effect’’ of some manipulation
must be done under the assumption of corresponding par-
ticipant random effects for that manipulation as well. In
our pet-food example, if there is a true effect of recipe—
that is, if on average a new, previously unstudied cat will
on average eat more of recipe 2 than of recipe 1—it should
be detectable above and beyond the noise introduced by
cat-specific recipe preferences, provided we have enough
data. Technically speaking, the fixed effect is tested against
an error term that captures the variability of the effect
across individuals.
Standard practices for data-analysis in psycholinguis-
tics and related areas fundamentally changed, however,
after Clark (1973). In a nutshell, Clark (1973) argued that
linguistic materials, just like experimental participants,
have idiosyncrasies that need to be accounted for. Because
in a typical psycholinguistic experiment, there are multiple
observations for the same item (e.g., a given word or sen-
tence), these idiosyncrasies break the conditional indepen-
dence assumptions underlying mixed-model ANOVA,
which treats experimental participant as the only random
effect. Clark proposed the quasi-F (F0 ) and min-F0 statistics
as approximations to an F-ratio whose distributional
assumptions are satisfied even under what in contempo-
rary parlance is called CROSSED random effects of participant
and item (Baayen, Davidson, & Bates, 2008). Clark’s paper
helped drive the field toward a standard demanding evi-
dence that experimental results generalized beyond the
specific linguistic materials used—in other words, the so-
called by-subjects F1 mixed-model ANOVA was not en-
ough. There was even a time where reporting of the min-
F0 statistic was made a standard for publication in the Jour-
nal of Memory and Language. However, acknowledging the
widespread belief that min-F0 is unduly conservative (see,
e.g., Forster & Dickinson, 1976), significance of min-F0
was never made a requirement for acceptance of a publica-
tion. Instead, the ‘normal’ convention continued to be that
a result is considered likely to generalize if it passes
p < 0.05 significance in both by-subjects (F1) and by-items
(F2) ANOVAs. In the literature this criterion is called F1  F2
(e.g., Forster & Dickinson, 1976), which in this paper we
use to denote the larger (less significant) of the two p val-
ues derived from F1 and F2 analyses.
Linear mixed-effects models (LMEMs)
Since Clark (1973), the biggest change in data analysis
practices has been the introduction of methods for simulta-
neously modeling crossed participant and item effects in a
single analysis, in what is variously called ‘‘hierarchical
regression’’, ‘‘multi-level regression’’, or simply ‘‘mixed-ef-
fects models’’ (Baayen, 2008; Baayen et al., 2008; Gelman &
Hill, 2007; Goldstein, 1995; Kliegl, 2007; Locker, Hoffman,
& Bovaird, 2007; Pinheiro & Bates, 2000; Quené & van den
Bergh, 2008; Snijders & Bosker, 1999b).1 In this paper we refer
to models of this class as mixed-effects models; when fixed ef-
fects, random effects, and trial-level noise contribute linearly
to the dependent variable, and random effects and trial-level
error are both normally distributed and independent for differ-
ing clusters or trials, it is a linear mixed-effects model (LMEM).
The ability of LMEMs to simultaneously handle crossed
random effects, in addition to a number of other advanta-
ges (such as better handling of categorical data; see Dixon,
2008; Jaeger, 2008), has given them considerable momen-
1
Despite the ‘‘mixed-effects models’’ nomenclature, traditional ANOVA
approaches used in psycholinguistics have always used ‘‘mixed effects’’ in
the sense of simultaneously estimating both fixed- and random-effects
components of such a model. What is new about mixed effects models is
their explicit estimation of the random-effects covariance matrix, which
leads to considerably greater flexibility of application, including, as clearly
indicated by the title of Baayen et al. (2008), the ability to handle the
crossing of two or more types of random effects in a single analysis.
 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
tum as a candidate to replace ANOVA as the method of
choice in psycholinguistics and related areas. But despite
the widespread use of LMEMs, there seems to be insuffi-
ciently widespread understanding of the role of random ef-
fects in such models, and very few standards to guide how
random effect structures should be specified for the analy-
sis of a given dataset. Of course, what standards are appro-
priate or inappropriate depends less upon the actual
statistical technique being used, and more upon the goals
of the analysis (cf. Tukey, 1980). Ultimately, the random ef-
fect structure one uses in an analysis encodes the assump-
tions that one makes about how sampling units (subjects
and items) vary, and the structure of dependency that this
variation creates in one’s data.
In this paper, our focus is mainly on what assumptions
about sampling unit variation are most critical for the use
of LMEMs in confirmatory hypothesis testing. By confirma-
tory hypothesis testing we mean the situation in which
the researcher has identified a specific set of theory-critical
hypotheses in advance and attempts to measure the evi-
dence for or against them as accurately as possible (Tukey,
1980). Confirmatory analyses should be performed accord-
ing to a principled plan guided by theoretical consider-
ations, and, to the extent possible, should minimize the
influence of the observed data on the decisions that one
makes in the analysis (Wagenmakers, Wetzels, Borsboom,
van der Maas, & Kievit, 2012). To simplify our discussion,
we will focus primarily on confirmatory analysis of simple
data sets involving only a few theoretically-relevant vari-
ables. We recognize that in practice, the complexity of
one’s data may impose constraints on the extent to which
one can perform analyses fully guided by theory and not by
the data. Researchers who perform laboratory experiments
have extensive control over the data collection process,
and, as a result, their statistical analyses tend to include
only a small set of theoretically relevant variables, because
other extraneous factors have been rendered irrelevant
through randomization and counterbalancing. This is in
contrast to other more complex types of data sets, such
as observational corpora or large-scale data sets collected
in the laboratory for some other, possibly more general,
purpose than the theoretical question at hand. Such data-
sets may be unbalanced and complex, and include a large
number of measurements of many different kinds. Analyz-
ing such datasets appropriately is likely to require more
sophisticated statistical techniques than those we discuss
in this paper. Furthermore, such analyses may involve
data-driven techniques typically used in exploratory data
analysis in order to reduce the set of variables to a manage-
able size. Discussion of such techniques for complex data-
sets and their proper application of is beyond the scope of
this paper (but see, e.g., Baayen, 2008; Jaeger, 2010).
Our focus here is on the question: When the goal of a
confirmatory analysis is to test hypotheses about one or
more critical ‘‘fixed effects’’, what random-effects structure
should one use? Based on theoretical analysis and Monte
Carlo simulation, we will argue the following:
1. Implicit choices regarding random-effect structures
existed for traditional mixed-model ANOVAs just
as they exist today for LMEMs.
257
2. With mixed-model ANOVAs, the standard has been
to use what we term ‘‘maximal’’ random-effect
structures.
3. Insofar as we as a field think this standard is appro-
priate for the purpose of confirmatory hypothesis
testing, researchers using LMEMs for that purpose
should also be using LMEMs with maximal random
effects structure.
4. Failure to include maximal random-effect structures
in LMEMs (when such random effects are present in
the underlying populations) inflates Type I error
rates.
5. For designs including within-subjects (or within-
items) manipulations, random-intercepts-only
LMEMs can have catastrophically high Type I error
rates, regardless of how p-values are computed from
them (see also Roland, 2009; Jaeger, 2011a; Schielz-
eth & Forstmeier, 2009).
6. The performance of a data-driven approach to deter-
mining random effects (i.e., model selection)
depends strongly on the specific algorithm, size of
the sample, and criteria used; moreover, the power
advantage of this approach over maximal models is
typically negligible.
7. In terms of power, maximal models perform surpris-
ingly well even in a ‘‘worst case’’ scenario where
they assume random slope variation that is actually
not present in the population.
8. Contrary to some warnings in the literature (Pinhe-
iro & Bates, 2000), likelihood-ratio tests for fixed
effects in LMEMs show minimal Type I error infla-
tion for psycholinguistic datasets (see Baayen et al.,
2008, FootNote 1, for a similar suggestion); also,
deriving p-values from Monte Carlo Markov Chain
(MCMC) sampling does not mitigate the high Type
I error rates of random-intercepts-only LMEMs.
9. The F1  F2 criterion leads to increased Type I error
rates the more the effects vary across subjects and
items in the underlying populations (see also Clark,
1973; Forster & Dickinson, 1976).
10. Min-F0 is conservative in between-items designs
when the item variance is low, and is conservative
overall for within-items designs, especially so when
the treatment-by-subject and/or treatment-by-item
variances are low (see also Forster & Dickinson,
1976); in contrast, maximal LMEMs show no such
conservativity.
Further results and discussion are available in an Online
Appendix.
Random effects in LMEMs and ANOVA: the same principles
apply
The Journal of Feline Gastronomy has just received a sub-
mission reporting that the feline palate prefers tuna to li-
ver, and as journal editor you must decide whether to
send it out for review. The authors report a highly signifi-
cant effect of recipe type (p < .0001), stating that they used
‘‘a mixed effects model with random effects for cats and
recipes’’. Are you in a position to evaluate the generality
258 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278

of the findings? Given that LMEMs can implement nearly
any of the standard parametric tests—from a one-sample
test to a multi-factor mixed-model ANOVA—the answer
can only be no. Indeed, whether LMEMs produce valid
inferences depends critically on how they are used. What
you need to know in addition is the random effects structure
of the model, because this is what the assessment of the
treatment effects is based on. In other words, you need
to know which treatment effects are assumed to vary
across which sampled units, and how they are assumed
to vary. As we will see, whether one is specifying a random
effects structure for LMEMs or choosing an analysis from
among the traditional options, the same considerations
come into play. So, if you are scrupulous about choosing
the ‘‘right’’ statistical technique, then you should be
equally scrupulous about using the ‘‘right’’ random effects
structure in LMEMs. In fact, knowing how to choose the
right test already puts you in a position to specify the cor-
rect random effects structure for LMEMs.
In this section, we attempt to distill the implicit stan-
dards already in place by walking through a hypothetical
example and discussing the various models that could be
applied, their underlying assumptions, and how these
assumptions relate to more traditional analyses. In this
hypothetical ‘‘lexical decision’’ experiment, subjects see
strings of letters and have to decide whether or not each
string forms an English word, while their response times
are measured. Each subject is exposed to two types of
words, forming condition A and condition B of the experi-
ment. The words in one condition differ from those in the
other condition on some intrinsic categorical dimension
(e.g., syntactic class), comprising a word-type manipulation
that is within-subjects and between-items. The question is
whether reaction times are systematically different be-
tween condition A and condition B. For expository pur-
poses, we use a ‘‘toy’’ dataset with hypothetical data from
four subjects and four items, yielding two observations
per treatment condition per participant. The observed data
are plotted alongside predictions from the various models
we will be considering in the panels of Fig. 1. Because we
are using simulated data, all of the parameters of the popu-
lation are known, as well as the ‘‘true’’ subject-specific and
item-specific effects for the sampled data. In practice,
researchers do not know these values and can only estimate
them from the data; however, using known values for
hypothetical data can aid in understanding how clustering
in the population maps onto clustering in the sample.
The general pattern for the observed data points sug-
gests that items of type B (I3 and I4) are responded to fas-
ter than items of type A (I1 and I2). This suggests a simple
(but clearly inappropriate) model for these data that re-
lates response Ysi for subject s and item i to a baseline level
via fixed-effect b0 (the intercept), a treatment effect via
fixed-effect b1 (the slope), and observation-level error esi
with variance r2:
Y si ¼ b0 þ b1 X i þ esi ;
ð1Þ
esi  Nð0; r2 Þ;

Fig. 1. Example RT data (open symbols) and model predictions (filled symbols) for a hypothetical lexical decision experiment with two within-subject/
between-item conditions, A (triangles) and B (circles), including four subjects (S1–S4) and four items (I1–I4). Panel (a) illustrates a model with no random
effects, considering only the baseline average RT (response to word type A) and treatment effect; panel (b) adds random subject intercepts to the model;
panel (c) adds by-subject random slopes; and panel (d) illustrates the additional inclusion of by-item random intercepts. Panel (d) represents the maximal
random-effects structure justified for this design; any remaining discrepancies between observed data and model estimates are due to trial-level
measurement error (esi).
 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
where Xi is a predictor variable2 taking on the value of 0 or
1 depending on whether item i is of type A or B respectively,
and esi  N(0, r2) indicates that the trial-level error is nor-
mally distributed with mean 0 and variance r2. In the pop-
ulation, participants respond to items of type B 40 ms faster
than items of type A. Under this first model, we assume that
each of the 16 observations provides the same evidence for
or against the treatment effect regardless of whether or not
any other observations have already been taken into ac-
count. Performing an unpaired t-test on these data would
implicitly assume this (incorrect) generative model.
Model (1) is not a mixed-effects model because we have
not defined any sources of clustering in our data; all obser-
vations are conditionally independent given a choice of
intercept, treatment effect, and noise level. But experience
tells us that different subjects are likely to have different
overall response latencies, breaking conditional indepen-
dence between trials for a given subject. We can expand
our model to account for this by including a new offset
term S0s, the deviation from b0 for subject s. The expanded
model is now
Y si ¼ b0 þ S0s þ b1 X i þ esi ;
S0s  Nð0; s200 Þ; ð2Þ
esi  Nð0; r2 Þ:
These offsets increase the model’s expressivity by allowing
predictions for each subject to shift upward or downward
by a fixed amount (Fig. 1b). Our use of Latin letters for this
term is a reminder that S0s is a special type of effect which
is different from the bs—indeed, we now have a ‘‘mixed-ef-
fects’’ model: parameters b0 and b1 are fixed effects (effects
that are assumed to be constant from one experiment to
another), while the specific composition of subject levels
for a given experiment is assumed to be a random subset
of the levels in the underlying populations (another instan-
tiation of the same experiment would have a different
composition of subjects, and therefore different realiza-
tions of the S0s effects). The S0s effects are therefore random
effects; specifically, they are random intercepts, as they al-
low the intercept term to vary across subjects. Our primary
goal is to produce a model which will generalize to the
population from which these subjects are randomly drawn,
rather than describing the specific S0s values for this sam-
ple. Therefore, instead of estimating the individual S0s ef-
fects, the model-fitting algorithm estimates the
population distribution from which the S0s effects were
drawn. This requires assumptions about this distribution;
we follow the common assumption that it is normal, with
a mean of 0 and a variance of s200 ; here s200 is a random effect
parameter, and is denoted by a Greek symbol because, like
the bs, it refers to the population and not to the sample.
2 deed, such a negative correlation, where we would have
For expository purposes, we use a treatment coding scheme (0 or 1) for
the predictor variable. Alternatively, the models in this section could be
expressed in the style more common to traditional ANOVA pedagogy,
where fixed and random effects represent deviations from a grand mean.
This model can be fit by using ‘‘deviation coding’’ for the predictor variable
(.5 and .5 instead of 0 and 1). For higher-order designs, treatment and
deviation coding schemes will lead to different interpretations for lower-
order effects (simple effects for contrast coding and main effects for
deviation coding).
259
Note that the variation on the intercepts is not con-
founded with our effect of primary theoretical interest
(b1): for each subject, it moves the means for both condi-
tions up or down by a fixed amount. Accounting for this
variation will typically decrease the residual error and thus
increase the sensitivity of the test of b1. Fitting Model (2) is
thus analogous to analyzing the raw, unaggregated re-
sponse data using a repeated-measures ANOVA with SSsub-
jects subtracted from the residual SSerror term. One could see
that this analysis is wrong by observing that the denomi-
nator degrees of freedom for the F statistic (i.e., corre-
sponding to MSerror) would be greater than the number of
subjects (see Online Appendix for further discussion and
demonstration).
Although Model (2) is clearly preferable to Model (1), it
does not capture all the possible by-subject dependencies
in the sample; experience also tells us that subjects often
vary not only in their overall response latencies but also
in the nature of their response to word type. In the present
hypothetical case, Subject 3 shows a total effect of
134 ms, which is 94 ms larger than the average effect in
the population of 40 ms. We have multiple observations
per combination of subject and word type, so this variabil-
ity in the population will also create clustering in the sam-
ple. The S0s do not capture this variability because they
only allow subjects to vary around b0. What we need in
addition are random slopes to allow subjects to vary with
respect to b1, our treatment effect. To account for this var-
iation, we introduce a random slope term S1s with variance
s211 , yielding
Y si ¼ b0 þ S0s þ ðb1 þ S1s ÞX i þ esi ;
" #!
s200 qs00 s11
ðS0s ; S1s Þ  N 0; ; ð3Þ
qs00 s11 s211
2
esi  Nð0; r Þ:
This is now a mixed-effects model with by-subject random
intercepts and random slopes. Note that the inclusion of the
by-subject random slope causes the predictions for condi-
tion B to shift by a fixed amount for each subject (Fig. 1c),
improving predictions for words of type B. The slope offset
S1s captures how much Subject s’s effect deviates from the
population treatment effect b1. Again, we do not want our
analysis to commit to particular S1s effects, and so, rather
than estimating these values directly, we estimate s211 ,
the by-subject variance in treatment effect. But note that
now we have two random effects for each subject s, and
these two effects can exhibit a correlation (expressed by
q). For example, subjects who do not read carefully might
not only respond faster than the typical subject (and have a
negative S0s), but might also show less sensitivity to the
word type manipulation (and have a more positive S1s). In-
q < 0, is suggested in our hypothetical data (Fig. 1): S1
and S3 are slow responders who show clear treatment ef-
fects, whereas S2 and S4 are fast responders who are
hardly susceptible to the word type manipulation. In the
most general case, we should not treat these effects as
coming from independent univariate distributions, but in-
stead should treat S0s and S1s as being jointly drawn from a
260 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
bivariate distribution. As seen in line 2 of Eq. (3), we follow
common assumptions in taking this distribution as bivari-
ate normal with a mean of (0, 0) and three free parameters:
s200 (random intercept variance), s211 (random slope vari-
ance), and qs00s11 (the intercept/slope covariance). For fur-
ther information about random effect variance–covariance
structures, see Baayen (2004, 2008), Gelman and Hill
(2007), Goldstein (1995), Raudenbush and Bryk (2002),
and Snijders and Bosker (1999a).
Both Models (2) and (3) are instances of what is tradi-
tionally analyzed using ‘‘mixed-model ANOVA’’ (e.g., Sche-
ffe, 1959, chap. 8). By long-standing convention in our
field, however, the classic ‘‘by-subjects ANOVA’’ (and anal-
ogously ‘‘by-items ANOVA’’ when items are treated as the
random effect) is understood to mean Model (3), the rele-
MST
vant F-statistic for which is F 1 ¼ MS TxS
, where MST is the
treatment mean square and MSTxS is the treatment-by-sub-
ject mean square. This convention presumably derives
from the widespread recognition that subjects (and items)
usually do vary idiosyncratically not only in their global
mean responses but also in their sensitivity to the experi-
mental treatment. Moreover, this variance, unlike random
intercept variance, can drive differences between condition
means. This can be seen by comparing the contributions of
random intercepts versus random slopes across panels (b)
and (c) in Fig. 1. Therefore, it would seem to be important
to control for such variation when testing for a treatment
effect.3
Although Model (3) accounts for all by-subject random
variation, it still has a critical defect. As Clark (1973) noted,
the repetition of words across observations is a source of
non-independence not accounted for, which would impair
generalization of our results to new items. We need to
incorporate item variability into the model with the ran-
dom effects I0i, yielding
Y si ¼ b0 þ S0s þ I0i þ ðb1 þ S1s ÞX i þ esi ;
" #!
s200 qs00 s11
ðS0s ; S1s Þ  N 0; ; model. A model like this is unidentifiable for the data it is
qs00 s11 s211 ð4Þ
2
I0i  Nð0; x 00 Þ;
2
esi  Nð0; r Þ:
This is a mixed-effect model with by-subject random inter-
cepts and slopes and by-item random intercepts. Rather
than committing to specific I0i values, we assume that
the I0i effects are drawn from a normal distribution with
a mean of zero and variance x200 . We also assume that
x200 is independent from the s parameters defining the
by-subject variance components. Note that the inclusion
of by-item random intercepts improves the predictions
from the model, with predictions for a given item shifting
3 that subject and treatment cannot interact—a model with
Note that in practice, most researchers do not compute MSTxS on the
raw data but rather aggregate their data first so that there is one
observation per subject per cell, and then perform an ANOVA (or t-test)
on the cell means. This aggregation confounds the random slope variance
with residual error and reduces the error degrees of freedom, making it
possible to perform a repeated-measures ANOVA. This is an alternative way
of meeting the assumption of conditional independence, but the aggrega-
tion precludes simultaneous generalization over subjects and items (see
online appendix for further details).
by a consistent amount across all subjects (Fig. 1d). It is
also worth noting that the by-item variance is also con-
founded with our effect of interest, since we have different
items in the different conditions, and thus will tend to con-
tribute to any difference we observe between the two con-
dition means.
This analysis has a direct analogue to min-F0 , which
tests MST against a denominator term consisting of the
sum of MSTxS and MSI, the mean squares for the random
treatment-by-subject interaction and the random main ef-
fect of items. It is, however, different from the practice of
performing F1  F2 and rejecting the null hypothesis if
p < .05 for both Fs. The reason is that MST (the numerator
for both F1 and F2) reflects not only the treatment effect,
 
but also treatment-by-subject
 2  variability s211 as well as
by-item variability x00 . The denominator of F1 controls
for treatment-by-subject variability but not item variabil-
ity; similarly, the denominator of F2 controls for item var-
iability but not treatment-by-subject variability. Thus,
finding that F1 is significant implies that b1 – 0 or
x200 – 0, or both; likewise, finding that F2 is significant im-
plies that b1 – 0 or s211 – 0, or both. Since x200 and s211 can
be nonzero while b1 = 0, F1  F2 tests will inflate the Type
I error rate (Clark, 1973). Thus, in terms of controlling Type
I error rate, the mixed-effects modeling approach and the
min-F0 approach are, at least theoretically, superior to sep-
arate by-subject and by-item tests.
At this point, we might wish to go further and consider
other models. For instance, we have considered a by-sub-
ject random slope; for consistency, why not also consider
a model with a by-item random slope, I1i? A little reflection
reveals that a by-item random slope does not make sense
for this design. Words are nested within word types—no
word can be both type A and type B—so it is not sensible
to ask whether words vary in their sensitivity to word type.
No sample from this experiment could possibly give us the
information needed to estimate random slope variance and
random slope/intercept covariance parameters for such a
applied to: there are (infinitely) many different values we
could choose for its parameters which would describe
the data equally well.4 Experiments with a within-item
manipulation, such as a priming experiment in which target
words are held constant across conditions but the prime
word is varied, would call for by-item random slopes, but
not the current experiment.
The above point also extends to designs where one
independent variable is manipulated within- and another
variable between- subjects (respectively items). In case of
between-subject manipulations, the levels of the subject
variable are nested within the levels of the experimental
treatment variable (i.e. each subject belongs to one and
only one of the experimental treatment groups), meaning
a by-subject random slope term would be unidentifiable.
In general, within-unit treatments require both the by-unit
4
Technically, by-item random slopes for a between-item design can be
used to capture heteroscedasticity across conditions, but this is typically a
minor concern in comparison with the issues focused on in this paper (see,
e.g., discussion in Gelman & Hill, 2007).
 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
intercepts and slopes in the random effects specification,
whereas between-unit treatments require only the by-unit
random intercepts.
It is important to note that identifiability is a property
of the model given a certain dataset. The model with by-
item random slopes is unidentifiable for any possible data-
set because it tries to model a source of variation that
could not logically exist in the population. However, there
are also situations where a model is unidentifiable because
there is insufficient data to estimate its parameters. For in-
stance, we might decide it was important to try to estimate
variability corresponding to the different ways that sub-
jects might respond to a given word (a subject-by-item
random intercept). But to form a cluster in the sample, it
is necessary to have more than one observation for a given
unit; otherwise, the clustering effect cannot be distin-
guished from residual error.5 If we only elicit one observa-
tion per subject/item combination, we are unable to
estimate this source of variability, and the model containing
this random effect becomes unidentifiable. Had we used a
design with repeated exposures to the same items for a gi-
ven subject, the same model would be identifiable, and in
fact we would need to include that term to avoid violating
the conditional independence of our observations given sub-
ject and item effects.
This discussion indicates that Model (4) has the maxi-
mal random effects structure justified by our experimental
design, and we henceforth refer to such models as maximal
models. A maximal model should optimize generalization
of the findings to new subjects and new items. Models that
lack random effects contained in the maximal model, such
as Models (1)–(3), are likely to be misspecified—the model
specification may not be expressive enough to include
the true generative process underlying the data. This type
of misspecification is problematic because conditional
independence between observations within a given cluster
is not achieved. Each source of random variation that is not
accounted for will tend to work against us in one of two
different ways. On the one hand, unaccounted-for variation
that is orthogonal to our effect of interest (e.g., random
intercept variation) will tend to reduce power for our tests
of that effect; on the other, unaccounted-for variation that
is confounded with our effect of interest (e.g., random
slope variation), can drive differences between means,
and thus will tend to increase the risk of Type I error.
A related model that we have not yet considered but
that has become popular in recent practice includes only
by-subject and by-item random intercepts.
Y si ¼ b0 þ S0s þ I0i þ b1 X i þ esi ;
 
S0i  N 0; s200 ;
  ð5Þ
I0i  N 0; x200 ;
esi  Nð0; r Þ:2
Unlike the other models we have considered up to this
point, there is no clear ANOVA analog to a random-inter-
5
It can also be difficult to estimate random effects when some of the
sampling units (subjects or items) provide few observations in particular
cells of the design. See Section ‘General Discussion’ and Jaeger, Graff, Croft,
and Pontillo (2011, Section 3.3) for further discussion of this issue.
261
cepts-only LMEM; it is perhaps akin to a modified min-F0
statistic with a denominator error term including MSI but
with MSTxS replaced by the error term from Model (2)
(i.e., with SSerror reduced by SSsubjects). But it would seem
inappropriate to use this as a test statistic, given that the
numerator MST increases as a function not only of the over-
all treatment effect, but also as a function of random slope
 
variation s211 , and the denominator does not control for
this variation.
A common misconception is that crossing subjects and
items in the intercept term of LMEMs is sufficient for meet-
ing the assumption of conditional independence, and that
including random slopes is strictly unnecessary unless it
is of theoretical interest to estimate that variability (see
e.g., Janssen, 2012; Locker et al., 2007). However, this is
problematic given the fact that, as already noted, random
slope variation can drive differences between condition
means, thus creating a spurious impression of a treatment
effect where none might exist. Indeed, some researchers
have already warned against using random-intercepts-
only models when random slope variation is present
(e.g., Baayen, 2008; Jaeger, 2011a; Roland, 2009; Schielzeth
& Forstmeier, 2009). However, the performance of these
models has not yet been evaluated in the context of simul-
taneous generalization over subjects and items. Our simu-
lations will provide such an evaluation.
Although the maximal model best captures all the
dependencies in the sample, sometimes it becomes neces-
sary for practical reasons to simplify the random effects
structure. Fitting LMEMs typically involves maximum like-
lihood estimation, where an iterative procedure is used to
come up with the ‘‘best’’ estimates for the parameters gi-
ven the data. As the name suggests, it attempts to maxi-
mize the likelihood of the data given the structure of the
model. Sometimes, however, the estimation procedure will
fail to ‘‘converge’’ (i.e., to find a solution) within a reason-
able number of iterations. The likelihood of this conver-
gence failure tends to increase with the complexity of the
model, especially the random effects structure.
Ideally, simplification of the random effects structure
should be done in a principled way. Dropping a random
slope is not the only solution, nor is it likely to be the best,
given that random slopes tend to account for variance con-
founded with the fixed effects of theoretical interest. We
thus consider two additional mixed-effects models with
simplified random effects structure.6 The first of these is al-
most identical to the maximal model (Model (4)) but with-
out any correlation parameter:
Y si ¼ b0 þ S0s þ I0i þ ðb1 þ S1s ÞX i þ esi ;
" #!
s200 0
ðS0s ; S1s Þ  N 0; ;
0 s211 ð6Þ
 
I0i  N 0; x200 ;
esi  Nð0; r2 Þ:
6
Unlike the other models we have considered up to this point, the
performance of these two additional models (Models (6) and (7)) will
depend to some extent on how the predictor variable X is coded (e.g.,
treatment or deviation coding, with performance generally better for the
latter; see Appendix for further discussion).
262 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
Note that the only difference from Model (4) is in the spec-
ification of the distribution of (S0s, S1s) pairs. Model (6) is
more restrictive than Model (4) in not allowing correlation
between the random slope and random intercept; if, for
example, subjects with overall faster reaction times also
tended to be less sensitive to experimental manipulation
(as in our motivating example for random slopes), Model
(6) could not capture that aspect of the data. But it does ac-
count for the critical random variances that are con-
founded with the effect of interest, s211 and x200 .
The next and final model to consider is one that has re-
ceived almost no discussion in the literature but is none-
theless logically possible: a maximal model that is
simplified by removing random intercepts for any with-
in-unit (subject or item) factor. For the current design, this
means removing the by-subject random intercept:
Y si ¼ b0 þ I0i þ ðb1 þ S1s ÞX i þ esi ;
 
S1s  N 0; s211 ;
 
I0i  N 0; x200 ;
2
esi  Nð0; r Þ:
This model, like random-intercepts-only and no-correla-
tion models, would almost certainly be misspecified for
typical psycholinguistic data. However, like the previous
model, and unlike the random-intercepts-only model, it
captures all the sources of random variation that are con-
founded with the effect of main theoretical interest.
The mixed-effects models considered in this section are
presented in Table 1. We give their expression in the syn-
tax of lmer (Bates, Maechler, & Bolker, 2011), a widely
used mixed-effects fitting method for R (R Development
Core Team, 2011). To summarize, when specifying random
effects, one must be guided by (1) the sources of clustering
that exist in the target subject and item populations, and
(2) whether this clustering in the population will also exist
in the sample. The general principle is that a by-subject (or
by-item) random intercept is needed whenever there is
more than one observation per subject (or item or sub-
ject-item combination), and a random slope is needed for
any effect where there is more than one observation for
each unique combination of subject and treatment level
(or item and treatment level, or subject-item combination
and treatment level). Models are unidentifiable when they
include random effects that are logically impossible or that
cannot be estimated from the data in principle. Models are
misspecified when they fail to include random effects that
create dependencies in the sample. Subject- or item-re-
lated variance that is not accounted for in the sample can
work against generalizability in two ways, depending on
whether on not it is independent of the hypothesis-critical
fixed effect. In the typical case in which fixed-effect slopes
are of interest, models without random intercepts will
have reduced power, while models without random slopes
will exhibit an increased Type I error rate. This suggests
that LMEMs with maximal random effects structure have
the best potential to produce generalizable results.
Although this section has only dealt with a simple single-
factor design, these principles extend in a straightforward
Table 1
Summary of models considered and associated lmer syntax.
No. Model lmer model syntax
(1) Ysi = b0 + b1Xi + esi n/a (Not a mixed-effects model)
(2) Ysi = b0 + S0s + b1Xi + esi Y  X+(1jSubject)
(3) Ysi = b0 + S0s + Y  X+(1 + XjSubject)
(b1 + S1s)Xi + esi
(4) Ysi = b0 + S0s + I0i + Y  X+(1 + XjSubject)+(1jItem)
(b1 + S1s)Xi + esi
(5) Ysi = b0 + S0s + I0i + b1Xi + esi Y  X+(1jSubject)+(1jItem)
(6)a As (4), but S0s, S1s Y  X+(1jSubject)+(0 + Xj
independent Subject)+(1jItem)
(7)a Ysi = b0 + I0i + Y  X+(0 + XjSubject)+(1jItem)
(b1 + S1s)Xi + esi
a
Performance is sensitive to the coding scheme for variable X (see
Online Appendix).
manner to higher-order designs, which we consider further
in Section ‘General Discussion’.
ð7Þ
Design-driven versus data-driven random effects specification
As the last section makes evident, in psycholinguistics
and related areas, the specification of the structure of ran-
dom variation is traditionally driven by the experimental
design. In contrast to this traditional design-driven ap-
proach, a data-driven approach has gained prominence
along with the recent introduction of LMEMs. The basic
idea behind this approach is to let the data ‘‘speak for
themselves’’ as to whether certain random effects should
be included in the model or not. That is, on the same data
set, one compares the fit of a model with and without cer-
tain random effect terms (e.g. Model (4) versus Model (5)
in the previous section) using goodness of fit criteria that
take into account both the accuracy of the model to the
data and its complexity. Here, accuracy refers to how much
variance is explained by the model and complexity to how
many predictors (or parameters) are included in the model.
The goal is to find a structure that strikes a compromise be-
tween accuracy and complexity, and to use this resulting
structure for carrying out hypothesis tests on the fixed ef-
fects of interest.
Although LMEMs offer more flexibility in testing ran-
dom effects, data-driven approaches to random effect
structure have long been possible within mixed-model AN-
OVA (see the online appendix). For example, Clark (1973)
considers a suggestion by Winer (1971) that one could test
the significance of the treatment-by-subjects interaction at
some liberal alpha level (e.g., .25), and, if it is not found to
be significant, to use the F2 statistic to test one’s hypothesis
instead of a quasi-F statistic (Clark, 1973, p. 339). In LMEM
terms, this is similar to using model comparison to test
whether or not to include the by-subject random slope (al-
beit with LMEMs, this could be done while simultaneously
controlling for item variance). But Clark rejected such an
approach, finding it unnecessarily risky (see e.g., Clark,
1973, p. 339). Whether they shared Clark’s pessimism or
not, researchers who have used ANOVA on experimental
data have, with rare exception, followed a design-driven
rather than a data-driven approach to specifying random
effects.
 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
We believe that researchers using ANOVA have been
correct to follow a design-driven approach. Moreover, we
believe that a design-driven approach is equally preferable
to a data-driven approach for confirmatory analyses using
LMEMs. In confirmatory analyses, random effect variance
is generally considered a ‘‘nuisance’’ variable rather than
a variable of interest; one does not eliminate these vari-
ables just because they do not ‘‘improve model fit.’’ As sta-
ted by Ben Bolker (one of the developers of lme4), ‘‘If
random effects are part of the experimental design, and if
the numerical estimation algorithms do not break down,
then one can choose to retain all random effects when esti-
mating and analyzing the fixed effects’’ (Bolker et al., 2009,
p. 134). The random effects are crucial for encoding mea-
surement-dependencies in the design. Put bluntly, if an
experimental treatment is manipulated within-subjects
(with multiple observations per subject-by-condition cell),
then there is no way for the analysis procedure to ‘‘know’’
about this unless the fixed effect of that treatment is
accompanied with a by-subject random slope in the analy-
sis model. Also, it is important to bear in mind that exper-
imental designs are usually optimized for the detection of
fixed effects, and not for the detection of random effects.
Data-driven techniques will therefore not only (correctly)
reject random effects that do not exist, but also (incor-
rectly) reject random effects for which there is just insuffi-
cient power. This problem is exacerbated for small
datasets, since detecting random effects is harder the few-
er clusters and observations-per-cluster are present.
A further consideration is that the there are no existing
criteria to guide researchers in the data-driven determina-
tion of random effects structure. This is unsatisfactory be-
cause the approach requires many decisions: What a-level
should be used? Should a be corrected for the number of
random effects being tested? Should one test random ef-
fects following a forward or backward algorithm, and
how should the tests be ordered? Should intercepts be
tested as well as slopes, or left in the model by default?
The number of possible random effects structures, and
thus the number of decisions to be made, increases with
the complexity of the design. As we will show, the partic-
ular decision criteria that are used will ultimately affect
the generalizability of the test. The absence of any ac-
cepted criteria allows researchers to make unprincipled
(and possibly self-serving) choices. To be sure, it may be
possible to obtain reasonable results using a data-driven
approach, if one adheres to conservative criteria. However,
even when the modeling criteria are explicitly reported, it
is a non-trivial problem to quantify potential increases in
anti-conservativity that the procedure has introduced
(see Harrell, 2001, chap. 4).
But even if one agrees that, in principle, a design-driven
approach is more appropriate than a data-driven approach
for confirmatory hypothesis testing, there might be con-
cern that using LMEMs with maximal random effects
structure is a recipe for low power, by analogy with min-
F0 , an earlier solution to the problem of simultaneous gen-
eralization. The min-F0 statistic has indeed been shown to
be conservative under some circumstances (Forster & Dick-
inson, 1976), and it is perhaps for this reason that it has not
been broadly adopted as a solution to the problem of
263
simultaneous generalization. If maximal LMEMs also turn
out to have low power, then perhaps this would justify
the extra Type I error risk associated with data-driven ap-
proaches. However, the assumption that maximal models
are overly conservative should not be taken as a forgone
conclusion. Although maximal models are similar in spirit
to min-F0 , there are radical differences between the estima-
tion procedures for min-F0 and maximal LMEMs. Min-F0 is
composed of two separately calculated F statistics, and
the by-subjects F does not control for the by-item noise,
nor does the by-items F control for the by-subjects noise.
In contrast, with maximal LMEMs by-subject and by-item
variance is taken into account simultaneously, yielding
greater prospects for being a more sensitive test.
Finally, we believe it is important to distinguish be-
tween model-selection for the purpose of data exploration
on the one hand and model-selection for the purpose of
determining random effects structures (in confirmatory
contexts) on the other; we are skeptical about the latter,
but do not intend to pass any judgement on the former.
Modeling of random effects in the current psycholinguistic
literature
The introduction of LMEMs and their early application
to psycholinguistic data by Baayen et al. (2008) has had a
major influence on analysis techniques used in peer-re-
viewed publications. At the time of writing (October
2012), Google Scholar reports 1004 citations to Baayen,
Davidson and Bates. In a informal survey of the 150 re-
search articles published in the Journal of Memory and Lan-
guage since Baayen et al. (from volume 59 issue 4 to
volume 64 issue 3) we found that 20 (13%) reported anal-
yses using an LMEM of some kind. However, these papers
differ substantially in both the type of models used and
the information reported about them. In particular,
researchers differed in whether they included random
slopes or only random intercepts in their models. Of the
20 JML articles identified, six gave no information about
the random effects structure, and a further six specified
that they used random intercepts only, without theoretical
or empirical justification. A further five papers employed
model selection, four forward and only one backward
(testing for the inclusion of random effects, but not fixed
effects). The final three papers employed a maximal ran-
dom effects structure including intercept and slope terms
where appropriate.
This survey highlights two important points. First, there
appears to be no standard for reporting the modeling pro-
cedure, and authors vary dramatically in the amount of de-
tail they provide. Second, at least 30% of the papers and
perhaps as many as 60%, do not include random slopes,
i.e. they tacitly assume that individual subjects and items
are affected by the experimental manipulations in exactly
the same way. This is in spite of the recommendations of
various experts in peer-reviewed papers and books (Baa-
yen, 2008; Baayen et al., 2008) as well as in the informal
literature (Jaeger, 2009, 2011b). Furthermore, none of the
LMEM articles in the JML special issue (Baayen et al.,
2008; Barr, 2008; Dixon, 2008; Jaeger, 2008; Mirman, Dix-
on, & Magnuson, 2008; Quené & van den Bergh, 2008) set a
264 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
bad example of using random-intercept-only models. As
discussed earlier, the use of random-intercept-only models
is a departure even from the standard use of ANOVA in
psycholinguistics.
The present study
How do current uses of LMEMs compare to more tradi-
tional methods such as min-F0 and F1  F2? The next sec-
tion of this paper tests a wide variety of commonly used
analysis methods for datasets typically collected in psy-
cholinguistic experiments, both in terms of whether result-
ing significance levels can be trusted—i.e., whether the
Type I error rate for a given approach in a given situation
is conservative (less than a), nominal (equal to a), or anti-
conservative (greater than a)—and the power of each meth-
od in detecting effects that are actually present in the
populations.
Ideally, we would compare the different analysis tech-
niques by applying them to a large selection of real data
sets. Unfortunately, in real experiments the true generative
process behind the data is unknown, meaning that we can-
not tell whether effects in the population exist—or how big
those effects are—without relying on one of the analysis
techniques we actually want to evaluate. Moreover, even
if we knew which effects were real, we would need far
more datasets than are readily available to reliably esti-
mate the nominality and power of a given method.
We therefore take an alternative approach of using
Monte Carlo methods to generate data from simulated
experiments. This allows us to specify the underlying sam-
pling distributions per simulation, and thus to have verid-
ical knowledge of the presence or absence of an effect of
interest, as well as all other properties of the experiment
(number of subjects, items and trials, and the amount of
variability introduced at each level). Such a Monte Carlo
procedure is standard for this type of problem (e.g., Baayen
et al., 2008; Davenport & Webster, 1973; Forster & Dickin-
son, 1976; Quené & van den Bergh, 2004; Santa, Miller, &
Shaw, 1979; Schielzeth & Forstmeier, 2009; Wickens &
Keppel, 1983), and guarantees that as the number of sam-
ples increases, the obtained p-value distribution becomes
arbitrarily close to the true p-value distribution for data-
sets generated by the sampling model.
The simulations assume an ‘‘ideal-world scenario’’ in
which all the distributional assumptions of the model class
(in particular normal distribution of random effects and
trial-level error, and homoscedasticity of trial-level error
and between-items random intercept variance) are satis-
fied. Although the approach leaves open for future research
many difficult questions regarding departures of realistic
psycholinguistic data from these assumptions, it allows
us great flexibility in analyzing the behavior of each ana-
lytic method as the population and experimental design
vary. We hence proceed to the systematic investigation
of traditional ANOVA, min-F0 , and several types of LMEMs
as datasets vary in many crucial respects including be-
tween- versus within-items, different numbers of items,
and different random-effect sizes and covariances.
Method
Generating simulated data
For simplicity, all datasets included a continuous re-
sponse variable and had only a single two-level treatment
factor, which was always within subjects, and either with-
in or between items. When it was within, each ‘‘subject’’
was assigned to one of two counterbalancing ‘‘presenta-
tion’’ lists, with half of the subjects assigned to each list.
We assumed no list effect; that is, the particular configura-
tion of ‘‘items’’ within a list did not have any unique effect
over and above the item effects for that list. We also as-
sumed no order effects, nor any effects of practice or fati-
gue. All experiments had 24 subjects, but we ran
simulations with both 12 or 24 items to explore the effect
of number of random-effect clusters on fixed-effects
inference.7
Within-item data sets were generated from the follow-
ing sampling model:
Y si ¼ b0 þ S0s þ I0i þ ðb1 þ S1s þ I1i ÞX si þ esi
with all variables defined as in the tutorial section above,
except that we used deviation coding for Xsi (.5, .5) rather
than treatment coding. Random effects S0s and S1s were
drawn from a bivariate normal distribution with means
lS = h0, 0i and variance–covariance matrix
 
s 2
00 qS s00 s11
T¼ . Likewise, I0i and I1i were also
qS s00 s11 2
s11
drawn from a separate bivariate normal distribution with
lI = h0, 0i and variance–covariance matrix
 
x 2
q x x
X¼ 00 I 00 11 . The residual errors esi were
qI x00 x11 2
x11
drawn from a normal distribution with a mean of 0 and
variance r2. For between-item designs, the I1i effects (by-
item random slopes) were simply ignored and thus did
not contribute to the response variable.
We investigated the performance of various analyses
over a range of population parameter values (Table 2). To
generate each simulated dataset, we first determined the
population parameters b0 ; s200 ; s211 ; qS ; x200 ; x211 ; qI ; and
r2 by sampling from uniform distributions with ranges gi-
ven in Table 2. We then simulated 24 subjects and 12 or
24 items from the corresponding populations, and simu-
lated one observation for each subject/item pair. We also as-
sumed missing data, with up to 5% of observations in a given
data set counted as missing (at random). This setting was as-
sumed to reflect normal rates of data loss (due to experi-
menter error, technical issues, extreme responses, etc.).
The Online Appendix presents results for scenarios in which
data loss was more substantial and nonhomogeneous.
For tests of Type I error, b1 (the fixed effect of interest)
was set to zero. For tests of power, b1 was set to .8, which
we found yielded power around 0.5 for the most powerful
methods with close-to-nominal Type I error.
We generated 100,000 datasets for testing for each of
the eight combinations (effect present/absent, between-/
7
Having only six items per condition, such as in the 12-item case, is not
uncommon in psycholinguistic research, where it is often difficult to come
up with larger numbers of suitably controlled items.
 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278 265

Table 2 Model selection analyses

Ranges for the population parameters; U(min, max) means the parameter We also considered a wide variety of LMEMs whose
was sampled from a uniform distribution with range [min, max].
random effects structure—specifically, which slopes to in-
Parameter Description Value clude—was determined through model selection. We also
b0 Grand-average intercept U(3, 3) varied the model-selection a level, i.e., the level at which
b1 Grand-average slope 0 (H0 true) or .8 (H1 slopes were tested for inclusion or exclusion, taking on
true) the values .01 and .05 as well as values from .10 to .80 in
s200 By-subject variance of S0s U(0, 3)
steps of .10.
s211 By-subject variance of S1s U(0, 3)
Our model selection techniques tested only random
qS Correlation between (S0s, S1s) U(.8, .8)
pairs slopes for inclusion/exclusion, leaving in by default the
x200 By-item variance of I0i U(0, 3) by-subject and by-item random intercepts (since that
x211 By-item variance of I1i U(0, 3) seems to be the general practice in the literature). For be-
qI Correlation between (I0i, I1i) U(.8, .8) tween-items designs, there was only one slope to be tested
pairs (by-subjects) and thus only one possible model selection
r2 Residual error U(0, 3)
algorithm. In contrast, for within-items designs, where
pmissing Proportion of missing U(.00, .05)
observations there are two slopes to be tested, a large variety of algo-
rithms are possible. We explored the model selection algo-
rithms given in Table 4, which were defined by the
direction of model selection (forward or backward) and
within-item manipulation, 12/24 items). The functions we whether slopes were tested in an arbitrary or principled
used in running the simulations and processing the results sequence. The forward algorithms began with a random-
are available in the R package simgen, which we have intercepts-only model and tested the two possible slopes
made available in the Online Appendix, along with a num- for inclusion in an arbitrary, pre-defined sequence (either
ber of R scripts using the package. The Online Appendix by-subjects slope first and by-items slope second, or vice
also contains further information about the additional R versa; these models are henceforth denoted by ‘‘FS’’ and
packages and functions used for simulating the data and ‘‘FI’’). If the p-value from the first test exceeded the mod-
running the analyses. el-selection a level for inclusion, the slope was left out of
the model and the second slope was never tested; other-
wise, the slope was included and the second slope was
Analyses
tested. The backward algorithm (‘‘BS’’ and ‘‘BI’’ models)
was similar, except that it began with a maximal model
The analyses that we evaluated are summarized in Ta-
and tested for the exclusion of slopes rather than for their
ble 3. Three of these were based on ANOVA (F1, min-F0 ,
inclusion.
and F1  F2), with individual F-values drawn from mixed-
For these same within-items designs, we also consid-
model ANOVA on the unaggregated data (e.g., using MStreat-
ered forward and backward algorithms in which the se-
ment/MStreatment-by-subject) rather than from performing re-
quence of slope testing was principled rather than
peated-measures ANOVAs on the (subject and item)
arbitrary; we call these the ‘‘best-path’’ algorithms because
means. The analyses also included LMEMs with a variety
they choose each step through the model space based on
of random effects structures and test statistics. All LMEMs
which addition or removal of a predictor leads to the best
were fit using the lmer function of the R package lme4,
next model. For the forward version, both slopes were
version 0.999375-39 (Bates et al., 2011), using maximum
tested for inclusion independently against a random-inter-
likelihood estimation.
cepts-only model. If neither test fell below the model-
There were four kinds of models with predetermined
selection a level, then the random-intercepts-only model
random effects structures: models with random intercepts
was retained. Otherwise, the slope with the strongest evi-
but no random slopes, models with within-unit random
dence for inclusion (lowest p-value) was included in the
slopes but no within-unit random intercepts, models with
model, and then the second slope was tested for inclusion
no random correlations (i.e., independent slopes and inter-
against this model. The backward best-path algorithm was
cepts), and maximal models.
the same, except that it began with a maximal model and
tested slopes for exclusion rather than for inclusion. (In
principle, one can use best-path algorithms that allow both
Table 3 forwards and backwards moves, but the space of possible
Analyses performed on simulated datasets.

Analysis Test statistics Table 4

0 0 Model selection algorithms for within-items designs.
min-F min-F
F1 F1 Model Direction Order
F1  F2 F1, F2
Maximal LMEM FS Forward By-subjects slope then by-items
t; v2LR
FI Forward By-items slope then by-subjects
LMEM, random intercepts only t; v2LR , MCMC
FB Forward ‘‘Best path’’ algorithm
LMEM, no within-unit intercepts (NWI) t; v2LR BS Backward By-subjects slope then by-items
Maximal LMEM, no random correlations (NRC) t; v2LR , MCMC BI Backward By-items slope then by-subjects
Model selection (multiple variants) t; v2LR BB Backward ‘‘Best path’’ algorithm
266 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
models considered here is so small that such an algorithm
would be indistinguishable from the forwards- or back-
wards-only variants.)
Handling nonconvergence and deriving p-values
Nonconverging LMEMs were dealt with by progres-
sively simplifying the random effects structure until con-
vergence was reached. Data from these simpler models
contributed to the performance metrics for the more com-
plex models. For example, in testing maximal models, if a
particular model did not converge and was simplified
down to a random-intercepts-only model, the p values
from that model would contribute to the performance
metrics for maximal models. This reflects the assumption
that researchers who encountered nonconvergence would
not just give up but would consider simpler models. In
other words, we are evaluating analysis strategies rather
than particular model structures.
In cases of nonconvergence, simplification of the ran-
dom effects structure proceeded as follows. For between-
items designs, the by-subjects random slope was dropped.
For within-items designs, statistics from the partially con-
verged model were inspected, and the slope associated
with smaller variance was dropped (see the Online Appen-
dix for justification of this method). In the rare (0.002%) of
cases that the random-intercepts-only model did not con-
verge, the analysis was discarded.
There are various ways to obtain p-values from LMEMs,
and to our knowledge, there is little agreement on which
method to use. Therefore, we considered three methods
currently in practice: (1) treating the t-statistic as if it were
a z statistic (i.e., using the standard normal distribution as
a reference); (2) performing likelihood ratio tests, in which
the deviance (2LL) of a model containing the fixed effect
is compared to another model without it but that is other-
wise identical in random effects structure; and (3) by Mar-
kov Chain Monte Carlo (MCMC) sampling, using the
mcmcsamp() function of lme4 with 10,000 iterations. This
is the default number of iterations used in Baayen’s
pvals.fnc() of the languageR package (Baayen, 2011).
This function wraps the function mcmcsamp(), and we
used some of its code for processing the output of mcmcs-
amp(). Although MCMC sampling is the approach recom-
mended by Baayen et al. (2008), it is not implemented in
lme4 for models containing random correlation parame-
ters. We therefore used (3) only for random-intercept-only
and no-random-correlation LMEMs.
Performance metrics
The main performance metrics we considered were
Type I error rate (the rate of rejection of H0 when it is true)
and power (the rate of failure to reject H0 when it is false).
For all analyses, the a level for testing the fixed-effect slope
(b1) was set to .05 (results were also obtained using a = .01
and a = .10, and were qualitatively similar; see online
appendix).
It can be misleading to directly compare the power of
various approaches that differ in Type I error rate, because
the power of anticonservative approaches will be inflated.
Therefore, we also calculated Power0 , a power rate cor-
rected for anticonservativity. Power0 was derived from the
empirical distribution for p-values from the simulation for
a given method where the null hypothesis was true. If the
p-value at the 5% quantile of this distribution was below
the targeted a-level (e.g., .05), then this lower value was
used as the cutoff for rejecting the null. To illustrate, note
that a method that is nominal (neither anticonservative
nor conservative) would yield an empirical distribution of
p-values for which very nearly 5% of the simulations would
obtain p-values less than .05. Now consider that a given
method with a targeted a-level of .05, 5% of the simulation
runs under the null hypothesis yielded p-values of .0217 or
lower. This clearly indicates that this method is anticonser-
vative, since more than 5% of the simulation runs had p-val-
ues less than the targeted a-level of .05. We could correct
for this anticonservativity in the power analysis by requir-
ing that a p-value from a given simulation run, to be
deemed statistically significant, must be less than .0217 in-
stead of .05. In contrast, if for a given method 5% of the runs
under the null hypothesis yielded a value of .0813 or lower,
this method would be conservative, and it would be unde-
sirable to ‘correct’ this as it would artificially make the test
seem more powerful than it actually is. Instead, for this case
we would simply require that the p-value for a given simu-
lation run be lower than .05.
Because we made minimal assumptions about the rela-
tive magnitudes of random variances, it is also of interest to
examine the performance of the various approaches as a
function the various parameters that define the space. Gi-
ven the difficulty of visualizing a multidimensional param-
eter space, we chose to visually represent performance
metrics in terms of two ‘‘critical variances’’, which were
those variances that can drive differences between treat-
ment means. As noted above, for between-item designs,  
this includes the by-item random intercept variance x200
and the by-subject random slope variance (s211 ); for with-
in-items designs, this includes the by-item and by sub-
ject-random slope variance (x211 and s211 ). We modeled
Type I error rate and power over these critical variances
using local polynomial regression fitting (the loess func-
tion in R which is wrapped by the loessPred function in
our simgen package). The span parameter for loess fitting
was set to .9; this highlights general trends throughout the
parameter space, at the expense of fine-grained detail.
Results and discussion
An ideal statistical analysis method maximizes statisti-
cal power while keeping Type I error nominal (at the stated
a level). Performance metrics in terms of Type I error,
power, and corrected power are given in Table 5 for the be-
tween-item design and in Table 6 for the within-item de-
sign. The analyses in each table are (approximately)
ranked in terms of Type I error, with analyses toward the
top of the table showing the best performance, with priority
given to better performance on the larger (24-item) dataset.
Only min-F0 was consistently at or below the stated a
level. This is not entirely surprising, because the tech-
niques that are available for deriving p-values from LMEMs
are known to be somewhat anticonservative (Baayen et al.,
2008). For maximal LMEMs, this anticonservativity was
 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
quite minor, within 1–2% of a.8 LMEMs with maximal ran-
dom slopes, but missing either random correlations or with-
in-unit random intercepts, performed nearly as well as
‘‘fully’’ maximal LMEMs, with the exception of the case
where p-values were determined by MCMC sampling. In
addition, there was slight additional anticonservativity rela-
tive to the maximal model for the models missing within-
unit random intercepts. This suggests that when maximal
LMEMs fail to converge, dropping within-unit random inter-
cepts or random correlations are both viable options for sim-
plifying the random effects structure. It is also worth noting
that F1  F2, which is known to be fundamentally biased
(Clark, 1973; Forster & Dickinson, 1976), controlled overall
Type I error rate fairly well, almost as well as maximal
LMEMs. However, whereas anticonservativity for maximal
(and near-maximal) LMEMs decreased as the data set got
larger (from 12 to 24 items), for F1  F2 it actually showed
a slight increase.
F1 alone was the worst performing method for between-
items designs, and also had an unacceptably high error rate
for within-items designs. Random-intercepts-only LMEMs
were also unacceptably anticonservative for both types of
designs, far worse than F1  F2. In fact, for within-items de-
signs, random-intercepts-only LMEMs were even worse than
F1 alone, showing false rejections 40–50% of the time at
the .05 level, regardless of whether p-values were derived
using the normal approximation to the t-statistic, the like-
lihood-ratio test, or MCMC sampling. In other words, for
within-items designs, one can obtain better generalization
by ignoring item variance altogether (F1) than by using an
LMEM with only random intercepts for subjects and items.
Fig. 2 presents results from LMEMs where the inclusion
of random slopes was determined by model selection. The
figure presents the results for the within-items design,
where a variety of algorithms were possible. Performance
for the between-items design (where there was only a sin-
gle slope to be tested) was very close to that of maximal
LMEMs, and is presented in the Online Appendix.
The figure suggests that the Type I error rate depends
more upon the algorithm followed in testing slopes than
the a-level used for the tests. Forward-stepping ap-
proaches that tested the two random slopes in an arbitrary
sequence performed poorly in terms of Type I error even at
relatively high a levels. This was especially the case for the
smaller, 12-item sets, where there was less power to detect
by-item random slope variance. In contrast, performance
was relatively sound for backward models even at rela-
tively low a levels, as well as for the ‘‘best path’’ models
regardless of whether the direction was backward or for-
ward. It is notable that these sounder data-driven ap-
proaches showed only small gains in power over
maximal models (indicated by the dashed line in the back-
ground of the figure).
8
This anticonservativity stems from underestimation of the variation
between subjects and/or items, as is suggested by generally better
performance of the maximal model in the 24- as opposed to 12-item
simulations. In Appendix, we show that as additional subjects and items are
added, the Type I error rate for LMEMs with random slopes decreases
rapidly, while for random-intercepts-only models, it actually increases.
267
From the point of view of overall Type I error rate, we
can rank the analyses for both within- and between-items
designs in order of desirability:
1. min-F0 , maximal LMEMs, ‘‘near-maximal’’ LMEMs miss-
ing within-unit random intercepts or random correla-
tions, and model selection LMEMs using backward
selection and/or testing slopes using the ‘‘best path’’
algorithm.
2. F1  F2.
3. forward-stepping LMEMs that test slopes in an arbitrary
sequence.
4. F1 and random-intercepts-only LMEMs.
It would also seem natural to draw a line separating
analyses that have an ‘‘acceptable’’ rate of false rejections
(i.e., 1–2) from those with a rate that is intolerably high
(i.e., 3–4). However, it is insufficient to consider only the
overall Type I error rate, as there may be particular prob-
lem areas of the parameter space where even the best anal-
yses perform poorly (such as when particular variance
components are very small or large). If these areas are
small, they will only moderately affect the overall error
rate. This is a problem because we do not know where
the actual populations that we study reside in this param-
eter space; it could be that they inhabit these problem
areas. It is therefore also useful to examine the perfor-
mance metrics as a function of the critical random variance
parameters that are confounded with the treatment effect,
i.e., s211 and x200 for between-items designs and s211 and x211
for within-items designs. These are given in the ‘‘heatmap’’
displays of Figs. 3–5.
Viewing Type I error rate as a function of the critical
variances, it can be seen that of models with predeter-
mined random effects structures, only min-F0 maintained
the Type I error rate consistently below the a-level
throughout the parameter space (Figs. 3 and 4). Min-F0 be-
came increasingly conservative as the relevant random ef-
fects got small, replicating Forster and Dickinson (1976).
Maximal LMEMs showed no such increasing conservativi-
ty, performing well overall, especially for 24-item datasets.
The near-maximal LMEMs also performed relatively well,
though for within-item datasets, models missing within-
unit intercepts became increasingly conservative as slope
variances got small. Random-intercepts-only LMEMs de-
graded extremely rapidly as a function of random slope
parameters; even at very low levels of random-slope vari-
ability, the Type I error rate was unacceptably high.
In terms of Type I error, the average performance of the
widely adopted F1  F2 approach is comparable to that of
maximal LMEMs (slightly less anti-conservative for 12
items, slightly more anti-conservative for 24 items). But
in spite of this apparent good performance, the heatmap
visualization indicates that the approach is fundamentally
unsound. Specifically, F1  F2 shows both conservative and
anticonservative tendencies: as slopes get small, it be-
comes increasingly conservative, similar to min-F0 ; as
slopes get large, it becomes increasingly anticonservative,
similar to random-intercepts-only LMEMs, though to a les-
ser degree. This increasing anticonservativity reflects the
fact that (as noted in the introduction) subject random
268 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278

Table 5
Performance metrics for between-items design. Power0 = corrected power. Note that corrected power for random-intercepts-only MCMC is not included
because the low number of MCMC runs (10,000) combined with the high Type I error rate did not provide sufficient resolution.

Type I Power Power0

Nitems
12 24 12 24 12 24
Type I: Error at or near a = .05
min-F0 .044 .045 .210 .328 .328 .328
LMEM, maximal, v2LR .070 .058 .267 .364 .223 .342
LMEM, no random correlations, v2LR a .069 .057 .267 .363 .223 .343
LMEM, no within-unit intercepts, v2LR a .081 .065 .288 .380 .223 .342
LMEM, maximal, t .086 .065 .300 .382 .222 .343
LMEM, no random correlations, t .086 .064 .300 .382 .223 .343
LMEM, no within-unit intercepts, ta .100 .073 .323 .401 .222 .342
F1  F2 .063 .077 .252 .403 .224 .337
Type I: Error far exceeding a = .05
LMEM, random intercepts only, v2LR .102 .111 .319 .449 .216 .314
LMEM, random intercepts only, t .128 .124 .360 .472 .472 .314
LMEM, no random correlations, MCMCa .172 .192 .426 .582
LMEM, random intercepts only, MCMC .173 .211 .428 .601
F1 .421 .339 .671 .706 .134 .212
a
Performance is sensitive to coding of the predictor (see the Online Appendix); simulations use deviation coding.

Table 6
Performance metrics for within-items design. Note that corrected power for random-intercepts-only MCMC is not included because the low number of MCMC
runs (10,000) combined with the high Type I error rate did not provide sufficient resolution.

Type I Power Power0

Nitems
12 24 12 24 12 24
Type I: Error at or near a = .05
min-F0 .027 .031 .327 .512 .327 .512
LMEM, maximal, v2LR .059 .056 .460 .610 .433 .591
LMEM, no random correlations, v2LR a .059 .056 .461 .610 .432 .593
LMEM, no within-unit intercepts, v2LR a .056 .055 .437 .596 .416 .579
LMEM, maximal, t .072 .063 .496 .629 .434 .592
LMEM, no random correlations, t .072 .062 .497 .629 .432 .593
LMEM, no within-unit intercepts, ta .070 .064 .477 .620 .416 .580
F1  F2 .057 .072 .440 .643 .416 .578
Type I: Error far exceeding a = .05
F1 .176 .139 .640 .724 .345 .506
LMEM, no random correlations, MCMCa .187 .198 .682 .812
LMEM, random intercepts only, MCMC .415 .483 .844 .933
LMEM, random intercepts only, v2LR .440 .498 .853 .935 .379 .531
LMEM, random intercepts only, t .441 .499 .854 .935 .379 .531
a
Performance is sensitive to coding of the predictor (see the Online Appendix); simulations use deviation coding.

slopes are not accounted for in the F2 analysis, nor are item
random slopes accounted for in the F1 analysis (Clark,
1973). The fact that both F1 and F2 analyses have to pass
muster keeps this anti-conservativity relatively minimal
as long as subject and/or item slope variances are not large,
but the anti-conservativity is there nonetheless.
Fig. 5 indicates that despite the overall good perfor-
mance of some of the data-driven approaches, these mod-
els become anticonservative when critical variances are
small and nonzero.9 This is because as slope variances be-
9
Some readers might wonder why the heatmaps for these models show
apparent anticonservative behavior at the bottom left corner, where the
slopes are zero, since performance at this point should be close to the
nominal level. This is an artifact of the smoothing used in generating the
heatmaps, which aids in the detection of large trends at the expense of
small details. The true underlying pattern is that the anticonservativity
ramps up extremely quickly from the point where the slope variances are
zero to reach its maximal value, before beginning to gradually taper away
(as the variances become more reliably detected).
come small, slopes are less likely to be kept in the model.
This anticonservativity varies considerably with the size of
the dataset and the algorithm used (and of course should
also vary with the a-level, not depicted in the figure). The
anticonservativity is present to a much stronger degree for
the 12-item than for the 24-item datasets, reflecting the fact
that the critical variances are harder to detect with smaller
data sets. The forward stepping models that tested slopes
in an arbitrary sequence were the most anticonservative
by far. The model testing the by-subject slope first per-
formed most poorly when that slope variance was small
and the by-item slope variance was large, because in such
cases the algorithm would be likely to stop at a random-
intercepts-only model, and thus would never test for the
inclusion of the by-item slope. By the same principles, the
forward model that tested the by-item slope first showed
worst performance when the by-item slope variance was
small and the by-subject slope variance large.
 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278 269

BB BI BS FB FI FS

Type I Error Power Corrected Power

.20 .70 .70
.18 .64 .64
.16 .59 .59
.13 .53 .53
.11 .48 .48
.09 .42 .42
.07 .37 .37
.04 .31 .31
.02 .26 .26
.00 .20 .20
.01
.05
.10
.20
.30
.40
.50
.60
.70
.80

.01
.05
.10
.20
.30
.40
.50
.60
.70
.80

.01
.05
.10
.20
.30
.40
.50
.60
.70
.80
α α α

.20 .70 .70

.18 .64 .64
.16 .59 .59
.13 .53 .53
.11 .48 .48
.09 .42 .42
.07 .37 .37
.04 .31 .31
.02 .26 .26
.00 .20 .20
.01
.05
.10
.20
.30
.40
.50
.60
.70
.80

.01
.05
.10
.20
.30
.40
.50
.60
.70
.80

.01
.05
.10
.20
.30
.40
.50
.60
.70
.80
α α α

Fig. 2. Performance of model selection approaches for within-items designs, as a function of selection algorithm and a level for testing slopes. The p-values
for all LMEMs in the figure are from likelihood-ratio tests. Top row: 12 items; bottom row: 24 items. BB = backwards, ‘‘best path’’; BI = backwards, item-
slope first; BS = backwards, subject-slope first; FB = forwards, ‘‘best path’’; FI = forwards, item-slope first; FS = forwards, subject-slope first.

In sum, insofar as one is concerned about drawing con- showed a considerable power advantage for within-item
clusions likely to generalize across subjects and items, only designs, with corrected power levels for a = .05 (.433 and
min-F0 and maximal LMEMs can be said to be fundamen- .592) that were 16–32% higher than the uncorrected power
tally sound across the full parameter space that we sur- values for min-F0 (.327 and .512). This additional power of
veyed. F1-only and random-intercepts-only LMEMs are maximal LMEMs cannot be attributed to the inclusion of
fundamentally flawed, as are forward stepping models that cases where slopes were removed due to nonconvergence,
follow an arbitrary sequence, especially in cases with few since in the worst case (the within-item dataset with 12
observations. All other LMEMs with model selection items) virtually all simulation runs (99.613%) converged
showed small amounts of anticonservativity when slopes with the maximal structure.
were small, even when the slopes were tested in a princi- Note that the corrected power for random-intercepts-
pled sequence; however, this level of anticonservativity is only LMEMs was actually below that of maximal LMEMs
probably tolerable for reasonably-sized datasets (so long (between-items: .216 and .314 for 12 and 24 items respec-
as there is not an extensive amount of missing data; see tively; within-items: .380 and .531). This means that most
the Online Appendix). The widely-used F1  F2 approach of the apparent additional power of maximal LMEMs over
is flawed as well, but may be acceptable in cases where min-F0 is real, while most of the apparent power of ran-
maximal LMEMs are not applicable. The question now is dom-intercepts-only LMEMs is, in fact, illusory.
which of these analyses best maximizes power (Tables 5 Our results show that it is possible to use a data-driven
and 6; Figs. 3 and 4). approach to specifying random effects in a way that mini-
Overall, maximal LMEMs showed greater power than mizes Type I error, especially with ‘‘best-path’’ model
min-F0 . When corrected for their slight anticonservativity, selection. However, the power advantage of this approach
maximal LMEMs exhibited power that was between 4% for continuous data, even when uncorrected for anticon-
and 6% higher for the between-items design than the servativity, is very small. In short, data-driven approaches
uncorrected values for min-F0 . Although there does not can produce reasonable results, but their very small benefit
seem to be a large overall power advantage to using max- to power may not be worth the additional uncertainty they
imal LMEMs for between-item designs, the visualization of introduce as compared to a design-driven approach.
power in terms of the critical variances (Fig. 3) suggests The above analyses suggest that maximal LMEMs are in
that the power advantage increases slightly as the critical no way conservative, at least for the analysis of continuous
variances become small. In contrast, maximal LMEMs data. To dispel this suspicion entirely it is illustrative to
270 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278

min−F' LMEM, Max LMEM, Max−NRC LMEM, Max−NWI

2.5 2.5 2.5 2.5

2.0 2.0 2.0 2.0

ω002

ω002

ω002

ω002
1.5 1.5 1.5 1.5

1.0 1.0 1.0 1.0

0.5 0.5 0.5 0.5

0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112 τ112

F1+F2 LMEM, RI−only F1 LMEM, Model Selection

2.5 2.5 2.5 2.5

2.0 2.0 2.0 2.0

ω002

ω002

ω002

ω002
1.5 1.5 1.5 1.5

1.0 1.0 1.0 1.0

0.5 0.5 0.5 0.5

0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112 τ112

min−F' LMEM, Max LMEM, Max−NRC LMEM, Max−NWI

2.5 2.5 2.5 2.5

2.0 2.0 2.0 2.0

ω002

ω002

ω002

ω002
1.5 1.5 1.5 1.5

1.0 1.0 1.0 1.0

0.5 0.5 0.5 0.5

0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112 τ112

F1+F2 LMEM, RI−only F1 LMEM, Model Selection

2.5 2.5 2.5 2.5

2.0 2.0 2.0 2.0

ω002

ω002

ω002

ω002

1.5 1.5 1.5 1.5

1.0 1.0 1.0 1.0

0.5 0.5 0.5 0.5

0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112 τ112

Fig. 3. Type I error (top two rows) and Power (bottom two rows) for between-items design with 24 items, as a function of by-subject random slope variance
s211 and by-item random intercept variance x200 . The p-values for all LMEMs in the figure are from likelihood-ratio tests. All model selection approaches in
the figure had a = .05 for slope inclusion. The heatmaps from the 12-item datasets show similar patterns, and are presented in the Online Appendix.

consider the performance of maximal LMEMs in an ex-
treme case for which their power should be at its absolute
worst: namely, when the random slope variance is negligi-
ble, such that the underlying process is best described by a
random-intercepts-only LMEM. Comparing the perfor-
mance of maximal LMEMs to random-intercepts-only
LMEMs in this circumstance can illustrate the highest
‘‘cost’’ that one could possibly incur by using maximal
LMEMs in the analysis of continuous data. Maximal LMEMs
might perform badly in this situation because they overfit
the underlying generative process, loosely akin to assum-
ing too few degrees of freedom for the analysis rather than
too many. But they might also perform tolerably well to
the extent that the estimation procedure for LMEMs can
detect that a random effect parameter is effectively zero.
In this situation, it is additionally informative to compare
the power of maximal LMEMs not only to random-inter-
cepts-only LMEMs, but also to that of min-F0 , since min-F0
is generally regarded as conservative, as well as to that of
F1  F2, since F1  F2 is generally regarded as sufficiently
powerful. If maximal LMEMs perform better than min-F0
and at least as well as F1  F2, then that would strongly ar-
gue against the idea that maximal LMEMs are unduly con-
servative. To address this, we conducted an additional set
of simulations, once again using the data-generating
parameters in Table 2, except that we set all random-slope
 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278 271

min−F' LMEM, Max LMEM, Max−NRC LMEM, Max−NWI

2.5 2.5 2.5 2.5

2.0 2.0 2.0 2.0

ω112

ω112

ω112

ω112
1.5 1.5 1.5 1.5

1.0 1.0 1.0 1.0

0.5 0.5 0.5 0.5

0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112 τ112

F1+F2 LMEM, RI−only F1

2.5 2.5 2.5

2.0 2.0 2.0

ω112

ω112

ω112
1.5 1.5 1.5

1.0 1.0 1.0

0.5 0.5 0.5

0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112

min−F' LMEM, Max LMEM, Max−NRC LMEM, Max−NWI

2.5 2.5 2.5 2.5

2.0 2.0 2.0 2.0

ω112

ω112

ω112

ω112
1.5 1.5 1.5 1.5

1.0 1.0 1.0 1.0

0.5 0.5 0.5 0.5

0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112 τ112

F1+F2 LMEM, RI−only F1

2.5 2.5 2.5

2.0 2.0 2.0

ω112

ω112

ω112

1.5 1.5 1.5

1.0 1.0 1.0

0.5 0.5 0.5

0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112

Fig. 4. Type I error (top three rows) and power (bottom three rows) for within-items design with 24 items, as a function of by-subject random slope
variance s211 and by-item random slope variance x211 . The p-values for all LMEMs in the figure are from likelihood-ratio tests. All model selection approaches
in the figure had a = .05 for slope inclusion. The heatmaps from the 12-item datasets show similar patterns, and are presented in Online Appendix.

variances to 0, so that the model generating the data was a
random-intercepts-only LMEM; also, we varied the true
fixed effect size continuously from 0 to 0.8.
The results were unambiguous (Fig. 6): even in this
worst case scenario for their power, maximal LMEMs con-
sistently showed higher power than min-F0 and even F1 
F2; indeed, for within-items designs, they far outstripped
the performance of F1  F2, an approach whose power is
rarely questioned. For between-items designs, maximal
LMEMs incurred a negligible cost relative to random-inter-
cepts-only LMEMs, while for within-items designs, there
was only a minor cost that diminished as the number of
items increased. Overall, the cost/benefit analysis favors
maximal LMEMs over other approaches. Note that over
the four different sets of simulations in Fig. 6, our maximal
LMEM analysis procedure stepped down to a random-
intercept model (due to convergence problems) on less
than 3% of runs. Thus, these results indicate good perfor-
mance of the estimation algorithm when random slope
variances are zero.
The case of within-item designs with few items showed
the biggest difference in power between random-inter-
cepts-only models and maximal LMEMs. The size of this
difference indicates the maximum benefit that could be
obtained, in principle, by using a data-driven approach.
However, in practice, the ability of a data-driven approach
to detect random slope variation diminishes as the dataset
gets small. In other words, it is in just this situation that
272 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278

Backward, Best Path Backward, Subjects First Backward, Items First Forward, Best Path Forward, Subjects First Forward, Items First
12 items 12 items 12 items 12 items 12 items 12 items

2.5

2.5

2.5

2.5

2.5

2.5
2.0

2.0

2.0

2.0

2.0

2.0
ω112

ω112

ω112

ω112

ω112

ω112
1.5

1.5

1.5

1.5

1.5

1.5
1.0

1.0

1.0

1.0

1.0

1.0
0.5

0.5

0.5

0.5

0.5

0.5
0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112 τ112 τ112 τ112

Backward, Best Path Backward, Subjects First Backward, Items First Forward, Best Path Forward, Subjects First Forward, Items First
24 items 24 items 24 items 24 items 24 items 24 items
2.5

2.5

2.5

2.5

2.5

2.5
2.0

2.0

2.0

2.0

2.0

2.0
ω112

ω112

ω112

ω112

ω112

ω112
1.5

1.5

1.5

1.5

1.5

1.5
1.0

1.0

1.0

1.0

1.0

1.0
0.5

0.5

0.5

0.5

0.5

0.5
0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112 τ112 τ112 τ112

Backward, Best Path Backward, Subjects First Backward, Items First Forward, Best Path Forward, Subjects First Forward, Items First
12 items 12 items 12 items 12 items 12 items 12 items
2.5

2.5

2.5

2.5

2.5

2.5
2.0

2.0

2.0

2.0

2.0

2.0
ω112

ω112

ω112

ω112

ω112

ω112
1.5

1.5

1.5

1.5

1.5

1.5
1.0

1.0

1.0

1.0

1.0

1.0
0.5

0.5

0.5

0.5

0.5

0.5
0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112 τ112 τ112 τ112

Backward, Best Path Backward, Subjects First Backward, Items First Forward, Best Path Forward, Subjects First Forward, Items First
24 items 24 items 24 items 24 items 24 items 24 items
2.5

2.5

2.5

2.5

2.5

2.5
2.0

2.0

2.0

2.0

2.0

2.0
ω112

ω112

ω112

ω112

ω112

ω112
1.5

1.5

1.5

1.5

1.5

1.5
1.0

1.0

1.0

1.0

1.0

1.0
0.5

0.5

0.5

0.5

0.5

0.5
0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5 0.5 1.0 1.5 2.0 2.5
τ112 τ112 τ112 τ112 τ112 τ112

Fig. 5. Type I error (top two rows) and power (bottom two rows) for data-driven approaches on within-items data, as a function of by-subject random slope
variance s211 and by-item random slope variance x211 . The p-values for all LMEMs in the figure are from likelihood-ratio tests. All approaches in the figure
tested random slopes at a = .05.

the power for detecting random slope variation is at its ity of the test. Including only those random components
worst (see Fig. 4 and the 12-item figures in Appendix); that reduce noise but not those that are confounded with
thus, in this case, the payoff in terms of statistical power the effect of interest will lead to drastically anticonserva-
over the maximal approach does not outweigh the addi- tive behavior, as seen by random-intercepts-only LMEMs,
tional risk of anticonservativity. which had the worst Type I error rates overall.
In sum, our investigation suggests that for confirmatory
hypothesis testing, maximal LMEMs yield nearly optimal General discussion
performance: they were better than all other approaches
except min-F0 at maintaining the Type I error rate near Recent years have witnessed a surge in the popularity of
the nominal level. Furthermore, unlike F1  F2 and certain LMEMs in psycholinguistics and related fields, and this
model selection approaches, maximal LMEMs held the er- growing excitement is well deserved, given the great flex-
ror rate relatively close to nominal across the entirety of ibility of LMEMs and their ability to model the generative
the parameter space. And once corrected for anticonserv- process underlying one’s data. However, there has been
ativity, no other technique exceeded the power of maximal insufficient appreciation of how choices about random ef-
LMEMs. ‘‘Best-path’’ model selection also kept Type I errors fect structure impact generalizability, and no accepted
largely at bay, but it is not clear whether they lead to gains standards for the use of LMEMs in confirmatory hypothesis
in statistical power. testing are currently in place. We have emphasized that
The near-optimal performance of maximal models may specifying random effects in LMEMs involves essentially
be explained as follows. Including the random variances the same principles as selecting an analysis technique from
that could potentially be confounded with the effect of the menu of traditional ANOVA-based options. The stan-
interest is critical to controlling the Type I error rate, by dard for ANOVA has been to assume that if an effect exists,
ensuring that the assumption of conditional independence subjects and/or items will vary in the extent to which they
is met. Including random variances that are not con- show that effect. This is evident in the fact that researchers
founded with that effect (e.g., within-unit intercept vari- using ANOVA have tended to assume the maximal (or
ances) is not critical for reducing Type I error, but near-maximal) random effects structure justified by the
nonetheless reduces noise and thus increases the sensitiv- design.
 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278 273

Between Item, N=12 Between Item, N=24

1.0 Maximal 1.0 Maximal
Random Intercept Random Intercept
min−F' min−F'
F1xF2 F1xF2
0.8 0.8

0.6 0.6

power
power

0.4 0.4

0.2 0.2

0.0 0.0
(.000, .042]
(.042, .084]
(.084, .126]
(.126, .168]
(.168, .211]
(.211, .253]
(.253, .295]
(.295, .337]
(.337, .379]
(.379, .421]
(.421, .463]
(.463, .505]
(.505, .547]
(.547, .589]
(.589, .632]
(.632, .674]
(.674, .716]
(.716, .758]
(.758, .800]

(.000, .042]
(.042, .084]
(.084, .126]
(.126, .168]
(.168, .211]
(.211, .253]
(.253, .295]
(.295, .337]
(.337, .379]
(.379, .421]
(.421, .463]
(.463, .505]
(.505, .547]
(.547, .589]
(.589, .632]
(.632, .674]
(.674, .716]
(.716, .758]
(.758, .800]
effect effect

Within Item, N=12 Within Item, N=24

1.0 Maximal 1.0 Maximal
Random Intercept Random Intercept
min−F' min−F'
F1xF2 F1xF2
0.8 0.8

0.6 0.6
power
power

0.4 0.4

0.2 0.2

0.0 0.0
(.000, .042]
(.042, .084]
(.084, .126]
(.126, .168]
(.168, .211]
(.211, .253]
(.253, .295]
(.295, .337]
(.337, .379]
(.379, .421]
(.421, .463]
(.463, .505]
(.505, .547]
(.547, .589]
(.589, .632]
(.632, .674]
(.674, .716]
(.716, .758]
(.758, .800]

(.000, .042]
(.042, .084]
(.084, .126]
(.126, .168]
(.168, .211]
(.211, .253]
(.253, .295]
(.295, .337]
(.337, .379]
(.379, .421]
(.421, .463]
(.463, .505]
(.505, .547]
(.547, .589]
(.589, .632]
(.632, .674]
(.674, .716]
(.716, .758]
(.758, .800]

effect effect

Fig. 6. Statistical power for maximal LMEMs, random-intercepts-only LMEMs, min-F0 , and F1  F2 as a function of effect size, when the generative model
underlying the data is a random-intercepts-only LMEM.

Our survey of various analysis strategies for confirma-
tory hypothesis testing on data with crossed random ef-
fects clearly demonstrates that the strongest contenders
in avoiding anti-conservativity are maximal LMEMs and
min-F0 (see below for related discussion of data-driven ap-
proaches). These were the only approaches that consis-
tently showed nominal or near-nominal Type I error
rates throughout the parameter space. Although maximal
LMEMs showed some minor anticonservativity, our analy-
sis has uncovered a hidden advantage of maximal LMEMs
over ANOVA-based approaches. This advantage was evi-
dent in the diverging performance of these approaches as
a function of the critical variances (the variances con-
founded with the effect of interest). As these variances be-
came small, maximal LMEMs showed better retention of
their power relative to ANOVA-based approaches, which
became increasingly conservative. In the limit, when the
generative process did not include any random slope vari-
ation, the power of maximal LMEMs substantially out-
stripped that of ANOVA-based approaches, even F1  F2.
An apparent reason for this power advantage of maximal
LMEMs is their simultaneous accommodation of by-subject
and by-item random variation. ANOVA-based approaches
are based on two separately calculated statistics, one for
274 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
subjects and one for items, each of which does not control
for the variance due to the other random factor. Specifi-
cally, the prominent decrease in power for ANOVA-based
approaches when critical random variances are small could
be due to the fact that separate F1  F2 analyses cannot dis-
tinguish between random intercept variation on the one
hand and residual noise on the other: by-item random
intercept variation is conflated with trial-level noise in
the F1 analysis, and by-subject random intercept variation
is conflated with trial-level noise in the F2 analysis. Maxi-
mal LMEMs do not suffer from this problem.
The performance of LMEMs depended strongly on
assumptions about random effects. This clearly implies
that researchers who wish to use LMEMs need to be more
attentive both to how they specify random effects in their
models, and to the reporting of their modeling efforts.
Throughout this article, we have argued that for confirma-
tory analyses, a design-driven approach is preferable to a
data-driven approach for specifying random effects. By
using a maximal model, one adheres to the longstanding
(and eminently reasonable) assumption that if an effect ex-
ists, subjects and/or items will most likely vary in the ex-
tent to which they show that effect, whether or not that
variation is actually detectable in the sample. That being
said, it seems likely that effect variation across subjects
and/or items differs among research areas. Perhaps there
are even effects for which such variation is effectively neg-
ligible. In such cases, one might argue that a maximal
LMEM is ‘‘overfitting’’ the data, which is detrimental to
power. However, our simulations (Fig. 6) show that maxi-
mal LMEMs would not be unreasonably conservative in
such cases, at least for continuous data; indeed, they are
far more powerful than even F1  F2.
Many researchers have used a data-driven approach to
determining the random effects structure associated with
confirmatory analysis of a fixed effect of particular inter-
est.10 Our simulations indicate that it is possible to obtain
reasonable results with such an approach, if generous crite-
ria for the inclusion of random effects are used. However, it
is important to bear in mind that data-driven approaches al-
ways imply some sort of tradeoff between Type-I and Type-II
error probability, and that it is difficult to precisely quantify
the tradeoff that one has taken. It is partly for this reason
that the use of data-driven approaches is controversial, even
among statistical experts (Bolker et al., 2009; see also Har-
rell, 2001 for more general concerns regarding model selec-
tion). Our results show that data-driven approaches yield
varying performance in terms of Type I error depending on
the criteria and algorithm used, the size of the dataset, the
extent of missing data, and the design of the study (which
determines the number of random effect terms that need
to be tested). It may be difficult for a reader lacking access
to the original data to quantify the resulting Type I/II error
10
We note here that arguments for data-driven effects to random-effects
structure have been made within the ANOVA literature as well (e.g.,
Raaijmakers, Schrijnemakers, & Gremmen, 1999), and that these arguments
have not won general acceptance within the general psycholinguistics
community. Furthermore, part of the appeal of those arguments was the
undue conservativity of min-F0 ; since maximal LMEMs do not suffer from
this problem, we take the arguments for data-driven random-effects
specification with them to be correspondingly weaker.
tradeoff, even when the inclusion criteria are known. Still,
there are situations in which data-driven approaches may
be justifiable, such as when the aims are not fully confirma-
tory, or when one experiences severe convergence problems
(see below).
Overall, our analysis suggests that, when specifying ran-
dom effects for hypothesis testing with LMEMs, research-
ers have been far too concerned about overfitting the data,
and not concerned enough about underfitting the design.
In fact, it turns out overfitting the data with a maximal
model has only minimal consequences for Type I error
and power—at least for the simple designs for typical psy-
cholinguistic datasets considered here—whereas underfit-
ting the design can incur levels of anticonservativity
ranging anywhere from minor (best-path model selection)
to extremely severe (random-intercepts-only LMEMs) with
little real benefit to power. In the extreme, random-inter-
cepts-only models have the worst generalization perfor-
mance of any approach to date when applied to
continuous data with within-subjects and within-item
manipulations. This goes to show that, for such designs,
crossing of random by-subject and by-item intercepts
alone is clearly not enough to ensure proper generalization
of experimental treatment effects (a misconception that is
unfortunately rather common at present). In psycholin-
guistics, there are few circumstances in which we can
know a priori that a random-intercepts-only model is truly
justified (see further below). Of course, if one wishes to
emphasize the absence of evidence for a given effect, there
could be some value in demonstrating that the effect is sta-
tistically insignificant even when a random-intercepts-
only model is applied.
Our general argument applies in principle to the analy-
sis of non-normally distributed data (e.g., categorical or
count data) because observational dependencies are
mostly determined by the design of an experiment and
not by the particular type of data being analyzed. However,
practical experience with fitting LMEMs to datasets with
categorical response variables, in particular with mixed lo-
git models, suggests more difficulty in getting maximal
models to converge. There are at least three reasons for
this. First, the estimation algorithms for categorical re-
sponses differ from the more developed procedures in esti-
mating continuous data. Second, observations on a
categorical scale (e.g., whether a response is accurate or
inaccurate) typically carry less information about parame-
ters of interest than observations on a continuous scale
(e.g., response time). Third, parameter estimation for any
logistic model is challenging when the underlying proba-
bilities are close to the boundaries (zero or one), since
the inverse logit function is rather flat in this region. So
although our arguments and recommendations (given be-
low) still apply in principle, they might need to be modi-
fied for noncontinuous cases. For example, whereas we
observed for continuous data that cases in which the ran-
dom effect structure ‘‘overfit’’ the data had minimal impact
on performance, this might not be the case for categorical
data, and perhaps data-driven strategies would be more
justifiable. In short, although we maintain that design-dri-
ven principles should govern confirmatory hypothesis test-
ing on any kind of data, we acknowledge that there is a
 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
pressing need to evaluate how these principles can be best
applied to categorical data.
In our investigation we have only looked at a very simple
one-factor design with two levels. However, we see no rea-
son why our results would not generalize to more complex
designs. The principles are the same in higher-order de-
signs as they are for simple one-factor designs: any main ef-
fect or interaction for which there are multiple
observations per subject or item can vary across these units,
and, if this dependency is not taken into account, the p-val-
ues will be biased against the null hypothesis.11 The main
difference is that models with maximal random effects struc-
ture would be less likely to converge as the number of with-
in-unit manipulations increases. In the next section, we offer
some guidelines for how to cope with nonconverging models.
Producing generalizable results with LMEMs: Best practices
Our theoretical analyses and simulations lead us to the
following set of recommended ‘‘best practices’’ for the use
of LMEMs in confirmatory hypothesis testing. It is impor-
tant to be clear that our recommendations may not apply
to situations where the overarching goal is not fully confir-
matory. Furthermore, we offer these not as the best possi-
ble practices—as our understanding of these models is still
evolving—but as the best given our current level of
understanding.
Identifying the maximal random effects structure
Because the same principles apply for specifying subject
and item random effects, to simplify the exposition in this
section we will only talk about ‘‘by-unit’’ random effects,
where ‘‘unit’’ stands in for the sampling unit under consid-
eration (subjects or items). As we have emphasized
throughout this paper, the same considerations come into
play when specifying random effects as when choosing
from the menu of traditional analyses. So the first question
to ask oneself when trying to specify a maximal LMEM is:
which factors are within-unit, and which are between? If a
factor is between-unit, then a random intercept is usually
sufficient. If a factor is within-unit and there are multiple
observations per treatment level per unit, then you need
a by-unit random slope for that factor. The only exception
to this rule is when you only have a single observation for
every treatment level of every unit; in this case, the ran-
dom slope variance would be completely confounded with
trial-level error. It follows that a model with a random
slope would be unidentifiable, and so a random intercept
would be sufficient to meet the conditional independence
assumption. For datasets that are unbalanced across the
levels of a within-unit factor, such that some units have
very few observations or even none at all at certain levels
of the factor, one should at least try to estimate a random
slope (but see the caveats below about how such a situa-
11
To demonstrate this, we conducted Monte Carlo simulation of a 24-
subject, 24-item 2  2 within/within experiment with main fixed effects,
no fixed interaction, and random by-subject and by-item interactions.
When analyzed with random-intercepts-only LMEMs, we found a Type I
error rate of .69; with maximal LMEMs the Type I error rate was .06. A
complete report of these simulations appears in the Online Appendix.
275
tion may contribute to nonconvergence). Finally, in cases
where there is only a single observation for every unit, of
course, not even a random intercept is needed (one can just
use ordinary regression as implemented in the R functions
lm() and glm()).
The same principles apply to higher-order designs
involving interactions. In most cases, one should also have
by-unit random slopes for any interactions where all factors
comprising the interaction are within-unit; if any one factor
involved in the interaction is between-unit, then the ran-
dom slope associated with that interaction cannot be esti-
mated, and is not needed. The exception to this rule, again,
is when you have only one observation for every subject in
every cell (i.e., unique combination of factor levels). If some
of the cells for some of your subjects have only one or zero
observations, you should still try to fit a random slope.
Random effects for control predictors
One of the most compelling aspects of mixed-effects
models is the ability to include almost any control predic-
tor—by which we mean a property of an experimental trial
which may affect the response variable but is not of theo-
retical interest in a given analysis—desired by the research-
er. In principle, including control variables in an analysis
can rule out potential confounds and increase statistical
power by reducing residual noise. Given the investigations
in the present paper, however, the question naturally
arises: in order to guard against anti-conservative infer-
ence about a predictor X of theoretical interest, do we need
by-subject and by-item random effects for all our control
predictors C as well? Suppose, after all, if there is no under-
lying fixed effect of C but there is a random effect of C—
could this create anti-conservative inference in the same
way as omitting a random effect of X in the analysis could?
To put this issue in perspective via an example, Kuperman,
Bertram, and Baayen (2010) include a total of eight main
effects in an LME analysis of fixation durations in Dutch
reading; for the interpretation of each main effect, the
other seven may be thought of as serving as controls. Fit-
ting eight random effects, plus correlation terms, would re-
quire estimating 72 random effects parameters, 36 by-
subject and 36 by item. One would likely need a huge data-
set to be able to estimate all the effects reliably (and one
must also not be in any hurry to publish, for even with
huge amounts of data such models can take extremely long
to converge).
To our knowledge, there is little guidance on this issue
in the existing literature, and more thorough research is
needed. Based on a limited amount of informal simulation,
however (reported in the Online Appendix), we propose
the working assumption that it is not essential for one to
specify random effects for control predictors to avoid anti-
conservative inference, as long as interactions between the
control predictors and the factors of interest are not pres-
ent in the model (or justified by the data). Once again,
we emphasize the need for future research on this impor-
tant issue.
Coping with failures to converge
It is altogether possible and unfortunately common that
the estimation procedure for LMEMs will not converge
276 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
with the full random-effects specification. In our experi-
ence, the likelihood that a model will converge depends
on two factors: (1) the extent to which random effects in
the model are large, and (2) the extent to which there
are sufficient observations to estimate the random effects.
Generally, as the sizes of the subject and item samples
grow, the likelihood of convergence will increase. Of
course, one does not always have the luxury of using many
subjects and items. And, although the issue seems not to
have been studied systematically, it is our impression that
fitting maximal LMEMs is less often successful for categor-
ical data than for continuous data.
It is important, however, to resist the temptation to step
back to random-intercepts-only models purely on the
grounds that the maximal model does not converge. When
the maximal LMEM does not converge, the first step should
be to check for possible misspecifications or data problems
that might account for the error. It may also help to use
standard outlier removal methods and to center or sum-
code the predictors. In addition, it may sometimes be effec-
tive to increase the maximum number of iterations in the
estimation procedure.
Once data and model specification problems have been
eliminated, the next step is to ask what simplification of
one’s model’s random-effects structure is the most defensi-
ble given the goals of one’s analysis. In the common case
where one is interested in a minimally anti-conservative
evaluation of the strength of evidence for the presence of
an effect, our results indicate that keeping the random slope
for the predictor of theoretical interest is important: a max-
imal model with no random correlations or even missing
within-unit random intercepts is preferable to one missing
the critical random slopes. Our simulations suggest that
removing random correlations might be a good strategy,
as this model performed similarly to maximal LMEMs.12
However, when considering this simplification strategy,
it is important to first check whether the nonconvergence
might be attributable to the presence of a few subjects (or
items) with small numbers of observations in particular
cells. If this is the case, it might be preferable to remove
(or replace) these few subjects (or items) rather than to re-
move an important random slope from the model. For exam-
ple, Jaeger et al. (2011, Section 3.3) discuss a case involving
categorical data in which strong evidence for random slope
variation was present when subjects with few observations
were excluded, but not when they were kept in the data set.
For more complex designs, of course, the number of pos-
sible random-effects structures proliferate. Research is
needed to evaluate the various possible strategies that
one could follow when one cannot fit a maximal model.
Both our theoretical analysis and simulations suggest a
general rule of thumb: for whatever fixed effects are of crit-
ical interest, the corresponding random effects should be
present in that analysis. For a study with multiple fixed ef-
fects of theoretical interest, and for which a model includ-
12
We remind the reader that the no-correlation and RS-only models are
sensitive to the coding used for the predictor; our theoretical analysis and
simulation results indicate that deviation coding is generally preferable to
treatment coding; see the Online Appendix.
ing random effects for all these key effects does not
converge, separate analyses can be pursued. For example,
in a study where confirmatory analysis of fixed effects for
X1 and X2 is desired, two separate analyses may be in or-
der—one with (at least) random slopes for X1 to test the evi-
dence for generalization of X1, and another with (at least)
random slopes for X2 to test X2. (One would typically still
want to include the fixed effects for X1 and X2 in both mod-
els, of course, for the established reason that multiple
regression reveals more information than two separate
regressions with single predictors; and see also the previ-
ous section on random effects for control predictors.)
One fallback strategy for coping with severe conver-
gence problems is to use a data-driven approach, building
up in a principled way from a very simple model (e.g., a
model with all fixed effects but only a single by-subjects
random intercept, or even no random effects at all). Our
simulations indicate that it is possible to use forward mod-
el selection in a way that guards against anticonservativity
if, at each step, one tests for potential inclusion all random
effects not currently in the model, and include any that
pass at a relatively liberal a-level (e.g., .20; see Method sec-
tion for further details about the ‘‘best path’’ algorithm).
We warn the reader that, with more complex designs,
the space of possible random-effects specifications is far ri-
cher than typically appreciated, and it is easy to design a
model-selection procedure that fails to get to the best
model. As just one example, in a ‘‘within’’ 2  2 design
with factors A and B, it is possible to have a model with a
random interaction term but no random main-effect
slopes. If one is interested in the fixed-effect interaction
but only tests for a random interaction if both main-effect
random slopes are already in the model, then if the true
generative process underlying the data has a large random
interaction but negligible random main-effect slopes then
forward model selection will be badly anti-conservative.
Thus it is critical to report the modeling strategy in detail
so that it can be properly evaluated.
A final recommendation is based on the fact that maxi-
mal LMEMs will be more likely to converge when the ran-
dom effects are large, which is exactly the situation where
F1  F2 is anti-conservative. This points toward a possible
practice of trying to fit a maximal LMEM wherever possi-
ble, and when it is not, to drop the concept of crossed ran-
dom effects altogether and perform separate by-subject
and by-item LMEMs, similar in logic to F1  F2, each with
appropriate maximal random effect structures. In closing,
it remains unresolved which of these strategies for dealing
with nonconvergence is ultimately most beneficial, and we
hope that future studies will investigate their impact on
generalizability more systematically.
Computing p-values
There are a number of ways to compute p-values from
LMEMs, none of which is uncontroversially the best.
Although Baayen et al. (2008) recommended using Monte
Carlo Markov Chain (MCMC) simulation, this is not yet
possible in lme4 for models with correlation parameters,
and our simulations indicate that this method for obtaining
p-values is more anticonservative than the other two
methods we examined (at least using the current imple-
 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
mentation and defaults for MCMC sampling in lme4 and
languageR).13 Also, it is important to note that MCMC sam-
pling does nothing to mitigate the anticonservativity of ran-
dom-intercept-only LMEMs when random slope variation is
present.
For obtaining p-values from analyses of typically-sized
psycholinguistic datasets—where the number of observa-
tions usually far outnumbers the number of model param-
eters—our simulations suggest that the likelihood-ratio
test is best approach. To perform such a test, one compares
a model containing the fixed effect of interest to a model
that is identical in all respects except the fixed effect in
question. One should not also remove any random effects
associated with the fixed effect when making the compar-
ison. In other words, likelihood-ratio tests of a fixed effect
with k levels should have only k  1 degrees of freedom
(e.g., one degree of freedom for the dichotomous single-
factor studies in our simulations). We have seen cases
where removing the fixed effect causes the comparison
model to fail to converge. Under these circumstances,
one might alter the comparison model following the proce-
dures described above to attempt to get it to converge, and
once convergence is achieved, compare it to an identical
model including the fixed effect. Note that our results indi-
cate that the concern voiced by Pinheiro and Bates (2000)
regarding the anti-conservativity of likelihood-ratio tests
to assess fixed effects in LMEMs is probably not applicable
to datasets of the typical size of a psycholinguistic study
(see also Baayen et al., 2008, footNote 1).
Reporting results
It is not only important for researchers to understand
the importance of using maximal LMEMs, but also for them
to articulate their modeling efforts with sufficient detail so
that other researchers can understand and replicate the
analysis. In our informal survey of papers published in
JML, we sometimes found nothing more than a mere state-
ment that researchers used ‘‘a mixed effects model with
random effects for subjects and items.’’ This could be any-
thing from a random-intercepts-only to a maximal LMEM,
and obviously, there is not enough information given to as-
sess the generalizability of the results. One needs to pro-
vide sufficient information for the reader to be able to
recreate the analyses. One way of satisfying this require-
ment is to report the variance–covariance matrix, which
includes all the information about the random effects,
including their estimates. This is useful not only as a check
on the random effects structure, but also for future meta-
analyses. A simpler option is to mention that one at-
tempted to use a maximal LMEM and, as an added check,
also state which factors had random slopes associated with
them. If the random effects structure had to be simplified
to obtain convergence, this should also be reported, and
the simplifications that were made should be justified to
the extent possible.
13
MCMC simulations for random-slopes and more complex mixed-effects
models can be run with general-purpose graphical models software such as
WinBUGS (Lunn, Thomas, Best, & Spiegelhalter, 2000), JAGS (Plummer,
2003), or MCMCglmm (Hadfield, 2010). This approach can be delicate and
error-prone, however, and we do not recommend it at this point as a
general practice for the field.
277
If it is seen as necessary or desirable in a confirmatory
analysis to determine the random effects structure using
a data-driven approach, certain minimal guidelines should
be followed. First, it is critical to report the criteria that have
been used, including the a-level for exclusion/inclusion of
random slopes and the order in which random slopes were
tested. Furthermore, authors should explicitly report the
changed assumptions about the generative process under-
lying the data that result from excluding the random slope
(rather than just stating that the slopes did not ‘‘improve
model fit’’), and should do so in non-technical language that
non-experts can understand. Readers with only back-
ground in ANOVA will not understand that removing the
random slope corresponds to pooling error across strata
in a mixed-model ANOVA analysis. It is therefore preferable
to clearly state the underlying assumption of a constant ef-
fect, e.g., ‘‘by excluding the random slope for the priming
manipulation, we assume that the priming effect is invari-
ant across subjects (or items) in the population.’’
Concluding remarks
In this paper we have focused largely on confirmatory
analyses. We hope this emphasis will not be construed as
an endorsement of confirmatory over exploratory ap-
proaches. Exploratory analysis is an important part of the
cycle of research, without which there would be few ideas
to confirm in the first place. We do not wish to discourage
people from exploiting all the many new and exciting
opportunities for data analysis that LMEMs offer (see Baa-
yen, 2008 for an excellent overview). Indeed, one of the sit-
uations in which the exploratory power of LMEMs can be
especially valuable is in performing a ‘‘post-mortem’’ anal-
ysis on confirmatory studies that yield null or ambiguous
results. In such circumstances, one should pay careful
attention to the estimated random effects covariance
matrices from the fitted model, as they provide a map of
one’s ignorance. For instance, when a predicted fixed effect
fails to reach significance, it is informative to check
whether subjects or items have larger random slopes for
that effect, and to then use whatever additional data one
has on hand (e.g., demographic information) to try to re-
duce this variability. Such investigation can be extremely
useful in planning further studies (or in deciding whether
to cut one’s losses), though of course such findings should
be interpreted with caution, and their post hoc nature
should be honestly reported (Wagenmakers et al., 2012).
At a recent workshop on mixed-effects models, a prom-
inent psycholinguist14 memorably quipped that encourag-
ing psycholinguists to use linear mixed-effects models was
like giving shotguns to toddlers. Might the field be better
off without complicated mixed-effects modeling, and the
potential for misuse it brings? Although we acknowledge
this complexity and its attendant problems, we feel that
one of the reasons why researchers have been using
mixed-effects models inappropriately in confirmatory analy-
ses is due to the misconception that they are something en-
tirely new, a misconception that has prevented seeing the
14
G.T.M. Altmann.
278 D.J. Barr et al. / Journal of Memory and Language 68 (2013) 255–278
continued applicability of their previous knowledge about
what a generalizable hypothesis test requires. As we hope
to have shown, by and large, researchers already know most
of what is needed to use LMEMs appropriately. So long as
we can continue to adhere to the standards that are already
implicit, we therefore should not deny ourselves access to
this new addition to the statistical arsenal. After all, when
our investigations involve stalking a complex and elusive
beast (whether the human mind or the feline palate), we
need the most powerful weapons at our disposal.
Acknowledgments
We thank Ian Abramson, Harald Baayen, Klinton Bic-
knell, Ken Forster, Simon Garrod, Philip Hofmeister, Florian
Jaeger, Keith Rayner, and Nathaniel Smith for helpful dis-
cussion, feedback, and suggestions. This work has been
presented at seminars and workshops in Bielefeld, Edin-
burgh, York, and San Diego. CS acknowledges support from
ESRC (RES-062-23-2009), and RL was partially supported
by NSF (IIS-0953870) and NIH (HD065829).
Appendix A. Supplementary data
Supplementary data associated with this article can be
found, in the online version, at http://dx.doi.org/10.1016/
j.jml.2012.11.001.
References
Baayen, R. H. (2004). Statistics in psycholinguistics: A critique of some
current gold standards. Mental Lexicon Working Papers, 1, 1–45.
Baayen, R. H. (2008). Analyzing linguistic data: A practical introduction to
statistics using R. Cambridge: Cambridge University Press.
Baayen, R. H. (2011). languageR: Data sets and functions with analyzing
linguistic data: A practical introduction to statistics. R package version 1.2.
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects
modeling with crossed random effects for subjects and items. Journal
of Memory and Language, 59, 390–412.
Barr, D. J. (2008). Analyzing ‘visual world’ eyetracking data using
multilevel logistic regression. Journal of Memory and Language, 59,
457–474.
Bates, D., Maechler, M., & Bolker, B. (2011). lme4: Linear mixed-effects
models using S4 classes. R package version 0.999375-41. http://
CRAN.R-Project.org/package=lme4.
Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens,
et al. (2009). Generalized linear mixed models: A practical guide for
ecology and evolution. Trends in Ecology & Evolution, 24, 127–135.
Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of
language statistics in psychological research. Journal of Verbal
Learning and Verbal Behavior, 12, 335–359.
Davenport, J. M., & Webster, J. T. (1973). A comparison of some
approximate F-tests. Technometrics, 15, 779–789.
Dixon, P. (2008). Models of accuracy in repeated-measures designs.
Journal of Memory and Language, 59, 447–496.
Fisher, R. A. (1925). Statistical methods for research workers. Edinburgh:
Oliver & Boyd.
Forster, K., & Dickinson, R. (1976). More on the language-as-fixed-effect
fallacy: Monte carlo estimates of error rates for F1, F2, F0 , and min F0 .
Journal of Verbal Learning and Verbal Behavior, 15, 135–142.
Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/
hierarchical models. Cambridge, MA: Cambridge University Press.
Goldstein, H. (1995). Multilevel statistical models. Kendall’s library of
statistics (vol. 3). Arnold.
Hadfield, J. D. (2010). MCMC methods for multi-response generalized
linear mixed models: The MCMCglmm R package. Journal of Statistical
Software, 33, 1–22.
Harrell, F. E. (2001). Regression modeling strategies: With applications to
linear models, logistic regression, and survival analysis. New York:
Springer.
Jaeger, T. F. (2008). Categorical data analysis: Away from ANOVAs
(transformation or not) and towards logit mixed models. Journal of
Memory and Language, 59, 434–446.
Jaeger, T. F. (2009). Post to HLP/Jaeger lab blog, 14 May 2009. <http://
hlplab.wordpress.com/2009/05/14/random-effect-structure>.
Jaeger, T. F. (2010). Redundancy and reduction: Speakers manage
syntactic information density. Cognitive Psychology, 61, 23–62.
Jaeger, T. F. (2011a). Post to the R-LANG mailing list, 20 February 2011.
<http://pidgin.ucsd.edu/pipermail/r-lang/2011-February/
000225.html>.
Jaeger, T. F. (2011b). Post to HLP/Jaeger lab blog, 25 June 2011. <http://
hlplab.wordpress.com/2011/06/25/more-on-random-slopes>.
Jaeger, T. F., Graff, P., Croft, W., & Pontillo, D. (2011). Mixed effect models
for genetic and areal dependencies in linguistic typology. Linguistic
Typology, 15, 281–320.
Janssen, D. P. (2012). Twice random, once mixed: applying mixed models
to simultaneously analyze random effects of language and
participants. Behavior Research Methods, 44, 232–247.
Kliegl, R. (2007). Toward a perceptual-span theory of distributed
processing in reading: A reply to Rayner, Pollatsek, Drieghe,
Slattery, and Reichle (2007). Journal of Experimental Psychology:
General, 136, 530–537.
Kuperman, V., Bertram, R., & Baayen, R. H. (2010). Processing trade-offs in
the reading of Dutch derived words. Journal of Memory and Language,
62, 83–97.
Locker, L., Hoffman, L., & Bovaird, J. (2007). On the use of multilevel
modeling as an alternative to items analysis in psycholinguistic
research. Behavior Research Methods, 39, 723–730.
Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. (2000). WinBUGS – A
Bayesian modelling framework: Concepts, structure, and
extensibility. Statistics and Computing, 10, 325–337.
Mirman, D., Dixon, J. A., & Magnuson, J. S. (2008). Statistical and
computational models of the visual world paradigm: Growth curves
and individual differences. Journal of Memory and Language, 59, 475–494.
Pinheiro, J. C., & Bates, D. M. (2000). Mixed-effects models in S and S-PLUS.
New York: Springer.
Plummer, M., 2003. JAGS: A program for analysis of Bayesian graphical
models using Gibbs sampling. In Proceedings of the 3rd International
workshop on distributed statistical computing.
Quené, H., & van den Bergh, H. (2004). On multi-level modeling of data
from repeated measures designs: A tutorial. Speech Communication,
43, 103–121.
Quené, H., & van den Bergh, H. (2008). Examples of mixed-effects
modeling with crossed random effects and with binomial data.
Journal of Memory and Language, 59, 413–425.
R Development Core Team (2011). R: A language and environment for
statistical computing. Austria: R Foundation for Statistical Computing
Vienna. ISBN 3-900051-07-0.
Raaijmakers, J. G. W., Schrijnemakers, J. M. C., & Gremmen, F. (1999). How
to deal with the language-as-fixed-effect fallacy: Common
misconceptions and alternative solutions. Journal of Memory and
Language, 41, 416–426.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models:
Applications and data analysis methods. Thousand Oaks, CA: Sage.
Roland, D., 2009. Relative clauses remodeled: The problem with mixed-
effect models. Poster presentation at the 2009 CUNY sentence
processing conference.
Santa, J. L., Miller, J. J., & Shaw, M. L. (1979). Using quasi F to prevent alpha
inflation due to stimulus variation. Psychological Bulletin, 86, 37–46.
Scheffe, H. (1959). The analysis of variance. New York: Wiley.
Schielzeth, H., & Forstmeier, W. (2009). Conclusions beyond support:
Overconfident estimates in mixed models. Behavioral Ecology, 20,
416–420.
Snijders, T., & Bosker, R. J. (1999a). Multilevel analysis: An introduction to
basic and advanced multilevel modeling. Thousand Oaks, CA: Sage.
Snijders, T. A. B., & Bosker, R. J. (1999b). Multilevel analysis: An introduction
to basic and advanced multilevel modeling. Sage.
Tukey, J. W. (1980). We need both exploratory and confirmatory. The
American Statistician, 34, 23–25.
Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., &
Kievit, R. A. (2012). An agenda for purely confirmatory research.
Perspectives on Psychological Science, 7, 632–638.
Wickens, T. D., & Keppel, G. (1983). On the choice of design and of test
statistic in the analysis of experiments with sampled materials.
Journal of Verbal Learning and Verbal Behavior, 22, 296–309.
Winer, B. J. (1971). Statistical principles in experimental design (2nd ed.).
New York: McGraw-Hill.