This article was downloaded by: [University of Waterloo]

On: 11 October 2014, At: 14:33

Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office:
Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Cognitive Neuropsychology
Publication details, including instructions for authors and subscription
information:
http://www.tandfonline.com/loi/pcgn20

Making sense of deviance: Identifying

dissociating cases within the case series
approach
a
Simon Fischer-Baum
a
Department of Psychology, Rice University, Houston, TX, USA
Published online: 29 Oct 2013.

To cite this article: Simon Fischer-Baum (2013) Making sense of deviance: Identifying dissociating
cases within the case series approach, Cognitive Neuropsychology, 30:7-8, 597-617, DOI:
10.1080/02643294.2013.846903

To link to this article: http://dx.doi.org/10.1080/02643294.2013.846903

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”)
contained in the publications on our platform. However, Taylor & Francis, our agents, and our
licensors make no representations or warranties whatsoever as to the accuracy, completeness, or
suitability for any purpose of the Content. Any opinions and views expressed in this publication
are the opinions and views of the authors, and are not the views of or endorsed by Taylor &
Francis. The accuracy of the Content should not be relied upon and should be independently
verified with primary sources of information. Taylor and Francis shall not be liable for any
losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities
whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or
arising out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Any substantial
or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or
distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use
can be found at http://www.tandfonline.com/page/terms-and-conditions
 Cognitive Neuropsychology, 2013
Vol. 30, Nos. 7 –8, 597 –617, http://dx.doi.org/10.1080/02643294.2013.846903

Making sense of deviance: Identifying dissociating cases

within the case series approach
Simon Fischer-Baum
Department of Psychology, Rice University, Houston, TX, USA

The case series approach in cognitive neuropsychology provides a means to test theories that make
quantitative predictions about associations between different components of the cognitive system
[Schwartz, M. F., & Dell, G. S. (2010). Case series investigations in cognitive neuropsychology.
Downloaded by [University of Waterloo] at 14:33 11 October 2014

Cognitive Neuropsychology, 27, 477 –494]. However, even when the predicted association is borne
out the study may include outliers—observations that deviate significantly from the rest of the
data. These outliers may reveal individual cases whose cognitive impairments dissociate from
other cases included in the study. These dissociating cases can pose a significant challenge to
the theory being tested. Using a recent case series that investigated the underlying causes of
letter perseveration in spelling [Fischer-Baum, S., & Rapp, B. (2012). Underlying cause(s) of
letter perseveration errors. Neuropsychologia, 50, 305–318], I discuss statistical and theoretical
issues that arise when using outlier detection techniques to identify dissociating cases in a case
series study.

Keywords: Cognitive neuropsychology; Outlier detection; Case series; Single-case studies;

Associations and dissociations.

A common critique of the single-case approach to
neuropsychology has been its focus on identifying
dissociations between cognitive functions at the
expense of identifying how cognitive functions
are associated (Patterson & Plaut, 2009). Case
series studies offer a complementary approach to
the single-case design, providing a clear means
for testing theoretically motivated associations. A
typical case series study combines data from mul-
tiple behavioural measures across a large number
of individual cases and analyses how these
measures covary across participants (Schwartz &
Dell, 2010; cf. Olson & Romani, 2011). To the
extent that these different behavioural measures
tap into different cognitive functions, associations
between these measures reflect associations
between the underlying cognitive functions. The
relative merits of the case series methodology in
cognitive neuropsychology have been discussed
extensively in this journal over the past several

Correspondence should be addressed to Simon Fischer-Baum, Department of Psychology, Rice University, MS-25, P.O. Box
1891, Houston, TX 77251, USA. (E-mail: sjf2@rice.edu).
I am grateful to Gary Dell, Matt Goldrick, Michael McCloskey, Fred Oswald, and Brenda Rapp for helping me work through
these issues of case series, outliers, and cognitive neuropsychology, and to David Kajander for reading through drafts. Portions of this
work were presented in a symposium on case series investigations at the 50th annual meeting of the Academy of Aphasia. The
remaining work was moulded by the discussion that took place at that meeting.

# 2013 Taylor & Francis 597

 FISCHER-BAUM
years (Bub, 2011; Dell & Schwartz, 2011;
Goldrick, 2011; Lambon Ralph, Patterson, &
Plaut, 2011; Nickels, Howard, & Best, 2011;
Olson & Romani, 2011; Rapp, 2011; Schwartz
& Dell, 2010; Shallice & Bulatti, 2011).
Generally, these researchers agree that case series
studies provide a means to test certain types of
hypotheses that are difficult to test with the
single-case approach. However, a number of
these researchers noted the importance of evaluat-
ing not only the associations that exist in the case
series, but also whether each individual in the
sample is a legitimate function of the underlying
Downloaded by [University of Waterloo] at 14:33 11 October 2014
pattern of association (Dell & Schwartz, 2011;
Goldrick, 2011; Nickels et al., 2011; Rapp, 2011;
Schwartz & Dell, 2010, cf. Lambon Ralph et al.,
2011). According to Nickels et al. (2011, p. 481),
the logic of cognitive neuropsychology dictates
that even a single individual who deviates signifi-
cantly from the rest of the sample “may be a real
example of a dissociating pattern that can falsify
a theory.”
The goal of this paper is to provide a frame-
work for evaluating case series studies for these
dissociating cases. I discuss a recent case series
on the underlying causes of letter perseverations
in spelling as a concrete example (Fischer-Baum
& Rapp, 2012). I then propose a statistical
method that uses the recursive externally studen-
tized residual test (Weisberg, 1985) as a tool for
detecting outliers, individual cases that deviate
significantly from the association being tested in
the case series. This method is similar to a
recent proposal by Crawford and Garthwaite
(2006, 2007a, 2007b) for comparing a single
neuropsychological case study to a population of
unimpaired control participants whose
performance is characterized by a regression
equation.
The second half of the paper explores this
outlier detection method in more detail by apply-
ing it to simulated data. Across six simulations I
explore the effects of the number of participants,
the number of observations per participant, the
frequency of dissociating cases in the population,
the placement of the decision criterion, and the
role of follow-up testing on the ability to detect
598 Cognitive Neuropsychology, 2013, 30 (7 –8)
dissociating cases in a case series study. By carefully
analysing our case series for dissociating cases, we
can use case series data to develop theories that
explain not only the associations that are observed
across large samples of brain-damaged individuals,
but also the dissociations.
THE CASE SERIES APPROACH TO
LETTER PERSEVERATIONS IN
DYSGRAPHIA
Fischer-Baum and Rapp (2012) addressed the
issue of dissociating cases in a case series in an
investigation of the underlying causes of letter per-
severation errors in dysgraphia. Perseveration
errors are the inappropriate repetition of a previous
response in the place of the current target. The
errors investigated in that paper were the perse-
veration of individual letters from a previous
written response into a subsequent response (e.g.,
the perseveration of the R in the error “edge”
spelled as ERGE, immediately following the
response FRENCE). Two accounts for why
these perseveration errors were produced were
considered: a “failure-to-inhibit” account, in
which perseverations arise because of a failure to
inhibit previous responses (e.g., Hauser, 1999)
and a “failure-to-activate” account, in which perse-
veration errors arise because the current target
receives abnormally little activation (e.g., Cohen
& Dehaene, 1998; Dell, Burger, & Svec, 1997).
To assess these two accounts, Fischer-Baum
and Rapp (2012) analysed the spelling errors pro-
duced by a series of dysgraphic individuals.
Following work on verbal perseverations by
Martin and Dell (2004, 2007), they reasoned
that a failure-to-activate deficit should lead to
both perseverative and nonperseverative intru-
sions, and they predicted an association between
an individual’s rate of producing both of these
error types. This predicted association was borne
out across the case series. Figure 1 reproduces
the association between perseverative and nonper-
severative intrusions in the 12 individuals with
acquired dysgraphia studied by Fischer-Baum
and Rapp (2012).

Downloaded by [University of Waterloo] at 14:33 11 October 2014
Figure 1. Observed relationship between the rate of producing letter
perseveration errors and nonperseverative intrusions in spelling-to-
dictation in a sample of 12 dysgraphic individuals. [Reprinted from
Neuropsychologia, 50, Fischer-Baum, S., & Rapp, B. Underlying
cause(s) of letter perseveration errors, 305– 318. Copyright (2012),
with permission from Elsevier.]
What can really be concluded from this associ-
ation? It would be incorrect to conclude from this
analysis that a failure-to-activate deficit is the only
deficit that produces letter perseverations or that
all individuals in the sample perseverate because
of a failure-to-activate deficit. The association
alone also cannot tell us whether all individuals
who have a failure-to-activate deficit produce a
significant number of perseverations. At best, it
indicates that it is unlikely that a failure-to-
inhibit deficit explains all of the letter persevera-
tions. But the goal of these studies was to identify
the deficit that caused individuals to perseverate. If
the theoretical claim being tested is that a failure-
to-activate deficit is both necessary and sufficient
for persons with aphasia or dysgraphia to
produce perseverations, then a single individual
whose pattern of performance deviates signifi-
cantly from the predicted association would
falsify that claim. An individual whose rate of per-
severation was much higher than expected by their
rate of nonperseverative intrusions would chal-
lenge the claim that the failure-to-activate deficit
is the only deficit that can cause perseveration,
while an individual who produced few persevera-
tions despite many nonperseverative intrusions
CASE SERIES OUTLIERS
would challenge the claim that having a failure-
to-activate deficit alone is enough for persevera-
tion to occur.
A cursory glance at Figure 1 reveals a single
point with an extremely high rate of perseveration
errors (25% of all letters produced are persevera-
tions) that would not be predicted from his rela-
tively low rate of nonperseverative intrusions
(4% of all letters produced). This deviating
data point may be the type of dissociating case
that falsifies the necessary and sufficient failure-
to-activate theory. But how can we determine
that this data point truly deviates from the pre-
dicted association? And even if we determine
that this data point is an outlier, how do we
know whether or not the individual who generated
that data point is a “real example of a dissociating
pattern”? The remainder of the paper outlines stat-
istical methods that address these questions.
RECURSIVE EXTERNALLY
STUDENTIZED RESIDUAL TEST:
A METHOD FOR OUTLIER
DETECTION
The issue of identifying outliers in correlational
analyses has received a great deal of attention in
the statistics literature. Freeman (1980, p. 350)
defines outliers as “any observation that has not
been generated by the mechanism that generated
the majority of the observations in the dataset”.
Following Freeman’s definition of outliers, the
identification of an outlier in the case series
described above seems to indicate that the data
point was not generated by the failure-to-activate
deficit, but rather by some other deficit. Of
course, like any detection procedure, a sensitive
outlier detection technique maximizes hits—prop-
erly identifying those individuals generated by a
different mechanism from that for the majority
of observations in the data set—while avoiding
false alarms—identifying as an outlier data points
generated by the same mechanism as that for the
majority of the data points.
Weisberg (1985) proposes the externally stu-
dentized residual test as an outlier detection
Cognitive Neuropsychology, 2013, 30 (7 –8) 599
 FISCHER-BAUM

method suited to determining whether individual
data points lie surprisingly far from a fitted
regression line.1 This test requires the assumptions
typical for linear regression models—namely that
the error from the fit of the regression line is nor-
mally distributed, and that errors are mutually
uncorrelated. Using this method, we can evaluate
whether an individual’s rate of producing perse-
veration errors differs significantly from the rate
expected given that individual’s rate of producing
nonperseverative intrusions. Included in
Appendix A and Appendix B are two Matlab pro-
grams that can be used, in combination with
Downloaded by [University of Waterloo] at 14:33 11 October 2014
rate of nonperseverative intrusion. To say the
residual is calculated externally means that when
evaluating how far the nth case deviates from a
predicted point, that prediction is derived using
all of the data except for the nth case. In the
example case series above, to determine the
residual of the apparent outlier, that individual’s
observed rate of producing perseverations
(25%) is compared to a predicted rate of produ-
cing perseverations, derived from (a) that individ-
ual’s observed rate of producing nonperseverative
intrusions (4%) and (b) the regression line
given the other 11 individuals in the study.

Statistics Toolbox, to carry out this outlier detec-
tion technique. Other statistical packages (includ-
ing free packages like R) come equipped with
functions that calculate the externally studentized
residual.
The basic logic of any outlier detection test is as
follows: First, each point has to be evaluated for
how far it deviates from the other data points in
the sample. Second, a decision criterion must be
set to evaluate whether the degree to which a
data point deviates from the rest of the sample is
large enough to merit identification as an outlier.
The externally studentized residual is a measure
of how far a single data point deviates from the
rest of the sample. The residual of each data
point is simply how far that point falls from the
point predicted by the regression line. For the
case series described above, the residual is a
measure of how far an individual’s observed rate
of producing perseveration deviates from the rate
that would be predicted given that individual’s
Finally, the residuals are studentized, or converted
to a t-statistic that takes into account the variance
in residuals—calculated over all of the data except
for the nth case—as well as leverage of the data
point (i.e., the influence of the data point on its
fitted value; high leverage indicates that its residual
matters more).2 The apparent outlying case in
Figure 1 has an externally studentized residual of
6.3. An externally studentized residual is calcu-
lated for all individuals included in the case
series; from the data shown in Figure 1, an exter-
nally studentized residual is calculated for all 12 of
the cases, with the other 11 cases ranging from
21.2 to 0.5.
The second stage of outlier detection is to define
a decision criterion for determining whether a
single data point that deviates enough from the
rest of the sample is an outlier. Because the exter-
nally studentized residuals have been converted to
t-statistics, the decision criterion can be set using
the null hypothesis statistical testing approach;

1
Fischer-Baum and Rapp (2012) describe a Mahalanobis distance procedure to detect outliers following Penny (1996). However,
a further review of the statistics literature, and some simple simulations, has led me to the conclusion that the externally studentized
residual test would have been more appropriate with our data set. The Mahalanobis distance technique measures how far a given data
point is from the centre of mass of the whole data set, while the externally studentized residual test measures how far the data point is
from the regression line. Consider a data set that includes a data point with a much greater rate of perseverative and nonperseverative
intrusions than the rest of the individuals in the sample, but whose data point falls near the regression line. Given the theory being
tested, it would be incorrect to conclude that the individual represented by this data point suffers a qualitatively different deficit from
the others in the sample; instead, it appears that this individual has a quantitatively more extreme version of the same deficit. Yet,
because this data point is far from the centre of mass of the whole data set, it will be identified as an outlier by the Mahalanobis
distance technique. Because it falls near the regression line, it will not be identified as an outlier by the externally studentized residual
test.
2
Error is larger for more extreme values of the predictor variable, causing the studentized residual to be smaller. Therefore, a
larger difference between the observed and predicted values is needed for a point to be identified as an outlier at these extreme
values (see Crawford & Garthwaite, 2006, for further discussion of this point).

600 Cognitive Neuropsychology, 2013, 30 (7 –8)


critical t-values can be identified such that we are
more than 95% confident that an externally studen-
tized residual is different from zero. Because we
need to independently determine whether each of
the 12 residuals is significantly different from
zero, it is important to make some correction for
multiple comparisons. To address this problem,
Weisberg (1985) suggests Bonferroni correction.
Most of the simulations described below follow
Weisberg’s suggestion and use this conservative
decision criterion for identifying outliers, though
the fifth simulation explores the effect of relaxing
the decision criterion on the effectiveness of
Downloaded by [University of Waterloo] at 14:33 11 October 2014
outlier detection. For the specific example above,
because there are 12 values being evaluated, for a
significant level of .05, with a Bonferroni correc-
tion, the critical value is based on an alpha of .05/
12 or .0042. The degrees of freedom for the signifi-
cance tests, in the case of simple linear regression, is
(n 2 2), where n is the number of data points in the
analysis. In the example case series, the critical
t-values for a two-tailed test with a p , .0042 and
10 degrees of freedom are +3.70 and 23.70.
Since the observed externally studentized residual
for the apparent outlying case (6.3) is more
extreme than the critical value, the null hypothesis
can be rejected that this residual is no different from
0, and this data point is identified as an outlier.
Even though no other participant in the study
has a residual value more extreme than the critical
value, it may be premature to conclude that none
of these other individuals are outliers. The
problem of masking—in which one extreme
outlier increases the overall variability of the
dataset, making the detection of other outliers
more difficult—is well known in the statistics lit-
erature (Ben-Gal, 2005). To address this
problem, I suggest recursive application of the
statistic described above until no more outliers
are detected. Those outliers detected during a
single run of the statistical analysis should be
deleted from the data set, and the statistical analy-
sis should be run again.3 In the case series example,
3
One concern about this recursive application of the externally studentized residual test is that it will make false alarms more
common; once one or two nonoutliers are incorrectly labelled as outliers, the technique may start to identify false alarms on other
data points. The results of the simulation discussed below show that such false alarms are rare with this technique.
CASE SERIES OUTLIERS
the one outlying case is deleted from the analysis,
and externally studentized residuals are recalcu-
lated for the remaining 11 cases. While these stu-
dentized residuals are now larger, none are
significantly different from 0. The application of
the recursive externally studentized residual test
identifies a single outlier in the Fischer-Baum
and Rapp (2012) case series.
If we can show that this outlier is generated by a
different mechanism from the mechanism that
generated the data for the other 11 participants in
the study (Freeman, 1980), then its very presence
poses a significant problem for the necessary-
and-sufficient failure-to-activate hypothesis.
However, there remains a possibility that this
outlier is a false alarm; the same mechanism that
generated the rest of the data generated this data
point, and it is because of random noise that this
individual is identified as an outlier. With real
data, identifying whether an outlier is a hit or a
false alarm is an ill-defined problem. We certainly
cannot claim that an outlier is a false alarm simply
because we do not understand the deficit that is
giving rise to this pattern of performance.
Simulation studies provide a clear opportunity to
investigate the problem of false alarms in outlier
detection. In a simulation, the mechanism that
generates the data for all of the participants can
be prespecified, so we know which individuals in
the simulation are truly dissociating cases and
which are not. False alarms in simulation studies
are simply those individuals identified as outliers,
despite not being one of the individuals prespeci-
fied as a truly dissociating case.
How much we need to worry about false alarms
depends on the statistical power of the outlier
detection technique. In part, the statistical power
of the outlier detection technique depends on the
design of the case series investigation—how
many participants are included in the study?
How much data are collected for each participant?
Other aspects of the investigation outside of the
control of the experimenter—that is, the
Cognitive Neuropsychology, 2013, 30 (7 –8) 601
 FISCHER-BAUM
prevalence of the different underlying deficits in
the population—will also affect the statistical
power of outlier detection technique. These vari-
ables are explored in a series of simulation
studies below.
BENEFITS OF MORE PARTICIPANTS
AND MORE DATA PER
PARTICIPANTS
As case series researchers, a trade-off exists
between including more participants in the study
Downloaded by [University of Waterloo] at 14:33 11 October 2014
and collecting more data from each participant.
Fischer-Baum and Rapp (2012) collected hun-
dreds of spelling trials for each of the participants
included in the case series. Because of the intensive
investigation of each individual in the study, it was
infeasible to include many participants. However,
there may be statistical concerns with having too
small a sample in a case series, as it may increase
sampling error variance. The goal of a case series
study is to draw inferences from our sample that
generalize to the population of all individuals
that meet the inclusion criteria for the study.
The importance of inclusion criteria has been
raised in recent discussions of the case series meth-
odology (Dell & Schwartz, 2011; Olson &
Romani, 2011; Shallice & Buiatti, 2011).
Regardless of inclusion criteria, it is unlikely that
the entire population is being investigated in the
study. Even when a random and representative
sample is drawn from the population of interest,
statistics will vary randomly from sample to
sample. As the size of the sample increases, the
sampling error variance will decrease.
Researchers who include more participants in
their study, in order for their results to better gen-
eralize to the population as a whole, may collect
fewer data from each participant. This decision
introduces an additional source of noise, measure-
ment error variance. For each participant in the
sample, we are trying to measure some true under-
lying value—in my example an individual’s ten-
dency to perseverate. But we can only measure
that underlying value from the actual data col-
lected from each individual. As with sampling
602 Cognitive Neuropsychology, 2013, 30 (7 –8)
error variance, because our actual measurement is
based on a limited number of observations, these
measures will vary randomly. Consider an individ-
ual whose true underlying tendency is to produce a
perseveration error on 10% of the trials. If this
individual was given only 10 trials of the task,
how likely is it that she would make exactly one
perseveration error? Following the binomial distri-
bution, the probability that she will produce
exactly one perseveration error (and therefore
have an estimated perseverative tendency of .1,
matching her true underlying tendency) is
roughly .4. The probability that she will not
produce a single perseveration error (and therefore
have an estimated perseverative tendency of .0) is
about .35, and the probability that she will
produce at least two perseverations in 10 trials
(and have an estimated perseverative tendency of
at least .2) is about .25. Because only limited
data are being collected from this individual, our
measure of the tendency to perseverate is going
to differ from the underlying value.
As with sampling error variance, one way to
reduce measurement error variance is to collect
more observations from each individual in the
sample. The effects of collecting more data are
illustrated in Figure 2. In Fischer-Baum and
Rapp (2012), each participant was asked to spell
at least 300 words (sometimes many more), and
each produced more than 1000 letters. The error
bars in Figure 2a show the bootstrapped 95% con-
fidence intervals (on x- and y-values indepen-
dently) around each participant’s estimated values
of the tendency to produce perseverative and non-
perseverative intrusions (Simon & Bruce, 2000);
these confidence intervals indicate the range of
true underlying values that could plausibly yield
the observed error rates given the actual number
of letters produced. The narrow error bars indicate
that there is little measurement error variance in
this study; estimates of each individual’s tendency
to produce preservative and nonperseverative
intrusions appear to be quite precise. Figure 2b
shows the bootstrapped 95% confidence interval
assuming that only 75 observations had been col-
lected from each individual. The resulting error
bars are wide, indicating that there is a much

Downloaded by [University of Waterloo] at 14:33 11 October 2014
Figure 2. Bootstrapped 95% confidence interval around both the
rate of perseveration errors and the rate of nonperseverative
intrusions based on the data shown in Figure 1. Figure 2a shows
the confidence intervals given the actual number of letters
produced by each individual in the sample, while Figure 2b shows
the confidence intervals assuming that each individual only
produced 75 responses.
larger range of true underlying values that could
have plausibly yielded the observed result.
Both sampling and measurement error variance
can contribute to poor performance of outlier
detection techniques. Collecting more data mini-
mizes each source of noise in the sample, but the
different sources of noise benefit from different
types of additional data collection. The first three
simulation studies examine the relative contri-
butions of running more participants and collect-
ing more data for each participant on the
sensitivity of outlier detection.
CASE SERIES OUTLIERS
The general structure of the simulations
For all of the simulations in this paper, a simplified
situation was considered in which a participant’s
response was (a) a correctly produced letter, (b) a per-
severation error, or (c) an intrusion that was not a
perseveration. Two types of participants were
included in the simulations: truly associated partici-
pants whose probability of producing a perseveration
error and probability of producing a nonpersevera-
tive intrusion were identical, and dissociating cases,
who were participants for whom those error rates
were uncorrelated. These dissociating cases are the
outliers in these stimulations, as their data are gener-
ated by a different mechanism from that for the truly
associated participants. The goal of these stimulation
studies is to evaluate how well our outlier detection
technique performs at correctly identifying these
cases without falsely identifying the associated
participants as outliers. For all simulations except
Simulation 4, 90% of the participants were truly
associated participants.
For all participants, a random value between 0.0
and 1.0 was generated for their probability of cor-
rectly producing a letter. For the truly associated
participants, the remaining proportion was then
evenly divided between perseverative and nonper-
severative intrusions. For example, a truly associ-
ated participant could have had a probability of
.6 for correctly producing a letter, .2 for producing
perseverations, and .2 for producing nonpersevera-
tive intrusions. For the dissociating cases, all three
true were randomly selected, and normalized,
ensuring that they added up to 1.0.
Error was introduced into the simulation
through resampling procedures. Each simulation
specified the number of observations collected for
each participant. Each observation was classified
as a perseveration, a nonperseverative intrusion, or
a correct response, following the underlying prob-
ability distribution for that participant. An observed
perseveration rate and rate of producing nonperse-
verative intrusions were then calculated for each
participant by dividing the number of observed per-
severations and the number of observed nonperse-
verative intrusions by the total number of
observations. In the limit, the observed data
Cognitive Neuropsychology, 2013, 30 (7 –8) 603
 FISCHER-BAUM

should match the underlying probabilities of these

error types. However, because a finite number of
observations were collected, the observed values
deviate from the underlying probabilities, with
error that approximates a normal distribution.
Using this procedure, a single randomly gener-
ated data set was created given (a) a number of par-
ticipants, (b) the number of observations for each
participant, and (c) the proportion of dissociating
cases. The correlation between the perseverative
and nonperseverative intrusion rates was then cal-
culated and tested for significance. The recursive
externally studentized residual test was applied to
Downloaded by [University of Waterloo] at 14:33 11 October 2014

the data set in order to identify outliers, and

both hit rate—the proportion of dissociating
cases correctly identified—and false-alarm rate—
the proportion of associated cases falsely identified
as outliers—were calculated. For a given number
of participants, number of observations for a par-
ticipant, and proportion of dissociating cases,
10,000 randomly generated data sets were
created, and the correlation coefficient, signifi-
cance, hit rate, and false-alarm rate were calculated
for each of the 10,000 data sets.

Simulation 1: Varying number of

participants
The first simulation looked at the effects of reducing
sampling error variance by increasing the number of
participants in the study. Held constant in this simu-
lation was the proportion of dissociating cases (10%)
and the number of observations collected per partici-
pant (100). The number of participants varied in
increments of 10 from 10 to 200 participants.
Overall, the ability to detect the underlying
association between perseverative and nonperse-
verative intrusions was good across number of
participants. The average correlation ranged
from .72, when the sample included 10 partici-
pants, to .69, when the sample included 200 par-
ticipants. When at least 30 participants were
included in the sample, that correlation was signifi-
cant in more than 95% of the randomly generated
data sets. With 10 participants, the correlation
was only significant in 72% of the data sets, and
Figure 3. (a) Hit rate, (b) false-alarm rate, and (c) d ′ for
simulated data sets from Simulation 1 with a constant number of
observations per participant (100) and proportion of the
dissociating cases in the sample (10%) and a variable number of
participants in the study (range: 10– 200).
that number increased to 89% of the data sets
when 20 participants were included in the study.
Figure 3 shows the results of applying the
outlier detection technique to these simulated
data sets in terms of hit rate (Figure 3a) and
false-alarm rate (Figure 3b). The first result to
note is that varying the number of participants
has no effect on the hit rate in these simulations.
Whether there are 10 participants or 200 partici-
pants in the study, the outlier detection technique

604 Cognitive Neuropsychology, 2013, 30 (7 –8)

 CASE SERIES OUTLIERS

correctly identifies approximately 35% of the dis-

sociating cases. The second result to note is how
low the false-alarm rate is in all of these simulated
data. Because of Bonferroni correction, the detec-
tion technique proposed here is quite conservative.
At most about 1% of the associated individuals are
falsely identified as outliers. It should be noted that
increasing the number of participants in the study
decreases the false-alarm rate. With 10 partici-
pants an average of 1.1% of the associative partici-
pants were identified as outliers, while with 200
participants the rate of false alarms was less than
0.1%. Following signal detection theory, the sensi-
tivity of outlier detection can be expressed with d ′ ,
Downloaded by [University of Waterloo] at 14:33 11 October 2014

a measure that takes into account both hit rate and

false-alarm rate. Figure 3c shows how the d ′ of the
recursive externally studentized residual detection
technique varies as a function of number of partici-
pants; as the number of participants increases, d ′
increases as well, from a d ′ of 1.9 for data sets
generated with 10 participants to a d ′ of 2.8 for
data sets generated with 200 participants. All else
being equal, reducing sampling error variance by
increasing the number of participants in the
study makes the recursive externally studentized
residual test more accurate, as it decreases the like-
lihood of producing a false alarm.

Simulation 2: Varying number of

observations per participant
The second simulation looked at the effect of low-
ering measurement error variance by increasing
observations per participant on outlier detection.
The number of participants was held constant (at
50) across the simulated data sets, as was the per-
centage of dissociating cases (10%, or 5 in all data
sets), while the number of observations collected
for each participant varied by increments of 20
from 20 to 1000 observations.
Significant correlations between rate of perse-
verative and rate of nonperseverative intrusions
were observed in nearly all of the randomly gener-
ated data sets. The average correlation coefficients
range from .45, with 20 observations per partici-
pant, to .77, with 1000 observations per partici-
pant. At all levels of observation-per-participant,
Figure 4. (a) Hit rate, (b) false-alarm rate, and (c) d ′ for simulated
data sets from Simulation 2 with a constant number of participants
in the study (50) and proportion of the dissociating cases in the
sample (10%) and a variable number of observations per
participants (range: 20– 1000).
more than 90% of the random data set had signifi-
cant correlations. When at least 40 observations
were collected per participant, more than 95% of
the random data sets had a significant correlation.
Figure 4 shows the performance of the outlier
detection technique over these random data sets.
Unlike increasing the number of participants,
there is a clear benefit of increasing the number

Cognitive Neuropsychology, 2013, 30 (7 –8) 605

 FISCHER-BAUM
of observations per participant on hit rate (Figure
4a). With 20 observations per participant, only
7% of the dissociating cases were correctly ident-
ified. However, when 1000 observations were col-
lected per participant, 71% of the dissociating cases
were correctly identified. Again, the false-alarm
rate in these simulations was extremely low, con-
sistently less than 0.5% across all conditions
(Figure 4b). There appears to be a slight cost, in
terms of false alarms, of increasing the number
of observations, from approximately 0.2% of the
truly associated individuals when 20 observations
were collected per participant to about 0.4%
Downloaded by [University of Waterloo] at 14:33 11 October 2014
when 1000 observations were collected per partici-
pant. However, as shown in Figure 4c, this slight
increase in false-alarm rate was overwhelmed by
the large increases in hit rate when d ′ was calcu-
lated, with the d ′ -sensitivity of the outlier detec-
tion technique around 1.4 with 20 observations
per participant and 3.2 with 1000 observations
per participant.
Simulations 1 and 2 demonstrate that both
sampling and measurement error variance cause a
decrease in the sensitivity of the recursive exter-
nally studentized residual outlier detection tech-
nique. The sensitivity of the technique is
improved both by increasing the number of par-
ticipants, decreasing sampling error variance, and
by increasing the number of observations per par-
ticipant, decreasing measurement error variance.
Strikingly, the benefits of increasing the number
of participants and increasing the observations
per participant appear to affect outlier detection
differentially. In Simulation 1, the effects of
increasing the number of participants shows up
in a decrease in false-alarm rate, while in
Simulation 2, the effects of increasing observations
per participants shows up primarily in an increase
in hit rate. If one of the goals of the case series
investigation is to be able to detect dissociating
cases, as many researchers have argued, there are
independent motivations for testing more partici-
pants and for collecting more data for each partici-
pant. Of course, there are costs, in terms of both
time and other resources, of running studies that
are too large. Finding additional participants for
the study and bringing the same participants
606 Cognitive Neuropsychology, 2013, 30 (7 –8)
back for multiple testing sessions can be time con-
suming. Simulation 3 looks at the trade-off
between including more participants in the study
and collecting more data per participant. With
finite time and resources, is it better to collect
more data from a smaller sample, or include
more participants but test each participant less?
Simulation 3: Trade-off between
participants and observations per
participation
Simulation 3 looks at the direct trade-off between
including more participants in the study and col-
lecting more observations per participant. A new
value—the total number of trials in the study—is
defined as the product of the number of partici-
pants and the observations per participant. For
example, if the study includes 50 participants,
and 40 observations are collected for each partici-
pant, then the study includes a total of 2000 trials.
The same number of trials could be distributed in
different ways; an alternative 2000-trial study
could have included 200 participants, with only
10 observations collected for each participant.
Assuming an equal cost for each trial, how
should trials be distributed between the number
of participants and the number of observations
per participant if the goal is to maximize the per-
formance of outlier detection?
To address this question, I looked at eight levels
of total numbers of trials—1000, 2000, 3000, 4000,
6000, 8000, 12,000, and 20,000 trials per study.
The proportion of dissociating cases was held con-
stant at 10%, and random data sets were created for
different combinations of number of participants
and observations per participant, resulting in these
numbers of trials. For example, the 2000-trial simu-
lations included random data sets generated with 10
participants and 200 observations per participant,
20 participants and 100 observations per partici-
pant, 40 and 50, 50 and 40, 80 and 25, 100 and
20, and, finally, 200 participants and 10 obser-
vations per participant. The maximum number of
participants was set to be 200, as studies including
more participants than that seem infeasible. For
each of these number of participants and

Figure 5. The d ′ for a given number of total number of trials in the
study as the number of participants increases (and the number of
observations per participant decreases). Darker lines reflect more
Downloaded by [University of Waterloo] at 14:33 11 October 2014
total trials in the study.
observations per participant pairs, 10,000 random
data sets were generated, and hit rate, false-alarm
rate, and d ′ were calculated.
Figure 5 plots the d ′ as a function of number of
participants and total number of trials.
Unsurprisingly, increasing the total number of
trials (the darkening lines) increased d ′ . The
greater the total number of trials in the study,
the better the recursive externally studentized
residual test performed at detecting dissociating
cases. However, within a given total number of
trials (shown here as a single line in the figure),
it was generally the case that it is better to run
fewer participants and collect more observations
per participant. For small total numbers of trials
(1000– 4000), the maximum d ′ was found with
only 10 participants (and 100, 200, 300, and 400
observations per participants), and d ′ decreased
monotonically as the number of participants
increases. With 6000 and 8000 trials, d ′ was
greater with 20 participants (300 and 400 obser-
vations per participant, respectively) than with 10
participants, but monotonically decreased with
more than 20 participants. For 12,000 trials, the
same level of sensitivity was observed with 20 par-
ticipants (and 600 observations/participant) and
30 participants (and 400 observations for partici-
pants), with decreasing sensitivity for smaller or
larger numbers of participants. For 20,000 trials,
the maximum sensitivity was observed with 40
participants (and 500 observations/participant).
CASE SERIES OUTLIERS
Assuming an even trade-off in the costs associ-
ated with running more participants and collecting
more data from each participant, the results of
Simulation 3 suggest that it is better to collect
more data for each participant than it is to
include more participants. While it is not always
the case that the outlier detection technique per-
forms best with the fewest number of participants
(10), it appears that increasing the number of par-
ticipants is only beneficial once there is a relatively
large number of observations (400 –600) collected
per participant.
The assumption of an even trade-off between
participants and observations may not hold for all
case series studies. For most case series studies, it
is more costly, in terms of both time and money,
to recruit and consent additional participants than
it is to collect more data per participant by bringing
them back for additional testing sessions or even
keeping them longer in a single session. This
problem is particularly salient in those case series
with restrictive inclusion and exclusion criteria,
requiring an individual either to show intact per-
formance in a number of cognitive tasks (e.g.,
Nickels et al., 2011) or to have specifically localized
brain damage (e.g., Shallice & Buiatti, 2011).
However, there are circumstances in which it is
impossible to collect a large number of observations
per participant—that is, case series with individuals
in the acute stage of stroke aphasia in which all of
the behavioural testing has to be carried out
within a small window of time immediately follow-
ing the stroke (e.g., Hillis, Kane, Barker,
Beauchamp, & Wityk, 2001). In those cases, the
costs of increasing the number of observations per
participant may be greater than the costs of includ-
ing more participants.
Summary of Simulations 1– 3
The first three simulations provide some sugges-
tions for how a case series study should be designed
to identify dissociating cases present in the sample.
Simulations 1 and 2 show that there are indepen-
dent benefits on the ability to detect outliers with
both increasing the number of participants and
increasing the number of observations per
Cognitive Neuropsychology, 2013, 30 (7 –8) 607
 FISCHER-BAUM
participant. Simulation 3 shows that the benefit of
collecting more data per participant mostly out-
weighs the benefits of including more participants.
Of course, identifying dissociating cases is not
the only goal of a case series investigation.
Another goal is being able to identify associations
that exist in the data. Simulation 1 shows that
small sample sizes have higher correlations
between perseverative and nonperseverative intru-
sions on average than larger sample sizes, but those
correlations are less likely to be significant. With
small sample sizes, there is less power to detect
the true association that exists for 90% of the indi-
Downloaded by [University of Waterloo] at 14:33 11 October 2014
viduals in the sample. Given these two goals in a
case series study, the simulations show that the
study should include a sample of at least 30 partici-
pants and at least 400 observations for each indi-
vidual, assuming a 90 – 10 split between
associated and dissociating cases. In real case
series, there is no way to know how frequently
the dissociating pattern will be observed.
Simulation 4 explores the effects of different pro-
portions of dissociating cases in the sample.
However, before moving onto the next stimu-
lation, I consider the situation in which no outliers
are identified in the sample.
WHAT DOES IT MEAN IF NO
OUTLIERS ARE DETECTED?
As Nickels et al. (2011) highlight, the presence of
a single dissociating case in a case series has the
possibility of falsifying the theory being tested.
But what if no such case is identified? Such a
situation appears to support the theory being
tested; however, there are several issues that arise
from taking this failure to identify a dissociating
case as strong evidence supporting the hypothesis.
The first is that the outlier detection technique
outlined above is a form of the null hypothesis test.
A failure to identify an outlier should have the
same status as a failure to reject the null hypoth-
esis; it may be consistent with a theory but it
does not constitute strong evidence in favour of
that theory. Indeed, even at its best, the outlier
detection technique only identifies around 70%
608 Cognitive Neuropsychology, 2013, 30 (7 –8)
of dissociating cases. In Simulation 1, regardless
of the number of participants in the study, only
35% of the dissociating cases were detected,
meaning that approximately twice as many disso-
ciating cases went undetected as were identified.
In order to limit false alarms in the outlier detec-
tion technique, many dissociating cases are
missed, and these misses make interpreting the
null result difficult.
Including more participants in the study
increases the certainty that the failure to find an
outlier means that no dissociating case exists in
the sample. The simulations above stipulated
that 90% of the individuals in the sample had
the true underlying association, while the remain-
ing 10% were dissociating cases. This means that
in a sample of 10 participants, only a single disso-
ciating case was present, while in a sample of 100
participants, 10 dissociating cases were present.
While the likelihood of missing an individual dis-
sociating case was relatively high (65%) regard-
less of the number of dissociating cases in the
simulation, the likelihood of missing both outliers
in a 20-participant study is lower than the likeli-
hood of missing the single outlier in a 10-partici-
pant study, and the likelihood of missing all 10
outliers in a 100-participant study is lower still.
The results of Simulation 1 can be reanalysed to
look at the likelihood that no outliers were
detected in samples with different numbers of par-
ticipants. The percentage of data sets with no out-
liers detected fell drastically as the number of
participants in the study increases, from 60.4% of
the studies with only 10 participants to 12.9%
with 50 participants, 2.2% with 100 participants,
and 0.1% with 200 participants.
The simulation stipulates not only that disso-
ciating cases are present, but also that they are rela-
tively frequent, consisting of exactly 10% of the
cases in the sample. If these dissociating cases are
more infrequent, say affecting only 2% of the
cases in the sample, then even with large samples
it may be unlikely that even a single dissociating
case will be identified. We have no way of estimat-
ing the frequency with which certain rare patterns
are observed, and, historically, cognitive neuropsy-
chology has not been concerned by the relative
 CASE SERIES OUTLIERS

Table 1. Effect of the frequency of associated cases on the

likelihood of a significant correlation and the sensitivity of the
outlier detection method

% Mean % Significant % True outliers

Associated correlation (a ¼ .05) detected

0 2.14 ,0.1 —
10 2.09 0.2 0.3
20 2.03 1.6 0.5
30 .04 4.5 0.8
40 .12 14.1 1.4
50 .21 31.3 2.7
60 .31 57.6 6.8 Figure 6. Percentage of data sets with a significant correlation
70 .44 83.8 21.0 between perseverative and nonperseverative intrusions and
80 .58 97.0 48.9 without a single outlier detected as a function of the frequency of
Downloaded by [University of Waterloo] at 14:33 11 October 2014

90 .76 99.9 62.9 dissociating cases in the sample (number of participants ¼ 50,
98 .93 100 64.1 observations per participant ¼ 500).
100 .98 100 —

Simulation 4: Varying frequency of

frequency of different patterns of cognitive impair-
ment (e.g., Coltheart, 2001). Therefore, at best
what can be concluded from a case series study in
which no outliers are detected is that there has
yet to be evidence presented that contradicts the
theory supported by the case series.
Rareness of the dissociating pattern is not the
only cause of a failure to detect any outliers.
Indeed, having too many individuals in the
sample with the dissociating pattern can also
limit the effectiveness of the outlier detection
technique. One assumption of any outlier detec-
tion technique is that a relatively small number
of data points are produced by this alternative
mechanism (Freeman, 1980). As a result, outlier
detection techniques perform poorly in situations
in which the frequency of dissociating cases is
high. But even in data sets with large numbers of
dissociating cases, correlations supporting the
underlying association for some of the individuals
in the sample may still be significant. Simulation 4
explores the effects of the frequency of dissociating
cases on both the likelihood of finding a significant
correlation and the hit rate of detecting outliers.
dissociating cases
The fourth simulation holds constant both the
number of participants (50) and the number of
observations (500) while varying the frequency of
dissociating cases in the sample. As in Simulations
1–3, for each random data set, the correlation coef-
ficient, the significance of the correlation, and the hit
and false-alarm rates were calculated. Additionally,
the percentage of random data sets in which (a) the
correlation was significant, and (b) there were no
outliers detected was calculated.
Table 1 shows the average correlation coefficient,
the percentage of the random data sets with signifi-
cant correlations, and the percentage of dissociating
cases identified as outliers for different relative pro-
portions of dissociating and associated cases. When
the proportion of associated cases was low (,30%
of all participants), the likelihood of finding a signifi-
cant correlation between perseverative and nonper-
severative intrusions was also low (,5% of the
data sets had a significant correlation), as was the
ability to detect dissociating cases (,1% of the dis-
sociating cases were identified as outliers).4 When
the proportion of associated cases was high (.80%

4
When there is no underlying association between perseverative and nonperseverative intrusions, the correlation is slightly nega-
tive; the rates of perseverative and nonperseverative intrusions are not entirely independent in this sample. The sum of perseverative
and nonperseverative intrusion rates and the proportion of correct responses must be equal to 1.0. Therefore, if the perseverative error
rate is very high (e.g., .50%) then the nonperseverative rate must be lower (e.g., ,50%) and vice versa.

Cognitive Neuropsychology, 2013, 30 (7 –8) 609

 FISCHER-BAUM
of all participants), the correlations were significant
in nearly all of the data sets (.95% of the data
sets), and the ability to detect dissociating cases as
outliers was reasonably good (.45% of the disso-
ciating cases). Problems arose when associated and
dissociating cases were roughly equally common in
the sample. For example, when the sample had a
50–50 split of associated and dissociating individ-
uals, nearly a third of the random data sets had a sig-
nificant correlation, while only 2.7% of the
dissociating cases were detected as outliers. The
results of this simulation highlight two critical
issues when interpreting case series studies. First,
Downloaded by [University of Waterloo] at 14:33 11 October 2014
finding a significant correlation in a case series inves-
tigation does not mean that even the majority of
individuals in the sample show the underlying
association. Second, too many dissociating cases
limit the ability to detect outliers.
This second issue is further illustrated in
Figure 6. This figure plots the proportion of data
sets in which (a) there was a significant correlation,
and (b) no outliers were detected as a function of
the percentage of the sample made up of associated
cases. Such data sets are common both when dis-
sociating cases are very rare (e.g., when 98% of
the sample are associated cases, 30% of the data
sets have a significant correlation without a
single outlier detected) and when dissociating
cases are reasonably common (e.g., when 60% of
the sample are associated cases, 23% of all data
sets have a significant correlation without a
single outlier detected). The size of the correlation
coefficient can be used to distinguish these two
causes of a failure to detect an outlier. When dis-
sociating cases are rare, correlations will be very
strong, and when dissociating cases are common,
correlations will be weak to moderate. In
Table 1, the average correlation coefficient is .93
when 98% of the data are generated from associ-
ated cases and only .31 when 60% of the data are
generated from associated cases. In a high-
powered case series, if the predicted association
has a moderate correlation coefficient, and no out-
liers are detected, it is more likely that no outliers
are detected because the dissociating pattern is so
common than because the dissociating pattern
does not exist.
610 Cognitive Neuropsychology, 2013, 30 (7 –8)
SHIFTING THE DECISION
CRITERION
One potential concern with the outlier detection
technique discussed in this paper is that it is
overly conservative. In detection theory, the
decision criterion can be shifted to decrease the
number of misses, at the expense of increasing
the number of false alarms. For the previous four
simulations, the criterion is based on the conserva-
tive Bonferroni-corrected critical value, which errs
against identifying data points as outliers and
limits the number of false alarms at the expense
of increasing the number of misses. A fifth simu-
lation explores the effects that shifting the decision
criterion has on hits and false alarms in outlier
detection.
Simulation 5: Varying the decision criterion
In this simulation, 10,000 data sets were created
with 50 participants, 100 data points per partici-
pant and a 90– 10 split between associated and dis-
sociating cases. Each of the 10,000 data sets was
evaluated for outliers using 11 different uncor-
rected alpha-values for statistical testing (.2, .1,
.05, .02, .01, .005, .002, .001, .0005, .0002,
.0001). Note that included in here is an uncor-
rected alpha value (.001) that is equivalent to a
Bonferroni-correct alpha value of .05 with 50 par-
ticipants. An average false-alarm rate and hit rate
were calculated for each of these different alpha-
levels and were plotted as an ROC-curve
(Figure 7). In addition, for each decision criterion,
the proportion of the 10,000 data sets that con-
tained at least one false alarm was calculated.
There is a clear cost, in terms of false alarms,
when the decision criterion is made more
liberal—that is, by not correcting for multiple com-
parisons. With an uncorrected alpha of .05, the rate
of hits is relatively high (68%), but so is the rate of
false alarms (20%). Because associated cases are nine
times more common in each data set than dissociat-
ing cases, at this hit and false-alarm rate an individ-
ual identified as an outlier is nearly three times more
likely to be a false alarm than a hit. When using this

Figure 7. Receiver-operating characteristic (ROC) curve based on
Downloaded by [University of Waterloo] at 14:33 11 October 2014
the same sets of simulated data with outlier detection run at
different alpha levels. Highlighted in the figure is the point that
represents a Bonferroni-corrected alpha of .05 and an uncorrected
alpha of .05 (number of participants ¼ 50, observations per
participant ¼ 100, frequency of dissociating cases ¼ 10%).
uncorrected alpha, 99% of the data sets had at least
one false alarm. Without correcting for multiple
comparisons, it becomes difficult to say with any
confidence whether truly dissociating cases exist in
the data set.
In contrast, when a Bonferroni-corrected alpha
of .05 is used as the decision criterion, the hit rate
is much lower (35%), the false-alarm rate is lower
still (,0.3%), and only 11% of all data sets contain
at least one false alarm. With these hit rates and
false-alarm rates, a data point detected as an
outlier is 15 times more likely to be a truly disso-
ciating case than a false alarm. In this paper, I
have focused on outlier detection in case series
investigations with the goal of identifying single
cases who show the dissociating pattern; this
goal is better served by setting a conservative
decision criterion, as the cost of mistakenly reject-
ing the null hypothesis that all of the individuals
suffer from the same underlying deficit is high.
Incorrectly identified dissociations limit the
impact of single-case cognitive neuropsychology.
Less conservative corrections that still keep the
familywise error rate below a specified alpha-
level, like the Holm – Bonferroni correction
5
Keep in mind, an outlier should not be discarded as a false alarm simply because the researcher cannot think of a theoretical
explanation for why that individual differs from the rest of the sample.
CASE SERIES OUTLIERS
(Holm, 1979), may be suitable alternatives to the
Bonferroni correction for the outlier procedures
outlined here.
A case-series investigator may want to identify
outliers with goals other than identifying these dis-
sociating cases. For example, a researcher may want
to limit the investigation only to those individuals
who best show the underlying association. In
these investigations, a miss—including in the analy-
sis an individual who actually suffers from a differ-
ent deficit from the others in the sample—may have
more impact than a false alarm. The externally stu-
dentized residual test may still be appropriate for
outlier detection with this goal in mind, but the
decision criterion should be made less conservative.
In these sorts of studies, it may be appropriate to use
an uncorrected alpha for outlier detection.
THE ROLE OF FOLLOW-UP
TESTING IN DISTINGUISHING
MISSES AND FALSE ALARMS
As the simulations above illustrate, simply detect-
ing an outlier in a case series is not enough to con-
clude that the theory being tested is wrong; there is
always a possibility that the data point detected as
an outlier is actually a false alarm. Nickels et al.
(2011) point out the importance of follow-up
testing to determine whether cases identified as
outliers are truly dissociating cases, or simply
appear to be dissociating cases due to some con-
founding factor. For these apparent dissociating
cases, some theoretical account must be made to
explain how the mechanism that is generating
their responses differs from the mechanism gener-
ating the associations in the other cases in the
series.5 Follow-up testing, both for the cases
detected as outliers and for the other cases in the
sample, can be used to show that the case detected
as an outlier truly has a different deficit from the
other individuals in the sample (see Dell &
Schwartz, 2011, for additional discussion of this
point).
Cognitive Neuropsychology, 2013, 30 (7 –8) 611
 FISCHER-BAUM
Nickels et al. (2011) argue that the outlier needs
to dissociate on follow-up testing from all of the
others in the sample in order to demonstrate that
it is a truly dissociating case. This argument
ignores the fact that the outlier detection technique
may have missed some truly dissociating cases. As
the simulations show, the Bonferroni-corrected,
externally studentized residual test is conservative,
making the likelihood of missing truly dissociating
cases surprising high. Follow-up testing can help to
identify these misses as well. Despite not being
detected as outliers, misses will pattern with the
identified dissociating case on the follow-up
Downloaded by [University of Waterloo] at 14:33 11 October 2014
testing rather than patterning with the rest of the
sample. A sixth simulation examines the role of
additional testing in distinguishing hits from false
alarms and identifying misses.
Simulation 6: The role of additional testing
The sixth simulation took a constant number of
participants (50), number of observations (100),
and frequency of dissociating cases in the sample
(10%). The outlier detection technique was
applied to these simulated data sets. Follow-up
testing was then simulated in the simplest possible
manner: The same simulated participants, with the
same underlying rates of producing perseverative
and nonperseverative intrusions, were adminis-
tered a second experiment with 500 observations.
Outlier detection is then applied to these new
data sets, and a computer program tabulated how
many individuals identified as outliers in the first
experiment were also identified as outliers in the
second experiment.
In the first experiment, an average of 1.9 out-
liers were detected for each data set, with 1.8
being truly dissociating cases (hit rate: 35%) and
0.1 being associated cases (false-alarm rate:
,0.3%). Outliers in Experiment 1 also tended to
be outliers in Experiment 2. An average of 1.5
cases were identified as outliers in both exper-
iments (77%), and nearly all of these cases were
truly dissociating cases (.99.9%). Indeed, over
80% of the truly dissociating cases identified as
an outlier in Experiment 1 were also identified as
an outlier in Experiment 2, while this was true of
612 Cognitive Neuropsychology, 2013, 30 (7 –8)
only 1% of the associated cases. Being identified
as an outlier in both experiments is an excellent
indicator that the first result was a hit and not a
false alarm; an individual that was an outlier in
both experiments was more than 1000 times
more likely to be a truly dissociating case than an
associated case mistakenly identified as an outlier.
A second issue is whether follow-up testing can
help distinguish misses from correct rejections. In
the first experiment, an average of 48.1 individuals
were not identified as outliers; of these, 44.9 were
correctly rejected associated cases, and 3.2 were
missed truly dissociating cases. Of these individ-
uals, an average of 0.4 cases were identified as out-
liers in the second experiment (,1%). However,
the majority of the cases that were detected as an
outlier in Experiment 2, despite not being detected
as an outlier in Experiment 1, were truly dissociat-
ing cases (75%). Being identified as an outlier in
the second experiment, despite not being ident-
ified as an outlier in the first experiment, is a
decent indication that the first result was a miss
and not a correct rejection.
This simulation shows the role that additional
testing can play in distinguishing hits and misses
from false alarms and correct rejections. Outlier
detection on a second experiment provides clear
information about how to interpret the results of
the first experiment. Individuals who are detected
as outliers on both experiments are almost cer-
tainly truly dissociating cases while individuals
detected as outliers in the second experiment but
not the first were probably misses. Nickels et al.
(2011) and Schwartz and Dell (2010) argue that
follow-up testing should be used to better under-
stand the underlying deficit of the dissociating
cases. This stimulation shows the power that
even simple retesting—a direct replication of the
findings with the same individuals performing
the same task—can have in identifying truly disso-
ciating cases in a case series.
CONCLUSIONS
Case series investigations provide a clear tool for
testing predicted associations between cognitive

functions in individuals with cognitive impair-
ments. When testing for these associations, it is
critical to additionally look for dissociating cases,
individuals whose pattern of performance deviates
from the predicted association, particularly for
those studies in which a single dissociating case
can falsify the theory being tested. Determining
whether a case series includes individuals whose
data are unpredicted from the underlying associ-
ation—called outliers in the statistics literature—is
a difficult statistical problem. This paper offers
one formal approach to testing case series for out-
liers: the recursive externally studentized residual
Downloaded by [University of Waterloo] at 14:33 11 October 2014
test. Through a series of simulation studies, I have
identified some statistical concerns that arise when
testing a case series for outliers. Briefly, it is impor-
tant for case series investigations to include large
numbers of participants. Increasing the number of
participants decreases the likelihood of falsely iden-
tifying an individual as a dissociating case and
increases confidence that the dissociating pattern
is not attested when no outliers are detected.
Furthermore, it is important in a case series investi-
gation to collect many observations from each par-
ticipant. Increasing the number of observations per
participant has a sizeable effect on the likelihood of
correctly detecting the truly dissociating cases in the
sample. Finally, if outliers are detected, it is impor-
tant to do follow-up testing, in order to determine
whether the detected outlier is a truly dissociating
case or a false alarm, as well as identifying other dis-
sociating cases that may have been missed by the
outlier detection technique in a single experiment.
While the Fischer-Baum and Rapp (2012)
study of perseveration errors was used as a concrete
example of how outlier detection might be applied
to case series data, the methods proposed in this
paper should be applicable to a wide range of
case series studies. This method is clearly appro-
priate for detecting outliers in case series studies
that use linear regression analyses, in which there
is a clear prediction being tested about the causal
relationship of one of the variables on the other
(e.g., the lower a patient’s performance on test X,
6
The degrees of freedom for the generalized externally studentized residual test is n – p – 1, in which n is the total number of
data points, and p is the total number of parameters (2 in simple linear regression, .2 in multiple linear regression).
CASE SERIES OUTLIERS
the worst their predicted performance on test Y).
However, this method is not appropriate for
studies in which the hypothesis being tested does
not attribute a causal role to one of the variables
and treats all of the variables as having equal
status (see Crawford & Garthwaite, 2005, 2007a,
2007b; Crawford, Garthwaite, & Porter, 2010
for discussion of appropriate tests to employ in
these conditions).
While the example discussed in this paper looks
only at the relationship between two variables, the
externally studentized residual test generalizes to
multiple linear regression (Weisberg, 1985).
Specifically, the method generalizes to cases in
which multiple causal variables are used to make
a prediction about a single dependent variable.
As with the simple linear regression, the observed
value on that dependent variable can be compared
to the predicted value based on the other explana-
tory variables, by computing an externally studen-
tized residual and comparing that value to some
decision criteria.6 Extending this technique to
multiple regression is critical for case series inves-
tigations, as individuals in a large case series are
likely to differ on many variables (i.e., age, edu-
cation, speed of processing) that may be related
to the measures being tested. The problem of cov-
ariates is beyond the scope of the present paper,
but I refer the reader to Crawford, Garthwaite,
and Ryan (2011) for a discussion of this issue, as
well as methods for comparing a case study to a
control population in the presence of covariates.
Following all of the directives from this paper
makes for infeasibly large case series investigations;
while running hundreds of participants on dozens
of tests each composed of thousands of trials will
result in the best data for case series investigations,
a single lab has neither the time nor the money to
carry out such a study. Simulation 3 offered some
suggestions for how to balance some of these com-
peting recommendations, advocating increasing
the number of observations per participant and
the expense of including more participants. An
alternative perspective would be to make our case
Cognitive Neuropsychology, 2013, 30 (7 –8) 613
 FISCHER-BAUM
series investigations an ongoing community
project. New cases investigated in labs around
the world can be added to existing case series by
administering the same or very similar tests and
carrying out the same analysis techniques. These
new cases can be added to the case series database,
and those case series can be reassessed for both
associations and outliers. The Moss Aphasia
Psycholinguistics Project Database already oper-
ates in this spirit, making available both the tests
administered to a large sample of aphasic individ-
uals and the data from those individuals (Mirman
et al., 2010). Rapp (2011) pointed out the tempta-
Downloaded by [University of Waterloo] at 14:33 11 October 2014
tion to ignore potentially relevant case studies
when drawing conclusions from the results of a
large case series. A mechanism by which these
individual cases can be integrated to make larger
case series could minimize this temptation.
REFERENCES
Ben-Gal, I. (2005). Outlier detection. In O. Maimon &
L. Rockach (Eds.), Data mining and knowledge dis-
covery handbook: A complete guide for practitioners
and researchers (pp. 131– 146). Kluwer: Academic
Publishers.
Bub, D. (2011). Facing the challenge of variation in
neuropsychological populations: Lessons from
biology. Cognitive Neuropsychology, 28, 445 –450.
Cohen, L., & Dehaene, S. (1998). Competition between
past and present. Assessment and interpretation of
verbal perseverations. Brain, 121, 1641–1659.
Coltheart, M. (2001). Assumptions and methods in
cognitive neuropsychology. In B. Rapp (Ed.),
Handbook of cognitive neuropsychology (pp. 3– 21).
Philadelphia: Psychology Press.
Crawford, J. R., & Garthwaite, P. H. (2005). Testing for
suspected impairments and dissociations in single-case
studies in neuropsychology: Evaluation of alternatives
using Monte Carlo simulations and revised tests for
dissociations. Neuropsychology, 19, 318–331.
Crawford, J. R., & Garthwaite, P. H. (2006).
Comparing patients’ predicted test scores from a
regression equation with their obtained scores: A sig-
nificance test and point estimate of abnormality with
accompanying confidence limits. Neuropsychology, 20,
259– 271.
614 Cognitive Neuropsychology, 2013, 30 (7 –8)
Crawford, J. R., & Garthwaite, P. H. (2007a).
Comparison of a single case to a control or normative
sample in neuropsychology: Development of a
Bayesian approach. Cognitive Neuropsychology, 24,
343–372.
Crawford, J. R., & Garthwaite, P. H. (2007b). Using
regression equations built from summary data in
the neuropsychological assessment of the individual
case. Neuropsychology, 21, 611–620.
Crawford, J. R., Garthwaite, P. H., & Porter, S. (2010).
Point and interval estimates of effect sizes in the
case-controls design in neuropsychology: Rationale,
methods, implementations, and proposed reporting
standards. Cognitive Neuropsychology, 27, 245–260.
Crawford, J. R., Garthwaite, P. H., & Ryan, K. (2011).
Comparing a single case to a control sample:
Testing for neuropsychological deficits and dissociations
in the presence of covariates. Cortex, 47, 1166–1178.
Dell, G. S., Burger, L., & Svec, W. (1997). Language
production and serial order: A functional analysis
and a model. Psychological Review, 104, 123–147.
Dell, G. S., & Schwartz, M. F. (2011). Who’s in and
who’s out? Inclusion criteria, model evaluation, and
the treatment of exceptions in case series. Cognitive
Neuropsychology, 28, 515–520.
Fischer-Baum, S., & Rapp, B. (2012). Underlying cause(s)
of letter perseveration errors. Neuropsychologia, 50,
305–318.
Freeman, P. R. (1980). On the number of outliers in
data from a linear model. Trabajos de estadı́stica y de
investigación operativa, 31, 349– 365.
Goldrick, M. (2011). Theory selection and evaluation in
case series research. Cognitive Neuropsychology, 28,
451–465.
Hauser, M. D. (1999). Perseveration, inhibition and the
prefrontal cortex: A new look. Current Opinion in
Neurobiology, 9, 214–222.
Hillis, A. E., Kane, A., Barker, P., Beauchamp, N., &
Wityk, R. (2001). Neural substrates of the cognitive
processes underlying reading: Evidence from
Magnetic Resonance Perfusion Imaging in hypera-
cute stroke. Aphasiology, 15, 919–931.
Holm, S. (1979). A simple sequentially rejective mul-
tiple test procedure. Scandinavian Journal of
Statistics, 6, 65– 70.
Lambon Ralph, M. A., Patterson, K., & Plaut, D. C.
(2011). Finite case series or infinite single-
case studies? Comments on “Case series investi-
gations in cognitive neuropsychology” by Schwartz
and Dell (2010). Cognitive Neuropsychology, 28,
466–474.

Martin, N., & Dell, G. S. (2004). Perseveration and
anticipations in aphasia: Primed intrusions from
the past and future. Seminars in Speech and
Language, 26, 349 –362.
Martin, N., & Dell, G. S. (2007). Common mechanisms
underlying perseverative and non-perseverative sound
and word substitutions. Aphasiology, 21, 1002–1017.
Mirman, D., Strauss, T. J., Brecher, A., Walker, G. M.,
Sobel, P., Dell, G. S., & Schwartz, M. F. (2010). A
large, searchable, web-based database of aphasic
performance on picture naming and other tests of
cognitive function. Cognitive Neuropsychology, 27,
495 –504.
Nickels, L., Howard, D., & Best, W. (2011). On the use
of different methodologies in cognitive neuropsy-
Downloaded by [University of Waterloo] at 14:33 11 October 2014
chology: Drink deep and from several sources.
Cognitive Neuropsychology, 28, 475 –485.
Olson, A. C., & Romani, C. (2011). Model evaluation and
case series data. Cognitive Neuropsychology, 28, 486–499.
Patterson, K., & Plaut, D. C. (2009). “Shallow draughts
intoxicate the brain”: Lessons from cognitive science
CASE SERIES OUTLIERS
for cognitive neuropsychology. Topics in Cognitive
Science, 1, 39–58.
Penny, K. I. (1996). Appropriate critical values
when testing for a single multivariate outlier by
using the Mahalanobis distance. Applied Statistics,
45, 73 –81.
Rapp, B. (2011). Case series in cognitive neuropsychol-
ogy: Promise, perils and proper persepective.
Cognitive Neuropsychology, 28, 435–444.
Schwartz, M. F., & Dell, G. S. (2010). Case series
investigations in cognitive neuropsychology.
Cognitive Neuropsychology, 27, 477–494.
Shallice, T., & Buiatti, T. (2011). Types of case series –
The anatomically based approach: Commentary on
M.F. Schwartz & G.S. Dell: Case series investi-
gations in cognitive neuropsychology. Cognitive
Neuropsychology, 28, 500– 514.
Simon, J. L., & Bruce, P. (2000). Resampling stats.
Arlington, VA: Resampling Stats Inc.
Weisberg, S. (1985). Applied linear regression (2nd ed.).
New York: Wiley.
Cognitive Neuropsychology, 2013, 30 (7 –8) 615
 FISCHER-BAUM

APPENDIX A

Matlab function for computing externally studentized residuals

function [studentized_residuals] ¼ externally_studentized_residuals(Data, alpha)
% Input is DATA – a N X 2 matrix for N subjects in a case series, with x and y values in each column and alpha –
the significance level for rejecting the null hypothesis, typically .05
% externally_studentized_residuals: Detection of outliers in a linear regression
% Computes an exernally-studentized residual for each subject in an analysis and compares that to a critical
value from a Bonferroni corrected alpha
% Output is two columns. Left column is the studentized residual for each case.
% The right column marks with (-1) values significantly below predicted value and(+1) for values significantly
higher than the predicted value.

[Number_of_subjects Dimensions_in_Data] ¼ size(Data);

Downloaded by [University of Waterloo] at 14:33 11 October 2014

studentized_residuals ¼ zeros(Number_of_subjects, 2);

Critical_values ¼ tinv([(alpha/2)/Number_of_subjects 1-(alpha/2)/Number_of_subjects],Number_of_subjects-

Dimensions_in_Data-1-1);

reg_results ¼ regstats(Data(:,2),Data(:,1),’linear’,’studres’);

studentized_residuals(:,1) ¼ reg_results.studres;

for i ¼ 1:Number_of_subjects
if studentized_residuals(i) . Critical_values(2)
studentized_residuals(i,2) ¼ 1;
elseif studentized_residuals(i) , Critical_values(1)
studentized_residuals(i,2) ¼ -1;
else
end
end
end

APPENDIX B

Matlab function for recursive application of

the externally studentized residual test
function [recursive_outlier_detection] ¼ externally_studentized_recursive(Data, alpha);
% Input is DATA – a N X 2 matrix for N subjects in a case series, with x and y values in each column and alpha –
the significance level for rejecting the null hypothesis, typically .05
% Program applies the externally_studentized_residuals program recursively.
% It outputs a single column, with one value for each case in the case series.
% A value of 0 means that the individual is not identified as an outlier
% A value of +1 means that the individual is an outlier in the positive direction
% A value of -1 means that the individual is an outlier in the negative direction

616 Cognitive Neuropsychology, 2013, 30 (7 –8)

 CASE SERIES OUTLIERS

[recursive_outlier_detection] ¼ zeros(length(Data),1);
[studentized_residuals] ¼ externally_studentized_residuals(Data, alpha);
DataPrime ¼ Data;
IndexData ¼ [1:1:length(Data)];
while sum(abs(studentized_residuals(:,2))) . 0
DataPrime2 ¼ [];
IndexData2 ¼ [];
for i ¼ 1:length(studentized_residuals)
if studentized_residuals(i,2) ¼¼ 0
DataPrime2 ¼ vertcat(DataPrime2, DataPrime(i,:));
IndexData2 ¼ horzcat(IndexData2, IndexData(i));
else
recursive_outlier_detection(IndexData(i)) ¼ studentized_residuals(i,2);
end
end
DataPrime ¼ DataPrime2;
Downloaded by [University of Waterloo] at 14:33 11 October 2014

IndexData ¼ IndexData2;
[studentized_residuals] ¼ externally_studentized_residuals(DataPrime, alpha);
end
end

Cognitive Neuropsychology, 2013, 30 (7 –8) 617