See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/235412093

The Problem of Circularity in Evidence, Argument, and Explanation

Article  in  Perspectives on Psychological Science · March 2011

DOI: 10.1177/1745691611400240

CITATIONS READS

42 1,708

1 author:

Ulrike Hahn
Birkbeck, University of London
188 PUBLICATIONS   3,902 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Rational Argument View project

Argumentation View project

All content following this page was uploaded by Ulrike Hahn on 17 February 2015.

The user has requested enhancement of the downloaded file.

 Perspectives on Psychological Science
6(2) 172–182
The Problem of Circularity in ª The Author(s) 2011
Reprints and permission:
Evidence, Argument, and Explanation sagepub.com/journalsPermissions.nav
DOI: 10.1177/1745691611400240
http://pps.sagepub.com

Ulrike Hahn
School of Psychology, Cardiff University, United Kingdom

Abstract
A number of recent articles (Fiedler, 2011, this issue; Gigerenzer, 2009; Kriegeskorte, Simmons, Bellgowen, & Baker, 2009; Vul &
Kanwisher, 2010) have highlighted seemingly circular arguments and explanations in psychological research, which suggests that
the problem is rife within psychology. The article reviews the literature on circularity, and, in light of this, evaluates these recent
examples from psychology to determine whether it is indeed circularity that is the underlying problem and to make suggestions
for what paths improvements might pursue.

Keywords
circular arguments, circular explanation, circular analysis

‘‘That is completely circular...’’ is one of the most damning
things a critic can say. No further evidence is typically needed
to establish that a thought, an argument, or an explanation is
worthless. Circular reasoning or ‘‘question begging’’ is the ulti-
mate reasoning fallacy, so it is of considerable concern that
recent articles have identified circularities in psychological
research from a range of different areas (Fiedler, 2011, this
issue; Gigerenzer, 2009; Kriegeskorte, Simmons, Bellgowen,
& Baker, 2009; Vul & Kanwisher, 2010), suggesting that the
error is widespread.
The more severe an accusation, the more important it is that it
is well-founded. Without a clear understanding of the notion of
circularity, psychology runs the risk of either unfairly rejecting
work that is actually sound or of rejecting work on the wrong
grounds. Misconceptions about the nature of a problem seem
undesirable even in those cases where it is obvious that a prob-
lem exists, because misconceptions are likely to impede prog-
ress: Inaccurate diagnoses rarely prompt successful remedies.
Either way, the prevalence of the charge of circularity
within psychological research suggests a need for greater
awareness of circularity. Hence, this article reviews the notion
of circularity in detail and critically reevaluates recent exam-
ples of putative circularities. The article proceeds in two main
parts. The first half provides the necessary theoretical treatment
of circularity. The concept is more difficult than one might
first presume and has consequently received considerable
amounts of theoretical attention. This section identifies the
formal characteristics of circularity and demonstrates how
and why not all circularities are equally poor. The second half
of the article then draws on this material to examine a range of
recent examples from cognitive psychology, social psychol-
ogy, and cognitive neuroscience.
What Is Circularity?
Circular arguments: What are they?
Circular arguments are the most well known of the so-called
fallacies of reasoning or argumentation. The fallacies are traps
for unwary reasoners: They might fool the inexperienced into
finding them persuasive, but they do not provide sufficient rea-
son for a claim. The most straightforward example of a circular
argument is this:
(1) God exists, because God exists.
Here, the reason or evidence given for a claim is simply a
(re)assertion of the claim itself.
More frequent in practice are so-called self-dependent justi-
fications such as the following standard textbook example:
(2) God exists because the Bible says so, and the Bible is the
word of God.
Here, the ‘‘evidence’’ presupposes the truth of the claim it is
trying to support. Logically, the Bible cannot be the word of
Corresponding Author:
Ulrike Hahn, School of Psychology, Cardiff University, Tower Building Park
Place, Cardiff, CF10 3AT United Kingdom
E-mail: hahnu@cardiff.ac.uk
Circularity
God if God does not, in fact, exist. Hence, the evidence depends
on the self-same claim it intends to support.
The third kind of circular argument involves the use of
terms that are identical or ‘‘clearly convertible’’ (Aristotle,
Prior Analytics II, p. xvi); a common textbook example of this
follows:
(3) Opium induces sleep because it has a soporific quality.
Here, an argument uses two terms that, in fact, mean the same
thing. What superficially seem like two distinct propositions
(‘‘opium induces sleep’’ and ‘‘opium has a soporific quality’’)
are semantically and, hence, logically equivalent. It is logically
impossible for something to be sleep inducing but not have
soporific qualities and vice versa.
Note that circular arguments are logically valid arguments.
Trivially, any statement logically implies itself: From the fact
that p, we can infer p. Likewise, logically equivalent statements
imply one another. Self-dependent justifications are logically
valid once the presupposition (an implicit premise) is spelled
out. This defining feature of circular arguments sets them apart
from other fallacies in the catalog, none of which constitute
logically valid argument forms.1
Circular arguments: When and why are
they poor?
Logical validity, in general, is a sign of maximum argument
strength and, historically, logic provided the sole normative
standard for the evaluation of arguments (see, e.g., Hamblin,
1970; Heysse, 1997). Hence, the fact that circular arguments
are deductively valid has been puzzling to philosophers, given
that they nevertheless seem so poor as arguments. Furthermore,
the more transparent their validity, the poorer they seem:
(1) seems a worse argument than (2).
As a consequence, the philosophical literature contained
widely diverging opinions that remained more or less unrecon-
ciled. Some authors denied that circular arguments were indeed
fallacious precisely because they are logically valid (Robinson,
1971). More widely shared, however, was the opposite
perspective that they were always fallacious (e.g., Copi, 1986).
A third group, finally, sought to argue that most, but not all,
instances of circularity are fallacious (e.g., Betz, 2010; Colwell,
1989; Kahane, 1986; Sanford, 1972; Walton, 1985, 1991).
What characterizes the middle ground suggesting that circu-
lar arguments might sometimes be acceptable is a wider view
toward argumentative contexts. It is consideration of the gen-
eral practical, or pragmatic, function of arguments that enables
solution of the circularity puzzle. As standard definitions of
argumentation make clear, the function or goal of argumenta-
tion is to rationally convince others of a position (see, e.g.,
Perelman & Olbrechts-Tyteca, 1969; van Eemeren & Grooten-
dorst, 2004).2 Argumentation, by definition, takes place where
people do not (yet) agree.
Argumentation is fundamentally about belief change. Once
this is taken into account, analysis of circular arguments readily
173
falls into place. The reason (1) fails as an argument is that it
cannot bring about any change in the addressee’s belief
(Hahn & Oaksford, 2006, 2007; Hahn, Oaksford & Corner,
2005). Though deductively valid, it falls short of the ultimate
goal of argumentative discourse: Whatever evaluation is
attached to the premise is transmitted to the conclusion,
where it remains the same; no increase in degree of belief
takes place.
It is worth stressing that logical inferences can straightfor-
wardly be coupled with changes in degree of belief for a con-
clusion.3 However, when belief change and logical validity
dissociate, arguments are generally viewed as poor—not just
in the case of circularity (for other examples and further discus-
sion, see Oaksford & Hahn, 2007).
Once belief change is considered, the case of self-dependent
justification (2) emerges as distinct from the direct restatement
case (1) above. This difference is important because self-
dependent justification is, as philosophers had noted (e.g.,
Brown, 1993, 1994), widespread throughout science:
(4) Electrons exist because their signature effects can be seen
in a cloud chamber.
shares the logical structure of (2)
God exists, because the Bible says so and the Bible is the word
of God.
but is a typical example of the kinds of inferences scientists
draw about entities we cannot directly observe. The evidence
in both cases presupposes the very conclusion that is being
asserted. Wholesale rejection of all circularities would conse-
quently rule out much of current scientific practice.
However, with the right formal tools it can be shown how
self-dependent justification cases differ fundamentally from
direct restatements such as (1); furthermore, the intuitive dif-
ference in relative strength between (2) and (4) can be
explained. The probability calculus, and with it Bayes’ theo-
rem, provides a normative framework for belief revision—that
is, how evidence should change degrees of belief. It has conse-
quently become the standard tool by which to evaluate the
impact of evidence both within philosophy (e.g., Bovens &
Hartmann, 2003; Earman, 1992; Howson & Urbach, 1993)4
and within the longstanding tradition of psychological investi-
gation of people’s actual response to evidence (e.g., Edwards,
1968; Peterson & Miller, 1965; Peterson, Schneider & Miller,
1964; Tversky & Kahneman, 1974; readers wanting to remind
themselves how belief change relates to evidence in this frame-
work and what makes evidence diagnostic or informative
should see the Appendix). This analytic framework has been
applied to the classic catalog of fallacies of argumentation
(e.g., Hahn & Oaksford, 2006, 2007; Korb, 2004; Oaksford
& Hahn, 2007) on the basis that arguments such as (1) to (4)
above can be viewed as claims or hypotheses (‘‘God exists’’)
and reasons or evidence for those claims (‘‘The Bible says so
and the Bible is the word of God’’).
174
Crucially, it has been shown that the presence of circulari-
ties does not preclude increases in degree of belief: Arguments
can involve circular dependencies but nevertheless provide
probabilistic support (Atkinson & Peijnenburg, 2010; Hahn
& Oaksford, 2006, 2007; Shogenji, 2000).
The traditional, purely logical analysis has obscured this fact
because in classical logic, statements are simply either true or
false. This means that a claim already has to be assumed to be
true in order to be invoked as a premise in an actual argument,
leaving no room for changes in belief for that claim considered
as a conclusion. However, from a probabilistic perspective, a
claim can be held tentatively, and further information can
increase our belief in that claim, even if this information some-
how involves the very same claim.
From a probabilistic perspective, there is a fundamental dis-
tinction between direct restatements and other types of circular
arguments. Direct restatements (‘‘God exists, because God
exists’’) are maximally poor because the reason is the claim;
no further information is provided that would allow a change
in degree of belief. There is, however, further information in
the case of self-dependent justification as in (2) and (4), and
as a result the circular dependence does not preclude changes
in degree of belief. Although the evidence cannot be true if the
conclusion is not (i.e., the Bible cannot be the word of God if
God does not exist), actually obtaining the evidence can conse-
quently increase our posterior degree of belief.
Technically, the conclusion is involved in interpreting the
evidence (Shogenji, 2000). There is uncertainty associated with
the desired interpretation of the evidence: The Bible might not
be the word of God, and what is seen in the cloud chamber
might not be the signature effects of electrons. Hence, this
interpretation cannot simply be assumed to be true. This uncer-
tainty can be captured through a hierarchical, probabilistic
model, of which the uncertain interpretation of the data is an
intermediate variable (see Hahn & Oaksford, 2007, for details
and discussion) in a way that is directly analogous to standard
treatments of uncertain evidence in other contexts, such as tes-
timony (see e.g., Pearl, 1988; Schum, 1981). Testimony is a
report of evidence as opposed to direct observation of that evi-
dence: for example, a nurse reporting the outcome of a labora-
tory test. One cannot in this case take the test outcome as given
and use Bayes’ theorem to calculate a posterior degree of belief
on its basis, as the person reporting may be less than fully reli-
able.5 One can, however, take as given the report of the out-
come in question and calculate a posterior degree of belief
given that report. The test outcome itself is then an intermediate
(not directly observed and hence uncertain) variable between
hypothesis and report. In all other ways, belief revision pro-
ceeds as normal, and the only consequence of the extra level
of indirection is that the belief change brought about will be
smaller than if the test outcome had been observed directly
(or the witness was known to be perfectly reliable).
The self-dependent justification case is analogous: Staring
at the cloud chamber, one cannot simply assume that the traces
one is seeing are the signature effects of electrons, not only
because the existence of electrons is the claim in question, but
Hahn
because even if they do exist, what the cloud chamber shows
may not actually be caused by electrons in the way our theories
assume. The desired interpretation of what one is seeing, how-
ever, is made more likely by our observation, and that, in turn,
increases our degree of belief in the claim.
Observing things in the cloud chamber thus rightly makes us
more convinced that electrons exist, and there is nothing wrong
with such scientific practice, despite the circularity involved.
Moreover, it becomes clear why (2) seems weaker than (4).
The overall strength of each argument is determined by the
probabilistic relationships between each of the variables. The
fact that there are plausible alternative explanations for the
Bible other than that it is the word of God will lower the prob-
abilities one assigns to the positive interpretation even where
the hypothesis is true. Lowering this conditional probability
will lessen the belief change brought about. In other words, the
argument will seem weaker as long as one finds it plausible that
the Bible might not be God’s literal word (even if God exists)
but have little in the way of alternative interpretations for what
is seen in the cloud chamber.6
Finally, the (logical) self-dependence inherent in these argu-
ments manifests itself only in one particular way: If the hypoth-
esis is false, then, by logical necessity, the interpretation of the
data that would confirm the hypothesis is false as well (i.e., the
Bible cannot be the word of God if God does not exist, nor can
one see signature effects of electrons if there is no such thing).
This dependency places a cap on the degree of belief one can
have in this interpretation: Because the desired interpretation of
the data logically requires the hypothesis, it can be no more prob-
able than the hypothesis itself. This in turn limits the strength of
the inference from data to hypothesis, whatever the probability of
the data itself. In other (i.e., non-self-dependent) cases of hier-
archical modeling, such as the testimony example, no such cap
is in place. The consequence of this is that observing the data can
have a greater confirmatory effect (see Hahn & Oaksford, 2007).
Thus, a probabilistic analysis clarifies also that there is a restric-
tion on how strong self-dependent arguments can be. A self-
dependent argument for a claim will never provide the theoreti-
cally strongest possible reason, though it might be the best reason
actually available.
In summary, whereas direct restatements are viciously
circular, and hence utterly worthless as arguments, this is not
true of self-dependent justifications. What then of circularities
involving interchangeable terms such as those seen in (3)?
These have not been analyzed in detail within the recent philo-
sophical literature; however, the preceding discussion makes
the answer clear. As noted, swapping terms that are inter-
changeable in meaning will lead to statements that are logically
equivalent. Logically equivalent statements, however, must—
on pain of inconsistency—be assigned equivalent degrees of
belief. One cannot be more convinced that opium ‘‘makes one
sleep’’ than that it is ‘‘soporific.’’ Hence, this case parallels
direct restatements such as (1); there is nothing here to rationally
change degrees of belief with.
However, it seems questionable whether many genuine
examples of semantic interchangeability exist. Different ways
Circularity
of saying things are typically not equivalent (see, e.g., Ooi,
1998; Sher & McKenzie, 2006). They tend to differ in when
they are used, and seeming synonyms rarely mean exactly the
same thing. Likewise, definitions are rarely entirely accurate.
For example, one might take the word bachelor to be defined
as, and hence mean, ‘‘unmarried man,’’ but examples can read-
ily be generated where only one of these clearly applies:
(5a) The Pope is an unmarried man.
(5b) The Pope is a bachelor.
Whereas (5a) is patently true, (5b) generates some unease in that
the Pope lacks other features typically associated with bachelors
(Lakoff, 1987). As a further example, one might imagine some-
one whose marriage was by legal technicality null and void, but
who had spent decades living with his ‘‘wife.’’ Full semantic
equivalence between terms thus seems exceptional. In its absence,
however, there is no logical implication relation between state-
ments containing those terms, and hence also no circularity.
In conclusion, the circular arguments that have been identi-
fied by logicians and philosophers as fallacies of reason are
deductively valid arguments. Not all types of these circular argu-
ments, however, involve ‘‘vicious circularities.’’ Although mere
restatements are worthless, self-dependent justifications can sup-
port rational change in degrees of belief and thus be convincing.
How convincing they are depends (as with non-circular argu-
ments) on the diagnosticity of the evidence involved.
Circularity: Arguments and explanations
The preceding overview of the philosophical literature on cir-
cularity has, in line with the emphasis of that literature, focused
on circular argumentation. Circularities, however, might also
readily arise in explanation, as will be seen in some of the
psychological examples. Should they be viewed as equally
troubling in the context of explanation?
Arguments and explanations not only share close conceptual
links in that both constitute reasons,7 they are often hard to tell
apart in practice. ‘‘Because’’ is used in natural language to pre-
cede both the provision of argumentative (evidential) support
and the provision of an explanation. Moreover the same state-
ments could be linked by an explanatory or by an evidential
support relationship, as in the following:
(6a) There is a bug in the system, because the computer crashed.
(6b) The computer crashed, because there is a bug in the
system.
Which relationship is intended may require both context and
background knowledge to disambiguate. Both types of reasons
are concerned with the question ‘‘Why?’’, but reasons that are
arguments seek to address why a claim should be endorsed, and
reasons that are explanations seek to elucidate why something
occurred or is the way it is. As commonly noted, arguments are
designed to increase conviction (or knowledge), and explana-
tions are designed to increase understanding (see, e.g., Hempel,
175
1965). Consequently, one can seek explanation for something
already completely believed (i.e., a theory for data), whereas it
seems pointless to collect further evidence for a claim that is
already fully accepted. That said, it is a commonly cited property
of explanations that they enhance belief in the occurrence of a
phenomenon (see, e.g., Lombrozo, 2006), and whether there
is a sense in which ‘‘explanatory goodness’’ should increase
the degree of belief associated with a theory above and
beyond that brought about by the supporting evidence itself
is presently the focus of a lively debate about inference to
the best explanation within philosophy (e.g. Harman,
1965; Lipton, 1991; Okasha, 2000; van Fraassen, 1989;
Weisberg, 2009).
Although explanation and argumentation are closely linked,
they are nevertheless distinct and serve different goals. Hence,
the extent to which circularity is damaging may differ across
argument and explanation (see also Walton, 2006). It thus
seems necessary to consider in slightly more detail what consti-
tutes ‘‘good explanation.’’
Given the close ties between evidence and explanation, it
seems unsurprising that there are historical parallels in the the-
orizing about each area. In the same way that argument quality
was long dominated by the measure of logic, scientific expla-
nation was viewed as a deductive process, whereby a phenom-
enon to be explained formed the conclusion of a logical
inference involving at least one general ‘‘law’’ of nature that
was true (e.g., Hempel, 1965; Hempel & Oppenheim, 1948).
This turned out to be problematic (see, e.g., Okasha, 2002);
consequently, for explanation too, it was argued that attention
must be paid to the goals or functions that explanations serve
(e.g., Koertge, 1992) and that pragmatic considerations might
play a role. Specifically, pragmatic views of explanation have
noted that what counts as explanatory is determined, in part, by
relevant contrasts a speaker has in mind. Explanations are
answers to a question (Sintonen, 1999; van Fraassen, 1980).
Hence, the question ‘‘Why did the DOG bury the bone?’’ prag-
matically requests a different answer than the question ‘‘Why
did the dog BURY the bone?’’ and so on. However, explanation
seems distinct from argument in the extent to which theories of
explanation have emphasized the role of causation (e.g.,
Ruben, 1990; Salmon, 1984).
Theorizing about explanation has also developed a list of
‘‘explanatory virtues’’ that characterize good explanations, in
particular: simplicity, elegance, unification of otherwise dispa-
rate theories or data, support for prediction, and coherence with
other theories or bodies of knowledge (see, e.g., Weisberg,
2009). These virtues suggest that there is more to good expla-
nation than causation (see e.g., Lombrozo, 2006). They also
distinguish explanation from argumentation in that not all of
these attributes readily apply, or seem equally desirable, in the
case of arguments.
In sum, although it is clear that explanations seek to enhance
understanding, what form explanations should take, and what
makes them ‘‘good,’’ seems, at present, beyond a simple answer,
and involves a whole range of considerations. Nevertheless, it is
clear that a direct restatement such as the following:
176
(7) I went to the store, because I went to the store.
is as unsatisfactory as an explanation as it is as an argument
because it already violates the most basic goal of explanation:
creating understanding. In the same way that it cannot effect
changes in belief when considered as an argument, it cannot
increase understanding, because no new information is offered.
Given the obvious defectiveness, however, such examples
seem unlikely to arise in scientific practice. More likely is the
possibility that data are ‘‘explained’’ by re-describing them—
that is, a circularity in the form of (3) above. Indeed the first
examples from psychological research are potentially of this
kind.
Circularity Within Psychology
The recent examples in the psychology literature fall naturally
into three categories: examples of circular explanation, exam-
ples of circular data analysis, and, finally, circular methods
more generally. These will be examined in turn.
Circular explanations
Examples of purported circular explanations are described in
Gigerenzer’s (2009) rather despairing assessment of the sci-
ence of psychology. Gigerenzer sees much of psychology as
engaged in ‘‘theoretical minimalism,’’ in which little attempt
at theory development is made, and research ‘‘limps along on
one leg and a crutch: the one leg is experimentation, the crutch
is a surrogate for theory.’’ (p. 21). Gigerenzer maintains that
such theory surrogates can, in contrast to genuine theory,
account for almost everything after the fact but cannot make new
predictions and are so vague that they cannot be wrong. One of
the three kinds of surrogate theories that he identifies is ‘‘circular
restatements,’’ such as (3), whereby the explanation is the phe-
nomenon, merely couched in slightly different terms. He identi-
fies several examples of this in current research, most notably
‘‘matching’’ (Evans, 1998) and ‘‘availability’’ (Tversky &
Kahneman, 1974):
. . . When participants were given the Wason selection task and
a rule ‘‘if there is an A on one side, then there is [not] a 3 on the
other,’’ the finding was that many checked the ‘‘A’’ and the ‘‘3’’
card, whether the ‘‘not’’ was present or not. This observation
was explained by a ‘‘matching bias,’’ which means that the
participants consider the two cards mentioned in the rule as rel-
evant information. These explanations are as close to circular
restatements as they can be. When two different external repre-
sentations of the same information (framing) resulted in differ-
ent performances, the restated explanation was that one of these
makes the solution more ‘‘transparent’’ or ‘‘salient.’’ That a
representation makes a solution transparent is hardly an
explanation, however, but rather what needs to be explained.
. . . Finally, the aforementioned availability heuristic was
invoked to explain various cognitive biases, but recently the
term availability bias has come to be used interchangeably.
Here, the same term refers to the phenomenon and its
Hahn
explanation. Like one-word explanations, circular restatements
are immune to improvement. They can’t be wrong. (p. 22).
Data, theory, and explanation. However, do these examples
really involve circular re-descriptions like (3)? Specifically,
do they involve the necessary logical equivalence? In both the
matching and availability examples, psychologists have used
the same name to refer to an empirical phenomenon and to a
mental heuristic put forward as an explanation (in some cases
within the same article; see Evans, 1998). This may be confus-
ing, but there is no semantic equivalence, and hence no circu-
larity, if the same name is used to refer to two very different
things. Closer reading of the texts confirms that there are mul-
tiple concepts at work here. For Tversky and Kahneman
(1974), heuristics are clearly mental ‘‘rules’’ (p. 1124) or stra-
tegies that people actively use in completing a task. Asserting
that people draw on the availability heuristic is to claim that
(in some situations) ‘‘people assess the frequency of a class
or the probability of an event by the ease with which instances
or occurrences can be brought to mind’’ (p. 1127). This process
in turn gives rise to systematic errors or ‘‘bias’’—that is, viola-
tions of normative responding. These behavioral errors arise
because though availability is correlated with frequency, it is
also affected by other factors. In Tversky and Kahneman’s
words: ‘‘The reliance on availability leads to predictable
biases’’ (p. 1127) of which ‘‘biases due to the retrievability
of instances’’ or ‘‘biases due to the effectiveness of the search
set’’ are specific examples. Though functionally related, the
mental heuristic and the biases are not the same thing: Applica-
tion of the heuristic causes error.
The same is true of Evans (1998): Performance on a range of
different tasks (the selection task, the truth table task, perfor-
mance on tasks involving not just a standard conditional but
also ‘‘only if’’ or universals, etc.) is unified in the sense that
it is viewed as representing a single phenomenon; this in turn
is explained by a heuristic strategy that people supposedly use.
It is essential to distinguish clearly between what the authors
are offering as data, theory, and explanation. The data are the
performance patterns of individual participants in particular
tasks. Statistical inference is used to establish an inference
about an underlying population, and data from different stud-
ies, involving quite different tasks, are then linked by a theory
that claims that data across these tasks are manifestations of a
single, general phenomenon. All of these data are then
explained by positing a mental rule—a heuristic—that partici-
pants invoke in generating their behavior, that is, a mental rep-
resentation that functions causally in determining performance.
The logical implication relations necessary for circularity
are absent here at every step. The theoretical claim about a gen-
eral phenomenon is not implied by the data and could easily be
false. For example, experimental findings may not replicate
with other materials (as has in fact been argued in the case of
availability bias due to effectiveness of the search set; Sedlme-
ier, Hertwig, & Gigerenzer, 1998). Likewise, the explanation by
internal heuristic is in no way logically implied by the wider
theoretical phenomenon itself, let alone the data; other,
Circularity
alternative explanations for those data are possible and have
been suggested (e.g., information gain in the context of the
matching bias; see Evans, 1998, for discussion).
Rules and regularities. The availability and matching exam-
ples do not involve circularity; rather, they are typical instances
of behavioral explanation through mental rules. Crucial to rules
is the distinction between rules and regularities, or the distinc-
tion between rule-governed (rule-following) behavior and
behavior merely described by rules (see e.g., Hahn, 2003).
Rule-following is exemplified by legal systems: Here, docu-
ments encoding the law cause particular behaviors such as pay-
ing taxes. Merely rule-describable, but not rule-guided,
behavior is exemplified by planetary motion: Planetary orbits
are well-described by physical laws, but planets do not them-
selves consult these laws to guide their behavior. Statements
of the rules of planetary motion do not function causally in the
generation of the planets’ behavior, so planetary motion is
unaffected by whether or not the appropriate physical laws are
known; legal rules, by contrast, must be known for them to pro-
duce the behaviors in question.
The availability and matching heuristics are no different to
the many rule-based explanations that have been proposed in
other domains. Languages, for example, possess regularities
such as the fact that the past tenses of the majority of English
verbs are formed by adding /-ed/ to the stem (walk ! walked);
a simple explanation for this regularity is that speakers possess
an internal mental rule (‘‘add /-ed/ to stem to form past tense’’),
which is applied, albeit unconsciously, when a past tense is pro-
duced (e.g., Pinker, 1991). This internal rule is distinct from the
regularity itself—hence, there is no circularity present. The
regularity is a data pattern, the rule is an agent-internal mental
representation that functions causally in the generation of
behavior. Other, alternative explanations for such regularities
typically exist, and psychology has a long tradition of seeking
to distinguish empirically between mental rules and other alter-
native mechanisms (e.g., similarity-based generalization).
Though this is difficult (for discussion, see Hahn & Chater,
1998), it is possible. Testimony to this is the fact that the once
nearly universally held view that cognition is generally rule
based has long been abandoned. There is contemporary debate
about rule use for very specific tasks only.
In conclusion, explanation by heuristics as mental rules is
not circular. It is true that, in cognitive contexts at least, men-
tal rules are a kind of default explanation, in that any observed
regularity in behavior could be based on a mental rule encod-
ing the regularity itself (see Hahn, 2003). However, whether
or not it actually is seems, in first instance, an empirical
question. It is unclear how considerations of interest or
informativeness enter into that. Nor is it clear on what grounds
some heuristics and mental rules should be seen as ‘‘interesting’’
but others should not.
If matching and availability heuristics are indeed unsatisfac-
tory as theories and explanations, the reasons for this have yet
to be elucidated. As seen in the previous section, the question
of what scientific explanation should consist of presently has
no agreed-upon answer. Developing criteria for good
177
explanation in psychology thus seems imperative: It is hard
to see how psychology can ultimately fulfill its goals without
clear articulation of what they are.
Circular analysis
Circularity has also recently been identified as the underlying
problem for certain procedures in data analysis (Kriegeskorte
et al., 2009; Vul & Kanwisher, 2010).
Preselection of data and statistical independence. Recent
scrutiny of the fMRI literature, prompted in part by implausibly
high observed correlations between measures of brain activity
and behavior, has revealed a worrying practice: Considerable
numbers of studies effectively preselect data points on the basis
of their ability to maximize some statistical criterion and then
use those same data points to conduct, in full, the statistical
analysis in question (Kriegeskorte, et al., 2009; Vul, Harris,
Winkielman, & Pashler, 2009; Vul & Kanwisher, 2010). This
‘‘double-dipping’’—the use of the same data for selection and
a subsequent selective analysis—bears resemblance to the clas-
sic circularities discussed earlier, in that the preselection (pre)-
determines more or less entirely what is subsequently found. In
the extreme, were, for example, only regions of interest (ROIs)
displaying a correlation of .8 with the phenomenon of interest
to be pre-selected, then the subsequent analysis would ‘‘dis-
cover’’ exactly that correlation of .8.
However, closer consideration suggests that ‘‘circular anal-
ysis’’ cases are not typically circular arguments in the technical
sense introduced in the first half of this article (but see Vul &
Kanwisher, 2010). Examples provided by Vul et al. (2009)
generally have the following structure: The researcher prese-
lects ROIs with a correlation above a threshold of .7; the
researcher then establishes the mean correlation within that
preselected set, and finally reports the observed value, say for
example, .835. This does not involve a deductively valid
inference. It is an empirical question what mean correlation
is ‘‘observed’’; logical necessity requires only that this mean
be greater than the selection threshold (i.e., 7). Moreover, the
process is neither explanation nor argument: The pre-
selection phase is not evidence offered in support of the claim
that the mean correlation in ROIs is .835, nor are preselection
or analysis offered (by the researcher!) as an explanation for
one another.
In contrast to standard cases of circular argument such as
(1), (2), and (3) above, the dependent ‘‘claim’’ that is being
established is itself data or evidence. It is the intrinsic value
of that evidence that is problematic in these studies, not the
relationship between evidence and claim. Accordingly, there
is a subtle difference in the pragmatic deficiencies present. In
standard viciously circular arguments, a disputed claim is not
made more probable. However, in the nonindependent data
analysis case, the claim about the data (the outcome of the
subsequent, final analysis) can be entirely true and beyond
doubt. In contrast to standard, viciously circular arguments, the
violation of independence in data analysis seems, at the most
general level, a pragmatic violation of what one might call the
178
logic of discovery. In effect, something is being ‘‘found’’ only
because it was put there in the first place. This difference to
viciously circular arguments matters, because it is crucial in
establishing exactly why certain procedures are bad.
In the first instance, there is nothing inherently wrong with
selecting data points that maximize a certain statistical criter-
ion. If one were to find it informative that a subset of voxels,
ROIs, or participants displayed correlations above a certain
threshold, there would be nothing wrong either with empiri-
cally establishing this or clearly reporting it as such. Viewed
simply as data, it might or might not be interesting that a certain
proportion of voxels, ROIs, or participants display a correlation
of .8 or more with a particular predictor. Whether or not it is, is
an empirical matter. Hence, the two excellent papers that have
brought these issues to light, Vul et al. (2009) and Kriegeskorte
et al. (2009), contain simulations to provide baseline compari-
sons. If even random data are likely to contain data points with
high correlations, for example, then finding such data points in
actual empirical studies seems uninformative. It is not wrong,
but simply pointless.
What is wrong, however, is any description of this that does
not make absolutely clear that one has simply selected data
points to maximize a given statistical criterion. This error is a
pragmatic error of misrepresentation. Disguising the fact that
one has put something there when reporting its discovery is,
in communicative terms, disingenuous and threatens the ability
of those in receipt of the information to assess and appreciate
the discovery’s actual value.
Finally, it would also be wrong to report a p value for the
final mean correlation, because the requirement of random
sampling on which calculation of p values rests has not been
met (see Kriegeskorte et al., 2009; Vul et al., 2009). In statisti-
cal parlance, there would be a violation of independence.
Acceptable and unacceptable preselection. Beyond that, how-
ever, it is not clear that the statistical problems of ‘‘putting
things there’’ via preselection have been solved. Kriegeskorte
et al. (2009) suggested that ‘‘ . . . a circular analysis is one
whose assumptions distort its results’’ (p. 540). This character-
ization extends beyond cases of double-dipping and raises the
question of what other practices might be questionable. At the
same time, however, Kriegeskorte et al. stressed that not all
preselection is bad:
selective analysis is a powerful tool and is perfectly justified
whenever the results are statistically independent of the selec-
tion criterion under the null hypothesis (p. 535).
Null hypothesis testing relies on the assumption that samples
are randomly, and hence measure-independently, sampled
from the underlying population. Selecting further within a
sample violates the logic of significance testing if the resultant
selection is no longer equally representative of the population
given the null hypothesis were true. As Vul and Kanwisher
(2010) noted: ‘‘If X is a random variable representing our data,
and P(X) reflects the probability distribution assumed by the
null hypothesis, then P(X|C) where C is the selection criteria,
Hahn
must be equal to P(X), the null hypothesis distribution.’’ Where
selection distorts a sample, statistical analysis will no longer
provide a legitimate inference about the population that it pur-
ports to be about.
However, how can one know in practice whether or not this
criterion is met? The null distribution is typically a hypothe-
tical distribution (see e.g., Howell, 2007). Often it is strictly
speaking known to be false even before testing begins (e.g.,
Jones & Tukey, 2000). Thus it is not at all obvious how criter-
ion independence can be ascertained. Indeed, in discussions
about the value of null hypothesis testing in general, it has been
argued that—when it comes to human beings—correlations
between variables are not the exception but the norm
(Nickerson, 2000). Thus, selecting on one variable seems
likely to affect another.
Kriegeskorte et al. (2009) stated that ‘‘one way to ensure sta-
tistical independence of the results under the null hypothesis is
to use an independent data set for final analysis.’’ However, this
does not seem sufficient. Consider, for example, a common
practice such as the removal (as outliers) of response times that
are more than two standard deviations from the participant
mean. Imagine also that in one case an analysis is conducted
both with and without these outliers (as Kruskal, 1960, recom-
mends), and a significant effect is found only with the outliers
removed. In a second case, outliers are removed directly, with-
out exploration, because this has been found to be beneficial in
bringing about ‘‘good results’’ in the past. In the third case,
finally, the outliers are removed directly because the researcher
has simply read that this is standard practice. Removal of out-
lier RTs is fine if the truncated sample is no more or less likely
to arise through random sampling if the null hypothesis is
indeed true. The differences between the three cases are irrele-
vant to this criterion; neither the subjective motivation of the
researcher nor the presence or absence of multiple data sets
play any role. The criterion is simply about the fact of indepen-
dence. Whether or not it is met depends only on the determi-
nants of outlier RTs and hence their distribution. Using
different data sets for selection and for statistical test is conse-
quently not sufficient to avoid circular analysis (see also Vul &
Kanwisher, 2010).
Nor is using an independent data set always necessary. In
their supplemental materials, Kriegeskorte et al. (2009) sought
to provide a comprehensive set of questions and answers about
circularity; they noted that they see use of independent data sets
as an instance of a general rule that exploration and confirma-
tion must involve different data:
Hypotheses generated by exploring the data require confirmation
by means of independent data, because a relationship observed in
a data set will be consistent with that data set, whether or not it
reflects a true relationship or just the noise. (p. 3).
It is not clear that there is any such general rule. It may feel
circular that data could confirm a hypothesis that was
constructed specifically to explain them. However, whether
or not they do has been the subject of long-standing debate
Circularity
within the philosophy of science (e.g., for references and
discussion, see chapter 15 in Howson & Urbach, 1993). Model
fitting provides an important example here, and it is featured
prominently in Kriegeskorte et al.’s (2009) discussion.
Model fitting and circularity. In fitting a model to data, splitting
test and training sets is useful, but not conceptually essential.
Researchers optimizing a model with regard to an entire data
set and reporting the resultant fit are not making a circular argu-
ment. The interestingness of the resultant data (the degree of
fit), again, depends entirely on an empirical issue, namely the
model’s flexibility. If any data can be fit almost perfectly,
observing high levels of fit in any individual case is uninforma-
tive. Specifically, the observed fit is not diagnostic with regard
to the model’s status as a true theory or explanation. A high
level of fit is itself intended to provide data in support of the
hypothesis or theory embodied by the model. For any evidence,
the degree of belief in the hypothesis it can bring about is deter-
mined by the associated hit rate, P(ejh), and false positive rate,
P(ej¬ h). For a model sufficiently flexible to fit any data, the hit
rate is extremely high; that is, a good fit will be obtained when
data has truly been generated from the underlying model. How-
ever, because of the model’s flexibility, the fit will be equally
good when the data were not generated from the model; the
false positive rate is effectively equivalent to the hit rate. In
other words, the likelihood ratio equals 1, and no change in
belief is brought about through observation of the evidence
of a good fit.
The critical problem here is the high false positive rate
brought about by the model’s flexibility. A systematic exami-
nation of model flexibility (e.g., through further simulation)
demonstrating that not all possible data could be fit will reduce
the estimated false positive rate, increase the likelihood ratio,
and establish that good fits, when obtained, are diagnostic and
informative.
Splitting training and test data partially addresses this issue;
the model has not been optimized to the test data at all, which
means that the fits obtained are not equally dominated by the
model’s flexibility. Hence, it also seems plausible to assume
that good fits are more likely if the model were the true under-
lying model than if it were not, which makes good fits diagnos-
tic and hence informative. However, the extent to which this is
true depends directly on the homogeneity of the data and train-
ing set. For illustration, assume a hypothetical participant who
repeatedly sees the same probes and is entirely consistent in
responding. Using this participant’s first block for training and
the second block for testing contributes nothing to the wider
question of whether the observed level of fit is or is not inter-
esting given the model’s underlying flexibility.
The true benefit of techniques such as splitting training and
test data or cross-validation lies in providing better estimates of
model parameters and better estimates of model performance
by guarding against overfitting of noise, which in turn
improves generalization (e.g., Ripley, 1996). These techniques
are not, however, requirements without which modeling is
pointless, nor do they alone provide sufficient answers to the
179
problem of model flexibility, which has rightly been given pro-
minence in recent years (Roberts & Pashler, 2000).
Model fitting and the general case of hypothesis generation
and confirmation raised by Kriegeskorte et al. (2009), however,
seem directly analogous. Concern about the evidential support
given by sets of data to theories developed for those very data,
seems (at least in part) to rest on the idea that any number of
plausible (but incompatible) accounts could be developed,
again making the false positive rate, P(ej¬ h), for the particular
theory put forward very high. Once more, whether this is
indeed the case is an empirical question, but one not typically
pursued. Clearly, however, noise and its overfitting are not the
only, nor necessarily the most important, concerns involved for
either model fitting or theory generation.
It thus seems questionable whether general considerations
about the scientific process necessitate separate test and training
sets in the way that Kriegeskorte et al. (2009) have suggested. Of
course, testing on further data sets will provide additional evi-
dence. However, lack of separation does not obviously preclude
confirmation from a Bayesian perspective (see also Howson &
Urbach, 1993). Nor does its absence imply circularity.
Summary. In conclusion, the circular analysis cases are not
circular arguments in the technical sense introduced in the first
half of this article. Reminiscent of circularity, however, there
is a sense of potentially illicit dependence: What was found was
in some sense put there in the first place. Such a dependence
might also be perceived in model fitting or, even more generally,
in viewing data as confirming a theory designed specifically to
explain them. Use of independent data sets has thus been recom-
mended in all these contexts, but it does not distinguish success-
fully between acceptable and unacceptable dependencies. It
does not seem necessary in the case of model fitting and the
data-theory relationship. It does not seem sufficient in the case
of preselection. The real problem with preselection stems from
the particular logic of classical null-hypothesis testing and the
role of random sampling therein. It is not clear that there are sim-
ple solutions here; arguably, it needs to be demonstrated fully
why data selection is compatible with classical inferential statis-
tics in any given case.
Circular methods
Finally, Fiedler (2011) raises the question of circular methodol-
ogies more generally. Fiedler views the nonindependence prob-
lems identified in fMRI research as special cases of a more
general class of sampling problems that includes the use of aus-
picious tasks figured out in careful pretesting and the fact that
weaker outcomes or null effects obtained under different con-
ditions are frequently ignored.
Again, none of these involve circularity in any technical
(logical) sense. However, they do fit under the header of con-
cerns about the interestingness of discovery that arise from a
sense that findings are there only because they have been ‘‘put
there’’ in the first place, and in this lies the link with the pre-
selection issue. Thus, concerns about technical misuse of sta-
tistical measures and concerns about misrepresentation again
180
seem relevant. For example, communicating to others that
something has worked while neglecting to mention that more
often it has failed arguably violates conversational maxims.
Yet this seems widespread in the publishing of psychological
experiments. Moreover, in these circumstances, a meta-
analytic analysis would be the analysis to yield an accurate
picture (Rosenthal, 1991).
Even beyond such clear errors, however, Fiedler’s (2011)
examples raise important questions about the extent to which
we as researchers are discovering anything of interest in our
experiments and not merely creating what we subsequently
find. Again, flexibility is a concern. Human behavior is incred-
ibly variable and adaptive. Some of the unease about explana-
tion by heuristics such as those discussed above might, for
example, stem from a feeling that they may merely reflect
affordances in otherwise difficult or unusual cognitive tasks;
that is, they are readily available special purpose procedures
that participants discover for experimental tasks, not cognitive
tools that participants use in everyday life. The only way to rule
this out is to examine such heuristics in a broad range of
circumstances. As Fiedler (2011) rightly stresses, sampling a
variety of measures and materials is essential to determining
the scope and relevance of what we find, and psychology must
move beyond the reification of tasks that is so often seen.
Conclusions
Three domains of putative circularity were distinguished:
circular argumentation, circular explanation, and potential cir-
cularities within the data-generating process itself. As a review
of the philosophical literature on circular argument revealed,
true circularities in the technical sense are logically valid argu-
ments. Whether or not circularities are damaging depends on
the pragmatic goals at hand. In the case of rational argument,
this is change in degree of belief. For explanation, ‘‘increase
of understanding’’ or ‘‘answer to a question’’ are the most fre-
quently cited general, pragmatic goals. Finally, in the context
of scientific data gathering, the goal is evidence that is informa-
tive and diagnostic; this goal may be violated where the nature
of the evidence obtained depends unduly on our efforts.
Acceptable circularities exist where these pragmatic goals
can be met despite some element of self-dependence, as
demonstrated in the case of self-dependent arguments that
nevertheless allow belief change. At the same time, these prag-
matic goals can be violated readily without the involvement of
circularity. Consequently, it is not surprising that closer scru-
tiny of accusations of circularity within the present psychology
literature found the charge of circularity somewhat overused.
Clarifying this is hopefully more than an argument about
words. Because circularity has the reputation of the ultimate
intellectual flaw, the mere label often suffices (in the review
process, for example) to devalue something so completely that
no further elaboration is required. However, the existence of
benign circularities makes clear that merely labeling something
as circular is not enough to dismiss it; an argument for why the
thing in question is bad still needs to be made.
Hahn
Data gathering and explanation are critical tasks in our dis-
cipline; thus, it is important to understand precisely where and
why they are going wrong. Hence, this article is intended not as
criticism, but as a constructive contribution to debate that we
need much more of, not less.
Appendix
According to Bayes’ theorem, one’s posterior degree of
belief in a hypothesis, h, in light of the evidence, P(hje), is a
function of one’s initial, prior degree of belief, P(h), and how
likely it is that the evidence one observed would have occurred
if one’s initial hypothesis was true, P(ejh), as opposed to if it
was false, P(ej¬h):
PðhÞ Pðe j hÞ
Pðh j eÞ ð1Þ
PðhÞ Pðe j hÞ þ Pð¬hÞ Pðe j ¬hÞ
P(ejh) and P(ej¬h) determine the informativeness of the evi-
dence with regard to the hypothesis in question. They corre-
spond to the ‘‘hit rate’’ and the ‘‘false positive rate’’ known
from signal detection theory (Green & Swets, 1966), and their
ratio, the so-called likelihood ratio, indicates the diagnosticity
of the evidence. The most basic aspect of diagnosticity is that if
P(ejh) > P(ej¬h), then receipt of the evidence will result in an
increase in belief in h, whereas if P(ejh) < P(ej¬h), then receipt
of the evidence will result in a decrease. If both are equal, the
evidence is entirely uninformative and no change will ensue
upon its receipt. Stronger, in the sense of more diagnostic, evi-
dence for a given hypothesis will lead to greater posterior
degrees of belief, P(hje).
Acknowledgments
The author would like to thank Jonah Schupbach, Hal Pashler, Tania
Lombrozo, Bobbie Spellman, and Klaus Fiedler for very helpful com-
ments on earlier versions of this manuscript, and Adam Corner and
Mike Oaksford for many discussions on circularity
Declaration of Conflicting Interests
The author declared that she had no conflicts of interest with respect to
her authorship or the publication of this article.
Notes
1. Other well-known examples from the catalog of fallacies are the
argument from ignorance (‘‘Ghosts exist, because nobody has pro-
ven that they do not’’), the appeal to popular opinion (‘‘Everyone
thinks this, so it must be true’’), or the appeal to authority (see
e.g., Woods, Irvine, & Walton, 2004, for the complete catalog).
2. It is the emphasis on rationality that distinguishes argumentation
from persuasion (see also, e.g., Corner & Hahn, 2010; Hahn &
Oaksford, 2007; O’Keefe, 1995, 1997, 2007).
3. One example is modus ponens ‘‘if p then q, p ¼> therefore q.’’ One
can be uncertain about q, but believe ‘‘if p then q’’; being informed
that p will then lead one to believe that q.
Circularity
4. The normative status of Bayesian conditionalization has not gone
unchallenged in the philosophy literature (for arguments and coun-
terarguments, see, e.g., Howsen & Urbach, 1993). However, it seems
unquestionable that no other framework has been applied to the wide
range of problems concerning the practice of evidence generation
and evaluation that have been subject to Bayesian analysis.
5. Alternatively, one might use Jeffrey conditionalization (Jeffrey, 1965).
6. From a Bayesian perspective, probabilities reflect subjective
degrees of belief, so people may differ in the specific probabilities
they assign, but the probability calculus constrains what can count
as rational degrees of belief, and hence argument evaluations,
given such assignments (see e.g., Hahn & Oaksford, 2007).
7. It is worth noting that philosophy has seen debate about whether or
not explanations are arguments (e.g., Ruben, 1990). However, the
notion of argument invoked in this debate is considerably narrower
than that used here and typically centers on deductive argument.
References
Atkinson, D., & Peijnenburg, J. (2010). Justification by infinite loops.
Notre Dame Journal of Formal Logic, 51, 407–416.
Betz, G. (2010). Petitio principia and circularity as seen from a theory
of dialectical structures. Synthese, 175, 327–349.
Bovens, L., & Hartmann, S. (2003). Bayesian epistemology. Oxford,
England: Oxford University Press.
Brown, H. (1993). A theory-laden observation can test a theory.
British Journal for the Philosophy of Science, 44, 555–559.
Brown, H. (1994). Circular justifications. Proceedings of the Biennial
Meeting of the Philosophy of Science Association, 1, 406–414.
Colwell, G. (1989). God, the Bible and circularity. Informal Logic, 11,
61–73.
Copi, I.M. (1986). An introduction to logic (7th ed.). New York:
MacMillan.
Corner, A.J., & Hahn, U. (2010). Message framing, normative advo-
cacy and persuasive success. Argumentation, 24, 153–163.
Earman, J. (1992). Bayes or bust? Cambridge, MA: MIT Press.
Edwards, W. (1968). Conservatism in human information processing.
In B. Kleinmuntz (Ed.), Formal representation of human judgment
(pp. 17–52). New York: Wiley.
Evans, J.S.B.T. (1998). Matching bias in conditional reasoning: Do we
understand it after 25 years? Thinking and Reasoning, 4, 45–110.
Fiedler, K. (2011). Voodoo correlations are everywhere—not only in
neuroscience. Perspectives in Psychological Science, 6, 163–171.
Gigerenzer, G. (2009). Surrogates for theory. Observer, 22(2), 21–23.
Green, D.M., & Swets, J.A. (1966). Signal detection theory and
psychophysics. New York: Wiley.
Hahn, U. (2003). Rule-based thought. Encyclopedia of cognitive sci-
ence. New York: Wiley.
Hahn, U., & Chater, N. (1998). Similarity and rules: Distinct? Exhaus-
tive? Empirically distinguishable? Cognition, 65, 197–230.
Hahn, U., & Oaksford, M. (2006). A Bayesian approach to informal
reasoning fallacies. Synthese, 152, 207–223.
Hahn, U., & Oaksford, M. (2007). The rationality of informal argumen-
tation: A Bayesian approach to reasoning fallacies. Psychological
Review, 114, 704–732.
Hahn, U., Oaksford, M., & Corner, A. (2005, June). Circular argu-
ments, begging the question and the formalization of argument
181
strength. In A. Russell, T. Honkela, K. Lagus, & M. Pöllä (Eds.),
Proceedings of AMKLC’05, International Symposium on Adaptive
Models of Knowledge, Language and Cognition (pp. 34–40).
Helsinki, Finland: Helsinki University of Technology.
Hamblin, C.L. (1970). Fallacies. London: Methuen.
Harman, G. (1965). The inference to the best explanation. Philosophi-
cal Review, 64, 88–95.
Hempel, C. (1965). Aspects of scientific explanation. New York: Free
Press.
Hempel, C.G. & Oppenheim, P. (1948). Studies in the logic of expla-
nation. Philosophy of Science, 15, 135–175.
Heysse, T. (1997). Why logic doesn’t matter in the (philosophical)
study of argumentation. Argumentation, 11, 211–224.
Howell, D.C. (2007). Statistical methods for psychology. Belmont,
CA: Wadsworth.
Howson, C., & Urbach, P. (1993). Scientific reasoning: The Bayesian
approach (2nd ed.). La Salle, IL: Open Court.
Jeffrey, R. (1965). The logic of decision. New York: McGraw Hill.
Jones, L.V., & Tukey, J.W (2000). A sensible formulation of the sig-
nificance test. Psychological Methods, 5, 411–414.
Kahane, H. (1986). Logic and philosophy: A modern introduction.
Belmont, CA: Wadsworth.
Koertge, N. (1992). Explanation and its problems. British Journal for
the Philosophy of Science, 43, 85–98.
Korb, K. (2004). Bayesian informal logic and fallacy. Informal Logic,
24, 41–70.
Kriegeskorte, N., Simmons, W.K., Bellgowen, P.S.F., & Baker, C.I.
(2009). Circular analysis in systems neuroscience: The dangers
of double dipping. Nature Neuroscience, 12, 535–540.
Kruskal, W.H. (1960). Some remarks on wild observations. Techno-
metrics, 2, 1–3.
Lakoff, G. (1987). Cognitive models and prototype theory. In
U. Neisser (Ed.), Concepts and conceptual development (pp.
63–100). Cambridge, England: Cambridge University Press.
Lipton, P. (1991). Inference to the best explanation. London: Routledge.
Lombrozo, T. (2006). The structure and function of explanations.
Trends in Cognitive Sciences, 10, 464–470.
Nickerson, R.S. (2000). Null hypothesis significance testing: A review
of an old and continuing controversy. Psychological Methods, 5,
241–301.
Oaksford, M., & Hahn, U. (2007). Induction, deduction and argument
strength in human reasoning and argumentation. In A. Feeney, &
E. Heit (Eds.), Inductive reasoning (pp. 269–301). Cambridge,
England: Cambridge University Press.
Okasha, S. (2000). Van Fraassen’s critique of inference to the best
explanation. Studies in the History of Philosophy, 31, 691–710.
Okasha, S. (2002). Philosophy of science: A very short introduction.
Oxford, England: Oxford University Press.
O’Keefe, D.J. (1995). Argumentation studies and dual-process models
of persuasion. In F.H. van Eemeren, R. Grootendorst, J.A. Blair, &
C.A. Willard (Eds.), Proceedings of the 3rd ISSA conference on
argumentation: Vol. 1: Perspectives and approaches (pp. 3–17).
Amsterdam: Sic Sat.
O’Keefe, D.J. (1997). Standpoint explicitness and persuasive effect: A
meta-analytic review of the effects of varying conclusion articula-
tion in persuasive messages. Argumentation & Advocacy, 34, 1–12.
182
O’Keefe, D.J. (2007). Potential conflicts between normatively-
responsible advocacy and successful social influence: Evidence
from persuasion research. Argumentation, 21, 151–163.
Ooi, V.B.Y. (1998). Computer corpus lexicography. Edinburgh,
Scotland: Edinburgh University Press.
Pearl, J. (1988). Probabilistic reasoning in intelligent systems. San
Mateo, CA: Morgan Kaufman.
Perelman, C., & Olbrechts-Tyteca, L. (1969). The new rhetoric: A
treatise on argumentation. Notre Dame, IN: University of Notre
Dame Press.
Peterson, C.R., & Miller, A.J. (1965). Sensitivity of subjective prob-
ability revision. Journal of Experimental Psychology, 70, 117–121.
Peterson, C.R., Schneider, R., & Miller, A.J. (1964). Sample size and
the revision of subjective probabilities. Journal of Experimental
Psychology, 69, 522–527.
Pinker, S. (1991). Rules of language. Science, 253, 230–235.
Ripley, B.D. (1996). Pattern recognition and neural networks.
Cambridge, England: Cambridge University Press.
Roberts, S., & Pashler, H. (2000). How persuasive is a good fit? A
comment on theory testing. Psychological Review, 107, 358–367.
Robinson, R. (1971). Begging the question. Analysis, 31, 113–117.
Rosenthal, R. (1991). Meta-analytic procedures for social research.
London: Sage.
Ruben, D.-H. (1990). Arguments, laws and explanations. In D.-
H. Ruben (Ed.), Explaining explanation (pp. 181–208). New
York: Routledge.
Salmon, W. (1984). Scientific explanation and the causal structure of
the world. Princeton, NJ: Princeton University Press.
Sanford, D.H. (1972). Begging the question. Analysis, 32, 197–199.
Schum, D.A. (1981). Sorting out the effects of witness sensitivity and
response-criterion placement upon the inferential value of testimo-
nial evidence. Organizational Behaviour and Human Performance,
27, 153–196.
Sedlmeier, P., Hertwig, R., & Gigerenzer, G. (1998). Are judgments of
the positional frequencies of letters systematically biased due to
View publication stats
Hahn
availability? Journal of Experimental Psychology: Learning,
Memory, and Cognition, 24, 754–770.
Sher, S., & McKenzie, C. (2006). Information leakage from logically
equivalent frames. Cognition, 101, 467–494.
Shogenji, T. (2000). Self-dependent justification without circularity.
British Journal for the Philosophy of Science, 51, 287–298.
Sintonen, M. (1999). Why questions, and why just why-questions?
Synthese, 120, 125–135.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty:
Heuristics and biases. Science, 185, 1124–1131.
van Eemeren, F.H., & Grootendorst, R. (2004). A systematic theory of
argumentation. The pragma-dialectical approach. Cambridge,
England: Cambridge University Press.
van Fraassen, B.C. (1980). The scientific image. Oxford, England:
Oxford University Press.
van Fraassen, B.C. (1989). Laws and symmetry. Oxford, England:
Oxford University Press.
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly
high correlations in fMRI studies of emotion, personality, and social
cognition. Perspectives in Psychological Science, 4, 274–290.
Vul, E., & Kanwisher, N. (2010). Begging the question: The
non-independence error in fMRI data analysis. In S. Hanson, &
M. Bunzl (Eds.), Foundational issues for human brain mapping.
(pp. 71–92). Cambridge, MA: MIT Press.
Walton, D.N. (1985). Are circular arguments necessarily vicious?
American Philosophical Quarterly, 22, 263–274.
Walton, D.N. (1991). Begging the question: Circular reasoning as a
tactic in argumentation. New York: Greenwood Press.
Walton, D.N. (2006). Epistemic and dialectical models of begging the
question. Synthese, 152, 237–284.
Weisberg, J. (2009). Locating IBE in the Bayesian framework. Synth-
ese, 167, 125–143.
Woods, J., Irvine, A., & Walton, D.N. (2004). Argument: Critical
thinking, logic and the fallacies (rev. ed.). Toronto, Canada:
Prentice Hall.