CLIVE ORTON

TESTING SIGNIFICANCE OR TESTING CREDULITY?

Summary.The paper examines two recent attempts to analyse a set of data

relating to Icenian coin hoards. The methodology of the second attempt, which
is based on significance tests made on ‘permillia’ data (i.e. coins per
thousand), is shown to be fatally flawed, and its conclusions unfounded. The
danger of using analytical statistical techniques without consideration of the
assumptions on which they are based, is stressed. Alternative approaches, one
analytical and one graphical, are presented: they suggest an interpretation
which differs from those of both the original analyses.

The use of statistical techniques in
archaeology is full of pitfalls, as many
archaeologists know to their cost. Occasion-
ally, papers are published which attempt to
‘set the record straight’ and bring back
straying archaeologists to the straight and
narrow path (e.g. Thomas 1978). It is
especially important that such papers should
themselves be free of serious statistical error,
as they are likely to be taken as a model by
archaeologists seeking for some certainty in a
sea of statistical confusion.
It is particularly unfortunate, therefore, that
a recent paper in this journal (van Arsdell
1996), while attempting to improve the meth-
odology of an earlier contribution (Creighton
1994), has only succeeded in digging an even
deeper pit to entrap unwary archaeologists. It
is necessary to point out the fatal flaws in the
methodology before it has a chance to
become an accepted technique.
The issue concerns the statistical analysis
of a simple dataset: counts of coins of 9 types
found in 12 hoards. The traditional view is
that the hoards are very similar to each other,
and all date to around the Boudican rebellion.
Creighton (1994) suggested a chronological
ordering of the hoards, based on a simple
seriation. In turn, van Arsdell (1996) criti-
cised Creighton on the grounds that he had
not demonstrated that there were statistically
significant differences between the hoards,
and went on to apparently show that there
were no such differences. The first point is
perfectly valid: it is all to easy to spot
patterns in ‘random’ data, and an objective
check that any possible patterns are ‘real’ is
an essential prerequisite to their interpret-
ation. The problem comes with the method
used to demonstrate the apparent lack of
statistical significance.
The first step of this method was to convert
all the counts to permillia (i.e. coins per
thousand, in contrast to the more common
percentages, or coins per hundred), following
the example of Reece (1981). This is a
perfectly respectable technique of data
presentation, and lends itself well to the

OXFORD JOURNAL OF ARCHAEOLOGY 16(2) 1997

ß Blackwell Publishers Ltd. 1997, 108 Cowley Road, Oxford OX4 1JF, UK
and 350 Main Street, Malden, MA 02148, USA. 219
 TESTING SIGNIFICANCE OR TESTING CREDULITY?
production of comparative graphs. The next
step was to compare each hoard against the
aggregate of all the others, using the
permillia of each coin type as the bases for
comparison. For each coin type in turn, a
range of  two standard deviations was
plotted about the mean permillia figure for
‘other hoards’, and the corresponding figure
for the chosen hoard was plotted to see if it
fell within the range; if it did, this was taken
as evidence that this difference was not
statistically significant.
This is where the method breaks down,
because two key assumptions, on which this
‘2 sigma’ rule is based, do not hold in this
particular dataset. First, the data are not
normally distributed, but in fact have a
skewed distribution (this can be seen from
the way in which the  2 standard deviations
bars frequently encompass impossible
negative values). Second, and more serious,
the individual permillia values from which
each standard deviation is calculated do not
have the same statistical distribution. Even
under the implicit null hypothesis that all the
hoards have the same pattern (and hence all
hoards have the same mean permillia for any
chosen coin type), they all have different
standard deviations because they are based
on hoards of different sizes. The calculation
of the standard deviation in this way (i.e. on
percentages or permillia), and its use in
constructing a hypothesis test (in effect, a t-
test, although it is not called such), is invalid.
Discussion of this point led Reece to stop
using this technique (Reece 1988, 22–3).
To the archaeologist, this may seem like an
obscure technical point, but it is a technicality
that ‘pulls the rug from beneath the feet’ of
the method. At a common-sense level, the
fatal weakness of the method can be seen in
the observation that the apparent significance
of the differences in no way depends on the
sizes of the hoards. If, for example, all the
220
hoards were ten times the size, the conclusion
would be exactly the same. But with ten times
more evidence, it is much more likely that the
observed pattern would be judged to be
statistically significant. The method fails to
take account of the size of the dataset and thus
of the weight of the evidence. The problem of
the relationship between sample size and
statistical significance has been widely
discussed, for example by Shennan (1988,
77–8).
The error is compounded by van Arsdell’s
misleading explanation of the meaning of a
significance test (e.g. 1996, 236). If a hoard is
proven to be different from the others at the
two standard deviation level, it is not true that
‘there will still be a five percent chance that
they are actually the same’. What can be said
is that if they were the same, there would be a
five percent chance of the difference between
them appearing to be as big as (or bigger
than) the observed difference. This misinter-
pretation is repeated at several stages in the
argument.
So what can be done? In statistical terms,
the dataset is a small two-way contingency
table, amenable to analysis by any of the
techniques designed for such tables, e.g. the
2 (chi-squared) test, the related G2 test, or
log-linear analysis (Bishop et al. 1975). Here
I shall first use the simplest and most
familiar, the 2 test, which compares the
observed data with the figures that would
have been obtained if (in this case) each coin
type had occurred in the same proportions in
all the hoards, and gives a measure of the
statistical significance of the difference
between the two (known as the ‘observed’
and the ‘expected’ respectively).
If we apply this test to the data as they
stand, we obtain a value of 2 = 696 on 88
degrees of freedom (d.f.), which is statisti-
cally significant at the 0.0001 level, i.e. if the
hoards were really ‘the same’, there would be
OXFORD JOURNAL OF ARCHAEOLOGY
ß Blackwell Publishers Ltd. 1997
 CLIVE ORTON

a chance of less than 0.0001 (1 in 10,000) of
them appearing as different as they do.
Unfortunately, it is not quite that simple.
The calculation of 2 is only an approxi-
mation, which can become inaccurate if some
of the ‘expected’ values are ‘too small’
(Cochran 1954). This implies that two small
hoards (Brettenham, 5 coins, and March, 8
coins) and three rare coin types (D, 8
examples, E, 7 examples, and M, 9 examples)
must either be deleted before the analysis, or
merged with other categories. If they are
deleted, the resulting 6-by-10 table gives 2
= 149 on 45 d.f., and p is again < 0:0001. My
preference would be to delete the two small
hoards and to merge coin types D and E with
C, and M with LN (as done by van Arsdell in
his Tables 1 and 2), again leading to a 6-by-
10 table, but this time with 2 = 226, and yet
again p < 0:0001. So there is clearly a
statistically significant pattern in the data; the
question is — where is it?
We can approach this question by a closer
examination of the dataset itself, or by trying
to make a picture of it. I shall do each I turn.
The 2 statistic is made up from individual
‘contributions’ from each cell (i.e. each
combination of hoard and coin type) in the
dataset. We can examine these contributions
to see which particular combinations of hoard
Table 1
Contributions to the overall chi-squared statistic from each cell of the ’reduced’ dataset. Very large contributions are shown in bold. The . . . indicates a
number smaller than 0.01 but greater than zero.
and coin type contribute most to the statistical
significance of the overall value of 2 that we
have already observed. There is unfortunately
no statistical test which would enable us to
say which of these contributions are statisti-
cally significant, but one can often form a
clear impression of which cells contribute
strongly. It is these cells which give a dataset
its distinctive pattern. The values of the con-
tributions to 2 for the reduced dataset (i.e. as
immediately above) are given in Table 1.
Table 1 shows that some coin types con-
tribute strongly to 2 and to the pattern, and
others scarcely at all. Type C+D+E makes
several strong contributions, indicating that
there are more coins than ‘expected’ of this
group of types in hoards Honingham,
Lakenheath, Weston and Wimblington, and
fewer in hoards Field Baulk, Scole, and
possibly Eriswell. There are individual large
contributions arising from there being more
coins than ‘expected’ (i) of type L+M+N
(more specifically, type M) at Joist Fen, and
(ii) of type F at Eriswell. Type G shows a
remarkably uniform pattern, as does the
Fring hoard.
Even with this approach, we have the task
of scanning tables to look for interesting
features. A visual approach might give us a
more accessible route into this dataset. The

Coin type
C+D+E F G HIJK L+M+N O
Honingham 6.21 1.00 ... 1.03 0.47 1.07
Lakenheath 6.19 3.68 0.02 0.05 0.08 0.01
Joist Fen 0.01 0.01 0.05 0.91 53.33 1.05
Weston 11.47 1.75 0.02 4.62 0.50 1.52
Santon Downham 2.69 1.19 0.61 0.17 0.02 0.06
Wimblington 39.53 0.84 1.01 2.09 0.41 1.96
Fring 0.28 1.71 0.09 0.81 2.48 1.06
Eriswell 4.24 29.04 0.16 2.42 0.50 4.71
Field Baulk 14.39 1.18 0.69 2.61 0.04 1.82
Scole 7.32 0.70 0.79 1.50 1.07 0.62

OXFORD JOURNAL OF ARCHAEOLOGY

ß Blackwell Publishers Ltd. 1997 221

 TESTING SIGNIFICANCE OR TESTING CREDULITY?

Figure 1
Plot of the first two axes of a correspondence analysis of the ‘reduced’ dataset.

appropriate statistical technique for the sort
of dataset is correspondence analysis
(Greenacre 1984; 1992), which represents
the rows and columns of a contingency table
as points on a scatterplot, in which (roughly
speaking) the point representing a particular
row will lie ‘in the direction of’ the points
representing the columns in which it is
prominent, and vice versa (1984, 6). In our
example, points representing coin types will
lie ‘in the direction of’ the hoards in which
they occur more frequently than ‘expected’.
The first run of this technique was with the
‘reduced’ dataset as described above, and
produced the plot shown as Figure 1. Here
we can see a central core of types and hoards,

OXFORD JOURNAL OF ARCHAEOLOGY

222 ß Blackwell Publishers Ltd. 1997

 CLIVE ORTON

Figure 2
Plot of the first two axes of a correspondence analysis of the ‘reduced’ dataset, after the omission of coin type M.

with two sets of outliers defining a pattern:
(i) in the horizontal axis, type C+D+E and the
hoards Wimblington and to a lesser extent
Weston, indicating an association between
this type and these hoards, (ii) on the vertical
axis, type L+M+N and the Joist Fen hoard,
indicating an association between this type
and this hoard. In other words, there are more
coins of type C+D+E at Wimblington and
Weston than ‘expected’, and more of type
L+M+N at Joist Fen. This agrees with the
high contributions shown in Table 1.
It is clear that pattern is to some extent
dominated by the exclusive relationship
between type M and Joist Fen (i.e. type M
is only found there in this dataset). There is

OXFORD JOURNAL OF ARCHAEOLOGY

ß Blackwell Publishers Ltd. 1997 223

 TESTING SIGNIFICANCE OR TESTING CREDULITY?
no more to be said statistically about this
type, so we delete it and re-run the analysis,
creating Figure 2.
Figure 2 shows a central core of types and
hoards, with C+D+E and Wimblington and
Weston to its right, and type F and Eriswell
to the top of the plot, reflecting the
association already noted from Table 1. The
odd position of Weston reflects partly the
high proportion of C+D+E and partly the low
proportion of HIJK in this hoard. Some
detailed patterning can be observed within
the central core, for example, Santon
Downham and Honingham are to the right,
towards type C+D+E, of which they have an
above-average proportion.
Overall, the pattern is clear from both
Table 1, and from Figures 1 and 2 together:
there are large and statistically significant
associations between (i) type C+D+E and the
Wimblington and Weston hoards, (ii) type M
and the Joist Fen hoard, and (iii) type F and
the Eriswell hoard. If a 2 test is carried out
on the ‘core’ types and hoards (as shown in
Figure 2), it gives a value of 2 = 13.8 on 18
d.f., which is not at all significant
statistically.
Van Arsdells’ claim that there are no
statistically significant differences between
the hoards must therefore be rejected as the
product of a faulty methodology. Four hoards
are statistically significantly different from
the others, in each case because they possess
a higher-than-expected proportion of a
particular coin type. However, the remaining
REFERENCES
bishop, y.m.m. fienberg, s.e. and Holland, P.W.
1975: Discrete Multivariate Analysis: Theory and
Practice (Cambridge, Massachusetts, The MIT Press).
cochran, w.g. 1954: Some methods for strengthening
the common chi-squared tests. Biometrics 10, 417–451.
224
hoard do look ‘remarkably similar’,
especially as it is quite unusual to find coin
hoards of an appreciable size that are not
statistically significantly different from each
other (Lockyear, pers. comm.).
Because van Arsdell’s criticism of
Creighton’s seriation is ill-founded, that does
not necessarily mean that the data support
Creighton’s view. A seriation of types in
assemblages (e.g. hoards) usually shows
itself as a distinctive parabolic curve (the
‘horse-shoe’) in the correspondence analysis
plot (Madsen 1988, 24). No such curve is
apparent in either Figure 1 or Figure 2. From
a statistical point of view, we can just
demonstrate as clearly as possible the nature
of the patterning in this dataset. The reasons
for this pattern remain an open archae-
ological question.
A final methodological point: the use of
analytical statistical techniques, such as
significance tests, on datasets that are
composed of percentages or permillia should
be viewed with the deepest suspicion. They
are almost always invalid.
Acknowledgements
I am grateful to Kris Lockyear for his comments on a
draft of this note, and for preparing Figures 1 and 2.
Institute of Archaeology
University College London
31–34 Gordon Square
London W0H 0PY
creighton, j.d. 1994: A time of change: the Iron Age
to Roman monetary transition in East Anglia. OJA 13,
325–333.
greenacre, m.j. 1984: Theory and Applications of
Correspondence Analysis (London, Academic Press).
greenacre, m.j. 1992: Correspondence Analysis in
Practice (London, Academic Press).
OXFORD JOURNAL OF ARCHAEOLOGY
ß Blackwell Publishers Ltd. 1997
 CLIVE ORTON

thomas, d.h. 1978: The awful truth about statistics in
archaeology. American Antiquity 43, 231–244.
madsen, t. 1988: Multivariate statistics and
archaeology. In Madsen, T. (ed.), Multivariate
Archaeology (Jutland Archaeological Society
Publications XXI), 7–27.
reece, r. 1981: The ‘Normal’ Hoard. PACT 5.
Statistics and Numismatics.
reece, r. 1988: My Roman Britain. Cotswold Studies
3.
shennan, s. 1988: Quantifying Archaeology.
(Edinburgh, Edinburgh University Press).
van arsdell, r. 1996: A statistical analysis of Icenian
coin hoards. OJA 15, 235–242.

OXFORD JOURNAL OF ARCHAEOLOGY

ß Blackwell Publishers Ltd. 1997 225