Contingency Tables Involving Small Numbers and the χ2 Test

Author(s): F. Yates
Reviewed work(s):
Source: Supplement to the Journal of the Royal Statistical Society, Vol. 1, No. 2 (1934), pp.
217-235
Published by: Wiley for the Royal Statistical Society
Stable URL: http://www.jstor.org/stable/2983604 .
Accessed: 02/01/2013 18:34

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.

Wiley and Royal Statistical Society are collaborating with JSTOR to digitize, preserve and extend access to
Supplement to the Journal of the Royal Statistical Society.

http://www.jstor.org

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
1934.] 217

CONTINGENCY TABLES INVOLVING SMALL NUMBERS AND THE

X2 TEST.

By F. YATES, B.A.

Introduction.
THERE has in the past been a good deal of argumentas to the ap-
propriate statistical tests of independence for contingencytables,
particularlythose in which each classificationis a simple dichotomy
(i.e. 2 X 2 tables). It is probably now almost universallyadmitted
that when the numbers in the various cells are large the x2 test,
introduced by K. Pearson in 1900,4 with the vital modification
as to degrees of freedom established by R. A. Fisher in 1922,1
and confirmedby Yule,5 is the appropriate one. The necessity
for this modification has been amply demonstrated both from
theoretical considerations1, 5 and by actual tests on material
knownto be almost if not quite freefromassociation,2 5 and there
is no need to discuss the matter furtherhere. Several other forms
of test for 2 X 2 tables have been shown to be equivalent to the
X2 test.'
The x2test is admittedly approximate,for in order to establish
the test it is necessary to regard each cell value as normally dis
tributed with a variance equal to the expected value, the whole
set of values being subject to certain restrictions. The accuracy
of this approximationdepends on the numbersin the various cells,
and in practice it has been customaryto regard x2 as sufficiently
accurate if no cell has an expectancy of less than 5. It is with the
question of the applicability of x2 to 2 X 2 contingency tables
involving small expectancies that we are directlyconcernedin this
paper.
It was suggestedto me by ProfessorFisher that the probability
of any observedset of values in a 2 X 2 contingencytable withgiven
marginal totals can be exactly determined. The method will be
explained in the next section. Armed with the exact distribution
the divergenceof the x2test in any special case can be tested. It
will be shown that although the test as ordinarilyapplied becomes
inaccurate even with moderatelysmall numbersin the cells, a simple
modificationenables the range of usefulness to be considerably
extended.
The problem of testing the independence of contingencytables
involving small numbers is of considerable practical importance

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
218 YATEs-Continqency Tables [No. 2,

in manybranchesof biologicalexperimentation, wheresuppliesof

experimentalmaterialare limited. Medicineoffersan extreme
example,whereconsiderations quite otherthan those of expense
serveto limitthe numberof subjectsavailable fortestinga new
treatment or methodof operation. We have therefore thoughtit
worthwhileto investigate the errorsof the modified
test in some
detail. The resultsof thisinvestigation in the form
are presented
ofa table whichservesto extendtherangeofthisteststillfurther.
In thecase ofcontingencytablesinvolving morethanone degree
of freedom, reasonsare givenforbelievingthatthe ordinaryx2test
is considerablymorereliable. As an exampletheexactdistribution
is workedout fora 2 X 3 table,and the agreement withthe dis-
tributiongiven by x2 is shown to be good.
Contingency
2 X 2 Tables.
We willdefinethe table by the followingnotation:
I A not A Total
B a b N-n
not B c d n
Total I N-n' n' N
wheren < n' < IN. It is easilyseen that d can take all integral
values fromo to n. We now proposeto findthe frequency dis-
tributionof thesevalues whenthe table is regardedas consisting
of two samplesof N - n and n respectively froma binomialdis-
tribution p ofA occurring,
withprobability subjectto therestriction
thatthetotalofA in bothsamplesshallbe N - n'.
The probabilityof obtainingthe values a and b in the sample
ofN - n is
(N - n
a b! pa( p)b,
thevaluesc and d in thesampleofn is
and ofobtaining

c! d! pc(l p)d

The combinedprobability is therefore

(N- n) ! n! pa+C p)b+d
a!b!c! d! p+(pbd
where a, b, c, and d are subject to the restrictions
a + b = N - n, and c + d _ n.
If the additional restrictionthat a + c = N - n' or b + d n'
is imposed, only those terms of the combined probability series
must be selected which satisfy the resfriction. But the factors

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
1934.1 InvolvingSmall Numbersand thex2 Test. 219

containing the requiredproba-

p are now constant,and therefore
bilitiesare equal to
__ __ ,d=n 1
a ! b!c !d l n
_a!b

subjectto the aboverestrictions.

The summation forit willbe givenby the
is easilyperformed,
ofpa + c in theexpansionof
coefficient
(p + q)Nr-n (p + q)nl(N -n) ! n !
i.e. ofpNn' in
(p + q)N/(N - n) ! n !.
Thisis
N!
n ! n' ! (N - n) ! (N -n') I
The successiveprobabilities fromd = o to d,= n are therefore
(N - n) ! (N -n') !(N -n) ! (N -n') ! n . n'
N! (N-n-n')!' N! (N-n-n'+ 1)!1!'
(N-n) !(N-n') !gn(n-1) n'(n'-1) (N_-n) ! n'!
N! (N-n-n'+ 2) ! 2 ! ' * * * N (n'-n)
The successiveprobabilities are therefore proportional to the terms
ofthehypergeometric seriesF(- n, - n', N - n - n' + 1, 1).
Alternatively statedthe probability corresponding to any term
a, b,c, d, is
n! n' ! (N-n) ! (N-n')!
N! a! b! c! d!
i.e.,theproductofthe factorials ofthefourmarginaltotalsdivided
by the productof thefactorials of thegrandtotal and the fourcell
numbers.
In cases whereN is not too large the distribution with any
particularnumerical valuesofthe marginaltotalscan be computed
quite quickly,usinga table of factorialsto determinesome con-
venientterm,and workingout the restof thedistribution termby
term,bysimplemultiplications anddivisions. If a tableoffactorials
is not available we may startwithany convenient termas unity,
and divideby the sumofthetermsso obtained.
Thenumerical distribution couldbe usedto providea directtest
of significance,but even whenthe marginaltotalsare quite small
the evaluationof x2is muchmoreexpeditious, and it is thereforeof
someinterest to determine thelimitswithinwhichx2maybe safely
employed.
ValidityofAssumptionofConstancyofMarginalTotals.
No demonstrationhas yet been giventhat we are justifiedin
assumingthe constancyof the marginaltotals. From one point

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
220 YATEs-Contingency Tables [No. 2,

of view the matteris indeed almost obvious, forthereis no inherent

distinctionbetween the rows and columns of the table, and if the
marginaltotals of the rows may be assumed fixed,then as an alter-
native the totals of the columns may equally be assumed fixed.
If the two tests ofsignificancearisingfromtheseseparate assumptions
are to be identical, then the test must ultimatelyinvolve the fixity
of all marginal totals. But since it is perhaps not immediately
clear that the tests of significanceneed be identical it may be worth
while to examine the matter a little more closely.
All possible sets of observationshaving N - n and n as the totals
of the rows can be classifiedaccordingto the value of n'. For each
n' a 2-5 per cent. (or other) level of significancecan be assigned
for each tail (subject to discontinuity)and all sets of observations
fallingoutside these limitswill be counted significant. If 5 per cent.
of each class is judged significant,then 5 per cent. of the whole
population of sets will be judged significant, whateverthe differences
in the relative frequenciesofthe various classes. Clearly,therefore,
the proposed test of significanceis a valid test, in the sense that if
there is no association a fixed proportionof the sets of observation
will be judged significant,whateverp.
Leaving aside the question of discriminationbetween the two
tails it is clear that when all the marginaltotals are fixedthe test is
efficient, the probabilityof obtain-
in the sense that ifp and p' differ,
ing a verdict of significanceis as great as possible. For there is
then only one degree of freedom,which must give rise to a definite
distrib-ution, of which the tails will provide the appropriateregions
of significance.
That the marginaltotals of the columns are necessarilyassumed
constant can be established without appeal to symmetry. Instead
of regarding the table as generated by two random samples of
N - n and n respectivelyfrom a population with probabilityp,
we may regardit as formedof a single sample N fromsuch a popu-
lation. In this sample each observationwill be eitherA or not -A.
The observations of class B may then be obtained by a random
drawing of N - n fromthis sample, for if there is no association
there can be nothing in the attributes of any single observation
whichwill determinewhetherit is also B or not -B. The remaining
n observations will be consigned to class not -B. By this pro-
cedureofconstructingthe table we securethe fixityofall the marginal
totals.
It is interestingto note that the marginaltotals are in the nature
of ancillary statistics as defined by Fisher. They determinethe
accuracy of the informationsupplied, but they are not subject to
erroror other variation themselves. From what has already been

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
1934.] Involving Small Numbers and thex2 Test. 221

said it will be apparent that thereis no need to estimatep in order

to make a test of significance. The distributionestablishedin the
previous section is independentof the actual value of p.
If a verdict of non-significance is obtained and this is regarded
as conclusive, i.e. if we are prepared to assume that p and p' are
identical, then the efficient(indeed sufficient)estimate of p is given
by (N - n')/N. If we are not prepared to make this assumption,
or if the probabilitiesare judged significantlydifferent,
the sufficient
estimates of p and p' are a/(N - n) and c/n,and the interpretation
of the differencebetween them can be determinedin the light of
their numericalvalues and the nature of the data.

Binomial Distributionswithknownp.
The examination of the x2 test in the case of simple binomial
distributionswith known p will illustratesome importantpoints.
We will firstconsider the symmetricaldistribution
(1 _ 1) 10.

Such a distributionwould be obtained, forexample, forthe number

of heads in groups of iO tosses with a coin.

TABLE I.

Successes. P' P. P(W). Discrepancy. P(X'). Discrepancy.

0 10 0 0010 0 0010 0 0008 -0 0002 0 0022 +0 0012

1 9 0 0098 0 0108 0 0057 -0 0051 0 0134 +0 0027
2 8 0.0439 0 0547 0 0290 -0 0257 0 0569 +0 0022
3 7 01172 0 1719 0 1030 -0 0689 0 1714 -0-0005
4 6 02051 0 3770 0 2635 -0 1135 0 3759 -0 0011
5 02461 _

Table I shows this distribution (which is symmetricalabout

5 successes). p' is the probability of obtaining a given number of
successes,and P the summationof p' fromeithertail, i.e. the proba-
bilityof obtaininga numberof successes deviatingfromthe expected
number5 by a given or any greateramount in one direction. Thus
the probability-of obtaining exactly 8 successes is o0o439, and the
probabilityof obtaining 8 or more successes is 0o0547.
The probabilitiesgiven by the ordinaryx2test are given under
the heading P(Z). These are one-halfthe ordinaryx2probabilities,
so as to be comparable with a single tail of the distribution. In
the computationof x2the numberof failuresas well as the number
of successes must be taken into account, giving a value for 8 suc-
cesses, for example, of 32/5 + 32/5 or 3-6. Since X is distributed
normallyforone degreeoffreedomthe exact probabilitycorrespond-

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
222 YATEs-Contingency Tables [No. 2,

ing to this value of X can, be obtained by taking the square root

and using a table of normal probabilityintegral.
It will be seen that x2 always under-estimatesthe probability,
the discrepancies being quite large except at the extreme tails.
The true probabilityof 8 or more successes, for example, is 0X0547,
whereas Xgives a value of 0X0290. We should thus on the x2test
judge such a deviation as almost reaching the 5 per cent. level of
significance(2X5 per cent. for one tail), whereas in realityits proba-
bility of occurrenceis over io per cent.
These discrepanciesare primarilydue to the fact that Xis a con-
tinuous distribution,whereas the distributionit is endeavouringto
approximateis discontinuous. If we groupthe Xdistribution,taking
the half units ofdeviation fromexpectationas the groupboundaries,
we may expectto obtain a much closerapproximationto the true dis-
tribution. This is equivalentto computingthe values of x2fordevia-
tions halfa unitless than the truedeviations,8 successes,forexample,
being reckoned as 7, 2 as 2'1. This correctionmay be styled the
correction for continuity,and the resultantvalue of Xdenoted by x'.
The resultant probabilities are shown under the heading P(z').
The agreementis now remarkablygood, although the expectation
of successes is only 5. x' somewhatover-estimatesthe probability
at the tails, and under-estimatesit near the centreofthe distribution.
Only forthe extremevalues is the discrepancyreally large, relative
to the true probability,but inasmuch as in tests of significancewe
are generally concerned with the region (of one tail) of between
o 5 per cent. and 2-5 per cent., this is not of importance. In the
critical region the discrepancies of this distribution are almost
sufficiently small to be neglected in ordinarytests of significance,
and were it only with distributionsof this type that we had to deal
we mightleave the matterhere.
There is, however,anothersource of discrepancywhich does not
appear in the above example. The X distributionfor one degree
of freedomis necessarilysymmetricalin terms of deviations from
the expected values, whereas many of the distributionswhich we
wish to test arc not. Table II exhibitsthe binomial distribution
(.3+ 1) 20.

Here, as before, P(Z), in general, seriously under-estimatesthe

probabilities. P(z'), however,is not now an adequate approxima-
tion in the criticalregion.
The magnitudeofthe errorsto whichP(Z') is subjectin the various
types of distributionwill be discussed in the next section. Here it
is sufficientto point out that P(Z') over-estimatesthe probability
on the shortertail, and under-estimatesit on the longertail, these

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
1934.] Involving Small Numbers and the x2 Test. 223

TABLE II.
Successes. p'. P. P(X). Discrepancy. P(%'). Discrepancy.

0 0-0032 0 0032 0 0049 +0-0017 0-0102 +0 0070

1 0-0211 0-0243 0-0192 -0-0051 0-0351 +0-0108
2 0-0669 0-0912 0-0618 -0-0294 0-0985 +0 0073
3 0-1339 0-2251 0-1515 -0-0736 0-2192 -0 0059
4 0-1897 0-4148 0-3129 -0-1019 0-3983 -0-0165
5 0-2023 - -
6 0-1686 0-3829 0-3129 -0 0700 0-3983 +0-0154
7 0-1124 0-2143 0-1515 -0-0628 0-2192 +0 0049
8 0-0609 0-1019 0-0618 -0-0401 0-0985 -0 0034
9 0-0271 0-0410 0-0192 -0-0218 0-0351 -0 0059
10 00099 0-0139 0 0049 -0 0090 0-0102 -0 0037
11 00030 0 0040 0 0010 -0 0030 0-0023 -0-0017
12 0.0008 0 0010 0 0001 -0 0009 0 0004 -0-0006
13 0-0002 0-0002 - -

discrepancies being reversed near the centre of the distribution.

In symmetricaland nearly symmetricaldistributionsP(Z') over-
estimates the probabilities at both tails and under-estimatesthem
near the centre of the distribution. Such discrepancies,however,
are small compared with those arising in violently unsymmetrical
cases.
From the nature of these discrepanciesit is clear that no simple
modificationof the correctionforcontinuitywill materiallyimprove
the approximationsobtained.

for Continuity.
Discrepanciesof thex2 Test aftercorrecting
If any given contingencydistributionis calculated by means of
the appropriate hypergeometricseries the discrepancies between
the values of x correspondingto the true probabilities and the
equivalent values of Z' can be evaluated. These discrepancies
are more convenientto work with than the discrepanciesbetween
the true and X' probabilities which were considered in the last
section: clearly they are immediatelyconvertiblethe one into the
otherby referenceto a table of the normalprobabilityintegral.
There are, of course, in general no discrepancies corresponding
to the exact 2X5 per cent. and 05 per cent. points, but it is possible
to determineapproximate hypotheticaldiscrepanciescorresponding
to these points on the true probability scale by interpolation. If
the discrepancies are plotted against the logarithms of the true
probabilitiesthe resultant points lie on remarkablyregular curves,
making graphical or other interpolationeasy. Fig. 1 shows the
graph of these discrepanciesin the case of the binomial of Table II.
For every distributiongenerated by a 2 X 2 contingencytable
with fixed marginaltotals but variable 'lass numbers,therefore,a

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
 224 YATES-ContinQInc?U Tabhle [No. 9%

hypothetical discrepancy in X' for any given level of significance

can be obtained. Clearlythe varietyof contingencytables met with
in practice is very large, but the variation of the X' discrepancycan
be exhibitedin quite a compact table in sufficient detail forpractical
purposes.
The distributiongeneratedby any contingencytable with fixed

+ 02

+ 0.1

P ~~~~0.5% 2.5%
0
LogP- 30 -25 -20 -'15 _10 0.5

-0-1 _/

-02-02

- 03

- 04

FIG. 1.-Discrepancies of X' forbinomial( + 3)20.

marginal totals may be definedin various ways. We might, for

instance,use the marginaltotals (onlythreeofwhichare independent)
forthispurpose,but any threefunctionsof themwill do equally well.
There is, in fact, a three-dimensionalfieldof distributionsand any

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
1934.] InvolvingSmall Numbersand thex2 Test. 225

position in this field can be definedby threeco-ordinates. For our

purposeit will be most convenientto use co-ordinateswhichrepresent
propertiesof the distributionratherthan of the table.
Considerthe general table discussed in the second section. The
distributionwill have n + 1 termscorrespondingto the n + 1 values
fromo to n of d. The expectation ford will be nn'/N,and this will
always be less than I n. All distributions,therefore,
may be classified
according to the smallest expectation of the table, and according
to the numberoftermsor range. In each class therewill be a whole
series of distributionshaving fixedexpectation m and range n + 1,
but with varying N (and n'). N and n' can, in fact, assume all
integralsolutionsof
m = nn'/N.
with n' > n. The contingencydistributionwith smallest possible
N will be called the limiting contingency distribution
of that class.
The smallest N must be equal to or greaterthan n2/m, but there is
no upperlimit. As N tends to infinitythe distributionapproximates
to the binomial distribution
(p + q) ,

where p rn/n, i.e. the expectation divided by the range less one.
In what followsit will be convenientto use m and p instead of m
and n + 1 as definingthe class.
With expectation 4 and range 12 + I, for example, we obtain
the series with sets of marginaltotals (24, 12; 24, 12), (27, 12; 26,
13), (30, 12; 28, I4), etc., with the limiting binomial (3 + 9)12.
With expectation32- and range io + i we obtain the series (30, Ia;
26, 14), (50, Ia; 39, 2i), etc. In each case the first-named distri-
bution is the limitingcontingencydistribution.
The utilityof the above classificationlies in the fact (established
by examinationof special cases) that the X' discrepanciesare similar
forall distributionsin anyone class,and in generaldecreaseorincrease
steadilywithincreasingN. Thus the knowledgeofthe X' discrepancy
forthelimitingcontingencydistributionand the binomialof any class
sets definitelimits to its value for any distributionof that class.
Moreoverthe X' discrepancies for the limiting contingencydis-
tributionsand the binomials vary in a regular manner as m and p
are varied. Fig. 2 illustrates this variation with variations of
p whenmis equal to 4. Thereare fourseparatediagramscorrespond-
ing to 2-5 per cent. and 0-5 per cent. points of the longerand shorter
tails. In each case the values actually calculated are marked. It
will be seen that the values fall very satisfactorilyon to smooth
curves (the curves of binomial values age shown full, those of the
limitingcontingencyseries values dotted).

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
 226 YATEs-Contingency Tables [No. 2,

Practically we are only interestedin the values of p between o

and o 5, but highervalues of p have been included where possible
to illustrate the continuity. In the series of limiting contingency

n
+02 2
+ 01

0S
o
0-1 0-2 0-3
O !
0
I I

0-2
I
0-3 0-4 0-
, '
07
08
1
00 00 0.0

L-00
-0-1 v

1S
-0-2 Jd

03_
2-5% point Longer Tail. 2-5% point ShorterTail.

043 X'9 4 -

+03 - +02 _ /

+0, _ +01 _

O~~
p- 0
I
01
.
00
~ 03
'm
05
I
06
0
0 01 02
I
03
'
04
I
00
',S
- 07 08

-03 _

0-5% point Longer Tail. 0.5% point ShorterTail.

FIG. 2.-Discrepancies of X',expectation4.
The verticalscales representdiscrepancies,the horizontalscales values of m
The fullcurvesare thosegiven by the binomials,the dottedcurvesthose given
by the limitingcontingencydistributions.

distributionsonly values such that N =,9n2/m have been included,

except in the case of the points marked with a query, where no

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
1934.] InvolvingSmall Numbersand thex2 Test. 227

integralvalues of N exactly equal to n2/Mwere available, but the

pointsgivenby thenexthigherintegralvaluesappearedofsufficient
interestforconclusion.
Thesediagramsare similarto thoseobtainedwithotherexpecta-
tions, subjectto the provisothat the largerthe expectationthe
smallerthe discrepancies.Severalinteresting factsmaybe at once
deducedfromthem. Thus of all the binomialdistributions with
given expectationand p :< o5 the Poisson distribution(p= o)
givesthe greatestdiscrepancies, the discrepanciesin X'forexpecta-
tion 4 being+ 0o205 and- o0283 forthe 2-5 per cent.pointon
the longerand shortertail respectively, corresponding to X' pro-
babilitiesof I-52 per cent.and 4-68per cent.respectively.Also
the discrepancieson the two tails of distributionswithsmallp are
approximately equal but ofoppositesign. The discrepancies ofthe
symmetrical binomial(p = o.5), on the otherhand,are quitesmall,
namely- o-o6owitha corresponding probabilityof 2-87per cent.
Thus asymmetry is the mostpowerfuldisturbing factorof the X'
test. This confirms the conclusion ofthe fourthsection.
The discrepanciesofthelimiting contingency seriesareless than
thoseofthe corresponding binomialseriesexceptfora smallregion
in the neighbourhood of zero discrepancy.In general,therefore,
the binomialmay be regardedas furnishing the upperlimits,and
the limitingcontingency distribution the lowerlimitsto the dis-
crepancieswithany givenm and p.
Theapplicationoftheaboveresultsto theconstruction ofpractical
testsof significance
willnowbe considered.Clearlyifit is desired
to test the significanceof a table withsome particularmarginal
totals(thatis,withsomeparticular m,p and N) and the2'5 percent.
and o 5 per cent. values of X' forthe corresponding distribution
are known,the test can be made immediately by comparing the
value of X'calculatedfromthetable withthe 2-5 per cent.and o 5
percent.values. Ideally,therefore, a table ofthe 2 5 percent.and
o 5 percent.valuesofX'foreachtailis required, covering all possible
contingencydistributions.This would be a three-dimensional
table, and we have not attemptedto constructit. Instead,two
two-dimensional tableshave been computed,one givingthe values
of X'forthe'binomialdistributions, the otherthoseforthelimiting
contingency distributions.Both the 2-5 per cent.and the o 5 per
cent.pointsaretabulatedforbothtailsforvaluesofm rangingfrom
i to 96 and valuesofp equal to o, 0-25, and o 5. Sincethe longer
and shortertail are identicalwhenp= o 5 and the binomialand
limitingcontingency distributionare identicalwhen p= o, the
wholeset ofvaluesforeitherthe2-5 perceint.or o 5 percent.points
can be most conveniently"given in a singletable. This is done

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
 228 YATEs-Contingency Tables rNo. 2,

in Table III. In the case of the binomial with m - 5, p =o025,

for instance, Fig. 1 shows that the discrepancies for the 2-5 per
cent. and o 5 per cent. points are - Oi62 and -- 343 on the shorter

TABLE III.
The 2-5per cent.and o 5 per cent.points of x'.
The values forthe binomialdistributionsare shownin ordinarytype,those
forthe limitingcontingencydistributionsin italics.
thesmallest
the sma.llestexpecta.tion
m = the smallestexpectation. p = the sma.llestm arin total'
ma.rgina.l
2 5 per cent. points. 0 5 per cent. points.

Longer tail. Shorter tail. Longer tail. Shorter tail.

m 0. 0-25. 0.5. 0-25. 0. 0. 0-25. 0.5. 0-25. 0.

1 232 210 - _ 333 279 - _

-

2.04 - -2-67 - -

2 2.24 208 - - - 313 2679 - -

2-03 1-91 -2-67 - -

3 2 19 2 07 1.88 1 73 - 3 05 2 78
202 193 183 267 2-48
4 2 16 2 06 1 90 1 77 1.68 2 97 2 76 2 41 2 18
2 02 194 1 85 2 67 2 50 2 32
5 2 14 2 06 1 91 1.80 1 71 2 95 2 75 2 44 2 23 2 06
2 01 194 1-86 2 66 2 52 2 36
6 2 13 2 05 1 92 1 82 1 74 2 92 2 73 2 47 2 27 2-13
,2 01 194 1 87 2 66 2 53 2 38
8 2 11 2 04 1 93 1 84 1-77 2 88 2 72 2 50 2 32 2 19
2 00 1 95 1-89 2 65 254 242
12 2 08 2 02 1-94 1 87 1.81 2 83 2-70 2-52 2 38 2 27
199 195 190 264 255 246
24 2 05 2 01 1 95 1 90 1 86 2*76 2 67 2 55 2 45 2 37
1 99 1 95 1 92 2 63 2 56 2 50
48 2 03 2 00 1 96 1-91 1-89 2 70 2 64 2 56 2 49 2 43
198 196 1-94 262 257 252
96 2-01 1 99 1 96 1 93 1 91 2 67 2 63 2 57 2 52 2 48
1 97 1 96 1 94 2 60 2 57 2 54

tail, and + 0o095 and + OI73 on the longer tail. Adding the
2-5 per cent. discrepancies to I96o (the 2-5 per cent. point of X)
and the o*5 per cent. discrepanciesto 2-576 (the o 5 per cent. point
of X) gives the tabulated values of iv8o, 2-23, 2o6 and 2-75 respec-
tively.
Since the table only contains values for the binomials and the
limiting contingencydistributionsit only serves to provide upper
and lower limits to the actual 2-5 per cent. and o 5 per cent points
of X' for other contingencydistributions. In testingany particular

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
1934.] InvolvingSmall Numbersand thex2 Test. 229

contingency table interpolationwill be necessaryto determine

theseupperand lowerlimits,unlessthe m and p ofthe tableto be
tested happen to coincidewith the tabulatedvalues. Thereis,
however, no needforanygreataccuracyin thisinterpolation, which
can be mademoreor less by inspection.
Should the actual value of X' calculatedfromthe particular
contingency table beingtestedhappento fall betweenthe upper
and lowerlimitsof eitherthe 2-5 per cent.or ? 5 per cent.points,
thenthe appropriatehypergeometric seriesmustbe calculated,at
east ifan exacttestis required.
Table III also servesto determine thelimitsofapplicabilityof
the X'testwhenthestandardx2distribution forone degreeoffree-
dom is used (i.e. of the ordinaryx2test aftercorrection forcon-
tinuity). Thus if it is consideredthat errorsin the X' test up to
i 0o5per cent.may be toleratedwhenthe trueprobability is 2-5
percent.,thelimitsofthe2 5 percent.pointofX'givenby TableIII
mustbe 2-054 and i-88i, thesebeingthe 2-0 per cent.and 3-0 per
cent.pointsof the ordinaryXdistribution.Whenp equals o (the
worstcase), theselimitsare exceededwhenm is less than 40. In
the symmetrical distributions(p 05) m can be as small as 3.
At theselimitstherelativediscrepancies at the o 5 per cent.points
are decidedlygreater,forthe X' probabilitiesrangebetweeno-8
per cent.and o03per cent.roughly.
Table IV givesthelimitsforpermissible discrepanciesof i o 5
percent.and ? o025 percent.at the2-5 percent.points.
TABLE IV.
Limitsofm within
whichthestandard
x2 tablemaybeused.
Permissible discrepancy
at 2.5% point. P =o ?5L. P > 0 375. P > 0 25. P > 0.

i 0-5% m> 3 m> 7 m > 16 m > 40

+ 0 25% m> 6 m> 20 m > 75 m > 125

In view of the wide variationin the limitsof m fordifferent

p it
willprobablybe moreconveiientin practiceto use Table III if it
is available wheneverthe smallestexpectationis less than ioo.
On the otherhand,whenTable III is not availablethe workerwill
not be led badly astrayif he applies the ordinaryx2test (after
correctingforcontinuity) to tables givingexpectationsas low as
IO, so long as the correspondingdistributionsare reasonably sym-
metrical. Moreover, in view of the factthatthe discrepanciesare
of oppositesignson the two tails of the distribution,
no consistent
under-or over-estimatesofsignificance willbe madewhenapplying

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
230 YATES-ContiqnqencyTables FNo. 2,

theordinary testcorrectedforcontinuityto a heterogeneous

collection
ofdata, evenwithverysmallexpectations.
The correctionfor continuityinvolves-practically no extra
workand shouldtherefore be applied unlessthe expectationsare
verylarge. Even witha smallestexpectation of500 thediscrepancy
introduced intothe x2test at the 2X5 per cent.pointof eithertail
by omitting thiscorrectionmaybe as muchas o26 percent.
The sequenceof operationsin makinga practicaltest of any
2 X 2 tableis thereforeas follows:
1. If the smallestexpectationis less than 500, calculate
x'2 i.e. x2 corrected
forcontinuity,instead of x2.
2. If xZ2is in the neighbourhoodof the 5 per cent. or I per
cent. point and the smallest expectationis less than ioo, calcu-
late p, take the square root of x'2,and referto Table III, inter-
polating if necessary. (Note that interpolationwill oftenbe
unnecessarybecause the value of X'calculated fromthe observa-
tions is greateror less than all the values of Table III between
which interpolationwould be made.) AlternativelyTable IV
may firstbe consulted to decide if referenceto Table III is
really necessary.
3. If the value of X' calculated from the observationslies
between the upper and lower limits given by Table III and a
more precise test is desired, calculate the hypergeometric series
and performthe exact test.
Example. The followingfiguresfor malocclusionof the teeth in
infantswere obtained by M. Hellman,3 who draws fromthem the
conclusion that bottle-feedingis one of the factors causing maloc-
clusion.

Normal Teeth. |Malocclusion. Total.

Breast-fed ... ... ... ... 4 16 20

Bottle-fed ... ... ... ... 1 21 22
Breast and bottlefed ... ... 3 47 50
Total ... ...8 84 92

Considering firstthe wholly breast-fed and wholly bottle-fed

only we have the fourfoldtable :

Normal Teeth.I Malocclusion. Total.

.
Breast-fed . 4 16 20
Bottle-fed . .. ... ... 1 21 22

Total .5 37 42

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
1934.] InvolvingSmall Numbersand thex2 Test. 231

ITere 2=(4 x 21-1 .20

x 16)2. 422*386
~~~~~5.-37.22
X 1-545,P(Z) = 0 0612, P(2) - 2P(X) = 0X1224
(31 X 20- - 1l X 16)2. 42
X .37 .20 .22
~~~5
X'-1-068, P(x') = 0-1427,P(X'2) = 2P(X') = 0X2854.
Note that P(Z) and P(Z') are obtained fromX and x' by reference
to a table of the normalprobabilityintegral.
The exact distributionis computed as follows. The probability
of obtaining no normal breast-fedchildrenis
5! 37! 20! 22! u-0309568.
42 ! 0 ! 20 ! 5 ! 17!
Using the successive multipliers shown we obtain the complete
table:

No. of Normal
Breast-fed Children. Multiplier. Probability.

0 0-0309568
1 5.20/1.18 0-171982
2 4.19/2.19 0-343964
3 3.18/3.20 0-309568
4 2.17/4.21 0 125301}0.143527
5 1.16/5.22 0*018226f
0-999998

Thus the true probabilityof obtainingfour or more normal breast-

fedchildrenis 01435. P(x') givesOI427, an excellentapproxima-
tion, whereasP(Z) gives o0o6I2, which,thoughnot in itselfattaining
significance,is less than halfthe truevalue; thiswould be exceedingly
misleading if a number of such probabilities fromdifferentclasses
of experimentwere to be combined.
x' is so far fromthe 2-5 per cent. point for any mnand p that
exact referenceto Table III is unnecessary. The smallest value
ofthe 2'5 per cent. of X' in Table III is i-68.
If malocclusionis due to mechanical action on the teeth it may
be considered that children both breast-fed and bottle-fed are
exposed to much the same risk of damage as those whollybottle-fed.
Combiningthese two classes we have the followingtable:

Normal Teeth. Malocclusion. Total.

Breast-fed ... ... ... ... 4 16 20

Bottle or breast and bottlefed ... 4 68 72

Total ... ... ... 8 84 92

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
 232 YATEs-Contingency Tables [No. 2,

This table gives the followingresults:

X2 4113, X-2X028, P(Z) = 0X0213.
Z'2 2X495,X' 1X580,P(z') = 0 0571.

No. of Normal .
Breast-fed Children. Probablity.

0 0-12859
1 0-31652
2 0-31892
3 0-17136
4 0 05355
5 0 00993
6 0-00106 1006460
7 0-00006 I
8 0 00000J

0.99999

Here the ordinaryx2 test attains the 5 per cent. level of significance
(2.5 per cent. on one tail). The true probability,however,oo646
on the one tail, is nowhere near this, and again the correctionfor
continuity,whichgives a probabilityof 0-0571, is a good approxima-
tion. Here also X' is sufficientlysmall to make referenceto Table III
unnecessary.
Thus it will be seen that even on the most favourable grouping
of the data association of the degree observed mighthave arisen by
chance about once in eight times, so that Hillman's conclusions
cannot be regarded as established.
In neitherof these examples has it been necessaryto make any
exact referenceto Table III. This is the case withthe greatmajority
of tests, but to illustratethe application of the table we will suppose
that the results obtained in the second case were as follows:

Normal Teeth. Malocclusion. Total.

Breast-fed. ... 5 15 20
Breast or breast and bottlefed ... 3 69 72
Total .. ... ... 8 84 92

Here X' 2-477and P(Z') ooo66, whichon the face of it indicates

significanceapproachingthe o 5 per cent.point,but since the smallest
expectation m 8 X 20/92 174, which is very small, and
p =74/8 0-22, so that the distributionwill be decidedly skew,
some assurance is needed that the approximationis good enough.
Referringto Table III we see that for m 1 and p- 0the 25per
cent. point of X' on the longertail is 2-32, forp = 0-25 it lies between

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
1934.] InvolvingSmall Numbersand thex2 Test. 233

21io and 2-O4, and for m 2 the correspondingvalues are 2-24,

2-o8, and 2-03. The value 2-477 calculated fromthe observations
is greater than all these, and thereforethe 2-5 per cent. level of
significanceis attained. The o 5 per cent. level of significance,on
the otherhand, is not attained, since the value 2-477 is less than all
the values of the o 5 per cent. point between which interpolation
would have to be made. The true value of the probabilitygiven
by the exact distributionalready workedout is ooiio.

Tables involvingmorethanone Degree of Freedom.

Contingency
The exact method of determiningthe probabilities of everyset
of values conforming to given marginaltotals can easily be extended
to tables withmorethan two classes in either or both classifications.
The probabilityof obtainingany given set of values is, as in the case
of 2 X 2 tables, the product of the factorials of all the marginal
totals divided by the product of the factorials of the grand total
and the individual cell numbers.
In the case of a 3 X 2 table, for instance, which involves two
degrees of freedom,the probabilitiesof each particularset of values
can be set out in a two-dimensionaldiagram which consists of two
sets of interlocking hypergeometric series. The computation
naturally becomes tedious if the number of possible sets of values
is at all large, but it involves no new principle.
The discontinuousnature of the distributionhas in general not
so seriousan effecton the x2test. For even withquite small marginal
totals there are far more possible sets of values, and it is only in
exceptional cases that more than one set of values has the same x2.
If we regard the probabilityof obtaining any one set of values as
the result of groupinga continuous frequencydistribution,the x2
distributionmay be considered as an approximation to this con-
tinuous distribution. With any considerable deviation from ex-
pectation the surfacesof equal x2cut across many of the cells repre-
sentingthe sets of values; some of these cells will be included in
the true probability for that x2,and others excluded. Inasmuch,
however, as the probability density falls off with increasing x2,
x2may as beforebe expected to under-estimatethe true probability
except at the tails, where other sources of disturbance become
important.
As an illustrationof what is likely to occur in practice take the
contingencytable-
a1 a2 a3 17
bi b2 b3 1 13
13 11 6 30

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
234 YATEs-Contingency Tables [No. 2,

TABLE V.
Discrepanciestn a 2 X 3 Contingency
Table.

a2. a3. | 2. p. p(X2)

9 2 4-750 0-1067 0-0930

7 1 4-963 0-0948 0-0836
8 1 5-192 0-0830 0-0746
5 6 5-736 0.0724 0-0568
6 1 6-102 0-0658 0-0473
3 4 6-285 0-0592 0-0432
6 6 6-363 0-0532 0-0415
3
8 5}
5 6-442 0-0483 0-0399
4 6 6-476 0-0365 0-0392
9 4 6-630 0-0317 0-0363
9 1 6-786 0-0268 0-0336
4 2 7-311 0-0221 0-0258
3 3 8-113 0-0188 0-0173
10 2 8*338 0-0167 0-0155
7 6 8-358 0-0149 0-0153
3 6 8-583 0-0130 0-0137
5 1 8-608 0-0120 0-0135
10 3 8-914 0-0094 0-0116
10 1 9-748 0-0081 0-0076
8 0 9-838 0-0071 0-0073
7 0 10-236 0-0061 0-0060
2 5
10-546 0-0053 0-0051
9 Si

Table V gives the true probabilities in the critical region of

obtaining as great or greaterx2 than that given in the firstcolumn.
P(X2) is given in the third column. It will be noted that x2 here
serves as a criterionby which deviations fromexpectation may be
arranged in order of magnitude, and performsan additional and
entirelydiflerentfunctionto that in the case of a single degree of
freedomwhere its only functionhas been' to serve as an approxi-
mation to the actual distribution. It could, however, have been
made to performthis additional functionin the case of a single
degree of freedom,when it would serve to combine the deviations
of the two tails on the basis of equal x2. This would eliminatethe
effectsof skewness in the true distribution and the agreement
would be much improved. The same elimination is undoubtedly
effectedin the case of more than one degree of freedom. Whether
x2 is really the best criterionof a deviation fromexpectation we do
not propose to discuss here.
The distributiongiven here is likely to be an example which is
fairly favourable to x2. Cases where some of the marginal totals
are large, and other's small, so that certain degrees of freedom
approximate to the binomial or Poisson distributions,may be ex-
pected, in the light of the resultsobtained forone degreeof freedom,

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions
1934.] InvolvingSmall Numbersand theX2 Test. 235

to give much more unfavourable results. Another source of dis-

crepancy between the x2 and the exact probabilityis that due to
several cells having the same value of x2. This will occur in sym-
metrical tables, particularly those with integral expectations in
the various cells.

Summary.
1. A method of obtaining the exact probability distribution
associated with a 2 X 2 contingencytable with given marginal
totals is developed.
2. It is shown that the ordinaryx2test is liable to considerable
errors when the expectations are moderately small. A simple
modificationis suggested which considerablyincreases the accuracy
of x2. Tables are given which enable the limits of applicability
of the modifiedtest to be determined,and serve as a means of in-
creasingthe accuracy of the modifiedtest.
3. The applicabilityof the x2test to contingencytables involving
morethan one degree of freedomis brieflydiscussed.

References.
1 R. A. Fisher (1922). " On the Interpretationof x2 from Contingency
Tables, and the Calculationof P." Journ.Roy. Stat. Soc., Vol. LXXXV, pp.
87-94.
2 R. A. Fisher (1926). " Bayes' Theorem and the Fourfold Table."
EugenicsRev., 18, pp. 32-33.
3 M. Hellman (1914). " A Study of some Etiological Factors of Maloc-
clusion." Dental Cosmos,Vol. LVI, pp. 1017-1032.
4 K. Pearson (1900). " On the criterionthat a given systemof deviations
fromthe probablein the case of a correlatedsystemof variablesis such that it
can be reasonably supposed to have arisen fromrandom sampling." Phil.
Mag., 1, Series 5, Vol. L, pp. 157-175.
5 G. U. Yule (1922). " On the Applicationof the x2Methodof Association
and ContingencyTables, with ExperimentalIllustrations." Journ.Roy. Stat.
Soc., Vol. LXXXV, pp. 95-104.

SUPP. VOL. I. NO. 2. K

This content downloaded on Wed, 2 Jan 2013 18:34:02 PM

All use subject to JSTOR Terms and Conditions