A Comparison of Chi-Square and Kolmogorov-Smirnov Tests

Author(s): Bruce Mitchell

Source: Area, Vol. 3, No. 4 (1971), pp. 237-241
Published by: Blackwell Publishing on behalf of The Royal Geographical Society (with the Institute of
British Geographers)
Stable URL: http://www.jstor.org/stable/20000590 .
Accessed: 12/05/2011 21:14

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless
you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you
may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at .
http://www.jstor.org/action/showPublisher?publisherCode=black. .

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed
page of such transmission.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.

Blackwell Publishing and The Royal Geographical Society (with the Institute of British Geographers) are
collaborating with JSTOR to digitize, preserve and extend access to Area.

http://www.jstor.org
A comparison of Chi-square and
Kolmogorov- Smirnov tests
Bruce Mitchell, University of Waterloo, Ontario

Summary. The characteristics of the two statistical techniques are described and compared
It is suggested that the Kolmogorov-Smirnov technique facilitates statistical testing when
some assumptions of Chi-square are not satisfied. Examples are provided in order to
illustrate themanner inwhich the two techniques complement each other ingeographical
analysis.

A frequent problem in behavioural research is the choice of statistical technique

for data analysis. In particular, studies of perceptions and attitudes, even when
sophisticated scaling techniques are applied, often result in data which are only
amenable to analysis with nonparametric techniques. And, just as it is important
to be aware of trade-offs involving flexibility, ease of calculation and power
efficiency when choosing between parametric and nonparametric tests, so it is
essential to consider the relative advantages of various non-parametric alterna
tives. With this in mind, the purpose of this paper is to compare two non
parametric tests, Chi-square and Kolmogorov-Smirnov. The former is probably
one of the most frequently used statistical tests in geographic research in general
and behavioural studies in particular. The latter test appears to be used infre
quently, despite the fact that in some situations it ismore flexible than Chi-square.
Consequently, in this paper an attempt is made to compare the characteristics
of the two techniques, and then to illustrate how the Kolmogorov-Smirnov
technique may facilitate statistical analysis when the assumptions associated
with Chi-square are not satisfied.

Characteristics of the tests

Numerous references are available which discuss the characteristics of the two
nonparametric tests under study.2' 3, 4, 5, 6, 7, 9, 10, 11, 13, 14, 15 As a result,
only a brief description of the techniques will be provided here. Chi-square is
used to test for significance of differences among 1 to k independent groups
when the data have been assembled in the form of frequencies in discrete cate
gories. From the reference to frequencies in discrete categories it will thus be
clear that Chi-square is a technique capable of accommodating data at a nominal
level of measurement. Since measurement levels are cumulative, the testmay
also be applied to data at higher levels of measurement if some loss of detail is
acceptable.
The major operational constraint associated with Chi-square is that it re
quires the expected frequencies in each cell not to be too small.9' 10, 14 It is
often accepted that if the degrees of freedom for a contingency table are larger
than 1 then the test should only be used if less than 20 % of the cells have expected
frequencies smaller than 5 and if no cell has an expected frequency of less than 1 .14
If the degrees of freedom equal 1, then each cell should have an expected fre
quency of greater than or equal to 5. If this requirement of minimum size for
237
238 Comparison of Chi-square and Kolmogorov-Smirnov tests

expected frequencies is not satisfied, cellsmay be aggregated, especially if data

have been collected at an ordinal scale. If, when the degrees of freedom equal l,
the expected frequency is not satisfied, and categories cannot be combined, it is
possible in the one-sample case to use the binomial test and in the two-sample
case the Fisher test as alternatives. An alternative for the k-sample situation is
not easily found, especially if the data are at a nominal level.
The Kolmogorov-Smirnov testmay be used in one- or two-sample situations.
In the one-sample case, the technique tests for a significant difference between
the distribution of sample observations and a specified theoreticaldistribution.
In the two-sample case the test focuses upon the differences between two inde
pendent samples in order to determine whether or not the two samples come
from the same population or from populations with the same distribution. The
test, which requires data at an ordinal level of measurement, is conducted by
comparing the cumulative distributions of the two samples.
When compared toChi-square, itwill be noted that theKolmogorov-Smirnov
test appears to have at least two drawbacks. First, the test requires ordinal data
while Chi-square may be used with nominal data. Second, Kolmogorov
Smirnov is confined to one- and two-sample situations, whereas Chi-square
may be used for one-, two- or k-sample cases. These two constraints-level of
measurement and number of sample groups-clearly reduce the flexibility of
theKolmogorov-Smirnov technique for behavioural studieswhen comparisons
among more than two groups on dichotomous scales are sought. On the other
hand, theKolmogorov-Smirnov test does not have the constraint of expected
frequencies which is usually associated with Chi-square. As a result, the use of
the former test does not necessitate the loss of detail through the need to aggre
gate categories, or result in an inability to test for significant differences when
aggregated or disaggregated data do not satisfy the expected frequency assump
tion. In this regard, therefore, it would appear as if the Kolmogorov-Smirnov
technique offers considerable potential for application in pilot studies or in
investigationsutilizing small sample sizes.The following sections present several
examples to illustrate the relative advantages of the two tests.

The data
A study of perceptions and attitudes towards water management in southwestern
Ontario provided the data which will be used to compare the two tests.16 The
purpose of the study was to obtain an understanding of how the public and the
professional or technical water managers perceived a range of water problems,
as well as their attitudes concerning citizen participation in the management
process. The objective of the project was therefore to obtain insight into the
views of two specified populations-the public and the professionals-in the
hope that such information could be used to improve proposed or existing
participation programmes. A subsequent objective was to study different sub
groups of the public component to determine whether prevailing statements
that urban and rural residents held different views on problems in the county
were valid.
During the summer of 1970, four hundred interviews were conducted with
residents ofWaterloo County. For purposes of the study, the public was stratified
into three categories; urban, urban-rural, and rural. To complete the investiga
tion, forty interviews were completed with professional water managers in the
same area. Having randomly sampled the public within each of the above strata,
Comparison of Chi-square and Kolmogorov-Smirnov tests 239

and having interviewed the professionals, it was possible to conduct tests of

significant differences between the public and professional groups and within
the public group.

Application of the tests

The following tables demonstrate the relative merits of the two tests. In
Table 1, a situation is portrayed where only Chi-square could be applied.
It should be emphasized that while the data are presented as percentages in the
tables in order to facilitate visual comparison of responses, the Chi-square tests
were conducted upon the data in raw frequency form. In both Tests A and B the
expected frequencies satisfy the assumptions and the data are separated into
discrete categories. The Kolmogorov-Smirnov test could not be used because

Table 1. Preferences for levels at which decisions should be made

City/ Township County Province Other

(percentages of respondents)
A. Public: total 26 38 34 2
Professionals 12 37 37 14
B. Public: urban 22 34 42 2
Public: urban-rural 35 35 24 6
Public: rural 27 45 27 1

Test A: Ho-no significant difference between the public and the profes
sionals.
Note: (1) 2-sample case; (2) nominal measurement level; (3)
degrees of freedom = 3; (4) 13% of cells have expected
frequencies less than 5.
Chi-square: significant at 0-005.
Kolmogorov-Smirnov: no test, assumptions not met.

Test B: Ho-no significant difference among subgroups of the public.

Note: (1) 3-sample case; (2) nominal measurement level; (3)
degrees of freedom = 6; (4) 17% of cells have expected frequencies
less than 5.
Chi-square: significant at 0 01.
Kolmogorov-Smirnov: no test, assumptions not met.

the data are at a nominal level of measurement. In addition, even if the measure
ment level were satisfied, in Test B three groups are involved, and the Kolmo
gorov-Smirnov test can only accommodate one- or two-sample situations.
Table 2 presents a somewhat different situation. Test A represents a 2-sample
case with data at an ordinal measurement level. Fifty per cent of the cells were
found to have expected frequencies of less than 5. Consequently, Chi-square
could not be used. In this case, however, the Kolmogorov-Smirnov test may be
used and a significant difference may be demonstrated.
Conversely, Test B appears to be incapable of testing by either technique.
The Kolmogorov-Smirnov test cannot be used because a 3-sample case occurs
and the Chi-square test is not appropriate because the expected frequency
assumption is not met. In this situation, the problem could be resolved in either
one of two ways. To satisfy the Kolmogorov-Smirnov requirements of a 2
sample case it would be possible to conduct three paired tests. An alternative
method would be to aggregate the categories into three classes-agree, neutral,
240 Comparison of Chi-square and Kolmogorov-Smirnov tests

Table 2. The average person may not know what is best for him where
technical problems are concerned and should rely upon professionals

Strongly Strongly
Agree Agree Neutral Disagree Disagree
(percentage of respondents)
A. Public: Total 9 80 7 4 0
Professionals 18 55 8 16 3
B. Public: urban 7 84 5 4 0
Public: urban-rural 19 73 6 2 0
Public: rural 7 77 9 6 0

Test A: Ho-no significant difference between the public and the profes
sionals.
Note: (1) 2-sample case; (2) ordinal measurement level; (3) de
grees of freedom = 4; (4) 50% of cells have expected frequencies
less than 5.
Chi-square: no test, assumptions not met.
Kolnogorov-Smirnov: significant at 0 001.

Test B: Ho-no significant difference among subgroups of the public.

Note: (1) 3-sample case; (2) ordinal measurement level; (3)
degrees of freedom = 8; (4) 33% of cells have expected frequencies
less than 5.
Chi-square: no test, assumptions not met.
Kolmogorov-Smirnov: no test, assumptions not met.

disagree-and then determine whether or not Chi-square could be used. In

fact, after aggregating it can be shown that with the resultant 3 x 3 contingency
table there are 4 degrees of freedom and 11% of the cells with expected fre
quencies less than 5. The requirements of Chi-square are therefore met, and
testing can proceed. In this example, no significant difference can be shown at a

Table 3. In the past, who do you think has been the most important inmaking
decisions about water problems in the County?

Elected
Planners Officials Businessmen Engineers Public Other
(percentage of respondents)
A. Public: Total 26 25 12 32 4 1
Professionals 5 35 0 60 0 0
B. Public: urban 19 26 15 36 3 1
Public: urban-rural 25 25 6 35 6 3
Public: rural 39 22 11 23 5 0

Test A: Ho-no significant difference between the public and the professionals.
Note: (1) 2-sample case; (2) nominal measurement level; (3) degrees of freedom =
5; (4) 28% of cells have expected frequencies less than 5.
Chi-square: no test, assumptions not met.
Kolmogorov-Smirnov: no test, assumptions not met.

Test B: Ho-no significant difference among subgroups of the public.

Note: (1) 3-sample case; (2) nominal measurement level; (3) degrees of free
dom = 10; (4) 33% of cells have expected frequencies less than 5.
Chi-square: no test, assumptions not met.
Kolmogorov-Smirnov: no test, assumptions not met.
Comparison of Chi-square and Kolmogorov-Smirn( v tests 241

levelof significanceof 0 05. It is appropriate to note, however, that aggregation

of data in Table 2 has resulted in important loss of detail. With the 5-point
ordinal scale it was possible to measure intensity as well as direction of view
point about the statement. After aggregating it is only possible to measure
direction. This may well be critical in a behavioural study where intensity may
be as important as, or even more important than, direction of feeling.
Finally, Table 3 is included to emphasize that even careful planning of a
research design may result in data which are not amenable to testing by either
technique. The data for Tests A and B are not suitable for Chi-square because
of their violation of the expected frequency assumption. Furthermore, because
the data have been collected at a nominal level of measurement it is not possible
to achieve horizontal aggregation. The Kolmogorov-Smirnov technique is
not an operational alternative for these data because a ranked measurement
level has not been attained.

Conclusions
An attempt has been made to draw attention to the relative advantages of the
Chi-square and Kolmogorov-Smirnov tests. It is felt that the latter technique
has charactersiticswhich facilitate statistical testingwhen the assumptions of
the former are not met. Specifically, Kolmogorov-Smirnov is not hampered
by an expected frequency requirementwhich is a considerable advantagewhen
a sample size is small or else is scattered throughout a relatively large number
of discrete categories. Conversely, it has two requirements associated with
measurement level and number of testable independentgroups that are limiting
constraints. Nevertheless, it is believed that the results of this paper suggest
that the Kolmogorov-Smirnov technique, while certainly not a panacea for all
limitations of Chi-square, deserves greater attention in geographical analysis.
References and Acknowledgments
This study has been supported by the National Advisory Committee Water Resources
Research, Department of Energy, Mines and Resources, Ottawa, Canada.
1. Blum, J. R., and Fattu, C. A., Nonparametric methods, Review of educational research,
24, 467-87.
2. Bradley, J. V., 1968. Distribution-free statistical tests. Englewood Cliffs: Prentice Hall.
3. Edington, E. S., 1969. Statistical inference: the distribution-free approach. New York:
McGraw Hill.
4. Fraser, D. A. S., 1957. Nonparametric methods in statistics. New York: JohnWiley.
5. Gibbons, J. D., 1971. Nonparametric statistical inference.New York: McGraw Hill.
6. H-ajek, J., 1969. A course in nonparametric statistics. San Francisco: Holden-Day.
7. Kraft, C. H., and van Eeden, C., 1968. A nonparametric introduction to statistics. New York:
Macmillan.
8. Labovitz, S., 1967. Some observations on measurement and statistics. Social Forces, 46,
151-60.
9. Lewis, D., and Burke, C. J., 1969. The use and misuse of theChi-square test. Psychological
Bulletin, 46, 433-89.
10. Lieberman, B. (ed), 1970. Contemporaryproblems in statistics. Toronto: Oxford University
Press.
11. Maxwell, A. E., 1961. Analysing qualitative data. London: Methuen.
12. Moses, L. E., 1952. Non-parametric statistics for psychological research, Psychological
Bulletin, 48, 122-43.
13. Noether, G. E., 1967. Elements of nonparametric statistics. New York: JohnWiley.
14. Siegel, S., 1956. Nonparametric statistics. New York: McGraw-Hill.
15. Walsh, J. E., 1962. Handbook of nonparametric statistics. Princeton: D. van Nostrand.
16. Mitchell, B., 1971. Behavioural aspects of water management: a paradigm and a case study.
Environment and Behavior, 3, 135-53.