REMOTE SENS. ENVIRON.
37:35-46 (1991)
A Review of Assessing the Accuracy of
Classifications of Remotely Sensed Data
Russell G. Congalton
Department of Forestry and Resource Management, University of California, Berkeley
T h i s paper reviews the necessary considerations
and available techniques for assessing the accuracy
of remotely sensed data. Included in this review
are the classification system, the sampling scheme,
the sample size, spatial autocorrelation, and the
assessment techniques. All analysis is based on the
use of an error matrix or contingency table. Exam-
ple matrices and results of the analysis are pre-
sented. Future trends including the need for assess-
ment of other spatial data are also discussed.
INTRODUCTION
With the advent of more advanced digital satellite
remote sensing techniques, the necessity of per-
forming an accuracy assessment has received re-
newed interest. This is not to say that accuracy
assessment is unimportant for the more traditional
remote sensing techniques. However, given the
complexity of digital classification, there is more of
a need to assess the reliability of the results.
Traditionally, the accuracy of photointerpretation
has been accepted as correct without any confir-
mation. In fact, digital classifications are often
assessed with reference to photointerpretation. An
Address correspondence to R. G. Congalton, 145 Mulford Hall,
Department of Forestry and Resource Management, University of
California, Berkeley, CA 94720.
Received 15 October 1990; revised 14 April 1991.
oo34-42s7/91/$3.50
obvious assumption made here is that the photoin-
terpretation is 100% correct. This assumption is
rarely valid and can lead to a rather poor and
unfair assessment of the digital classification
(Biging and Congalton, 1989).
Therefore, it is essential that researchers and
users of remotely sensed data have a strong knowl-
edge of both the factors needed to be considered
as well as the techniques used in performing any
accuracy assessment. Failure to know these tech-
niques and considerations can severely limit one's
ability to effectively use remotely sensed data. The
objective of this paper is to provide a review of the
appropriate analysis techniques and a discussion of
the factors that must be considered when perform-
ing any accuracy assessment. Many analysis tech-
niques have been published in the literature; how-
ever, I believe that it will be helpful to many
novice and established users of remotely sensed
data to have all the standard techniques summa-
rized in a single paper. In addition, it is important
to understand the analysis techniques in order to
fully realize the importance of the various other
considerations for accuracy assessment discussed
in this paper.
TECHNIQUES
Until recently, the idea of assessing the classifica-
tion accuracy of remotely sensed data was treated
35
36 Congalton

more as an afterthought than as an integral part of
any project. In fact, as recently as the early 1980s
many studies would simply report a single number
to express the accuracy of a classification. In many
of these cases the accuracy reported was what is
called non-site-specific accuracy. In a non-site-
specific accuracy assessment, locational accuracy is
completely ignored. In other words, only total
amounts of a category are considered without re-
gard for the location. If all the errors balance out,
a non-site-specific accuracy assessment will yield
very high but misleading results. In addition, most
assessments were conducted using the same data
set as was used to train the classifier. This training
and testing on the same data set also results in
overestimates of classification accuracy.
Once these problems were recognized, many
more site specific accuracy assessments were per-
formed using an independent data set. For these
assessments, the most common way to represent
the classification accuracy of remotely sensed data
is in the form of an error matrix. Using an error
matrix to represent accuracy has been recom-
mended by many researchers and should be
adopted as the standard reporting convention. The
reasons for choosing the error matrix as the stan-
dard are clearly demonstrated in this paper.
An error matrix is a square array of numbers
set out in rows and columns which express the
number of sample units (i.e., pixels, clusters of
pixels, or polygons) assigned to a particular cate-
gory relative to the actual category as verified on
the ground (Table 1). The columns usually repre-
sent the reference data while the rows indicate the
classification generated from the remotely sensed
data. An error matrix is a very effective way to
represent accuracy in that the accuracies of each
category are plainly described along with both the
errors of inclusion (commission errors) and errors
of exclusion (omission errors) present in the classi-
fication.
Descriptive Techniques
The error matrix can then be used as a starting
point for a series of descriptive and analytical
statistical techniques. Perhaps the simplest de-
scriptive statistic is overall accuracy which is com-
puted by dividing the total correct (i.e., the sum of
the major diagonal) by the total number of pixels
in the error matrix. In addition, accuracies of
individual categories can be computed in a similar
manner. However, this case is a little more com-
plex in that one has a choice of dividing the
number of correct pixels in that category by either
the total number of pixels in the corresponding
row or the corresponding column. Traditionally,
the total number of correct pixels in a category is
divided by the total number of pixels of that
category as derived from the reference data (i.e.,
the column total). This accuracy measure indicates
the probability of a reference pixel being correctly

Table 1. A n E x a m p l e Error Matrix

Reference Data
lOW

D C BA SB total
D 65 4 22 24 115 Land Cover Categories
C 6 81 5 8 100 D = deciduous
BA 0 11 85 19 115 C = conifer

SB 4 7 3 90 104 BA = b a r r e n

column 75 103 115 141 434 SB = s h r u b

total

OVERALL ACCURACY =
3 2 1 / 4 3 4 = 74%

PRODUCER'S ACCURACY USER'S ACCURACY

D=65/75= 87% D=65/115= 57%
C = 81/103 = 79% C= 81/100= 81%
BA = 8 5 / 1 1 5 = 74% BA=85/115= 74%
SB = 9 0 / 1 4 1 = 64% SB = 9 0 / 1 0 4 = 87%

classified and is really a measure of omission error.
This accuracy measure is often called "producer's
accuracy" because the producer of the classifica-
tion is interested in how well a certain area can be
classified. On the other hand, if the total number
of correct pixels in a category is divided by the
total number of pixels that were classified in that
category, then this result is a measure of commis-
sion error. This measure, called "user's accuracy"
or reliability, is indicative of the probability that a
pixel classified on the map/image actually repre-
sents that category on the ground (Story and
Congalton, 1986).
A very simple example quickly shows the ad-
vantages of considering overall accuracy, "pro-
ducer's accuracy," and "user's accuracy.'" The er-
ror matrix shown in Table 1 indicates an overall
map accuracy of 74%. However, suppose we are
most interested in the ability to classify deciduous
forests. We can calculate a "producer's accuracy"
for this category by dividing the total number of
correct pixels in the deciduous category (65) by
the total number of deciduous pixels as indicated
by the reference data (75). This division results in
a "producer's accuracy" of 87%, which is quite
good. If we stopped here, one might conclude
that, although this classification has an overall
accuracy that is only fair (74%), it is adequate for
the deciduous category. Making such a conclusion
could be a very serious mistake. A quick calcula-
tion of the "user's accuracy" computed by dividing
the total number of correct pixels in the deciduous
category (65) by the total number of pixels classi-
fied as deciduous (115) reveals a value of 57%. In
other words, although 87% of the deciduous areas
have been correctly identified as deciduous, only
57% of the areas called deciduous are actually
deciduous. A more careful look at the error matrix
reveals that there is significant confusion in dis-
criminating deciduous from barren and shrub.
Therefore, although the producer of this map can
claim that 87% of the time an area that was
deciduous was identified as such, a user of this
map will find that only 57% of the time will an
area he visits that the map says is deciduous will
actually be deciduous.
Analytical Techniques
In addition to these descriptive techniques, an
error matrix is an appropriate beginning for many
Review: Assessing Classification Accuracy 37
analytical statistical techniques. This is especially
true of the discrete multivariate techniques. Start-
ing with Congalton et al. (1983), discrete multi-
variate techniques have been used for performing
statistical tests on the classification accuracy of
digital remotely sensed data. Since that time many
others have adopted these techniques as the stan-
dard accuracy assessment tools (e.g., Rosenfield
and Fitzpatrick-Lins, 1986; Hudson and Ramm,
1987; Campbell, 1987). Discrete multivariate tech-
niques are appropriate because remotely sensed
data are discrete rather than continuous. The data
are also binomially or multinomially distributed
rather than normally distributed. Therefore, many
common normal theory statistical techniques do
not apply. The following example presented in
Tables 2-9 demonstrates the power of these dis-
crete multivariate techniques. The example begins
with three error matrices and presents the results
of the analysis techniques.
Table 2 presents the error matrices generated
from using three different classification algorithms
to map a small area of Berkeley and Oakland,
California surrounding the University of California
campus from SPOT satellite data. The three classi-
fication algorithms used included a traditional su-
pervised approach, a traditional unsupervised ap-
proach, and a modified approach that combines
the supervised and unsupervised classifications to-
gether to maximize the advantages of each
(Chuvieco and Congalton, 1988). The classification
was a simple one using only four categories; forest
(F), industrial (I), urban (U), and water (W). All
three classifications were performed by a single
analyst. In addition, Table 3 presents the error
matrix generated for the same area using only the
modified classification approach by a second ana-
lyst. Each analyst was responsible for performing
an accuracy assessment. Therefore, different num-
bers of samples and different sample locations
were selected by each.
The next analytical step is to "normalize" or
standardize the error matrices. This technique uses
an iterative proportional fitting procedure which
forces each row and column in the matrix to sum
to one. In this way, differences in sample sizes
used to generate the matrices are eliminated and,
therefore, individual cell values within the matrix
are directly comparable. In addition, because as
part of the iterative process the rows and columns
are totaled (i.e., marginals), the resulting normal-
38 Congalton

Table 2. Error Matrices for the Three Classification Approaches from Analyst #1

Supervised Approach
Reference Data
F I U W

F 68 7 3 0

Classified I 12 112 15 10 Overall Accuracy =

Data 3 2 5 / 3 9 1 = 83%
U 3 9 89 0

W 0 2 5 56

UnsupervisedApproach
Reference Data
F I U W
F 60 11 3 4

Classified I 15 102 14 8 Overall Accuracy =

Data U 6 13 90 2 3 0 4 / 3 9 1 = 78%

W 2 4 5 52

Modified Approach
Reference Data
F I U W

F 75 6 1 0

Classified I 4 116 11 3 Overall Accuracy =

Data U 3 7 96 2 3 4 8 / 3 9 1 = 89%

W 1 1 4 61

Table 3. Error Matrix for the Modified Classification Approach from Analyst #2

ModifiedApproach - - Analyst # 2
Reference Data
F AG U W

F 35 6 1 0

Classified AG 3 82 5 10 overall accuracy =

Data 2 0 8 / 2 4 6 = 85%
U 4 2 54 0
W 0 5 2 37

ized matrix is more indicative of the off-diagonal
cell values (i.e., the errors of omission and com-
mission). In other words, all the values in the
matrix are iteratively balanced by row and column
thereby incorporating information from that row
and column into each individual cell value. This
process then changes the cell values along the
major diagonal of the matrix (correct classifica-
tions) and therefore a normalized overall accuracy
can be computed for each matrix by summing the
major diagonal and dividing by the total of the
entire matrix. Consequently, one could argue that
the normalized accuracy is a better representation
of accuracy than is the overall accuracy computed
from the original matrix because it contains infor-
mation about the off-diagonal cell values. Table 4
presents the normalized matrices from the same
three classification algorithms for analyst # 1 gen-
erated using a computer program called MARGFIT
(marginal fitting). Table 5 presents the normalized
 Review: Assessing ClassificationAccuracy 39

Table 4. Normalized Error Matrices for the Three Classification Approaches from Analyst #1

Supervised Approach
Reference Data
F I U W
F 0.8652 0.0940 0.0331 0.0073
Classified I 0.0845 0.7547 0.0784 0.0824 Normalized Accuracy =
Data
U 0.0435 0.1171 0.8319 0.0072 3 . 3 5 4 9 / 4 = 84%
W 0.0069 0.0342 00567 0.9031

UnsupervisedApproach
Reference Data
F I U W
F 0.7734 0.1256 0.0387 0.0622

Classified I 0.1242 0.7014 0.1006 0.0824 Normalized Accuracy =

Data U 0.0656 0.1163 0.7094 0.0273 5.1022/4 = 78%

W 0.0369 0.0567 0.0702 0.8370

Modified Approach
Reference Data
F I U W
F 0.9080 0.0687 0.0152 0.0076

Classified I 0.0372 0.8460 0.0801 0.0366 Normalized Accuracy =

Data U 0.0370 0.0697 0.8598 0.0334 3.5362/ 4 = 88%

W 0.0178 0.0156 0.6450 0.9224

Table 5. Normalized Error Matrix for the Modified Approach from Analyst #2

ModifiedApproach- - Analyst # 2
Reference Data
AG U W
F 0.8519 0.1090 0.0287 0.0113

Classified AG 0.0464 0.7641 0.0581 0.1313 Overall Accuracy =

Data U 0.0897 0.0348 0.8655 0.0094 3.3295/ 4 = 83%

W 0.0120 0.0921 0.0477 0.8480 I

matrix for the modified approach performed by
analyst #2.
In addition to computing a normalized accu-
racy, the normalized matrix can also be used to
directly compare cell values between matrices.
For example, we may be interested in comparing
the accuracy each analyst obtained for the forest
category using the modified classification ap-
proach. From the original matrices we can see that
analyst #1 classified 75 sample units correctly
while analyst # 2 classified 35 correctly. Neither of
these numbers means much because they are not
directly comparable due to the differences in the
number of samples used to generate the error
matrix by each analyst. Instead, these numbers
would need to be converted into percent so that a
comparison could be made. Here another problem
arises: Do we divide the total correct by the row
total (user's accuracy) or by the column total (pro-
ducer's accuracy)? We could calculate both and
40 Congalton
compare the results or we could use the cell value
in the normalized matrix. Because of the iterative
proportional fitting routine, each cell value in the
matrix has been balanced by the other values in its
corresponding row and column. This balancing has
the effect of incorporating producer's and user's
accuracies together. Also since each row and col-
umn add to 1, an individual cell value can quickly
be converted to a percentage by multiplying by
100. Therefore, the normalization process provides
a convenient way of comparing individual cell
values between error matrices regardless of the
number of samples used to derive the matrix.
Another discrete multivariate technique of use
in accuracy assessment is called KAPPA (Cohen,
1960). The result of performing a KAPPA analysis
is a KHAT statistic (an estimate of KAPPA), which
is another measure of agreement or accuracy. The
KHAT statistic is computed as
/~= i=1 i=1
F
N2- E (x,+ • x+,)
i=1
where r is the number of rows in the matrix, xii is
the number of observations in row i and column i,
x i+ and x +i are the marginal totals of row i and
column i, respectively, and N is the total number
of observations (Bishop et al., 1975). The KHAT
equation is published in this paper to clear up
some confusion caused by a typographical error in
Congalton et al. (1983), who originally proposed
the use of this statistic for remotely sensed data.
Since that time, numerous papers have been pub-
lished recommending this technique. The equa-
tions for computing the variance of the KHAT
statistic and the standard normal deviate can be
found in Congalton et al. (1983), Rosenfield and
Fitzpatriek-Lins (1986), and Hudson and Ramm
(1987) to list just a few. It should be noted that
the KHAT equation assumes a multinomial sam-
pling model and that the variance is derived using
the Delta method.
Table 6 provides a comparison of the overall
accuracy, the normalized accuracy, and the KHAT
statistic for the three classification algorithms used
by analyst # 1. In this particular example, all three
measures of accuracy agree about the relative
ranking of the results. However, it is possible for
these rankings to disagree simply because each
Table 6. A Comparison of the Three Accuracy Measures for
the Three Classification Approaches
Classification Overall KHAT Normalized
Algorithm Accuracy Accuracy Accuracy
Supervised
approach 84% 77% 83%
Unsupervised
approach 78% 70% 78%
Modified
approach 88% 85% 89%
measure incorporates various levels of information
from the error matrix into its computations. Over-
all accuracy only incorporates the major diagonal
and excludes the omission and commission errors.
As already described, normalized accuracy directly
includes the off-diagonal elements (omission and
commission errors) because of the iterative pro-
portional ftting procedure. As shown in the KHAT
equation, KHAT accuracy indirectly incorporates
the off-diagonal elements as a product of the row
and column marginals. Therefore, depending on
the amount of error included in the matrix, these
three measures may not agree. It is not possible to
give cleareut rules as to when each measure should
be used. Each accuracy measure incorporates dif-
ferent information about the error matrix and
therefore must be examined as different computa-
tions attempting to explain the error. My experi-
ence has shown that if the error matrix tends to
have a great many off-diagonal cell values with
zeros in them, then the normalized results tend to
disagree with the overall and Kappa results. Many
zeros occur in a matrix when an insuflqeient sam-
ple has been taken or when the classification is
exceptionally good. Because of the iterative pro-
portional fitting routine, these zeros tend to take
on positive values in the normalization process
showing that some error could be expected. The
normalization process then tends to reduce the
accuracy because of these positive values in the
off-diagonal cells. If a large number of off-diagonal
cells do not contain zeros, then the results of the
three measures tend to agree. There are also times
when the Kappa measure will disagree with the
other two measures. Because of the ease of com-
puting all three measures (software is available
from the author) and because each measure re-
fleets different information contained within the
error matrix, I recommend an analysis such as the

Table 7. Results of the KAPPA Analysis Test of Significance
for Individual Error Matrices
Test of Significance of Each Error Matrix
Classification Algorithms KHAT Statistic Z Statistic Result"
Supervised approach .7687 29.41
Unsupervised approach .6956 24.04
Modified approach .8501 39.23
~At the 95% confidence level.
bS = significant.
Table 8. Results of KAPPA Analysis for Comparison be-
tween Error Matrices for Analyst # 1
Test of Significant Differences between Error Matrices
Comparison Z Statistic
Supervised vs. unsupervised 1.8753
Supervised vs. modified 2.3968
Unsupervised vs. modified 4.2741
°At the 95% confidence level.
I~S= significant, NS = not significant.
one performed here to glean as much information
from the error matrix as possible.
In addition to being a third measure of accu-
racy, KAPPA is also a powerful technique in its
ability to provide information about a single matrix
as well as to statistically compare matrices. Table 7
presents the results of the KAPPA analysis to test
the significance of each matrix alone. In other
words, this test determines whether the results
presented in the error matrix are significantly bet-
ter than a random result (i.e., the null hypothesis:
KHAT = 0). Table 8 presents the results of the
KAPPA analysis that compares the error matrices
two at a time to determine if they are significantly
different. This test is based on the standard normal
deviate and the fact that, although remotely sensed
data are discrete, the KHAT statistic is asymptoti-
cally normally distributed. A quick look at Table 8
shows why this test is so important. Despite the
overall accuracy of the supervised approach being
6% higher than the unsupervised approach (84%
- 7 8 % = 6%), the results of the KAPPA analysis
show that these two approaches are not signifi-
cantly different. Therefore, given the choice of
only these two approaches, one should use the
easier, quicker, or more efficient approach because
the accuracy will not be the deciding factor. Simi-
lar results are presented in Table 9 comparing the
modified classification approach for analyst #1
with analyst #2.
Review: Assessing Classification Accuracy 41
Table 9. Results of KAPPA Analysis for Comparison be-
tween Modified Approach for Analyst # 1 vs. Analyst # 2
Test of Significant Differences between Error Matrices
Comparison Z Statistic Result ~
Sb Modified #1 vs. modified #2 1.6774 NS b
S °At the 95% confidence level.
/'NS = not significant.
S
In addition to the discrete multivariate tech-
niques just presented, other techniques for assess-
ing the accuracy of remotely sensed data have also
been suggested. Rosenfield (1981) proposed the
use of analysis of variance techniques for accuracy
Result ~ assessment. However, violation of the normal the-
NS/~ ory assumption and independence assumption
S when applying this technique to remotely sensed
S data has severely limited its application. Aronoff
(1985) suggested the use of a minimum accuracy
value as an index of classification accuracy. This
approach is based on the binomial distribution of
the data and is therefore very appropriate for
remotely sensed data. The major disadvantage of
the Aronoff approach is that it is limited to a single
overall accuracy value rather than using the entire
error matrix. However, it is useful in that it this
index does express statistically the uncertainty
involved in any accuracy assessment. Finally,
Skidmore and Turner (1989) have begun work on
techniques for assessing error as it accumulates
through many spatial layers of information in a
GIS, including remotely sensed data. These tech-
niques have included using a line sampling method
for accuracy assessment as well as probability the-
ory to accumulate error from layer to layer. It is in
this area of error analysis that much new work
needs to be performed.
CONSIDERATIONS
Along with the actual analysis techniques, there
are many other considerations to note when per-
forming an accuracy assessment. In reality, the
techniques are of little value if these other factors
are not considered because a critical assumption of
all the analysis described above is that the error
matrix is truly representative of the entire classi-
fication. If the matrix is improperly generated,
then all the analysis is meaningless. Therefore, the
42 Congalton
following factors must be considered: ground data
collection, classification scheme, spatial autocorre-
lation, sample size, and sampling scheme. Each of
these factors provide essential information for the
assessment and failure to consider even one of
them could lead to serious shortcomings in the
assessment process.
Ground Data Collection
It is obvious that in order to adequately assess the
accuracy of the remotely sensed classification, ac-
curate ground, or reference data must be col-
lected. However, the accuracy of the ground data
is rarely known nor is the level of effort needed to
collect the appropriate data clearly understood.
Depending on the level of detail in the classifica-
tion (i.e., classification scheme), collecting refer-
ence data can be a very difficult task. For example,
in a simple classification scheme the required level
of detail may be only to distinguish residential
from commercial areas. Collecting reference data
may be as simple as obtaining a county zoning
map. However, a more complex forest classifica-
tion scheme may involve collecting reference data
for not only species of tree, but size class, and
crown closure as well. Size class involves measur-
ing the diameters of trees and therefore a great
many trees may have to be measured to estimate
the size class for each pixel. Crown closure is even
more difficult to measure. Therefore, in this case,
collecting accurate reference data can be difficult.
A traditional solution to this problem has been
for the producer and user of the classification to
assume that some reference data set is correct. For
example, the results of some photointerpretation
or aerial reconnaissance may be used as the refer-
ence data. However, errors in the interpretation
would then be blamed on the digital classification,
thereby wrongly lowering the digital classification
accuracy. It is exactly this problem that has caused
the lack of acceptance of digital satellite data for
many applications. Although no reference data set
may be completely accurate, it is important that
the reference data have high accuracy or else it is
not a fair assessment. Therefore, it is critical that
the ground or reference data collection be care-
fully considered in any accuracy assessment. Much
work is yet to be done to determine the proper
level of effort and collection techniques necessary
to provide this vital information.
Classification Scheme
When planning a project involving remotely sensed
data, it is very important that sufficient effort be
given to the classification scheme to be used.
In many instances, this scheme is an existing
one such as the Anderson classification system
(Anderson et al., 1976). In other cases, the classi-
fication scheme is dictated by the objectives of the
project or by the specifications of the contract. In
all situations a few simple guidelines should be
followed. First of all, any classification scheme
should be mutually exclusive and totally exhaus-
tive. In other words, any area to be classified
should fall into one and only one category or class.
In addition, every area should be included in the
classification. Finally, if possible, it is very advan-
tageous to use a classification scheme that is hier-
archical in nature. If such a scheme is used, cer-
tain categories within the classification scheme can
be collapsed to form more general categories. This
ability is especially important when trying to meet
predetermined accuracy standards. Two or more
detailed categories of lower than the minimum
required accuracy may need to be grouped to-
gether (collapsed) to form a more general category
that exceeds the minimum accuracy requirement.
For example, it may be impossible to separate
interior live oak from canyon live oak. Therefore,
these two categories may have to be collapsed to
form a live oak category to meet the required
accuracy standard.
Because the classification scheme is so impor-
tant, no work should begin on the remotely sensed
data until the scheme has been thoroughly re-
viewed and as many problems as possible identi-
fied. It is especially helpful if the categories in the
scheme can be logically explained. The difference
between Douglas fir and Ponderosa pine is easy to
understand; however, the difference between
Density Class 3 (50-70% crown closure) and Den-
sity Class 4 ( > 70% crown closure) may not be. In
fact, many times these classes are rather artificial
and one can expect to find confusion between a
forest stand with a crown closure of 67% that
belongs in Class 3 and a stand of 73% that belongs
in Class 4. Sometimes there is little that can be
done about the artificial delineations in the classi-
fication scheme; other times the scheme can be
modified to better represent natural breaks. How-
ever, tZailure to try to understand the classification

scheme from the every beginning will certainly
result in a great loss of time and much frustration
in the end.
Spatial Autocorrelation
Spatial autoeorrelation is said to occur when the
presence, absence, or degree of a certain charac-
teristic affects the presence, absence, or degree of
the same characteristic in neighboring units (Cliff
and Ord, 1973). This condition is particularly im-
portant in accuracy assessment if an error in a
certain location can be found to positively or nega-
tively influence errors in surrounding locations
(Campbell, 1981). Work by Congalton (1988a) on
Landsat MSS data from three areas of varying
spatial diversity (i.e., an agriculture, a range, and a
forest site) showed a positive influence as much as
30 pixels (over 1 mile) away. These results are
explainable in an agricultural environment where
field sizes are large and typical misclassification
would be to make an error in labeling the entire
field. However, these results are more surprising
for rangeland and forested sites. Surely these re-
suits should affect the sample size and especially
the sampling scheme used in accuracy assessment,
especially in the way this autocorrelation affects
the assumption of sample independence. This au-
tocorrelation may then be responsible for periodic-
ity in the data that could effect the results of any
type of systematic sample. In addition, the size of
the cluster used in cluster sampling would also be
effected because each new pixel would not be
contributing independent information.
Sample Size
Sample size is another important consideration
when assessing the accuracy of remotely sensed
data. Each sample point collected is expensive and
therefore sample size must be kept to a minimum
and yet it is critical to maintain a large enough
sample size so that any analysis performed is
statistically valid. Of all the considerations dis-
cussed in this paper, the most has probably been
written about sample size. Many researchers, no-
tably van Genderen and Lock (1977), Hay (1979),
Hord and Brooner (1976), Rosenfield et al. (1982),
and Congalton (1988b), have published equations
and guidelines for choosing the appropriate sam-
ple size. The majority of researchers have used an
Review: Assessing Classification Accuracy 43
equation based on the binomial distribution or the
normal approximation to the binomial distribution
to compute the required sample size. These tech-
niques are statistically sound for computing the
sample size needed to compute the overall accu-
racy of a classification or even the overall accuracy
of a single category. The equations are based on
the proportion of correctly classified samples
(pixels, clusters, or polygons) and on some allow-
able error. However, these techniques were not
designed to chose a sample size for filling in an
error matrix. In the case of an error matrix, it is
not simply a matter of correct or incorrect. It is a
matter of which error or, in other words, which
categories are being confused. Sufficient samples
must be acquired to be able to adequately repre-
sent this confusion. Therefore, the use of these
techniques for determining the sample size for an
error matrix is not appropriate. Fitzpatrick-Lins
(1981) used the normal approximation equation to
compute the sample size for assessing a land u s e /
land cover map of Tampa, Florida. The results of
the computation showed that 319 samples needed
to be taken for a classification with an expected
accuracy of 85% and an allowable error of 4%. She
ended up taking 354 samples and filling in an
error matrix that had 30 categories in it (i.e., a
matrix of 30 rows × 3 0 columns or 900 possible
cells). Although this sample size is sufficient for
computing overall accuracy, it is obviously much
too small to be represented in a matrix. Only 35 of
the 900 cells had a value greater than zero. Other
researchers have used the equation to compute the
sample size for each category. Although resulting
in a larger sample, the equation still does not
account for the confusion between categories.
Because of the large number of pixels in a
remotely sensed image, traditional thinking about
sampling does not often apply. Even a one-half
percent sample of a single Thematic Mapper scene
can be over 300,000 pixels. Not all assessments are
performed on a per pixel basis, but the same
relative argument holds true if the sample unit is a
cluster of pixels or a polygon. Therefore, practical
considerations more often dictate the sample size
selection. A balance between what is statistically
sound and what is practically attainable must be
found. It has been my experience that a good rule
of thumb seems to be collecting a minimum of 50
samples for each vegetation or land use category
in the error matrix. If the area is especially large
44 Congalton
(i.e., more than a million acres) or the classifica-
tion has a large number of vegetation or land use
categories (i.e., more than 12 categories), the mini-
mum number of samples should be increased to 75
or 100 samples per category. The number of sam-
ples for each category can also be adjusted based
on the relative importance of that category within
the objectives of the mapping or by the inherent
variability within each of the categories. Some-
times it is better to concentrate the sampling on
the categories of interest and increase their num-
ber of samples while reducing the number of
samples taken in the less important categories.
Also it may be useful to take fewer samples in
categories that show little variability such as water
or forest plantations and increase the sampling in
the categories that are more variable such as un-
even-aged forests or riparian areas. Again, the
object here is to balance the statistical recommen-
dations in order to get an adequate sample to
generate an appropriate error matrix with the time,
cost, and practical limitations associated with any
viable remote sensing project.
Sampling Scheme
In addition to the considerations already dis-
cussed, sampling scheme is an important part of
any accuracy assessment. Selection of the proper
scheme is absolutely critical to generating an error
matrix that is representative of the entire classified
image. Poor choice in sampling scheme can result
in significant biases being introduced into the
error matrix which may over or under estimate the
true accuracy. In addition, use of the proper sam-
pling scheme may be essential depending on the
analysis techniques to be applied to the error
matrix.
Many researchers have expressed opinions
about the proper sampling scheme to use (e.g.,
Hord and Brooner, 1976; Ginevan, 1979; Rhode,
1978; Fitzpatrick-Lins, 1981). These opinions vary
greatly among researchers and include everything
from simple random sampling to stratified system-
atic unaligned sampling. Despite all these opin-
ions, very little work has actually been performed
in this area. Congalton (1988b) performed sam-
pling simulations on three spatially diverse areas
and concluded that in all cases simple random
without replacement and stratified random sam-
pling provided satisfactory results. Despite the
nice statistical properties of simple random sam-
pling, this sampling scheme is not always that
practical to apply. Simple random sampling tends
to undersample small but possibly very important
areas unless the sample size is significantly in-
creased. For this reason, stratified random sam-
piing is recommended where a minimum number
of samples are selected from each strata (i.e.,
category). Even stratified random sampling can be
somewhat impractical because of having to collect
ground information for the accuracy assessment at
random locations on the ground. The problems
with random locations are that they can be in
places with very difficult access and they can only
be selected after the classification has been per-
formed. This limits the accuracy assessment data
to being collected late in the project instead of in
conjunction with the training data collection,
thereby increasing the costs of the project. In
addition, in some projects the time between the
project beginning and the accuracy assessment
may be so long as to cause temporal problems in
collecting ground reference data. In other words,
the ground may change (i.e., the forest harveste d )
between the time the project is started and the
accuracy assessment is begun.
Therefore, some systematic approach would
certainly help make this ground collection effort
more efficient by making it easier to locate the
points of the ground and allowing data to be
collected simultaneously for training and assess-
ment. However, results of Congalton (1988a)
showed that periodicity in the errors as measured
by the autocorrelation analysis could make the use
of systematic sampling risky for accuracy assess-
ment. Therefore, perhaps some combination of
random and systematic sampling would provide
the best balance between statistical validity and
practical application. Such a system may employ
systematic sampling to collect some assessment
data early in a project while random sampling
within strata would be used after the classification
is completed to assure that enough samples were
collected for each category and to minimize any
periodicity in the data.
In addition to the sampling schemes already
discussed, cluster sampling has also been fre-
quently used in assessing the accuracy of remotely
sensed data, especially to collect information on
many pixels very quickly. However, cluster sam-
piing must be used intelligently. Simply using

very large clusters is not a valid method of collect-
ing data because each pixel is not independent of
the other and adds very little information to the
cluster. Congalton (1988b) recommended that no
clusters larger than 10 pixels and certainly not
larger than 25 pixels be used because of the lack
of information added by each pixel beyond these
cluster sizes.
Finally, some analytic techniques assume that
certain sampling schemes were used to obtain the
data. For example, use of the Kappa analysis as-
sumes a multinomial sampling model. Only simple
random sampling completely satisfies this assump-
tion. The effect of using another of the sampling
schemes discussed here is unknown. An interest-
ing project would be to test the effect on the
Kappa analysis of using a sampling scheme other
than simple random sampling. If the effect is
found to be small, then the scheme may be appro-
priate to use within the conditions discussed above.
If the effect is found to be large, then that sam-
pling scheme should not be used to perform Kappa
analysis. To conclude that some sampling schemes
can be used for descriptive techniques and others
for analytical techniques seems impractical. Accu-
racy assessment is expensive and no one is going
to collect data for only descriptive use. Eventually,
someone will use that matrix for some analytical
technique.
CONCLUSIONS
This paper has reviewed the factors and tech-
niques to be considered when assessing the accu-
racy of classifications of remotely sensed data. The
work has really just begun. The factors discussed
here are certainly not fully understood. The basic
issues of sample size and sampling scheme have
not been resolved. Spatial autocorrelation analysis
has rarely been applied to any study. Exactly what
constitutes ground or reference data and the level
of effort needed to collect it must be studied.
Research needs to continue in order to balance
what is statistically valid within the realm of prac-
tical application. This need becomes increasingly
important as techniques are developed to use re-
motely sensed data over large regional and global
domains. What is valid and practical over a small
area may not apply to regional or global projects.
Up to now, the little experience we have has been
Review: Assessing Classification Accuracy 45
on relatively small remote sensing projects. How-
ever, there is a need to use remote sensing for
much larger projects such as monitoring global
warming, deforestation, and environmental degra-
dation. We do not know all the problems that will
arise when dealing with such large areas. Cer-
tainly, the techniques described must be extended
and refined to better meet these assessment needs.
It is critical that this work and the use of quantita-
tive analysis of remotely sensed data continue. We
have suffered too long because of the oversell of
the technology and the underutilization of any
quantitative analysis early in the digital remote
sensing era. Papers such as Meyer and Werth
(1990) that state that the digital remote sensing is
not a viable tool for most resource applications
continue to demonstrate the problems we have
created by not quantitatively documenting our
work. We must put aside the days of a casual
assessment of our classification. "It looks good" is
not a valid accuracy statement. A classification is
not complete until it has been assessed. Then and
only then can the decisions made based on that
information have any validity.
In addition, we must not forget that remotely
sensed data is just a small subset of spatial data
currently being used in geographic information
systems (GIS). The techniques and considerations
discussed here need to be applied over all spatial
data. Techniques developed for other spatial data
need to be tested for use with remotely sensed
data. The work has just begun, and if we are going
to use spatial data to help us make decisions, and
we should, then we must know about the accuracy
of this information.
The author would like to thank Greg Biging and Craig Olson
for their helpful reviews of this paper. Thanks also to the two
anonymous reviewers whose comments significantly improved
this manuscript.
REFERENCES
Anderson,J. R., Hardy, E. E., Roach,J. T., and Witmer, R. E.
(1976), A land use and land cover classificationsystem for
use with remote sensor data, U.S. Geol. Survey Prof. Paper
964, 28 pp.
Aronoff, Stan (1985), The minimum accuracy value as an
index of classificationaccuracy, Photogramm. Eng. Remote
Sens. 51(1):99-111.
Biging, G. and Congalton, R. (1989), Advances in forest
inventory using advanced digital imagery, in Proceedings
46 Congalton
of Global Natural Research Monitoring and Assessments':
Preparing for the 21st Century, Venice, Italy, September,
Vol. 3, pp. 1241-1249.
Bishop, Y., Fienberg, S., and Holland, P. (1975), Discrete
Multivariate Analysis--Theory and Practice, MIT Press,
Cambridge, MA, 575 pp.
Campbell, J. (1981), Spatial autocorrelation effects upon the
accuracy of supervised classification of land cover, Pho-
togramm. Eng. Remote Sens. 47(3):355-363.
Campbell, J. (1987), Introduction to Remote Sensing, Guilford
Press, New York, 551 pp.
Chuvieco, E., and Congalton, R. (1988), Using cluster analysis
to improve the selection of training statistics in classifying
remotely sensed data, Photogramm. Eng. Remote Sens.
54(9): 1275-1281.
Cliff, A. D., and Ord, J. K. (1973), Spatial Autocorrelation,
Pion, London, 178 pp.
Cohen, J. (1960), A coefficient of agreement for nominal
scales, Educ. Psychol. Measurement 20(1):37-46.
Congalton, R. G. (1988a), Using spatial autocorrelation analy-
sis to explore errors in maps generated from remotely
sensed data, Photogramm. Eng. Remote Sens. 54(5):
587-592.
Congalton, R. G. (1988b), A comparison of sampling schemes
used in generating error nmtrices for assessing the accu-
racy of maps generated from remotely sensed data, Pho-
togramnt. Eng. Remote Sens. 54(5):593-600.
Congalton, R. G., Oderwald, R. G., and Mead, R. A. (1983),
Assessing Landsat classification accuracy using discrete
multivariate statistical techniques, Photogramm. Eng. Re-
mote Sens. 49(12):1671-1678.
Fitzpatrick-Lins, K. (1981), Comparison of sampling proce-
dures and data analysis for a land-use and land-cover map,
Photogramm. Eng. Remote Sens. 47(3):343-351.
Ginevan, M. E. (1979), Testing land-use map accuracy: an-
other look, Photogramm. Eng. Remote Sens. 45(10):
1371-1377.
Hay, A. M. (1979), Sampling designs to test land-use map
accuracy, Photogramm. Eng. Remote Sens. 45(4):529-533.
Hord, R. M., and Brooner, W. (1976), Land use map accuracy
criteria, Photogramm. Eng. Remote Sens. 42(5):671-677.
Hudson, W., and Ramm, C. (1987), Correct formulation of the
kappa coefficient of agreement, Photogramm. Eng. Remote
Sens. 53(4):421-422.
Meyer, M., and Werth, k (1990), Satellite data: management
panacea of potential problem?, J. Forestry 88(9):10-13.
Rhode, W. G. (1978), Digital image analysis techniques for
natural resource inventory, in National Computer Confer-
ence Proceedings, pp. 43-106.
Rosenfield, G. (1981), Analysis of variance of thematic map-
ping experiment data, Photogramm. Eng. Remote Sens.
47(12):1685-1692.
Rosenfield, G., and Fitzpatrick-Lins, K. (1986), A coefficient
of agreement as a measure of thematic classification accu-
racy, Photogramm. Eng. Remote Sens. 52(2):223-227.
Rosenfield, G. H., Fitzpatriek-Lins, K., and Ling, H. (1982),
Sampling tbr thematic map accuracy testing, Photogramm.
Eng. Remote Sens. 48(1):131-137.
Skidmore, A., and Turner, B. (1989), Assessing the accuracy
of resource inventory maps, in Proceedings of Global
Natural Resource Monitoring and Assessments: Preparing
for the 21st Century, Venice, Italy, September, Vol. 2, pp.
524-535.
Story, M., and Congalton, R. (1986), Accuracy assessment: a
user's perspective, Photogramm. Eng. Remote Sens.
52(3):397-399.
van Genderen, J. L., and Lock, B. F. (1977), Testing land use
map accuracy, Photogramm . Eng. Remote Sens.
43(9):1135-1137.