LECTURE 10:

ACCURACY ASSESSMENT
IN REMOTELY SENSED
CATEGORICAL
INFORMATION
 A quick review of Lecture 9
(Classification)
• Thematic mapping based on mage classification
1) use spectral (radiometric) differences to
distinguish spatial classes
2) supervised / unsupervised mode
 We will discuss

• Why ?
• What ?
• How ?
for accuracy assessment
• Key points:
to construct confusion matrix
to compute accuracy measures
• Difficult points:
Statistics for sampling design and
spatial analysis
Why do we bother accuracy assessment?
• Thematic maps of land cover, forest types,
and others can be derived from classification
of remotely sensed imagery in combination
with ancillary data sets
• You need to tell map users how well it
actually represents what’s out there
• “Without an accuracy assessment, a
classified map is just a pretty picture.”
 ACCURACY ASSESSMENT
• The accuracy of information derived from
remotely sensed images will always be asked
• Sometime is very difficult to address
• Accuracy of a map or image can be evaluated
and identifies
• Accuracy defines “correctness”; it measures
the agreement between a standard assumed to
be correct and classified image of unknown
quality.
 Accuracy Assessment
• If the image classification corresponds closely
with the standard, it is said to be “accurate”
• The accuracy assessment task can be defined
as one of comparing two maps.
• One based upon analysis of remotely sensed
data (the map to be evaluated) and another
upon a different source of information. The
second map is the reference map, assumed to
be accurate, that forms the standard for
comparison.
What is accuracy assessment?

• Assess how well a classifier

works
• Interpret the usefulness of
someone else’s classification
How do we do accuracy assessment?

• Collect reference data, i.e., “ground

truth”
determining class types at specific
locations
• Compare a map with the reference to
compute accuracy measures
• Interpretation of the results
 Reference Data - possible sources

• Aerial photo interpretation

• Ground truthing with GPS
• GIS layers

• Make sure we can actually extract from the

reference source the information needed
• For discriminating four species of grass, we may
need ground surveys not aerial photographs
Determining size of reference plots

• Match spatial scale of reference plots and

remotely-sensed data
• Ground plots (5 meters on a side) may not
be useful for remotely-sensed imagery
(1km)
may need aerial photos or even other
satellite imagery.
• Take into account spatial frequencies of
image
consider photo reference plots that
cover an area 3 pixels on a side

Example 1: Low spatial Example 2: High spatial

frequency Homogeneous frequency Heterogeneous
• Implication for positional accuracy
consider the situation where accuracy
of position of the image is +/- one pixel

Example 1: Low Example 2: High

spatial frequency spatial frequency
 Determining number and position of
samples
• Make sure to adequately sample the landscape
• Variety of sampling schemes:
Random, stratified random, systematic, etc.
• The more reference plots, the better
You can estimate how many you need statistically
In reality, you can never get enough
Lillesand and Kiefer: suggest 50 per class as rule
of thumb
 Sample Size (N) Calculation
Fitzpatrick-Lins (1981) suggests that the sample size N to be
used to assess the accuracy of a land-use classification map
be determined from the formula for the binomial probability
theory:
Z 2 ( p)(q)
N
E2
where p is the expected percent accuracy of the entire map,
q = 100 – p, E is the allowable error, and Z = 2 from the
standard normal deviate of 1.96 for the 95% two-sided
confidence level.
For a sample for which the expected accuracy is 85% at an
allowable error of 5% (i.e., it is 95% accurate), the number of
points necessary for reliable results is:

Z 2 ( p)(q)
N
E2
2
2 (85)(15)
N 2
 a minimum of 203points.
5
With expected map accuracies of 85% and an acceptable
error of 10%, the sample size for a map would be 51:

2
2 (85)(15)
N 2
 51 points
10
 Accuracy Assessment
• Sampling Approaches: to reduce analyst
bias
– simple random sampling: every pixel has equal
chance
– stratified random sampling: # of points will be
stratified to the distribution of thematic layer
classes (larger classes more points)
– equalized random sampling: each class will have
equal number of random points
– Sample size: at least 30 samples per land cover
class
Sampling Methods

Simple Random Sampling

observations are :
randomly placed.
Sampling Methods

Systematic Sampling :
Observations are placed at equal
intervals according to a strategy

Systematic Non-Aligned Sampling

a grid provides even distribution of
randomly placed observations
Sampling Methods

Cluster Sampling
Randomly placed “centroids”
used as a base of several nearby
Observations, which can be
Selected randomly or
systematically
 The Error Matrix
• The standard form for reporting site-specific
error is the error matrix sometimes referred
to as the confusion matrix.
• It identifies not only overall errors for each
category, but also misclassifications (due to
confusion between categories) by category.
• Thomlinson et al. (1999) set a target of an
overall accuracy of 85% with no class less
than 70% accurate
 An example confusion matrix
Class types determined from
reference
# Plots Conifer deciduous grass Totals User’s
Class types Accuracy
determined
from classified
map
Conifer 50 5 2 57 88%
deciduous 14 13 0 27 48%
grass 3 5 8 16 50%
Totals 67 23 10 100
Producer’s Accuracy 75% 57% 80% Total (PCC):
71%
 Interpreting results of accuracy
assessment
• Misclassification in remotely-sensed data:
Classes are land use, not land cover
Classes not spectrally separable
Spatial scale of remote sensing instrument
does not match classification scheme
• Error in reference data:
Interpreter error
Subjectivity
 Improving Classification
• Land use/land cover: incorporate other data
 Elevation, temperature, ownership, etc.
 Context
• Spectral inseparability
 Hyperspectral
 Multiple dates
• Scale
 Different sensor
 Aggregate pixels
• Classifiers
 Use HIERARCHICAL CLASSIFICATION scheme
 In Maximum Likelihood classification, use Prior Probabilities to
weigh minority classes more
 Summary
• Accuracy assessment to add value to remote
sensing information products and to ensure their
proper use
• Ground truth is itself difficult to acquire
• Sampling design is important for cost-
effectiveness in accuracy assessment
• Accuracy assessment as an integral component in
the information process
 References
• Cochran, W.G. 1977.  Sampling techniques.  Wiley, New York.
• Congalton, R. 1991.  A review of assessing the accuracy of
classifications of remotely sensed data.  Remote Sensing of
Environment 37:35-46.
• Nusser, S.M., and E.E. Klaas.  2002.  Final performance report to
EPA Region 7, Part II:  GAP accuracy assessment pilot study. 
Environmental Protection Agency Contract X997387-01 Final
Report.  Iowa Cooperative Fish and Wildlife Research Unit, Iowa
State University, Ames, Iowa.  77 pp.
• Stehman, S.V. 1997.  Selecting and interpreting measures of
thematic classification accuracy.  Remote Sensing of Environment
62:77-89.
THANK YOU!!
 EXAMPLE CALCULATION OF ACCURACY
ASSESSMENT

Table 1: Error matrix data