8.4.

VALIDATION OF THE RESULT 306

8.4 Validation of the result

Image classification results in a raster file in which the individual raster elements
are class labelled. As image classification is based on samples of the classes, the
actual quality of the result should be checked. This is usually done by a sam-
pling approach in which a number of raster elements of the output are selected
and both the classification result and the true world class are compared. Com-
parison is done by creating an ‘error matrix’, from which different accuracy mea-
sures can be calculated. The ‘true world class’ is preferably derived from field
observations. Sometimes sources of an assumed higher accuracy, such as aerial
photos, are used as a reference.

Various sampling schemes have been proposed to select pixels to test. Choices to
be made relate to the design of the sampling strategy, the number of samples
required, and the area of the samples. Recommended sampling strategies in
the context of land cover data are simple random sampling or stratified random
sampling. The number of samples may be related to two factors in accuracy as- Sampling schemes
sessment: (1) the number of samples that must be taken in order to reject a data
set as being inaccurate; or (2) the number of samples required to determine the
true accuracy, within some error bounds, for a data set. Sampling theory is used
to determine the number of samples required. The number of samples must be
traded-off against the area covered by a sample unit. A sample unit can be a
point but also an area of some size; it can be a single raster element but may
also include the surrounding raster elements. Among other considerations the
optimal sample area size depends on the heterogeneity of the class.

previous next back exit contents index glossary bibliography about

8.4. VALIDATION OF THE RESULT 307
A B C D Total Error of User Ac- Table 8.1: The error ma-
Com- curacy trix with derived errors
mission (%) and accuracy expressed
(%) as percentages. A, B, C
and D refer to the refer-
a 35 14 11 1 61 43 57
ence classes; a, b, c and d
b 4 11 3 0 18 39 61
refer to the classes in the
c 12 9 38 4 63 40 60
classification result. Over-
d 2 5 12 2 21 90 10
all accuracy is 53%.
Total 53 39 64 7 163
Error of 34 72 41 71
Omission
Producer 66 28 59 29
Accuracy

Once the sampling has been carried out, an error matrix can be established (Ta-
ble 8.1). Other terms for this table are confusion matrix or contingency matrix. In Error matrix
the table, four classes (A, B, C, D) are listed. A total of 163 samples were col-
lected. From the table you can read that, for example, 53 cases of A were found
in the real world (‘reference’) while the classification result yields 61 cases of a;
in 35 cases they agree.

The first and most commonly cited measure of mapping accuracy is the overall
accuracy, or Proportion Correctly Classified (PCC). Overall accuracy is the num-
ber of correctly classified pixels (ie, the sum of the diagonal cells in the error Overall accuracy
matrix) divided by the total number of pixels checked. In Table 8.1 the overall
accuracy is (35 + 11 + 38 + 2)/163 = 53%. The overall accuracy yields one figure

previous next back exit contents index glossary bibliography about

8.4. VALIDATION OF THE RESULT 308
for the classification result as a whole.

Most other measures derived from the error matrix are calculated per class. Er-
ror of omission refers to those sample points that are omitted in the interpretation
result. Consider class A, for which 53 samples were taken. 18 out of the 53 Omission
samples were interpreted as b, c or d. This results in an error of omission of
18/53 = 34%. Error of omission starts from the reference data and therefore re-
lates to the columns in the error matrix. The error of commission starts from the Commission
interpretation result and refers to the rows in the error matrix. The error of com-
mission refers to incorrectly classified samples. Consider class d: only two of the
21 samples (10%) are correctly labelled. Errors of commission and omission are
also referred to as type I and type II errors respectively.

Omission error is the corollary of producer accuracy, while user accuracy is the Producer accuracy
corollary of commission error. The user accuracy is the probability that a certain User accuracy
reference class has also been labelled that class. The producer accuracy is the
probability that a sampled point on the map is that particular class.

Another widely used measure of map accuracy derived from the error matrix
is the kappa or κ0 statistic. Kappa statistics take into account the fact that even
assigning labels at random results in a certain degree of accuracy. Based on Kappa
Kappa statistics one can test if two data sets have different accuracy. This type
of testing is used to evaluate different RS data or methods for the generation of
spatial data.

previous next back exit contents index glossary bibliography about