Characterizing Uncertainty and Enhancing Utility in Remotely Sensed Land Cover Using Error Matrices Localized in Canonical Correspondence Analysis Ordination Space

Abstract

In response to uncertainty in remotely sensed land cover products, there is continuing research on accuracy assessment and analysis. Given reference sample data, accuracy indicators are commonly estimated based on error matrices, from which areal extents of different cover types are also estimated. There are merits to explore the ways utilities of land cover products may be further enhanced beyond map face values and conventional area estimation. This paper presents an integrative method (CCAErrMat) for uncertainty characterization and utility enhancement. This works through reference-map cover type co-occurrence analyses based on error matrices localized in canonical correspondence analysis (CCA) ordination space rather than in geographic space to overcome the sparsity of reference sample data. The aforementioned co-occurrence analyses facilitate quantification of accuracy indicators, identification of correctly classified and perfectly misclassified pixels, and prediction of reference class probabilities, all at individual pixels. Moreover, these predicted reference class probabilities are used as auxiliary variables to formulate model-assisted area estimation, further enhancing map utilities. Extensions to CCAErrMat are also investigated as a way to bypass the pre-computing of map class occurrence pattern indices as candidate explanatory variables for CCAErrMat, leading to two variant methods: CCACCAErrMat and CNNCCAErrMat. A case study based in Wuhan municipality, central China was undertaken to compare the proposed method against alternative methods, including CCA-separate and CNN-separate. The advantages of CCAErrMat and CCACCAErrMat were confirmed. The proposed method is recommendable for characterizing uncertainty and enhancing utilities in land cover maps by analyzing locally constrained error matrices. The method is also cost-effective in terms of reference sample data, as requirements for them are similar to those for conventional accuracy assessments.

1. Introduction

Land cover information is important for geoscientific research and various spatial applications. A variety of land cover information products are produced by organizations around the world at both finer spatial resolution, such as GlobeLand30 [1] and LCMAP (Land Change Monitoring, Assessment, and Projection) [2], and coarser spatial resolution, such as Copernicus global land cover [3] and MODIS [4,5]. They provide information support for natural resource monitoring and environmental modeling at different scales.

However, land-cover maps are subject to uncertainty, which hampers spatial analyses and applications. There is growing research on accuracy assessment and analysis [6,7,8,9], which are useful not only for judging maps’ fit for use, but also for better understanding occurrences of misclassifications and thus informing about how land cover mapping may be improved. In this paper, uncertainty is a term with broad meanings and includes misclassification, ambiguity and fuzziness in labeling, and other types of errors in classification, though it can be used interchangeably with inaccuracy here.

Accuracy is conventionally assessed in a non-spatial way using error matrices constructed by cross-tabulating map classifications and reference classifications. Accuracy indicators, such as overall accuracy (OA), user’s accuracy (UA), and producer’s accuracy (PA), are often reported.

Local accuracy characterization complements non-spatial accuracy assessments mentioned above and provides pixel-level information about spatial variations of classification accuracy. For this, there have been continuing research efforts [10,11,12,13,14]. Three types of methods are often applied: spatial interpolation, empirical modeling, and localized error matrices, as reviewed by Khatami et al. [12], Stehman and Foody [14], and Zhang et al. [15], among other researchers. Spatial interpolation may be based on kriging, as in Steele et al. [16]. Empirical modeling-based methods seek to model relationships between map-reference classification agreements/disagreements and various predictors, either in the image spectral domain [9] or the map domain [13,17]. For empirical modeling, machine learning methods were employed, such as logistic regression [10,13], generalized additive modeling [18], and random forest [9]. The spatially constrained error matrices-based method was proposed by Foody [11] to map local classification accuracies (e.g., local OA, local UA, and local PA). Along this line, Comber et al. [19] computed localized and geographically weighted correspondence matrices to compute local accuracy indicators. The local error matrices-based methods are advantageous over alternative methods for their versatility in modeling multiple accuracy indicators altogether. With per-pixel accuracy analyzed, it is possible to demarcate low-accuracy locations for well-targeted re-mapping. For example, an accuracy increase was achieved by combined use of computer image re-processing over regions relatively easy to map and visual image interpretation over regions difficult to map [20].

Map-wise accuracy assessment and per-pixel accuracy modeling are seldom the ultimate goal themselves. It is also important to investigate how maps’ utilities may be enhanced beyond their face values. Here, utilities refer to the information and functionalities attainable with a map. An example is area estimation based on error matrices along with accuracy assessment [8]. Quantities that can be estimated from error matrices include map-reference class co-occurrences, map-reference class transition probabilities (with UA and PA being their respective special cases), and reference class occurrences, which actually pertain to area estimation.

The quantities mentioned above can be estimated not only for a map as whole but also around individual pixels. This study seeks to analyze map-reference class co-occurrences locally so that not only accuracy indicators but also map-reference cover type co-occurrences, transitions, and marginals (reference class probabilities) can be predicted, all at individual pixels. For map-reference cover type transitions, there are two extreme cases. One concerns correctly classified pixels, which can be used as classification training data. The other is perfectly misclassified pixels (i.e., refinable pixels discussed in [21], which can be corrected and then added to the aforementioned training data. Such pixels would be difficult to identify on globally estimated error matrices, as misclassifications are rarely patterned up, except for cases like in Campos and Brito [22]. In between, a fuller spectrum of map-reference transition probabilities can be computed for a land cover map individually as in this research or for fusion of multiple land cover maps [23], although the latter is beyond the scope of this research.

The predicted per-pixel reference class probabilities can be used to augment land cover representations. They also help identify mixed pixels, which are common for land cover products at 30 m resolution or coarser, especially over fragmented landscapes. The information about mixed pixels and low-accuracy pixels identified in accuracy assessment can be used together for well-targeted land cover re-mapping. Moreover, reference class probabilities are useful for area estimation. While stratified estimators with map classes being strata are highly recommended [8,24], model-assisted estimators are often applied for improving area estimation [25,26]. As reference class probabilities mentioned above are predicted in alignment to reference classifications via reference sample data, they are presumably closer to the true class proportions than original map classifications. It then makes sense to formulate model-assisted area estimators by using these predicted probabilities as auxiliary variables to increase estimation precision.

As discussed above, properly constructed local error matrices are versatile for uncertainty characterization and map-reference cover type co-occurrence analyses, which lead to enhanced map utilities as described above. However, reference cover conditions are often only sparsely sampled, meaning that reference sample data are often inadequate for constructing error matrices localized in geographic space, as revealed in our preliminary experiments. This research proposes constructing error matrices localized in synthetic feature space (i.e., ordination space) defined through CCA (thus named CCAErrMat) to overcome the hurdle of sample data sparsity. This leads to a unifying framework for predictions and analyses concerning local accuracy indicators, correct classifications, perfect misclassification, spectral confusions, and reference class probabilities, which can be further used as auxiliary variables to facilitate model-assisted area estimation for improved precision. The remainder of the paper describes the methods and experiments before some discussion and concluding remarks.

2. Methods

In conventional CCA modeling, response variables are specified according to the objectives of modeling: map-reference classification agreements (conditional or non-conditional) for mapping local accuracies and reference classifications for predicting reference class probabilities. Thus, the conventional CCA method is named CCA-separate to highlight its difference from CCAErrMat (for which response variables are always reference classifications). Similar to CCA-separate, CNN is also compared with CCAErrMat, thus named CNN-separate.

To bypass the pre-computing of map class occurrence pattern indices for CCA modeling, novel use was made of CCA and CNN as feature extractors for CCAErrMat in this study. These two variants of CCAErrMat were named CCACCAErrMat and CNNCCAErrMat, respectively. A flowchart for the methods proposed is shown in Figure 1.

As indicated in Figure 1, CCA-separate and CNN-separate perform the modeling for different local accuracy indicators and reference class probabilities only and separately. By CCAErrMat and its two variant methods, map-reference cover types co-occurrence analyses are performed based on CCA feature space localized error matrices, proceeding to training data extraction (by locating correctly classified and perfectly misclassified pixels) and reference class probability predictions. Reference class probabilities can be used to augment representations of land cover. They can also be used along with local accuracy surfaces to demarcate locations of uncertainty to inform land cover mapping. The predicted reference class probabilities can be further used to drive model-assisted area estimation, as shown in Figure 1. The aforementioned methods are described in more detail below while CCA and CNN are described in Appendix A and Appendix B, respectively.

2.1. CCA Feature Space Local Error Matrices: CCAErrMat

The basis for this method is CCA [27,28]. With reference classifications used as response variables and map classifications (usually along with map class occurrence pattern indices) used as explanatory variables, CCA modeling is described in Legendre and Legendre [28] and other related literature [29,30,31,32,33]. See also Appendix A for a concise introduction.

As described in Appendix A, CCA site scores determine the canonical coordinates of each pixel in the canonical space (CCA feature space) defined by the canonical axes. Consider a pixel x₀. At its location O(x₀) in CCA feature space, a subset of the sample pixels (say k of them) falling within a distance range Dis are weighted by their distance to O(x₀) to construct the error matrix. The resulting values are feature space distance-weighted counts of CCA sample pixels in each reference class (column j) assigned to each map class (row i). In other words, cell counts $N{P}_{O\left(x0\right)}\left(i,j\right)$ are estimated. Cell counts are the counts of pixels belonging to map-reference cover type combinations in a localized error matrix. These counts are sums of numbers of pixels weighted by distances between unknown locations and sample locations in CCA ordination space. They are non-integers when distance-weighting is applied:

$${\widehat{NP}}_{O\left(x0\right)}\left(i,j\right)={{\displaystyle \sum }}_{k^{\prime }=1}^k{W}_{k^{\prime }}\ast I\left(i=c^{\prime }\left(x_{k^{\prime }}\right),\hspace{0.17em}j=c\left(x_{k^{\prime }}\right)\right),\hspace{0.17em}\mathrm{when}\hspace{0.17em}{d}_{k^{\prime }{x}_0}^{}\hspace{0.17em}<\hspace{0.17em}\mathrm{Dis}$$

(1)

where I() is an indicator; $c^{\prime }\left(x_{k^{\prime }}\right)$ and $c\left(x_{k^{\prime }}\right)$ refer to map class and reference class for sample pixel x_k′, respectively; $W_{k^{\prime }}$, being the weight assigned to sample pixel x_k′, is based on the distance $d_{k^{\prime }x0}^{}\left(\mathrm{between}\hspace{0.17em}\mathrm{sample}\hspace{0.17em}\mathrm{pixel}\hspace{0.17em}{x}_{k^{\prime }}\hspace{0.17em}\mathrm{and}\hspace{0.17em}x0\right)$, which is itself eigenvalue-weighted; and Dis refers to the search radius containing k sample pixels. The estimated cell counts $N{P}_{O\left(x0\right)}\left(i,j\right)$ are standardized by their grand total say TP to get standardized cell probabilities ${\widehat{p}}_{O\left(x0\right)}\left(i,j\right)$ ($\mathrm{which}\hspace{0.17em}\mathrm{equals}\hspace{0.17em}N{P}_{O\left(x0\right)}\left(i,j\right)$/TP).

Once the error matrices centered at individual pixels are populated with cell proportions properly estimated, it is straightforward to calculate local accuracy indicators for pixel x₀. For example, we can compute local OA as:

$$P_{OA,\hspace{0.17em}o\left(x0\right)}={{\displaystyle \sum }}_{i=1}^{CLS}{\widehat{p}}_{O\left(x0\right)}\left(i,i\right)$$

(2)

where CLS represents the number of candidate classes considered. Local UA for map class c′ and local PA for reference class c are predicted, respectively, as:

$$P_{UA,\hspace{0.17em}o\left(x0\right)|c^{\prime }}={\widehat{p}}_{O\left(x0\right)}\left(c^{\prime },c^{\prime }\right)\hspace{0.17em}/{{\displaystyle \sum }}_{j=1}^{CLS}{\widehat{p}}_{O\left(x0\right)}\left(c^{\prime },j\right)$$

(3)

and

$$P_{PA,\hspace{0.17em}o\left(x0\right)|c}={\widehat{p}}_{O\left(x0\right)}\left(c,c\right)\hspace{0.17em}/{{\displaystyle \sum }}_{i=1}^{CLS}{\widehat{p}}_{O\left(x0\right)}\left(i,c\right)$$

(4)

We can predict probabilities of all candidate reference classes’ occurrences from the aforementioned feature space localized error matrices. This amounts to computing the sum of cell proportions along a particular column (corresponding to a reference class say j):

$${\widehat{p}}_{o\left(x0\right)}\left(+,j\right)\hspace{0.17em}={{\displaystyle \sum }}_{i=1}^{CLS}{\widehat{p}}_{O\left(x0\right)}\left(i,j\right)$$

(5)

The resultant class probabilities are useful for identifying mixed pixels, which along with local accuracy surfaces mapped above, can inform land cover re-mapping. They can also be used for augmented representations of land cover by showing primary and alternate class labels for mixed pixels; among them, pixels showing virtually equal class memberships to more than two classes can be further identified. Furthermore, based on localized error matrices, it is straightforward to examine pixels if they are correctly classified or perfectly misclassified (can be corrected easily). From such pixels and the pixel segments (3 by 3) they fall in, training data are extracted and used for improving classification.

Note that for different modeling objectives (e.g., different local accuracy indicators, reference class probabilities), the k’s (optimum number of nearest neighboring sample pixels) for constructing local error matrices need to be determined individually. Optimization was carried out through cross-validation in this study.

2.2. CCA and CNN Used as Feature Extractors for CCAErrMat: CCACCAErrMat and CNNCCAErrMat

For modeling by CCAErrMat, CCA-separate, and CNN-separate, candidate explanatory variables usually include map classes, geospatial coordinates, and map class occurrence pattern indices, such as class proportions, homogeneity, heterogeneity, dominance, entropy, and contagion [34,35,36]. For this study, map class occurrence pattern indices were computed over moving window sizes of 3 by 3, 5 by 5, 7 by 7, and 9 by 9 pixels. This maximum window size of 9 pixels was set because smaller window sizes are more informative for pixel-level modeling and help to reduce the computational burden. A total of 63 explanatory variables were considered in model selection: map class (6), class proportions in different-sized windows (28), pattern indices computed in different-sized windows (20), class occurrence frequencies of neighbor polygons (7), and sample pixel’s geospatial coordinates (2) [21].

The computing of class occurrence pattern indices is, however, computationally intensive. As CNN is becoming easy to use and is often considered as a feature extractor, there seem to be merits in applying CNN using map class indicators alone as explanatory variables to predict reference class probabilities, which are then used as explanatory variables along with map classifications for CCAErrMat. This leads to a method named CNNCCAErrMat. Likewise, CCA is also used as a feature extractor for CCAErrMat, leading to Method CCACCAErrMat.

2.3. Improved Model-Assisted Area Estimation

Area estimators include design-based estimators, model-based estimators, and model-assisted estimators [25,37,38]. The most common design-based estimator is the π estimator (also known as the Horvitz–Thompson estimator), which uses probability-based samples to estimate areas. It is unbiased and has relatively high precision (i.e., low variance). When stratified sample data with map classes being the strata were organized in an error matrix, areas for individual cover types were easily estimated [8]. See Appendix C for further detail. When full-coverage auxiliary variables were available or could be assembled, model-assisted estimators were used [37,39], such as the difference estimator and the regression estimator, which are also described in Appendix C.

A model-assisted area estimator, which uses predicted reference class probabilities (Equation (5)) as auxiliary variables, was proposed in this study. Since the predicted class probabilities presumably approximated true class proportions more closely than original map classifications, they were expected to lead to increased precision in area estimation. This was pursued through the difference estimator and regression estimator described in Appendix C.

3. Experiments

3.1. The Study Site and Datasets

GlobeLand30 2010 land-cover dataset (http://www.globallandcover.com (accessed on 26 February 2023)) for Wuhan, China was used for this study. The city of Wuhan is about 8495 km² in areal extent, located in the middle reach of the Yangtze, as shown in Figure 2 (where the inset map of China is shown lower right corner). For Wuhan, of seven classes in total, the dominant class is cultivated land, followed by water, forest, and artificial surface; they account for about 60%, 15%, 12%, and 7% of the total area, respectively, according to map classifications. Grassland, wetland, and bare land together take about 6% of Wuhan’s areal extent. Further detail can be found in Wan et al. [21].

Model training and testing were based on the sample data acquired by Zhang et al. [40], which were collected following a class-heterogeneity-stratified random sampling design (StRS). This design uses a local heterogeneity index to stratify pixels within a class (stratum) into relatively homogeneous and heterogeneous sub-strata [40,41], resulting in a total of 14 sub-strata for a total of 7 strata (cover types) in the study area. The full training sample has a total of 3000 pixels, and the full test sample has a total of 1020 pixels, as shown in

Table 1. Model-training sample full set and subsets I and II, the test sample (for model testing and area estimation), and sample subset III (for area estimation).

Strata and Sub-Strata	Nstrata	Training Sample Full Set (Sampling Intensity)	Sample Subset I	Sample Subset II	Test Sample Full Set (Sampling Intensity)	Sample Subset III
Cultivt_E	56,721	120 (0.21)	15	60	60 (0.11)	15
Cultivt_O	5,739,203	1095 (0.02)	132	160	160 (0.003)	132
Forest_E	133,655	140 (0.10)	18	70	70 (0.05)	18
Forest_O	1,001,767	280 (0.03)	27	100	100 (0.01)	27
Grass_E	70,366	120 (0.17)	15	60	60 (0.09)	15
Grass_O	248,912	170 (0.07)	21	80	80 (0.03)	21
Wetland_E	2033	80 (3.93)	9	40	40 (1.97)	9
Wetland_O	133,735	140 (0.10)	18	70	70 (0.05)	18
Water_E	23,895	100 (0.42)	12	50	50 (0.21)	12
Water_O	1,395,043	285 (0.02)	33	110	110 (0.01)	33
Artfct_E	19,324	100 (0.52)	12	50	50 (0.26)	12
Artfct_O	699,787	200 (0.03)	24	90	90 (0.01)	24
Bare_E	3658	80 (2.19)	12	40	40 (1.10)	12
Bare_O	6994	90 (1.29)	12	40	40 (0.58)	12
All	9,535,093	3000	360	1020	1020	360

, where number of sample pixels belonging to individual sub-strata are indicated, with sampling intensities shown in parentheses for the full training sample and the full test sample. Sampling intensities are percentages of the total number of pixels (N_strata) belonging to specific sub-strata (E-Heterogeneous, O-Homogeneous). It is shown that sampling intensities for E sub-strata were much greater than their O counterparts, being a feature of the adopted StRS design by Zhang et al. [40]. Sample size allocations for the training and test samples were configured following the Neyman method [40]. In Table 1, Cultivt and Artfct are abbreviations for cultivated land and artificial surfaces, respectively, while grass, water, and bare for grassland, water bodies, and bare land, respectively. These abbreviations are the same through the remainder of this paper, including the appendices.

In this study, for evaluating the influence of sample size on model performances, sample subsets I and II (of 360 pixels and 1020 pixels, respectively) were selected from the full training sample for experiments, as shown in Table 1. For model testing, the full test sample was adopted. For model-assisted area estimation, sample subset III (of 360 pixels) was selected from the full test sample and was used along with the latter, as shown in Table 1. Note that the aforementioned sample subsets I and II (for model training) were different from the samples used for area estimation (i.e., sample subset III and full test sample), although with the same sizes (strata-allocations) of 360 and 1020 pixels, respectively, as shown in Table 1.

3.2. Results

Modeling and estimation were carried out using methods described in Section 2. Some of the results, especially the intermediate ones, are shown in Appendix D, as indicated in the text below wherever relevant.

3.2.1. Mapping Local Accuracy

Conventional error matrix-based accuracy assessment was carried out, as shown in

Table A1. Error matrix and accuracy assessment for the original map based on the full model-training sample set (cell proportions and accuracy measures in %).

Map Class	Reference Class
	Cultivt	Forest	Grass	Wetland	Water	Artfct	Bare	Strata Accuracy	UA
Cultivt (E)	0.2	4.5$\times 10$−2	6.4$\times 10$−2	3.5$\times 10$−2	0.1	0.1	2.5$\times 10$−2	36.7	71.8
Cultivt (O)	43.4	2.5	1.8	2.3	5.8	3.6	0.8	72.1
Forest (E)	0.3	0.4	0.2	0.1	0.4	0.1	3.0$\times 10$−2	26.4	73.1
Forest (O)	0.8	8.3	0.8	0.2	0.3	0.1	3.8$\times 10$−2	79.3
Grass (E)	0.1	0.1	0.3	3.7$\times 10$−2	0.1	3.1$\times 10$−2	6.1$\times 10$−3	38.3	48.3
Grass (O)	0.3	0.2	1.3	4.6$\times 10$−2	0.4	0.2	0.1	51.2
Wetland (E)	2.7$\times 10$−4	0	5.3$\times 10$−4	1.1$\times 10$−2	9.1$\times 10$−3	0	0	53.8	69.1
Wetland (O)	0.1	1.0$\times 10$−2	2.0$\times 10$−2	1.0	0.3	0	0	69.3
Water (E)	2.3$\times 10$−2	7.5$\times 10$−3	1.3$\times 10$−2	3.0$\times 10$−2	0.2	0	0	71.0	85.4
Water (O)	0.4	0.1	0.6	0.6	12.5	0.1	0.4	85.6
Artfct (E)	2.0$\times 10$−2	3.9$\times 10$−2	1.6$\times 10$−2	0	1.6$\times 10$−2	0.1	6.1$\times 10$−3	52.0	83.1
Artfct (O)	0.2	0.1	0.4	0	0.1	6.2	0.4	84.0
Bare (E)	7.2$\times 10$−3	9.6$\times 10$−3	8.2$\times 10$−3	0	4.8$\times 10$−4	0	1.3$\times 10$−2	33.8	45.1
Bare (O)	9.0$\times 10$−3	1.2$\times 10$−2	1.3$\times 10$−2	0	1.6$\times 10$−3	0	3.7$\times 10$−2	51.1
PA	95.1	73.0	29.5	22.9	62.8	59.8	2.9	73.9

in Appendix D. While it is possible to identify classes of relatively lower UAs and/or PAs from inspecting the error matrix, pixel-level information about spatial variations in accuracy would be possible only through local accuracy mapping.

Local accuracies were mapped using the methods described in Section 2. In Appendix D,

Table A2. Selected explanatory variables and optimum k for KNN (Method CCA-separate).

Re-mapping	Sample Sets	Selected Variables	k
	360 pixels	mapclass1+mapclass2+mapclass3+mapclass4+mapclass5+	20
	1020 pixels	mapclass6+hom3+con3+het3+dom3+ent3+p1w3+p2w3+	47
Local accuracy mapping (360 sample pixels)		Selected Variables	k
	OA	ent3+mapclass6+mapclass5+p6w9+mapclass4+p5w7+p2w5	43
	UA Cultivt	p6w7	3
	UA_Forest	con3+patch5+hom3	8
	UA_Grass	con5+patch2	10
	UA_Wetland	patch3+patch4+dom3+p5w3+con3+het3+patch1+p2w7	7
	UA_Water	p1w7+het5+het7+area+patch3+con9	6
	UA_Artfct	p8w3	20
	UA_Bare	con5+het5+patch4+dom3+het7+p9w3+ent5+patch2+area+ patch5+patch1+dom7+con3+p6w9+ent3+p2w3+patch3+ hom9+het9+p2w5+p2w7+p1w9+hom3	7
	PA Cultivt	mapclass1+p1w3+p1w5+p1w7+p1w9	1
	PA_Forest	mapclass2+p1w3+p1w5+p1w7+p1w9	1
	PA_Grass	mapclass3+p1w3+p1w5+p1w7+p1w9	1
	PA_Wetland	mapclass4+p1w3+p1w5+p1w7+p1w9	1
	PA_Water	mapclass5+p1w3+p1w5+p1w7+p1w9	1
	PA_Artfct	mapclass6+p1w3+p1w5+p1w7+p1w9	1
	PA_Bare	mapclass7+p1w3+p1w5+p1w7+p1w9	1
Local accuracy mapping (1020 sample pixels)		Selected Variables	k
	OA	ent3+mapclass5+mapclass6+lsihet2_3+mapclass4+ mapclass1+p6w9+p5w7+POINT_Y+hom9+p5w3+p5w9	37
	UA Cultivt	ent3+het3+p5w9+p6w5	21
	UA_Forest	hom9+patch5+p3w9+p8w3+con3	40
	UA_Grass	p1w5+dom5+dom9+patch2+area	44
	UA_Wetland	p6w3+patch4+patch3+patch1+p3w9+p1w3	36
	UA_Water	p1w3+dom3+p3w5+patch6+p2w3	14
	UA_Artfct	patch4+dom3+p8w5+hom3+p8w3	19
	UA_Bare	patch3+p1w3+hom5+area	14
	PA Cultivt	mapclass1+p1w3+p1w5+p1w7+p1w9	1
	PA_Forest	mapclass2+p1w3+p1w5+p1w7+p1w9	1
	PA_Grass	mapclass3+p1w3+p1w5+p1w7+p1w9	1
	PA_Wetland	mapclass4+p1w3+p1w5+p1w7+p1w9	1
	PA_Water	mapclass5+p1w3+p1w5+p1w7+p1w9	1
	PA_Artfct	mapclass6+p1w3+p1w5+p1w7+p1w9	1
	PA_Bare	mapclass7+p1w3+p1w5+p1w7+p1w9	1

shows the optimal explanatory variables selected and optimum k’s for KNN (K nearest neighbors) in CCA-separate while shows optimum k’s for CCAErrMat. Some of the local accuracy surfaces obtained with method CCAErrMat based on the training sample subset of 360 pixels are shown in Figure 3, where surfaces of local OAs, local UAs for cultivated land, and local PAs for water bodies are shown in Figure 3a–c, respectively.

Water pixels appear to have higher local OAs while pixels of artificial surfaces and cultivated land tend to have lower local OA, as shown in Figure 3a. For the class of cultivated land, there exist variations in local UAs, as shown in Figure 3b. For the class of water bodies, higher local PAs tend to cluster around water bodies of larger areal extents.

Accuracies of local accuracy indicators predicted using different methods were evaluated based on the full test sample. This was performed based on area under the curve (AUC) (i.e., the receiver operating characteristic curve, ROC) [42] commonly applied for continuously valued quantities such as local accuracies. Results are shown in

Table 2. AUC-based evaluation for local accuracy characterization.

		CCAErrMat		CCA-Separate		CNN-Separate		CCACCAErrMat		CNNCCAErrMat
		360	1020	360	1020	360	1020	360	1020	360	1020
Local OA		0.71	0.72	0.75	0.79	0.74	0.77	0.74	0.77	0.72	0.76
Local UA	Cultivt	0.72	0.71	0.67	0.75	0.78	0.75	0.70	0.74	0.62	0.76
	Forest	0.79	0.83	0.78	0.87	0.82	0.86	0.77	0.82	0.82	0.87
	Grass	0.61	0.50	0.57	0.54	0.66	0.61	0.67	0.65	0.61	0.56
	Wetland	0.61	0.68	0.61	0.71	0.40	0.70	0.54	0.67	0.36	0.70
	Water	0.45	0.62	0.66	0.81	0.55	0.54	0.56	0.56	0.54	0.54
	Artfct	0.70	0.56	0.69	0.66	0.73	0.71	0.71	0.70	0.69	0.70
	Bare	0.63	0.65	0.58	0.80	0.72	0.87	0.62	0.70	0.73	0.79
Local PA	Cultivt	1.00	1.00	1.00	1.00	0.99	1.00	1.00	1.00	1.00	1.00
	Forest	1.00	1.00	1.00	1.00	0.99	1.00	1.00	1.00	1.00	1.00
	Grass	1.00	1.00	1.00	1.00	0.98	1.00	1.00	1.00	1.00	1.00
	Wetland	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	Water	0.99	0.99	1.00	1.00	0.99	1.00	1.00	1.00	1.00	1.00
	Artfct	1.00	1.00	1.00	1.00	0.98	1.00	1.00	1.00	1.00	1.00
	Bare	0.98	1.00	1.00	1.00	0.96	1.00	1.00	1.00	1.00	1.00

.

As shown in Table 2, accuracies increase generally as the training sample size increases. For local OAs, all methods tested achieve AUCs higher than 0.70. CCA-separate with 1020 training sample pixels attained the highest accuracy (0.79). For local UAs, relatively accurate predictions are generated by most methods for cultivated land, forest, and artificial surfaces. For local PAs, prediction accuracies are quite high. This is similar to that reported in Wickham et al. [13].

Then, we compare CCAErrMat and CCACCAErrMat. Although they predict local accuracies with comparable accuracies, CCACCAErrMat predicts local OAs and local UAs (for grassland, artificial surfaces, and bare land) with greater accuracy. Thus, it is highly recommended for use given its reduced cost in computing class occurrence pattern indices.

3.2.2. Analyzing Map-Reference Class Co-Occurrences

Alternative methods were applied with training sample subsets I and II detailed in Table 1. Unless stated otherwise, the results obtained using CCAErrMat with the sample of 360 pixels were used as examples below.

Reference class probabilities predicted were used to identify mixed pixels. They were identified by setting thresholds on the maximum class probabilities. The threshold applied was 0.60. Mixed pixels identified are shown in Figure 4a, indicating many pixels were viewed as mixed as the threshold value applied was relatively high. Based on local OAs, locations of relatively low accuracy (thresholding at 0.20) can be marked, resulting in the map shown in Figure 4b, where fewer locations than those in Figure 4a were considered as of low-accuracy, as the threshold value applied was relatively low. Combining the maps shown in Figure 4a,b, locations of low-accuracy and mixed cover were identified. This leads to the map shown in Figure 4c, where the locations marked indicate the locations where re-mapping should be targeted.

Predicted reference class probabilities were further used to augment land cover representations. The most probable cover types (i.e., primary classes) were determined based on the predicted class probabilities, as shown in Figure 5a. For mixed pixels identified as in Figure 4a, the second most probable class labels (i.e., alternate classes) were determined, resulting in Figure 5b. Furthermore, the mixed pixels exhibiting almost equal probabilities to more than two classes were further identified (called highly mixed pixels), as shown in Figure 5c, where the third most probable classes are depicted.

In addition, based on local error matrices, locations that appear to be correctly classified or can be corrected based on local refinability can be identified. A total of 359,780 pixels were identified as being correctly classified while only 768 pixels were classified as refinable. All 3 by 3 pixel segments (excluding edge pixels) in which correctly classified pixels fall were extracted (Figure 6a). For pixels found refinable, homogeneous 3 by 3 pixel segments (again excluding edge pixels) they fall in were corrected accordingly (Figure 6b). The union of correctly classified pixel segments (Figure 6a) and corrected pixel segments (Figure 6b) are shown in Figure 5c, leading to the set of pixels potentially usable as classification training data.

3.2.3. Improved Area Estimation

Based on reference class probabilities predicted using CCAErrMat (trained with the training sample subset I of 360 pixels described in Section 3.1), model-assisted area estimation was carried out and compared with that obtained with the π estimator. For this, two samples (the full test sample set of 1020 pixels and its subset of 360 pixels (sample subset III), as shown in Table 1) were used. The estimated area proportions and the corresponding standard errors (SEs) relative to area proportions estimated are shown in

Table 3. Area proportions estimated with different estimators (%).

Area		Cultvt Land	Forest	Grass	Wetland	Water	Artificial Surfaces	Bare Land
360 pixels	π estimator	50.1	12.9	6.8	3.4	17.1	8.9	0.8
	Difference estimator	50.4	12.6	6.9	3.5	16.9	8.8	0.8
	Regression estimator	50.4	12.6	6.9	3.6	16.9	8.8	0.8
1020 pixels	π estimator	49.1	12.6	6.7	2.6	19.5	8.9	0.6
	Difference estimator	49.9	12.6	6.5	2.5	18.9	9.1	0.6
	Regression estimator	50.0	12.6	6.5	2.6	18.7	9.1	0.6

and

Table 4. Standard error (SE) estimates (relative to area proportions estimated) obtained with different estimators (%).

SE		Cultvt Land	Forest	Grass	Wetland	Water	Artificial Surfaces	Bare Land
360 pixels	π estimator	5.0	14.1	24.0	38.3	12.4	18.3	102.4
	Difference estimator	3.9	11.6	21.7	28.1	8.9	12.8	91.5
	Regression estimator	3.8	11.6	21.7	28.2	8.7	12.8	96.5
1020 pixels	π estimator	2.6	8.9	16.1	32.2	7.0	12.0	81.9
	Difference estimator	2.1	6.8	15.0	26.5	5.1	7.4	78.7
	Regression estimator	2.0	6.8	15.0	26.0	5.1	7.5	80.6

, respectively.

As shown in Table 3, area proportions of seven classes estimated from multiple methods are quite similar. However, there are noticeable differences in SEs concerning the area estimates, as shown in Table 4 (where smaller SEs mean greater precision). Overall, SEs for area estimates obtained with a larger sample size (1020 pixels) are smaller than those with smaller sample size (360 pixels), indicating improved precision with a greater sample size. With respect to the differences of SEs among the three estimators tested, the π estimator leads to the greatest SEs, the regression estimator the smallest SEs (especially for the classes of cultivated land and water bodies), with the difference estimator in between, on the whole. Exceptions are observed with the class of bare land, due to it being the smallest areal extent and prone to misclassification.

4. Discussions

4.1. Comparisons with Related Work

This study promotes method CCAErrMat for uncertainty characterization and utility enhancement. The method’s major advantages lie in its multiple functionalities.

For local accuracy mapping, CCAErrMat allows predictions of multiple local accuracy indicators without having to re-run modeling separately, although optimum k’s for different accuracy indicators need to be determined separately. The proposed method complements those devised for image-domain per-pixel accuracy analyses, such as Ebrahimy et al. [9].

Concerning performances of local accuracy modeling, some remarks are in order. Although not reported here, a preliminary experiment was undertaken. Results indicated that geospatially constrained local error matrices are inferior to CCAErrMat due to the sparsity of sample data used in this study (even with the full model-training sample set of 3000 pixels). In addition, the initial results showed that CCAErrMat was comparable to random forest in prediction performances. Nevertheless, it is important to undertake more comprehensive comparative studies on competing methods, although this is out of the scope of this study.

Predicted reference class probabilities can be used to augment land cover representations by providing information about potential land cover distributions through multiple alternative class labels. They can also be used to locate mixed pixels, providing useful information about locations prone to misclassification. The demarcation of low-accuracy locations and mixed pixels is useful for well-targeted land cover re-mapping. This is complementary to the work done by Yu et al. [43], in which difficult-to-map classes are identified and analyzed based on multiple classifications of land cover, and by Huang et al. [20], where areas of uncertainty were identified using multiple map consistency analysis.

Corrections for misclassification errors can be done to increase classification accuracies when misclassification errors exhibit certain patterns [22]. However, map-wise misclassification patterns are rarely apparent or discernible. By the proposed method, patterns of map-reference cover type associations can be analyzed based on localized error matrices. Pixels that are correctly classified or perfectly misclassified can be identified to create classification training datasets, although our preliminary experiments using augmented training data indicated only a modest accuracy increase. Nevertheless, this method provides a map-domain strategy for classification improvement, complementing to image-domain strategies as in Huang et al. [20], in which image interpretation is adopted over uncertain regions while training sample data for image classification over certain regions are extracted from consistency analyses of multiple products. Furthermore, confusion among cover types can be analyzed locally to aid in searching for potential informative features for land cover mapping, although this was not investigated further in this study.

With the proposed method, the enhanced map utilities were also demonstrated by an improved area estimation through the use of model-predicted reference class probabilities. Specifically, cover type probabilities were used as auxiliary variables in models for predicting per-pixel areal extents of candidate classes. Areas of multiple cover types were estimated by correcting model predictions using reference sample data in a model-assisted estimation framework. This strategy complements existing model-assisted area estimators, which are often based on remote-sensing images and/or biophysical variables and are mostly applied to a single theme (e.g., forest) [38]. According to Sales et al. [44], consistent area estimates can be obtained using class membership probabilities estimates from a random forest classification, with the error of the predicted class membership probabilities converging to zero given a large sample and proper set of explanatory variables. This model-based strategy can be usefully extended with the model-assisted strategy promoted in this study.

4.2. Recommendation for Further Research

CCAErrMat may be improved by integrating it with geographically localized error matrices for improved performances when denser reference data can be furnished. CCA can also be improved by considering stratified modeling approaches according to distinct patterns of map-reference cover type associations. This is relevant for large-area applications where regionalization is to be discerned concerning map-reference cover type associations. As a further note, feature space does not need to be synthesized through CCA only. Other kinds of synthetic feature spaces are certainly worth studying.

Land cover information fusion is a long-standing research theme. There is continuing work on the fusion of multiple land cover products by making use of a variety of methods, as exemplified by the relevant literature [45,46,47,48,49,50,51]. Other potentially useful methods are worth exploring. For instance, in addition to original maps, class probabilities predicted can be used as contextual information or as land cover primitives [52] for the fusion of multiple land cover products [53], especially when in combination with accuracy characterization. This may proceed by having individual products accuracy-characterized and thematically aligned using reference sample data before having them fused through accuracy-based weighting [47]. Land cover information fusion likely also benefits from exploiting a fuller spectrum of map-reference class co-occurrence statistics, which can be computed using local error matrices, such as map-reference cover type transition probabilities, as investigated by Zhang et al. [23].

Change detection and analysis are becoming increasingly important nowadays [53,54,55]. It is certainly worth exploring how CCAErrMat (and its variant CCACCAErrMat) is extended from single-date applications to change detection and multi-temporal applications [56]. Both direct and indirect methods are worth exploring. The former treat changes as “from-to” classes (using methods originally designed for single-date land cover to change detection directly) while the latter handle single-date land cover separately followed by proper synthesis of single-date results. For the latter, work by Zhang et al. [15] on geostatistical modeling of spatial-temporal correlation (for local accuracy characterization in land cover change) may be usefully extended.

In this study, we considered only discrete classes of land cover. Percent cover information products are also important [13,57]. Numerical ecology and machine learning methods can in principle be used for modeling percent covers of candidate land cover classes by noting the analog between percentages (or fractions) and abundances. Although there seems to be no consensus on how error matrices should be constructed for percent covers, some work has been done on error matrices for soft classifications [58]; class probabilities can be directly used as explanatory variables for CCA, indicating CCAErrMat’s potential extensibility. Nevertheless, further investigations are needed.

5. Conclusions

The novelty of this study lies in having proposed a method based on CCA ordination space localized error matrices for accuracy characterization (with multiple local accuracy indicators), analyses of map-reference cover type transitions, prediction of reference class probabilities, and improved area estimation. For bypassing the pre-computing of map class occurrence pattern indices in modeling, CCACCAErrMat and CNNCAAErrMat are also proposed as useful extensions to CCAErrMat.

Results obtained from experiments in this study are summarized below. First, for local accuracy characterization, local OAs, local UAs, and local PAs were all predicted with reasonable accuracy using alternative methods. However, in terms of cost-effectiveness, CCErrMat and CCACCAErrMat are preferred. Second, reference class probabilities predicted using CCAErrMat were used to augment land cover representations with primary and alternate labels. They were also used to identify mixed pixels, which, along with low-accuracy pixels, can be used to inform re-mapping. Third, local error matrices were analyzed to extract training data by locating correctly classified and perfectly misclassified pixels. Lastly, standard errors in area estimation were greatly reduced by model-assisted area estimation using reference class probabilities predicted by CCAErrMat as auxiliary variables. The regression estimator gave rise to the greatest precision on the whole. Although CCAErrMat was tested using GlobeLand30 land cover data over Wuhan municipality, it is transferable to other regions and other land cover products with the understanding that the specific results and performances are likely different over different regions and with different products.

Through map-reference co-occurrence analyses via local error matrices, the proposed method, CCAErrMat, provides fuller information about uncertainty in map classification, map-reference cover type transitions (e.g., differentiation among locations correctly classified, perfectly misclassified, or prone to more complex confusion), and reference class probabilities, which lead to further improved area estimation by building model-assisted area estimators. The enhanced map utilities pertain not only to the maps individually, but also collectively when they are used in combination (e.g., for change detection and data fusion). These come at no extra cost with respect to the reference data used, which are usually available or can be easily furnished, as in conventional accuracy assessments.