Spatially Balanced Sampling for Validation of GlobeLand30 Using Landscape Pattern-Based Inclusion Probability

Abstract

Global and local land-cover mapping products provide important data on land surface. However, the accuracy of land-cover products is the key issue for their further scientific application. There has been neglect of the relationship between inclusion probability and spatial heterogeneity in traditional spatially balanced sampling. The aim of this paper was to propose an improved spatially balanced sampling method using landscape pattern-based inclusion probability. Compared with other global land-cover datasets, Globeland30 has the advantages of high resolution and high classification accuracy. A two-stage stratified spatially balanced sampling scheme was designed and applied to the regional validation of GlobeLand30 in China. In this paper, the whole area was divided into three parts: the Tibetan Plateau region, the Northwest China region, and the East China region. The results show that 7242 sample points were selected, and the overall accuracy of GlobeLand30-2010 in China was found to be 80.46%, which is close to the third-party assessment accuracy of GlobeLand30. This method improves the representativeness of samples, reduces the classification error of remote sensing, and provides better guidance for biodiversity and sustainable development of environment.

1. Introduction

Land cover is the natural basis for the survival and development of human beings, and it affects the energy and material circulation of the Earth. It is also intrinsically related to global climate change, biological habitat protection, sustainable practices suitable for increasing soil protection and reducing pollution, and so on [1,2,3,4,5,6,7]. Global-scale land-cover mapping and monitoring is a complex remote sensing project. At present, the United States and the European Union have successively developed five sets of more commonly used global land-cover products, namely: (1) IGBP-DISCover (International Geosphere–Biosphere Program’s Data and Information System Cover) produced by the U.S. Geological Survey [8,9]; (2) UMD (University of Maryland) produced by the University of Maryland [10]; (3) MODIS (Moderate resolution Imaging Spectro-radiometer) produced by Boston University [11,12]; (4) GLC2000 (Global Land Cover 2000) produced by the European Commission’s Joint Research Centre [13]; (5) GlobCover (Global Land Cover Product) produced by the European Space Agency [14]. In order to support the development of global change research, China completed the world’s first 30 m global land-cover product GlobeLand30 in 2014. Someone generated nine LC maps of China from seven global LC datasets (IGBP DISCover, UMD, GLC, MCD12Q1, GLCNMO, CCI-LC, and GlobeLand30), while Globeland30 had the highest accuracy [15]. In addition, there were some other high-precision land-cover products made in China for classification accuracy evaluation [16,17,18].

The key issue for the further scientific application of land-cover products is accuracy. Without validation, any land-cover product, either at a global scale or at a regional scale, is only an unconfirmed hypothesis [19]. It is therefore necessary to ensure product quality by assessing the similarities between the classification results and the real land surface through quantitative means. At present, basic accuracy assessment is carried out by means of a confusion matrix, which is derived from the spatial comparison between classified data and reference data. A confusion matrix can be constructed by pixel-to-pixel comparison or by a sampling survey. Reference data of a whole region are not usually available, so that a sampling strategy is needed for selecting samples from the study area. The sample size (number of samples) can be determined according to the available resources/time/cost or by a rigorous statistical model [20,21]. Stratified sampling and cluster sampling are the most widely used sampling methods for land-cover product validation [22,23,24,25,26]. However, how to consider the land-cover spatial autocorrelation in the sample distribution and increase the flexibility of the sample size is still one of the main challenges for land-cover product validation [27,28,29]. Most populations are in fact distributed over space but many sampling designs, such as simple random sampling (SRS), do not incorporate the spatial aspect into the design [30]. The spatial distribution of a population represents an important tool in sampling designs that use the geographical coordinates of the units in the frame as auxiliary information [31]. In particular, our interest is focused on probability samples that are well spread over the population in every dimension, which in the recent literature is defined as spatially balanced samples [32]. Spatially balanced sampling can produce evenly distributed samples, and it also allows the user to use a priori knowledge to create an efficient sampling scheme [28,33]. This method has been applied to natural and environmental resource surveying [34,35,36].

The most common simple random sampling (SRS), although simple in theory, has low levels of error and of computational complexity of the target estimation, and it is difficult to operate in practice. For example, it needs a sampling frame containing all the basic units in the population. In addition, the variability among SRS samples is very high, and the spatial distribution of samples is uneven, resulting in poor representativeness of samples and unsatisfactory estimation accuracy; although cluster sampling can save costs in a large-scale investigation, it requires small differences among primary units (PSU) and large differences among secondary units (SSU), which is very difficult to achieve. The variability between the actually collected samples is high, and many redundant samples with high similarity will be obtained. Compared with the methods above, systematic sampling can obtain samples with uniform spatial distribution, but it cannot obtain unbiased estimation of estimators, and the adaptability of systematic sampling is very poor. If one is to increase or decrease samples, one must increase or decrease the grid density, which is inflexible. The principle of stratified sampling is to keep the intra-layer variance as small as possible and the inter-layer variance as large as possible, which makes its estimation accuracy higher and the variability between samples less. However, there is also the problem of uneven spatial distribution of samples; spatial balanced sampling is a random, equiprobable, efficient, low-cost, well-balanced, and adaptable sampling method. However, spatial balanced sampling still gives the same inclusion probability to the patches with different spatial heterogeneity. This cannot improve the phenomenon that the land-covers with broken spatial structures are more wrong than the classification results with regular spatial structures. In this paper, the idea of unequal inclusion probability is introduced on the basis of spatial balanced sampling through the landscape shape index. The inclusion probability of spatial units in patches is directly proportional to the degree of spatial heterogeneity. The higher the spatial heterogeneity of patches (the larger the landscape shape index), the higher the inclusion probability of spatial units in patches. The reverse is also true. This makes up for the defect of spatially balanced sampling and makes the sampling design more reasonable.

The classification error of remote sensing images follows certain rules: for those objects with a complete structure and large area, the classification error is small; for structural fragmentation and fragmentation, there is a greater classification error. In view of this phenomenon, the sampling method proposed in this paper can reasonably consider the relationship between the inclusion probability and the degree of spatial heterogeneity. By improving the inclusion probability of patches with strong heterogeneity and severe fragmentation, and increasing the number of their sample points, a more reasonable accuracy evaluation can be obtained. Considering the spatial characteristics of the land-cover data in a 30 m resolution, we propose a two-stage stratified spatially balanced sampling scheme. In order to ensure the representativeness of the sample points, the landscape shape index (LSI) was introduced into the unequal inclusion probability design in spatially balanced sampling. Then, in the regional validation of GlobeLand30 in China, we considered the validation in three parts, namely, sampling design, response design, and accuracy assessment. In addition, we also studied the factors affecting the product accuracy, including reference data, sampling methods, classification methods and phenological conditions. Finally, we provide suggestions and guidance for the improvement of land-cover products.

2. Methods

2.1. Spatially Balanced Sampling

At present, the most popular algorithm for spatially balanced sampling is the generalized random tessellation stratified (GRTS) method. Reversed randomized quadrant-recursive raster (RRQRR) is a refinement of or variation on GRTS. The idea behind GRTS is to map the two-dimensional locations into one dimension while preserving some spatial order. The sample is then selected in one dimension, using systematic pps sampling, and mapped back into two dimensions [37,38]. A primary difference between the RRQRR and GRTS algorithms is that GRTS is feature based, so that quadrant-recursion occurs only in areas where features are located and only to a level k that resolves features in the sampling frame so that the sum of inclusion probabilities in a cell is less than 1. In contrast, RRQRR uses a raster to provide a discrete, but fine-grained, approximation of features—which can include any combination of geographic feature types. Again, the RRQRR and GRTS approaches differ. In GRTS, inclusion probabilities are used as weights, before reversing the Morton addresses. In RRQRR, inclusion probabilities are relative inclusion probabilities that specify the probability that a given location (raster cell) will be selected, relative to other locations [38]. Since GRTS and RRGRR share a general resemblance, with only minor differences, we take RRQRR as an example and list the main steps of RRQRR below (see Figure 1):

• Morton code

Divide the study area into four parts, and then assign every unit a sequencing number of 1, 2, 3, or 4, which follows the “N” or “Z” order, thus obtaining the first layer ${\mathrm{L}}_1$. Do the same in the four parts of ${\mathrm{L}}_1$, thus obtaining the second layer ${\mathrm{L}}_2$. Go on until all the units in the study area are included. From the bottom to the top, each unit has a series of layer numbers which make a unique code, i.e., the Morton code.

2.

Linear address

Starting from the top-left corner, the Morton code of each unit is converted into a one-dimensional linear address which also follows the same “N” or “Z” order.

3.

Reversed Morton code

Reverse the layer numbers of each unit, and the reversed Morton code is made. For example, the reversed Morton code of ${\mathrm{M}}_{1234}$ is ${\mathrm{M}}_{4321}^{\prime }$. At the same time, the linear address of the corresponding unit is also changed.

4.

Hierarchical random sorting

Firstly, undertake random sorting of the four parts of ${\mathrm{L}}_1$, then carry on random sorting of all the four pars of ${\mathrm{L}}_2$ until all four parts of ${\mathrm{L}}_{\mathrm{k}}$ are finished. At the same time, the linear address of each unit is changed again.

5.

Sampling

In order to ensure that samples of different importance have different probabilities of being chosen, an inclusion probability raster is introduced (the value of the raster means the possibility of a unit being sampled relative to others). By the filtering calculation between the inclusion probability and the random raster ([0, 1] random even distribution), units with a higher inclusion probability than random probability are saved. According to the new linear address produced by Step 4, randomly choose a number as a starting point, and then extract n consecutive units. Spatially balanced sampling is thus achieved.

The inclusion probability raster plays a decisive role in spatially balanced sampling. For different applications, the factors considered in the design of the inclusion probability raster are distinct. For example, in field surveys, accessibility is the key factor influencing the design of the sampling method, and road network data are usually introduced. By defining the accessibility of the road buffers in different ranges, points far from the road have a small possibility of being chosen, thus avoiding the no-response sampling units. For land-cover products of a 30 m resolution, the landscape in transitional areas often exhibits high fragmentation. If patches of different fragmentation degrees are given an equal inclusion probability, the distribution of the sample points in spatially balanced sampling will be as shown in Figure 2. In contrast, if the patches are given different inclusion probabilities according to the complexity, the distribution of the sample points in spatially balanced sampling will be as shown in Figure 3. Therefore, the sampling method proposed in this paper reasonably considers the relationship between the inclusion probability of samples and the degree of spatial heterogeneity. The patches with large heterogeneity can improve the sampling probability of samples. Otherwise, reduce the inclusion probability of the samples.

2.2. Using the Landscape Pattern-Based Inclusion Probability to Improve Spatially Balanced Sampling

As is well known, the classification errors of remotely sensed images are not evenly distributed, but follow certain rules. For spatial objects with a more regular structure and larger area, the classification results are likely to be better, while objects with more fragmented and sporadic structures are more likely to be misclassified. Therefore, this paper introduces the concept of patches to landscape patterns, regarding land-cover types as patches or corridors with different sizes and shapes. Considering the relationship between landscape complexity and classification error, a landscape which is made up of complex, heterogeneous, and broken patches will have a larger inclusion probability than a landscape which is composed of simple, uniform, and regular patches. The complexity of the landscape can be measured by the LSI, which is calculated by the perimeter ratio of a patch to its equal size square. The calculation of the LSI is as shown in Equation (1):

$$\mathrm{LSI}=\frac{0.25E}{\sqrt{A}},$$

(1)

where E means the perimeter of the patch, A means the area of the patch, and the range of the LSI is equal to or larger than 1. When the LSI equals 1, this indicates that the shape of the patch is the simplest square, and when the LSI value increases, the shape of the patch becomes more complex and irregular.

Since the inclusion probability indicates the possibility of a unit being sampled relative to other units in the population, it is necessary to normalize the LSI. The steps are as follows:

• Calculate the LSI values of all the water patches in the area and draw the corresponding frequency distribution histogram, as shown in Figure 4. Affected by the spatial distribution of the land cover on the surface, the number of broken patches is large but the area proportion is small, while the number of complex patches is small but the area proportion is large. Therefore, the frequency histogram usually shows a left-sided distribution, and the statistics such as mean, median, and mode are small and not suitable for stratification.
• Therefore, in order to narrow the gap between the patches, they are clustered into four groups according to the LSI values, as shown in Figure 5b.
• The higher the landscape fragmentation, the greater the probability of misclassification. Therefore, the unequal inclusion probability is introduced according to the fragmentation of the patches. As shown in Figure 5c, the patches are divided into four layers, and the inclusion probabilities of 1, 0.8, 0.6, and 0.4 are given, respectively, according to the LSI. The patches with strong heterogeneity and large fragmentation have more samples, thus improving the representativeness of the sample points.

3. Experiment

There are three basic components of the accuracy assessment of land-cover products, namely, sampling design, response design, and analysis [39,40,41]. Sampling design refers to how to select the typical sample points from the study area. Response design means how to interpret the reference label of the sample points through a ground survey or visual interpretation of finer images. Analysis involves comparing the difference between the classification and reference data and then expressing it in quantitative ways, so that users can better understand the classification result. Figure 6 depicts the overall design of the experiments. The three main parts are described in detail in the following.

3.1. Sampling Design

Land cover usually has an uneven distribution on the Earth’s surface, and study areas vary widely, thus making it difficult for the determination of sample size, sample allocation, and the spatial layout of sample points. To ensure the representativeness of sample points, a zoning strategy is introduced. In this paper, a comprehensive physical geographical regionalization is used to divide the whole area into three parts: the Tibetan Plateau region, the Northwest China region, and the East China region. The sampling design adopts a top-down design and divides the whole process into two stages. In the first stage, spatial stratification is made and some primary sample units are randomly chosen from each strata. In the second stage, secondary sample units are further chosen from the primary sample units.

In the first stage, 1° × 1° grids serve as the sampling framework under the original map sheet (5° × 6°). At the same time, the broken grids on the edge are merged into a new grid, ensuring that each grid is of the same size. Stratified random sampling is then employed with the sampling ratio of five grids per map sheet, and the spatial stratification is as shown in Figure 7.

In the second stage, the land-cover types are used as strata for their differences in classification accuracy. In addition, considering the characteristics of classification error, the LSI is introduced into the design of the inclusion probability raster to ensure the spatial representativeness and uniformity of the sample points. About 50 sampling points are drawn from each grid, and then sample allocation of the different land-cover types is made according to the proportion of the square root of the area.

3.2. Response Design

Visual interpretation is the main method of sample judgment in this paper. When loading the sample points in Google Earth, first of all, we browsed the available images in this area and selected the best-quality images which had obvious characteristics of the land-cover features and which were taken around the vegetation growth period of the year 2010. We then drew a 3 × 3 pixel block (90 m × 90 m rectangle), with the sampling point as the center. When the sampling unit fell into a homogeneous area, i.e., only one land-cover type in the box, we recorded the land-cover type as the reference label. When the sampling unit lay in an heterogeneous area, i.e., more than one land-cover type in the box, we recorded the primary label and secondary label according to the area size. In addition, in order to facilitate the follow-up analysis, the level of degree of trust of the judgment results was scored (1 for uncertain, 2 for basically certain, and 3 for very certain).

3.3. Accuracy Assessment

The accuracy assessment in this paper is mainly based on the confusion matrix and common accuracy measures, such as overall accuracy, producer’s accuracy, user’s accuracy, and the Kappa coefficient [41,42,43,44]. These indexes describe the classification accuracy from different aspects, and they are also simple and feasible. The overall accuracy is one of the most popular measures, and it indicates the percentage of correctly classified pixels in the population, as shown in Equation (2), where n means the total number of pixels, m means the number of land-cover types, and ${\mathrm{n}}_{\mathrm{ii}}$ means the number of correctly classified pixels in the diagonal.

$$\mathrm{OA}={\sum }_{\mathrm{i}=1}^{\mathrm{m}}\frac{{\mathrm{n}}_{\mathrm{ii}}}{\mathrm{n}},$$

(2)

In addition, the accuracy of each land-cover type can be calculated in the same way as proposed in Equations (3) and (4), where ${\mathrm{n}}_{+\mathrm{i}}$ and ${\mathrm{n}}_{\mathrm{i}+}$ represent the marginal sum of columns and the marginal sum of rows, respectively. The producer’s accuracy represents the probability of each land-cover type being correctly classified in the reference data. Similarly, the user’s accuracy means the probability of each land-cover type being correctly classified in the classification data.

$${\mathrm{PA}}_{\mathrm{i}}=\frac{{\mathrm{n}}_{\mathrm{ii}}}{{\mathrm{n}}_{+\mathrm{i}}},\mathrm{i}=1,2,\dots ,\mathrm{m},$$

(3)

$${\mathrm{UA}}_{\mathrm{i}}=\frac{{\mathrm{n}}_{\mathrm{ii}}}{{\mathrm{n}}_{\mathrm{i}+}},\mathrm{i}=1,2,\dots ,\mathrm{m},$$

(4)

Kappa analysis is a multivariate statistical analysis method which can reflect whether the classification result is statistically significant compared to a random classification result [45]. Differing from the overall accuracy, Kappa analysis takes a comprehensive consideration of all the elements in the confusion matrix (not only the elements in the diagonal, but also the omission and commission errors off the diagonal). These indexes should usually be calculated at the same time, in order to obtain more information about the classification.

$$\mathrm{Kappa}=\frac{\mathrm{n}{\sum }_{\mathrm{i}=1}^{\mathrm{m}}{\mathrm{n}}_{\mathrm{ii}}-{\sum }_{\mathrm{i}=1}^{\mathrm{m}}{\mathrm{n}}_{\mathrm{i}+}{\mathrm{n}}_{+\mathrm{i}}}{{\mathrm{n}}^2-{\sum }_{\mathrm{i}=1}^{\mathrm{m}}{\mathrm{n}}_{\mathrm{i}+}{\mathrm{n}}_{+\mathrm{i}}},$$

(5)

4. Results

4.1. Selected Samples

As shown in Figure 5, the city of Suzhou in China, which is located in the Taihu River Basin, is rich in water resources (Figure 5a). We take the water as an example to show how LSI-based spatially balanced sampling works.

As shown in Figure 5d, the inclusion probability raster was set up, and 300 spatially balanced sample points were extracted using the RRQRR algorithm.

China was selected as the validation region due to its vast territory, large latitude range, complex terrain, diverse climate, rich vegetation species, and complex spatial distribution. GlobeLand30-2010 for China includes nine types of land cover, namely, cropland, forest, grass, shrub, wetland, water, artificial, bare land, and snow and ice (the land cover in China, not including tundra). The reference data refer to the dataset representing the ground truth, which can be used to judge whether the classification is right or wrong. This paper takes Google Earth as the main reference data source.

Based on the two-stage stratified spatially balanced sampling method, a total of 160 primary sample units and 7490 sample points were selected in China. The sample size calculation is shown in

Table 1. Sample distribution based on two-stage stratified spatially balanced sampling.

Region	Area Ratio	Primary Sampling Units	Sampling Points
Tibetan Plateau	28.68%	47	2198
Northwest China	26.30%	42	1967
East China	45.01%	71	3325
Total	1	160	7490

and the spatial distribution of each region is shown in Figure 8.

According to the degree of trust system in the sample judgment results, there were 6336 points with a high level of trust, accounting for 84.59% of the total. There were 906 points in the middle level of degree of trust, with most in the Tibetan Plateau area (416 in the Tibetan Plateau, 274 in Northwest China, and 216 in East China). The remaining points were of low quality and could be excluded from the confusion matrix (248 in total, 124 in the Tibetan Plateau, 20 in Northwest China, and 104 in East China).

According to the results of the 7490 sample points, about 6.84% of the total samples lacked high-resolution images, and the images were characterized by a dim tone, blurred texture, and no shadow in Google Earth. A statistical analysis of the capture time of the images of all the sample points was conducted. From the statistics of the year distribution, the time span was quite long, with a range of 17 years. As shown in

Table 2. Statistics of the yearly distribution of the images.

Region	Year
	2000–2002	2003–2005	2006–2008	2009–2011	2012–2014	2015–2017	No Image	Total
Tibetan Plateau	2	29	59	1369	494	27	218	2198
Northwest China	3	33	61	1039	620	76	135	1967
East China	7	38	164	1350	1442	165	159	3325
Total	12	100	284	3758	2556	268	512	7490

, 1.50% of the total images were recorded early in 2000–2002 and 2003–2005, and 88.10% of the total were recorded in 2006–2008, 2009–2011, and 2012–2014. From the statistics of the yearly distribution, each month was relatively balanced, except for October and December, as shown in

Table 3. Statistics of the monthly distribution of the images.

Region	Month
	1	2	3	4	5	6	7	8	9	10	11	12	No Image	Total
Tibetan Plateau	99	198	234	117	63	25	49	104	111	244	169	567	218	2198
Northwest China	48	71	177	156	149	115	195	176	215	276	97	157	135	1967
East China	205	167	305	347	238	225	164	186	287	353	261	428	159	3325
Total	352	436	716	620	450	365	408	466	613	873	527	1152	512	7490

. For the period of vegetation growth, only 24.73% of the images were taken between June and September.

4.2. Accuracy Assessment

As a result of location or registration errors, more than one label was recorded in the 90 m × 90 m sampling unit. The confusion matrix is based on hard classification results, which means that only one reference label corresponds to the sample point, so that it is necessary to define the consistency between the classification label and the reference label. When compared with the classification data, the classification is correct as long as the classified label is the same as the primary label or secondary label. Otherwise, the primary label is considered the reference land-cover type. According to the confusion matrix based on the 7242 sample points (sample points with a lower level of trust are not included), the overall accuracy of GlobeLand30 in China was 80.46% and the Kappa coefficient was 0.76. The sum of each row in

Table 4. Confusion matrix of GlobeLand30 in China.

	Cropland	Forest	Grass	Shrub	Wetland	Water	Artificial	Bare Land	Snow and Ice	Total
Cropland	1286	53	25	8	1	7	30	22	0	1432
Forest	71	1132	55	18	5	9	3	12	1	1306
Grass	143	197	1445	50	10	14	16	165	5	2045
Shrub	4	1	4	106	1	1	0	45	0	162
Wetland	7	3	18	3	55	6	0	6	0	98
Water	12	5	22	3	4	256	0	15	0	317
Artificial	34	6	5	3	0	2	271	5	0	326
Bare land	25	18	158	9	2	9	2	1176	7	1406
Snow and Ice	0	9	14	0	0	2	0	25	100	150
Total	1582	1424	1746	200	78	306	322	1471	113	7242
UA (%)	81.29	79.49	82.76	53.00	70.51	83.66	84.16	79.95	88.50	-
PA (%)	89.80	86.68	70.66	65.43	56.12	80.76	83.13	83.64	66.67	-

is the classified number of the object class, and each column is the real number of the object class. As shown in Table 4, the highest user’s accuracy was 88.50% (snow and ice), while the lowest was 53.00% (shrub). The highest producer’s accuracy was 89.80% (cropland), while the lowest was 56.12% (wetland).

The confusion between GlobeLand30 and the reference data is established on the overlay analysis. As shown is Figure 9a, the horizontal axis represents the land-cover type in the reference data, and the vertical axis represents the percentage of land cover i in the reference data misclassified as other land-cover types, which corresponds to the omission errors. Similarly, as shown in Figure 9b, the horizontal axis represents the land-cover type in GlobeLand30, and the vertical axis represents the percentage of land cover i in GlobeLand30 classified as other land-cover types, which corresponds to the commission errors. When the land-cover type described in the vertical axis is inconsistent with what is in the horizontal axis, confusion between land-cover types occurs. As shown in Figure 9a,b, confusion between forest/grass/shrub, wetland/water, and grass/bare land are important factors affecting the classification accuracy of GlobeLand30.

In addition, according to the regional statistics, the classification accuracies of the three regions are shown in Figure 10a–c. In terms of vertical comparison, the classification accuracy of cultivated land in the three regions was the most stable, and the user’s and producer’s accuracies were all about 80%. The reason for this was the image characteristics of cropland, which show an obvious striped texture in the images after cultivation and harvesting. The cropland accounts for a small proportion in the Tibetan Plateau and Northwest China regions, and it is easily affected by image quality, resulting in a relatively low producer’s accuracy, while the irrigated lands in East China are easily confused with wetland and water body. The seasonal rivers in the Tibetan Plateau and Northwest China regions are easily confused with wetlands. In addition, economic crops such as tea and fruit trees in the East China region, which should be classified as cropland, are easily confused with forest, while in the Northwest China region, they are confused with pasture. From a horizontal perspective, the classification accuracy of forest and shrub in the Northwest China region was poor, which is closely related to the vegetation morphology (low, sparsely populated, low canopy coverage) and a sparse distribution pattern in this arid region.

The results from the confusion matrix based on 7242 sample points with a high degree of trust show that the overall accuracy of GlobeLand30 in China was 80.46% and the Kappa consistency coefficient was 0.76, which is close to the 84.2% in Wang et al. [45], but the total number of samples used in this paper was 7490, which is slightly less than the number of samples (8400) in Wang et al. (See Figure 11). Compared with stratified sampling, the improved spatial balanced sampling proposed in this paper gives higher inclusion probability to areas with high spatial heterogeneity, and distributes more samples while ensuring the uniformity of samples in this area, so as to improve the classification accuracy. As shown in Figure 8, the samples of the sampling design method in this paper were more evenly distributed, especially in places where the fragmentation degree of patches is high, and the samples were denser. In addition, the omission errors of shrubs, wetlands, and artificial land in this paper were obviously less than under stratified random sampling. Yang et al. [15] evaluated the accuracy of GlobeLand30 products in China by collecting 600 m × 600 m validation sample units (VSUs) in homogeneous areas. Although its accuracy was 82.39%, which is slightly higher than in this paper, this method does not consider the heterogeneity of landscape (which is also one of the defects of the verification unit mentioned in this paper). For example, most rivers in China are small patches and are long and thin, so that this 600 m × 600 m verification unit excludes these patches, which makes the estimated classification accuracy of the corresponding land-cover higher, and the accuracy of this paper is comparable to this higher accuracy. In addition, VSUs were selected according to normal distribution, and the distribution of samples was not more uniform than that collected in this paper.

5. Discussion and Conclusions

In this paper, we evaluated the accuracy of the Chinese GlobeLand30 product using a two-stage stratified spatially balanced sampling plan, obtaining 7242 valid samples and 248 invalid samples according to the degree of trust in sample judgment.

For sampling design, compared with simple random sampling (SRS), cluster sampling, systematic sampling, and stratified sampling, spatial balanced sampling is a random, equiprobable, efficient, low-cost, well-balanced, and adaptable sampling method. We used landscape pattern-based inclusion probability to improve the spatially balanced sampling in this paper. The improved spatially balanced sampling provides a higher inclusion probability to areas with high spatial heterogeneity, and distributes more samples while ensuring the uniformity of samples in this area, so as to improve the classification accuracy.

For accuracy analysis, the results from the confusion matrix based on 7242 sample points with a high degree of trust show that the overall accuracy of GlobeLand30 in China was 80.46% and the Kappa consistency coefficient was 0.76, which is close to the third-party assessment accuracy of GlobeLand30.

In this paper, the sampling method was applied to the regional validation of GlobeLand30 in China. Compared with other scholars’ research, the results show that the method is effective and reasonable.

Besides the classification error, the raw remote sensing images and the phenological conditions also influenced the accuracy of GlobeLand30. There are two main factors that can influence the validation accuracy of GlobeLand30: (1) the quality of the reference data, i.e., the coverage and the time distribution of the high-resolution images may not be comprehensive enough to support all the validation points; and (2) the sample unit design, i.e., when the sample falls on the adjacent area due to positioning or matching error, it will be judged as another type of surface coverage, and the positioning error will be confused as a classification error. According to the survey results of the 7490 sample points, 12.71% of the 3 × 3 sample units (952 in total) were mixed with two or more kinds of land cover, and mixed units were easily found in cropland, forest, grass, and bare land.

Table 2 shows the distribution of the reference data of sample points in the acquisition years. The reference data acquired around 2010 accounted for 88.10% of all the reference data, that is, the phase of most reference data was close to that of Global30 products, which provided guarantees for the verification of land cover data products. This is because the extent to which the time phase is reasonable and rich is one of the key factors of image reference data quality. The land cover will of course change constantly with time, for example, cultivated land becomes building area, deforestation, lakes dry up, and so on. If there is no image of the corresponding year, it is easy to make the wrong judgment because of the change of land cover. Moreover, the land cover shows seasonal and periodic changes, such as seasonal changes of water and grassland, and melting of snow and ice. The occurrence of these phenomena will have an impact on both remote sensing mapping and subsequent accuracy evaluation. Therefore, the rationality and richness of image phase are also of great significance to the verification of land cover data products. Table 3 summarizes the distribution of all reference data in each month in the past 17 years. The number of reference data in other months is relatively balanced except for individual months, which ensures that the sample points have corresponding images in each month and prevents most of the reference images from being concentrated in a few months. Otherwise, for some seasonal land cover, the time phase is not rich enough, which will lead to classification errors.