Urban Vegetation Extraction from High-Resolution Remote Sensing Imagery on SD-UNet and Vegetation Spectral Features

Abstract

Urban vegetation plays a crucial role in the urban ecological system. Efficient and accurate extraction of urban vegetation information has been a pressing task. Although the development of deep learning brings great advantages for vegetation extraction, there are still problems, such as ultra-fine vegetation omissions, heavy computational burden, and unstable model performance. Therefore, a Separable Dense U-Net (SD-UNet) was proposed by introducing dense connections, separable convolutions, batch normalization layers, and Tanh activation function into U-Net. Furthermore, the Fake sample set (NIR-RG), NDVI sample set (NDVI-RG), and True sample set (RGB) were established to train SD-UNet. The obtained models were validated and applied to four scenes (high-density buildings area, cloud and misty conditions area, park, and suburb) and two administrative divisions. The experimental results show that the Fake sample set can effectively improve the model’s vegetation extraction accuracy. The SD-UNet achieves the highest accuracy compared to other methods (U-Net, SegNet, NDVI, RF) on the Fake sample set, whose ACC, IOU, and Recall reached 0.9581, 0.8977, and 0.9577, respectively. It can be concluded that the SD-UNet trained on the Fake sample set not only is beneficial for vegetation extraction but also has better generalization ability and transferability.

1. Introduction

In urban areas, vegetation holds a crucial position within the urban ecosystem, assuming multifaceted and considerable important roles [1,2], including alleviating environmental pollution [3,4,5], protecting biodiversity [6,7], regulating the ecosystem [8], reducing the urban heat island effect [9,10], and improving the overall quality of life [11]. The high-precision vegetation information is not only beneficial for decision-makers to exploit the vegetation resources but also helps them to evaluate the urban ecological environment [12], plan urban life [13], and establish sustainable cities [14]. How to efficiently, accurately, and intelligently obtain urban vegetation information has aroused the interest of many researchers.

In recent years, the rapid advancements in space optical remote sensing technology have greatly promoted the Earth Observation System [15] and it has emerged as a potent and impactful tool for acquiring regional and global vegetation information [16,17,18,19]. Currently, a variety of optical, hyperspectral, and radar remote sensing data have been widely utilized for vegetation extraction, such as Gaofen and ZY series data [20,21,22] from China, Sentinel series data from the European Space Agency [23,24,25], and Landsat series data from the United States [26]. For applying remote sensing technology to urban vegetation research, a critical aspect is to extract ground truth vegetation information from images swiftly and effectively. Traditional methods can be categorized into three kinds: visual interpretation, pixel-based classification [27], and object-oriented classification [28,29,30]. Manual visual interpretation is usually considered the most accurate extraction method, but there are some disadvantages such as human error, low efficiency, and high cost. The pixel-based classification method was proposed by Roger M. Herold et al. in 1983. It can identify targets based on pixel values, features, and variations. Bey classified land use temporally via the spectral and textural features of Landsat data [31]. Shen proposed a 3D Dabor-wavelet-based method to identify each pixel and finally achieved hyperspectral image classification [32]. Hadi found that maximum likelihood algorithms could achieve higher accuracy by using a combination of original bands and the Ratio Vegetation Index [27]. Although relative research demonstrates that pixel-based classification methods are more efficient than manual visual interpretation, they are limited by “the different objects with the same spectrum” and “the same object with different spectrum” problems. Some researchers also have demonstrated that object-oriented classification methods surpass pixels-based classification methods in accuracy [33,34,35,36]. Object-oriented classification methods aim to classify objects by calculating each object’s features, such as image spectrum, shape, texture, and size [37]. However, the computational process is too time-consuming and labor-intensive to extract ground truth information swiftly.

With the development of artificial intelligence (AI), deep learning has emerged as a popular approach for extracting remote sensing information, owing to its remarkable precision and efficiency. Deep learning is an important branch of AI, and its development can be traced back to the 1950s. However, deep learning did not experience a breakthrough until the 21st century; many world-renowned universities have invested huge human and financial resources to conduct related research in the field. In recent years, with the proposal of deep learning theory, the Convolutional Neural Network (CNN) has ushered in an era of rapid development. CNN is one of the most successful and important deep learning algorithms whose main network architecture consists of input layer, convolution layer, pooling layer, full connection layer, and output layer [38]. The reasonable architecture ensures powerful learning abilities to automatically learn features from massive samples, making it a main algorithm for academic research such as image classification [39], semantic segmentation [40,41], and land use classification [42,43]. Additionally, a series of semantic segmentation convolutional neural networks have been proposed for providing more options to extract urban vegetation information, including the full convolution network (FCN) [44], U-Net [45], SegNet [46], etc. The study shows that deep learning algorithms extract vegetation information with high accuracy and low cost, whose extraction accuracy has obvious advantages over shallow machine learning algorithms such as Support Vector Machine (SVM) and Random Forest (RF) [47,48], and which can also monitor large-scale vegetation information [49]. However, previous research has primarily focused on visible image data, which possesses limited spectral information for effectively distinguishing vegetation and non-vegetation areas. To enhance the ability of deep learning models to identify vegetation information [50], several scholars have utilized the spectral features of vegetation in the near-infrared band (NIR) to enhance the gap between vegetation and other ground objects. Nezami employed hyperspectral data to construct a training set and determine the most efficient combination of features for tree species classification [51]; Li introduced the red-edge band of GF-6WFV when establishing the training set [52]; Chen evaluated the performance of a proposed neural network by using samples with NIR from Sentinel-2A images [53]; Xu utilized the NIR from Gaofen-2 satellite images to establish a dataset for urban green space classification [54]; Roberto E. Huerta evaluated the potential of two deep learning models for semantic segmentation of urban green spaces using 12 three-band compositions. The bands mainly include vegetation indices (normalized difference vegetation index (NDVI), two-band enhanced vegetation index (EVI2), normalized difference water index (NDWI)) and spectral single bands (near-infrared (NIR), red (R), green (G), blue (B)) obtained from the original WV2 data [55]. The findings indicate that introducing red-edge bands or relevant vegetation indices into the dataset can further enhance the accuracy of vegetation extraction. However, the most suitable sample set may vary depending on the encoder used in different deep learning networks [55]. Therefore, researchers often optimize the encoders of deep learning networks to achieve better accuracy when dealing with irregular urban vegetation boundaries, complex vegetation shapes, uneven vegetation coverage, and scattered vegetation distribution. Liu proposed a multiple architecture method (DeepLabv3plus) for extracting urban green spaces from Gaofen-2 images, employing an Atrous Spatial Pyramid Pooling (ASPP) module within the encoder to capture rich contextual semantic information [47]. Men built a new model known as Concatenated Residual Attention UNet (CRAUNet), which introduced a residual module to preserve more feature information from the original image during the feature extraction process [49]. While these methods demonstrate superior performance in extracting urban green spaces, the price in exchange is that the excessive sampling rate of atrous convolution in the ASPP module leads to the loss of details, and the introduction of the residual network module increases the parameters, resulting in a higher computer burden.

To reduce ultra-fine vegetation omissions, minimize the computational burden, and improve vegetation extraction accuracy, a deep learning network named separable densely UNet (SD-UNet) is proposed, and it utilizes the vegetation spectral features in NIR during network training. The general research process was divided into four steps. Firstly, the acquired Gaofen-1 remote sensing images are preprocessed to ensure the accuracy of subsequent experiments. Secondly, three high-resolution urban vegetation sample sets are established to research the effect of NIR on vegetation extraction results. Each sample set contains 3060 samples of 256 × 256 in size to support the training and verification of the urban vegetation extraction model. Thirdly, SD-UNet, which consists of an encoder and a decoder, is proposed. Specifically, SD-UNet introduces dense connections, separable convolutions, batch normalization layers, and the Tanh activation function. This framework can capture more ultra-fine vegetation features and reduce the computational burden. Finally, the SD-UNet is compared with other methods. The SD-UNet is applied to four different scenes and the best model is applied to two administrative divisions.

The key contributions of this article can be summarized as follows:

• An optimized convolutional neural network (SD-UNet) was proposed to effectively extract urban vegetation from Gaofen-1 remote sensing images.
• Three sample sets were established to evaluate the influence of the vegetation spectral features on the model extraction results. The SD-UNet was trained on three sample sets, finally obtaining the best model.
• The SD-UNet’s performance on urban vegetation extraction was compared with U-Net, SegNet, NDVI, and RF. The SD-UNet trained on three sample sets was applied to four scenes and the best model was applied to two administrative divisions to evaluate their generalization ability in vegetation extraction.

The remaining sections of this article are outlined as follows. Section 2 illustrates the proposed network structure, experimental materials, and related works. Section 3 shows the experimental results and provides a comprehensive analysis. Section 4 provides the discussion, and Section 5 offers a conclusion.

2. Materials and Methods

2.1. Study Area and Data Sources

The main urban area of Chongqing is situated in western Chongqing (Figure 1), belonging to the subtropical wet monsoon climate. It is located among the Jinyun, Zhongliang, Tongluo, and Mingyue mountains, at the intersection of the Yangtze River and Jialing River, forming the Yuzhong Peninsula. The geographical scenes are complex: the buildings in the mountainous area are densely arranged and the terrain is undulating; the urban vegetation has serious fragmentation. Therefore, vegetation information extraction is more difficult under cloud and misty conditions.

The Gaofen-1 satellite, launched on 31 March 2018, takes images with a panchromatic band and four multispectral bands, offering spatial resolution of 2 m and 8 m, respectively. The study data are Gaofen-1 remote sensing images of Chongqing city, obtained from the Application Center of the Chinese High-Resolution Earth Observation System. The imaging time was in September 2021 and the cloud cover was less than 5%. Images were preprocessed by radiometric calibration, atmospheric correction, orthorectification, and image fusion. Finally, a multispectral image with 2 m spatial resolution was obtained, and the digital number (DN) quantization depth is 16 bit, ranging from 0 to 65,535.

2.2. Experimental Process

The research workflow is shown in Figure 2. The research process was divided into four parts, including preprocessing the Gaofen-1 images, establishing the vegetation sample sets, training the models, and evaluating the model’s generalization ability. For remote sensing image interpretation, preprocessing is a very important step. The goal of preprocessing is to eliminate or reduce the influences (such as noise, uneven brightness, and geometric distortions), ensuring the reliability and accuracy of subsequent vegetation extraction results. Therefore, in the first part, Gaofen-1 images are preprocessed by radiometric calibration, atmospheric correction, orthorectification, and image fusion. Finally, a multispectral image with 2 m spatial resolution is obtained. In the second part, three urban vegetation sample sets are established by combining different bands, namely the Fake sample set (NIR-RG), NDVI sample set (NDVI-RG), and True sample set (RGB). In the third part, SD-UNet is proposed, and the specific structure and content are shown in Section 2.5.1. Additionally, for evaluating the advantages of SD-UNet and the impacts of different vegetation spectral features on the model’s vegetation extraction accuracy, this part trains different networks using different sample sets, resulting in nine models. In the last part, nine models are validated. The validation results are compared and analyzed. For evaluating SD-UNet’s performance in practical application scenes, the SD-UNet is applied to four different scenes. To make the experimental results more convincing, the vegetation of two administrative divisions is extracted by using the best model.

2.3. Sample Sets

Deep learning requires massive samples to automatically learn features and extract targets. Three urban vegetation sample sets were employed to train and validate the deep learning model. The following briefly describes the sample set.

The urban vegetation sample sets are composed of images and labels. The labels consist of vegetation and background areas, where white areas represent urban vegetation and black areas represent non-vegetation. All labels are established by manual visual interpretation. The process is mainly realized through three steps. Firstly, the vector boundaries of vegetation in the study area were delineated on Gaofen-1 images using ArcGIS 10.2 software, with reference to high-resolution Google images. Secondly, these vector boundaries were corrected through field surveys (Figure 3). Notably, not all labels were field surveyed, only the uncertain ground truth types in Gaofen-1 remote sensing images were physically verified. Finally, the results are stored in the form of raster images as labels. The labels (Figure 4a) in the sample set are binary images, where 1 represents urban vegetation and 0 represents background. Since the sample size is fixed, both images and labels were clipped to a size of 256 × 256.

Three sample sets are used to train and validate SD-UNet, as well as to calculate vegetation extraction accuracy. These sample sets include the standard false-color image vegetation sample set (Fake sample set, Figure 4c), the normalized difference vegetation index sample set (NDVI sample set, Figure 4d), and the true-color sample set (True sample set, Figure 4b). The Fake sample set was synthesized by three bands: NIR, R, and G. Vegetation is particularly prominent in standard false-color images due to its higher reflectance and distinct spectral features in the near-infrared region [56]. The NDVI sample set was synthesized by NDVI, R, and G bands. NDVI is widely applied as a key parameter to distinguish vegetation and non-vegetation areas [57,58]. Its value equals the ratio of the difference between NIR and R to the sum of NIR and R. Generally, NDVI values for vegetation are greater than 0, while most of the values for non-vegetation are less than 0 (with some exceptions, such as barren land of rock, sand, snow, and soil [59,60,61]). Thus, the NDVI values were determined to range from 0.355 to 0.854 through numerical statistics. Finally, the True sample set with R, G, and B bands was used as the research control group for model training and scene application. Considering that training data must contain as many different samples as possible to avoid over-fitting the generated model, data augmentation techniques such as geometric transformations (flipping, rotation, and random cropping), adding noise, and fuzzy transformations were applied. The final sample sets consisted of 2448 training samples and 612 validation samples, with a ratio of 8:2.

2.4. Scene Data

To verify the transferability and generalization ability of SD-UNet, four 1000 × 1000 pixel images with typical mountainous urban scene characteristics were used to extract vegetation and analyze the results. Notably, all scene images did not participate in the sample set construction. In addition, to make the verification results more convincing, the vegetation of Yuzhong District and Nan’an District was extracted. In order to eliminate the influence of the atmosphere condition and sunlight, four images were preprocessed via radiometric calibration, atmospheric correction, orthorectification, and image fusion.

As shown in Figure 5: (a) represents a scene image with cloud and misty conditions. Chongqing City, a famous “fog city”, the biggest interference factors are clouds and misty when extracting vegetation from remote sensing images. Hence, extracting vegetation in this scene can evaluate the transferability of SD-UNet. (b) represents a scene image located in the urban center where the vegetation and buildings are mixed. Thus, the biggest challenge is how to avoid interference from buildings in the process of extracting vegetation. Extracting vegetation in this scene can evaluate SD-UNet’s performance in accurately extracting fine vegetation within building gaps. (c) represents a suburban scene located at the edge of the city, where residential areas, factories, cultivated land, and roads are intertwined, and the vegetation is distributed randomly. Extracting vegetation in this scene can evaluate SD-UNet’s performance in a complex background. (d) represents park vegetation in the city where vegetation is concentrated and covers a large area, but is affected by interference from the lake. Extracting vegetation in this scene can evaluate SD-UNet’s performance in large-scale areas.

2.5. Methods

2.5.1. SD-UNet Model

U-Net, originally introduced in 2015 by Ronneberger et al. [45], is an improved network based on a fully convolutional network (FCN), named after its U-shaped architecture. Compared to other semantic segmentation networks, U-Net possesses significant advantages, particularly its symmetric encoder–decoder structure and skip connections. The encoder mainly reduces the spatial resolution of the input image through down-sampling and gradually extracts high-level semantic features, while the decoder mainly restores the spatial resolution of the feature maps through up-sampling. Skip connections enable the decoder to directly obtain feature information from different levels of the encoder and integrate these features with its own extracted features, thereby improving the capability of image semantic segmentation. To address U-Net’s overfitting and accuracy fluctuations during training with small samples, SD-UNet (Figure 6) was proposed to enhance the model’s generalization ability and transferability. The letter “S” in SD-UNet indicates two meanings: one refers to the separable convolutions, and the other refers to “Septal” which represents the alternating arrangement of ordinary convolution and separable convolution. The letter “D” represents the dense connections.

The design of SD-UNet mainly focuses on optimizing the encoder by increasing the network depth and introducing dense connections to enhance vegetation extraction accuracy. For better fitting vegetation features and expanding the receptive field of the network, the number of convolutions was increased at each level of the encoder. This enhancement enables the network to capture global semantic contextual information more effectively. At the same time, to better connect the feature information between the encoders, the idea of dense connections was introduced. Specifically, a dense connection layer was added at the input of each level to enhance its relationship with all the previous levels. The dense connection can enhance the feature transfer and alleviate gradient disappearance. By introducing dense connections, the utilization of feature maps at each scale can be maximized, and low-level vegetation features can be retained as much as possible, thereby reducing the loss of fine vegetation during the convolution process. However, the aforementioned two methods also increased network parameters, so that the computational burden was increased significantly. To address this issue, separable convolutions were introduced [62]. Compared to conventional convolutions, separable convolutions have fewer parameters, which streamline the network backbone. However, some details may be overlooked by the overuse of separable convolutions. Through research, the separable convolutions finally were positioned in the second and fourth levels of the network, which not only increases the network depth but also reduces computational parameters, while preserving SD-UNet’s ability to identify and extract fine vegetation. Furthermore, a batch normalization (BN) layer [63] was introduced after each convolution layer. BN plays a crucial role in alleviating the issues of gradient disappearance and exploding, while also accelerating the convergence speed of the network. It can not only stabilize the network’s training process by normalizing the inputs and outputs at each convolution operation but it can also effectively avoid overfitting and improve the network’s overall performance. Particularly in scenes with limited samples, overfitting is prone to occur in traditional U-Net architectures, but SD-UNet can reduce the dependence on the data and enhance the convergence and fitting ability for urban vegetation extraction. Finally, for a better convergent and fitting network, the Relu activation function [64] was replaced with Tanh.

2.5.2. Experimental Environment

The TensorFlow framework was used to build SegNet, U-Net, SD-UNet, and train network models. The hardware configurations mainly included an Inter Core i9-10900F CPU and an NVIDIA GeForce RTX2060 GPU with 12-GB frame buffer in the experiment. The image and label sizes in the sample set were set to 256 × 256 pixels, the initial learning rate was set to 0.001, and the epoch was set to 200. Considering the limitation of GPU memory size, the batch size was set to 4. The learning rate determined the step size of parameter update during each iteration to decrease the model gradient until the best. Batch size represents the sample number in one iteration. Epoch represents the number of complete training samples.

2.5.3. Assessment Measures

A single image element is typically used to calculate accuracy evaluation metrics. Four metrics were selected to evaluate the vegetation extraction results, namely accuracy (ACC), intersection over union (IOU), Recall, and loss which is calculated by the cross-entropy loss function. ACC is the proportion of the correctly identified subjects among all ground subjects in model extraction results, IOU is the proportion of the correctly identified vegetation in all actual and identified vegetation, and Recall is the proportion of all actual vegetation that is correctly identified as vegetation.

Three metrics are calculated as follows:

$$\mathrm{A}\mathrm{C}\mathrm{C}=\frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}$$

(1)

$$\mathrm{I}\mathrm{O}\mathrm{U}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}$$

(2)

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}$$

(3)

where TP indicates the number of vegetation regions correctly classified as belonging to vegetation (i.e., true positives), FN indicates the number of vegetation regions incorrectly classified as belonging to non-vegetation (i.e., false negatives), FP indicates the number of non-vegetation regions incorrectly classified as belonging to vegetation (i.e., false positives), and TN indicates the number of non-vegetation regions correctly classified as belonging to non-vegetation (i.e., true negatives).

3. Results and Analysis

To evaluate the advantages of SD-UNet, SD-UNet was compared with SegNet, U-Net, NDVI, and RF.

3.1. Results

3.1.1. SD-UNet Results

As shown in Figure 2, nine models were obtained. In this section, the nine models are validated. The vegetation extraction results are shown in

Table 1. Vegetation extraction results from different deep learning methods.

Methods	Fake Samples		NDVI Samples		True Samples
	Block 1	Block 2	Block 1	Block 2	Block 1	Block 2
Images
Labels
SegNet
U-Net
SD-UNet
Note: The red areas of the images represent the vegetation. The colorful boxes in Block 1 and Block 2 represent the marked areas, the white pixels represent the vegetation, and the black pixels represent the background.

, where white areas represent vegetation and black represent non-vegetation.

From the vegetation extraction results of all models, it can be seen that the SegNet performed the worst in urban vegetation extraction. The vegetation boundaries are difficult to accurately fit the labels, and the misclassification and omissions are pretty clear. Especially in block 2, there was a loss of detail in the circular road region (red box), and the extracted results were obviously different from the ground truth delineated by the labels. In contrast, the SD-UNet and U-Net showed significantly better results in extracting urban vegetation information, with most of the extraction results matching labels. However, there were still minor differences in the extraction results for fine vegetation. In block 1, the SD-UNet showed an apparent ability to identify and extract finer strip vegetation along the sides of roads (green box). In contrast, the U-Net model performed slightly less effectively in this aspect. Taking a different perspective to compare the results obtained by models trained on three sample sets, as shown in block 2 (yellow box), the model trained on the True sample set showed obvious misclassification such as the playground, whose artificial turf was misclassified as vegetation. In contrast, the models trained on the Fake sample set and NDVI sample set successfully avoided such errors. Finally, the vegetation extraction results of the same network trained on the Fake sample set and the NDVI sample set were compared, as shown in the green box. It can be seen that the network trained on the Fake sample set performed better. In conclusion, the SD-UNet trained on the Fake sample set is the best model.

3.1.2. NDVI Results

The NDVI is a widely used traditional method in the realm of vegetation monitoring. Its principle is to segment vegetation and non-vegetation by setting a threshold. Through numerical statistics, the range of NDVI value was determined to be between 0.355 and 0.854. In the previous section, the best model was obtained. Comparing the best model with the NDVI, it can be seen that there is a clear difference in the vegetation extraction results of the two methods, as shown in Figure 7. (a) is the images, (b) is the fake samples; (c) is the labels, (d) is the extraction results of NDVI, and (e) is the extraction results of the best model. It can be clearly seen that the edges of the vegetation extracted by NDVI are not smooth enough and appear obviously jagged. The overall results did not align well with the labels, and many non-vegetation areas were misclassified as vegetation. For example, when comparing the vegetation extraction results on the sides of narrow roads, the obvious differences between the NDVI extraction results and labels can be seen just from the human eye. In contrast, the vegetation results extracted by the best model were closer to the labels. In general, the vegetation extraction results only relying on NDVI were unsatisfactory, particularly the phenomenon of “same object with different spectra” and “different objects with same spectra” in remote sensing images had an apparent impact on NDVI. Therefore, to improve classification accuracy and enhance the model’s discriminative capability between vegetation and non-vegetation, it is necessary to fuse multiple bands of information [65].

3.1.3. Random Forest Results

Random Forest is a relatively representative machine learning method to extract targets or specific information from remote sensing images. It has been extensively applied in various fields, including target detection, land cover classification, vegetation extraction, etc. The principle of RF is to achieve target extraction by classifying or regressing pixels in an image. The best model was compared with RF, and it was found that the vegetation extraction results of the best model were significantly better than RF. To provide reliable evaluation results, the same images were used, as shown in Figure 8. (a) shows the images, (b) shows the fake samples; (c) shows the labels, (d) shows the extraction results of RF, and (e) shows the extraction results of the best model. In Figure 8d, it can be clearly seen that the extraction results not only exhibited a “salt and pepper” phenomenon but also showed numerous omissions with noticeable “foggy” vegetation boundaries. Specifically, as shown in the red box of the fourth row, omissions on narrow roads and buildings could be observed. In contrast, the vegetation results extracted by the best model showed more integrity, and the omission phenomenon was rarely produced in the same images. In addition to the above-mentioned problems, the vegetation results extracted by NDVI and RF exhibited opposite misclassification characteristics. The former misclassified more vegetation as non-vegetation, while the latter misclassified more non-vegetation as vegetation.

3.2. Analysis

3.2.1. SD-UNet vs. U-Net, SegNet

SD-UNet, U-Net, and SegNet were trained on three urban vegetation sample sets with the same hyperparameter settings. The accuracy curves show the fitting process of the network macroscopically, as shown in Figure 9. It can be seen that each model was trained for 200 epochs, and the accuracy curve trends of models obtained by training the same network with different sample sets are similar. (a–c) show the accuracy curve during the training SegNet. The three models showed noticeable fluctuations during the fitting process, and all the loss curves showed a phenomenon from a downward trend to an upward trend. (d–f) show the accuracy curve during the training U-Net. It can be seen that the accuracy of the three models remained stable during the initial few epochs. Subsequently, the loss curve sharply decreased and the other accuracy curves rapidly rose. Eventually, all the curves gradually reached a stable fitting state. (g–i) show the accuracy curve during the training SD-UNet. Compared to the previous accuracy curves, the accuracy curve of SD-UNet appeared smoother and more uniform changes. Meanwhile, SD-UNet achieved an accuracy of over 0.9 in the early few epochs, followed by a gradual increase.

Nine models extracted the urban vegetation from the validation images and calculated the accuracy based on labels. The final results are shown in

Table 2. Accuracy comparison of nine models.

Method	Fake Sample Set			NDVI Sample Set			True Sample Set
	ACC	IOU	Recall	ACC	IOU	Recall	ACC	IOU	Recall
SegNet	0.8902	0.7962	0.8895	0.8910	0.5659	0.8910	0.8673	0.5544	0.8568
U-Net	0.9432	0.7703	0.9419	0.9429	0.7658	0.9422	0.9182	0.8305	0.9163
SD-UNet	0.9581	0.8977	0.9577	0.9577	0.8942	0.9572	0.9447	0.8740	0.9442

. Overall, SegNet showed the lowest extraction accuracy, followed by U-Net, while SD-UNet achieved the highest extraction accuracy. In the same network, the model trained on the Fake sample set showed higher extraction accuracy than other models trained on the NDVI sample set and True sample set. In conclusion, the SD-UNet trained on the Fake sample set (best model) achieved the highest results: ACC (0.9581), IOU (0.8977), and Recall (0.9577).

3.2.2. SD-UNet vs. RF, NDVI

To further intuitively evaluate the best model, it was compared with traditional methods in accuracy, as shown in

Table 3. Accuracy comparison of the best model and traditional methods.

Method	ACC	IOU	Recall
NDVI	0.8312	0.5996	0.8124
RF	0.8903	0.6553	0.8734
The best model	0.9581	0.8977	0.9577

. The ACC of the best model achieved 0.9581, which was 0.0678 higher than the RF and 0.1269 higher than the NDVI, indicating that it showed better accuracy. The IOU of the best model achieved 0.8977, which was 0.2424 higher than the RF and 0.2981 higher than the NDVI, indicating that its extraction results had a higher overlap with the labels. The Recall of the best model achieved 0.9577, which was 0.0843 higher than the RF and 0.1453 higher than the NDVI, indicating that it had a stronger ability to correctly identify targets.

3.3. Scene Application

To further research the generalization ability of the SD-UNet in practical application scenes, four images with dimensions of 1000 × 1000 pixels were selected. The four images represent different urban scenes, respectively, namely the high-density buildings scene, cloud and misty conditions scene, park scene, and suburban scene, as shown in Figure 10. It can be seen that the extraction results of the SD-UNet trained on different sample sets are significantly different. (1) is a scene with high-density buildings. In this scene, the vegetation extraction results of SD-UNet trained on the True sample set generally showed interference from buildings, and many buildings were misclassified as vegetation (red box). In contrast, the SD-UNet trained on the Fake sample set and the NDVI sample set had better performance. (2) is a scene with cloud and misty conditions. In this scene, the differences between the extraction results were very obvious. Especially for the vegetation along the river banks (red box), the extraction results of the SD-UNet trained on the Fake sample set were obviously better than SD-UNet trained on the other two sample sets. This is because vegetation has high reflectivity and water has strong absorption in the near-infrared spectrum, so adding the NIR to the sample set not only improves the SD-UNet’s anti-interference to water but also enhances the differences between vegetation and background, thereby reducing misclassifications. (3) is the park scene, which has the highest vegetation extraction accuracy among the four images. The main drawback of the vegetation results extracted by SD-UNet trained on the True sample set was the blurred vegetation boundaries (red box) and poor integrity when extracting large-area vegetation (yellow box). The vegetation extraction results of the SD-UNet trained on the Fake sample set and NDVI sample set were similar, but the difference was the omission of fine vegetation (green box). The vegetation extraction result of SD-UNet trained on the NDVI sample set had a higher omission rate. (4) is a suburban scene, which is the most difficult area from which to extract vegetation. In this scene, the vegetation is too fragmented. It can be seen that the extraction results of the SD-UNet trained on the True sample set had many misclassifications (red box), while the extraction results of the SD-UNet trained on the NDVI sample set had many omissions (yellow box). In contrast, the extraction results of the SD-UNet trained on the Fake sample set were closer to the labels. Moreover, the OA and kappa of the three models were shown to more intuitively compare their performance, as shown in

Table 4. Accuracy comparison of SD-UNet trained on different sample sets in four scene images.

Scenes	Fake Sample Set				NDVI Sample Set				True Sample Set
	Clouds and Misty	Building	Suburb	Park	Clouds and Misty	Building	Suburb	Park	Clouds and Misty	Building	Suburb	Park
OA	0.9256	0.9301	0.9214	0.9392	0.9085	0.9210	0.8605	0.9286	0.8238	0.8821	0.8817	0.8823
KAPPA	0.8631	0.8750	0.8832	0.8893	0.8055	0.8574	0.8102	0.8565	0.6589	0.7635	0.8325	0.7645

. The SD-UNet trained on the True sample set performed the worst, the SD-UNet trained on the NDVI sample set outperformed it, and the SD-UNet trained on the Fake sample set was the best model.

3.4. Qualitative Evaluation of Administrative Divisions

To further study the SD-UNet’s applicability to regions of different scales and complexities, the qualitative analysis is carried out in this section by using the best model from Section 3.3. For more contrast, Yuzhong District and Nan’an District were selected for vegetation extraction. Both districts belong to the main urban area of Chongqing. Yuzhong District is located in the center of the main urban area, covering an area of 23.24 km². Nan’an District is adjacent to Yuzhong District, covering an area of 262.43 km². Both districts are characterized by high temperatures, limited sunshine, a long lengthy rainy season, high humidity, and the frequent presence of clouds and mist. From the image selection perspective, two districts contain the four scenes. The Yuzhong District’s geography is particularly complex, with dense and scattered buildings, undulating terrain, and extremely fragmented vegetation. In contrast, the distribution of vegetation in Nan’an District is relatively good. Although the scene near the residential buildings is complicated, the vegetation in most areas is relatively complete, and the terrain is relatively flat. From the validation perspective, the urban vegetation extraction results of two districts can be evaluated visually by using the available ground truth. Due to time and human resource limitations, large-scale ground truths are lacking to evaluate accuracy, the accuracy metrics cannot be calculated. The ground truth is obtained by manual visual interpretation, so it has a certain human error. In addition, to better demonstrate the advantages of the best model, the vegetation extraction results of the best model and deep learning (DL) framework [66] are compared. The DL framework enables large-scale and general urban vegetation extraction. Its vegetation extraction results can be downloaded from https://doi.org/10.57760/sciencedb.07049, accessed on 10 September 2023. The visualization comparisons in the two districts are shown in Figure 11 and Figure 12.

From the overview images of Yuzhong Districts, it can be seen that the vegetation extraction results were relatively good overall. Specifically, the best model performed well in the extraction of ultra-fine vegetation alongside roadsides, riverbanks, and building gaps, although they were complex and broken in morphology. In contrast, the vegetation extraction results of the DL framework in these aspects were slightly insufficient. This may be because the DL framework is more suitable for large-scale and general vegetation extraction, so it has some limitations in ultra-fine vegetation. Similarly, the vegetation extraction results of the best model also have some shortcomings. For example, the urban vegetation extraction results of the best model are mainly influenced by the building shadows. As shown in Figure 11f, the urban vegetation features were blocked by building shadows, resulting in a relatively poor extraction effect. However, according to Figure 11e, at the same location in the image, it can be seen that the building shadows were extracted as vegetation and the vegetation was misclassified as non-vegetation. This may be caused by the different angles of the images taken, and Shi’s article also validated this. Additionally, the vegetation extraction results were also affected by the vegetation sparseness. When an area is covered by sparser vegetation, it is more easily misclassified due to its similar appearance to other land types. From the overview images of Nan’an Districts, it can be seen that the vegetation extraction results were generally average. Compared to the Yuzhong District, its results are slightly inferior, especially the vegetation near dense buildings. There may be three reasons for this. Firstly, the area of Nan’an District is relatively large and the vegetation distribution is uneven, so some details may be ignored when the model captures the overall vegetation distribution. Secondly, the increase in the diversity of ground features can make it difficult for the model to accurately distinguish vegetation and non-vegetation. Finally, it can be seen from Figure 12c,f that the Nan’an District’s vegetation extraction results are also affected by building shadows and vegetation sparseness. However, the vegetation extraction results of the best model also have many advantages. For example, the best model can extract large-area vegetation more completely without lake interference. When the buildings are arranged regularly, the vegetation between the building gaps can also be well extracted. In summary, despite the possible deficiencies, the best model still has a strong generalization ability.

4. Discussion

Currently, the combination of high-resolution remote sensing images and deep learning methods is widely used in semantic segmentation [67] and change detection [68]. High-resolution remote sensing images have many advantages, including providing rich geospatial information, enhancing classification accuracy, alleviating the problem of “the different objects with the same spectrum” and “the same object with different spectrum”, and supporting fine-grained classification. Especially in complex scenes, high-resolution images can provide more detailed geospatial features and contextual information for finer feature classification and detection. However, traditional methods selecting features artificially are not suitable for massive data and complex feature details. In contrast, deep learning networks have become the preferred choice. Deep learning networks exhibit stronger learning capabilities and higher levels of automation, enabling automatic extraction and analysis of various features from high-resolution images, such as vegetation, buildings, and roads. In addition, deep learning models have better generalization capabilities on new scenes. However, existing networks still have some problems in extracting vegetation, such as blurred vegetation edges, misclassification, and omissions of ultra-fine vegetation. To solve these problems, SD-UNet was proposed to improve the accuracy of urban vegetation extraction. Considering the spectral features of vegetation in the NIR and NDVI, two different spectral sample sets were additionally established. In this section, the following three aspects are discussed. Firstly, the sample sets with NIR and NDVI are compared with the True sample set, and the impact of NIR, sample sets quality, and quantity on the model’s vegetation extraction accuracy are discussed. Secondly, the advantages of SD-UNet’s structure are discussed in detail. Finally, SD-UNet’s generalization ability in four scenes and two administrative divisions is discussed.

4.1. Sample Set Evaluation

The NIR band makes the vegetation extraction model perform better, which is similar to Huerta’s research results using ResNet34 to extract urban vegetation information [55]. Moreover, Ayhan’s experiments also confirm this [69]. In the comparison of deep learning methods, the CNN model trained on the Vasiliko dataset (RGB and NIR bands) exhibited the best vegetation extraction results, followed by the DeeplabV3+ trained on the Vasiliko dataset (NDVI-GB). The CNN model and DeeplabV3+ trained on the Vasiliko dataset (RGB only) performed the worst. However, in previous deep learning studies, about half of the researchers only used the R, G, and B band information from remote sensing images to establish sample sets. Few researchers utilized the electromagnetic radiation features of vegetation in the NIR. Similarly, there were also fewer comparisons between the sample set with NIR and the True sample set (RGB) in the vegetation extraction results. In order to study the effect of NIR on SD-UNet’s vegetation extraction results, a Fake sample set, an NDVI sample set, and a True sample set were established. Training three deep networks on three sample sets, nine urban vegetation extraction models were obtained. Then, the validation images were used to extract urban vegetation, and the extraction results and accuracy of nine models were obtained. Through the previous analysis, it has been proven that the models trained on the Fake sample set and NDVI sample set can improve the urban vegetation extraction accuracy. This is because the spectral feature sample set with NIR and NDVI can provide a more comprehensive feature representation, which can enhance models’ identification and extraction capabilities for vegetation [70,71]. Moreover, the vegetation extraction results of the model trained on the Fake sample set were slightly superior to the model trained on the NDVI sample set under the same network and research area. This is because the reflectance difference between the NIR and R band was lower than normal, making NDVI unable to reach the threshold value for identifying vegetation. As a result, omissions were produced in the process of vegetation extraction. In general, the sample sets with different spectral features have many potential possibilities for the application. In the future, more detailed vegetation spectral features can be utilized to train deep learning networks, thereby achieving the classification of different vegetation types and monitoring vegetation changes (seasonal changes, degradation, etc.).

In addition, the quality and quantity of the sample set is also an important influencing factor for vegetation extraction results. Training different quality sample sets will lead to the SD-UNet’s accuracy difference. Deep learning is a big data-driven approach, so sample sets are particularly important for the extraction accuracy of deep learning models. The vegetation sample set established by manual visual interpretation inevitably has certain human errors, so it is impossible to completely guarantee that the ground truths fit the labels. However, the labels were delineated on the Gaofen-1 remote sensing images by using ArcGIS and referring to the high-precision Google images. This step ensures the quality of the sample set to a certain extent. High-quality samples can provide accurate label and feature information, while an adequate number of samples can help the model learn and generalize more effectively. For better learning the statistical characteristics and patterns of the data, numerous samples are required to train deep learning networks. Therefore, the sample data were augmented by geometric transformation, adding noise, and fuzzy transformations, and 2448 training samples and 612 validation samples were finally obtained.

4.2. SD-UNet’s Superiority in the Structure

The optimized SD-UNet is beneficial for urban vegetation extraction. It is an optimized version of U-Net, with several enhancements. Firstly, the SD-UNet introduces additional convolutions at each level of the U-Net’s encoder, capturing deeper semantic information and enhancing the accuracy, continuity, and integrity of vegetation extraction results. Secondly, dense connections are introduced at the input side of each encoder layer. By directly connecting all layers, multiscale feature reuse can be strengthened and the information flow between layers can be maximized [72,73]. Therefore, the SD-UNet can preserve vegetation details from low levels as much as possible and reduce the omissions in ultra-fine vegetation. Thirdly, the separable convolutions are reasonably arranged. Although adding traditional convolutional layers can capture deeper semantic information, a large number of parameters are generated [74]. To further reduce the computation cost of the networks, some traditional convolutions are replaced by separable convolutions [75,76]. Fourthly, a BN layer is introduced after each convolutional layer. The BN layer can improve the network training speed [77], prevent over-fitting [78], and enhance the stability of the model [79]. Therefore, adding BN can not only save SD-UNet’s training time and resources but also improve the accuracy and stability of the urban vegetation extraction model. Finally, the activation function Relu is replaced with Tanh. Experimental findings indicated that, compared to the Relu activation function, the Tanh activation function could reduce loss in SD-UNet and accelerate convergence.

4.3. SD-UNet’s Applicability

To verify the generalization ability of SD-UNet in practical applications, four 1000 × 1000 pixels scene images that do not participate in the construction of urban vegetation sample sets were selected to extract vegetation. Four images represent different urban scenes: high-density buildings, cloud and misty conditions, park, and suburban scenes. To more intuitively compare the urban vegetation extraction results, overall accuracy (OA) and Kappa were used to quantify the accuracy. The experimental results show that the best model achieved the highest vegetation extraction accuracy in the same scene. In other words, the sample set with NIR can avoid interference from other ground objects. In addition, the results and analysis in the preceding sections also indicate that the vegetation extraction accuracy was the highest in the park scene and the lowest in the suburban scene and the results’ contrast was the most obvious in the cloud and misty conditions scene. In the park, the vegetation is dense and has a high reflection in NIR, so the extensive vegetation areas can be more completely and accurately extracted. In contrast, the relationship among ground subjects in a suburban scene is highly complex, and the spectral difference between vegetation and non-vegetation is small, so the precision of vegetation extraction is the lowest. At the same time, the suburban environment varies from one city to another. The suburban scene only represents partial suburban features and lacks strict representativeness compared with the other three scenes. Therefore, the extraction results and conclusions about the suburban scene only have a certain reference value. In the cloudy and misty conditions scene, visible light images are significantly impacted. The cloud and mist can blur vegetation details and reduce the contrast between vegetation and non-vegetation. However, NIR with higher vegetation reflectance can penetrate clouds, so images with NIR can provide more vegetation features. In this case, SD-UNet trained on the Fake sample set can learn these vegetation features to distinguish vegetation and non-vegetation in the cloud and misty condition scene.

While the previous evaluations and comparisons have proven the best model’s advantages and validity in vegetation extraction, its potential utility is not very convincing due to the smaller scene images compared to the study area. Therefore, the vegetation in Yuzhong District and Nan’an District was extracted to further display the best model’s performance. Since the lack of large-scale and complete urban vegetation ground truth, a visual comparison was made. The experimental results showed that the vegetation extraction results in the Yuzhong District were good, while the vegetation extraction results in the Nan’an District were generally average. Considering the difference in complexity and area between the two regions, the following suggestion is made: To efficiently and accurately obtain urban vegetation in a large area (with complex areas and general areas), the vegetation area can be divided into several small areas according to the complexity. The best model is used to extract vegetation in complex regions, and other methods (such as the DL framework [66]) are used to extract vegetation in general regions. However, even though the best model has proven to be practicable for complex vegetation extraction, it still has to be pointed out that there may be a small number of missing labels in the sample set, especially for the vegetation covered by building shadows. As analyzed above, building shadows easily affect the model’s vegetation extraction performance. This also affects the establishment of labels. Fortunately, despite the possible deficiencies, the best model can still learn different vegetation features from a large number of accurate labels due to its strong generalization ability.

5. Conclusions

To effectively solve the problem of misclassification, ultra-fine vegetation omissions, and heavy computational burden, a new convolutional neural network SD-UNet is proposed to extract urban vegetation from Gaofen-1 remote sensing imagery. At the same time, three sample sets are established to evaluate the impact of the NIR on the model’s vegetation extraction accuracy, namely the Fake sample set, the NDVI sample set, and the True sample set. SD-UNet is also compared with U-Net, SegNet, NDVI, and RF on the sample sets created in Section 2.3. To ensure the fairness and reliability of the experiment, the same training data and parameters are used. Additionally, three metrics are selected to intuitively evaluate the vegetation extraction performance of the model, including ACC, IOU, and Recall. The experimental results show that the Fake sample set can effectively improve the accuracy of urban vegetation extraction results. The SD-UNet can achieve the highest performance; the ACC, IOU, and Recall of the SD-UNet trained on the Fake sample set reached 0.9581, 0.8977, and 0.9577, respectively. Based on the accuracy evaluation results, it can be concluded that the SD-UNet trained on the Fake sample set is the best model to extract urban vegetation. Finally, the SD-UNet trained on three sample sets are applied to four scenes to evaluate its generalization ability and transferability. To make the result more convincing, the best model is applied to two administrative divisions to extract vegetation. The result shows that the best model is suitable for small-scale vegetation extraction. It also shows that the research in this article can provide decision-making support for the sustainable development of the urban environment.

Although the SD-UNet can achieve satisfactory extraction results, limitations still exist. Firstly, the SD-UNet still produces omissions for ultra-fine vegetation among other ground subjects. The reason is probably that the ultra-fine vegetation information is continuously lost in the pooling process of down-sampling with the increasing network depth, so it cannot be correctly judged and identified. Secondly, the vegetation covered by building shadows also cannot be accurately identified and extracted. These problems need to be further improved in future research.