Building Polygon Extraction from High-Resolution Remote Sensing Imagery Using Knowledge Distillation

Abstract

Building polygons plays an important role in urban management. Although leveraging deep learning techniques for building polygon extraction offers advantages, the models heavily rely on a large number of training samples to achieve good generalization performance. In scenarios with small training samples, the models struggle to effectively represent diverse building structures and handle the complexity introduced by the background. A common approach to enhance feature representation is fine-tuning a pre-trained model on a large dataset specific to the task. However, the fine-tuning process tends to overfit the model to the task area samples, leading to the loss of generalization knowledge from the large dataset. To address this challenge and enable the model to inherit the generalization knowledge from the large dataset while learning the characteristics of the task area samples, this paper proposes a knowledge distillation-based framework called Building Polygon Distillation Network (BPDNet). The teacher network of BPDNet is trained on a large building polygon dataset containing diverse building samples. The student network was trained on a small number of available samples from the target area to learn the characteristics of the task area samples. The teacher network provides guidance during the training of the student network, enabling it to learn under the supervision of generalization knowledge. Moreover, to improve the extraction of buildings against the backdrop of a complex urban context, characterized by fuzziness, irregularity, and connectivity issues, BPDNet employs the Dice Loss, which focuses attention on building boundaries. The experimental results demonstrated that BPDNet effectively addresses the problem of limited generalization by integrating the generalization knowledge from the large dataset with the characteristics of the task area samples. It accurately identifies building polygons with diverse structures and alleviates boundary fuzziness and connectivity issues.

1. Introduction

Buildings are essential components of cities and serve as the primary places for residential and commercial activities [1]. Building polygons refer to the vector line information representing the planar outline of buildings when viewed from an overhead perspective. They play a crucial role in various fields such as urban planning [2], smart cities [3], 3D modeling [4], and disaster assessment [5]. Therefore, there is significant interest in extracting building polygon information rapidly and accurately.

In general, the process of extracting the polygon of a building is to first extract the raster mask, then use the texture, shape, and structure to design conversion rules, and finally convert the raster mask to a vector polygon within the constraints of the rules [6]. For example, firstly, remote sensing imagery is initially interpreted manually or through semi-automatic methods to extract the raster mask [7]. Secondly, mathematical morphology [8], edge detection operators such as Canny and Sobel [9,10] and Hough transform [11] are employed to extract building features, which are used to design rules for the transformation process. Finally, a rule set is used to determine the boundary positions of building raster masks and convert them into vector polygons. However, these methods can only extract accurate building polygons when predefined rules are satisfied. Besides, the methods are complex, resource-intensive, and have high labor costs, resulting in low automation levels [12].

In recent years, deep learning techniques have been widely applied in the field of remote sensing and have achieved remarkable results in building extraction [13]. For instance, using high-resolution remote sensing imagery, accurate building regions can be extracted using techniques such as semantic segmentation [14,15,16], object detection [17,18,19], and instance segmentation [20,21]. Building polygon extraction techniques have also gradually shifted towards deep learning-based methods. The basic workflow of such methods involves first obtaining a raster mask for the building class using deep learning-based semantic segmentation techniques and then vectorizing the raster results. Furthermore, researchers have integrated strategies such as multi-scale information [22], attention mechanisms [23], and multi-feature fusion [24] to extract more precise and effective features from high-resolution imagery to obtain more accurate raster masks. In the vectorization process, some studies have incorporated building corner and edge information to optimize building boundaries [25,26] or used Frame-Field to align the building segmentation maps for better edge representation [27,28]. To achieve end-to-end automatic building polygon extraction, some methods adopt the idea of multi-task learning by combining building segmentation with the vectorization process [29]. Certain approaches first detect the bounding boxes of building objects and then use LSTM to predict the building corners [30,31]. Additionally, some researchers treat building polygons as graphs [32], detecting building vertices and computing their adjacent connections to obtain polygonal contours.

However, deep learning methods require a large number of well-annotated labeled samples for support [33], and obtaining building polygon labels through manual annotation incurs significant labeling costs [34]. In practical applications, government agencies can only provide a small number of building polygon-labeled samples due to data confidentiality requirements. Training with a small number of samples leads to insufficient model generalization, making it difficult to handle the challenges of extracting diverse building polygons against complex backgrounds [35]. It is worth noting that some large and high-quality building extraction datasets are publicly available [26,36,37]. These datasets include building samples from different regions, encompassing various building styles, structures, and diverse backgrounds. Therefore, it is possible to extract generalization knowledge from these abundant and diverse building samples. However, directly applying models trained on other datasets to predict the target area may not yield satisfactory results.

To account for the building characteristics in the target area, it is possible to incorporate samples from the target area during the training process. A commonly used approach in current research is to first pre-train the model on publicly available datasets and then fine-tune the model on samples from the target area [38,39,40,41]. While this approach allows the model to learn the specific characteristics of buildings in the target area, it does not effectively inherit the generalization knowledge learned from the pretrained model on public datasets. Knowledge distillation techniques [42,43] offer a solution by pretraining a teacher neural network on a large dataset. During the training process, the knowledge learned by the teacher network is utilized to guide the student neural network, thereby improving the accuracy of the student network. For example, Wang et al. addressed the problem of insufficient labels in remote sensing scene classification by employing self-distillation to learn complex scene knowledge in diverse backgrounds [44]. Nabi et al. used knowledge distillation to extract visual knowledge in complex urban scenes [45]. By applying knowledge distillation, the model not only retains the generalization knowledge learned from the public datasets but also leverages the supervision from the limited samples in the target area to enhance the model’s adaptability to diverse types of buildings.

Therefore, this paper proposes the Building Polygon Distillation Net (BPDNet) to accurately extract building polygons in the target area with limited samples. The network consists of two structurally identical sub-networks known as the teacher network and the student network. The teacher network is trained on a large building polygon dataset that contains structurally diverse building samples, while the student network is trained on a small number of available samples from the target area. During the training process of the student network, the teacher network provides guidance, allowing the student network to access generalization knowledge. Furthermore, to address the issue of fuzzy, irregular, and fragmented building extraction caused by complex urban backgrounds, we employed the Dice Loss [46] to encourage the model to focus more on building boundaries. This helps alleviate the problem of building fragmentation caused by blurry boundaries.

The primary contributions of this work can be summarized as follows:

• We propose BPDNet for extracting building polygons in the area with limited high-resolution remote sensing image samples. This network enables the model to inherit the generalization knowledge from the large dataset via knowledge distillation;
• BPDNet employs Dice Loss to improve the extraction of buildings against the backdrop of complex urban context, characterized by fuzziness, irregularity, and connectivity issues.

The paper is structured as follows: Section 1 introduces the background to the methodology and the problem addressed in this paper; Section 2 describes the study area, the external dataset used, and the process and methods used to process the data; Section 3 presents the details of the methodology; Section 4 presents the experimental design, parameter settings, and analysis of the results; Section 5 provides a discussion of the methodology; and, finally, the conclusions are presented in Section 6.

2. Study Area and Data

2.1. Remote Sensing Images and Building Vector Data

Wenzhou, located in Zhejiang Province, China, is an important coastal commercial and regional center in southeastern China. In recent years, the urban area of Wenzhou has been continuously expanding. The intelligent and automated extraction of building polygons will effectively enhance the efficiency of building supervision. The remote sensing imagery and building vector data used in this study are derived from six regions within the urban area of Wenzhou, captured by unmanned aerial vehicles. Each region’s imagery has a size of 20,480 × 20,480 pixels with a spatial resolution of 0.2 m, including RGB bands. The study area and the corresponding imagery are shown in Figure 1a. In the high-resolution remote sensing imagery, various buildings with different shapes, arrangements, and heights can be observed in the six regions, as shown in Figure 1b. Moreover, the imagery exhibits challenges in extracting building polygons due to significant issues such as building tilts and shadow occlusions, which can cause confusion between building rooftops, facades, and the background.

In the data pre-processing stage, we cropped six remote sensing images into 2400 patches of 1024 × 1024 size and randomly divided the training and test sets according to the ratio of 7:3. The building contour label data is the building vector data corresponding to the image range, as shown in the sample in Figure 2.

2.2. WHU-Mix Dataset

In response to the lack of data diversity and poor label quality of current building datasets, the WHU-Mix dataset [47] collects images and polygon vector labels of over 754k buildings from around the world. The training set of this dataset contains 43,727 images, and the test set contains 8402 images from another five cities on five continents. The WHU-Mix dataset integrates data from the WHU dataset [37], Crowd AI [48], Open AI [49], SpaceNet [50], and Inria [36] datasets, with manual corrections for offsets and missing data. Sample data from the dataset are shown in Figure 3. To better capture the diversity of real-world scenarios and consider the wide range of building variations, we trained the teacher model based on the WHU-Mix dataset. This allowed us to transfer the knowledge of generalization to the target study area effectively.

3. Methods

3.1. Overall Architecture

Figure 4 illustrates the overall architecture of BPDNet, which consists of two structurally identical networks: the teacher network and the student network. The teacher network is trained on the WHU-Mix dataset for building polygon extraction. Once the teacher network is trained and reaches convergence, its model parameters are frozen. The student network is trained on the training set constructed from six images of the Wenzhou area and distilled using the trained teacher network. Therefore, the student model is capable of integrating the generalization knowledge from the large dataset and the feature distribution knowledge of the task area samples to address the issue of insufficient generalization of the building polygon extraction model caused by a small number of samples in the task area.

The teacher network and the student network are both constructed based on HiSup [49]. HiSup serves as the baseline model for our method, consisting of the HRNet [51] feature encoder, Channel Attention module, and three building representation branches. Details of HiSup and our improvements to it are described in Section 3.2. The teacher network was trained on the WHU-Mix dataset, so during the training of the student network, the parameters of the teacher network are no longer updated to guide the training of the student network. In the training process of the student network, the remote sensing images from the task area are input to both the student network and the teacher network with frozen parameters. The forward propagation data flow of the network is shown by the solid green arrows in Figure 4. The gradients of the network parameters are backpropagated through the purple dashed arrows in Figure 4. The loss of the student network consists of two parts: distillation loss and building polygon prediction loss. The distillation loss represents the difference between the output features of the student network and the teacher network in the three building representation branches, encouraging the student model to learn generalization knowledge from the teacher model and improve its ability to extract building polygons with diverse structures. The building polygon prediction loss represents the difference between the predicted building polygons by the student network and the ground truth building polygon labels, guiding the student model to learn the feature distribution knowledge of the data in the task area and enhance its accuracy in extracting building polygons in the task area.

3.2. Baseline Model: HiSup

This paper adopts HiSup as the baseline model for extracting building polygons from remote sensing images. The model structure is illustrated in Figure 5. HiSup consists of three feature learning branches, including the Mask Branch, Line Branch, and Vertex Branch. Additionally, HiSup incorporates modules for multi-scale feature extraction, channel attention, boundary enhancement, and polygon construction and simplification. Specifically, for an input remote sensing image $I\in R^{3\times H\times W}$, HRNet is employed as the multi-scale feature extractor to obtain the feature map $F\in R^{3\times H_S\times W_S}$. Here, $H_S=H/S$ represents the length of feature map F, and $W_S=W/S$ represents its width, with S being the downsampling factor. After obtaining the image feature map F through the multi-scale feature extractor, F is fed into three distinct branches to learn the vertex feature $F_{ver}$, line feature $F_{line}$, and semantic mask feature $F_{seg}$ of buildings. Each branch contains three consecutive processing units, including 3 × 3 convolutional layers, a batch normalization layer [52], and a ReLU [53] layer in each processing unit. Specifically, the vertex feature $F_{ver}$ and line feature $F_{line}$ are derived from mid-level image features, while the semantic mask feature $F_{seg}$ is obtained from high-level image features. Subsequently, the channel attention module integrates these three features to generate the vertex prediction heat map $M_H$, vertex distance offset field $M_O$, line attraction field [29] prediction map $M_A$, and building a semantic mask prediction map $M_S$. Next, by utilizing the line attraction field $M_A$, and image feature F to impose boundary constraints on $M_S$, a more regular segmentation result is obtained. Finally, the vertex, line, and semantic prediction results are utilized to construct vector polygons, which are further simplified to obtain the final building polygon extraction results.

To further reduce the influence of complex backgrounds on building boundaries, we improved HiSup. In the boundary enhancement module, Dice Loss was incorporated to alleviate the issue of building adhesion. This modification aims to improve the segmentation performance by effectively separating buildings from their surrounding environment, as shown in Figure 5.

3.2.1. Channel Attention Module

HiSup utilizes the channel attention mechanism [54] to enhance the feature representation of vertex maps $F_{ver}$ and semantic features $F_{seg}$, thereby improving the accuracy and consistency of building shape prediction. The computation method is as follows:

$$F_{seg}^e=\delta \left(C1D\left(GAP\left(F_{line}+F_{seg}\right)\right)\right)\times F_{seg}+F_{seg}$$

(1)

$$F_{ver}^e=\delta \left(C1D\left(GAP\left(F_{line}+F_{ver}\right)\right)\right)\times F_{ver}+F_{ver}$$

(2)

where $C1D(·)$ represents one-dimensional convolution, $GAP(·)$ represents global average pooling, and $\delta (·)$ represents the sigmoid activation function.

3.2.2. Boundary Enhancement

Due to the widespread irregularity and fuzzy boundaries of building shapes in the semantic mask results, HiSup employs the boundary enhancement module to utilize the line attraction field results $M_A$ to constrain the semantic mask results $M_S$, thereby making the building shapes in the semantic mask more regular and the boundaries clearer. Specifically, firstly, $M_A$ is concatenated with the feature map F extracted by the backbone network to obtain the fused feature map $M_{AF}$, incorporating the line constraint information into F. Then, $M_{AF}$ is fed into the semantic mask classification head to predict the semantic mask image $M_{AS}$. Subsequently, the semantic mask loss function is employed to calculate the discrepancy between $M_{AS}$ and the semantic mask labels. Finally, through backpropagation, $M_A$ receives additional supervision from the semantic mask labels, enabling it to have a better representation. After further gradient backpropagation, the backbone network enhances the expression of building shapes in F to facilitate the generation of $M_A$. Consequently, the backbone network learns the correct building shapes guided by $M_A$.

To further reduce the merging of buildings and confusion with the background, we improved the loss function of HiSup. Dice Loss [46] was added to the binary cross-entropy-based segmentation loss $L_{seg}$ to allow the model to focus more on the boundaries of building targets and reduce the adhesion between building polygons. The segmentation loss $L_{seg}$ of the model can be represented as follows:

$$L_{seg}\left(\theta \right)=BCE\left(M_{AS},M_S^{GT}\right)+BCE\left(M_S,M_S^{GT}\right)+\theta L_{dice}$$

(3)

where BCE represents the binary cross-entropy loss, $M_{AS}$ represents the semantic mask result guided by the line features, $M_S^{GT}$ represents the semantic mask label for the building, $L_{dice}$ represents the Dice Loss, $\theta$ is the coefficient. Dice Loss can be represented as follows:

$$L_{dice}=1-\frac{2TP}{2TP+FP+FN}$$

(4)

where TP represents the number of true positive pixels, FP represents the number of false positive pixels, and FN represents the number of false negative pixels.

3.3. Polygon Construction and Simplification

After extracting building features, it is necessary to construct the polygonal outlines of the buildings using post-processing methods. First, the semantic mask image $M_S$ is filtered using a given threshold value $\mathsf{\epsilon }\in \left(0,1\right)\left(\mathrm{reference} \mathrm{HiSup} \mathrm{sets} \mathsf{\epsilon }=0.6\right)$ to obtain the resulting image S, which contains n building polygons. Then, local non-maximum suppression is applied to filter out non-key vertices in the vertex prediction heatmap $M_H$, and based on S, the vertices located on the boundary pixels of building polygons are connected to construct the initial building outlines. Finally, the building outlines are simplified, with a focus on the vertices of the buildings. The rule for vertex simplification is as follows: if the distance between two vertices of the same building polygon is smaller than a threshold value $\mathsf{\tau }$ (reference HiSup sets $\mathsf{\tau }$ = 5), the midpoint of the line connecting the two points is taken as the simplified vertex, resulting in the final building polygon.

3.4. Knowledge Distillation

In this study, knowledge distillation was employed to learn generalization knowledge from the teacher network trained on a large dataset. The specific process of knowledge distillation is illustrated in Figure 6. The input images are passed through the HRNet feature extractors of both the teacher network and the student network, resulting in image features denoted as $F_{ }^T$ and $F_{ }^S$, respectively. For $F_{ }^T$ in the teacher network, it is separately inputted into the Mask Branch, Line Branch, and Vertex Branch. The feature $F_{ }^T$ in each branch undergoes three consecutive processing units, where each unit consists of a 3 × 3 convolutional layer, a batch normalization (BN) layer, and a rectified linear unit (ReLU) activation layer. The output of the last processing unit in each branch is, respectively, the semantic mask feature $F_{seg}^T$, the line feature $F_{line}^T$, and the vertex feature $F_{ver}^T$. Similarly in the student network, the feature $F^S$ is separately inputted into the Mask Branch, Line Branch, and Vertex Branch to obtain the semantic mask feature $F_{seg}^S$, the line feature $F_{line}^S$, and the vertex feature $F_{ver}^S$ in the student network. Due to the well-trained teacher network on the WHU-Mix dataset, it possesses the ability and knowledge to extract structurally diverse building polygons, including the features of building semantic masks, building lines, and building vertices. Therefore, we employed the CWD (Channel-wise Knowledge Distillation) [43] to distill these capabilities of the teacher network regarding building outlines into the student network. Taking the semantic mask feature as an example, we aligned the feature distributions of each channel in $F_{seg}^S$ from the student network with the corresponding channel in $F_{seg}^T$ from the teacher network. Similarly, both $F_{line}^S$ and $F_{ver}^S$ need to be aligned with $F_{line}^T$ and $F_{ver}^T$ in the same manner as described for $F_{seg}^S$. CWD, as it distills knowledge in the channel dimension, can effectively utilize the knowledge contained in each channel, making it suitable for dense prediction tasks such as building polygon prediction in this paper.

CWD minimizes the asymmetry Kullback–Leibler (KL) divergence between the channel-wise soft probability maps of the teacher and student networks to align the feature distributions of each channel. Therefore, the distillation loss can be defined as follows:

$$L_d\left(T\right)=\frac{T^2}{C}{{\displaystyle \sum }}_{c=1}^C{{\displaystyle \sum }}_{i=1}^{W·H}\varphi \left(F_{c,i}^T\right)·log\left[\frac{\varphi \left(F_{c,i}^T\right)}{\varphi \left(F_{c,i}^S\right)}\right]$$

(5)

where c = 1, 2, …, C indexes the channel; and i indexes the spatial location of a channel. W and H are the width and height of the feature map respectively. T is a hyperparameter. $F^T$ denotes the feature map from the teacher network and $F^S$ denotes the feature map from the student network. $\varphi (·)$ is used to convert the activation values into a probability distribution as below:

$$\varphi \left(T\right)=\frac{exp\left(\frac{F_{c,i}^{ }}{T}\right)}{{{\displaystyle \sum }}_{i=1}^{W·H}exp\left(\frac{F_{c,i}^{ }}{T}\right)}$$

(6)

where c = 1, 2, …, C indexes the channel; and i indexes the spatial location of a channel. W and H are the width and height of the feature map respectively. T is the temperature hyper-parameter. The probability becomes softer if we use a larger T, meaning that we focus on a wider spatial region for each channel.

3.5. Loss Function

The loss function of BPDNet consists of two parts: the building polygon prediction loss $L_p$ and the distillation loss $L_d$. The loss function for building polygon contour prediction is composed of three components: the semantic mask branch loss function $L_{seg}$, the line attraction field prediction branch loss function $L_{line}$, and the vertex prediction branch loss function $L_{ver}$. $L_{seg}$ as shown in Equation (3), and $L_{line}$ and $L_{ver}$ can be expressed as follows:

$$L_{line}=l_1\left(M_A,M_A^{GT}\right)$$

(7)

where $l_1$ represents the l-penalized loss, which corresponds to the absolute deviation loss. $M_A^{GT}$ refers to the line attraction field generated from the ground truth labels of the building polygon in the image.

$$L_{ver}\left(\alpha ,\beta \right)=\alpha BCE\left(M_H,M_H^{GT}\right)+\beta l_1\left(M_O·M_H^{GT},M_O^{GT}·M_H^{GT}\right)$$

(8)

where $\alpha$ and $\beta$ are coefficients, $M_H$ represents the predicted heat map for building vertices, $M_O$ corresponds to the short-distance offset field for vertices, and $M_H^{GT}$ and $M_O^{GT}$ refer to the ground truth heat map and short-distance offset field for vertices generated from the actual labels of the building polygon in the image.

The loss function for building polygon contour prediction can be expressed as follows:

$$L_p\left({\lambda }_1,{\lambda }_2,{\lambda }_3,\alpha ,\beta ,\theta \right)={\lambda }_1{L}_{seg}\left(\theta \right)+{\lambda }_2{L}_{line}+{\lambda }_3{L}_{ver}\left(\alpha ,\beta \right)$$

(9)

where ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, $\alpha$, $\beta$, and $\theta$ are coefficients.

The overall loss function consists of the building polygon contour prediction loss and the distillation loss and can be represented as follows:

$$L_{total}\left({\lambda }_1,{\lambda }_2,{\lambda }_3,\alpha ,\beta ,\theta ,\eta ,T\right)={\lambda }_1{L}_{seg}\left(\theta \right)+{\lambda }_2{L}_{line}+{\lambda }_3{L}_{ver}\left(\alpha ,\beta \right)+\eta L_d\left(T\right)$$

(10)

where ${\eta }_{ }$ represents the coefficient hyperparameter for $L_d$, and T represents the temperature hyperparameter.

4. Results

4.1. Experimental Design

To validate the effectiveness of the proposed BPDNet model, we conducted the following experiments. By comparing the results of each model on evaluation metrics, we were able to quantitatively analyze the strengths and weaknesses of each model in comparison to BPDNet. Additionally, we performed a visual analysis of the predicted results of each model to qualitatively assess the building extraction performance of each model.

(1)

Frame-field [27] is a building polygon extraction method proposed in 2020. It aligns the predicted frame field with the true outline. We implemented the model based on the code of the original paper. In our experiments, the frame-field was trained on a training set composed of samples from Wenzhou and was tested on a test set constructed from samples from Wenzhou;

(2)

Model_1 (Baseline) was trained on a training set composed of samples from Wenzhou using the HiSup model. It was then tested on a test set constructed from samples from Wenzhou. This serves as the baseline model for comparison;

(3)

Model_2 (w/o Finetune) uses the WHU-Mix dataset to train the HiSup model without using fine-tuning but directly on a test set constructed from the Wenzhou sample.

(4)

Model_3 (w/Fine-tune) was trained on the WHU-Mix dataset using the HiSup model. Subsequently, fine-tuning was performed on the training set constructed from samples from Wenzhou. Finally, Model_3 was tested on the test set composed of samples from Wenzhou;

(5)

Model_4 represents the proposed BPDNet method. It involves training the HiSup model on the WHU-Mix dataset, which serves as the teacher model. The CWD (Collaborative Weight Distillation) method was then employed to guide the training of the student model on the training set constructed from samples of Wenzhou. Finally, Model_4 is evaluated on the test set composed of samples from Wenzhou.

4.2. Evaluation Metrics

For instance, for segmentation tasks such as building contour extraction, it is important to evaluate not only pixel-level segmentation performance but also instance-level building extraction effectiveness. To quantify the performance of different models, we utilize the following metrics: Overall Accuracy (OA), IoU, Precision for IoU threshold > 0.5 called $P_{50}^{IoU}$, Precision for Boundary IoU [55] threshold > 0.5 called $P_{50}^{Boundary}$, Recall for IoU threshold > 0.5 called $R_{50}^{IoU}$ and Recall for Boundary IoU threshold > 0.5 called $R_{50}^{Boundary}$. Among the metrics used for evaluation, OA and IoU are employed to assess the pixel-level effectiveness of building extraction. $P_{50}^{IoU}$ is used to evaluate the precision of extracting building polygon instances, while $P_{50}^{Boundary}$ is utilized to evaluate the precision of the boundaries of the extracted building polygon instances. $R_{50}^{IoU}$ is used to evaluate the recall of extracting building polygon instances, while $R_{50}^{Boundary}$ is utilized to evaluate the recall of the boundaries of the extracted building polygon instances. The evaluation metrics are designed such that larger values indicate better performance of the models. The specific calculation methods for these evaluation metrics are as follows:

$$OA=\frac{TP+TN}{TP+TN+FP+FN}$$

(11)

where TP indicates the number of pixels predicted to be true positives, TN indicates the number of pixels predicted to be true negatives, FP indicates the number of pixels predicted to be false positives, and FN indicates the number of pixels predicted to be false negatives.

$$IoU\left(X,Y\right)=\frac{X{\displaystyle \cap }Y}{X{\displaystyle \cup }Y}$$

(12)

where X represents the predicted results, and Y represents the ground truth labels. IoU measures the intersection-to-union ratio between the predicted building polygon and the ground truth building polygon. A higher IoU value indicates a better match between the predicted building polygon and the ground truth, signifying more accurate building prediction results.

The Boundary IoU metric provides a reasonable measure to assess the accuracy of polygon boundaries. It helps to reveal the precision of the boundaries of the detected polygon instances in a meaningful way. For two polygon instances X and Y, Boundary IoU calculates the IoU only for pixels that are within a distance d of the boundaries of the two polygon instances. It focuses on evaluating the IoU specifically for the pixels that lie within a specified distance d from the boundaries of the polygon instances X and Y. Define the boundaries of X and Y as $X_d$ and $Y_d$. Similar to IoU, a larger value of Boundary IoU indicates better extraction performance. The calculation formula for Boundary IoU can be represented as follows:

$$Boundary IoU\left(X,Y\right)=\frac{\left|(X_d{\displaystyle \cap }X){\displaystyle \cap }(Y_d{\displaystyle \cap }Y)\right|}{\left|(X_d{\displaystyle \cup }X){\displaystyle \cup }(Y_d{\displaystyle \cup }Y)\right|}$$

(13)

The value of parameter d in the formula is set based on reference [55], where d = 0.2.

$$P_{50}^{IoU}=\frac{T{P}_{50}^{IoU} }{T{P}_{50}^{IoU}+F{P}_{50}^{IoU}}$$

(14)

where $T{P}_{50}^{IoU}$ represents the number of true positive pixels within instances where the IoU is greater than 0.5. $F{P}_{50}^{IoU}$ represents the number of false positive pixels within instances where the IoU is greater than 0.5.

$$P_{50}^{Boundary}=\frac{T{P}_{50}^{Boundary} }{T{P}_{50}^{Boundary}+F{P}_{50}^{Boundary}}$$

(15)

where $T{P}_{50}^{Boundary}$ represents the number of true positive pixels within instances where the Boundary IoU is greater than 0.5. $F{P}_{50}^{Boundary}$ represents the number of false positive pixels within instances where the Boundary IoU is greater than 0.5.

$$R_{50}^{IoU}=\frac{T{P}_{50}^{IoU} }{T{P}_{50}^{IoU}+F{N}_{50}^{IoU}}$$

(16)

where $T{P}_{50}^{IoU}$ represents the number of true positive pixels within instances where the IoU is greater than 0.5. $F{N}_{50}^{IoU}$ represents the number of false negative pixels within instances where the IoU is greater than 0.5.

$$R_{50}^{Boundary}=\frac{T{P}_{50}^{Boundary} }{T{P}_{50}^{Boundary}+F{N}_{50}^{Boundary}}$$

(17)

where $T{P}_{50}^{Boundary}$ represents the number of true positive pixels within instances where the Boundary IoU is greater than 0.5. $F{N}_{50}^{Boundary}$ represents the number of false negative pixels within instances where the Boundary IoU is greater than 0.5.

4.3. Experimental Parameter Setting

In this experiment, the HRNet model used for feature extraction is HRNetV2-W48 [51]. As for the variables ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, $\alpha$, and $\beta$ in the loss function of BPDNet, we set their values based on the reference HiSup [49]: ${\lambda }_1=1.0$, ${\lambda }_2=0.1$, ${\lambda }_3=1.0$, $\alpha =8.0$, $\beta =0.25$. Furthermore, for the variables $\theta$, $\eta$, and $T$ in the loss function, we conducted experiments to determine their optimal values for achieving the best results with BPDNet. Based on the experimental results, we found that setting $\theta$ = 1.0, $\eta$ = 1.0, and $T$ = 1.0 yielded the best performance for BPDNet.

All experiments in this paper were conducted on an NVIDIA RTX3090 GPU. The batch size for the models was set to 2, and the training process consisted of 60 epochs. Random flipping augmentation was applied during the training of all methods. The AdamW optimizer was used for training, with an initial learning rate of 6 × 10⁻⁵, ${\beta }_1$ = 0.9, ${\beta }_2$ = 0.999, and a weight decay coefficient of 0.01. The learning rate decay strategy employed the poly strategy.

4.4. Experimental Results

4.4.1. Quantitative Analysis

The experimental results are presented in

Table 1. Accuracy results on the test set using different methods. “w/Fine-tune ” denotes that fine-tune was employed. “w/o Fine-tune” denotes that fine-tune was not employed.

Method	OA	IoU	${\mathit{P}}_{50}^{\mathit{I}\mathit{o}\mathit{U}}$	${\mathit{P}}_{50}^{\mathit{B}\mathit{o}\mathit{u}\mathit{n}\mathit{d}\mathit{a}\mathit{r}\mathit{y}}$	${\mathit{R}}_{50}^{\mathit{I}\mathit{o}\mathit{U}}$	${\mathit{R}}_{50}^{\mathit{B}\mathit{o}\mathit{u}\mathit{n}\mathit{d}\mathit{a}\mathit{r}\mathit{y}}$
Frame-field	88.43	59.84	50.17	43.25	54.24	47.11
Model_1 (Baseline)	90.88	62.37	52.50	45.71	56.89	49.34
Model_2 (w/o Fine-tune)	90.34	61.89	52.01	45.18	56.03	48.17
Model_3 (w/Fine-tune)	91.47	63.54	53.22	49.98	57.65	51.29
Model_4 (Ours)	92.28	66.54	56.01	54.19	60.18	57.35

. From the results, we can observe the following findings. Firstly, our proposed BPDNet method achieved improvements compared to the frame-field and HiSup. It was shown that BPDNet is superior to the current state-of-the-art building polygon extraction algorithms when facing the problem of limited training sample sizes. Secondly, compared with HiSup, BPDNet showed an increase of 1.40% in OA, 4.17% in IoU, 3.51% in $P_{50}^{IoU}$, 8.48% in $P_{50}^{Boundary}$, 3.29% in $R_{50}^{IoU}$ and 8.01% in $R_{50}^{Boundary}$. The significant improvement in $P_{50}^{Boundary}$ and $R_{50}^{Boundary}$ indicated that BPDNet reduces missed and false extraction of building polygons and enhances the accuracy of building polygon extraction, including the precise positioning of building boundaries. This improvement effectively mitigates issues related to incorrect building polygon localization and blurry boundaries caused by diverse building structures and complex backgrounds. Thirdly, Model_2 performed the worst among the four comparative models, as indicated by the lowest evaluation scores, where the OA was 90.34%, IoU was 61.89%, $P_{50}^{IoU}$ was 52.01%, $P_{50}^{Boundary}$ was 45.18%, $R_{50}^{IoU}$ was 56.03% and $R_{50}^{Boundary}$ was 48.17%. This result indicates that the significant difference in the sample distribution between WHU-Mix and the study area greatly affected the performance of Model_2. Directly applying a model trained on WHU-Mix to predict the test set from the study area shows that the generalization capability obtained from a large dataset is still insufficient to overcome the impact of sample distribution differences. To address this issue, it is necessary to enable the model to learn the distribution characteristics of the samples in the task area. Compared to Model_2, Model_3 incorporates fine-tuning on the training set from the study area, enabling the model to learn the distribution characteristics of the samples in the task area. Although Model_3 showed improved performance compared to both Model_1 and Model_2, its evaluation metrics were still lower than those of Model_4 (BPDNet). Model_4 (BPDNet) achieved an OA of 92.28%, IoU of 66.54%, $P_{50}^{IoU}$ of 56.01%, $P_{50}^{Boundary}$ of 54.19%, $R_{50}^{IoU}$ of 60.18%, $R_{50}^{Boundary}$ of 57.35%. This indicates that although fine-tuning allows the model to learn the distribution characteristics of the task-specific dataset and partially overcome the differences in sample distribution, it does not effectively preserve the generalization performance obtained from the large dataset training. Model_4 (BPDNet) effectively distills the building feature representation capability obtained from diverse structured building samples in the large dataset to the student model through the CWD method. This enables the student model to achieve the best prediction results on the test set of the study area. Additionally, the inclusion of Dice Loss in the mask branch of Model_4 (BPDNet) further optimizes the boundaries of buildings, allowing for better differentiation between buildings and complex backgrounds. Therefore, based on the aforementioned analysis, the BPDNet method not only improves the pixel-level extraction results of buildings but also enhances the accuracy of the shape and position of building polygons. It reduces missed and false extraction of building polygons and demonstrates superior generalization ability compared to other methods for extracting building polygons with few samples in the Wenzhou area.

4.4.2. Qualitative Analysis

To facilitate visual comparison of the results, we have provided visualizations of prediction outputs (as shown in Figure 7). From Figure 7, it is evident that Model_4 (BPDNet) exhibited more accurate positioning and boundary delineation of building polygons. BPDNet demonstrated fewer instances of missed detections compared to other methods, which is consistent with the findings of Table 1. In Row (a), Model_4 accurately identified the boundaries of the buildings within the red box, while other methods mistakenly recognized the background as buildings, resulting in inaccurate building polygon results. The Hisup-based methods (Model_1, Model_2, Model_3, Model_4) were able to recognize the complete building polygon in the yellow box, but the Frame-field model mistakenly identified the one building polygon as two parts in the yellow box. In Row (b), based on the imagery and labels in the second column, it can be observed that the building within the red box is a composite structure consisting of three high-rise buildings and a low-rise building. The extracted contour should only represent the entire complex, and only BPDNet and Model_3 successfully identified it as a single entity. Regarding the small building within the yellow box in Row (b), both BPDNet and Model_3 successfully detected it, while Frame-field, Model_1, and Model_2 overlooked this small building. In Row (c), Frame-field, Model_1, Model_2, and Model_3 exhibited varying degrees of issues with building polygon merging within the yellow box, where buildings are incorrectly connected or misclassified as part of the background. However, BPDNet accurately identified the boundaries of the buildings. In Row (d), only BPDNet recognized and precisely extracted the polygon of the small building within the red box, while other models failed to detect it. In the yellow box region, Model_2 mistakenly identified the water body as a building, indicating the impact of the difference in sample distribution between WHU-Mix and the study area on the accuracy of building recognition. In conclusion, the results of building polygon extraction demonstrated that BPDNet effectively tackles the challenges posed by diverse building structures and complex backgrounds. It accurately identifies the positions of building polygon and alleviates issues such as boundary blurring and merging.

5. Discussion

5.1. Ablation Experiments

To validate the effectiveness of the proposed method, we conducted ablation experiments by removing individual modules used in this study. The experimental results are shown in

Table 2. Ablation experiments.

Method	Distillation	Dice Loss	OA	IoU	${\mathit{P}}_{50}^{\mathit{I}\mathit{o}\mathit{U}}$	${\mathit{P}}_{50}^{\mathit{B}\mathit{o}\mathit{u}\mathit{n}\mathit{d}\mathit{a}\mathit{r}\mathit{y}}$	Batch Time	Params
Baseline			90.88	62.37	52.50	45.71	0.84s	74.29M
Ours	√		91.37	64.63	54.11	51.98	0.88s	74.29M
Ours	√	√	92.28	66.54	56.01	54.19	0.90s	74.29M

below.

From the experimental results, it can be observed that when using knowledge distillation alone, compared to the baseline, the OA increased by 0.49%, IoU increased by 2.26%, ${\mathrm{P}}_{50}^{\mathrm{IoU}}$ increased by 1.61%, and ${\mathrm{P}}_{50}^{\mathrm{Boundary}}$ increased by 6.27%. All metrics showed improvement, with ${\mathrm{P}}_{50}^{\mathrm{Boundary}}$ showing a significant increase. This indicates that by using knowledge distillation, the accuracy of building boundary extraction can be greatly improved. The model effectively learns generalization knowledge from the large dataset, enabling more precise identification of diverse building boundaries. When both knowledge distillation and Dice Loss were used together, the model achieved the best performance. Compared to using knowledge distillation alone, OA increases by 0.91%, IoU increased by 1.91%, ${\mathrm{P}}_{50}^{\mathrm{IoU}}$ increased by 1.90%, and ${\mathrm{P}}_{50}^{\mathrm{Boundary}}$ increased by 2.21%. All metrics showed stable growth. This indicates that the inclusion of Dice Loss effectively mitigates the problem of building boundary adhesion, resulting in more complete extraction of building polygon instances and more accurate boundaries.

Additionally, we calculated the GPU processing time per batch for each method during training and the number of parameters of each model. From Table 2, the Baseline achieved the least GPU processing time per batch. With the addition of distillation loss and Dice Loss, the GPU processing time for each batch increased. Compared to the baseline, the processing time per batch for the model with distillation loss and Dice Loss increased from 0.84 s to 0.90 s. We believe that this small increase in GPU processing time is acceptable due to the significant improvement in modeling results. Meanwhile, Table 2 shows that the number of parameters in each model is equal and the use of distillation loss and Dice Loss does not increase the complexity of the model.

5.2. Distillation Methods

In this section, we extensively investigate the impact of feature selection and various parameter settings on the experimental results during the knowledge distillation process, aiming to explore effective approaches for knowledge distillation.

5.2.1. Selection of Feature to Distillate

During the training of the student network guided by the teacher network, it is necessary to select distilled features. In this paper, the available features for selection include semantic mask features $F_{seg}^{ }$, line features $F_{line}^{ }$, and vertex features $F_{ver}^{ }$. We aimed to ensure that the student network learns effective feature representations, and thus conducted ablation experiments to investigate different feature combinations.

Table 3. Distillation feature ablation experiments. √ denotes the feature is selected for knowledge distillation.

Distillation Feature			OA	IoU	${\mathit{P}}_{50}^{\mathit{I}\mathit{o}\mathit{U}}$	${\mathit{P}}_{50}^{\mathit{B}\mathit{o}\mathit{u}\mathit{n}\mathit{d}\mathit{a}\mathit{r}\mathit{y}}$
${\mathit{F}}_{\mathit{s}\mathit{e}\mathit{g}}^{ }$	${\mathit{F}}_{\mathit{l}\mathit{i}\mathit{n}\mathit{e}}^{ }$	${\mathit{F}}_{\mathit{v}\mathit{e}\mathit{r}}^{ }$
√			91.33	64.43	54.01	52.93
	√		92.08	66.31	55.35	53.34
		√	92.04	66.26	54.92	53.86
√	√		92.13	66.32	55.62	53.65
√		√	92.16	66.37	55.56	54.12
	√	√	92.21	66.45	55.61	53.74
√	√	√	92.28	66.54	56.01	54.19

presents the experimental results of different feature combinations during the distillation process. The student network achieved the best learning performance when using the semantic mask feature $F_{seg}^{ }$, line feature $F_{line}^{ }$, and vertex feature $F_{ver}^{ }$ simultaneously. The results demonstrated that all three features have a positive impact on building polygon extraction. The combination of these three features outperformed any pairwise combination, indicating their complementary nature and collective contribution to building polygon extraction.

5.2.2. Distillation Temperature Hyperparameter

In knowledge distillation, the temperature hyperparameter T is used to control the smoothness of the SoftMax function’s output during the feature map normalization process. A higher value of T indicates a larger spatial region of interest for each feature channel and leads to smoother outputs. We conducted experiments with different distillation temperatures, aiming to explore the appropriate values for the hyperparameter T and determine the optimal focus scale for feature channels during distillation.

Table 4. Experimental results for different distillation temperatures T.

Distillation Temperature	OA	IoU	${\mathit{P}}_{50}^{\mathit{I}\mathit{o}\mathit{U}}$	${\mathit{P}}_{50}^{\mathit{B}\mathit{o}\mathit{u}\mathit{n}\mathit{d}\mathit{a}\mathit{r}\mathit{y}}$
$T=0.02$	91.25	63.21	53.86	51.84
$T=0.50$	88.76	60.13	47.31	45.05
$T=0.70$	92.05	65.97	54.96	52.81
$T=1.00$	92.28	66.54	56.01	54.19
$T=1.50$	90.85	62.76	52.24	50.96

presents the experimental results with different hyperparameter T values. The model achieved the best performance when T = 1.00. When T < 1, the probability distribution became sharper as T decreased. On the other hand, when T > 1, the probability distribution became smoother. When T = 1, the SoftMax probability distribution remained consistent with the original distribution. In the results of this section, maintaining the original probability distribution actually yielded better performance. Comparing T = 0.50 with T = 0.02 and T = 0.70, there was a significant decrease in performance, indicating that a smaller focus scale for feature channels made it difficult for the student network to learn effective generalization knowledge from the teacher network.

5.2.3. Weighting of Distillation Loss

The total loss function of our proposed method consists of two components: the building polygon prediction loss and the distillation loss. The weight of the distillation loss $\eta$ is a hyperparameter that controls the importance of the distillation process. In this study, we conducted experiments with different values of the distillation loss weight $\eta$ to explore the optimal value for this hyperparameter.

Table 5. Distillation loss weighting and experimental results.

Weight of Distillation Loss	OA	IoU	${\mathit{P}}_{50}^{\mathit{I}\mathit{o}\mathit{U}}$	${\mathit{P}}_{50}^{\mathit{B}\mathit{o}\mathit{u}\mathit{n}\mathit{d}\mathit{a}\mathit{r}\mathit{y}}$
$\eta =0.20$	92.12	66.30	55.13	53.16
$\eta =0.40$	92.17	66.41	55.25	53.47
$\eta =0.60$	92.18	66.43	55.24	53.01
$\eta =0.80$	92.21	66.47	55.08	53.80
$\eta =1.00$	92.28	66.54	56.01	54.19

presents the experimental results with different values of the distillation loss weight. As the value of the weight $\eta$ increased, the model’s performance metrics steadily improved, indicating the crucial role of knowledge distillation in our proposed method. Particularly, when the weight value $\eta$ was set to 1, the model achieved the best performance, demonstrating the effectiveness of the knowledge distillation approach.

5.2.4. Application of the Proposed Distillation Method

In order to verify the effectiveness of the proposed distillation method for other models, we applied the present distillation method to the Frame-field multi-task learning building polygon extraction model based on ResNet-101. The experimental results are shown in

Table 6. The proposed distillation method’s experimental results.

Method	Knowledge Distillation	Backbone	OA	IoU	${\mathit{P}}_{50}^{\mathit{I}\mathit{o}\mathit{U}}$	${\mathit{P}}_{50}^{\mathit{B}\mathit{o}\mathit{u}\mathit{n}\mathit{d}\mathit{a}\mathit{r}\mathit{y}}$	${\mathit{R}}_{50}^{\mathit{I}\mathit{o}\mathit{U}}$	${\mathit{R}}_{50}^{\mathit{B}\mathit{o}\mathit{u}\mathit{n}\mathit{d}\mathit{a}\mathit{r}\mathit{y}}$
Frame-field		ResNet-101	88.43	59.84	50.17	43.25	54.24	47.11
Frame-field	√	ResNet-101	90.45	61.74	51.33	44.97	55.15	48.36
Baseline		HRNetV2-W48	90.88	62.37	52.50	45.71	56.89	49.34
Ours	√	HRNetV2-W48	92.28	66.54	56.01	54.19	60.18	57.35

.

From Table 6, it can be found that although the method framework and backbone of Frame-field and baseline are different, both Frame-field and baseline are multi-task learning models, which are able to apply our proposed multi-feature distillation. After applying the proposed distillation method, the Frame-field was also able to improve significantly in all indicators. Therefore, our proposed distillation method can be applied to other building profile extraction models based on a multi-task learning approach. Due to the limitation of the distillation method, it is difficult to apply the proposed distillation method to the building polygon extraction model that does not adopt the multi-task learning method.

5.3. Weight Setting of Dice Loss

The overall segmentation loss of our model in this paper is composed of both binary cross-entropy (BCE) loss and Dice Loss, with a coefficient $\theta$ controlling the contribution of the Dice Loss. The Dice Loss measures the similarity between the predicted mask and the ground truth mask in the semantic mask branch, and the value of $\theta$ reflects the importance of the Dice Loss in semantic mask learning. We conducted experiments with different values of $\theta$ to explore the optimal weight for the Dice Loss in our proposed method.

Table 7. Dice Loss weights and experimental results.

Weight of Dice Loss	OA	IoU	${\mathit{P}}_{50}^{\mathit{I}\mathit{o}\mathit{U}}$	${\mathit{P}}_{50}^{\mathit{B}\mathit{o}\mathit{u}\mathit{n}\mathit{d}\mathit{a}\mathit{r}\mathit{y}}$
$\theta =0.25$	90.95	62.78	53.01	51.52
$\theta =0.50$	91.38	64.64	54.13	52.09
$\theta =0.75$	92.10	66.25	55.05	53.13
$\theta =1.00$	92.28	66.54	56.01	54.19
$\theta =1.25$	92.11	66.26	55.87	53.62
$\theta =1.50$	92.16	66.32	55.46	54.01
$\theta =1.75$	89.57	60.45	51.42	49.47
$\theta =2.00$	91.44	64.73	54.18	52.23

presents the experimental results with different values of the weight coefficient $\theta$. When $\theta$ was set to 1, the model achieved the optimal performance in terms of all evaluation metrics. However, when $\theta$ was too small, there was a noticeable decline in model performance, indicating the important role of Dice Loss in the segmentation process. It is worth noting that setting $\theta$ too high can also lead to a decrease in model performance, suggesting that a larger value of $\theta$ is not necessarily better.

6. Conclusions

This paper proposed a knowledge distillation method called BPDNet for building polygon extraction. BPDNet utilizes the structural knowledge of buildings distilled from a teacher network trained on a large dataset containing diverse building polygons. By leveraging the generalization capability of the teacher network in extracting structurally diverse building polygons, the proposed method enables the student network to learn the characteristics of building polygons in the target region, thereby improving the student network to represent the features of buildings in the target region. This approach addresses the challenge of limited samples and diverse building structures, which can lead to insufficient generalization performance of the model. Moreover, the incorporation of Dice Loss in the model enhances the accuracy of the boundaries and reduces the blurring of building edges caused by complex backgrounds such as building shadows and impermeable surfaces. The experimental results demonstrated that BPDNet exhibits superior performance in terms of evaluation metrics compared to models trained solely on the target region samples or models fine-tuned using large datasets. The ablation experiments further validated the effectiveness of our knowledge distillation method and the use of Dice Loss. The discussion on hyperparameters also confirmed the rationality of the parameter settings in BPDNet. In addition, the method has the following limitation: the method embeds sample knowledge obtained from a large dataset on the task areas with an insufficient sample size for enhancing the model’s feature representation. However, if the sample size of the task areas is sufficient, the model can be trained without the sample knowledge from the large dataset to obtain a better representation of various styles of buildings. Therefore, the method is not applicable when the sample size of the task areas is sufficient. When the sample size is large enough, the extraction results of this method may not exceed the SOTA method by much.

In future work, our method can be further enhanced by incorporating prior knowledge about the structural regularities of buildings, such as the symmetry of building structures and the presence of right angles at building corners. By leveraging this prior knowledge, we can assist the inference process of building polygons and extract more accurate building polygons.