A Detection Method for Collapsed Buildings Combining Post-Earthquake High-Resolution Optical and Synthetic Aperture Radar Images

Abstract

The detection of collapsed buildings based on post-earthquake remote sensing images is conducive to eliminating the dependence on pre-earthquake data, which is of great significance to carry out emergency response in time. The difficulties in obtaining or lack of elevation information, as strong evidence to determine whether buildings collapse or not, is the main challenge in the practical application of this method. On the one hand, the introduction of double bounce features in synthetic aperture radar (SAR) images are helpful to judge whether buildings collapse or not. On the other hand, because SAR images are limited by imaging mechanisms, it is necessary to introduce spatial details in optical images as supplements in the detection of collapsed buildings. Therefore, a detection method for collapsed buildings combining post-earthquake high-resolution optical and SAR images was proposed by mining complementary information between traditional visual features and double bounce features from multi-source data. In this method, a strategy of optical and SAR object set extraction based on an inscribed center (OpticalandSAR-ObjectsExtraction) was first put forward to extract a unified optical-SAR object set. Based on this, a quantitative representation of collapse semantic knowledge in double bounce (DoubleBounceCollapseSemantic) was designed to bridge a semantic gap between double bounce and collapse features of buildings. Ultimately, the final detection results were obtained based on the improved active learning support vector machines (SVMs). The multi-group experimental results of post-earthquake multi-source images show that the overall accuracy (OA) and the detection accuracy for collapsed buildings (P_cb) of the proposed method can reach more than 82.39% and 75.47%. Therefore, the proposed method is significantly superior to many advanced methods for comparison.

1. Introduction

Timely and accurate evaluation of earthquake damages to buildings after earthquakes is an important part of disaster surveillance [1]. Compared with traditional field survey methods, the remote sensing technology that adopts a remote imaging mode has many advantages, such as timely acquisition of information and not being limited by field conditions, so it has become the main technical means for extracting earthquake damage information of buildings [2,3].

In recent years, the detection of buildings subjected to earthquake damages based on remote sensing images has mainly focused on the identification of collapsed buildings [4,5]. The reason is that collapsed buildings are usually severely damaged and people are trapped therein, which are primary targets in post-earthquake emergency response and rescue [6]. In complex post-earthquake scenarios, collapsed and non-collapsed buildings are generally significantly different in the height. Therefore, introducing elevation information to traditional high-resolution remote sensing images can provide direct evidence support for judging whether buildings collapse or not. Even so, the acquisition of digital elevation data, such as light detection and ranging (LiDAR), usually requires extracting ground control points, with high computation complexity and time costs. Therefore, it is difficult to meet the timeliness requirements for the detection of collapsed buildings after earthquakes [7,8]. For this reason, it is necessary to design a reliable detection method for collapsed buildings in the event of a lack of elevation data. In accordance with different data sources used, detection methods for collapsed buildings can be classified into three categories as a whole: (1) methods based on pre-earthquake and post-earthquake images; (2) methods based on post-earthquake images; and (3) methods combining elevation data.

(1)

Methods based on pre-earthquake and post-earthquake images: Such methods can be used to evaluate the damage degree of buildings by extracting changes of typical features from pre-earthquake/post-earthquake images [9]. Due to the introduction of pre-earthquake data for reference, other ground objects with features similar to collapsed buildings that have existed before earthquakes generally can be effectively eliminated from detection results by using such methods. In spite of this, normal urban evolution may also produce abundant changes in addition to earthquake impact. Furthermore, the lack of pre-earthquake data after earthquakes is often the bottleneck that restricts the popularization and application of such methods [10,11,12].

(2)

Methods based on post-earthquake images: Such methods eliminate the dependence on pre-earthquake data and have stronger universality compared with methods based on pre-earthquake and post-earthquake images [13]. Collapsed buildings are depicted by extracting manually defined or automatically extracted features such as spectra, texture and space and thus an appropriate classifier is selected for prediction [14]. Even so, the diversity of collapsed buildings and complexity of post-earthquake scenarios lead to more prominent problems of different objects with the same spectra and same object with different spectra, which requires establishing more discriminative classification models. Furthermore, the lack of elevation information, as the direct evidence to determine whether buildings collapse or not, is still the main challenge in practical application of such methods [15].

(3)

Methods combining elevation data: Based on remote sensing images, elevation information provided by elevation data, such as LiDAR and digital elevation model (DEM), is used in such methods as a strong basis for determining whether buildings collapse or not [16,17,18]. Although remote sensing images are strongly complementary with elevation data, it is not a common practice to specially collect and produce elevation data only for the detection of collapsed buildings in practical application. In addition, there is no reliable method for scanning and measuring collapsed buildings at present.

Compared with traditional machine learning, deep learning adopts a deep nonlinear network structure to achieve an approximation of a complex function through hierarchical learning, thus extracting advanced features [19]. In recent years, scholars have carried out research on semantic segmentation of the deep learning technique Mask Region-Based Convolutional Neural Network (Mask RCNN), and obtained a great deal of research findings in remote sensing applications. For example, Li et al. [20] proposed a novel Histogram Thresholding Mask Region-Based Convolutional Neural Network (HTMask R-CNN), which utilized the significant differences between old and new buildings in grayscale histogram to improve the classification ability of the model. Mahmoud et al. [21] proposed an adaptive Mask RCNN framework to detect multi-scale objects in optical remote sensing images. In this method, the standard convolutional neural network in Mask RCNN is replaced by ResNet50 to overcome the vanishing gradient problem. Zhao et al. [22] proposed a method combining Mask RCNN with building boundary regularization, which could produce better regularized polygons. In addition to building detection, many state- of-the-art Mask RCNN semantic segmentations have been proposed for other applications. For example, Bhuiyan et al. [23] developed a high-throughput mapping workflow to automatically detect and classify ice-wedge polygons (IWPs). Witharana et al. [24] gauged the influence of spectral and spatial artifacts on the prediction accuracies of CNN models by using Mask RCNN. Despite this, at present, training is usually carried out using deep learning methods based on training samples in specific areas, so the portability of the model remains unclear. In the meantime, the production and manual annotation of sample sets after earthquakes are very time consuming and laborious, which seriously restricts the application of such methods in the detection of collapsed buildings.

In conclusion, machine learning methods based on post-earthquake images neither rely on pre-earthquake data nor require a large number of training samples, so they have unique advantages in usability and timeliness. In view of the lack of elevation information in such methods, introducing the double bounce features can provide supplementary information. Among different scattering contributions present in high-resolution synthetic aperture radar (SAR) images, the double bounce (which is caused by the corner reflector assembled by the front wall and its adjacent ground) with linear characteristics indicates the presence of a building or other artificial target. However, the double bounce features are influenced by the orientation angle of the buildings and the ground material, as well as the polarization. Adamo Ferro demonstrated that the double bounce effect has a strong power signature for buildings which have a wall on the side closest to the sensor almost parallel to the SAR azimuth direction [25]. In polarized SAR images, the double bounce intensity of cross polarization is usually weaker than that of co-polarization, and the double bounce intensity decreases with the increase of the angle between the orientation of the buildings and the polarized SAR azimuth direction. In the post-earthquake scenario, the collapse of a building results in a reduction of the ‘ground-wall’ dihedral structure. The main difference of the scattering mechanism before and after building collapse is the change from the primary double-bounce scattering mechanism to the primary single-bounce scattering mechanism. This is embodied in the real post-earthquake SAR data in which the double bounce generally appears as a bright line parallel to the wall of a non-collapsed building. In comparison, the double bounce of collapsed buildings is not significant or exhibits disorderly distributed speckle noise [26]. To this end, by taking SAR and optical satellite images after an earthquake in Sendai, Japan in 2011 as examples, different manifestations of the double bounce of collapsed and non-collapsed buildings are displayed in Figure 1. Therefore, double bounce may indirectly reflect the elevation information and improve the accuracy of collapsed building detection.

However, SAR images inevitably have problems, such as lack of spectral information, complex noise, and blur degradation, so it is not reliable to detect collapsed buildings by only relying on SAR images. Meanwhile, the spectral and spatial details contained in high-resolution optical images are favorable for accurate location and profile extraction of buildings. Therefore, based on the combination of post-earthquake high-resolution optical and SAR images, traditional visual features of optical images such as spectra, texture and morphology features can be combined with double bounce features. This can provide a new technical path for accurately and reliably detecting collapsed buildings with a lack of elevation information. In particular, in complex environments such as urban areas, many scattering contributions from small structures with possibly different materials interfere, which are not considered in the currently reported theoretical models.

To achieve the complementary advantages of high-resolution optical and SAR images, it is prioritized to establish a unified object set from multi-source data. However, due to a great difference in imaging mechanisms between optical and SAR images, the same object may have significantly different manifestations in two types of data, so it is difficult to extract profile pairs belonging to the same object from heterologous images. In addition, at present, there are few quantitative representations and analysis methods for collapse semantic knowledge contained in double bounce. Finally, the combination of multi-source data means that the annotation of training samples is more time consuming and laborious. Therefore, a reliable effectiveness measure is needed to fully mine and select representative training samples, so as to improve the detection efficiency and accuracy of collapsed buildings.

In view of the above challenges, a non-deep learning method for collapsed buildings combining post-earthquake high-resolution optical and SAR images was proposed in this research. Firstly, a strategy of optical and SAR object set extraction based on inscribed center (OpticalandSAR-ObjectsExtraction) was designed to provide unified analysis elements for the subsequent feature modeling and detection of collapsed buildings. The inscribed center represents the center of the circle with the largest radius inside the boundary of the object. Based on this, a quantitative representation of collapse semantic knowledge in double bounce (DoubleBounceCollapseSemantic) was constructed according to spatial distribution of double bounce. After that, feature modeling of collapsed buildings was performed based on traditional visual features and double bounce features. Finally, the samples were refined based on a category uncertainty index (CUI) between the samples to be tagged and the tagged samples to optimize the active learning process, thus detecting collapsed buildings.

The novel contributions of the proposed method are shown as follows: (1) The proposed OpticalandSAR-ObjectsExtraction overcame imaging differences between heterologous images and extracted a unified object set from optical and SAR images. (2) The proposed DoubleBounceCollapseSemantic provided a way to quantitatively extract double bounce features from SAR images, which could significantly improve the accuracy of collapsed building detection. (3) The CUI was put forward to improve the training process of active learning support vector machines (SVMs), which was conducive to fully mining and selecting representative training samples.

2. Methodology

The proposed method mainly included four steps: building the unified optical-SAR object set based on OpticalandSAR-ObjectsExtraction; extracting double bounce features based on DoubleBounceCollapseSemantic; extracting traditional visual features based on morphological attribute profiles (MAPs); and detecting collapsed buildings based on improved active learning SVMs. The specific realization process is displayed in Figure 2.

2.1. Construction of the Unified Optical-SAR Object Set Based on OpticalandSAR-ObjectsExtraction

To construct the unified optical-SAR object set, the proposed OpticalandSAR-ObjectsExtraction was mainly divided into three steps, namely image segmentation, establishment of a coarse registration-based affine transformation equation, and projection of inscribed center of an object and region growing.

2.1.1. Image Segmentation

Firstly, two images were segmented, and the inscribed center of the object was taken as a feature point in segmentation results to establish the coarse registration-based affine transformation equation. The optical image was segmented by utilizing the well-known business software Ecognition to obtain the object set $R_{opt}$ of the optical image [27]. The segmentation was performed using the following parameters: scale parameter, 30; shape, 0.5; compactness, 0.2. Furthermore, an iterated conditional model (ICM) based on Markov is conducive to better highlighting foreground targets including buildings in SAR image segmentation, so this method was used to obtain the object set $R_{SAR}$ of the SAR image [28]. The reason why we adopted this method is that, on one hand, the method is an image segmentation algorithm based on statistics, which is spatially constrained and has fewer model parameters. On the other hand, this method has been successfully applied to SAR images with different polarizations, such as single polarimetry, dual polarimetry and full polarimetry and yields good results.

2.1.2. Establishment of the Coarse Registration-Based Affine Transformation Equation

In $R_{opt}$ and $R_{SAR}$, the matched object pairs are searched as a basis for establishing the affine transformation equation. Because of invariance to translation, rotation and scaling of moment invariants, the 7th-order Hu moment invariants are taken as measures for similarity between objects [29]. The specific steps are demonstrated as follows:

Step 1: Based on Equation (1), moment invariants of the $i$th object in $R_{opt}$ and $j$th object in $R_{SAR}$ are calculated and all possible combinations are traversed.

$$d_{ij}=\sqrt{{\displaystyle \sum _{n=1}^7{\left({\varphi }_i\left(n\right)-{\psi }_j\left(n\right)\right)}^2}}$$

(1)

where, ${\varphi }_i\left(n\right)$ and ${\psi }_j\left(n\right)$ represent the $n$th moment invariant of the $i$th object in the optical image and $n$th moment invariant of the $j$th object in the SAR image, respectively.

Step 2: An object with the smallest moment invariant is selected from $R_{SAR}$ for each object in $R_{opt}$ to construct a set of matched object pairs $R_{opt-SAR}$. An object with the minimum moment invariant is selected from $R_{opt}$ for each object in $R_{SAR}$ to constitute another set of matched object pairs $R_{SAR-opt}$.

Step 3: The same matched object pairs in $R_{opt-SAR}$ and $R_{SAR-opt}$ are retained as the final set of matched object pairs $R_{match}$.

Step 4: Because inscribed circles of each object are bound to exist and locate inside of the objects, the inscribed centers of each object can be calculated in $R_{match}$. On this basis, each matched object pair can obtain a pair of matched inscribed centers (feature points), thus obtaining the set of matched feature points $P_{match}$ required for establishing the affine transformation equation.

Step 5: By combining $P_{match}$ and Equation (2), the affine transformation equation between optical and SAR images can be established.

$$\{\begin{array}{c}{x}^{\prime }=a_0+a_{1}x+a_{2}y\\ y^{\prime }=b_0+b_{1}x+b_{2}y\end{array}$$

(2)

2.1.3. Projection of Inscribed Centers of Objects and Region Growing

The objects in the SAR image that match with each object in the optical image are searched. Based on coarse registration results, the inscribed centers of each object in $R_{opt}$ are directly projected into the SAR image according to the affine transformation equation, thus obtaining a set of project points in the SAR image. Based on region growing of project points, the SAR image is divided into connected regions corresponding to each object in $R_{opt}$ [30], so as to finally acquire the unified optical-SAR object set $R_{uni}$.

2.2. Extraction of Double Bounce Features Based on Double Bounce Collapse Semantic

In view of extraction of collapse semantic features contained in double bounce, this study mainly designed two parts, namely detection of potential double bounce pixels (PDBPs) and construction of a collapse semantic histogram.

2.2.1. Detection of PDBPs

Since double bounce is shown as a highlighted line in the SAR image, this study firstly used Hough transform for line detection [31], in which LOG operators were used for edge detection to obtain a set of initial potential double bounce pixels (IPDBPs). On this basis, for any one pixel $e$ in IPDBPs, pixels belonging to IPDBPs are searched in its eight neighborhoods. If there is only one pixel meeting the condition, the pixel $e$ is regarded as an endpoint. In this case, the pixels belonging to IPDBPs are searched continuously in a 5 × 5 window with $e$ as the center and the overlapped pixels in eight neighborhoods of these pixels and eight neighborhoods of the pixel $e$ are all taken as PDBPs. Traversing all pixels, the final set of PDBPs is extracted from the SAR image.

2.2.2. Construction of the Collapse Semantic Histogram

In the SAR image, a visual vocabulary based on collapse semantics was designed and the collapse semantic histogram was constructed by combining with a spatial relationship between $R_{uni}$ and PDBPs. When the total number of objects in $R_{uni}$ is $N$, for any one object $R_{uni}^c\left(c=1,2,3,\dots ,N\right)$, the set of visual words and DoubleBounceCollapseSemantic rules are defined as follows:

• Pixel set 1 of non-collapsed buildings. Double bounce of non-collapsed buildings usually appears as a highlighted line at the corner of a building. Therefore, line segments of double bounce with features of non-collapsed buildings overlap or are adjacent to profiles of the objects, showing a similar curvature and direction and a certain length. The specific search and discrimination steps are shown as follows:

Step 1: The blurred line segment $\tilde{L}$ that overlaps or is adjacent to the $R_{uni}^c$ profile is firstly searched. From any one pixel $g$ on the profile, PDBPs are searched from eight neighborhoods of $g$. If there is a PDBP, defined as $r$, PDBPs in eight neighborhoods of $r$ are searched. The newly searched PDBPs and $r$ are retained and the fitting line $\widehat{L}$ is obtained according to these pixels. On this basis, new PDBPs are searched continuously in the existing newly searched eight neighborhoods of each PDBP. If they exist, the distance from this point to $\widehat{L}$ is calculated. When the distance is smaller than $w$, this PDBP is retained. In a similar way, all possible pixels are traversed and all retained PDBPs constitute the blurred line segment $\tilde{L}$.

Step 2: For the next pixel $g^{\prime }$ on the profile, a blurred line segment corresponding to $g^{\prime }$ can be obtained by repeating Step 1. All points on the profile are traversed to form a candidate blurred line-segment set $S_1$. All blurred line segments with the length larger than $T_a$ are retained to constitute a blurred line-segment set $S_2$.

Step 3: For foot points of two endpoints of any one line segment $L_{S_2}$ in the set $S_2$ on the profile of the object, the line segment $L_{S_2}^{\prime }$ of the profile between foot points is intercepted and the line segment $L_{S_2}$ which simultaneously meets the following two conditions constitutes a blurred line-segment set $S_3$. (1) The difference in average curvatures of $L_{S_2}$ and $L_{S_2}^{\prime }$ is calculated and should be smaller than the threshold $T_b$. (2) $L_{S_2}$ and $L_{S_2}^{\prime }$ are fitted by straight lines using the ordinary least squares to calculate the slope difference of the two straight lines which should be smaller than the threshold $T_c$. $S_3$ is the constructed visual word. It should be pointed out that, in order to improve the automation degree of the proposed method, the following adaptive extraction strategy is adopted for $T_a$, $T_b$ and $T_c$. Compared with collapsed buildings, double bounce of non-collapsed buildings is usually longer and more complete. Based on this assumption, an objective function $F_{S_3}(T_a,T_b,T_c)$ is constructed, representing the number of pixels extracted from $S_3$ for an object under different combinations of $T_a$, $T_b$ and $T_c$. Let $T_a$, $T_b$ and $T_c$ be valued in intervals of [0, t], [0, 1] and [0, 1] and t indicate the length of the diagonal of the bounding rectangle for object $R_{uni}^c$. When $F_{S_3}$ is the maximum, $T_a$, $T_b$ and $T_c$ constitute the optimal parameter combination.

• Pixel set 1 of locally collapsed buildings. In $S_1$, blurred line segments with the length smaller than or equal to $T_a$ are retained, namely the constructed visual word.
• Pixel set 1 of completely collapsed buildings. Except for PDBPs which have been defined as visual words, the other PDBPs located on the $R_{uni}^c$ profile or within one pixel outside the profile are the constructed visual word.
• Pixel set 2 of non-collapsed buildings. Within pixels in $R_{uni}^c$ profile, a candidate blurred line-segment set $inner$ meeting conditions is searched from any one pixel $u$. Except for different starting points and scopes of search, other steps are exactly the same as $S_1$ above. Because the blurred line segments in $inner$ are located inside $R_{uni}^c$, the $inner$ is directly regarded as the constructed visual word.
• Pixel set 2 of locally collapsed buildings. In $R_{uni}^c$, PDBPs without being defined as visual words are defined as ${\mathrm{PDBP}}_{res}$ and the ratio of the number of ${\mathrm{PDBP}}_{res}$ to the total number of pixels is $\phi$. Furthermore, the ratio of the total number of PDBPs in the SAR image to the total number of pixels is defined as ${\phi }_{SAR}$. If $\phi \le {\phi }_{SAR}$, ${\mathrm{PDBP}}_{res}$ is the constructed visual word.
• Pixel set 2 of completely collapsed buildings. In $R_{uni}^c$, PDBPs without being defined as visual words are the constructed visual word.

Based on the above six-dimensional visual words, the collapse sematic histogram $I_{csh}$ of the double bounce of $R_{uni}^c$ can be obtained.

2.3. Extraction of Traditional Visual Features Based on MAPs

The area, diagonal, normalized moment of inertia (NMI) and standard deviation in MAPs have been proven to have strong discrimination ability in the detection of buildings. To this end, the traditional visual features in optical and SAR images are extracted based on the four attributes through the proposed automatic building detection from high-resolution remote sensing images based on joint optimization and decision fusion of MAPs (detailed steps can refer to previous studies [32]). On this basis, the multi-scale MAPs sets corresponding to optical and SAR images are obtained, including $MAP{s}_{opt}$ and $MAP{s}_{SAR}$. In $MAP{s}_{opt}$, the mean gray values of $R_{uni}^c$ in each attribute profile (AP) are calculated, thus obtaining the visual histogram $I_{osh}$ of optical images corresponding to $R_{uni}^c$. Similarly, the visual histogram $I_{ssh}$ of SAR images can be obtained.

2.4. Detection of Collapsed Buildings Based on Improved Active Learning SVMs

In the classification stage, $R_{uni}^{}$ is classified into non-collapsed buildings, collapsed buildings and others by using active learning SVMs. The decision-making function of each SVM classifier is shown as follows:

$$f\left(h_k\right)=sign\left({\displaystyle \sum _{m=1}^M{y}_m{\alpha }_{m}K\left(x_m,h_k\right)}+b\right)$$

(3)

Furthermore, when annotating samples by active learning SVMs, it is difficult to annotate samples that are always on the category boundary with the greatest uncertainties. Therefore, this study proposed the CUI, and the calculation process is demonstrated as follows:

Step 1: The probabilities of the samples $h_k$ to be annotated belonging to annotated positive samples $w_l^p$ and annotated negative samples $v_l^q$ are separately calculated:

$$D\left(w_l^p/h_k\right)=\frac{{\displaystyle \sum _{p=1}^P\frac{\langle h_k,w_l^p\rangle }{\Vert h_k\Vert \Vert w_l^p\Vert }}}{P}$$

(4)

$$D\left(v_l^q/h_k\right)=\frac{{\displaystyle \sum _{q=1}^Q\frac{\langle h_k,v_l^q\rangle }{\Vert h_k\Vert \Vert v_l^q\Vert }}}{Q}$$

(5)

where, $w_l^p$ indicates the $p\left(p=1,2\cdots P\right)$th sample in the $l$th category of positive samples; $v_l^q$ represents the $q\left(q=1,2\cdots Q\right)$th sample in the $l$th category of negative samples.

Step 2: On this basis, the CUI of $h_k$ on the $l$th classifier is calculated by the following formula:

$$\mathrm{CUI}\left(w_l^p,v_l^q\right)=2[(D^2\left(w_l^p/h_k\right)-1)\times (D^2\left(v_l^q/h_k\right)-1)]$$

(6)

where, the larger the CUI is, the greater the classification uncertainty of the sample $h_k$.

Step 3: Based on this, the categorical decision function $f_l\left(h_k\right)$ of the sample $h_k$ is calculated in accordance with Equation (3). When CUI is the minimum and $f_l\left(h_k\right)$ is the maximum, the sample $h_k$ is annotated. The annotated samples are added into the training samples to re-train the model. By repeating the above steps, the samples are refined to obtain the final detection results of collapsed buildings.

3. Experiments and Evaluation

3.1. Study Area and Dataset Description

The study area is located in Sendai, Japan. An earthquake (Mw 9.0) occurred on 11 March 2011. The epicenter was located in the Pacific Ocean to the east of Miyagi Prefecture, Japan, with a focal depth of 20 km. Sendai is one of the cities that were most seriously hit by the earthquake. The earthquake and tsunami damaged lots of buildings, including 9877 collapsed buildings.

The post-earthquake high-resolution optical images adopted in this study are IKONOS satellite images in Sendai, Japan, which were collected on 24 March 2011 and showed a spatial resolution of 1 m, as displayed in Figure 3a. Post-earthquake high-resolution SAR images were TerraSAR-X satellite images for the area, collected on 23 March 2011 with the spatial resolution of 3 m. The images were acquired in HH polarization in stripmap mode, as demonstrated in Figure 3b. In the experiments, for the difference in resolution between optical and SAR images, the images with the lower resolution were re-sampled in this research, so that multi-source images had the same resolution. On this basis, this study selected three representative regions for experiments. Dataset 1 is located in an industrial zone where buildings were large and sparsely distributed, as shown in Figure 4a. Compared with the industrial zone, the residential area is usually the most severely affected, and is usually the primary target of post-earthquake emergency response and post-disaster reconstruction. To this end, the constructed Datasets 2 and 3 are both located in the residential area, as displayed in Figure 4b,c. Buildings in this area are usually densely distributed and neatly arranged. Due to the significant radiometric and geometric differences between optical and SAR data, their exact registration and high geometric precision correction are not only very complex but also difficult to obtain the desired results. In addition, since each object is extracted separately in optical and SAR images in this paper, only the matching set of objects need to be found in both datasets. For this reason, the following strategy was adopted for producing the datasets: taking the cropped and segmented optical image as a reference, we cropped the corresponding area in the SAR image which could completely cover all the objects in the optical image according to the visual interpretation.

3.2. Experimental Settings and Methods for Comparison

In the experiments, four different advanced methods were selected for comparison. The first is the detection method of optical images based on sparse dictionary (SD-OPT). The spatial context information is further introduced by constructing the same and different pairs of words with this method, so as to construct multi-visual features to model collapsed buildings [33]. The second is the detection method of SAR images based on multi-texture feature fusion (RF-SAR). In this method, texture features are extracted by comprehensively utilizing the gray-level histogram, gray-level co-occurrence matrix (GLCM), local binary pattern (LBP) and Gabor filtering, and then post-earthquake collapse information of buildings is obtained by Random Forest [34]. This third method is a deep learning method based on object context and boundary enhanced loss (OCR-BE). In this method, a novel loss function, BE loss, is designed according to the distance between pixels and the boundary, forcing the network to pay more attention to learning of boundary pixels [35]. The fourth method is a deep learning method based on UNet 3+ (UNet 3+). UNet 3+ takes advantage of full-scale skip connections and deep supervisions to make full use of the multi-scale features [36]. Among the four methods, the first two are single-source imaging methods based on traditional machine learning. By comparing them, complementarity and combined advantages of two data sources (optical and SAR images) in the detection of collapsed buildings can be verified. The latter two methods belong to deep learning methods combining multi-source data. Comparing them, it is conducive to analyze the difference in performances between the proposed method and deep learning methods in the detection of collapsed buildings, especially under small sample conditions.

All experiments are based on the three datasets in Section 3.1. In order to ensure consistency of evaluation indexes for accuracy across different methods, semantic segmentation results obtained by OCR-BE and UNet 3+ were converted into object-level detection results according to the proportion of pixels belonging to different categories. In the experiments, Matlab 2018 was taken as a simulation platform in all traditional machine learning methods. The two deep learning methods were based on PyTorch-1.3.1 framework and implemented in Ubuntu 16.04.

3.3. General Results and Analysis

Based on the three datasets, the detection results of collapsed buildings by different methods are demonstrated in Figure 5, Figure 6 and Figure 7. In addition, the true ground map made through visual interpretation is regarded as a basis for accuracy evaluation, in which white, gray and black represent the collapsed buildings, non-collapsed buildings and others, respectively.

As shown in the above figures, the detection results obtained by the proposed method are significantly superior to those of the four methods for comparison as a the whole. As traditional machine learning methods, SD-OPT adopts optical images, while RF-SAR uses SAR images. Compared with the proposed method, SD-OPT and RF-SAR separately have prominent false negatives (FNs) and false positives (FPs) because they only rely on single-source data, as displayed in (c) and (d) in Figure 5, Figure 6 and Figure 7. As two deep learning methods, OCR-BE and UNet 3+ need massive training samples to fully train a deep network; otherwise, it is difficult to obtain ideal detection effects. In the experiments, the sample sizes of the three datasets are 1880, 2036 and 2058, and the proportions of the sample sizes of collapsed buildings in the total number of samples are only 9.2%, 10.6% and 12.8%. This results in serious over-fitting and poor generalization effects of the model on the test set. This is the main reason why the detection accuracies of collapsed buildings using OCR-BE and UNet 3+ are significantly lower than that of the traditional machine learning methods. It is considered that with the increase of the sample size of collapsed buildings, the accuracy of the deep learning methods gradually improves until the model converges. In addition, for plants with a large size and low detection difficulty in the industrial zone (Figure 5), except for the RF-SAR method with a lot of FNs and FPs, other methods have good detection effects on buildings with large sizes. For densely distributed small buildings that are difficult to detect in the residential area (Figure 6 and Figure 7), the proposed method and SD-OPT method are significantly superior to other methods regarding the FN rate and FP rate. This indicates that rich spatial details provided by optical images are favorable for accurate characterization of collapsed buildings in the complex context compared with SAR images.

Furthermore, six evaluation indexes, including overall accuracy (OA), FP rate and FN rate as well as detection accuracies of non-collapsed buildings (P_nb), collapsed buildings (P_cb) and others (P_o) were used for quantitative accuracy evaluation, and the results are shown in

Table 1. Detection accuracy based on Dataset 1.

Method/ Indicator	OA (%)	FP (%)	FN (%)	Pnb (%)	Pcb (%)	Po (%)
Evaluation Criteria	The Higher the Better	The Lower the Better	The Lower the Better	The Higher the Better	The Higher the Better	The Higher the Better
Proposed method	82.39	9.65	17.61	52.92	74.57	88.55
SD-OPT	75.85	14.59	25.15	41.35	46.82	92.62
RF-SAR	63.99	21.96	36.01	25.29	44.51	73.17
OCR-BE	78.46	13.31	15.35	78.94	29.48	90.10
UNet 3+	77.28	15.80	19.96	80.21	14.45	92.52

,

Table 2. Detection accuracy based on Dataset 2.

Method/ Indicator	OA (%)	FP (%)	FN (%)	Pnb (%)	Pcb (%)	Po (%)
Evaluation Criteria	The Higher the Better	The Lower the Better	The Lower the Better	The Higher the Better	The Higher the Better	The Higher the Better
Proposed method	80.60	10.74	19.40	50.22	73.94	85.68
SD-OPT	74.66	14.51	26.34	17.77	47.22	87.14
RF-SAR	63.41	22.39	36.59	24.38	29.63	74.02
OCR-BE	77.14	12.49	22.20	76.03	22.30	87.26
UNet 3+	76.80	11.87	24.59	50.00	21.76	89.73

and

Table 3. Detection accuracy based on Dataset 3.

Method/ Indicator	OA (%)	FP (%)	FN (%)	Pnb (%)	Pcb (%)	Po (%)
Evaluation Criteria	The Higher the Better	The Lower the Better	The Lower the Better	The Higher the Better	The Higher the Better	The Higher the Better
Proposed method	78.61	12.80	22.69	55.08	75.47	83.51
SD-OPT	66.81	19.90	33.19	45.72	41.13	77.17
RF-SAR	59.23	25.60	40.77	22.73	27.55	74.77
OCR-BE	76.39	13.92	22.89	63.98	33.32	85.04
UNet 3+	75.11	14.43	25.55	65.24	9.43	92.88

. In the three experiments, OAs of the proposed method separately rise to 82.39%, 80.60% and 78.61%. Particularly, P_cb concerned here reaches more than 73.94%, showing the best performance in all experimental methods and consistent conclusions with visual analysis. In comparison with the proposed method, SD-OPT and RF-SAR only depend on single-source data and their FN rates and FP rates increase by more than 3.77% and 6.94%, respectively. As deep learning methods, OCR-BE and Unet 3+ can only obtain better detection effects of non-collapsed buildings than the proposed method under small sample sizes, while other accuracy indexes reduce significantly, especially with the P_cb being the lowest at only 9.43%. Nevertheless, the detection effects of the two deep learning methods will be greatly improved under the condition of sufficient training samples. Therefore, the strategy of combining optical and SAR images proposed in this research is necessary, feasible and effective in the detection of collapsed buildings and can achieve ideal effects under small sample conditions.

3.4. Visual Comparison of Representative Patches

For further detailed visual analysis and discussion, representative patches were selected in the three datasets, as shown in Figure 8, Figure 9 and Figure 10. Yellow and red boxes separately represent the collapsed buildings and non-collapsed buildings.

The above figures demonstrate that buildings in the industrial zone are easy to detect because of their large sizes and sparse distribution, so better detection effects of collapsed buildings can be acquired by all experimental methods. However, only UNet 3+ (yellow box in Figure 8g) shows FNs and SD-OPT presents FPs (yellow box in Figure 8e). For non-collapsed buildings in the industrial zone (red boxes in Figure 8), the proposed method and the two deep learning methods do not incur FPs, while SD-OPT and RF-SAR separately show FPs (red box in Figure 8d) and FNs (red box in Figure 8e). In the residential area with neatly arranged and densely distributed buildings, only the proposed method yields completely correct detection results for collapsed buildings (yellow boxes in Figure 9 and Figure 10). By contrast, RF-SAR (yellow box in Figure 10e) and OCR-BE (yellow box in Figure 10f) present FPs, while SD-OPT (yellow box in Figure 9d) and UNet 3+ (yellow boxes in Figure 9g and Figure 10g) show FNs. For non-collapsed buildings, the visual analysis results are similar to that of the industrial zone and good effects are reached by different methods. However, only SD-OPT and RF-SAR have obvious FPs and FNs. In conclusion, the detection effects of non-collapsed buildings by the five methods are good, while the proposed method has higher P_cb and less FPs and FNs by combining with optical and SAR images, which is consistent with the quantitative analysis results.

4. Discussion

4.1. Validity Analysis of Combined Optical and SAR Images

To further verify the validity of combined optical and SAR images, experiments based on single-source data (optical or SAR images) were conducted by using the proposed method. The accuracy evaluation of experimental results obtained by combining optical and SAR images, based on the optical image and based on the SAR image, are separately shown in

Table 4. Comparison of detection accuracies of combining optical and SAR images with those based on single-source data.

Datasets	Method/Indicator	OA (%)	FP (%)	FN (%)	Pnb (%)	Pcb (%)	Po (%)
	Evaluation Criteria	The Higher the Better	The Lower the Better	The Lower the Better	The Lower the Better	The Higher the Better	The Higher the Better
Dataset 1	Optical and SAR	82.39	9.65	17.61	52.92	74.57	88.55
	Optical	66.49	30.56	27.12	40.86	46.59	75.79
	SAR	74.68	14.49	23.32	50.19	57.23	81.10
Dataset 2	Optical and SAR	80.60	10.74	19.40	50.22	73.94	85.68
	Optical	64.62	20.76	30.38	38.43	47.66	72.24
	SAR	74.71	14.48	25.29	45.04	54.91	83.33
Dataset 3	Optical and SAR	78.61	12.80	22.69	55.08	75.47	83.51
	Optical	65.40	19.95	31.65	69.79	50.57	67.02
	SAR	72.30	16.07	25.70	45.45	62.91	79.76

.

On this basis, after combining optical and SAR images, the OA in experiments based on three datasets increases by 6.31~7.71% and P_cb rises by 12.56~19.03% compared with that under single-source data. Therefore, earthquake damage characteristics of buildings were depicted from multiple perspectives by combining optical and SAR images after the earthquake, and the extracted complementary information could significantly improve the detection accuracy of collapsed buildings. Particularly, double bounce in the SAR image provides key evidence support for judging whether buildings collapse or not, so P_cb in the experiment based on the SAR image is always significantly higher than that based on the optical image only with traditional visual features.

In addition, two representative patches were selected for further visual analysis, as shown in Figure 11 and Figure 12. For collapsed buildings with fragmented distribution in both images (yellow boxes in Figure 11), correct results can be obtained by the three methods. For collapsed buildings with well-preserved roofs in optical images (yellow boxes in Figure 12), correct judgement can be made by only the proposed method and the method based on SAR images because the double bounce in SAR images shows the typical collapse semantic features. However, FPs appear when using the method based on optical images (yellow box in Figure 12d). Non-collapsed buildings shown in red boxes in Figure 11 illustrate complete profile and single texture in optical and SAR images, so correct results are obtained from the three methods. For non-collapsed buildings with intact roofs in optical images but regional fragmentation distribution in SAR images, only the proposed method and that based on optical images can make correct judgements, while the method based on SAR images have obvious FPs. Therefore, it is feasible and effective to improve P_cb by complementing advantages of optical and SAR images.

4.2. Validity Analysis of DoubleBounceCollapseSemantic

To clarify the validity of the constructed DoubleBounceCollapseSemantic, the comparative experiments were carried out by adding or not adding double bounce features extracted by DoubleBounceCollapseSemantic to traditional visual features of combined optical and SAR images. The results are shown in

Table 5. Validity analysis of DoubleBounceCollapseSemantic. √ and – separately represent that a feature is added and not added.

Datasets	Method/Indicator	OA (%)	FP (%)	FN (%)	Pnb (%)	Pcb (%)	Po (%)
	Evaluation Criteria	The Higher the Better	The Lower the Better	The Lower the Better	The Higher the Better	The Higher the Better	The Higher the Better
Dataset 1	√ DoubleBounceCollapseSemantic	82.39	9.65	17.61	52.92	74.57	88.55
	– DoubleBounceCollapseSemantic	78.83	11.86	20.81	48.64	67.63	85.52
Dataset 2	√ DoubleBounceCollapseSemantic	80.60	10.74	19.40	50.22	73.94	85.68
	– DoubleBounceCollapseSemantic	77.26	12.44	23.01	46.28	64.46	84.48
Dataset 3	√ DoubleBounceCollapseSemantic	78.61	12.80	22.69	55.08	75.47	83.51
	– DoubleBounceCollapseSemantic	74.69	15.29	25.40	42.25	68.68	81.47

.

As illustrated in the above table, the OA in the case of adding DoubleBounceCollapseSemantic increases by 3.34~3.92%, while FP rate and FN rate decrease by 1.7~1.49% and 2.71~3.61% compared with those in the case without adding DoubleBounceCollapseSemantic. P_cb rises by 6.49%, 9.48% and 6.79%, respectively. Therefore, the proposed DoubleBounceCollapseSemantic is effective. On this basis, six collapsed buildings and six non-collapsed buildings were selected from the three datasets and histogram statistics were created on pixels of double bounce belonging to different visual words, as displayed in Figure 13a,b.

Therefore, the histogram of collapsed buildings shows similar distribution, with low intra-class separability. Meanwhile, the proportion of the pixels of collapsed buildings is significantly higher than that of the non-collapsed ones, so it is conducive to obtaining correct identification results. For non-collapsed buildings, the proportions of pixels of collapsed to non-collapsed are contrary to collapsed buildings. Therefore, collapsed and non-collapsed buildings have good inter-class separability in the above histograms. Furthermore, for pixels of locally collapsed buildings, their proportion in collapsed buildings is significantly higher than that in non-collapsed buildings, so this is favorable for enhancing inter-class separability between collapsed and non-collapsed buildings.

4.3. Validity Analysis of CUI

To verify the validity of CUI, comparative experiments were conducted by adding or not adding CUI to the active learning SVMs, and the accuracy was evaluated. The results are shown in

Table 6. Validity analysis of CUI. √ and – separately represent that an index is added and not added.

Datasets	Method/Indicator	OA (%)	FP (%)	FN (%)
	Evaluation Criteria	The Higher the Better	The Lower the Better	The Lower the Better
Dataset 1	√ CUI	82.39	9.65	17.61
	– CUI	81.58	9.81	18.33
Dataset 2	√ CUI	80.60	10.74	19.40
	– CUI	79.07	12.45	20.01
Dataset 3	√ CUI	78.61	12.80	22.69
	– CUI	76.90	13.27	23.78

.

As listed in Table 6, OAs in the three experiments increase by 0.81%, 1.53% and 1.71%, while FP rate and FN rate reduce by 0.16~1.71% and 0.61~1.09%, respectively. Therefore, this suggests that the proposed CUI is conducive to selecting more representative samples for model training, which can significantly improve the classification accuracy.

4.4. Analysis of Effects of the Number of Initial Training Samples

In order to verify the performance of improved active learning SVMs proposed in this study under different numbers of initial training samples, the number of initial training samples in each category is valued in the interval [5, 50] at the step length of 5. The change trend of the OA with the increase of the number of training samples is shown in Figure 14.

Therefore, with the increase of the number of initial training samples, the OA rapidly rises in the interval [0, 20] and then tends to be stable. The OAs under Datasets 1 and 2 reach the peak (83.05% and 81.43%) when the sample size is 45. The OA under Dataset 3 reaches the peak of 79.14% when the sample size is 50. Although the peak OA rises by 0.53~0.83% compared with that under the sample size of 20, the number of the training samples required is more than doubled. In accordance with the above analysis, the number of training samples in each category is suggested to be 20.

5. Conclusions

With a lack of pre-earthquake data, the detection method for collapsed buildings relying on post-earthquake high-resolution optical and SAR images was proposed. To solve the challenge for accurately judging whether buildings collapse or not due to the lack of elevation data, the elevation information of buildings was indirectly acquired with this method by mining the double bounce features in SAR images. Moreover, automated detection of collapsed buildings was reached through combining multi-source traditional visual features. To this end, this research firstly designed the OpticalandSAR-Objects Extraction strategy to construct the unified optical-SAR object set. Based on this, the DoubleBounceCollapseSemantic was constructed, thus bridging the semantic gap between double bounce and collapse features of buildings. In the classification stage, the CUI was put forward, which was conducive to selecting more representative samples to optimize active learning SVMs and finally automatically detect collapsed buildings. Through multi-group comparative experiments on post-earthquake remote sensing images in different regions, the proposed method shows excellent performances compared with visual and quantitative analysis. The OA and P_cb can reach 82.39% and 75.47% at most, so the proposed method is superior to multiple methods for comparison. However, the proposed model does not dig into the influence of factors such as orientation angle of the building and polarization on double bounce intensity. In the future, we will focus on these issues to develop more complicated and advanced models.