An Artificial Intelligence-Based Method for Crack Detection in Engineering Facilities around Subways

Abstract

While the construction and operation of subways have brought convenience to commuters, it has also caused ground subsidence and cracks of facilities around subways. The industry mainly adopts traditional manual detection methods to monitor these settlements and cracks. The current approaches have difficulties in achieving all-weather, all-region dynamic monitoring, increasing the traffic burden of the city during the monitoring work. The study aims to provide a large-scale settlement detection approach based on PS-InSAR for the monitoring of subway facilities. Meanwhile, this paper proposes a crack detection method that is based on UAVs and the VGG16 algorithm to quantify the length and width of cracks. The experimental data of Shenzhen University Section of Metro Line 9 are used to verify the proposed settlement model and to illustrate the monitoring process. The developed model is innovative in that it can monitor the settlement of large-scale facilities around the subway with high accuracy around the clock and automatically identify and quantify the cracks in the settled facilities around the subway.

1. Introduction

In order to meet the growing demand for public transportation and to minimize the occupation of land space, the subway has become a priority option for public transportation systems globally. Since the opening of the London metro in 1980, subways have been built in 193 countries around the world [1]. For example, the London Underground has a long history, which now serves 270 stations and over 408 km of track [2]. In China, 40 cities across the country have 208 line subways, with 7253.73 km by 2021, to provide a service to 23.71 billion passengers annually [3]. Most of the subways run through the city′s center and they may cause deformation, subsidence, or even collapse of the engineering facilities, such as subway stations, roads, buildings, bridges, etc., along the subways [4]. It is reported that the land subsidence due to subway construction accounts for 30% of land subsidence in urban construction [5].

For instance, Shanghai has experienced many subsidence accidents caused by subway construction. The breakdown of Shanghai Metro Line 4 in 2003 was the most serious accident in China′s subway construction history. The subway has experienced significant subsidence, destroyed three buildings and caused an economic loss of 150 million yuan [6]. In 2013, an explosion during the construction of Guangzhou Metro Line 8 resulted in the collapse of 690 m² of tunnel face and the destruction of two buildings [7]. Concerns about the safety of engineering facilities along the subway lines stand out, along with the concerns surrounding subway construction and operation. The problems of land subsidence caused by subway construction have received extensive attention from scholars and governmental departments in many cities [8], such as Florence [9], Tokyo [8], Beijing [10], Hong Kong [11], and Shanghai [12]. The traditional methods have certain limitations, such as the lack of accurate and continuous characterization records of subway facilities; the difficulties of inspecting the hidden places of engineering facilities; the road sections need be closed for the inspection, which increases the traffic burden of the city; and the significant subjectivity and difference in the records among inspectors [13]. In order to ensure the safety of residents’ lives and property, it is urgent to establish new methods to monitor the health of engineering facilities along the subway lines.

Settlement screening and appearance inspection are two primary engineering facility monitoring methods [14,15]. PS-InSAR technology (Persistent Scatterer Interferometric Synthetic Aperture Radar technology) can provide millimeter-level motion information with superior performance, which can monitor abnormal behavior in a short period and motion information that cannot be found visually [16,17]. Therefore, PS-InSAR technology is extensively used in the settlement monitoring of infrastructure, including buildings [18], motorways [19], airports [20], subways [5], etc. For instance, Gao et al. [21] used PS-InSAR technology to analyze ground fissures and the land subsidence of Beijing Capital International Airport and found active ground fissures. In addition, based on PS-InSAR technology, Wang et al. [22] carried out long-term ground deformation monitoring and modeling of the Guangzhou metro networks. These methods, however, have certain shortcomings, i.e., although PS-InSAR technology can quickly and accurately obtain the settlement data of engineering facilities around the subway, it cannot capture intuitive crack images. Furthermore, traditional crack recognition and measurement methods are not able to accurately identify and measure cracks in facilities around the subway due to the complexity of the environment around the subway and the diversity of structures and materials. To overcome the limitations of the current methods, a method integrating PS-InSAR, UAVs and the VGG16 algorithm was developed to monitor the settlement along the subway, inspect the appearance of facilities around the subway, and identify and quantify cracks, which improves the accuracy and speed and reduces the cost of the monitoring of settlement and cracks for engineering facilities around subways.

2. Literature Review

2.1. InSAR Monitoring Technology

SAR (Synthetic Aperture Radar) is a measurement and imaging technique that uses the phase of coherent radar signals to measure the surface of the terrain, and the variation in surface detail features over time [23]. In contrast to GPS technology, SAR technology can be applied to an area of about 1000 km², which is regularly scanned by single or multiple satellites. SAR technology does not need to arrange monitoring points on site, and the monitoring cost is low, while the efficiency is high. However, the temporal and geometric decorrelation of SAR becomes the limiting factor for applying SAR technology in settlement monitoring. In addition, the inhomogeneity of the atmosphere can also strongly affect the accuracy of the results [24]. Based on the SAR technique, InSAR (Interferometric Synthetic Aperture Radar) time-series technology, a robust measurement remote sensing method for estimating millimeter-scale surface deformation from space, has been developed to solve temporal and geometric decoherence [25]. Although the interferometric phase can be obtained by using the geometric relationship between the two SAR acquisition points, InSAR technology presents a problem with signal decorrelation. InSAR time-series analysis techniques mainly address signal decorrelation to estimate deformation and atmospheric signals [26]. These techniques can be divided into three types. The first type, named the D-InSAR technique, separates the phase contribution terms other than the surface deformation phase in the interferometric phase [27]. The second type is PS-InSAR permanent scatterer (PS) InSAR technology, which mainly analyzes time-coherent point scatters in urban areas [28]. The PSP-InSAR (persistent scatterer pairs InSAR) persistent scatterer method is a new persistent scatterer interferometry method [29].

D-Insar is a low-cost deformation monitoring tool for large area coverage, but the temporal decoherence and atmospheric inhomogeneity limit its use of D-Insar [30]. To address the limitations of D-InSAR, Ferretti was the first to propose persistent scatter InSAR PSI. The PS-InSAR technology is a kind of InSAR technology, the main feature of which is to extract pixels with relative steady backward scattering characteristics from a range of interferograms, thus eliminating the effect of atmosphere interference on the beam signal. Validation activities have been performed in various studies since the first application of the PSI algorithm to confirm the accuracy of the results of the technique. The verification studies are mainly based on level measurements [30] and the Global Navigation Satellite System [31].

Subsidence monitoring in urban areas has received more and more attention from scholars such as those in Tokyo [8], Houston [32], Beijing [33], Shanghai [34], etc. Subsidence is mainly divided into endogenous subsidence and exogenous [35]. Endogenous subsidence is related to internal geological activities such as earthquakes, faults, volcanism, etc. [36]. Subsidence caused by human activities is often regarded as exogenous subsidence, such as groundwater extraction [37], subway construction, etc. [11]. Urban subsidence is usually caused by different reasons, such as groundwater or gas and oil extraction [38,39]. Zhou et al. [10] conducted a detailed study on Beijing based on time series using the data from 2012 to 2018 and found that groundwater is an important factor affecting subsidence. Peng et al. [40] collected Sar data in Shenzhen from 2004 to 2017. According to the InSAR time-series analysis of Shenzhen, most of the uneven subsidence in Shenzhen often occurs in reclamation and coastal areas. Peng et al. [40] found that the subway is also the main factor causing subsidence, especially in the reclamation area that the subway passes through.

PSI technology is often used in the settlement monitoring of infrastructure, including subways [5], buildings [18], airports [20], highways [19], etc. Gao et al. [21] used PS-InSAR technology to monitor the Beijing Capital Airport and found that the Shunyi fracture developed by the Shunyi-Liangxiang fracture structure is the main cause of the development of differential settlement at the airport. Ramirez et al. [16] analyzed the surface deformation related to tunneling using the PS-InSAR and Sentinel-1B SAR data, and compared the PS-InSAR results with those from the conventional leveling method. Yao et al. [41] proposed a cross-comparison method using SBAS-InSAR and PS-InSAR technology that refined the deformation features and verified the reliability of the InSAR results, which helps in the comprehensive identification and further mapping of landslides.

Real-time and accurate structural deformation monitoring plays a vital role in preventing traffic accidents, which makes PS-InSAR technology more and more widely used in linear transportation infrastructure. Lyu et al. [42] calculated the vertical surface displacements from 2010 to 2016 according to the TerraSAR-X images and discussed the relationship between groundwater level, temperature, and age with the seasonal deformation of the overpass. The conclusions show that the seasonal vertical deformation in the study area is influenced by the interaction of groundwater level, temperature, compressible layers and the age of the overpass. The seasonal deformation trends of the overpasses were consistent with the temperature trends. The authors also found that the thicker compressible soil thickness provided a conducive geologic environment.

2.2. Crack Recognition Model

2.2.1. Image Processing-Based Model

With the development of SAR technology, infrastructure settlement in large areas can be effectively monitored. Nevertheless, settlement information alone is insufficient to support experts in assessing the settlement status of infrastructure. Concrete structures play an essential role in infrastructure, but cracks are inevitable due to uneven settlement factors, construction quality, and other reasons. If cracks are allowed to develop, the durability and serviceability of the infrastructure can be severely compromised.

Manual inspection was a very common and traditional method for crack detection [43]. This method, however, is labor-intensive, error-prone, vulnerable to traffic threats and subjective. In addition, surveying engineers often move between work areas, often in heavy traffic areas, so their life safety is also a concern [44]. Crack detection is a hot research topic. Many studies have been conducted on the manual investigation of cracks. Traditional methods include image processing and deep neural networks and support vector machines [45,46]. The early use of computer vision technology to detect damage is the mainstream way to detect cracks with a high recognition rate of surface defects (e.g., cracks and corrosion). There are four edge detection methods that are commonly used: wavelet transform, Fourier transform, Sobel and Canny [47].

Lyasheva et al. [48] proposed a computer vision-based pavement crack detection algorithm, developed an automatic crack recognition software system based on the proposed algorithm, and verified the effectiveness of the method. Takafumi et al. [49] proposed a new full image processing method for the detection of cracks in concrete surfaces under various conditions. The method utilizes GP-assisted SACIF to develop a robust image filter (multi-sequence image filter) for crack detection. It automatically sets several simple image filters in multiple sequences by SACIF, then iterates and searches for the best combination to obtain the best multi-sequence image filter. Lu and Fu [50] used the improved Canny algorithm to judge the presence or absence of cracks, which performed well in removing noise and preventing edge loss. At the same time, the crack recognition rate of the algorithm has also been significantly improved.

However, the literature review found that edge detection is a complex problem since the results are largely influenced by the noise generated when using image processing, mainly due to complex illumination, distortion and background, and without optimization of its solution. One of the most effective ways to surmount these problems is the use of noise reduction methods, but with little success.

2.2.2. Detection Algorithms Based on Classification Tasks

With the rise of machine learning technology, machine learning technology has been gradually adopted to crack detection. The technic core of crack recognition is image classification. Due to the capacity of convolutional neural network (CNN)-based deep learning algorithms to efficiently extract image features through convolution operations, they have found extensive utility in image classification, object detection, image segmentation, and related domains. Many scholars have continually enhanced, innovated, and achieved significant advancements in CNN algorithms.

Chen and Jahanshahi [51] proposed a NB-CNN deep learning framework that uses CNNs to analyze individual videos and perform crack detection frame by frame, proposing a novel data fusion scheme to aggregate the data extracted from each video frame. The information enhances the overall performance and robustness of the system. The proposed framework achieves a false positive rate of 0.1 per frame and a hit rate of 98.3%. Fan et al. [52] suggested a CNN-based pavement crack detection method, which can predict the crack structure of each patch from the input images after training, and then combine all the results to collect probabilistic outputs. Finally, we can set the decision threshold to obtain the final binary result. Compared with other traditional defect detection methods, the method exhibits superior performance in all aspects and does not require preprocessing. However, it has so many parameters to set manually that it will significantly impact the overall performance. Liu et al. [53] implements a deep learning framework on the basis of the classical network architectures Alexnet, VGG and Resnet for crack detection. Liu et al. [53] summarized all the existing crack segmentation and detection datasets, and established the most extensive crack segmentation and detection benchmark dataset on the Internet, which is available to the world.

2.3. Crack Measurement Model

The method of convolutional neural network is only aimed at the classification of images, that is, whether there is a crack, but it has no countermeasures for the measurement of the length and width of the crack. By only identifying whether there is a crack, without quantifying the crack, the information provided is insufficient. This is because in the infrastructure of concrete, asphalt, etc., the length and width of cracks are a key parameter in evaluating whether reinforcement is required. Based on the existing methods for automatic concrete crack detection using convolutional neural networks, Dung and Anh [54] proposed bounding box and image classification methods. This crack detection method is based on a deep FCN (fully convolutional network) for semantic segmentation of concrete crack images. Following validation by the concrete crack dataset consisting of 40,000 images of 227 × 227 pixels, the average accuracy of the FCN network is about 90%. Deng et al. [55] proposed a faster region-based neural network for detecting cracks on the concrete bridges with contaminated backgrounds, which can successfully detect tiny cracks from contaminated background after training.

Although crack measurement based on deep learning has been a hot topic recently, there are still many unsolved problems, such as the large workload of crack measurement data annotation based on deep learning and the low accuracy of pixel segmentation. However, computer vision-based methods have the advantages of high efficiency and easy cost control. Lins and Givigi [56] proposed a method to measure cracks with handheld devices, which combines statistical filtering (particle filtering) technology to automatically identify cracks in images and machine vision technology for measuring crack width and length.

Image processing methods also have been extensively used in crack measurement. Sulistyaningrum et al. [57] applied the Gabor filter method to texture defect detection. In this study, fifteen cracks were tested experimentally and it was found that the accuracy of the method for the length of the crack area reaches 95.19%, and the accuracy for the width of the crack area reaches 87.94%.

In some dangerous areas and on higher floors, drone photography is more advantageous. Liu et al. [58] proposed a UAV-based crack detection scheme. This paper proposes a GAN (generative adversarial network)-based fuzzy model to overcome the degradation of image quality caused by UAV motion. In addition, the idea of using local jump connections is introduced by identifying the strong correlation between blurring and sharpening. The proposed deblurring model is experimentally validated by studying the effect of jump connections on deblurring. The model is tested in comparison with the existing deblurring models, and the results show that the model significantly improves the deblurring performance in terms of both the overall structure and feature details of the cracked image.

Based on the previous work, Yang et al. [59] proposed an image processing technique, which can automatically measure crack features, including the width, length, orientation, crack pattern, etc. In the technique proposed by Yang et al. [59], morphological techniques are used to rectify the inhomogeneous brightness of the background, and enhanced binarization and shape analysis are used to increase the detection performance. In addition, detailed algorithms for calculating crack length, width, and direction, as well as artificial neural networks to identify crack patterns, including vertical cracks, horizontal cracks, diagonal cracks, and random cracks, are also proposed. A series of experimental and analytical studies validate that the proposed technique can precisely measure and analyze the fracture characteristics.

The literature review shows that PS-InSAR, convolutional neural networks, and digital image processing have achieved some interesting research results. However, there is no settlement monitoring method that integrates PS-InSAR, convolutional neural networks, or digital image processing techniques. The framework we proposed integrates these three technologies, which can effectively identify dangerous and abnormal points along the subway where the settlement exceeds the threshold, inspect the appearance of facilities around the subway, and identify and quantify the cracks, providing a novel and efficient method for settlement and crack monitoring.

Table 1. Summary of prior work and description of the proposed method.

Technology	Previous Research	Authors	Limitations
PS-InSAR	Detect the subsidence and ground fissure	Gao et al.	The provided settlement information is not sufficient for experts to assess the settlement condition of the infrastructure
	Analyze the surface deformation	Ramirez et al.
	Calculate the vertical surface displacements	Yao et al.
Crack recognition base on image processing	Proposed a computer vision-based pavement crack detection algorithm	Lyasheva et al.	Recognition results are greatly influenced by noise
	Proposed a new robust image processing method for detecting cracks on concrete surfaces under various conditions	Takafumi et al.
Crack recognition base on classification	Proposed a NB-CNN deep learning framework for frame-by-frame crack detection in individual videos	Chen and Jahanshahi	The accuracy of measuring cracks is limited and the amount of labeling work is high
	Proposed a CNN-based method for pavement crack detection	Fan et al.
Crack measurement	Proposed a method to measure cracks with handheld devices	Lins and Givigi	Handheld devices have significant limitations for crack detection in hazardous areas and higher floors
	Applied the Gabor filter method to crack measurement	Sulistyaningrum et al.
The proposed method	Integrated PS-InSAR, convolutional neural network, and digital image processing techniques. Large-area, all-weather, all-day automated settlement monitoring Inspect the appearance of facilities around the subway for high-precision crack recognition Identify and quantify the cracks

provides a brief summary of prior work and the method proposed in the study.

3. Methods

3.1. Framework

Given the drawbacks of previous studies, such as the difficulty in accurately recognizing cracks in subway environments and the inability to obtain crack information, this study aims to provide a large-area settlement solution based on PS-InSAR technology, considering its ability for monitoring subway facilities with high accuracy over a long period of time. Meanwhile, a crack detection method based on UAVs, deep learning and digital image processing is proposed, which can accurately quantify the length and width of cracks in the complex environment around the subway. The research concept is shown in Figure 1.

• Scope and aims: By comprehensively reviewing relevant literature such as the subsidence monitoring model, traditional subsidence monitoring method, InSAR monitoring technology, digital image recognition model, deep learning fracture recognition model, digital image fracture measurement and fracture measurement model based on fracture segmentation, the scope and aims of the study were developed.
• Data collection, preprocessing and selecting research areas: This included determining typical and representative research areas, collecting photos of cracks through multiple channels, and scientifically conducting training sets, validation sets, and test sets. Preprocessing the photos can better train the model, improving the accuracy of model classification, and improving the accuracy of crack quantification.
• Building a settlement monitoring model, including:
  • Positioning of settlement points. The PS-InSAR technology was used to process satellite remote sensing data and calculate the settlement rate of buildings to obtain all abnormal settlement points in the research area. Subsequently, an uncrewed aerial vehicle was used to conduct on-site surveys of abnormal points and obtain appearance images of facilities.
  • Crack identification. Using crack images obtained from multiple channels as training data, a crack recognition model based on the VGG16 algorithm was trained, which can accurately identify whether there are cracks in the image.
  • Crack analysis. According to the recognition result of the previous step, information was extracted from the image containing the crack. Finally, the crack′s width, length and area can be calculated, laying the foundation for further hazard assessment.
• Empirical research: Taking the Shenzhen University Section of Metro Line 9 as an example, the specific process of implementing the settlement monitoring model proposed in this paper is demonstrated, and the model′s validity is verified.

3.2. Data Collection and Pre-Processing

3.2.1. Remote Sensing Data Preparation and Pre-Processing

The preparation and preprocessing of remote sensing data consist of 4 parts: preparation of SLC (Single Looked Complex) data, image alignment and cropping combinations, selection of DEM data and alignment and selection of master images.

Single Looked Complex (SLC) data are the basis of PS-InSAR. The data set in this study comes from the ESA Sentinel 1 Earth Observation Satellite. The satellite parameters are shown in

Table 2. Parameter table of sentinel No. 1.

Project	Parameter
Satellite	SENTINEL-1
Country/Organization	ESA
Imaging mode	Interference wide mode
Wavelength	C-band (5 cm)
Return to cycle	12 days
Incident angle	29–46°
Spatial resolution	5 m × 20 m
Polarization mode	Dual polarization: HH + HV, VV + VH, Single polarization: HH, VV
Scan width	250 KM

.

A total of 15 single-view complex images were selected in this study, and the period is between 22 September 2019 and 8 March 2020. The selected single-view images can cover the entire Shenzhen area.

3.2.2. Deep Learning and Image Processing Data Collection and Pre-Processing Data Collection

The study′s primary purpose is to realize the identification and measurement of dangerous points and abnormal point cracks in the subway and the engineering facilities along the subway line subsidence. Engineering facilities include roads, buildings, bridges, etc., and their materials are not the same. Therefore, the dataset we used for model training should contain as many materials as possible, such as asphalt, concrete, bricks, etc. In order to obtain a large and diverse set of high-quality crack images, the data sources for the study consisted of the following 4 sources:

• Using search engines such as www.baidu.com (accessed on 5 March 2021) and www.bing.com (accessed on 12 March 2021) to retrieve and collect crack images posted on the Internet;
• Handheld cameras in Shenzhen perform manual shooting;
• Use of DJI′s 4RTK drone to shoot high-rise and hidden places;
• Public data sets in existing research.

After collecting the data, the images need to be preprocessed by being cropping into a fixed size. The images are then labeled to distinguish whether they contain cracks or not. Finally, the crack information is highlighted by grayscaling and image enhancement.

Image Cropping and Labeling

When deep learning performs image classification, the input image needs to be cropped to a uniform size. In the data collected in this experiment, the pixels of the images taken by the camera are 3024 × 4032, the pixels taken by the drone are 4864 × 3648, the public data set is 227 × 227, and the sizes of the images downloaded from the Internet vary. However, the training of deep learning models requires uniform-size image data. At the same time, in order to avoid the impact of image compression and distortion on the model training effect, this study cuts the oversized image into 227 × 227 image samples, and filters out the images that do not contain concrete and crack information. Finally, a dataset with 8026 images was obtained. In this dataset, there are 1294 images taken manually, 912 images taken by drones, 2000 images downloaded from the Internet, and the public data set contains 4000, with positive and negative factors each accounting for 50% (see

Table 3. Dataset information.

Data Source	Total	After Clipping	Positive	Negative
Manual shooting	458	1294	670	624
Drone shooting	218	912	450	462
Internet search	2000	2000	1000	1000
Public data set	4000	4000	2000	2000

for details). Finally, according to the set-aside cross-validation method, the data were divided into training, validation and test sets in the ratio of 8:1:1. As shown in

Table 4. Training, validation and test data sets.

Data Sets	Percentage	Set Size
Training set	80%	6556
Validation set	10%	820
Test set	10%	820
Total	100%	8206

, the test set contains 6556 images, the validation set contains 820 images, and the test set contains 820 images.

3.2.3. Image Processing Data Pre-Processing

Grayscale

The color of a color image is jointly determined by the three components of R, G, and B, but RGB does not reveal the morphological characteristics of the image; instead, the color is adjusted according to the optical principle. Therefore, converting a color image into a grayscale image that contains only luminance information but not color information helps reduce computational effort. Grayscale can transform a color image into a grayscale image by deleting all the color information of the image and only retaining the brightness of each pixel. Grayscale processing methods primarily include the component method, maximum method, average method, weighted average method and so on. The weighted average method effectively enhances data features, thereby enabling the model to better learn crack information. Therefore, in this paper, we employ the weighted average method for grayscale preprocessing of images. The three-channel components of the color image are weighted and averaged according to the importance of each channel, and then the resulting weighted average is used as the gray value of the grayscale image. The gray value is shown in Equation (1).

$$\mathrm{f}(\mathrm{x},\mathrm{y})=0.299\mathrm{R}(\mathrm{x},\mathrm{y})+0.587\mathrm{G}(\mathrm{x},\mathrm{y})+0.114\mathrm{B}(\mathrm{x},\mathrm{y})$$

(1)

Image Enhancement

In the infrastructures, the difference between the crack and the surrounding background color is generally small, which can lead to a lack of crack contrast. In order to make the edges of the cracks more prominent, the image′s contrast needs to be enhanced. The grayscale range measures contrast. The larger the range, the higher the contrast and the clearer the image. Contrast enhancement is based on the grayscale histogram. The grayscale histogram is a function of the grayscale level of the image, which is used to describe the number or occupancy rate of each grayscale pixel in the image matrix, that is, the number of pixels corresponding to each gray level between 0 and 255 is counted.

Linear transformation is used to enhance contrast, which is a less computationally intensive method of enhancing dynamic range by linearly stretching the original gray level to stretch the range of the original gray level through linear transformations. The Equations (2)–(4) of a linear transformation are shown below.

$$\mathrm{g}\left(\mathrm{f}\right)=\mathrm{a}\mathrm{f}+\mathrm{b},\left[{\mathrm{s}}_1,{\mathrm{s}}_2\right]\to [{\mathrm{t}}_1,{\mathrm{t}}_2]$$

(2)

$$\mathrm{g}\left(\mathrm{f}\right)=\left(\mathrm{f}-{\mathrm{s}}_1\right)\times \left({\mathrm{t}}_2-{\mathrm{t}}_1\right)\times \left({\mathrm{s}}_2-{\mathrm{s}}_1\right)+{\mathrm{t}}_1$$

(3)

$$\mathrm{a}=\frac{\left({\mathrm{t}}_2-{\mathrm{t}}_1\right)}{\left({\mathrm{s}}_2-{\mathrm{s}}_1\right)},\mathrm{b}={\mathrm{t}}_1-{\mathrm{s}}_1·({\mathrm{t}}_2-{\mathrm{t}}_1)·({\mathrm{s}}_2-{\mathrm{s}}_1)$$

(4)

3.3. Settlement Point Identification Model

The deformation of the ground elevation is the basis for the selection of settlement points. InSAR is a space-to-earth observation technology, which can calculate the topography and surface changes in the target area by using the complex image pairs observed by the radar at different times or different angles in the same ground area [23]. Coherence and atmospheric inhomogeneity limit the use of InSAR [24,25]. The PS-InSAR algorithm, by placing the core of the calculation on a part of the target with high coherence, can ensure that the target maintains stable scattering characteristics over a long time sequence [30], which is highly spatially and temporally independent. In addition, it can acquire data over long-time spans with sub-millimeter accuracy and is capable of round-the-clock, all-day monitoring. These PS points are generally distributed on some hard ground objects without vegetation coverage, so this technology is suitable for urban areas.

After inputting the prepared 15 scene images data into the SARscape software (v.5.6), a connection map can be generated. The central scene image is selected as the basis of the other 14 scene images for registration to eliminate the relative offset between the primary and auxiliary images caused by orbital errors.

Subsequent differential interference processing is performed on the enslaver and enslaved person images to obtain the interference image set based on the familiar master image. The differential interference atlas of the region is obtained by subtracting the DEM information and the interference atlas. Then, the amplitude dispersion index (ADI) threshold method is used to identify the point target with stable radar scattering. After the above processing, the remaining phase includes the deformation phase, the noise phase, the elevation error and the atmospheric delay phase. The SARscape software can eliminate factors affecting monitoring accuracies, such as terrain residuals, deformation, elevation, and climate errors. In order to obtain more accurate deformation information, points with an error greater than 1 mm were deleted in ArcGIS, and 9,998,423 PS points were obtained by processing satellite data. The settling velocity ranges from −41.06 mm/y to 7.93 mm/y. Figure 2 shows the PS-InSAR technology processing flow.

3.4. Crack Recognition Model

Convolutional neural networks can be used to build a deeper network with a smaller number of parameters, thus reducing the time and computational cost of the training process. VGG16, as a classical convolutional neural network architecture, consists of 13 conventional layers and 3 fully connected layers. It employs small 3 × 3 convolutional kernels and incorporates maximum pooling layers, presenting a relatively streamlined yet deep architectural design, as shown in Figure 3. Consequently, in our crack recognition model, VGG16 was chosen as the classifier to convert the crack detection problem into a binary classification problem and classify the image samples with cracks and no cracks in the image data set.

Viewing the VGG16 model as a classifier allows us to formalize the crack detection problem as a binary classification task, distinguishing between crack and non-crack image samples. Constructing a crack recognition model using VGG16 primarily involves three steps: data collection, data preprocessing, and model training. The first two steps have already been completed in the previous sections.

There are two types of data: the data collected by dispatching drones based on the settlement value, and the second type is specially prepared for model training to improve the universality of the crack recognition model. The size of the existing data set is 227 × 227, so the input structure of the VGG16 model is adjusted to 227 × 227. In order not to miss the boundary information, 0 is added to the periphery of the original image matrix. After the convolution operation, the pixel value will be reduced accordingly, making the resulting matrix smaller than the original matrix. Padding can solve this problem well and obtain the matrix by convolution consistent with the original matrix. Since it is a grayscale image, it is a 3 × 3 single-channel convolution kernel. The 64 convolution kernels are convolved to generate a 64-channel feature map. The activation function ReLU makes the output content non-linear. In addition, 64-channel 3 × 3 convolution kernels generate 64-channel feature maps, and the number of channels in the output feature map always equals the number of convolution kernels. During the training, the initial learning rate was set to 0.001, the SGD optimizer was used, and as for other parameters, momentum was set to 0.5.

In order to further reduce the parameters, after the maximum pooling, average pooling can be performed. The uniform output size is 7 × 7 × 512 feature images. After flattening, there are 25,088 parameters (7 × 7 × 512), and the last three layers are fully connected. After the 13-layer convolution operation, the output of the last convolution layer is flattened and then input into the fully connected layer. The fully connected layer can map 25,088 parameters to 4096 neurons. Finally, the SoftMax layer is used to determine whether the input image contains cracks. Model codes are available in the Crack Recognition Model for Supplementary Materials.

The training results of the model are shown in Figure 4. The convolutional network inputs 8206 images, 6566 images in the training set, and 820 images in the validation set. A batch-size has 64 images, and an echo has a total of 103 batch-sizes. All the data of training set are used for training, and epochs are set to 20, 30, 40, 50, 70, 80, 90 and 100 to find the best trained model. In this study, training loss, validation loss, training accuracy, and validation accuracy are utilized to calculate the accuracy of crack recognition model. When training for 20 epochs, as shown in Figure 4a, there is no convergence trend for training accuracy and verification accuracy. At the 80th epoch, as shown in Figure 4b, the training accuracy has not yet converged but the validation accuracy is close to convergence. As shown in Figure 4c–e, after 100 epochs of training, the training loss and validation loss are 0.046 and 0.042, respectively, the training accuracy of the model converges to 99.48%, and the validation accuracy converges to 98.66%, which is a sufficiently accurate model for subsequent use.

Figure 5a is a photo of a crack, and the feature image is extracted by convolution processing. Figure 5 shows the process of convolution processing and the feature changes from clear to fuzzy, from edge feature extraction to regional feature extraction. Figure 5c is the 0th layer of convolution, which extracts edge information and shows 30 feature images. Figure 5d shows that the second convolution layer further extracts the edge information, but it is gradually blurred. Figure 5e shows the regional features extracted by the fifth layer of convolution.

3.5. Crack Measurement Model

After the convolutional neural network classification, a series of image labels containing cracks can be obtained. However, the cracks in the image cannot be further analyzed, so we need to use digital image processing technology to extract and analyze the crack information in the image.

First, the Canny edge detection, a classic edge detection algorithm, is performed on the image that has been grayed and contrast-enhanced, which can extract useful structural information and significantly reduce the amount of data. After edge detection, five morphological operations are required to highlight the crack information and remove noise. In the first morphology, the core size is determined to be 3 × 3, and then the expansion and closing operations are performed. During the expansion operation, the closing operation can effectively remove the noise. However, the cracks and noise will be highlighted simultaneously, so obtaining a closing operation with acceptable results is difficult. In the second morphological operation, a closing operation was performed, the noise information was obviously removed, and the crack information was further highlighted. In the third morphological operation, the cracks are screened and the following takes place: (1) determination of whether the area of the crack is larger than the area; (2) if the width is less than the height, then the width and height are swapped. If the width is greater than the WHRation times of the height, and if the width is greater than 100 and meets 1 and 2, it is considered a crack. Then, the image is opened. The opening operation is to expand and then corrode, which is to connect the broken cracks. Finally, two corrosion operations were carried out. After five morphological operations, all cracks have been screened. Figure 6a,b compares the images before and after the five morphological manipulations.

After further extracting the crack′s shape, the crack′s length, width and area need to be calculated. In our study, the length and width of cracks refer to the length and width of the minimum bounding rectangle around the cracks, while the area represents the en-closed pixel area. But most of the current UAVs are equipped with monocular cameras and monocular cameras cannot capture the distance between the lens and the object, so this study installed a laser rangefinder on the UAV platform, and made the shutter and the laser rangefinder work at the same time, which can capture the distance between the lens and the object at the same time. Then, based on the measured distance, the parameters of the camera and the focal length formula, the actual length and width formula of cracks can be obtained as shown in the Formula (5). The area is the area surrounded by pixels.

$$D=\frac{(L-V)s}{VS}\times d^{\prime }$$

(5)

where D is the actual length of the crack, L is the distance obtained by the laser rangefinder, V is the focal length of the camera, S is the number of pixels on the long side of the image sensor, d′ is the number of pixels in the image, and s is the physical size of the long side of the image sensor. DJI 4RTK UAV is used in this experiment. The pixel size of the UAV is 2.41 μm, and the focal length of the camera is 8.8 mm.

Finally, the fracture information extracted from the experiment is covered in the original image, and the actual information output is saved in TXT format for experts to evaluate, as shown in Figure 6c. Model codes are available in the Crack Measurement Model for Supplementary Materials.

4. Results and Discussion

4.1. Study Case

In recent years, the settlement of Shenzhen city mainly occurs in the reclamation area and along the subway, especially near the reclamation area. Located in the Yuehai Street office, Nanshan District, Shenzhen University is one of the most prosperous and populous areas in Shenzhen. The Yuehai campus of Shenzhen University is built in the reclaimed area. Excavation activities for subways in such fragile soil conditions can potentially lead to the movement and alteration of underground soil, causing subsidence in soil layers and affecting surface stability. Additionally, the vibrations and impact forces generated during subway train operations may act upon nearby aging structures, thereby accelerating the formation of cracks. Furthermore, for China, while the actual lifespan of a subway system can indeed be influenced by factors such as construction materials, maintenance practices, and operational conditions, it is generally designed with a structural lifespan of 100 years. In such an extended timeframe, the monitoring of subsidence and cracks becomes an essential concern that cannot be overlooked. Therefore, the settlement of engineering facilities in this area needs more attention. To verify the effectiveness of the proposed settlement and crack monitoring model and explain its operation process, this study takes the Shenzhen University section of Shenzhen Metro Line 9 as an example for case analysis, and the analysis area is shown in Figure 7.

4.2. Settlement Analysis

Based on the settlement model built in the early stage, a 300 m buffer zone is built in ArcGIS for the Shenzhen University section of Metro Line 9. The purpose of establishing the buffer zone is to identify the engineering facilities that may be affected in the operation stage of metro, and to cut PS points with the buffer zone. From Figure 8, it is not difficult to find many engineering facilities with settlement risk in the study area. The PS point is divided into dangerous, abnormal, and safe points according to their settlement rate. The PS point with a settlement rate of more than −20.00 mm/y is called a dangerous point, the PS point with a settlement rate of more than −8.00 mm/y and less than −19.99 mm/y is called an abnormal point, and the PS point with settlement rate less than −7.99 mm/y is called a safe point.

Through ArcGIS and InSAR technology, the settlement points processed by Sarcape are imported into ArcGIS software (v.10.8.1), and the pre-arranged vector box is used to cut and screen the settlement data. As shown in

Table 5. Settlement statistics of engineering facilities along the Shenzhen section of Metro Line 9. PS point with a settlement rate more than −20.00000 mm/y is called a dangerous point; PS point with a settlement rate more than −8.000000 mm/y and less than −19.999999 mm/y is called an abnormal point; PS point with a settlement rate less than −7.999999 mm/y is called a safe point.

NO.	Metro Line	Facility Name	Settlement (mm/y)	Monitoring or No Monitoring	NO. of PS Points
1	Line 9	Zhigong Building	−25.554718	Yes	44
2	Line 9	Zhiteng Building	−27.472183	Yes	57
3	Line 9	Zhaonan Bridge	−28.976548	Yes	31
4	Line 9	Baishi Road	−35.676012	Yes	526
5	Line 9	Ligong Building	−6.054376	No	89
6	Line 9	Zhizhen Building	−29.384743	Yes	25
7	Line 9	Zhixin Building	−26.788934	Yes	53
8	Line 9	Zhiyi Building	−21.344345	Yes	45
9	Line 9	Southern Sports Square	−6.543322	No	27
10	Line 9	Binhai community	−18.432556	Yes	49
11	Line 9	Qiaoyuan dormitory	−19.563729	Yes	64

, a total of 11 dangerous and abnormal facilities are selected: Zhigong building, Zhiteng building, Zhixin building, Zhiyi building, Baishi Road, Zhaonan bridge, science and technology building, Binhai community, Qiaoyuan dormitory, Zhizhen building and southern Sports Square. According to the screening results, the point of maximum settlement rate is located in Baishi Road, which is −35.68 mm/y, and the engineering facility is located directly above Metro Line 9. At the same time, seven dangerous engineering facilities, two abnormal engineering facilities and two safety engineering facilities are selected, as shown in Table 5. In this case, 44 PS points were selected, and the maximum settlement rate was −22.55 mm/y.

4.3. Building Image Acquisition and Processing

The data collection objects in this chapter are the dangerous engineering facilities and abnormal engineering facilities in the 300 M of Metro Line 9 as the buffer zone. The data mainly include the crack photos of the dangerous engineering facilities and abnormal engineering facilities and the distance between the camera and the engineering facilities when the crack photos are taken. When collecting data, it is necessary to collect the vertical distance between the camera and the target. Therefore, the lens and the body need to be adjusted to face the target without tilting, because tilt shooting will cause higher errors. While shooting the crack, the laser rangefinder measures the distance in real-time. Figure 9 shows the flight path of the UAV when collecting data. Among them, red engineering facilities are hazard monitoring points, green engineering facilities are abnormal monitoring points, and blue engineering facilities are safety monitoring points. The solid orange line indicates the flight route, starting from the Civil Engineering College and ending at Baishi Road. A total of 256 images were collected, and the pixels of this batch of images were unified to 4864 × 3684 pixels. Then, the image is cropped, and the small images cropped from the same image will be placed in the same folder and processed for grayscale and image enhancement.

4.4. Crack Identification and Measurement

We can input the above-processed image data into the pre-trained convolutional neural network in the previous chapter, and then use the pre-trained parameters to perform recognition work to determine whether the input image contains cracks. If any image in the folder with the same place name is identified as having a crack, the entire engineering facility is considered cracked. Finally, the photos are divided into two cases: cracks and no cracks. After the convolutional neural network screening process, 146 crack photos were identified.

The elevation information and phase information provided by integrating PS-InSAR technology and DEM data can quickly screen out possible settlement points, which solves the problem of slow and incomplete detection of settlement phenomena in the past by manual inspection. In this case study, we used the technical framework proposed in the previous paper to find out the multiple settlement points in Shenzhen University and then carried out an artificial investigation of these abnormal areas. The results showed that different degrees of the settlement were found in the abnormal areas forming settlement cracks, which successfully verified the effectiveness of the settlement screening model. In addition, it took less than one hour from the beginning of finding the abnormal point to the final arrival at the site for confirmation, which greatly sped up the efficiency of settlement detection.

After identifying the areas of concern, we employed drones to capture vertical-angle images of the cracks in subsided structures, thereby preventing distortion of crack images. Detailed shooting information was also documented. These images will be input into the crack recognition model trained after a series of preprocessing. In order to make the model more versatile, we used crack images from different materials and different engineering facilities to train the model. The results show that after the new image is input into the model, the accuracy of the model is as high as 98.48% in the classification of whether there are cracks, which is much higher than the accuracy of the human eye to distinguish cracks. Among them, some are cracks in the exterior wall of high-rise buildings with a width of less than 7 mm, which are almost impossible to find via manual observation and cannot be quantified in high-rise buildings.

After the fracture was found in the fracture recognition model, the corresponding region of the fractured image was classified into the abnormal region and used as the input of the fracture analysis model to further extract the fracture information. As shown in Figure 10i, a total of 38 cracks of different sizes were found in this area, and the specific information of the cracks was output as a text document (Figure 10j), which was stored together with the processed image. The length of the longest crack is 5.7 m, and the width is less than 1 cm. This crack information helps to evaluate the damage degree to engineering facilities further.

Table 6. The results of the crack analysis.

Number of Connected Domains: 38
Total Number of Cracks Detected: 38
Crack Number	Position	Area	Length	Width
1	(1,588,569)	52,790.00	5757.00	9.17
2	(2,991,214)	12,237.00	1495.00	8.19
3	(2,111,170)	3042.00	450.00	6.79
4	(2,823,91)	810.00	137.00	5.91
5	(471,95)	679.00	107.00	6.35
6	(4,190,213)	20,580.00	1889.00	10.89
7	(4,537,139)	1494.00	101.00	14.79
8	(2,322,196)	1260.00	157.00	8.03
9	(755,330)	1293.00	170.00	7.61
10	(3,005,505)	899.00	164.00	5.48
11	(4,720,572)	2687.00	273.00	9.84
12	(750,564)	881.00	110.00	8.01
13	(3,800,888)	5835.00	786.00	7.42
14	(3,184,1096)	11,103.00	1535.00	7.23
15	(2,848,647)	926.00	111.00	8.34
16	(1,590,792)	2679.00	345.00	7.77
17	(4,814,766)	837.00	111.00	7.54
18	(480,850)	2457.00	261.00	9.41
19	(1,014,1255)	8104.00	1107.00	7.32
20	(312,1022)	1831.00	213.00	8.60

shows the results of the crack analysis.

5. Conclusions

In the study, an automatic settlement and crack monitoring model was proposed to automatically obtain the settlement conditions of engineering facilities along the subway in a large area. After analyzing the settlement, UVA is employed to inspect the appearance of the target engineering facilities and collect crack data. The CNN model was used to recognize the appearance inspection data of the target facilities, which can effectively identify the crack information of the target facilities in the complex environment. Finally, the selected crack images are inputted into the quantitative model to obtain the length, width and area information of the cracks.

In the study, we developed a settlement monitoring model with data from Shenzhen Metro Line 9 and Qianhai section of Metro Line 5 and verified its accuracy. In the study case, 11 dangerous and abnormal facilities were precisely identified, which verified the accuracy of the model. As for the crack recognition model, a dataset including a variety of materials was first built, containing a total of 8206 photos, which was divided into training, validation, and test sets according to the ratio of 8:1:1. Meanwhile, a VGG16 network architecture consisting of 13 convolutional layers and 3 fully connected layers was built. Through diversified data sources and scientific training methods, the versatility of the crack recognition model in different scenarios was improved, and multi-material crack monitoring in engineering applications was realized. The accuracy of the final model is as high as 98.48%, which is much higher than the accuracy of the human eye to distinguish cracks.

A settlement and crack monitoring model integrating PS-InSAR, convolutional neural networks and digital image processing methods for engineering facilities along the subway is proposed in the paper. The model realizes the automated monitoring along the subway and identifying and quantifying cracks in multiple materials in complex scenarios, which improves the monitoring efficiency, effectively reduces the cost of monitoring and site investigation, and protects the safety of practitioners. Meanwhile, the monitoring results also provide more useful references for expert decision-making. However, there are still some limitations in the settlement and crack monitoring model. Firstly, the functions of this model are to provide references for experts to make decisions, in which only the quantification of settlement cracks has been realized so far. Secondly, although partial automation of crack identification and quantification has been realized, there is no automation of drone control. Finally, the visualization can be further enhanced to help better understand the results of the inspection.