Automatic Damage Detection and Diagnosis for Hydraulic Structures Using Drones and Artificial Intelligence Techniques

Abstract

Large-volume hydraulic concrete structures, such as concrete dams, often suffer from damage due to the influence of alternating loads and material aging during the service process. The occurrence and further expansion of cracks will affect the integrity, impermeability, and durability of the dam concrete. Therefore, monitoring the changing status of cracks in hydraulic concrete structures is very important for the health service of hydraulic engineering. This study combines computer vision and artificial intelligence methods to propose an automatic damage detection and diagnosis method for hydraulic structures. Specifically, to improve the crack feature extraction effect, the Xception backbone network, which has fewer parameters than the ResNet backbone network, is adopted. With the aim of addressing the problem of premature loss of image detail information and small target information of tiny cracks in hydraulic concrete structures, an adaptive attention mechanism image semantic segmentation algorithm based on Deeplab V3+ network architecture is proposed. Crack images collected from concrete structures of different types of hydraulic structures were used to develop crack datasets. The experimental results show that the proposed method can realize high-precision crack identification, and the identification results have been obtained in the test set, achieving 90.537% Intersection over Union (IOU), 91.227% Precision, 91.301% Recall, and 91.264% F1_score. In addition, the proposed method has been verified on different types of cracks in actual hydraulic concrete structures, further illustrating the effectiveness of the method.

1. Introduction

Large-volume hydraulic concrete structures such as concrete dams must bear the loads caused by water pressure, temperature, and earthquakes. It may be affected by dry–wet cycles under the action of many unfavorable factors during the service process [1,2]. The tensile strength of concrete is low, and the complex working conditions and the material properties of concrete make hydraulic concrete prone to cracks [3]. The existence and development of cracks can lead to leakage and corrosion damage or cause dam failure, endangering the safety of people’s life and property [4].

At present, the dam-crack detection method at home and abroad adopts mainly the traditional manual inspection [5]. Not only is this method time-consuming and laborious, but also, it is difficult to reach many high structures. The inspection method using manual inspection can inspect only the cracks existing in the appearance of the structure, and it is difficult to find hidden cracks inside the dam body, and the judgment of the status of the cracks depends on subjective experience [6]. Moreover, the manual inspection of high dams still has certain safety risks. In recent years, dam safety monitoring systems have been introduced to achieve automatic sensing of structural changes [7,8]. At present, cracks in hydraulic structures such as concrete dams are generally monitored by in situ monitoring instruments such as joint gauges and strain gauges. Since the occurrence of cracks is spatially random and uncertain, the aging of internal observation instruments cannot be easy [9]. It is hard to monitor cracks with this conventional point monitoring method without a good solution.

In recent years, with the development of artificial intelligence and information technology, computer vision (CV)-based structural crack recognition and detection technology has been gradually applied to civil engineering operations and maintenance [10,11,12]. Inspection equipment represented by drones can be equipped with various customized cameras to carry out structural defect scanning inspection work [13,14]. CV-based technology has significant advantages such as a large image acquisition range, high spatial resolution, and short detection time. Due to the large size of the dam body and numerous ancillary buildings, the single-machine inspection task will generate a large amount of image or video data [15,16]. Because the occurrence of cracks is a probabilistic event, the proportion of information directly related to structural damage is extremely low. It is necessary to combine advanced digital image processing technology to extract information closely related to cracks from these data.

In recent years, artificial intelligence technology has been widely used in many practical engineering fields to replace labor and reduce costs [17,18]. Among them, the deep learning method represented by the convolutional neural network (CNN) is widely used in the structural damage identification of civil infrastructure. For example, Cha et al. [19] developed a vision-based method using a deep architecture of CNN for detecting concrete cracks without calculating the defect features. Khani et al. [20] proposed a novel crack detection framework that utilizes techniques from both classical image processing and deep learning methodologies. Kim et al. [21] developed shallow convolutional neural network (CNN)-based architecture for surface-concrete crack detection. However, most of the above research focuses on crack recognition and classification and targets detection, and few studies consider the feature extraction of hydraulic concrete structure cracks in drone-based application scenarios. Due to the interference of various factors in the process of drone aerial photography, there are many blurred and low-quality problems in the crack images collected, so it is necessary to combine deep learning and artificial intelligence algorithms.

In this paper, considering these limitations, this paper uses the Xception backbone network instead of the conventional kernels in DeepLab V3+, the internal convolution kernel of which uses depth-separable convolution, and the hole rate of the convolution kernel can be set by itself. The adaptive attention mechanism module is embedded into the backbone network by utilizing the residual network and the channel attention strategy. The purpose of this module is to supplement the detailed information of the unit input feature map to the output feature map, to delay the trend of loss of detailed information.

The main contributions of this work are the following three points.

(1)

The Xception backbone-based crack automatic segmentation network achieves faster detection efficiency and few parameters because it uses depthwise separable convolution for its internal convolution kernel, and the hole rate of the convolution kernel can be set by itself.

(2)

The combination of the attention mechanism module and the Deeplab V3+ backbone network can significantly improve the accuracy of the model for identifying small-scale concrete cracks.

(3)

The proposed method shows strong crack pixel-level detection performance on a variety of different types and background roughness crack images.

The rest of the paper is organized as follows: Section 2 presents an overview of the methodology developed in this paper, including the Deeplabv3+ neural network. Section 3 details the experimental setup, and the experimental results are presented and discussed in Section 4. Conclusions are provided in the final section of the paper.

2. Methodology and Materials

2.1. The Deeplabv3+ Semantic Segmentation Network

The structure diagram of the classic DeepLabv3+ network model is shown in Figure 1. The DeepLab series is developed by Google and is currently a widely used image segmentation model, using an end-to-end training method [22]. DeepLabv3+ consists of two parts, including the Encoder and Decoder at the encoding end. The backbone network at the encoding end selects the Xception model. The ASPP (Atrous Spatial Pyramid Pooling) is designed to combined with atrous convolution can expand the receptive field of the convolution kernel without losing resolution (no downsampling) [23]. Together with the ASPP module as a feature extraction module, the feature maps (Feature maps) from Xception in the backbone network are divided into two branches: one is the feature map output by the last layer of the backbone network, which is sent to the ASPP module; the other part is the backbone network output from the middle layer. The shallow Feature maps are sent to the Decoder module, and the Decoder at the decoding end introduces the shallow Feature maps and fuses the high-level semantic features output by the ASPP module. This makes the two input feature maps through convolution (Conv) or upsampling (Upsample) operations. If the size is the same, then use Concat to splice them together and send them to a set of 3×3 convolution blocks for processing. Finally, perform linear interpolation and upsampling again to obtain an output segmentation result image with the same resolution as the original image.

2.2. The Xception Backbone

Considering that devices such as drones are difficult to achieve high-performance computing, deep learning needs to be improved to improve its real-time detection capabilities [24]. Considering the performance of the device, this paper uses the Xception backbone network, the internal convolution kernel of which uses depth-separable convolution, and the hole rate of the convolution kernel can be set by itself. The improved Xception-65 network has fewer parameters than the ResNet backbone network [25], showing the improvement of the DeepLabv3+ neural network, as shown in Figure 2. Equation (2) is to illustrate the introduction of hole convolution, which significantly reduces the amount of model weight parameters. The specific calculation formula is as follows:

$$W1=K*K*C1*C2$$

(1)

$$W2=K*K*C1+1*1*C1*C2$$

(2)

where the size of the ordinary convolution kernel and the depth-separable convolution kernel with holes are both $K*K$, the number of channels of the input feature map is $C1$, the number of channels of the output feature map is $C2$, and the parameter quantity generated after the convolution of the ordinary convolution kernel and the feature map is $W1$.

Figure 2 shows the flowchart of the improved crack pixel-level segmentation and quantification framework. It can be seen that on the left is the calculation process of encoding and decoding construction, and on the right is the detailed composition of each component. Inspired by the above research methods, for the current Deeplab V3+ semantic segmentation framework, it is easy to cause the loss of detailed information or small targets. In this paper, the adaptive attention mechanism module is embedded into the backbone network by utilizing the residual network and the channel attention strategy. The purpose of this module is to supplement the detailed information of the unit input feature map to the output feature map in order to delay the trend of loss of detailed information.

The method proposed in this paper consists of the following four parts. The method of using channel compression is inspired by the above research methods and aims at the problem that the current Deeplab V3+ semantic segmentation framework is prone to loss of detailed information or small targets. In this paper, the adaptive attention mechanism module is embedded into the backbone network by utilizing the residual network and the channel attention strategy. The purpose of this module is to supplement the detailed information of the unit input feature map to the output feature map to delay the trend of loss of detailed information.

Figure 3 shows the comparison of the standard convolution process and the depthwise separable convolution process. It can be seen from Figure 3a that the process of convolving an ordinary convolution kernel with a feature map, and Figure 3b shows the process of convolving a depthwise separable convolution kernel with a hole and a feature map [26].

2.3. The Adaptive Attention Mechanism

Most of the cracks in hydraulic concrete structures are tiny and hidden in complex backgrounds, making them difficult to identify [27]. In image semantic segmentation, it is usually necessary to obtain high-level semantic information about the image to strengthen the understanding of the whole image. However, in the down-sampling process of obtaining the high-level semantic information of the image, the detailed information of the image will inevitably be lost, especially for some small objects. The number of pixels occupied in the image is small, and it is easy to lose in the process of multiple downsampling. However, this detailed information and these small objects will exist in the channels of low-level feature maps.

In this paper, the residual network is used in combination with the channel compression method and the attention mechanism to build an adaptive attention mechanism module, and it is embed into the Xception backbone network. Specifically, the attention mechanism module is embedded in the input layer, middle layer, and output layer of the Deeplab V3+ backbone network, and a weight value is introduced to multiply each attention mechanism module to achieve the purpose of constraining the attention mechanism module. First, a global average pooling process is performed on the input feature map of the attention mechanism module. Then, after that, a 1 × 1 × C feature map is output, where C is the number of channels, which is the same as the channel number of the output feature map of the residual block. Equation (3) is to illustrate the calculation process of the residual block output feature map to obtain richer high-level semantic information. The specific calculation formula is as follows:

$$g_C=\frac{1}{H^{*}W}{\displaystyle \sum _{i=1}^H{\displaystyle \sum _{j=1}^W{G}_C}}(i,j)$$

(3)

where $g_c\in R^c$ from a feature graph G of size H × W and number of channels C, C (i, j) generated; (i, j) represents the coordinates of the pixel on the feature map.

Based on the above-mentioned methods, the calculation process of the self-attention mechanism is described. Firstly, the attention mechanism module is embedded in the input layer, middle layer, and output layer of the Deeplab V3+ backbone network, and a weight value is introduced to multiply each attention mechanism module to achieve the purpose of constraining the attention mechanism module. Secondly, the Deeplab V3+ embedding the attention module is trained on the constructed concrete crack dataset, so as to manually obtain the weight value (empirical value) of the attention mechanism module; then, the input layer, the middle layer, and the output layer are explored. Finally, the weight value of the attention mechanism module is changed to automatically update by backpropagation, so as to obtain the optimal weight value of the attention mechanism module and the optimal segmentation model.

2.4. Evaluation Indicators

In this study, three evaluation metrics are used to calculate the model’s ability to crack segmentation, including recall, precision, and F1_score. The details about these evaluation indicators are shown as follows:

$$\mathrm{Recall}=\frac{TP}{TP+FN}$$

(4)

$$\mathrm{Precision}=\frac{TP}{TP+FP}$$

(5)

$$F1\_\mathrm{score}=\frac{2\cdot \mathrm{Precision}\cdot \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

(6)

where TP is the True Positives, FN is the False Negative, and FP is the False Positives.

To quantitatively evaluate the effect of crack identification, an evaluation index of intersection ratio, which is called the Intersection over Union (IOU), is introduced. Figure 4 demonstrates the visual display of IOU indicators.

3. Experimental Setup

3.1. Project Description

Figure 5 shows the overall layout of the water conservancy project. This project is composed of a dam, a flood discharge structure, a power station, a navigation structure, and an irrigation canal head. This project is located in the middle reaches of the Huangbai River East branch of the first tributary of the Yangtze River. The rainwater collection area above the dam site is 862 km², the annual average rainfall is 1150 mm, and the annual average runoff is 388 million m³. The total storage capacity of the reservoir is 196 million m³. It is a large (2) type reservoir that focuses mainly on water storage and irrigation and also has comprehensive benefits such as flood control, power generation, and urban water supply. The reservoir is the main water source for farmland irrigation, and the average annual power generation of the power station is 55.1 million KW.h.

3.2. Drone Inspection

The project developed a set of high-precision UAV inspection systems for dam surface, which applies to dam surface from the aspects of dam surface inspection path planning and independent safety inspection control, dam surface defect identification, and health diagnosis system construction. Figure 6 shows the drone techniques used for dam inspection tasks.

Table 1. The related parameters of drones and cameras.

Parameters	Values
weight	570 g
size	180 × 97 × 74 mm (fold); 183 × 253 × 77 mm (stretch)
flight time	34 min
maximum ascent speed	4 m/s
maximum descent speed	3 m/s
maximum horizontal flight speed	19 m/s
maximum flight altitude	500 m
maximum wind resistance rating	Level 5 wind
maximum transmission distance	10,000 m
maximum supported storage	8 G (airborne memory) + 256 GB
camera pixels	3840 × 2160 pixels
equivalent focal length	24 mm
aperture	f/2.8
angle of view	84°

demonstrates the related parameters of drones. Independent inspection of the development of the prototype. Before carrying out the inspection work, drone path planning is carried out according to the actual layout of the dam, so as to realize the shooting of the whole range of the dam. The relevant parameters of the drone equipped with the camera are shown in Table 1. In the data post-processing stage, real-time return of on-site data such as ultra-high-definition images, cloud computing, artificial intelligence in-depth data analysis, graphic image detection, and other technologies to implement intelligent risk diagnosis, realize independent inspection of the dam surface, and seamlessly acquire honey image information on the dam surface.

3.3. Dataset Label and Generation

In this study, images of surface cracks of relevant hydraulic concrete structures were extensively collected through on-site shooting, report review, and network retrieval. These hydraulic structures include dam surface, face rockfill dam face, sluice pier, channel lining, and other hydraulic concrete structures. The original resolution of the collected images is high, and it is difficult to directly input the network for training. In this study, the original acquired image is divided into 200×200 low-resolution small images for further model annotation and model training.

Figure 7 shows some images from the constructed hydraulic concrete structure crack dataset. Moreover, it can be seen from Figure 7 that crack images acquired from the hydraulic structures suffer from severe environmental interference noises, such as water stains, alkaline substances precipitation, etc. It is of great significance to carefully label the crack information and morphological features under the inference of these unfavorable factors.

Figure 8 shows the comparison of the crack image and refined annotation image. It can be seen from the figure that there are significant differences in the background of the cracks in the constructed hydraulic concrete structure crack dataset. There are both smooth concrete surfaces and rough concrete surfaces. As observed from Figure 8, the pixel-wise annotations of cracks include mainly two parts, including the original images and annotation images at the pixel level.

4. Result and Discussion

4.1. Model Training

All experiments were implemented based on Pytorch, and the operating system is Windows 10. The dataset is randomly divided into a certain proportion, with a total of 5000 images, which are divided into the training set, verification set, and test set according to the ratio of 6:2:2.

Table 2. Experimental hardware and software environment configuration.

Lab Environment	Configuration Details
Hardware environment	CPU: Intel i7-12700KF
	GPU: NVIDIA GeForce RTX 3070
	Memory:32 GB
Software environment	Window 10 system
Development environment	VS code
Model Computing Environment	Pytorch

shows the hardware and software environment in which the comparative experiment was run. Specifically, the central processing unit (CPU) is Intel i7-12700KF, and the graphics card is NVIDIA GeForce RTX 3070. Note that the proposed and compared methods are trained on the same computing station. The number of model batches is uniformly set to 8, and the number of training rounds (Epoch) is set to 100. The learning rate is set to 0.0005.

Figure 9 shows the changes in the loss functions in both the training and validation sets during 100 iterations. As can be seen from the figure, with the increase in the number of iterations, the loss function of the model on the training set and verification set shows a gradual and smooth downward trend, indicating that the model effectively learns the rules from the dataset. After 100 iterations, the loss function of the model tends to converge, indicating that the model is fully trained.

4.2. Ablation Experiments

In this study, a series of ablation studies were implemented to analyze the importance of these proposed components.

Table 3. Contribution comparison of different components.

Xception Backbone	The Adaptive Attention Mechanism Network	IOU	Precision	Recall	F1
		74.820	84.350	82.170	83.246
√		86.094	90.178	81.070	85.385
	√	71.640	82.000	77.988	79.948
√	√	90.537	91.227	91.301	91.264

shows the contribution comparison of different improvements of the proposed method. It can be seen from the table that the detection effect of the crack identification model applying the improved backbone network and the attention enhancement mechanism on the test set is significantly better than that only applying the improved backbone network and the attention enhancement mechanism.

4.3. Model Comparison with Other Algorithms

In this study, the proposed method is compared with several advanced semantic segmentation methods to further verify the model performance. On the proposed hydraulic structure crack dataset, the method is compared with other state-of-the-art (SOTA) semantic segmentation methods, including Unet [28], Deeplab V3+, fully convolutional network (FCN) [29], and Canny [30]. All implementations of the SOTA algorithms are based on the published paper and fine-tuned hyperparameters using only the developed crack dataset.

Table 4. Comparative evaluation of crack detection results of different segmentation methods.

Models	IOU	Interference Speed/s
Proposed method	90.537	0.0192
Deeplabv3+	74.820	0.0224
FCN	78.204	0.0215
UNet	82.030	0.0201
Canny	69.242	0.5

shows the comparative evaluation of crack detection results of different segmentation methods. Figure 10 demonstrates the crack detection effect of the proposed method and other SOTA algorithm. It can be seen from the figure that the crack identification effect of the proposed method is better than that of the digital image processing method. Specifically, the recognition result of the proposed crack detection method is smooth and continuous, whereas the traditional digital image processing method has fracture and discontinuity. Compared with other DL-based methods, the proposed method tends to have fewer fractures in the recognition results, which is more in line with the real crack distribution.

It can be seen from Table 4 and Figure 10 that the proposed method using the improved backbone network and attention mechanism has stronger feature extraction and crack identification capabilities and has achieved better performance on the test set. Specifically, the introduction of the Xception-based backbone network effectively reduces the number of model calculation parameters, which effectively improves the inference speed of the model on a single crack image with a resolution of 200 × 200. In addition, the deep learning method significantly outperforms traditional image processing algorithms in crack image inference efficiency.

4.4. Test Result Visualization

To evaluate the recognition effect of the proposed method in crack segmentation better and effectively, three kinds of crack images containing different backgrounds and crack shapes are selected to evaluate the identification performance effect. Figure 11 demonstrates the identification effect of the proposed method on different types of cracks for hydraulic concrete structures. It can be seen from the figure that the proposed method has achieved good performance on different types of cracks in the test set, and the results of neural network segmentation and crack identification are basically consistent.

5. Conclusions

Crack detection for hydraulic concrete structures is of great significance to ensure the safety of dam operation. At present, the manual inspection method mainly used in crack detection is not only inefficient but also high-risk. Moreover, the traditional image processing method has a slow detection speed, low degree of automation, and weak generalization ability. Considering these limitations, this paper develops an improved Deeplab V3+ network via the Xception backbone and the adaptive attention mechanism network for the crack identification and segmentation of hydraulic concrete structures. The proposed method can effectively identify cracks of different types and background complexities. The specific contributions of this study are as follows.

(a)

The experimental results show that the proposed method can realize high-precision crack identification, and the identification results have been obtained in the test set, achieving 90.537 IOU, 91.227 precision, 91.301 recall, and 91.264 F1_score.

(b)

The fusion of a lightweight backbone network and attention mechanism can improve the accuracy of model crack identification and improve the speed of model crack detection.

(c)

The proposed method can effectively identify different types of s types of cracks in hydraulic concretes. It can be seen from the experimental results that the proposed method has a good recognition effect on wide, narrow, transverse, and longitudinal cracks.

However, some issues require further elaboration and clarification. The types of defects in hydraulic concrete structures are highly complex and diverse. In addition to cracks, there are also many types of defects such as spalling, collapse, and pitting. Therefore, it is necessary to combine deep learning and semantic segmentation network models to further study the identification methods of different types of defects. In addition, it is necessary to further combine aerial photography technologies such as drones to study automatic identification methods for defects to guide drone navigation.