Detection and Identification of Expansion Joint Gap of Road Bridges by Machine Learning Using Line-Scan Camera Images

Abstract

Recently, the lack of expansion joint gaps on highway bridges in Korea has been increasing. In particular, with the increase in the number of days during the summer heatwave, the narrowing of the expansion joint gap causes symptoms such as expansion joint damage and pavement blow-up, which threaten traffic safety and structural safety. Therefore, in this study, we developed a machine vision (M/V)-technique-based inspection system that can monitor the expansion joint gap through image analysis while driving at high speed (100 km/h), replacing the current manual method that uses an inspector to inspect the expansion joint gap. To fix the error factors of image analysis that happened during the trial application, a machine learning method was used to improve the accuracy of measuring the gap between the expansion joint device. As a result, the expansion gap identification accuracy was improved by 27.5%, from 67.5% to 95.0%, and the use of the system reduces the survey time by more than 95%, from an average of approximately 1 h/bridge (existing manual inspection method) to approximately 3 min/bridge. We assume, in the future, maintenance practitioners can contribute to preventive maintenance that prepares countermeasures before problems occur.

1. Introduction

The Korea Expressway Corporation is an institution that builds and maintains expressways in Korea, and the number of highway bridges in use reached about 9800 as of 2020. Recently, the number of narrow expansion joint gap occurrences has been increasing. It increased the most in 2018, when the number of days of summer heat waves was the greatest (see Figure 1,

Table 1. Average annual temperature, average annual maximum temperature, and number of heatwave days (national).

Year	2011	2012	2013	2014	2015	2016	2017	2018	Recent Average (2011–2018)	Average Year (1981–2010)	Difference	Ratio
Average Temperature (°C)	24.0	24.7	25.4	23.6	23.7	24.8	24.5	25.4	24.5	23.6	+0.8	3.8%
Maximum Temperature (°C)	36.7	38.7	39.2	37.9	38.7	39.6	39.7	41.0	38.9	37.5	+1.4	3.7%
Number of Heatwave Days (days)	14	15	18	6	10	22	14	32	14.2	9.8	+4.4	45%

). If such a lack of expansion joint gap occurs, it may adversely affect the structural behavior of the bridge and can be a major threat to traffic safety [1]. Considering the characteristics of the climate of Korea, which has four distinct seasons, we can see that the importance of maintenance and the role of the bridge expansion joint responding to temperature changes between the cold and hot seasons is growing very high.

The causes of narrow expansion joint gap occurrences vary. Representatively, there are construction errors, inappropriate pre-setting, deformation of backfill, and alkaline silica reaction; recently, abnormal high temperatures have also emerged as one of the causes. The year 2018 had the highest number of heatwave days since 2013, and accordingly, the number of cases of lacking bridge expansion joint gaps and pavement blow-up damage rapidly increased. The Korea Expressway Corporation conducted a complete survey of bridges under its management and found that narrow expansion joint gap occurred in 276 bridges (2.96% of the total of 9334 bridges).

Table 2. Analysis of causes of insufficient expansion joint gap.

	Sum (Bridges)	Major causes
		Expansion of cement concrete pavement ①	Deformation of backfill ②
	276	166 (60%)	110 (40%)

presents the main causes of the occurrence of narrow expansion joint gaps as analyzed through onsite investigations [2].

Most importantly, after bridge expansion joint damages occur, a huge budget is required for maintenance, and restoration is virtually impossible unless the bridge is remodeled. Experience shows that the life-cycle costs of a bridge expansion joint, over the life of the bridge, are many times greater than the initial costs of supply and installation, especially when consequential impacts such as traffic disruption during replacement works are considered [3].

Therefore, to solve these problems, the overall design/materials/construction/maintenance should be re-examined to supplement the relevant standards, but this has the problem of consuming a large amount of budget and time. Gaining a full understanding of the demands on the bridge’s expansion joints, and how well they are performing, can enable adjustments and maintenance measures to be tailored to maximize the length of the service life with a minimum of maintenance effort [4,5,6,7].

In this study, to detect problems and minimize the damage that may occur in the bridge expansion joint at an early stage, an optimal inspection method was developed by converting a vast number of bridge expansion joint gap maintenance inspection methods from existing manual inspections to automated inspections. To this end, it was decided to incorporate an M/V technique to switch from the existing manual inspection method of the bridge expansion joint device to a precise high-speed inspection method.

M/V refers to the technology and methods used to provide imaging-based automatic inspection and analysis for such applications as automatic inspection, process control, and robot guidance, usually in industry. It refers to many technologies, software and hardware products, integrated systems, actions, methods, and expertise.

Definitions of the term M/V vary, but all include the technology and methods used to extract information from an image on an automated basis, as opposed to image processing, where the output is another image. The information can be used for such applications as automatic inspection, robot and process guidance in industry, security monitoring, and vehicle guidance [8,9,10]. This field encompasses a large number of technologies, software and hardware products, integrated systems, actions, methods, and expertise [10,11]. M/V is practically the only term used for these functions in industrial automation applications. It attempts to integrate existing technologies in new ways and apply them to solve real-world problems in a way that meets the requirements of industrial automation and similar application areas [10,12].

The primary uses for M/V are imaging-based automatic inspection and sorting and robot guidance [13,14]. The imaging device (e.g., camera) can either be separate from the main image processing unit or combined with it, in which case the combination is generally called a smart camera or smart sensor [15,16]. While conventional (2D visible light) imaging is most commonly used, alternatives include multispectral imaging, hyperspectral imaging, imaging various infrared bands, line scan imaging, 3D imaging of surfaces, and X-ray imaging [13]. We used the method of line scan imaging, which we think is most effective in technology that can automatically acquire images while driving at high speeds. [17,18,19,20].

One of the purposes of the research is to use a line-scan camera in M/V to record road surfaces as images with a resolution of 1.0 mm per pixel in a high-speed (100 km/h) environment. Existing line-scanning cameras are used for the purpose of identifying counterfeit bills and inspecting semiconductor wafers’ surfaces on belt conveyors, so they can be said to be the best technology for introducing to fast-moving vehicles on roads [21,22]. This is suitable for this study because it is free from the problem of lens distortion that may occur in an area-scan camera [23]. According to this study, we developed a survey system equipped with line-scan cameras, Global Positioning System (GPS), and a distance measurement instrument sensor (DMI) in the vehicle, and as a result, we could obtain a safe and accurate image without blocking the road [24,25,26,27,28].

Recently, deep-learning-based approaches [29] have been applied to many problems in various industrial and academic fields. Visual recognition tasks, which extract information of interest from images, such as image classification [30], object detection [31], and semantic segmentation [32], have been actively studied. In particular, CNN (convolutional neural networks) have shown successful results in many visual recognition applications. In this study, models based on CNN and some deep learning techniques are applied to image analysis problems to determine the distance between expansion joints. Two CNN models are used to analyze the distance step by step. First, image classification categorizes images semantically into sub-groups. An image-classification CNN extracts important image features like texture cues and shape cues from each image, and a logistic regression [31] distinguishes them between categories based on the extracted features. Successful studies on image classification CNNs have been conducted focusing on the ImageNet benchmark dataset [33], and some design patterns have been proposed to achieve technical goals such as learning more complex patterns (ResNet) [34], light model weights (MobileNet) [35], and efficient scalability (EfficientNet) [36]. These design patterns, called CNN architectures, are applied to general image classification to reduce the need to design new models every time. Second, semantic segmentation is the process of classifying each pixel in an image belonging to a particular class. The recent success of CNN has also driven outstanding progress in semantic segmentation [37,38,39,40,41,42,43,44,45,46]. Based on CNN architecture, the semantic segmentation model extracts local contexts (a small area centered in a pixel to classify) and global contexts (overall semantics of input image) from an image and reconstructs contexts to create a class heatmap of the same size as the original image, which shows the probability of which class each pixel is. Therefore, it can be thought of as a classification problem for each pixel, considering local and global contexts.

Another purpose of the study is to develop software that measures gap width with more than 90% accuracy in the road image of the survey so that many bridges can be monitored quickly. Therefore, an algorithm was developed to measure the gap width after first finding the expansion point in the road image using machine learning. As a result, the identification accuracy was more than 95%, and the investigation time was reduced by more than 95%, from an average of about 1 h/bridge (the inspector’s existing manual inspection method) to about 3 min/bridge.

The main contribution of this study is to demonstrate the possibility of developing smart maintenance techniques for road structures, in other words, the successful development of a smart maintenance system that combines machine vision and machine learning technology, for the following purposes: first, to ensure the traffic safety of vehicles on roads that are already in use; and second, to obtain the condition of road structures at the level the investigator wants with figures and accurate images.

The article is organized as follows:

• Section 2 introduces a system developed to survey the road surface while driving at high speed (100 km/h) with a line-scan camera and an M/V imaging device. It introduces operating equipment and explains the main functions and test results.
• Section 3 describes the adequacy review of pre-setting by surveying newly constructed bridge construction joints with standard computer vision methods applied to the initial system.
• Section 4 describes another detection mechanism that uses machine learning.
• Section 5 concludes the paper and proposes future work.

2. Development of Monitoring Technology for Bridge Expansion Joint Using Line-Scan Cameras

Recently, the Korean government has been actively encouraging the introduction of the latest structural safety inspection technologies through the revision of various laws and regulations. For example, Annex 10 of the Enforcement Decree of the Special Act on the Safety and Maintenance of Facilities was amended in March 2020 to newly establish a provision for “appearance investigation and image analysis using new technology or inspection robot, etc.”. In other words, new technology can be applied for structure inspection. Further, the revision of Article 167 of the Occupational Safety and Health Act emphasized the safety management of internal employees by strengthening the punishment standards for employers (a maximum fine of $100,000 USD) in case of workplace accidents caused by negligence in worker safety management [1,2].

As a result, the structure management conditions of the Korea Expressway Corporation are becoming increasingly strict. The establishment of facility and performance evaluation following the revision of the Special Act on Safety and Maintenance of Facilities newly necessitated the performance evaluation of 2611 Type 3 facilities (bridges) and 4626 structures. As a result, structural inspectors performed 25,219 regular safety inspections, 2205 precise safety inspections, and 258 precise safety diagnoses in 2020, including in-house and external services. A shortage of management personnel is arising owing to this excessive workload (see

Table 3. Number of structural inspections (2020).

Safety Management	Sum	Bridges	Tunnels	Box Culverts
Sum (EA)	27,682	15,636	2118	9928
Regular safety inspection 1	25,219	13,648	1643	9928
Precision safety inspection 2	2205	1783	422	-
Precision safety diagnosis	258	205	53	-

¹ Regular safety inspection: an average of approximately 450 EA/yr/branches performed by the structural staff. ² Precision safety inspection (2020): self-manpower performance (684EA), performing external services (1521EA).

).

Given that the stricter structure management conditions are imposing greater demands on manpower, it is crucial to promote inspection methods and smart technologies that can replace manpower.

The Korea Highway Corporation Road Traffic Research Institute analyzed past safety inspection and technical advice data and found that 78.3% of expansion joint devices that had to be replaced lacked a gap. Since 2016, we have been researching monitoring methods to preemptively maintain a suitable gap between expansion joints [2].

For this purpose, by integrating high-speed line-scan cameras, which are widely used in various fields such as semiconductor inspection and road pavement investigation, image-processing technology to determine the length of expansion and contraction through AI-based (i.e., machine learning) image sensing in the analysis process, and automatic control technology, we developed the Nonstop bridge EXpansion joint gap measuring Utility System (NEXUS) to measure the distance between expansion joints with 1.0 mm resolution while driving at a high speed of 100 km/h [1] (see Figure 2).

Existing methods manually measure the gap in highway bridge expansion joints through traffic control and partial blockage. By contrast, the proposed method uses a high-speed line-scan camera mounted on a vehicle driving at 100 km/h to acquire an image across a 40 cm-wide car lane. This system uses the geographical information system data of the bridge (above, longitude coordinate system) to perform automatic measurements in conjunction with the mounted GPS while driving at high speed and thereby creates a database of the expansion gaps based on accurate survey images without affecting the traffic flow. In this study, a test survey was conducted on approximately 5000 bridges along the highway, and the analysis results were used for big-data-based machine learning for developing algorithms to accurately determine the length of the expansion joint gap depending on its type and site conditions (see Figure 3) [45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62]. The NEXUS system and the on-site test survey introduction is available on our YouTube channel [63].

The use of the NEXUS for performing automatic surveys of the bridge expansion joint device gap while driving at high speed reduces the survey time by more than 95%, from an average of approximately 1 h/bridge (existing manual inspection method) to approximately 3 min/bridge. In addition, if the accumulated gap is monitored and preemptive maintenance is performed before gap narrowing occurs, it can eliminate the risk factors in the future temperature expansion behavior of the bridge and contribute to traffic safety and cost reductions.

3. Initial Gap Measurement and Evaluation of New Bridge Expansion Joint Device

Previously, it was difficult to use data when necessary, as they were managed in the form of reports and fields. By contrast, by using the NEXUS device, initial data can be accumulated, and the adequacy of highway bridge expansion joints can be checked and evaluated online at any time. To confirm this, we tested the initial data for the adequacy of the pre-setting after 300 days, when the initial drying shrinkage and creep deformation roughly converge; in this regard, it can be beneficial to use the initial gap database (D/B) for the three routes completed at the end of 2016. As the pre-setting of the expansion joint device is not consistent with the daily average temperature during winter and summer at the time of construction, it is essential to adjust the gap in advance such that it has an intermediate value at the reference temperature (average daily temperature of 15 °C).

The route to be investigated has 302 expansion joint devices of three types, namely, rail type, steel finger type, and mono cell type (see

Table 4. Bridge expansion joint devices installation status.

Division	Total	Rail Type	Steel Finger Type	Mono Cell Type
Total	302 places	169 places	128 places	5 places
	100%	56%	42%	2%
lane A	171 places	98 places	71 places	2 places
	100%	57%	42%	1%
lane B	131 places	71 places	57 places	3 places
	100%	54%	44%	2%

). We converted the expansion gap and average temperature (obtained from the Korea Meteorological Administration) at 15 °C during the investigation and evaluated the adequacy of the pre-setting. After analyzing the initial values of the new bridges, we investigated some bridges that were not properly preset, potentially due to lacking gaps during summer (35–40 °C) or excessive gaps during winter (−15 °C) (see Figure 4 and Figure 5).

For example, the design temperature range Δ T is from −5 °C to 35 °C in the prestressed concrete girder type expansion bridge design movement calculation standard. Therefore, the reference temperature for pre-setting can be seen as 15°C, which is the median temperature. In fact, the temperature at the time of construction of the new joint is under various temperature environments, not 15 °C, so pre-setting is performed at the time of installation to increase or decrease the spacing at the time of installation. The y-axis “Percentage of the capacity of the new joint” is to ensure that pre-setting was performed properly at the time of construction:

(1)

After examining the joint gap and the average daily temperature on any day,

(2)

The joint gap converted into the reference temperature of 15 °C is expressed as a percentage of the capacity of the new joint.

This value means the following:

(a)

If it is close to 50%, it means that it is installed in the middle of the absolute value of the joint gap. (If it is an expansion and contraction joint with a capacity of 100 mm, it represents 50 mm when the joint gap is 15 °C, which is the reference temperature.)

(b)

If it is near 10%, it means that it is installed at a small value of the absolute value of the joint gap. (In the case of an expansion joint with a capacity of 100 mm, it shows 10 mm when the joint gap is 15 °C, which is the reference temperature, so the joint gap is insufficient in summer.)

(c)

If it is near 90%, it means that it is installed at a large value of the absolute value of the joint gap. (If it is a 100 mm stretchable joint, it is 90 mm when the joint gap is 15 °C, which is the reference temperature, so the gap is exceeded in winter.)

As such, if it is possible to investigate and analyze the expansion joint device gap while driving, without blocking a separate route, from the beginning, and to convert it into D/B in the bridge management system, by monitoring the changing trend, it is possible to take preemptive measures in case of emergency (such as when the gap narrows in a short time).

4. Advanced Identification of Expansion Gap Using Machine Learning

Expansion joints device has different details for each development and sales company, and typical expansion joint devices constructed on road bridges in South Korea include mono cell type, finger type, and rail type.

Table 5. Expansion joint device installation status (2014).

Installation	Sum	Mono Cell Type	Finger Type	Rail Type	Others
EA	14,784	7793	1786	4228	977
Prop (%)	100	53	12	29	7

lists the types of expansion joint devices installed on highway bridges in South Korea as of 2014.

The analysis of the expansion joint gaps of 4821 bridges and 12,825 devices on the highway mainline from 2017 to 2019 using NEXUS equipment revealed various types of expansion joint devices and gap identification errors depending on the condition of the expansion joint device body installed in the field. False-detection errors include the case of erroneously identifying another gap in the expansion joint body as a gap and the case of detecting a specific part of the road surface (portholes, repair marks, cracks, etc.). On average, the gap identification accuracy was 67.5%; that for the finger type with a complex shape was the lowest at 51% (see

Table 6. Gap identification accuracy by type of expansion joint (2017–2019).

Discrimination (%)	Average (%)	Mono Cell Type	Finger Type	Rail Type	Others
Accuracy	67.5	71	51	87	61
Loss	32.5	29	49	13	39

, Figure 5). False detections were attributed to limitations of the traditionally coded image analysis algorithm. Therefore, the image identification method must be improved using machine learning, as it is superior to traditional coding.

We aimed to overcome the limitations of the traditional gap identification algorithm. Machine learning can be used to learn previously investigated images as big data, and to thereby identify the expansion joint device from an input image and more accurately determine the gap value. The input image obtained using the NEXUS equipment was recorded as multiple 10,000 × 1024 pixels images, depending on the irradiation distance. The image analysis system consisted of cascaded AI vision modules, in which the work process analyzed high-resolution input images for realizing effective calculations. The AI process involves the following steps: original image input > expansion joint area extraction AI network (classification) > expansion joint area image area (cropping) input > gap extraction AI (segmentation) > gap measurement (algorithm) (see Figure 6).

To measure the gap value, the position of the expansion joint device must be specified in the image. Further, for achieving high measurement accuracy, the expansion joint device needs to be recognized at the pixel level. This can be done using image segmentation methods used in computer vision. Further, a deep learning model with extensive data learning is used for problems with large changes in illuminance, angle, image shape, and noise generation, such as road environments.

The AI machine learning model for expansion joint device discovery was implemented to solve the classification problem using CNN as a feature extractor. The original image with a size of 10,000 × 1024 pixels was cut into an image patch with a size of 1000 × 1024 pixels to be inputted to the AI model with minimal distortion. Then, 19 image patches were generated for each line-scan image, and classification was performed to find the image patches in which the expansion joint device exists (see Figure 7).

To find the point with the minimum gap distance, it is necessary to precisely segment the gap region from the image patch where the expansion joint device appears. The AI machine learning model for gap region extraction used U-Net, a representative segmentation model. The first AI model generated and learned training data by labeling pixels in the corresponding area to extract metal parts with characteristic textures from the line-scan images of the expansion joint device. The second AI model was trained to find the gap region between the extracted joint devices (see Figure 8).

After obtaining a binarized image with the gap region and the background, we found the point with the minimum gap distance. The minimum gap distance in pixels was converted to the final gap value in millimeters by multiplying the information in millimeters per pixel.

4.1. AI-Based Image Analysis

Line-scan images obtained using NEXUS were converted into high-resolution digital images with a length and width of 10,000 and 1024 pixels, respectively. The unit length (mm) of the pixels can be calculated by considering the vehicle speed at the time of the shooting. If the expansion joint device can be measured in pixels, the actual value of the gap distance can be inferred within the error range. Therefore, recognizing the position of the expansion joint device in the line-scan image using a program and accurately measuring the size of the gap in pixels determine the measurement accuracy.

To perform highly accurate analyses, a large number of line-scan images were trained using deep learning. To efficiently analyze high-resolution input images, cascade-type analysis procedures were configured. The analysis involved the following steps: (1) ultrafast line-scan image input, (2) extraction of small image patches in a square matrix, (3) recognition of expansion joint device among the image patches, (4) expansion gap division, and (5) gap distance analysis and actual value calculation (see Figure 9).

4.2. Expansion Joint Device Recognition

Line-scan images have a high resolution of more than 10,000,000 pixels and require considerable memory and computational resources when being analyzed using a deep neural network. Therefore, it is necessary to divide the image into small image patches for realizing effective computations. Through the window sliding method, images were extracted every 1000 pixels in the longitudinal direction to generate a total of 19 image patches with a size of 1000 × 1024 (see Figure 10). Each image patch is again converted to a size of 224 × 224 pixels and subjected to binary classification.

Each image patch is again converted to a size of 224 × 224 pixels for binary classification. By using a CNN, an effective neural network for image analysis, we labelled patches with the expansion joint device as y = 1 and patches without the expansion joint device as y = 0 and classified them by logistic regression. The logistic regression followed by CNN infers the probability P that there is an expansion joint in the image patch. The error of the model prediction for each image patch is calculated with binary cross entropy [64,65,66,67,68,69], which compares each of the predictions to the actual class, which can be either 0 or 1. The errors of each set of training data are summed up. In stochastic gradient methods [70,71,72,73,74,75,76,77], the cost and sum of errors is used to update current model parameters to reduce the distance from the optimal point in the parameter space. The equation of binary cross entropy is shown as follows:

$$L = -ylogP-\left(1-y\right)log\left(1-P\right)$$

(1)

Because class imbalance exists in a ratio of 1:9, a weight is assigned to each class.

$$L_{weighted} = -ϵ\times ylogP-\mu \times \left(1-y\right)log\left(1-P\right)$$

(2)

where $ϵ=9.0 , \mu = 1.0$.

The cost function J is calculated by averaging the errors for N training data and adding the L2 regularization to reduce overfitting [78,79,80,81,82,83,84,85] of the model.

$$J = \frac{1}{N}{\sum }_{j=1}^N{L}_{weighted}^j+\mathsf{\lambda }{\left|w\right|}^2$$

(3)

where $\lambda ={10}^{-4}$.

The gradient descent method updates the model parameters in the direction of reducing the cost function J as follows:

$$w \Leftarrow w - \alpha \cdot g$$

(4)

where $\alpha ={10}^{-4}$, $g=\frac{\partial J}{\partial w}$.

We tested ResNet, MobileNet, and EfficientNet, all of which are well-known CNN architectures [22,23,24]. Gradient vanishing is more likely to occur as the layers of the deep learning model deepen, and ResNet solved this problem by performing residual learning using skip connection. The structure obtained high accuracy compared to the scale of the model. MobileNet proposed a depth-wise separable convolution that reconstructed the existing convolution method to reduce the computational amount of the model. Compared to the popular models at the time of proposal, the amount of computation was significantly reduced, and the same accuracy was maintained. EfficientNet, which empirically reports a methodology to increase model complexity to improve performance, updated the state of the art for benchmark datasets.

We trained the three models to a binary classification of expansion joints and non-expansion joints (see Figure 11). Through the test, the EfficientNet model showed the highest performance, with recognition accuracy of 97.57% for the expansion joint device. Further, it was better to start learning from randomized initial parameters than through transfer learning for ImageNet. This is because the analysis image has different characteristics from the general characteristics of ImageNet. We employed EfficientNet for expansion joint detection, as it has the highest accuracy.

4.3. Expansion Joint Gap Segmentation

Even if the image patch of the expansion joint is extracted from the line-scan image, image segmentation is required within the detected image to accurately measure the expansion gap. Image segmentation is a pixel-level classification that deduces the class each pixel belongs to (i.e., expansion joint device or background). A masking image representing the pixel corresponding to the expansion joint device can be generated by the image segmentation algorithm.

Figure 12 shows the image patch (left) and correct mask (right) of the expansion joint device. The neural network structure receives image patches as an input and performs binary classification of the individual pixels. The deep learning model receives image patches as the input and learns to predict a masking that is close to the correct answer. In this case, the class of a pixel should be determined by considering the global and local characteristics of the image rather than individual pixel values.

This study uses a U-Net model, which was previously developed for sophisticated analyses of organ lesions in the field of biomedical science [36]. U-Net is suitable for the problem of precisely detecting the shape of the analysis target by simultaneously learning the global and local information of the image. The structure of U-Net was modified to develop EfficientNet, a neural network that extracts image information.

U-Net infers the probability that an arbitrary pixel is an expansion joint device and learns the correct answer for each pixel from the correct answer masking. Therefore, the prediction error for all pixels M of the image patch expressed as BCE is as follows:

$$L_{patch}={\sum }^{ }{L}_{pixel}={\sum }_{i=1}^M\left(-y_i\log P_i-\left(1-y_i\right)\log \left(1-P_i\right)\right), i=1,2,3, \dots , N$$

(5)

The final cost function J is expressed by summing the mean error for N image patches and the L2 regularization term:

$$J=\frac{1}{N}{\sum }_{j=1}^N\left(L_{patch}^j+\lambda {\left|w\right|}^2\right)$$

(6)

where $\lambda ={10}^{-4}$.

The gradient descent method updates model parameters in the direction of reducing the cost function J as follows:

$$w \Leftarrow w - \alpha \cdot g$$

(7)

where $\alpha ={10}^{-4}, g = \frac{\partial J}{\partial w}$.

We compared the performance between U-Net’s masking image and the correct masking image in the test dataset.

Table 7. Gap identification accuracy by type of expansion joint (2017–2019).

Accuracy by Type of Expansion Joint (%)	Precision	Recall	f1-Score
Positive (pixels of expansion joints)	96.61	94.38	95.49
Negative (other pixels)	99.23	99.55	99.39

shows the pixel-level classification performance for the test set; the pixel precision of the expansion joint device was 96.61%, recall rate was 94.38%, and f1-score was 95.49%. The f1-score of the expansion joint device pixel detection was within 5%. In the post-processing process, by correcting the error of the predicted masking image of U-Net, the minimum gap point was detected, and the distance was measured.

4.4. Gap Distance Analysis Algorithm

The texture of the pixels in the expansion gap area to be measured appears somewhat irregular depending on the gap, foreign matter, and type of expansion joint device. Texture irregularity is a factor that makes it difficult to distinguish pixels in the expansion gap area.

The metal surface constituting the expansion joint device has a consistent texture compared to the gap region. This means that it is easier to extract the expansion joint device than the gap area. Therefore, to analyze the gap distance, the expansion joint device is extracted first, and the gap area is extracted again from the resulting image. Image segmentation using U-Net was applied to both area extraction processes (see Figure 13).

Because the U-Net output represents the probability that each pixel is a gap area, binarization is performed by applying a threshold value of 0.5. When defining a binarization function as a response, the formula to search the x-coordinate of the minimum gap is as given below. Note that the same applies to rail-type expansion joint devices.

$$\underset{1\le x\le 512}{arg min}{\displaystyle \sum }_{y=1}^{512}response\left(x,y\right)$$

(8)

The input and output of the U-Net model have the size of 512 × 512. By binarizing the final output probability map, we search for the x-coordinate with the smallest number of pixels with a value of 1. The number of pixels at the coordinates is the gap distance in pixels, and the actual gap distance is obtained by multiplying the distance value per pixel.

$$\left(distance\right)=\left(\# of pixel\right)*\left(\frac{mm}{pix}\right)$$

(9)

Figure 14 shows the pseudocode for obtaining the minimum gap distance from the output probability map of U-Net.

4.5. Gap Identification Verification

Based on the abovementioned results of AI-based gap identification, we randomly selected 10,526 places among 12,825 expansion joint device big-data images obtained previously to determine the discrimination of the expansion joint device gap. After dividing and refining 10,526 line-scan images into 19 image patches, 289,495 sets of training data and 45,950 tests of the classification model were constructed. A total of 21,604 sets of training data and 4174 of test data of the segmentation model for measuring the expansion joint gap were refined. The results are presented below for each expansion joint device type. The result of the position where the minimum spacing was measured is indicated by a red line. For rail-type joints in which several gaps appear at once, the starting and end gaps of the part with the smallest actual gap value are indicated by red lines (see Figure 15).

We used Python 3 and TensorFlow 2 to implement and train a deep learning model using CNN, and development frameworks such as TensorFlow and PyTorch support libraries for implementing popular CNN layers and to support learning using GPUs.

We used a single NVIDIA Tesla V100 graphics card and Tensorflow to accelerate the training of the model. The EfficientNet B0 model for classification of expansion joints completed training in less than 30 epochs and took up to 4 h. A total of 259,495 training images and 30,000 validation images were used. Overall, 45,950 images for testing did not participate in the training. The U-Net model for gap region extraction completed training in less than 20 epochs and took up to 4 h, and 19,304 training images and 2300 validation images were used, while 4174 images for testing did not participate in the training.

In the environment for field application after training, one NVIDIA RTX 3070 graphics card and Tensorflow were used. In one line-scan image (1024 × 10,000), it took an average of 0.6 s to find the expansion joint and measure the gap.

After comparing the identification rate results from the existing traditional algorithm, we found that the finger type, which had the lowest discrimination accuracy, improved the most, by 41% to an accuracy of approximately 92%, and the overall average identification accuracy improved by 27.5% to 95%. Even though the identification accuracy was greatly improved, if a foreign substance existed inside the expansion gap, if a sealing agent was applied to the surface for preventing water leakage, if the body was damaged, and so on, these were included in the discriminant errors and manual correction would be required (see

Table 8. Accuracy of expansion gap identification using machine learning algorithm.

Accuracy (%)	Average	Mono Cell Type	Finger Type	Rail Type	Others
Conventional algorithm	67.5	71	51	87	61
AI algorithm (machine learning)	95	98	92	99	91
Improvement rate	↑27.5	↑27	↑41	↑12	↑30

).

5. Conclusions

In this research, an automatic-image-recognition-based survey system was established on road bridges and was successfully verified through field tests. In order to ensure 1.0 mm resolution performance per pixel in a high-speed (100 km/h) environment, M/V technology using a line-scan camera was used, and accurate images could be obtained almost in real time as a result of the test. Line-scan camera technology advances rapidly, so using a better camera could further improve system performance.

Technological advances will promote the implementation of the M/V system and lower maintenance costs. Therefore, one contribution of this research is providing a solution that can apply the M/V system to road maintenance using line-scan cameras.

A survey system equipped with line-scan cameras, GPS, and DMI was designed to perform automatic investigation when accessing the bridge while driving at high speeds. The use of NEXUS for performing automatic surveys of the bridge expansion joint device gap while driving at high speed reduces the survey time by more than 95%, from an average of approximately 1 h/bridge (existing manual inspection method by an inspector) to approximately 3 min/bridge. In addition, if the accumulated gap data is monitored and preemptive maintenance is performed before gap narrowing (lack of joint gap) occurs, it can eliminate the risk factors in the future temperature expansion behavior of the bridge and contribute to traffic safety and cost reductions. This is the second contribution.

Measuring the bridge expansion joint gap through survey images has limitations in traditional algorithmic methods. Images with various objects on the road surface and shapes of various expansion joint devices are similar to big data. Therefore, by creating an algorithm by artificial intelligence technology (machine learning), more accurate survey values can be stably obtained. The machine learning model for searching for expansion joint devices in survey images used Resnet as a feature extractor, and the representative segmentation model for searching for the gap area used U-Net. These were used to solve the classification problem. Testing with a random selection of 10,526 previously acquired big data images indicated that the expansion gap identification accuracy was improved by 27.5%, from 67.5% to 95.0%. This is another contribution.

However, the main contribution of this study is to demonstrate the possibility of developing smart maintenance techniques for road structures, in other words, the successful development of a smart maintenance system that combines machine vision and machine learning technology that serves the following purposes: first, to ensure the traffic safety of vehicles on roads that are already in us; and second, to obtain the condition of road structures at the level the investigator wants with figures and accurate images. This requires satisfaction with vehicle traffic, customer safety, investigator convenience, and traffic safety, so we believe that developing road maintenance technology should consider much more complex problems than developing smart construction technology.

If this study intensifies in the future, we will be able to create a wider variety of smart road maintenance systems based on this concept and our imagination.