Modeling the Optimal Maintenance Strategy for Bridge Elements Based on Agent Sequential Decision Making

Abstract

In addressing the issues of low efficiency in bridge maintenance decision making, the inaccurate estimation of maintenance costs, and the lack of specificity in decision making regarding maintenance measures for specific defects, this study utilizes data from regular bridge inspections. It employs a two-parameter Weibull distribution to model the duration variables of the states of bridge elements, thereby enabling the prediction of the duration time of bridge elements in various states. Referring to existing bridge maintenance and repair regulations, the estimation process of maintenance costs is streamlined. Taking into account the specific types and development state of bridge defects, as well as considering the adequacy of maintenance and the restorative effects of maintenance measures, an intelligent agent sequential decision-making model for bridge maintenance decisions is established. The model utilizes dynamic programming algorithms to determine the optimal maintenance and repair measures for elements in various states. The decision results are precise, all the way down to the specific bridge elements and maintenance measures for individual defects. In using the case of the regular inspection data of 222 bridges along a highway loop, this study further validates the effectiveness of the proposed research methods. By constructing an intelligent agent sequential decision-making model for bridge element maintenance, the optimal maintenance measures for 21 bridge elements in different states are obtained, thereby significantly enhancing the efficiency of actual bridge maintenance and the practicality of decision results.

1. Introduction

Bridges are critical infrastructure that connect transportation networks, and failures in bridge structures can result in significant economic, social, and environmental losses [1,2,3]. Therefore, it is crucial to scientifically manage, assess, and maintain bridges. One of the main reasons for inadequate bridge maintenance funding is the low efficiency of maintenance decision making, which leads to an improper allocation of funds [4]. Considering the severity of bridge degradation, the large number of bridges, increasing maintenance demands, and rapidly growing repair funds [5], effective bridge maintenance decision making, the rational allocation of limited repair funds, and the calculation of maintenance benefits to ensure maximum bridge performance at a satisfactory level are among the top concerns for government agencies, management, and academic research groups [6]. Bridge maintenance decision models serve as tools through which to assist scientific bridge management decisions, and they provide important support for the formulation of highway bridge maintenance plans. This study aims to analyze the actual condition of bridge element defects, establish a quantified and clear model for maintenance strategies, develop a cost estimation model for maintenance activities, and conduct a benefit analysis of element maintenance. Based on these analyses, an intelligent agent-based sequential decision-making model for bridge element maintenance is established. Finally, the optimal maintenance strategy for bridge elements is solved using dynamic programming algorithms.

1.1. Literature Review

At present, the prevailing approach for project-level bridge maintenance decision making is multi-objective decision making. In the context of multi-objective decision-making methods, Zoubir et al. [7] proposed a maintenance multi-objective decision-making method for single-span highway bridge deck panels. However, their study did not provide specific maintenance measures. Based on the multi-objective particle swarm optimization algorithm, Frangopol et al. [8] carried out a multi-objective optimization of an RC bridge superstructure based on the reinforcement years, failure probabilities of the main beam and bridge panels, and the FRP volume used for reinforcement. Lee et al. [9] introduced the applicability index, established a table corresponding to the repair methods for typical types of bridge deck diseases, and divided maintenance measures into four maintenance activity levels (daily maintenance, repair, reinforcement, and replacement). The multi-objective function consisted of a restoration effect, applicability, and maintenance cost, and it was optimized by a genetic algorithm. In Lee’s study, maintenance measures were divided into four activity levels: routine maintenance, repair, strengthening, and replacement. The multi-objective function comprised restoration effectiveness, applicability, and maintenance cost. Genetic algorithm optimization was employed to determine the appropriate measures for each bridge deck panel element. Furthermore, a multi-objective decision model was constructed, thereby optimizing the maintenance priorities for each bridge deck panel element and helping with the allocation ratio of funds.

Cost-effectiveness analysis in bridge maintenance encompasses an estimation of the maintenance measure costs that are incurred during the maintenance process and the assessment of benefits generated after maintenance. In the current research on bridge maintenance decision making, many studies have not specifically focused on the cost estimation of measures at the element and defect level. Cao et al. [10] defined cost estimation in bridge maintenance as the process of estimating the expenses associated with a specific maintenance strategy, which considers both the general category and detailed measures of the maintenance approach. This estimation is based on the bridge maintenance and repair quotas. The maintenance cost estimation comprises two parts: base price and workload. However, the study’s defined scope of maintenance work was relatively broad, and the cost estimation process for maintenance work was not presented in detail. Some researchers have focused on specific defect types and measures, whereby they quantified the cost of measures based on the volume of materials used. For instance, Frangopol et al. [8], in their research on lifecycle decision optimizations of the superstructure of RC bridges, considered a single maintenance measure (FRP strengthening), and its cost was quantified by the total volume of FRP material required. In the field of bridge engineering, there is a need to further specify and present the cost estimation process for maintenance work, which is achieved by considering specific defect types, measures, and the material volumes used. Zhao et al. [11] defined it such that each maintenance operation involves replacing 10% of the original materials, and the maintenance cost is calculated based on the volume of material replacement. However, the cost of maintenance measures not only includes material costs, but also encompasses expenses related to labor, machinery, and other factors such as regional rates. Therefore, quantifying the cost solely based on material volume may not be entirely accurate.

Scholars both domestically and internationally have different definitions of bridge maintenance benefits. For example, Frangopol et al. [8] used the cumulative time-dependent failure probability as a measure of lifecycle benefits. Mondoro et al. [12], for a given management strategy, defined the corresponding benefit as the probability of risk reduction achieved through structural retrofitting. Yadollahi et al. [13] assigned scores to various factors related to elements and conditions, health indexes, safety, and risk, as well as calculated the corresponding repair costs. They assigned different weights to each factor and calculated the benefit value using a benefit calculation formula. Ghodoosi et al. [14] defined maintenance benefit as an actual improvement in reliability before and after maintenance actions. Sabatino et al. [15] modified the specific parameters, the time-dependent cross-sectional area of internal reinforcement, and the plastic section modulus associated with steel beams, included in the performance function defined in maintenance effect studies, to assess the benefits of maintenance actions. Shim et al. [16] defined bridge maintenance benefits as an improvement in the NBI deck condition rating after the application of deck maintenance measures. In practical applications, some states in the United States have also provided relevant definitions of bridge maintenance benefits [17]. For example, Connecticut defines bridge maintenance benefits based on the improvement in bridge condition compared to no maintenance work, while New Mexico defines bridge maintenance benefits by measuring the ability to improve the NBI rating.

With the rapid development of artificial intelligence in recent years, there is a growing trend of integrating knowledge from the field of artificial intelligence with bridge maintenance management. This integration aligns with the development of intelligent bridge maintenance practices. In the field of artificial intelligence, planning and learning are central to many problems. In a discrete or continuous time domain, an intelligent agent interacts with the environment to choose appropriate sequences of actions to achieve predefined goals. This decision-making process is known as sequential decision making. In sequential decision making, the intelligent agent must balance between short-term greedy objectives and long-term planning, which aligns with the requirements of bridge maintenance decision making. In order to find the optimal maintenance strategies in bridge maintenance processes, attempts have been made to apply the sequential decision-making process of intelligent agents to the decision-making process of intelligent bridge maintenance management. This involves simulating the interaction between a bridge and its natural environment. The purpose of this interaction is to maximize the improvement in bridge structural performance. Dawid et al. [17] proposed a semi-Markov decision model that enables wind farm operators to automate the mid-term and short-term maintenance planning process for a batch or for all turbines rather than making decisions for individual turbines to determine their maintenance strategies. Wei et al. [18] combined sequential decision making with deep learning to explore maintenance strategies for simply supported beam bridge deck elements and long-span cable-stayed bridge elements. Maintenance strategies were classified into minor repair, major repair, and replacement. However, these strategies were not specifically tailored to the level of individual defects, and no detailed maintenance plans were formulated. Yu et al. [19] considered time-varying reliability in the degradation process of bridges and proposed a semi-Markov process model based on Weibull distribution for predicting the degradation of urban bridges.

1.2. Limitations of Existing Studies

Overall, both scholars have made numerous achievements in the research on the optimization of project-level bridge maintenance decision making. However, there are still some limitations and shortcomings:

(1) In the current research on project-level bridge maintenance decision making, the estimation of maintenance measure costs is mostly not specifically aligned with the actual occurrence of defects in bridge elements. The quantification of measure costs is often based on the volume of materials used. However, the cost of maintenance measures goes beyond material expenses, and it is also influenced by factors such as labor, machinery, and regional rates. Therefore, quantifying the cost solely based on material volume may lead to inaccuracies.

(2) In the current research on bridge maintenance decision making, the definition of maintenance strategies is often not very specific. It is often broadly defined as a category of maintenance activities, such as repairing the superstructure or replacing bridge deck panels. The cost estimation for such general maintenance activities can easily lead to significant deviations and may not fully consider all aspects of the actual engineering requirements. In practical applications, if the optimization process could be applied at the level of specific defects, and if maintenance strategies could be refined into concrete measures, it would greatly promote the refinement of bridge maintenance management. This would provide more accurate guidance for actual bridge maintenance decision making.

The remainder of this study is organized as follows. Section 2 summarizes the framework of bridge project-level maintenance that is established in this paper. Section 3 introduces the model and theory of the sequential decision making framework. In Section 4, the feasibility and validity of the proposed method are verified by the inspection data of a highway loop bridge. Section 5 includes the conclusions and main findings, as well as discusses the limitations of this work and the direction of future research.

2. The Bridge Project-Level Maintenance Decision Process

In terms of bridge project-level maintenance decision optimization, this paper aims to explore the optimal maintenance strategy for all elements of a bridge. Based on the actual occurrence of defects in bridges, the decision is made by considering the minimization of maintenance costs as the objective. The maintenance effect matrix, which represents the probability of technical condition transitions for elements after executing maintenance plans, is introduced. A sequential decision optimization model is established for bridge project-level maintenance, and the optimal maintenance plan is obtained using dynamic programming in the sequential decision making process. The decision optimization process is depicted in Figure 1.

(1) The analysis of actual defect conditions and technical conditions of bridges elements is based on the regular inspection reports of bridges. It involves gathering information on the types and quantities of defects observed in bridge elements, as well as the attributes of these defects. The technical condition level of each element is calculated, and this information serves as the foundation for maintenance decision making, thereby ensuring the practicality of the decisions.

(2) Based on all the observed defect types, corresponding maintenance plans were determined, where each plan may include multiple repair measures. By considering the cost of measures in the maintenance plans and the distribution of the elements’ technical condition levels, the cost of repairing elements with different condition levels is calculated, thereby generating a maintenance cost matrix (referred to as the reward matrix). By implementing different maintenance plans for elements with defects, the effects of various maintenance strategies are explored, thus forming a transition probability matrix (known as the maintenance effect matrix, which represents the transition of element conditions before and after maintenance).

(3) The state distribution of elements, the transition probability matrix for element conditions before and after maintenance, and the maintenance cost matrix are used as inputs to the sequential decision making model. A dynamic programming algorithm is employed to solve the model.

3. Bridge Element Maintenance Decision Optimization Model

3.1. Sequential Decision Making Theory

In addition to multi-objective decision-making methods, decision tree methods and a range of self-learning methods such as semi-Markov decision processes [20] (or Markov decision processes [21], artificial neural networks [22], probabilistic neural networks [23], analytic hierarchy process [24], and reliability methods [25]) have also been employed for project-level bridge maintenance decision making. The sequential decision making process (SDP) can also be used for bridge project-level maintenance decisions. SDP refers to the process of making a series of decisions in successive time steps to achieve a certain goal, and it is a mathematical model used to describe the problem of progressively optimal decision making in a dynamic environment. Sequential decision making processes can be applied to a variety of decision problems, including Markov decision processes (MDPs) and semi-Markov processes (SMDPs). MDPs consist of a tuple <S,A,P,R,γ>, and the conditional distribution of future states only depends on the current state, which is independent of past states, where S represents the set of states; A represents the set of actions; $P\left(s^{\prime }\right|s,a)$. represents the state transition function, which denotes the probability of transitioning from state s to state s′ when action a is taken; R represents the reward or values function, which depends on state s and action a; and γ represents the discount factor.

This paper introduces an improved Markov model that incorporates Weibull distribution and forms a semi-Markov model, where the duration time in each state follows the Weibull distribution, thus providing necessary support for the decision model.

3.2. Sequential Decision Making Model for Bridge Project-Level Maintenance Decisions

3.2.1. Bridge Element State Sets

The state set represents all the possible states of a bridge’s elements, and it also corresponds to the technical condition levels of the elements. Since a definition of element technical condition levels is not provided in “Standards for Technical Condition Evaluation of Highway Bridges” (JTG/T H21-2011) [26], this paper refers to the classification boundaries of bridge element technical conditions as are defined in JTG/T H21-2011. Moreover, this paper defines five states for bridge elements that correspond to five technical condition levels, as shown in

Table 1. SDP state set.

Element Technical Condition Score	State Set
EBCI	State 1	State 2	State 3	State 4	State 5
	[95, 100]	[80, 95)	[60, 80)	[40, 60)	[0, 40)

.

At time T, the ratio of the number of elements in each state to the total number of elements forms the state distribution vector D_T.

$$D_T=\{\frac{C_1}{C},\frac{C_2}{C},\frac{C_3}{C},\frac{C_4}{C},\frac{C_5}{C}\}.$$

(1)

With respect to the elements of the vector $\frac{C_i}{C}$, C_i represents the number of elements in state i at the current year and C represents the number of elements at the current year.

3.2.2. Maintenance Strategies Set

The core of an SDP is the interaction between an agent and its environment through actions. In the context of bridge maintenance decisions, executing maintenance plans can potentially transition bridge elements from one state to another. Therefore, maintenance plans can correspond to actions in an SDP. A set of maintenance plans was introduced, which was denoted as A, as shown in

Table 2. Examples of bridge element maintenance strategy set A.

Number	Bridge Deterioration Types	Maintenance Plan Description	Maintenance Measures
			Maintenance Measure Number	Maintenance Measure Description
A1	Defect Types 1	Repairing Elements	a1	Maintenance Measures 1
			a2	Maintenance Measures 2
A2	Defect Types 2	Repairing Elements	b1	Maintenance processes 1 + Maintenance processes 2 + Maintenance processes 3 +…+ Maintenance processes nb1
			b2	Maintenance processes 1 + Maintenance processes 2 + Maintenance processes 3 +…+ Maintenance processes nb2
A3	Defect Types 3	Repairing Elements	c1	Maintenance processes 1 + Maintenance processes 2 + Maintenance processes 3 +…+ Maintenance processes nc1
			c2	Maintenance processes 1 + Maintenance processes 2 + Maintenance processes 3 +…+ Maintenance processes nc2
			d1	Maintenance processes 1 + Maintenance processes 2 + Maintenance processes 3 +…+ Maintenance processes nd1

. The categorization of the maintenance plans depended on the actual technical condition levels and specific defects of the bridge, as well as through references to relevant maintenance regulations.

3.2.3. Maintenance Effectiveness Matrix

In this study, we established a maintenance effectiveness matrix as the transition probability matrix based on the actual defect conditions of bridge elements to enhance the accuracy of decision making. For bridge maintenance benefit analysis, maintaining and improving the technical condition of bridges is the most significant aspect of enhancing benefits. Shim et al. [16] compared the bridge deck condition levels before and after the application of maintenance action, and they analyzed the impact of bridge deck maintenance measures on the condition levels. They established a maintenance effectiveness matrix. In this paper, via referring to their research findings, we define the effectiveness matrix as the transition probability matrix after implementing maintenance plans, and this represents the repair effects of maintenance actions on bridge elements. Different maintenance plans applied to elements in different states result in varying repair effects, thus leading to different transition probabilities. The calculation formula for the effectiveness matrix of executing a maintenance plan k is as follows:

$$M_{A_k}=\left[\begin{array}{ccccc}\frac{C_{k11}}{C_1}& 0& 0& 0& 0\\ \frac{C_{k21}}{C_2}& \frac{C_{k22}}{C_2}& 0& 0& 0\\ \frac{C_{k31}}{C_3}& \frac{C_{k32}}{C_3}& \frac{C_{k23}}{C_3}& 0& 0\\ \frac{C_{k41}}{C_4}& \frac{C_{k42}}{C_4}& \frac{C_{k43}}{C_4}& \frac{C_{k44}}{C_4}& 0\\ \frac{C_{k51}}{C_5}& \frac{C_{k52}}{C_5}& \frac{C_{k53}}{C_5}& \frac{C_{k54}}{C_5}& \frac{C_{k55}}{C_5}\end{array}\right].$$

(2)

Assuming that, after executing a maintenance plan, the state of an element can only transition to a lower state, then the elements of the matrix can be denoted as $\frac{C_{kij}}{C_i}$, where C_i represents the quantity of elements in state i; C_kij represents the quantity of elements transitioning from state i to state j after executing maintenance plan k; $M_{A_k}$ is the element in the ith row; the jth column represents the probability of elements in state i transitioning to state j after implementing maintenance plan A_k; and k is the number of maintenance plans, and it is also determined by the actual number of defects present in the elements.

3.2.4. Reward Set (Cost Matrix) and Discount Factor

The reward set corresponds to the costs incurred by executing maintenance plans on bridge elements. In this study, reference is made to the “Regulations for Budgeting of Highway Maintenance” (JTG 5610-2020) [27] and “Budget Quota for Highway Bridge Maintenance Engineering” (JTG/T 5612-2020) [28]. The cost of maintenance measures should include labor costs, material costs, other direct and indirect costs, the management fees of the construction unit, preliminary work expenses for the construction project, as well as reserved costs and other initial construction costs.

In this paper, the budget unit price method based on the work process of maintenance measures is used to determine the direct cost unit price of each maintenance measure. This method involves calculating the unit prices of materials and the labor involved in the maintenance process to estimate the direct costs of the maintenance measures.

$$U{P}_x={\sum }_{j=1}^{n}U{P}_j,$$

(3)

where UP_x represents the cost unit price of maintenance measure x and UP_j denotes the base price of the jth work process in the description of maintenance measure x. The specific base prices for the maintenance work processes can be referred to in the “Budget Quota for Highway Bridge Maintenance Engineering” (JTG/T 5612-2020), where n represents the total number of work processes included in the description of maintenance measure x, as shown in Table 2.

Due to the fact that the maintenance plans defined in this paper are composed of a series of maintenance measures, it is necessary to develop specific maintenance plans that are based on the actual condition of the bridge and the specific defects present. By determining the corresponding maintenance measures for each identified defect and substituting them into Equation (3), we can estimate the unit cost UP_x of the maintenance measures. This will provide an estimation of the maintenance costs for each identified defect location.

$$c_{l_k,m}=Q_{l_k,m}\times U{P}_{x,m},$$

(4)

where c_lk,m represents the maintenance cost required for repairing the defect m on element k, Q_lk,m represents the total number of defect m on element k, UP_x,m represents taking action x for defect m, and the unit cost of action x is calculated according to Equation (3).

The specific cost incurred by element k when executing maintenance plan A_i is calculated according to the following equation:

$$C_{l_k,A_i}={\sum }_{m=1}^r{c}_{l_k,m},$$

(5)

where $C_{l_k,A_i}$  represents the cost incurred by element k when executing maintenance plan A_i, which includes repairing r defects on that element.

After calculating the required costs for different maintenance plans on each element, a maintenance cost matrix, also known as the reward matrix R, can be formed. In the context of SDPs, the objective is to find a maintenance strategy that maximizes the reward (and minimizes the cost). In this case, the reward can be defined as the negative value of the maintenance plan cost, and this allows us to solve for the optimal strategy that minimizes costs or maximizes the reward. The calculation formula is as follows:

$$C_{A_i,s}=\sum _{k=1}^{Q_s} C_{l_k,A_i}$$

(6)

$$R=\left[\begin{array}{ccccc}-\frac{C_{A_0,1}}{Q_1}& -\frac{C_{A_0,2}}{Q_2}& -\frac{C_{A_0,3}}{Q_3}& -\frac{C_{A_0,4}}{Q_4}& -\frac{C_{A_0,5}}{Q_5}\\ -\frac{C_{A_1,1}}{Q_1}& -\frac{C_{A_1,2}}{Q_2}& -\frac{C_{A_1,3}}{Q_3}& -\frac{C_{A_1,4}}{Q_4}& -\frac{C_{A_0,5}}{Q_5}\\ ⋮& ⋮& ⋮& ⋮& ⋮\\ -\frac{C_{A,1}}{Q_1}& -\frac{C_{A,2}}{Q_2}& -\frac{C_{A,3}}{Q_3}& -\frac{C_{A,4}}{Q_4}& -\frac{C_{A,5}}{Q_5}\end{array}\right],$$

(7)

where Q_s represents the quantity of elements in state s. The matrix elements are denoted as $-\frac{C_{A_i,s}}{Q_s}$, which represents the cost required to repair all elements in state s when executing maintenance plan $C_{A_i,s}$. In the cost matrix R, the element in the ith row and jth column represents the average maintenance cost incurred by executing maintenance plan A_i under state j.

3.2.5. Weibull Distribution for the Duration of States

Compared to traditional Markov processes, in a semi-Markov process, the random variable T, which represents the duration of staying in a state, follows a probability distribution. Let T₁,T₂,…,T₄ represent the random variable for the duration of bridge elements in state i(i = 1,2,3,4). It is assumed that duration T_i in any state i follows a two-parameter Weibull probability distribution, which is denoted as $T_i~Weibull(b_i,\frac{1}{a_i})$. Therefore, we have the following:

$$F_i(t)=Pr[T_i\le t]=1-e^{-({\alpha }_{i}t{)}^{b_i}}$$

(8)

$$S_i(t)=\mathrm{P}\mathrm{r}\left[T_i>t\right]=1-F_i(t)=e^{-{\left(a_{i}t\right)}^{b_i}}$$

(9)

$$f_i(t)=\frac{\delta F_i\left(t\right)}{\delta t}=a_i{b}_i{\left(a_{i}t\right)}^{b_i-1}{e}^{-{\left(a_{i}t\right)}^{b_i}},$$

(10)

where F_i(t) represents the Cumulative Distribution Function (CDF) associated with the duration, which refers to the probability that an element remains in state i for less than time t. S_i(t) represents the Survival Function (SF), also known as the cumulative survival rate, where the probability that an element remains in state i for longer than time t is indicated. f_i(t) represents the Probability Density Function (PDF) associated with the duration, which describes the probability density of the duration in state i at time t.

By utilizing the historical inspection data of bridges, it is possible to fit the parameters a_i and b_i for different states i. According to the properties of the Weibull distribution, the duration T_i of staying in each state of the elements can be calculated by substituting the values into Equation (11):

$$T_i=-\frac{{\left(\mathrm{l}\mathrm{n}\left(u\right)\right)}^{\frac{1}{b_i}}}{a_i}$$

(11)

Assuming that, without any maintenance, the elements can only degrade by one condition level, it can be assumed that the duration of staying in state s for an element is equal to the time it takes for the element to transition from state s to state $s^{\prime }$, where $s^{\prime }=s+1$. $T(s,s^{\prime },t^{\prime })$ represents the time distribution required to transition from state s to state $s^{\prime }$, and $T(s,s^{\prime },t^{\prime })=\{T_1,T_2,T_3,T_4,T_5\}$. $T(s,s^{\prime },t^{\prime })$ is used to calculate the state value function, which serves as a guide for determining maintenance priorities, formulating maintenance strategies, and allocating maintenance funds.

3.3. Sequential Decision Making and Solving

The reward function represents the cumulative discounted rewards starting from time t (the value of all rewards at the current time). In the context of bridge maintenance decision making, this corresponds to the present value of all maintenance plan costs.

$$G_t=R_t+\gamma R_{t+1}+\dots ={\sum }_{k=0}^{\mathrm{\infty }} {\gamma }^k{R}_{t+k},$$

(12)

where $R_t$ represents the immediate reward obtained by taking a maintenance plan at time t, which corresponds to the cost incurred by the maintenance plan in that year.

By substituting the formula of the state value function into the reward function, we can derive the Bellman equation and solve for the state value function:

$$\begin{array}{c}v\left(s\right)=\mathrm{E}\left[G_t\mid S_t=s\right]\\ =\mathrm{E}\left[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\dots \mid S_t=s\right]=\mathrm{E}\left[R_{t+1}+\gamma G_{t+1}\mid S_t=s\right]\\ =\mathrm{E}\left[R_{t+1}+\gamma v\left(S_{t+1}\right)\mid S_t=s\right]\end{array}.$$

(13)

As per the Bellman equation, the value of a state is composed of the expected immediate reward and the expected value of the next state. Therefore, by maximizing all action value functions, we can obtain the optimal strategy:

$${\pi }_{\ast }\left(a∣s\right)=\left\{\begin{array}{cc}1& \mathrm{if} a=\underset{a\in A}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}{q}_{\ast }\left(s,a\right)\\ 0& \mathrm{otherwise} \end{array}\right..$$

(14)

Dynamic programming is an algorithm used to solve SMDP problems when the model is known. It can be used to evaluate a given policy and iteratively obtain the optimal value function. The specific approaches include policy iteration and value iteration. Compared to value iteration, policy iteration tends to converge faster in large-scale problems.

This paper adopts a policy iteration method to solve for the optimal maintenance strategy, as illustrated in Figure 2. The solution process mainly consists of two steps: policy evaluation and policy improvement.

In the stage of policy evaluation, the method of value function iteration can be used to simplify the computation. Firstly, the value function of each bridge state under the current maintenance strategy is estimated. The state value function is used to evaluate the superiority of the policy, where a larger value indicates a better policy. Specifically, the value function v^π(s,t) for each bridge state is defined as the expected cumulative reward that can be obtained starting from state s and after time t. The value function of bridge states is recursively calculated using the Bellman equation:

$$v^{\pi }\left(s,t\right)=R(s)+D_T\cdot \pi (a|s)\cdot T(s,s^{\prime },t^{\prime })\cdot \sum \gamma \cdot v^{\pi }(s^{\prime },t^{\prime }+t),$$

(15)

where $R\left(s\right)$ represents the immediate maintenance cost starting from bridge state s; $D_T$ is the state distribution vector; $\pi (a|S)$ represents the probability of taking action a given the input state s, where action a represents a maintenance strategy a; T(s,s′,t′) represents the time distribution required to transition from state s to state s′; and γ is the discount factor that controls the importance of future rewards. The value is equal to the ratio of the existing maintenance funds to the estimated cost of repairing all diseases, and it also represents the adequacy of current maintenance funds. v(s,t′ + t) represents the value function of state s′, which is reached by state s after time t′. The above equation is a recursive equation that requires starting with an initial estimate of the value function for each state and gradually updating it until convergence is achieved.

In the stage of policy improvement, the current maintenance strategy $\pi$ should be improved based on the current state value function v^π(s,t) in order to make the maintenance strategy $\pi$ more optimal. For each bridge state s, the value function q(s,a,t) is calculated for all possible maintenance strategies a that can be taken under state s.

$$q(s,a,t)=R_{a,s}+\gamma {\sum }_{s^{\prime }\in S} M_{A_a}(s,s^{\prime })\cdot V^{\pi }(s^{\prime }),$$

(16)

where q(s,a,t) is related to v(s,t), and the maintenance strategy π(s) is chosen as the new policy by selecting the one with the following maximum value:

$$\pi \left(s\right)=ma{x}_{a}q\left(s,a,t\right),$$

(17)

where q(s,a,t) represents the expected cumulative reward that can be obtained after taking maintenance strategy a for a bridge in state s at time t.

The above two steps are performed iteratively until the policy no longer changes or the predetermined number of iterations is reached. Eventually, the optimal policy $\pi$ and its corresponding optimal value function $v\ast \left(s,t\right)$ can be obtained:

$$\nu \ast \left(s,t\right)=max\pi {\nu }_{\pi }\left(s,t\right)$$

(18)

where v_π(s,t) represents the value function starting from state s under policy $\pi$.

4. Case Study

4.1. Data Preparation

When using the historical inspection data of highway loop bridge in Jinan, Shandong Province, China as a case study, as shown in Figure 3, there were a total of 8 interchanges and 222 bridges along the route. In order to establish a degradation model of the bridge elements, 40 prestressed reinforced concrete continuous box girder bridges in the route were selected, including 285 box girder elements. The technical condition score data of the box girder members in the periodic inspection reports of 40 bridges from 2008 to 2019 were extracted for pre-processing, and the panel data formed were used as the input measured data of the degradation model. The bridges undergo regular inspections and maintenance every three years. In order to verify the validity of the project-level decision method for bridges, a continuous beam bridge on the route was selected as the case study bridge. The main girder of the continuous beam bridge was a prestressed reinforced concrete box girder, the span of which was 25 + 28 + 2×25 + 20 + 4 × 25 + 22.7 + 35 + 58 + 35 + 4 × 30 + 4 × 30 m. Consisting of 21 elements, the total length was 627.06 m, and the total width was 12.75 m, as shown in Figure 4. The purpose of this case was to find the optimal disease maintenance strategy for the pre-stressed reinforced concrete box girder of the bridge.

The state distribution vector ${\mathrm{D}}_6=\left(\begin{array}{ccc}\begin{array}{ccc}\frac{1}{21}& \frac{1}{21}& \frac{4}{21}\end{array}& \frac{12}{21}& \frac{3}{21}\end{array}\right)$ for the sixth year of the main girder elements of the bridge was known. By establishing an SMDP model and utilizing a dynamic programming algorithm, the optimal maintenance strategy for the box girder elements of the case study bridge for the next 3 years will be calculated and determined.

4.2. Element Maintenance Decision Model

4.2.1. Bridge Element State Sets

The state set S represents all the possible states of the bridge elements, which correspond with the five semi-Markov states $i=\left\{1, 2, 3, 4, 5\right\}$, as detailed in Section 3.2.1.

4.2.2. Maintenance Strategy Set

For the specific typical defects observed in the box girder elements of the bridges on this route, such as concrete cracks, exposed reinforcement corrosion, and surface defects, the maintenance strategy set $A=\left\{A_0,A_1,A_2,A_3\right\}$ was defined. This study refers to Section 4.3 of the “Specifications for Maintenance of Highway Bridges and Culverts” (JTG 5120-2021) [28] to define the maintenance and reinforcement methods for prestressed concrete beam bridges. The maintenance strategies were categorized as shown in

Table 3. Maintenance strategy set of prestressed reinforced concrete box girders.

Number	Maintenance Plan Description	Maintenance Measures and Costs
		Number	Maintenance Measure Description	Unit Cost (USD)
A0	No maintenance	/	/	/
A1	Repairing concrete cracks in elements	a1	Automatic low-pressure grouting	13.66
		a2	Pressure grouting	15.55
A2	Repairing exposed reinforcement corrosion in elements	b1	Manual derusting + Reinforcement protective agent + Manual chiseling + Cleaning + Polymer cement mortar	35.71
		b2	Manual derusting + Reinforcement protective agent + Rust inhibitor + Manual chiseling + Cleaning + Polymer cement mortar	44.15
A3	Repairing surface defects in elements	c1	Manual chiseling + Cleaning + Polymer cement mortar	23.89
		c2	Manual chiseling + Cleaning + Cast-in-place repairs	133.50
		d1	Manual chiseling + Cleaning + Reinforcement protective agent + Polymer cement mortar	35.43

, where A0 was defined as no maintenance.

The determination of a series of maintenance actions corresponding to each maintenance strategy in Table 3 was based on the recommended maintenance actions in the bridge regular inspection report, as well as guidelines such as the “Regulations for Budgeting of Highway Maintenance” (JTG 5610-2020) [27], the “Budget Quota for Highway Bridge Maintenance Engineering” (JTG/T 5612-2020) [28], and the “Specifications for Maintenance of Highway Bridges and Culverts” (JTG 5120-2021) [29].

4.2.3. Maintenance Effectiveness Matrix

For the 21 elements of the case study bridge, the 4 maintenance strategies described in the previous section were implemented individually. The changes in state before and after maintenance were recorded and are summarized in

Table 4. Comparison of the technical conditions after different maintenance schemes.

Number	Number of Concrete Cracks	Number of Exposed Reinforcement Corrosion	Number of Surface Defects	State before Maintenance	State after Implementing A1	State after Implementing A2	State after Implementing A3
R-8-1	0	0	3	2			1
R-2-1	0	2	0	3		1
R-5-1	4	0	0	3	1
R-6-1	2	0	0	3	1
R-13-1	10	0	0	3	1
R-1-1	3	2	0	4	3	4
R-4-1	3	0	0	4	1
R-7-1	17	1	0	4	3	4
R-9-1	13	0	3	4	2		3
R-10-1	4	0	1	4	3		2
R-14-1	11	3	9	4	4	4	4
R-16-1	0	1	1	4		3	3
R-17-1	9	0	5	4	3		3
R-18-1	2	1	6	4	4	4	4
R-19-1	7	0	3	4	2		4
R-20-1	5	1	2	4	4	4	4
R-21-1	1	1	0	4	3	3
R-11-1	6	2	3	5	5	4	4
R-12-1	7	0	1	5	2		4
R-15-1	5	1	10	5	4	4	4

.

By substituting the statistical results from Table 4 into Equation (2), the maintenance effectiveness matrix can be obtained as shown in Equations (19) to (22).

$$M_{A_0}=\left[\begin{array}{ccccc}1& 0& 0& 0& 0\\ 0& 1& 0& 0& 0\\ 0& 0& 1& 0& 0\\ 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 1\end{array}\right]$$

(19)

$$M_{A_1}=\left[\begin{array}{ccccc}1& 0& 0& 0& 0\\ 0& 1& 0& 0& 0\\ 0.75& 0.25& 0& 0& 0\\ 0.08& 0.17& 0.33& 0.42& 0\\ 0& 0.33& 0& 0.67& 0\end{array}\right]$$

(20)

$$M_{A_2}=\left[\begin{array}{ccccc}1& 0& 0& 0& 0\\ 0& 1& 0& 0& 0\\ 0& 0& 1& 0& 0\\ 0.08& 0.25& 0.67& 0& 0\\ 0& 0& 0& 1& 0\end{array}\right]$$

(21)

$$M_{A_3}=\left[\begin{array}{ccccc}1& 0& 0& 0& 0\\ 0& 1& 0& 0& 0\\ 0.25& 0& 0.75& 0& 0\\ 0& 0& 0.17& 0.83& 0\\ 0& 0& 0& 0.67& 0.33\end{array}\right]$$

(22)

4.2.4. Cost Matrix and Discount Factor

Based on the estimated unit prices UP_x for the maintenance measures in Table 3, the repair costs for different defects in each element could be calculated based on attributes such as area and length, as shown in

Table 5. Calculation of the maintenance program costs.

Elements Number	Element Grade	Cost (USD)			Measure Number
		Repairing Cracks	Surface Defects	Exposed Reinforcement Corrosion
R-1-1	4	23.45	0.00	1090.76	a1 + b2
R-2-1	3	0.00	0.00	1090.76	b2
R-3-1	1	0.00	0.00	0.00	/
R-4-1	4	22.42	0.00	0.00	a1 + a2
R-5-1	3	36.94	0.00	0.00	a1 + a2
R-6-1	3	97.03	0.00	0.00	a1 + a2
R-7-1	4	326.30	0.00	1636.14	a1 + a2 + b2
R-8-1	2	0.00	195.18	0.00	c1
R-9-1	4	239.57	321.16	0.00	a1 + a2 + c1
R-10-1	4	81.76	53.23	0.00	a2 + c1
R-11-1	5	76.30	21,048.78	299.96	a1 + a2 + b2 + c2
R-12-1	5	921.36	236.58	0.00	c1 + a1
R-13-1	3	137.90	0.00	0.00	a1 + a2
R-14-1	4	345.70	7403.53	207.24	a1 + a2 + b2 + c1 + c2
R-15-1	5	173.42	501.37	545.38	a1 + a2 + b2 + c1 + d1
R-16-1	4	0.00	177.44	13.63	b2 + c1
R-17-1	4	101.47	211.84	0.00	a1 + a2 + c1 + d1
R-18-1	4	19.31	115.58	43.63	a1 + a2 + b1 + c1 + d1
R-19-1	4	50.25	266.15	0.00	a1 + a2 + c1
R-20-1	4	83.76	34.30	5.45	a1 + a2 + b2 + c1
R-21-1	4	12.56	0.00	32.72	a1 + b2

. By substituting the results from Table 5 into Equation (6), the cost matrix R can be obtained, as shown in Equation (23). When the bridge had defects that could not be repaired by any of the three maintenance strategies, the maintenance cost was defined as the unit price for cleaning and washing, which was USD 1.09. For the same type of defect, different repair measures can be adopted. For example, for element R-4-1, according to Table 4, there were three concrete cracks in different positions of the box girder. Different repair actions, a₁ and a₂, can be chosen to repair the cracks based on construction conditions.

$$R=\begin{array}{c}{A}_0\\ A_1\\ A_2\\ A_3\end{array}\left[\begin{array}{ccccc}0& 0& 0& 0& 0\\ -7.80& -7.80& -486.16& -778.78& -2792.08\\ -7.80& -195.18& -7.80& -715.27& -7262.24\\ -7.80& -7.80& -272.69& -252.47& -281.78\end{array}\right]$$

(23)

In the case of a limited maintenance budget, the initial value of the discount factor $\gamma$ was set to 0.1, thus indicating the need to save future maintenance costs.

4.2.5. Duration of Each State

The duration of bridge element states reflects the degradation of the technical condition in the natural environment and should be universally applicable. In this paper, based on the historical inspection data of bridges on the northbound section of a highway, 40 prestressed reinforced concrete continuous box girder bridges were selected for ease of calculation. The percentages of the elements in each state at different ages were calculated and are summarized in

Table 6. The proportion of box girder members in each state.

Element Age	Proportion Of Each State
	1	2	3	4	5	1–2	1–3	1–4
0	100.00%	0.00%	0.00%	0.00%	0.00%	100.00%	100.00%	100.00%
1	66.67%	33.33%	0.00%	0.00%	0.00%	100.00%	100.00%	100.00%
2	36.84%	28.77%	34.39%	0.00%	0.00%	65.61%	100.00%	100.00%
3	29.82%	22.11%	16.84%	31.23%	0.00%	51.93%	68.77%	100.00%
4	20.00%	7.37%	31.93%	35.79%	4.91%	27.37%	59.30%	95.09%
5	19.30%	10.18%	28.42%	36.49%	5.61%	29.47%	57.89%	94.39%
6	18.60%	4.91%	30.18%	39.30%	7.02%	23.51%	53.68%	92.98%
7	18.60%	10.18%	20.00%	43.51%	7.72%	28.77%	48.77%	92.28%
8	18.25%	2.81%	24.56%	46.32%	8.07%	21.05%	45.61%	91.93%
9	17.54%	4.21%	20.35%	47.72%	10.18%	21.75%	42.11%	89.82%
10	14.39%	2.81%	18.25%	52.28%	12.28%	17.19%	35.44%	87.72%
11	13.68%	1.75%	19.65%	50.18%	14.74%	15.44%	35.09%	85.26%

Note: States 1–2 refers to State 1 and State 2; States 1–3 refers to State 1, State 2, and State 3; and States 1–4 refers to State 1, State 2, State 3, and State 4.

. The parameters for fitting the duration of each state could be obtained using the Weibull distribution, as shown in

Table 7. Weber distribution parameters.

State	Weibull Distribution Parameters
	ai	bi
State 1	0.3870	0.5745
States 1–2	0.2091	0.999
States 1–3	0.1040	1.1127
States 1–4	0.0261	1.5009

.

State 5 represents the final state of the element, and its duration is a deterministic value (the bridge design service life minus the total duration of States 1 to 4). It does not follow the Weibull distribution. By substituting the Weibull parameters into Equation (11), the durations of the elements in each state can be obtained, as shown in the table.

The durations in

Table 8. Status duration schedule.

State	a	b	Duration (years)
State 1	0.387	0.5745	1.36
States 1–2	0.2091	0.999	3.31
States 1–3	0.104	1.1127	6.91
States 1–4	0.0261	1.5009	30.01

represent the total time that bridge elements stay in multiple states without maintenance. The duration of State 1 was 1.36 years, thereby indicating that the element stays in State 1 for 1.36 years. The duration of States 1–2 was 3.31 years, meaning that the element stays in States 1 and 2 for a combined duration of 3.31 years. Therefore, the element stays in State 2 for 3.31 − 1.36 = 1.95 years. The duration of States 1–3 is 6.91 years, implying that the element stays in States 1, 2, and 3 for a combined duration of 6.91 years. Therefore, the element stays in State 3 for 6.91 − 3.31 = 3.6 years. The duration of States 1–4 is 30.01 years, thereby indicating that the element stays in States 1, 2, 3, and 4 for a combined duration of 30.01 years. Therefore, the element stays in State 4 for 30.01 − 6.91 = 23.1 years. Since the bridge on the highway is inspected and repaired every three years, the duration of each state cannot be exactly three years.

$$T\left(s,s^{\prime },t^{\prime }\right)=\{\mathrm{1.36,1.95,3},\mathrm{3,3}\}$$

(24)

4.2.6. Definition of the Initial Strategy

Using the policy iteration method obtained from the dynamic programming algorithm, we solved the sequential decision making problem. First, we defined the maintenance strategy, denoted as strategy $\pi \left(a\right|s)$, in this study. Strategy A is defined as the proportion of elements in each state that will be maintained using four maintenance actions every year in the future.

$$\pi (a\mid s)=P\left[A_t=a\mid S_t=s\right]=\left[\begin{array}{cccc}{a}_{11}& a_{12}& a_{13}& a_{14}\\ a_{21}& a_{22}& a_{23}& a_{24}\\ a_{31}& a_{32}& a_{33}& a_{34}\\ a_{41}& a_{42}& a_{43}& a_{44}\\ a_{51}& a_{52}& a_{53}& a_{54}\end{array}\right],$$

(25)

where the element in the ith row and kth column represents the proportion of bridge elements in state i that will be maintained using maintenance measure $A_{k-1}$. For example, the element a₁₂ in the maintenance strategy matrix represents the percentage of elements in State 1 that will be maintained using maintenance action A₁. The element a₂₃ represents the percentage of elements in State 2 that will be maintained using maintenance action A₂.

Before starting the iterative computation, an initial strategy needs to be arbitrarily defined as the input for the algorithm.

$$\pi (a\mid s)=P\left[A_t=a\mid S_t=s\right]=\left[\begin{array}{cccc}0.25& 0.25& 0.25& 0.25\\ 0.25& 0.25& 0.25& 0.25\\ 0.25& 0.25& 0.25& 0.25\\ 0.25& 0.25& 0.25& 0.25\\ 0.25& 0.25& 0.25& 0.25\end{array}\right]$$

(26)

5. Results and Discussion

5.1. Solving for the Optimal Strategy

The sequential decision making process elements defined in the above process, along with the initial strategy, were sequentially incorporated into the dynamic programming algorithm for iterative computation. In this study, the iterative calculation was implemented in the form of MATLAB code, as shown in the pseudo-code in Algorithm 1. The maximum number of iterations was set to 1000, and the policy iteration method was used for iterative computation to obtain the optimal strategy. The optimal strategy obtained from the iterative computation was as follows:

$$\pi (a\mid s)=P\left[A_t=a\mid S_t=s\right]=\left[\begin{array}{cccc}1& 0& 0& 0\\ 1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 1& 1& 0\\ 0& 1& 1& 1\end{array}\right]$$

(27)

The optimal strategy output above represents the optimal maintenance plans for elements in different states in the next three years. Specifically, the following observations were made: (1) For elements in States 1 and 2, the recommended maintenance plan is no maintenance. (2) For elements in State 3, the recommended maintenance plan is to repair all the concrete cracks in the elements that are in this state. (3) For elements in State 4, the recommended maintenance plan is to repair all concrete cracks in the elements that are in this state and to repair all exposed reinforcement corrosion in the elements in this state. (4) For elements in State 4, the recommended maintenance plan is to repair all of the concrete cracks in the elements that are in this state, as well as to repair all the exposed reinforcement corrosion and all the visible defects in the elements in this state.

According to Table 5 corresponding maintenance measures were applied to repair the elements. For example, if R-8-1 is in State 2 and there is a visible defect but there is an insufficient maintenance budget, then this element will not be repaired. If R-1-1 is in State 4, then it will be repaired using measure a1 to repair the concrete cracks and measure b2 to repair the exposed reinforcement corrosion. If R-11-1 is in State 5, it will be repaired using measure a1 and a2 to repair the concrete cracks, measure b2 to repair the exposed reinforcement corrosion, and measure c2 to repair the visible defect.

5.2. Maintenance Strategies under Different Parameters

By varying the discount factor based on the availability of maintenance funds and applying it to the dynamic programming algorithm mentioned above, the optimal maintenance strategies for elements in service for 6 years are obtained under different maintenance budgets and different states. The results are shown in Figure 5.

From Figure 5, it can be observed that, for the same state, the maintenance strategies differ when the discount factor varies. For example, for State 3, when the discount factor ranges from 0.1 to 0.3 (considering only minimizing the current maintenance cost without considering future cost savings), Maintenance Strategy 2 is used to repair the concrete cracks. When the discount factor ranges from 0.4 to 0.6 (which considers not only minimizing the current maintenance cost, but also considers some cost savings in the future), Maintenance Strategies 2 and 4 are used to repair concrete cracks and visible defects. When the discount factor ranges from 0.7 to 1 (which considers both minimizing the current maintenance cost and considering cost savings in the future), Maintenance Strategies 2 and 3 are used to repair concrete cracks and exposed reinforcement corrosion.

For bridge elements with different service durations, if the actual condition of the elements varies, their maintenance strategies also differ. Figure 6 illustrates the maintenance strategies for elements with a service duration of 2 years.

Algorithm 1. Pseudo code of the policy iteration al gorithm.

Objective: Find the optimal state and optimal strategy. Input: State distribution vector $D_6$, repair effect matrix $M_{A_k}$, maintenance cost matrix R, initial policy $\pi \left(a\right|s)$, initial value function $v^{\pi }(s,t)$, discount factor $\gamma$, and state duration distribution $T(s,s^{\prime },t^{\prime })=\{T_1,T_2,T_3,T_4,T_5\}$ While iteration < 1000 do $\hspace{1em}\Delta <0$; //Policy evaluation  for s = 1:5 do//5 technical condition levels $\hspace{1em}\hspace{1em}v(s,t)=v^{\pi }(s,t)$   //compute value function $\hspace{1em}\hspace{1em}New\_v(s,t) =\mathrm{R}\left(\mathrm{s}\right)+D_T·\pi \left(a\right|s)·T(s,s^{\prime },t^{\prime })·\sum \gamma ·v^{\pi }(s^{\prime },t^{\prime }+t)$ $\hspace{1em}\hspace{1em}\Delta =\mathrm{m}\mathrm{a}\mathrm{x}\left(\right|v\left(s,t\right)-Ne{w}_{v\left(s,t\right)|}, \Delta )$  Until Δ < θ//check if the value function has converged  //policy update  for s = 1:5 do//5 technical condition levels $\hspace{1em}\hspace{1em}old\_policy=\pi \left(a\right|s)$//previous maintenance strategy $$\begin{gathered} \hspace{1em}\hspace{1em}q(s,a,t)=R_{a,s}+\gamma {\displaystyle \sum _{s\prime \in S}}{M}_{A_a}(s,s\prime )\cdot New\_v(s,t) \\ \hspace{1em}\hspace{1em}new\_policy={max}_{a}q(s,a,t)//updated\; maintenance\; strategy \end{gathered}$$  End  If old_policy = new_policy do   return $New\_v(s,t)$//selecting the optimal value function and optimal strategy  else: $\hspace{1em}\hspace{1em}\pi \left(a\right|s)=new\_policy$//return to policy evaluation and repeat the process  End  iteration+ = 1 End Output: The optimal strategy.

When comparing Figure 5 and Figure 6, it can be observed that the maintenance strategies for elements with a service duration of 6 years and 2 years differ. Due to the lesser number of defects in elements with a service duration of 2 years, the maintenance strategies for elements in each state of the bridge were as follows: (1) For elements in State 3, execute Maintenance Strategy 4 to repair the visible defects in the concrete. (2) For elements in State 4, execute either Maintenance Strategy 3 or Strategy 4 to repair either the exposed reinforcement corrosion or the visible defects in the concrete. (3) For elements in State 5, execute Maintenance Strategies 2 and 3 to repair the concrete cracks and exposed reinforcement corrosion.

6. Conclusions

This article aims to minimize the maintenance costs incurred at the project level of bridges. Based on the actual condition of bridge defects, the costs and benefits of different maintenance strategies were analyzed, and an optimization model for maintenance decision making was constructed. The applicability of the dynamic programming–policy iteration algorithm in solving the optimization problem of maintenance decision making at the project level of bridges was studied. The following conclusions were drawn:

(1) In order to address the issues of unclear quantification and inaccurate estimations of costs and benefits in bridge maintenance decision analysis, this study simplifies the estimation process of maintenance costs. It improves the accuracy and reliability of estimations by considering the actual defect types and development of bridge defects, as well as the importance of future maintenance costs. The unit direct cost in the budget unit price method was used as the unit cost of maintenance measures, thereby aiming to improve the accuracy and reliability of estimations based on the actual problems of bridge maintenance.

(2) Considering the limitation of the widely used Markov model in not accurately representing the time aspect, the Weibull distribution was introduced to model the duration of the elements in different technical conditions. The duration of bridge elements in each technical condition was determined, and it served as a decision basis through which to track the degradation of bridges in real time, to promptly detect and address issues, and to ensure the safety and reliability of bridges.

(3) In the analysis of bridge maintenance decision making, where the decision results do not propose specific maintenance plans at the defect level, a sequential decision-making model was established and solved using the dynamic programming–policy iteration algorithm. The decision factors, such as the state distribution vector, cost matrix, and maintenance effectiveness matrix, were related to the actual bridge defects. The decision results can provide specific maintenance plans for elements with different service durations and different technical conditions, thereby achieving targeted decision making for the purpose of precise maintenance. The model can also adjust maintenance strategies based on the availability of maintenance funds to ensure the feasibility, economy, and sustainability of the decision results.

(4) The sequential model constructed in this study uses a reward function based on bridge maintenance costs. However, in bridge maintenance decision making at the project level, other factors need to be considered, such as the importance weight of elements and the construction difficulty of maintenance plans. Therefore, future research will focus on considering more influencing factors in bridge project-level maintenance decision making. In addition, the use case of this research is a prestressed concrete box girder. The adaptability of other parts and other bridge types such as arch bridges, cable-stayed bridges, and suspension bridges remain to be proven.