Health Status-Based Predictive Maintenance Decision-Making via LSTM and Markov Decision Process

Abstract

Maintenance decision-making is essential to achieve safe and reliable operation with high performance for equipment. To avoid unexpected shutdown and increase machine life as well as system efficiency, it is fundamental to design an effective maintenance decision-making scheme for equipment. In this paper, we propose a novel maintenance decision-making method for equipment based on Long Short-Term Memory (LSTM) and Markov decision process, which can provide specific maintenance strategies in different degradation stages of the system. Specifically, the LSTM model is firstly applied to predict the remaining service life of equipment to distinguish its health state quantitatively. Then, based on the bearing residual life prediction curve, the degradation process model is constructed, and the corresponding parameters of the model are identified. Finally, the bearing degradation curve is obtained by the degradation process model, based on which the Markov decision process model is constructed to provide accurate maintenance strategies for different health conditions of system. To demonstrate the effectiveness of the proposed method, an experimental study with the full life cycle data set of rolling bearings is carried out. The experimental results show that the proposed method can achieve efficient maintenance decisions for bearings under different health states, which provides a feasible solution for the maintenance of bearing systems.

1. Introduction

With the continuous improvement in modern industrialization, as well as the progress of society and the rapid development of science and technology, mechanical equipment is becoming more intelligent, systematic and modular. The functions of mechanical equipment have become increasingly diversified to meet the growing requirements of industrial production. In the process of long-term operation, mechanical equipment will be gradually aging, along with gradually declining operating performance and remaining life, the possibility of failure will increase. Once the failure occurs, it may cause costly industrial downtime, casualties or even serious social impact. Therefore, how to design effective maintenance decision-making scheme, in order to ensure the long-term safe and stable operation of the mechanical equipment is an urgent problem to be solved.

To ensure the reliable and safe operation of equipment, the existing research paid a lot of attention to fault detection and diagnosis for different equipment via various means [1,2,3,4]. Actually, further study on effective maintenance decision-making method is also of great importance. Due to the crucial role in mechanical equipment, maintenance decisions for bearings have drawn increasing attention of many scholars [5,6]. The maintenance decision-making scheme for the bearing system is also our focus in this paper.

To attain safe and reliable operation with high performance of equipment and achieve the lowest possible maintenance costs at the same time, a novel maintenance decision-making method for equipment based on LSTM and Markov decision process is proposed in this paper. To this end, the prediction curve of the bearing remaining life is firstly obtained by applying the LSTM model. Then, the degradation process model is constructed, and the corresponding parameters are estimated based on the bearing remaining life prediction curve. Finally, based on the bearing degradation curve acquired by the degradation process model, the Markov decision process model is applied to provide optimal maintenance strategies for different health conditions of the system. The main contributions of this paper are given as follows.

(1)

A novel maintenance decision-making method is developed for rotating mechanical system.

(2)

An LSTM model is adopted to predict the remaining life of system, and the remaining life prediction data are used as the input of the following degradation process model to identify the model parameters.

(3)

A maintenance decision-making model is constructed based on Markov decision process to provide an effective maintenance solution for equipment. Furthermore, the revenue of maintenance decisions under different health conditions is designed for the instruction of maintenance strategies. Moreover, the maintenance decision-making model is tested on the experimental platform of rolling bearings, and the effectiveness of the proposed method has been validated.

The remainder of this paper is organized as follows. In Section 2, the related work is reviewed, which summarizes the main research progress in the field of maintenance decision-making. Section 3 presents the framework of the proposed method in detail, including the prediction of remaining life based on LSTM and maintenance decision-making model for bearings. The effectiveness of the proposed method is verified by the experimental study in Section 4. Finally, the conclusions of this paper are summarized in Section 5.

2. Literature Review

With the development of science and technology, as well as increasing demand for economic and healthy operation of equipment, autonomous decision-making and equipment maintenance decision-making has drawn increasing attention from the academy [7,8,9]. In the past decades, the research topic of maintenance decision-making has been widely studied [10]. The existing methods can be mainly divided into two categories: time-based maintenance (TBM) and condition-based maintenance (CBM).

Many scholars have made in-depth research on TBM strategy optimization. Buchholz, Peter et al. [11] proposed a general model of partially observable states and non-exponential fault, maintenance and repair time based on phase distribution. D.E. Ighravwe et al. [12] proposed a fuzzy objective programming model and used it to establish a single objective function of maintenance optimization considering random constraints, so as to generate reliable information for fault maintenance plan. Considering the time-based preventive maintenance scheduling problem under the uncertainty of unit life distribution, De Jonge et al. [13] evaluated the long-term benefits of initially delaying preventive maintenance and made the benefits maximization through the numerical research. Yiming Chen et al. [14] proposed two optimization problems by taking the static availability or expected performance capacity of the system as the goal.

The condition-based maintenance (CBM) is based on the methods of integrating current state prediction, plan diagnosis and future state prediction. These methods can be classified into physical model-based methods, data-driven methods and hybrid methods. Guang Zou [15] developed a probabilistic maintenance optimization method using information value (VOI) calculation and Bayesian decision optimization. The VOI based approach explicitly quantifies the added value of future inspections and gives the best decision by directly modeling decision alternatives and evaluating their expected results.

In the field of CBM, more and more scholars use the Markov decision process to study the degradation process of equipment. Paté-Cornell et al. [16] applied Markov chains with four states to simulate the degradation process of production system, where time-based maintenance and three condition-based maintenance strategies are considered. The latter is based on product inspection, machine signals and signals provided by product in service. Minou C.A. Olde Keizer et al. [17] constructed a parallel system, which is subject to both fault dependence and economic dependence by maintenance cost through load sharing. The system is formulated as a Markov decision process, where the optimal replacement decision is obtained to minimize the long-term average cost per unit time. Yaqiong Lv and Qianwen Zhou et al. [9] proposed an intelligent predictive maintenance system for production equipment multi granularity fault based on BP neural network and fuzzy decision-making, which successfully realized the automatic predictive maintenance decision-making. Renny Arismendi et al. [18] explored the application of piecewise deterministic Markov process (PDMP) to cover different modeling assumptions, such as non-ignorable maintenance delay and inspection-based status monitoring.

In addition, some researchers consider the combination of the two types of methods in applications. Mckone and Weiss [19] combined CBM with TBM methods. The available status information is limited to potential fault signals that may be received before the actual fault. Therefore, the performance of CBM depends on the prediction accuracy. In some cases, TBM or the combination of CBM and TBM is preferred.

From the state of art and development of the study on equipment maintenance decision-making, existing research has been demonstrated by relatively ideal research results in some respects. However, in the field of equipment maintenance decision-making, less efforts have been reported to systematically map out the specific maintenance strategies in different degradation stages of the system, which is worthy to be further explored. Due to the superior ability to find a strategic solution with maximum return and broad application prospects in automatic control and recommendation systems, the Markov decision process has great potential in the field of equipment maintenance decision-making. Motivated by the aforementioned studies, this paper develops a novel maintenance decision-making scheme based on LSTM and Markov decision process, which can provide effective maintenance strategies in different degradation stages of the equipment.

3. Methodology

The framework of the maintenance decision-making method proposed in this paper is shown in Figure 1. Specifically, the LSTM model is applied to predict the remaining life curve of the equipment. Then, based on the bearing remaining life prediction curve, the degradation process model is constructed, and the parameters of the model are identified. Finally, the bearing degradation curve is obtained by the degradation process model, based on which the Markov decision process model is constructed to provide accurate maintenance strategies for different health conditions of system.

3.1. Prediction of Remaining Life Based on LSTM

LSTM is a special type of Recurrent Neural Network (RNN) that can learn long-term dependent information, which has been demonstrated by many successful applications [20,21].

The specific structure of LSTM is shown in Figure 2, where Xt is the input of cell state at time t and Ht is the output of cell state at time t. LSTM realizes information protection and control through three gate unit structures, including input gate, forgetting gate and output gate.

(1)

Forgetting gate

The first step in LSTM is to decide what information will be discarded from the cellular state. The decision is made through the forgetting gate. The gate will read the output of the hidden layer at the last moment and the input of the current cell, and then output a value between 0 and 1, where 1 means “completely preserved”, 0 means “completely discarded”.

(2)

Input gate

The next step is to decide how much new information will be added to the cellular state. To this end, there are two steps to be performed: first, the input gate determines which information needs to be updated. A tanh layer generates a vector, which is the alternative content for updating. In the second step, the two parts are combined to update the cell state.

(3)

Output gate

Finally, we need to determine the output value. This output will be based on the cell state. Firstly, we run a sigmoid layer to determine which part of the cell state will be output. Then, we deal with the cell state through tanh (get a value between −1 and 1) and multiply it with the output of the sigmoid gate. Finally, we just output the part of the output we determined.

Through the above three gating units, LSTM realizes the selective retention and output of information, and meanwhile solves the problem of gradient disappearance of RNN.

The remaining life prediction based on LSTM can integrate the original learning samples with the new learning mode to realize the re-training of samples. It can not only improve the accuracy of remaining life prediction, but also has the characteristics of fast convergence and high stability. Due to the great advantages in the processing of serial data, LSTM is applied for remaining life prediction of bearings by making use of the vibration signals in operation, which also have serial characteristics.

In what follows, the remaining life prediction data obtained by LSTM model will be used to quantify the health status of the bearing.

3.2. Degradation Process Model

The bearing degradation curve in ideal conditions is shown in Figure 3. According to the curve, the trend of the bearing degradation has the following characteristics [22]:

(1)

The normal operation time of bearing is long, accounting for 80–90% of the whole life cycle of the bearing.

(2)

When a small crack appears on the surface of the bearing rolling elements or raceways, the bearing begins to enter the degradation stage.

(3)

When the degree of bearing degradation accumulates to a certain extent, the probability of bearing damage and equipment failure will increase significantly

The degradation quantity of rolling bearing in a certain period $\Delta t$ is expressed as Z$\left(\Delta t\right)$, including both continuous degradation quantity and sudden degradation quantity in the process of bearing degradation. The degradation process of bearing follows the Gauss–Poisson process:

$$Z\left(\Delta t\right)=X\left(\Delta t\right)+\beta Y\left(\Delta t\right)$$

(1)

where $X\left(\Delta t\right)$ denotes the continuous degradation of bearings, and $X\left(\Delta t\right)$~N$\left(\mu ,{\sigma }^2\right)$. $Y\left(\Delta t\right)$ represents the quantity of degradation due to sudden factors, and $Y\left(\Delta t\right)$~Poisson(λ). β is the average degradation amount generated by each sudden degradation.

In order to evaluate the health state of the system, the health score is introduced in the construction of degradation process model. The initial health score of the bearing is set to be 1. After operation time t, the normal continuous degradation of the bearing is denoted by $X\left(t\right)$, and the quantity of sudden degradation is $Y\left(t\right)$, then the health score of the bearing is given by:

$$Ht=1-{\displaystyle \sum }_{t=0}^t\left(X\left(t\right)+\beta Y\left(t\right)\right)$$

(2)

The parameters of the health state degradation process can be identified by the historical health score degradation data, which is discussed in the following.

After obtaining the remaining life prediction data, the bearing health score degradation data can be obtained from the following formula:

$$H_t\left(n\right)=H\left(t\right)-H(t+1)$$

(3)

Assume that $H_N\left(n\right)$(n = 1, 2, 3, ..., N) is a group of historical degradation data of health score, where n represents the state number. According to the health score degradation data $H_N\left(n\right)$, the parameters in Equation (2) are estimated by calculating the central moments of each order of $H_N\left(n\right)$. The estimation of parameters is given as follows:

$$E\left(H_N\right)=\mu +\lambda \beta$$

(4)

$$D\left(H_N\right)={\sigma }^2+\lambda {\beta }^2$$

(5)

$$E{\left(H_N-E\left(H_N\right)\right)}^3={\beta }^3\lambda$$

(6)

$$\begin{array}{c}E{\left(H_N-E\left(H_N\right)\right)}^4\hfill \\ =3{\sigma }^2+3{\beta }^4{\lambda }^2+{\beta }^4\lambda +6{\sigma }^2{\beta }^2\lambda \end{array}$$

(7)

where μ, σ, λ, β are the parameters of rolling bearing degradation process. The central moments of each order of the group of data are calculated by the health score degradation data, which can be recorded as H₁, H₂, H₃, …, H_n. The obtained central moments are expressed as a₁, a₂, a₃, a₄ respectively, which can be calculated as follows:

$$a_1=E\left(H_N\right)=\frac{1}{n}{\displaystyle \sum }_{N-1}^n{H}_N$$

(8)

$$a_2=D\left(H_N\right)=\frac{1}{n}{\displaystyle \sum }_{N-1}^n{(H_N-a_1)}^2$$

(9)

$$a_3=E{\left(H_N-E\left(H_N\right)\right)}^3=\frac{1}{n}{\displaystyle \sum }_{N-1}^n{(H_N-a_1)}^3$$

(10)

$$a_4=E{\left(H_N-E\left(H_N\right)\right)}^4=\frac{1}{n}{\displaystyle \sum }_{N-1}^n{(H_N-a_1)}^4$$

(11)

Based on the above equations, each parameter of the Gauss–Poisson process model is given by:

$$\lambda =\frac{a_3^4}{{\left(a_4-3a_2^2\right)}^3}$$

(12)

$$\sigma =\sqrt{a_2-\frac{a_3^2}{a_4-3a_2^2}}$$

(13)

$$\mu =a_1-\frac{a_3^3}{{\left(a_4-3a_2^2\right)}^2}$$

(14)

$$\beta =\frac{a_4-3a_2^2}{a_3}$$

(15)

According to the above discussions, the parameters of the bearing degradation process are completely identified.

3.3. Maintenance Decision-Making Model

3.3.1. Markov Decision Process Model

The health score (0–1) of the system can be obtained in Section 3.2. Higher health score indicates better system health state. Health score 1 means that the system is completely healthy, and health score 0 indicates that the system is failed.

The health score can effectively represent the deterioration of the system, motivating us to use to evaluate the health status of the system. The health score is divided into four intervals: [1, 0.8), [0.8, 0.6), [0.6, 0.4) and [0.4, 0], corresponding to four different health states of the bearing:

Healthy (that is, the bearing is under a completely healthy state with only slight degradation),

Good (the bearing begins to deteriorate but is not obvious),

Sub-health (the bearing has been seriously degraded and its performance has been obviously reduced),

Damaged (the bearing is completely damaged and cannot be used).

Their health states are recorded as 1, 2, 3, 4 respectively. Therefore, the health state set of rolling bearing can be defined as S = {1, 2, 3, 4}, which is a continuous Markov process. Since the bearing degradation process is continuous, the rolling bearing must be in a certain state (health, good, sub-health, damage) at any time in its full life cycle [23]. The health state transition process of rolling bearing is shown in Figure 4, where each circle represents different health states, and the value in the circle represents the benefit of remaining in each state.

3.3.2. Transition Probability

In this paper, the Monte Carlo method is used to calculate the transition probability of the Markov process [24]. The transition probability can be calculated as follows:

$$P_{ij}=\frac{M_{ij}}{M_i}$$

(16)

where $P_{ij}$ is the transition probability of state from i to j; $M_{ij}$ is the number of samples transferring from state i at the last moment to state j at the next moment, and $M_i$ is the total number of samples in state i.

3.3.3. Maintenance Effect

According to the impact of different maintenance modes on bearing service life, the maintenance effect of different maintenance modes can be represented, as well as the impact of different maintenance modes on the health status of the bearing.

In this paper, the effect of different maintenance modes in this paper is given as follows. Simple maintenance applied to rolling bearings can prolong the bearing service life by 10% on average. If the bearings are repaired by complete maintenance, the health score can directly change to 1. If we apply state maintenance to repair rolling bearings, the bearing service life can be extended by 40% on average. The health states transition probability matrix under different maintenance states can be obtained through the health score represented by the life extension.

3.3.4. Cost Analysis

Different maintenance modes of bearings under different health conditions brings different cost, which has significant impact on the decision-making process. The cost includes three parts:

(1)

Maintenance costs (the maintenance costs incurred by various maintenance activities);

(2)

Continuous maintenance costs (the costs incurred from continuous care and maintenance of rolling bearings to keep them healthy and effective);

(3)

Signal detection costs (the costs caused by the vibration signal detection of the bearing to identify the current health status).

At present, there is no uniform standard for the maintenance mode and cost of mechanical equipment, and the maintenance mode setting in this paper is only to verify the effectiveness of this method. Therefore, this paper formulates the maintenance cost based on some maintenance experience. To sum up, the costs of each simple maintenance, state maintenance and complete maintenance are 15, 40 and 300, respectively, where the relative value is selected to facilitate the calculation of the total reward.

For a Markov decision process, Gt is defined as the cumulative reward of the system, which can be expressed as:

$$G_t=R_{t+1}+R_{t+2}+R_{t+3}+\cdots ={\displaystyle \sum }_{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}$$

(17)

where γ represents the discount factor, which is set as 1. Rt denotes the income at time t.

4. Experiment Analysis

4.1. Bearing Data Acquisition

The data used in this paper are the life cycle experimental data of bearings from Xi’an Jiaotong University [25]. The experimental platform is shown in Figure 5 [26]. The accelerated life tests for various types of bearings (including rolling bearings and sliding bearings) under different working conditions can be carried out on the experimental platform, where the life cycle data of the test bearings can be collected. The main bearing operating parameters, including the radial force and the rotating speed, which can be adjusted by the test-bed. The test bearing type is LDK UER204 rolling bearing, whose parameters are shown in

Table 1. LDK UER204 Bearing parameters.

Parameters	Numerical Value
Diameter of inner race/mm	29.30
Outer ring raceway diameter/mm	39.80
Bearing pitch diameter/mm	34.55
Basic dynamic load rating/N	12,820
Ball diameter/mm	7.92
Number of balls	8
Contact angle/(°)	0
Basic static load rating/kN	6.65

.

4.2. Prediction of the Remaining Useful Life of Bearings

The aforementioned data are used for the verification of the proposed method. Several groups of data samples are selected as the training set from the bearing life cycle data of Xi’an Jiaotong University, including Bearing 1_1, earing1_2 and Bearing1_4. While Bearing1_5 is selected as the test set. (Operating condition: speed 2100 r/min, radial force 12 kN, sampling frequency 25.6 kHz, sampling interval 1 min, sampling duration 1.28 s).

The actual remaining life of the bearing is used as the training and testing label value y. The process of label construction is discussed as follows. label 1 represents the bearing state that it is in good condition, and label 0 means that the bearing is in complete failure. For example, Bearing1–2 dataset has a total of 2496 groups of data, which means the total life of the bearing is 2496 min. If the current sample is the 1000th datum, then the remaining life of the bearing is 1496 min, and the value of the corresponding label y under the sample is 1496/2496 = 0.599358. According to the remaining life of the rolling bearing, the data samples, are labeled in the same manner.

The LSTM model is designed based on the Python open-source deep learning framework. In the experiment, the Adam optimizer is selected to optimize the training loss of LSTM model. Adam is a popular optimizer in the current architecture. Compared with other optimizers, it can learn parameters adaptively, which has the advantages of fast convergence, small memory requirements, and better processing of noise samples. The obtained life prediction curve of Bearing1_5 is shown in Figure 6, and the prediction accuracy rate is 96.7%.

To illustrate, the status of bearing is provided. As shown in Figure 7, at time point 400, the bearing status is shown as the left bearing, while at time point 1400, the bearing status is shown as the right bearing. It can be seen that the left bearing is in good condition, while the right bearing has been severely worn, which is consistent with the life prediction results by the LSTM model. Therefore, the method in this paper can fit well with the whole life degrading trend of the bearing so as to predict the remaining life of it.

4.3. Parameters Estimation

At present, we have obtained the predicted value of the remaining life of the bearing. Based on this, we subtract the predicted value of the remaining life of the bearing at adjacent time points to obtain the deterioration of the bearing health score ($H_N\left(n\right)$), then we can calculate the relevant parameters of the model.

Based on the health score of bearing life prediction curve obtained in Figure 6, which represents the degradation quantity of bearings, the parameters of the bearing degradation process model are identified as follows:

$$\mu =0.000243, o=0.0208 \beta =0.000596, \lambda =0.400$$

According to the obtained bearing degradation process model, we can estimate the bearing degradation curve as shown in Figure 8.

Based on the bearing degradation curve, the transition probability of Markov decision process can be calculated. According to the maintenance effect in Section 3.3, the impact of each maintenance mode on the bearing health state transition is discussed as follows:

The state transition probability matrix after simple maintenance is:

$$\mathrm{A}1=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0.64& 0.36& 0& 0\\ 0& 0.27& 0.63& 0\\ 0& 0& 0.09& 0.91\end{array}\right]$$

The state transition probability matrix after condition-based maintenance is:

$$\mathrm{A}2=\left[\begin{array}{cccc}1& 0& 0& 0\\ 1& 0& 0& 0\\ 0.14& 0.71& 0.15& 0\\ 0& 0& 0.29& 0.71\end{array}\right]$$

The state transition probability matrix after complete maintenance is:

$$\mathrm{A}3=\left[\begin{array}{cccc}1& 0& 0& 0\\ 1& 0& 0& 0\\ 1& 0& 0& 0\\ 1& 0& 0& 0\end{array}\right]$$

According to Equation (16), the health state transition probability matrix of the rolling bearing can be obtained as:

$$\mathrm{A}4=\left[\begin{array}{cccc}0.977& 0.023& 0& 0\\ 0.067& 0.916& 0.017& 0\\ 0& 0.034& 0.933& 0.033\\ 0& 0& 0.013& 0.987\end{array}\right]$$

The row of the above matrix represents the original state, and the column is the state after transition. The value means the probability of transition from the original state to the new state. Finally, the Markov decision process model of the entire bearing degradation process is obtained as shown in Figure 9. Each circle of the figure represents the different health states of the bearing, in which the value represents the benefit of remaining in each state, and the value on the line of circles represents the transition probability of each state.

To calculate the value of each maintenance decision on each state, the Bellman equation is used to iteratively calculate the value function of each state, and the following results are obtained: Revenue in healthy state R1 = 4631.84, revenue in good state R2 = 4195.92, revenue in sub-health state R3 = 2141.21, and revenue in damaged state R4 = 0.

The benefits of different maintenance modes under different conditions are obtained by combining the effects of the above maintenance decisions on different health status, as shown in

Table 2. Revenue from different maintenance decisions.

Maintenance Modes	Health	Good	Sub-Health	Damage
Simple maintenance	4616.84	4429.91	2466.86	177.71
Condition-based maintenance	4591.84	4591.84	3908.72	580.95
Complete maintenance	4331.84	4331.84	4331.84	4331.84

.

4.4. Summary

It can be seen from Table 2 that when the rolling bearing is in healthy state, and simple maintenance is applied, i.e., routine maintenance, the maximum benefit can be obtained. While the benefit of condition-based maintenance is only slightly lower than that of simple maintenance. When the rolling bearing is under good condition, the maximum benefit can be obtained by carrying out appropriate condition maintenance according to its condition, and considerable benefit can be gained by carrying out simple maintenance or complete maintenance under this condition. If the rolling bearing is under sub-health state, the benefit of complete maintenance, i.e., directly replacing the bearing, is the largest, which is far greater than that of the other two maintenance modes. However, if the rolling bearing has been damaged, only when the bearing is completely repaired, that is to say, the replacement of the bearing can obtain greater benefits.

Our conclusions obtained above are consistent with the historical experience of bearing maintenance, verifying that the proposed maintenance decision-making method can provide effective guidance for the maintenance strategy of rolling bearings under different states.

5. Conclusions

In this paper, a maintenance decision-making scheme for equipment is proposed based on LSTM and Markov decision process, which can provide effective maintenance decisions for system under different degradation stages. First, the LSTM model is adopted to predict the remaining service life to distinguish the health state quantitatively. Then, the degradation process model is constructed, and the parameters of the model are identified. With the aid of the degradation curve obtained from the degradation process model, the maintenance decision-making model is established based on the Markov decision process. Moreover, to facilitate more appropriate maintenance strategy identification, the revenue of maintenance decisions under different health conditions is analyzed. Experimental study with the full life cycle data set of bearings is carried out to demonstrate the effectiveness of the proposed method. Besides the rotating mechanical systems, the application of the proposed method can be further extended to other industrial fields.