A Two-Stage Transfer Regression Convolutional Neural Network for Bearing Remaining Useful Life Prediction

Abstract

Recently, deep learning techniques have been successfully used for bearing remaining useful life (RUL) prediction. However, the degradation pattern of bearings can be much different from each other, which leads to the trained model usually not being able to work well for RUL prediction of a new bearing. As a method that can adapt a model trained on source datasets to a different but relative unlabeled target dataset, transfer learning shows the potential to solve this problem. Therefore, we propose a two-stage transfer regression (TR)-based bearing RUL prediction method. Firstly, the incipient fault point (IFP) is detected by a convolutional neural network (CNN) classifier to identity the start time of degradation stage and label the training samples. Then, a transfer regression CNN with multiloss is constructed for RUL prediction, including regression loss, classification loss, maximum mean discrepancy (MMD) and regularization loss, which can not only extract fault information from fault classification loss for RUL prediction, but also minimize the probability distribution distance, thus helping the method to be trained in a domain-invariant way via the transfer regression algorithm. Finally, real data collected from run-to-failure bearing experiments are analyzed by the TR-based CNN method. The results and comparisons with state-of-the-art methods demonstrate the superiority and reliable performance of the proposed method for bearing RUL prediction.

1. Introduction

As a critical component in almost all forms of rotating machinery, the health operation of bearings has a major impact on the performance of mechanical systems [1,2,3]. The tough working environment of high temperature, high pressure and humidity will inevitably cause bearings degradation [4]. Thus, to avoid the associated economic losses and catastrophic failures, the remaining useful life (RUL) prediction of bearings plays an increasing crucial role in many industrial fields, including mining, aviation, railway and so on [5,6].

In the literature, a great variety of techniques have been proposed for RUL prediction, which coarsely include model-based methods and data-driven methods [7]. Owing to the era of big data, more and more data types and datasets are available with the development of various sensing techniques [8], which have led to the enormous development of data-driven RUL prediction methods in recent decades [9]. Instead of using failure mechanisms in model-based methods, data-driven methods rely on historical data and attempt to directly derive the degradation pattern of a machine from the data for RUL prediction [10]. In traditional data-driven RUL prediction methods, a signal processing technique is always applied firstly for feature extraction, such as statistic analysis [11], empirical mode decomposition (EMD) [12], wavelet transform [13], sparse representation [14,15] and Gaussian mixture model (GMM) [16]. Then, the extracted degradation features are used as the inputs of a regression model for RUL prediction. Machine learning methods are widely used as the prediction model, such as neural networks [17], Bayesian methods [18], support vector regression (SVR) [19] and hidden Markov model [20]. Recently, deep learning has shown its potential in fault diagnosis of mechanical systems because of its strong capacity for feature extraction, which aims to extract high-level features from raw data directly and automatically [21]. Since 2016, deep learning methods have been successfully introduced to fault diagnosis of many mechanical systems or components [22], including bearing [23], motor [21] and gearbox [24]. The research on deep-learning-based RUL prediction is relatively scarce. Deep convolutional neural network (CNN) was applied for RUL estimation of aero-engine unit prognostics. The experimental results verified the superiority of the deep CNN [25]. J. Zhu et al. proposed a multiscale CNN to keep both the local and global information synchronously, then estimated the RUL of bearings [8]. Y. Wu et al. proposed a long short-term memory (LSTM) network-based RUL prediction method. The performance improvement of the proposed method was verified by monitoring the health conditions of aircraft turbofan engines [26]. The major challenge in applying deep learning in bearing RUL prediction lies in the degradation pattern of bearings that can be much different from each other due to many complex factors, such as materials, manufacturing process and operational environment [27]. Therefore, the training data and test data for RUL prediction model obey significantly different distributions. Thus, the deep learning model trained with the vibration signals of a bearing usually cannot work well directly for RUL prediction of another bearing. In this case, transfer learning provides a good way to solve the problem [28,29]. Transfer learning aims to learn a model with better performance for the target domain by transferring the knowledge learned from the related source domain. The model constructed by knowledge transfer can reduce the requirement of large labeled samples of the target domain and obtain good generalization performance [30]. Therefore, it holds the ability to overcome the aforementioned challenge in recent data-driven RUL prediction methods. Ding et al. proposed an RUL prediction method for rolling bearing based on a deep transfer auto-encoder, which can easily determine the degradation stage of bearings under different working conditions [31]. Mao et al. proposed a transfer learning approach to improve the RUL prediction performance across different working conditions based on phase space warping (PSW), dynamic time warping (DTW), and meta-degradation information [32]. Huang et al. constructed a transfer depth-wise separable convolution recurrent network for bearing RUL prediction from the same public datasets considering different work conditions [33]. Meng et al. proposed a dynamic reweighted domain adaption method for cross-domain bearing fault diagnosis, which was verified via extensive experiments on several bearing fault diagnosis datasets [34]. However, most of the existing methods predict the RUL directly, without considering the start time of failure. Since little fault information can be found from the vibration signals under normal condition, it is important to find the start time of failure, which is called the incipient fault (IF) point in this paper.

Therefore, we propose a two-stage transfer regression (TR) CNN method for RUL prediction of an unknown bearing with the help of the historical data of training bearings. Firstly, a CNN classifier is constructed for IF point identification. Thus, the vibration data of training bearings can be labeled based on the IF point. Then, a TR CNN is proposed in this paper, which is trained by the labeled training data and used for an unknown bearing RUL prediction. The innovations of this paper are summarized as follows.

1.

An end-to-end CNN classifier is firstly constructed for IF point identification, which can detect the IF point without the help of feature extractor.

2.

A two-stage TR-CNN method is proposed based on transfer learning for bearing RUL prediction, which can help the method to be trained in a domain-invariant way by minimizing the probability distribution distance, namely the maximum mean discrepancy (MMD). Since the distributions of the bearing vibration datasets are usually different from each other, the proposed method is expected to be a promising method for RUL prediction.

3.

Via proposing the regression loss, classification loss, MMD term and regularization term, a multiloss CNN model is constructed as the backbone architecture to extract the fault information from the fault diagnosis for RUL prediction, thus making full use of the historical data and increasing the performance of the proposed method.

4.

Experimental results of the publicly available PRONOSTIA dataset [35] demonstrate the effectiveness of the proposed method.

The remaining parts of this article are organized as follows. Section 2 describes the proposed TSTR-CNN model. The effectiveness and the transfer ability of the proposed TSTR-CNN method are validated on a motor experimental platform. At last, Section 4 concludes the paper.

2. Proposed Approach

2.1. Problem Description

A machinery RUL prediction system is generally composed of four stages: data acquisition, health indicator construction, IF point identification and RUL prediction [36]. In this paper, the experiment data come from the publicly available PRONOSTIA dataset, and the proposed method is constructed based on a deep learning method, which can extract features automatically without the need for the health indicator construction, so we mainly focus on the last two stages.

Firstly, since we cannot monitor the condition all the time during the bearing operation, the IF point should be determined first. Thus, the data after IF point contain the degradation information and can be used to train the prediction model. In addition, the training samples can be labeled based on the IF point.

For the RUL prediction stage, let $D^s$ and $D^t$ denote the source domain and target domain, respectively. $X^s$ and $X^t$ represent the sample spaces of source domain and target domain with marginal probability distribution $P(X^s)$ and $P(X^t)$, respectively. $x^s$ and $x^t$ are the data samples. Because the degradation pattern of bearings can be much different from each other due to many complex factors, the training data and test data for the prediction model are subject to substantially different data spaces and marginal distributions, namely, $X^s\ne X^t$ and $P(X^s)\ne P(X^t)$, as shown in Figure 1a. The label space is defined as $y=(1,2,3,\dots ,k)$. In this paper, the relative RUL percentage was applied instead of the absolute RUL value, therefore, the labels of the source and target domains were the same, that is, $y^s=y^t=y\in [0,1]$. The goal of the transfer regression algorithm is to learn a kernel function F that satisfies $P(F(X^s))=P(F(X^t))$ and $P(y^s|F(X^s))=P(y^t|F(X^t))$, as shown in Figure 1b. In this way, the trained regression model is also expected to work well for the RUL prediction of new bearings.

2.2. Convolutional Neural Network

The proposed RUL prediction framework was constructed based on a CNN backbone; therefore, the CNN model is briefly introduced here. As one of the most commonly used model of deep learning methods, a CNN can learn how to extract feature and recognize patterns of different tasks directly and automatically. Generally, a CNN is a multistage neural network. The former part consists of several layers, which contains the convolutional layer and the pooling layer. Classification is implemented by the latter part, in which fully connected layers are employed.

2.2.1. Convolutional Layer

The convolutional layer generates the feature maps by sliding the kernels on the input signals. Each of these kernels outputs a feature map. Then, by filtering these feature maps, the desired fault features can be extracted and rearranged according to the similarity of the embedding characteristics. By means of a series of convolutions, the desired features can be rearranged and mined by similar statistical characteristic among the feature maps. Then, a nonlinear activation function is imposed after convolution to obtain the output feature map [8]. In this paper, the ReLU function was applied as the activation function. Thus, the output feature map of the convolutional layer can be written as:

$$a_j^l=ReLU(\sum _i{a}_i^{l-1}\ast w_{ij}^l+b_j^l)$$

(1)

where ∗ denotes the convolution operator; $a_i^{l-1}$ and $a_j^l$ are the ith input feature map of layer $l-1$ and the jthe output feature map of layer l; $w_{ij}^l$ and $b_j^l$ are the convolutional kernel and bias; $ReLU$ is the rectified linear unit activation function, which is defined as:

$$ReLU(z)=\left\{\begin{array}{cc}0,ifz<0\hfill & \\ z,otherwise\hfill \end{array}\right.$$

(2)

2.2.2. Pooling Layer

In the pooling layer, a statistical value of a local region is calculated as the output, which can reduce the size of feature maps and increase the computational efficiency. In addition, the subsampling operation can also make the output feature maps invariant to small variance. In this paper, max-pooling was utilized:

$$a_j^{l+1}(c,d)=\underset{0\le p,q<m}{max}\left\{a_j^l(c·m+p,d·m+q)\right\}$$

(3)

where m is the step size of pooling and p and q are the pixel of x and y direction, respectively. c and d are the coordinate of x and y direction, respectively.

2.2.3. Fully Connected Layer

The fully connected layer is usually introduced after the feature extraction layers that consist of convolutional layers and pooling layers. A soft-max classifier is usually applied at the last layer of the CNN for classification or regression.

2.3. Incipient Fault Point Identification

IF point identification is the first task for RUL prediction, which is often detected experimentally due to the difference of failure time between machines, which can be time-consuming. Therefore, to make the RUL prediction framework more intelligent, a CNN classifier is firstly proposed to detect the IF point in this stage, which can extract the fault features automatically via the deep networks. As an end-to-end deep learning method, a CNN can detect the IF point automatically without the demand of prior knowledge, physical model or human labor.

In this stage, the normal samples and the samples with obvious fault of the training bearings were labeled as 0 and 1, respectively, and used to train a CNN classification model ${\mathrm{CNN}}_c$. After obtaining the detection model ${\mathrm{CNN}}_c$, all the samples of the training bearings were used as the inputs of ${\mathrm{CNN}}_c$. When the output of ${\mathrm{CNN}}_c$ changed to 1 from 0, the IF point was detected. After obtaining the IF points, all the samples of the training bearings were labeled from 0 to 1 as the relative RUL based on the IF points, for training the CNN regression model. Meanwhile, all the samples were also labeled as 0 or 1 to train the fault classification model. Thus, every sample had two labels: the first label was the relative RUL and the second label the fault type.

The flowchart of the CNN classifier for IF point identification is shown in Figure 2. In the process of RUL prediction, once the IF point of the test bearing has been detected, the vibration data can be used for RUL prediction, which is described in the next two sections.

2.4. Maximum Mean Discrepancy

MMD is a nonparametric distance to estimate the discrepancy between two distributions [37]. Specifically, MMD introduces a reproducing kernel Hilbert space (RKHS) for discrepancy determination. For example, for the samples with two different distributions, the mean values of function f can be obtained by looking for the continuous function f. Then, the mean discrepancy to function f can be obtained by calculating the difference between the two mean values. MMD aims to find a function f that can maximize the mean discrepancy. Thus, MMD can be used as a criterion to estimate whether the two distributions are the same or not. If the MMD of these two distributions is small enough, then they can be regarded as the same distribution and vice versa.

Let F be a continuous function library. For the data from $D^s$ and $D^t$, the MMD is defined as:

$$\begin{array}{c}\hfill MMD[F,P(X^s),P(X^t)]:=\underset{f\in F}{sup}(E_{X^s\backsim P(X^s)}\left[f(X^s)\right]-E_{X^t\backsim P(X^t)}\left[f(X^t)\right])\end{array}$$

(4)

where $:=$ means the value on the right of the equal sign is equal to the value on the left. f is the nonlinear transformation from the original feature space to the RKHS.

In the case where $X^s$ and $X^t$ are independent identically distributed (IID), and there are n and m samples in $X^s$ and $X^t$, respectively, the MMD can be calculated as:

$$MMD[F,X^s,X^t]:=\underset{f\in F}{sup}(\frac{1}{n}\sum _{i=1}^{n}f(x_i^s)-\frac{1}{m}\sum _{i=1}^{m}f(x_j^t))$$

(5)

where $x_i^s$ represents the ith sample from the source domain and $x_j^t$ represents the jth sample from the target domain.

Then, the characteristic kernels guide the RKHS via a kernel mean embedding of the distribution. Thus, the MMD based on the kernel mean embedding is defined as:

$$\begin{array}{cc}\hfill MM{D}_{\mathcal{H}}^2[X^s,X^t]& ={\parallel \frac{1}{n}\sum _{i=1}^{n}f(x_i^s)-\frac{1}{m}\sum _{i=1}^{m}f(x_j^t)\parallel }_{\mathcal{H}}^2\hfill \\ & =\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^{n}k(x_i^s,x_i^s)-\frac{2}{nm}\sum _{i=1}^n\sum _{j=1}^{m}k(x_i^s,x_j^t)+\frac{1}{m}\sum _{i=1}^m\sum _{j=1}^{m}k(x_j^t,x_j^t)\hfill \end{array}$$

(6)

where $\mathcal{H}$ is a universal RKHS and $k(·,·)$ is the characteristic kernel.

2.5. Transfer Regression Method

Although transfer learning provides a promising tool to realize the domain adaption and improve the performance of the target learner by transferring the knowledge extracted from the source domain, it is mostly used for classification tasks. Therefore, a transfer regression (TR) method is proposed based on transfer learning theory for bearing RUL prediction. Considering there may exist a synergistic effect between fault diagnosis and RUL prediction, a multiloss transfer network was constructed in this stage as the backbone architecture. This multiloss structure can learn a shared feature between parallel tasks, which provides an inductive transfer way to use the domain-specific data characteristic of different but related tasks [38]. For example, two simple independent neural networks for fault diagnosis and RUL prediction were constructed, respectively. The parameters of the two neural networks should be optimized separately, which increases the computational requirement. Moreover, because they are trained separately, the feature maps cannot be shared, that is, the information of one task cannot be used in another task. However, as we all know, RUL is related to the degradation condition, and a failure can lead to the bearing degradation. As a result, much information used for fault diagnosis can also be utilized for the RUL prediction task [39]. Therefore, a multiloss network was constructed in this paper to extract the fault information from fault diagnosis, then used for the RUL prediction.

Specifically, the flowchart of the TR-CNN prediction method is graphically illustrated in Figure 3. During the training process, the vibration signals from the two domains were cut into segments as the samples. After the IF point identification step, the labels of the training dataset from the source domain for both fault classification and RUL prediction were obtained. Thus, the training samples from the source domain with corresponding labels, as well as the unlabeled samples from the target domain were used to train the multiloss model. The label of the first task was the relative RUL obtained based on the IF point; the label of second task was the fault type. Therefore, there were two loss functions needed to be optimized: the regression loss and the classification loss. In addition, to adapt the differences between the source domain dataset and target domain dataset, the MMD term should also be minimized. As a result, the final loss function consisted of four parts: regression loss, classification loss, MMD term and regularization term.

2.5.1. Regression Loss

The objective of this task is to fit the degradation process with nonlinear functions. Firstly, after the IF points detection, the samples before the IF points were labeled with 1, and the samples after the IF points were labeled with the healthy percentages:

$$y_2(i)=\left\{\begin{array}{cc}1,T_i\le T_s\hfill & \\ 1-\frac{T_i-T_s}{T_e-T_s},T_i>T_s\hfill \end{array}\right.$$

(7)

where $y_2(i)$ is the label at time $t_i$; $T_e$ is the total time of the test; $T_s$ is the start time of IF; and $T_i$ is the current time.

Then, the samples were labeled with the health percentage and input to the multiloss CNN model. For RUL prediction, the selected loss function was the mean squared error.

$$L_1=\frac{1}{n}\sum _{i=1}^n(y_i-f(x_i))$$

(8)

where $y_i$ is the ith label, and $x_i$ is the ith input. n is the number of samples.

2.5.2. Classification Loss

In this task, the vibration datasets of the training bearings were labeled as 0 or 1 based on IF points, where 0 represents the normal bearing and 1 represents the faulty bearing. Then, the labeled training samples were used as the inputs of the multiloss model. For this task, the selected loss function was the cross-entropy:

$$L_2=H(p(x),q(x))=-\sum _{x}p(x)logq(x)$$

(9)

where $p(x)$ is the target distribution and $q(x)$ is the estimated distribution.

2.5.3. MMD Term

The transfer layer was placed after the fully connected layer $fc7$. In the transfer layer, the discrepancy metric of labeled samples was realized by the MMD term:

$$L_3=MM{D}_{\mathcal{H}}^2[a^{s(fc7)},a^{t(fc7)}]$$

(10)

where $a^{s(fc7)}$ represents the input of the transfer layer $fc7$ in the source domain and $a^{t(fc7)}$ represents the input of the transfer layer $fc7$ in the target domain.

2.5.4. Regularization Term

The regularization term (also called weight decay term) is a commonly used loss, which can decrease the magnitude of the weights and help prevent overfitting. The regularization term is defined as:

$$L_4=\frac{1}{2}\sum _{l=1}^{m-1}\sum _{i=1}^{s_l}\sum _{j=1}^{s_{l+1}}{(W_{ij}^{(l)})}^2$$

(11)

where m is the number of layers in the network; $s_l$ is the size of a hidden unit in layer l; $W_{ij}^{(l)}$ is the weight that connect the ith unit in layer l and the jth unit in layer $l+1$.

2.5.5. Final Loss Function

Therefore, the final loss function of the TR-based RUL prediction model consisted of four parts:

1.

The regression loss $L_1$;

2.

The classification loss of CNN model $L_2$;

3.

The MMD term $L_3$ for domain adaption between the source domain and target domain.

4.

The regularization term $L_4$.

Thus, the final loss function was given as follows:

$$\begin{array}{cc}\hfill L(W,b)=& L_1+{\lambda }_1{L}_2+{\lambda }_2{L}_3+{\lambda }_3{L}_4\hfill \\ \hfill =& \frac{1}{n}\sum _{i=1}^n(y_i-f(x_i))+{\lambda }_1\sum _{x}p(x)logq(x)+{\lambda }_{2}MM{D}_{\mathcal{H}}^2[a^{s(fc7)},a^{t(fc7)}]\hfill \\ & +{\lambda }_3\frac{1}{2}\sum _{l=1}^{L-1}\sum _{i=1}^{s_l}\sum _{j=1}^{s_{l+1}}{(W_{ij}^{(l)})}^2\hfill \end{array}$$

(12)

where ${\lambda }_1$, ${\lambda }_2$ and ${\lambda }_3$ controlled the trade-off among these four loss functions and were determined empirically. b is the bias parameter.

After the training was over, the TR-based RUL prediction model took test data as input to calculate the fault condition and the estimated RUL.

2.6. Training Process

During the training process of the TR-based RUL prediction model, the Adam optimization algorithm was applied to minimize the loss function in Equation (12) [40]. Let $g_t$ represent the gradient vector at time t of the loss function $L(W,b)$, the estimations of the first moment $m_t$ (the mean) and second moment $v_t$ (the uncentered variance) of the gradients $g_t$ can be described as:

$$m_t=\mu [{\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t]$$

(13)

$$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)·g_t^2$$

(14)

where $\mu$ is the control weight; ${\beta }_1$ and ${\beta }_2$ are exponential decay rate of first order and second order, respectively; ${\beta }_1,{\beta }_2\in [0,1]$; $m_0$ and $v_0$ are set with bias correction:

$$\widehat{m_t}=\frac{m_t}{1-{\beta }_1^t}$$

(15)

$$\widehat{v_t}=\frac{v_t}{1-{\beta }_2^t}$$

(16)

Then, the parameters can be updated:

$${\theta }_{t+1}={\theta }_t-\frac{1}{\sqrt{\widehat{v_t}}+ϵ}\widehat{m_t}$$

(17)

where $\theta =(W,b)$ is the parameter set; $ϵ$ is a constant with small positive value used to avoid the division for zero. Thus, the parameters can be optimized more stably.

The flowchart of the proposed RUL prediction framework is presented in Figure 4.

3. Experiments, Results and Discussion

3.1. Experimental Setup and Data Description

The dataset came from the PRONOSTIA experimental platform in the IEEE PHM 2012 Data Challenge [35]. Figure 5 shows the overview of the experimental platform. The main objective of PRONOSTIA is to provide real data related to accelerated degradation of bearings performed under constant and/or variable operating conditions, which are online controlled. The operating conditions are characterized by two sensors: a rotating speed sensor and a force sensor. In the PRONOSTIA platform, the bearing’s health monitoring is ensured by gathering online two types of signals: temperature and vibration (horizontal and vertical accelerometers). The experiment was conducted under three conditions. In condition 1, the radial load force was 4000 N and the shaft speed was 1800 r/min; in condition 2, the radial load force was 4200 N and the shaft speed was 1650 r/min; in condition 3, the radial load force was increased to 5000 N and the shaft speed was 1500 r/min. The vibration sensors collected 0.1 s vibration signals every 10 s with a sampling frequency of 25.6 kHz. In this paper, each segment with 2560 points (0.1 s) was regarded as a sample.

To verify the effectiveness of the proposed method and its transfer ability between different conditions, two cases were analyzed. The vibration signals of four tests in condition 1 were selected as the source domain datasets, as shown in Figure 6. In case 1, one test in condition 2 was used as the target domain dataset. The vibration waveforms and the corresponding RMS are shown in Figure 7. In case 2, one test in condition 3 was used as the target domain dataset, as shown in Figure 8.

Table 1. Description of datasets.

	Source Domain	Target Domain Dataset of Case 1	Target Domain Dataset of Case 2
Conditions	Condition 1	Condition 2	Condition 3
Datasets	1_1, 1_2, 1_3, 1_4	2_1	3_1

describes the datasets in the source domain and target domain in detail. The sample number of each dataset is shown in

Table 2. Sample number of each dataset.

Dataset	1_1	1_2	1_3	1_4	2_1	3_1
Samples	871	1138	2448	2230	906	1637

.

Firstly, CNN${}_c$ was trained on the training samples to extract the features from the vibration signal as the input of the RUL prediction model. Then, the IF points of different training datasets in source domain were identified by CNN${}_c$. The degradation times of the four source domain datasets were identified to be 8250 s, 10,820 s, 24,110 s and 22,070 s, respectively. The source samples were labeled by the IF points. In this step, the sample $x_i^s$ corresponds to two labels: the classification label (0 or 1) and the regression label (the relative RUL from 0 to 1). That is, the sample $x^s(i)$ becomes $(x^s(i),y_1^s(i),y_2^s(i))$. The target domain datasets were unlabeled because the test dataset cannot be labeled in practical applications.

3.2. Case 1

In this case, one test dataset in condition 2 was used as the target dataset. Since the operational condition in the target domain was different from the source domain, the proposed TR-based prediction method was introduced for the RUL prediction. The proposed method was implemented in Python 3.5 based on a NVDIA GeForce GTX 1060 GPU. The parameters of the Adam optimization method were selected as: ${\beta }_1=0.9,{\beta }_2=0.999,ϵ=1\times {10}^{-8}$. The learning rate was selected as 0.001, the weight decay was set to 0 and the batch size was set to 32. There were 50 epochs in this experiment. The prediction results are shown in Figure 9. In order to further demonstrate the effectiveness of the proposed method, three state-of-the-art methods, including a traditional CNN, an SVR and a health index (HI)-based prediction method, were introduced to the same data as baseline methods for comparison. In order to verify the improvement of the prediction accuracy via multiloss architecture and transfer learning, the architecture of the traditional CNN was the same as that of the proposed method except that there was no F7-2 output and MMD domain loss in the last layer. The inputs of the SVR were set to be eleven time and frequency domain features, including RMS, kurtosis, mean and so on. Furthermore, the detail of the HI-based prediction method is illustrated in [41]. The results of these three methods are also drawn in Figure 9 for easier comparison.

It can be observed from Figure 9 that during the normal process from 0 s, the SVR does not perform as good as the other methods. The predicted relative RUL of the normal stage should be 1. The results of the SVR (the green line) show shocks around 0.8. The HI-based method also shows some shocks, but better than the SVR. The traditional CNN and the proposed method can provide a more accurate prediction result in this stage. Furthermore, during the degradation process, the HI-based method shows a higher variance than other methods. The prediction results of the two-stage TR-CNN are closer to those of the real RUL. Therefore, it can be concluded that the main limitation of the SVR is that it cannot detect if the bearing begins to degrade or not based on the historical data. Furthermore, the main drawback of the HI-based method is the low prediction accuracy during the degradation process.

For a better illustration and numerical comparison, the mean square error (MSE) of the two-stage TR-CNN and the three baseline methods was calculated as an estimation index, as shown in

Table 3. MSE of the proposed method, CNN, SVR and the HI-based method in case 1.

	Proposed Method	CNN	SVR	HI-Based Method
Full life cycle	0.0014	0.0023	0.381	0.0080
Degradation stage	0.0350	0.0617	0.0810	0.1007

. Matching with the prediction results in Figure 9, the MSE of the SVR is highest in the full life cycle, which is mainly caused by the prediction error during the normal stage. Furthermore, the MSE of the HI-based method is highest during the degradation stage. It can also be seen that the MSE of the two-stage TR-CNN is smallest both during full life cycle and degradation stage, which demonstrates that the proposed method not only can detect if the current bearing is faulty or not based on the collected signal, but that it also provides a more accurate predicted relative RUL.

3.3. Case 2

In this case, one test dataset of condition 3 was used as the unlabeled target dataset. Then, the proposed method, as well as three comparison methods were employed to predict the relative RUL of the target dataset. The prediction results are shown in Figure 10.

It can be found from Figure 10a that the SVR (green line) is still not good at estimating the relative RUL during the normal stage, which is similar to case 1. From the local enlarged area in Figure 10b, it can be seen that both the SVR and HI-based prediction method (blue line) show higher variance because the prediction results of these two methods are farther from the real relative RUL (black line). This is because the degradation pattern of bearings can be very different due to many factors, such as the changeable operational conditions, processing techniques and so on, which can lead to the instability of features, thus influencing the prediction performance of feature-based methods. The traditional CNN (yellow line) performs better than SVR and HI-based method because the CNN can automatically learn high feature hierarchies via its deep architecture without the help of a feature extractor, thus avoiding the influence of unstable features. It can also be found that the prediction results of the two-stage TR-CNN (red line) are closest to the real RUL both during the full life cycle and degradation stage, which verifies the effectiveness of the proposed method.

In this case, MSE was also used as the estimation index, which is shown in

Table 4. MSE of the proposed method, CNN, SVR and the HI-based method in case 2.

	Proposed Method	CNN	SVR	HI-Based Method
Full life cycle	0.0005	0.0015	0.0150	0.0031
Degradation stage	0.0105	0.0258	0.0413	0.0503

. It can be seen that the proposed TR-based RUL prediction method shows the lowest MSE both in full life cycle and degradation stage. This is because the proposed method was constructed based on a multiloss CNN backbone, which made it not only possess the superiority of CNN, but also able to learn fault features from fault classification, which can be used for RUL prediction. More than that, transfer learning can bridge the discrepancy between the dataset from different domains, thus learn transferable features and provide a more accurate estimated RUL results.

4. Conclusions

The main challenge for bearing RUL prediction is the difference in the degradation process between two bearings. Inspired by transfer learning, this paper proposed a TR-based RUL prediction method for bearing RUL prediction. A multiloss CNN was constructed as the backbone to make full use of historical data by extracting fault features from fault classification and then using them for RUL prediction. The application of a transfer learning algorithm also enabled the proposed method to learn transferable features between different domains. The effectiveness and transfer ability of the two-stage TR-CNN were verified by two experiments under different conditions. The experimental results illustrated that the two-stage TR-CNN was able to predict the RUL of the bearing tested under different conditions. The comparisons with state-of-the-art RUL prediction methods also illustrated the superiority of the proposed method. However, since the proposed method was constructed based on deep learning, more training datasets can improve the prediction performance. Because there are several parameters still needed to be determined manually and empirically, a more adaptive approach for the parameter selection shall be developed in the future.