Functional Subspace Variational Autoencoder for Domain-Adaptive Fault Diagnosis

Abstract

This paper presents the functional subspace variational autoencoder, a technique addressing challenges in sensor data analysis in transportation systems, notably the misalignment of time series data and a lack of labeled data. Our technique converts vectorial data into functional data, which captures continuous temporal dynamics instead of discrete data that consist of separate observations. This conversion reduces data dimensions for machine learning tasks in fault diagnosis and facilitates the efficient removal of misalignment. The variational autoencoder identifies trends and anomalies in the data and employs a domain adaptation method to associate learned representations between labeled and unlabeled datasets. We validate the technique’s effectiveness using synthetic and real-world transportation data, providing valuable insights for transportation infrastructure reliability monitoring.

1. Introduction

The use of data-driven maintenance strategies in sophisticated intelligent transportation systems, which make use of IoT-enabled resources, has gained considerable attention both industry and academia [1]. This is achieved by integrating embedded sensors that collect operational data from the systems being monitored, making it possible to constantly identify or predict any damage to the system. The commonly used approach is to install many interconnected sensors at fixed locations to gather information about facility operations. These sensors are synchronized through a global reference clock, which allows for the simultaneous timestamping of digitized sensor data from different locations. Consequently, conventional machine learning methods can be employed to extract valuable insights through time series data analysis at each location and spatial cross-correlation analysis between different locations [2,3].

In the context of transportation system monitoring, this study focuses on a unique situation, where sensors are deployed on mobile assets, such as vehicles. The vehicle’s trajectory is segmented into discrete anchor points, with the path segment between two successive anchor points forming a basic unit for spatial indexing shown in Figure 1. In a path consisting of L segments containing time series data, one can extract data vectors from each segment for further analysis. However, due to uncontrollable motion precision, the time series sensor data from different vehicle journeys will exhibit diverse phase differences or timing misalignment [4]. This issue is further complicated by inconsistencies in the sampling clock rate during each journey. Without an inherent standard for aligning time series data, it becomes a considerable challenge to construct reliable training vectors of time series data, which are crucial for developing machine learning models for monitoring transportation systems.

The process of annotating IoT sensor data in transportation systems also poses a substantial challenge for fully exploiting the capabilities of machine learning models. As the transportation sector increasingly adopts IoT sensor-based monitoring systems, an immense volume of data is generated [5]. While these data have the potential to significantly enhance the training of machine learning models, the scarcity of labeled examples constitutes a major impediment in developing data-driven solutions for these systems [6]. The difficulties stem from various factors, including privacy concerns, disputes over data ownership, the expenses associated with annotation, and the inherent dynamics of transportation systems.

The current work proposes a novel solution to tackle the challenge of learning from limited labeled data in various domains and resolves data quality issues that arise in real-world data. The challenge of creating an efficient domain adaptation strategy in the current research lies in the absence of complete correlations among latent features across various domains. As shown in Figure 2, our approach address this issue by using split latent subspace encoders in the architecture of a variational autoencoder (VAE) [7,8] and integrating a domain adaptation strategy [9,10] between the source and target domains. Specifically, we associate one VAE with the source domain and another with the target domain. The domain adaptation strategy employs the latent space alignment technique to extend the classification model trained on the source domain, using labeled synthetic data, to the target domain of unlabeled real-world data. To handle the issue of time series misalignment in real-world data that can potentially impact data analysis and machine learning processes, we use the functional data analysis (FDA) technique [11,12]. This technique considers sensor data time series as a sequence of smooth curves or functional data in Hilbert space, assuming that the sensor data points are causally related and behave as samples of smooth functions. This data model of presenting vectorial data as smooth functions allows a formulation of dynamic optimization method [13,14] to address the irregularity in the sensor data sampling interval and phase variations with improved computational efficiency.

The following are the contributions of this paper:

• We propose using an efficient functional data-based dynamic programming method as a machine learning pipeline pre-processing step to remove timing misalignment in moving sensor data.
• We modify learning rules in conventional deep learning models to handle functional data and propose a novel split encoder variational autoencoder that combines functional data classifications and clustering for prognostic management in transport systems.
• We evaluate the proposed technique using synthetic and real-world sensor data by a domain adaptation method and validate its efficacy using different performance metrics.

The goal of developing the functional subspace variational autoencoder (FS-VAE) is to identify meaningful latent space characteristics within time series data from the moving sensor by utilizing the two complementary latent subspaces as shown in Figure 2. In addition to the classification of the temporal profile as a means to detect anomalies, we also hypothesize that the long-term stability of a signal-generating mechanism captured by the sensor data can be assessed by analyzing the evolution in the size of clusters in the latent space. This is demonstrated in this paper. The output from the two subspaces can be useful in performing downstream analysis tasks, such as comprehending the progression of equipment malfunctions or system breakdowns [15].

2. Related Work

Time series sensor data gathered using a fixed sensor array can be considered analogous to imagery data, as they are both the result of time-synchronized sampling by a global clock. Convolutional variational autoencoder-based deep learning models have proven to be effective for anomaly detection and forecasting across a variety of industrial applications [16,17]. However, the underlying data model assumes that the pixels in the imagery data are independent of each other. As a result, comparing pixels becomes computationally expensive, and the process is more susceptible to noise corruption.

To migrate the time series data quality issues in the machine learning algorithms, sequence-to-sequence models, such as recurrent neural networks (RNNs) are proposed [18,19]. However, their training complexity grows linearly with the length of the sequence, which can result in prolonged computational times, especially for long sequences. Another method is called dynamic time warping (DTW) [20], which aligns sequences with varying speeds by ’warping’ the time axis. Its standard implementation, however, has a complexity of $O\left(N^2\right)$, making it computationally intensive for long sequences, despite the existence of variants designed to reduce this complexity. Both methods offer viable solutions for addressing misalignment in time series data from moving sensors, but they come with trade-offs in terms of computational complexity.

In transportation systems, substantial volumes of unlabeled data frequently arise, necessitating the employment of domain adaptation strategies to enhance model performance across diverse and novel data distributions. Domain adaptation techniques [9,10] for aligning latent spaces have been utilized in the field of machine learning. As mentioned previously, there is a lack of complete correlations among latent features across various domains. To address this issue, we extend the two-stage disentanglement method [21] and interweave domain adaptation and disentanglement without the mini-max game played in adversarial training between a generator and a discriminator network. The result is a less complex network. As illustrated in Figure 2, the input data are projected into private and shared latent subspaces using the split autoencoder as proposed in [21]. Rather than relying on adversarial training, this disentanglement is facilitated by employing a softmax classifier, thereby enhancing the efficiency of the process.

The proposed FS-VAE and domain adaptation techniques offer improvements over previous related works. Here are the main reasons why the functional data analysis (FDA) approach can provide a better data model:

• Dimensionality reduction: FDA can reduce the dimensionality of time series data by representing the data as functions with a limited number of parameters. This is particularly useful for machine learning tasks, as it reduces computational complexity and can help prevent overfitting.
• Alignment and warping: One common challenge when clustering time series data from moving sensors is that the data can have different time scales or be misaligned due to the varying speeds of the sensors. FDA offers a functional alignment technique that can help align the time series data before clustering, leading to more accurate results.
• Functional similarity: FDA allows for the comparison and clustering of time series based on their functional properties, such as shape or overall trend, rather than just pointwise similarities. This can lead to more meaningful clusters, as it focuses on the underlying structure of the time series rather than just their pointwise values.

3. FS-VAE Deep Learning Network

FS-VAE differs from conventional VAE in that the encoder input layer and the decoder output layer deal with functional data rather than vectorial data. As illustrated in Figure 1, each neural node in the input layer performs convolution operations on the input functional datasets and outputs a numerical result to deeper neural layers. The decoder works on the combined latent subspaces generated by the split encoders, which produce two sets of independent latent variables: private (unsupervised) and shared (supervised). This is also shown in Figure 2. These variables facilitate the discovery of low-dimensional latent representations of functional data. The unsupervised latent variables capture clustering correlation features in the functional datasets, while the supervised latent variables identify temporal features of functional data. To achieve these two aims, the encoder–decoder pair in the FS-VAE must reconstruct the original sequences of functional data from the union of supervised and unsupervised latent variables by optimizing an appropriately designed loss function, and the factorization of latent space is enforced by a softmax classifier loss function [22]. In the subsequent section, we will present the mathematical formulation of the FS-VAE.

3.1. Time Warping to Compensation Phase Variations

Given a set of functions $\left\{f_i\left(t\right)\right\}$ representing the functional dataset, warping functions $\left\{{\gamma }_i\left(t\right)\right\}$ are introduced such that

$${\tilde{f}}_i\left(t\right)=f_i\left({\gamma }_i\left(t\right)\right)i=1,2,\dots ,n,$$

(1)

to correct phase variations. These corrections improve the validity of statistical analyses and predictions derived from the functional data by providing a more accurate reflection of the data structure. Computing the warping function in the square-root velocity function (SRVF) [13,14] space simplifies the optimization problem involved in finding the optimal alignment. In the SRVF space, defined as

$$q_i\left(t\right)=\frac{f_i^{\prime }\left(t\right)}{\sqrt{\left|f_i^{\prime }\left(t\right)\right|}}i=1,2,\dots ,n,$$

(2)

the optimization problem becomes a standard shortest-path problem, which can be efficiently solved using dynamic programming. The distance to be minimized is the $L^2$ norm in Hilbert space between the template function $q_0\left(t\right)$ and warped functions:

$$d\left(q_0,q_i\circ {\gamma }_i\right)={\int }_{\mathcal{T}}{∥q_0\left(t\right)-q_i\left({\gamma }_i\left(t\right)\right)∥}^{2}dti=1,2,\dots ,n.$$

(3)

The template function is then updated from computing the average of these aligned SRVFs:

$$q_0^{\mathrm{new}}\left(t\right)=\frac{1}{n}\sum _{i=1}^n{q}_i\left({\gamma }_i\left(t\right)\right).$$

(4)

The template function can be initialized as the average of the functional dataset and its SVRF. After the convergence of the iterative loop formed by Equations (3) and (4), the resulting optimal warping functions $\left\{{\gamma }_i^{*}\left(t\right)\right\}$ can generate an aligned set of functions, $\left\{x_i\left({\gamma }_i^{*}\left(t\right)\right)\right\}$, which mitigate the phase variations and improve the accuracy of statistical analyses and predictions. An illustrative example of the convergence process inherent in the SRVF method can be found in Appendix A.

3.2. Explicit Functional Data Formulation

According to the notion of a multi-layer deep learning network, information flows from one layer to subsequent layer. For a particular layer, the mathematical rule relating input vector $\widehat{x}\in {\mathbb{R}}^n$ to output vector $\widehat{y}\in {\mathbb{R}}^m$ is

$$\widehat{y}=\sigma \left(W\widehat{x}+\widehat{b}\right),$$

(5)

where m is the number of neural processing units on this layer, $W\in {\mathbb{R}}^{m\times n}$ is the weight matrix, $\widehat{b}\in {\mathbb{R}}^m$ is the network bias in this layer and $\sigma$ is the non-linear activation function. The network parameters are all time independent. In the case of functional time series data, $n\to \infty$ in the input layer, and $m\to \infty$ in output, and we need to replace the time-discretization coordinates with Hilbert space basis coordinates in the learning rule for these two layers.

For the functional data input layer, the learning rule for functional datasets ${\left\{f_k\left(t\right)\right\}}_{k=1}^L$ can be defined as

$$v_i=\sigma \left(\sum _{k=1}^K{\int }_{\mathcal{T}}{\mathsf{\Psi }}_{i,k}\left(s\right)f_k\left(s\right)ds+b_i\right).$$

(6)

The time-dependent weight function ${\mathsf{\Psi }}_{i,k}\left(t\right)$ will be learned by this network to extract features from the functional inputs. By using the series expansion in terms of the orthonormal basis functions $\left\{{\phi }_1,{\phi }_2,\dots ,{\phi }_M\right\}$ with a inner product ${(,)}_H$, the output of the input layer is

$$v_i=\sigma \left(\sum _{j=1}^M\sum _{k=1}^K{w}_{j,k}^i{\int }_{\mathcal{T}}{\phi }_j\left(s\right)f_k\left(s\right)ds+b_i\right).$$

(7)

The hyperparameters or the weight tensor on the 1st hidden layer to be solved are

$$W=w_{j,k}^i,i\in [1,K],j\in [1,M],k\in [1,K],$$

(8)

where M is the number of basis function to represent the functional weights, and K is the number of input functional data. The input is a matrix

$${\mathsf{\Psi }}^{j,k}={\int }_{\mathcal{T}}{\phi }_j\left(s\right)f_k\left(s\right)ds.$$

(9)

In compact form, the input vector $\widehat{v}$ of the 1st hidden layer can be written as

$$\widehat{v}=G_1(W\wedge \mathsf{\Psi }+b_1),$$

(10)

where the ∧ operator is the

$${(W\wedge \mathsf{\Psi })}_i=w_{j,k}^i{\mathsf{\Psi }}^{j,k},$$

(11)

and the repeated indices are implicitly summed over. From a programming standpoint, Equation (11) is implemented as the i-th node, which is connected to $M\times K$ input nodes. These input nodes receive input from ${\mathsf{\Psi }}^{j,k}$ as illustrated in Figure 1. The subsequent deep layers can be dense network or convolutional network as defined by a transformation

$$y_i=\sigma \left(\sum _{j=1}^r{W}_j{x}_{1+(i-1)s+j}+b_j\right),$$

(12)

with the stride s, kernel size r, weight W and bias vector b. In this context, x is the input feature from the 2nd layer in Figure 1.

For the functional data output layer,

$${\mathsf{\Psi }}^{j,k}=\sigma \left(W_{j,k}^i{x}_i+b_{i,k}\right).$$

(13)

Then, the output functional data can be obtained as

$$f_k\left(t\right)=\sum _{j=1}^M{\mathsf{\Psi }}^{j,k}{\phi }_j\left(t\right),k\in [1,L].$$

(14)

Deep learning models for functional data can be extended in various ways [23,24]. The current approach utilizes a functional input and output layer, where the input represents the functional data, and the output provides the predictions. The deep learning layers in this model are numerical, meaning that they process numeric inputs and perform mathematical operations on them. These layers learn from the data patterns and improve the prediction accuracy over each epoch in the model training by the stochastic gradient descent optimizer [25].

In our functional data analysis of vibration signals, we parameterize the weight functions using a truncated Fourier series. Given that the frequency spectrum of our data ranges from 0 Hz to 100 Hz, we carefully select the number of basis functions to effectively capture the underlying frequency content while maintaining computational efficiency. To balance these considerations, we opt for 32 cosine and 32 sine functions, resulting in 64 basis functions. This selection results in a frequency resolution of approximately 3.125 Hz, offering a meaningful compromise between detail and model complexity. Furthermore, as the projections onto sine and cosine basis functions can yield both positive and negative values, we should choose activation functions that allow for negative values. In the present work, we used the hyperbolic tangent (tanh) as the activation function.

3.3. Latent Subspaces Domain Adaptation

Suppose that we have a training set ${\mathcal{T}}^s={\left\{\left({\mathsf{\Psi }}_n^s,{\mathbf{Y}}_n^s\right)\right\}}_{n=1}^{N_s}$, which consists of labeled samples $\left({\mathsf{\Psi }}_n^s,{\mathbf{Y}}_n^s\right)\in$ a supervised source domain. As mentioned previously, real-world time series data from a moving sensor are usually unlabeled. We represent these target domain data as ${\mathcal{T}}^t={\left\{{\mathsf{\Psi }}_n^t\right\}}_{n=1}^{N_t}$.

FS-VAE projects the functional data into two latent subspaces as illustrated in Figure 2. The shared subspace is used to classify the temporal feature of the data. The private subspace is used to determine the clustering correlations of the data. We introduce a novel method of applying softmax classification training to facilitate the factorization of latent subspaces by the split encoders. Additionally, the reconstruction loss of the FS-VAE training can ensure private representations to capture low-dimensional projections of input data. With the labeled training dataset, the source domain branch of Figure 2 is self-sufficient in training the split encoder, softmax classifier and decoder to accomplish temporal feature classification and clustering analysis. The trained networks in source domain can be transferred to the target domain as an initial condition for the domain adaptation training.

4. Model Training

The joint FS-VAE architecture depicted in Figure 2 simultaneously performs dimensionality reduction clustering (unsupervised) and classification tasks (supervised). The total loss function of the architecture comprises two reconstruction error terms, a single coupling error term and a classification term:

$$\begin{array}{cc}\hfill {\mathcal{L}}_S=& \frac{{\lambda }_r}{N}\sum _{i=1}^N{∥{\mathbf{x}}_i^s-D_s\left(E_s^c\left({\mathbf{x}}_i^s\right),E_s^p\left({\mathbf{x}}_i^s\right)\right)∥}^2+\frac{{\lambda }_r}{N}\sum _{i=1}^N{∥{\mathbf{x}}_i^t-D_t\left(E_t^c\left({\mathbf{x}}_i^t\right),E_t^p\left({\mathbf{x}}_i^t\right)\right)∥}^2+\hfill \\ \hfill & {\lambda }_{\mathrm{st}}\sum _{\begin{array}{c}i,j\in N\end{array}}{∥E_s^c\left({\mathbf{x}}_i^s\right)-E_t^c\left({\mathbf{x}}_j^t\right)∥}^2-\frac{{\lambda }_c}{N}\sum _{i=1}^{N_s}{\mathbf{y}}_i^T\ln C\left(E_s^c\left({\mathbf{x}}_i^{s0}\right)\right).\hfill \end{array}$$

(15)

The hyperparameters ${\lambda }_r$, ${\lambda }_{st}$ and ${\lambda }_c$ explicitly control this trade-off in our formulation. We set all three parameters to 1.0 for all central analyses in this manuscript.

Our model is trained by updating several components alternately using the stochastic gradient descent (SGD) algorithm. These components include the shared encoders $E_s^c(·)$/$E_t^c(·)$, the private encoders $E_s^p(·)$/$E_t^p(·)$, the decoders $D_s(·)$/$D_t(·)$, and the softmax classifier $C(·)$. Our model is implemented using the PyTorch framework [26].

5. Results

5.1. Pre-Processing of Target Domain Data

The real-world data used in our study came from accelerometers mounted on trains. This target domain vibration dataset records variations in the signal generation mechanism as a result of the interaction between train wheels and railway tracks. Investigating the temporal features and the long-term trends in clustering correlations can provide valuable insights for maintenance decision making and improving system reliability. To ensure the accuracy of our analysis, the data are pre-processed to eliminate timing misalignment issues according to Section 3.1. The results of our analysis of a segment of real-world data are presented in Figure 3. As illustrated in this figure, it is evident that the misalignment, or phase noise, is largely removed upon inspection. This improved data quality allows for a more accurate assessment of temporal features and long-term trends. The impact of the pre-processing step on the unsupervised learning of clustering is further investigated and compared against other standard methods. The results are tabulated in the table in Section 5.7. It can be observed that there is a consistent improvement of about $30\%$ in the clustering performance metrics in the benchmark comparison when pre-processing is not applied. More details about the construction of the deep learning model can be found in Section 5.3.

5.2. Labeled Synthetic Dataset Preparation

We evaluate our approach to temporal feature classification using simulated random vibration time series data generated by the method described in [27] for the source domain. We create 3000 samples of functional datasets, each of which contains three segments of functional data, denoted as $f^k\left(t\right)$, generated according to

$$f_k\left(t\right)={\mathsf{\Phi }}_m\left(t\right)\times S_{\pi }\left(t\right),$$

(16)

$$k\in [1,\dots ,3000]m\in [1,\dots ,30],$$

$$\pi \in \left\{\mathrm{impulsive},\mathrm{non}-\mathrm{impulsive}\right\}$$

where $S_{\pi }\left(t\right)$ is the random vibration signal, and ${\mathsf{\Phi }}_m\left(t\right)$ is the envelope function. The envelope function emulates the spatial feature for each functional data segment. We prepare 30 different types of envelope functions of different modulation lengths, and mix them randomly with vibration signals. The labeling of the time series data in the training dataset into impulsive signals and non-impulsive signals is determined by the spectral content on the high-frequency side. Given that the frequency spectrum of our data ranges from 0 Hz to 100 Hz, the high-frequency side is above 30 Hz, which corresponds to a pulse duration of less than 0.1 s.

According to the experimental results shown in the figure in Section 5.4, the accuracy of our technique to classify impulsive signals and non-impulsive signals remains consistent across different envelope functions, with a fluctuation of within $15\%$. More details are discussed in Section 5.4.

5.3. Decoder Selection and Evaluation

Our proposed split encoder FS-VAE model comprises two distinct encoders: a feedforward network and a convolutional network. The structure of the encoders is shown in

Table 1. The architecture of the encoders in the split encoder FS-VAE is as follows. The input layer, depicted in Figure 1, is labeled as “input to next layer”. This layer is responsible for receiving the overlap integration between the functional input data and the basis functions. The network parameters that determine the functional weights are optimized alongside the other network parameters using backpropagation and Adam optimization.

Component	Type	Architecture
Encoder 1	Feedforward	Input: 40, Hidden Layers: 15,
		Output: 6
Encoder 2	Convolution	Input: 40, Conv Layers: [8, 16],
		Fully Connected: 6,
		Output: 5

. Both encoders transform the output of the functional input layer (refer to Equation (9)), which is 192-dimensional numerical data from the overlap of functional data into different basis functions, as mentioned in Section 3.2, into separate latent spaces of dimensions 6 and 5, respectively. These outputs are combined into a joint latent space of 11 dimensions. A latent variable is sampled from this joint distribution using the reparameterization trick [7], serving as the input to the decoder for data reconstruction.

The architecture of the encoders does not explicitly determine the structure of the decoder in the split encoder FS-VAE model. To identify an appropriate decoder architecture, we evaluate two candidates: a feedforward network and a convolutional network. The structures of the two decoder options are shown in

Table 2. The architecture of the candidate decoders in the split encoder FS-VAE differs from that of the split encoder itself. Two options are explored in this regard. The results obtained from these options are evaluated and verified using the reconstruction loss, similar to the conventional VAE approach.

Component	Type	Architecture
Decoder (Option A)	Feedforward	Input: 11, Hidden
		Layers: [15, 25],
		Output: 40
Decoder (Option B)	Convolution	Input: 11, Fully Connected: 16,
		Deconv Layer: [8, 4],
		Output: 40

. Our assessment focused on training error decay and reconstruction error, which is the Euclidean norm between the matrices in Equations (9) and (13). According to the experimental results in Figure 4, the feedforward decoder demonstrates a slower training error decay and a higher reconstruction error compared to the convolutional decoder. The convolutional decoder shows a rapid training error decay and a $100\%$ reduction in reconstruction error, indicating more efficient learning and accurate reconstruction. Based on these results, the convolutional network is identified as the more suitable decoder for our split encoder FS-VAE model.

5.4. Classification Accuracy with Synthetic Dataset

The effectiveness of the supervised learning feature of FS-VAE by the softmax classifier is investigated in this section. This multi-label classification problem is trained by minimizing the cross-entropy loss function in Equation (15). For this purpose, we consider a case with three segments (L = 3) as depicted in Figure 1. Each segment exhibits two behavior types: “impulsive” and “non-impulsive.” The output of the softmax consists of six nodes, representing the probabilities of the two temporal features within each of the three segments. The classifier receives input from the 6-dimensional latent space generated by the feedforward encoder.

The architecture of the classifier can be found in

Table 3. The architecture of the softmax classifier for segment = 3 is illustrated in Figure 1. This classifier is responsible for determining the likelihood of detecting “impulsive” or “non-impulsive” signals within each segment. The input to the classifier is derived from the latent subspace of the feedforward decoder in the split encoder as depicted in Figure 2.

Layer	Size
Input	6
Hidden Layer 1	6
Hidden Layer 2	6
Output (Softmax)	6

. The effectiveness of our model is evaluated using the F1 score. Our findings, as shown in Figure 5, indicate that the F1 scores remain steady across different combinations of envelope functions and impulsive function profiles. The fluctuations in the F1-score are within $15\%$. The cases with relatively low F1-scores come from the impulsive function profiles that, under Fourier analysis, have similar spectral content to non-impulsive functions. This suggests the stability of coupling the softmax classifier with the feedforward encoder to encode salient features and underlying structures of data into a lower-dimensional shared latent subspace.

In addition, we also performed an in-depth comparison analysis against the standard time series classification methods with vectorial data and functional data input. The results are shown in

Table 4. A comparison of the performance of of five classifiers on labeled synthetic data. The evaluation is conducted using both the vectorial/functional data model and the functional variational autoencoder (FS-VAE) approach. FS-VAE demonstrates the highest overall performance. Note that, except for SVM, all classifiers exhibit improved performance when applied to the functional data approach.

Method	Metric
	Accuracy	F1 Score	AUC Score	Precision	Recall	Specificity
Vectorial Data
Decision Trees	0.72	0.69	0.75	0.70	0.68	0.76
k-NN	0.68	0.66	0.71	0.67	0.65	0.73
Naive Bayes	0.69	0.67	0.72	0.68	0.66	0.74
SVM	0.78	0.76	0.82	0.77	0.74	0.82
Random Forests	0.79	0.77	0.83	0.78	0.75	0.84
MLP	0.75	0.76	0.79	0.77	0.76	0.86
Functional Data
Decision Trees	0.77	0.75	0.80	0.76	0.74	0.81
k-NN	0.82	0.80	0.86	0.81	0.79	0.87
Naive Bayes	0.84	0.82	0.88	0.83	0.81	0.89
SVM	0.78	0.76	0.78	0.77	0.75	0.82
Random Forests	0.86	0.84	0.90	0.85	0.83	0.91
MLP	0.82	0.80	0.84	0.81	0.79	0.86
FS-VAE	0.93	0.91	0.96	0.93	0.89	0.98

, and we observe a variety of performance patterns.

For the decision trees algorithm, we find similar performance across functional and direct (discrete) approaches, with a slightly better outcome using functional data. This algorithm’s AUC score is $80\%$ for functional data and $75\%$ for vectorial data, suggesting that decision trees algorithm algorithms may have slight benefits from the dimensionality reduction offered by the functional approach, although it can also perform satisfactorily in higher-dimensional spaces.

Support vector machine (SVM) shows a small advantage in performance with the vectorial data approach. Given that SVMs are inherently designed to handle high-dimensional data and capture complex decision boundaries, they may not significantly benefit from the functional approach’s dimensionality reduction. In fact, the vectorial approach outperforms the functional approach by about $5\%$, with AUC scores of $82\%$ and $78\%$, respectively.

In contrast, the k-nearest neighbors (kNNs), naive Bayes (NB) and multi-layer perceptron (MLP) algorithms demonstrate enhanced performance with the functional approach. These algorithms typically perform better in lower-dimensional spaces, with AUC scores of $86\%$ for k-NN, $88\%$ for NB and $84\%$ for MLP with functional data, versus $71\%$, $72\%$, and $79\%$, respectively, with discrete data.

The versatile random forest (RF) algorithm shows a slight improvement with the functional approach compared to the direct one. Despite its capability to handle high-dimensional data and complex relationships, it achieves an AUC of $90\%$ with functional data, compared to $83\%$ with vectorial data. This indicates that robust and adaptable algorithms such as random forest can also benefit from the more concise data representation offered by functional methods.

Our current technique, FS-VAE, involves performing softmax classification on the split encoder variational autoencoder’s latent subspace and shows remarkable performance improvements. For decision tree, FS-VAE increases the AUC to $96\%$, a gain of $16\%$ over the functional approach and $21\%$ over the vectorial approach. Even for SVM, where the vectorial data approach performs exceptionally well, FS-VAE surpasses it with an AUC of $96\%$, indicating a performance gain of about $14\%$. For algorithms such as kNN and NB, which show significant performance improvement with the functional approach, FS-VAE drives AUCs up to around $96\%$, a performance gain of 10–8%, respectively. Finally, for the random forest algorithm, which shows a slight preference for the functional approach, FS-VAE pushes the AUC to $96\%$, a performance gain of $6\%$.

On average, FS-VAE improves performance by about $14.5\%$ over the best method in the vectorial data category (random forests), and by about $7.9\%$ over the best method in the functional data category (random forests). Thus, the FS-VAE approach, which combines functional data analysis and variational autoencoders, provides a compelling strategy that could outperform both the functional and direct approaches in time series classification tasks.

5.5. Accuracy Dependence on Segment Size

We evaluate the classifier’s performance with different numbers of segments in the model, ranging from 3 to 9, with the results presented in the Figure 6. The network layer sizes are also adjusted in each case, while the number of hidden layers remains the same. Our analysis shows that a model with 6 branches or less achieves over 50% accuracy. However, with an increase in segments to 7 or more, simply adjusting the network layer sizes is insufficient to preserve the accuracy, which declines to below $50\%$. While there are several potential enhancements, such as adding more network layers, integrating regularization techniques, such as dropout, and fine-tuning hyperparameters, such as learning rate, batch size, or training epochs, these extensive modifications are out of the scope of the current study. For the discussions in this paper, we set the value of L to 3 when it is not explicitly mentioned.

5.6. Latent Subspaces Alignment

In our study, we implement a domain adaptation strategy using a split encoder and one decoder variational autoencoder structure for both the source and target domain branches. The split encoder produces a shared subspace and a private subspace. The shared subspace is connected to a softmax classifier to enforce the factorization of the latent space. The shared subspaces in both branches are connected by the latent space domain adaptation method, facilitating the transfer of knowledge from the source to the target domain.

The source domain is initially trained with labeled synthetic data for 100 epochs. Upon convergence, the network weights, which are achieved with a final training loss of 0.05, are used as initial values for the network weights in the target domain branch. This transfer learning trick, which provides an educated guess of the initial network parameters of the target domain branch in Figure 2, can facilitate domain adaptation training with a modestly sized real-world dataset. Both branches are then trained together for another 50 epochs with the domain adaptation method connecting the shared subspaces, achieving a final training loss of 0.03. During the domain adaptation training process, we encounter a situation known as “latent space collapse” [28], where the diversity of latent features is suppressed to be very small. This issue is mitigated by incorporating batch normalization [29] into the training process, which helps to maintain the diversity of the latent space and improves the model’s performance on the target domain data.

The result of domain adaptation training with and without batch normalization is studied with a labeled target domain dataset comprising 30 samples. The shared subspace in the trained target domain branch is connected to the softmax classifier. The classification evaluation metrics are tabulated in

Table 5. The evaluation of domain adaptation performance metrics involves the assessment of batch normalization’s impact. Domain adaptation networks are trained using both labeled synthetic and unlabeled real-world data. Subsequently, the results are compared with and without batch normalization. The classification accuracy in the target domain branch is computed by connecting the softmax classifier to the shared latent subspace in this branch, utilizing 30 labeled real-world data. A clear improvement is demonstrated in all the metrics as a result of incorporating batch normalization.

Metric	Method
	Without Batch Normalization	With Batch Normalization
Accuracy	0.60	0.85
Precision	0.55	0.82
Recall	0.65	0.88
F1 Score	0.60	0.85
Mean Absolute Error	0.45	0.20
Root Mean Squared Error	0.60	0.35

.

In Table 5 we can see that the inclusion of batch normalization significantly improves all the metrics. The accuracy increases from $60\%$ to $85\%$, precision from $55\%$ to $82\%$, recall from $65\%$ to $88\%$, and F1 score from $60\%$ to $85\%$. The error metrics also decrease, with the mean absolute error reducing from 0.45 to 0.20 and root mean squared error from 0.60 to 0.35. These improvements demonstrate the effectiveness of the domain adaptation method, achieving an accuracy of about $85\%$.

5.7. Clustering in Private Latent Subspace and Prognostic Implications

This section delineates the operational principles of the FS-VAE private subspace model, specifically, its ability to extract clustering correlations from input functional data. As shown in Figure 1, the model projects the input data into separate subspaces. A portion of this projection is handled by the 1D convolutional neural network (1DCNN) decoder, which maps the data into the private subspace. This private subspace is then leveraged for clustering analysis, capturing and exploring the variations within the functional dataset. The successful factorization into subspaces is evidenced by the low reconstruction error, as generated by the total variational autoencoder (VAE). This can be visually confirmed in Figure 4, which shows a comparative representation of the original and reconstructed data, thus illustrating the effectiveness of our approach.

The neural network responsible for generating the clusters is trained concurrently with the procedures for classification and domain adaptation as previously described. This simultaneous training approach ensures that the network’s learning is harmonized across these different tasks, thus leading to more cohesive and effective performance.

Our study aims to evaluate the performance of various dimensionality reduction techniques on time series data. We primarily focus on the convolutional variational auto-encoder (CVAE) [30] and principal component analysis (PCA) [31], specifically for vectorial time series data, in addition to the FS-VAE private latent subspace model. These models are examined based on their capacity to extract meaningful clusters from the time series data. To quantify the quality of these clusters, we employ measures such as accuracy, mutual information, and purity metrics, following the standards defined in the well-established literature [32,33,34].

The initial stages involve the pre-processing and transformation of our time series data into a lower-dimensional representation. Here, we note a significant distinction between the techniques. CVAE and PCA, which are designed to handle discrete data, require larger network sizes—approximately 5394 and 5000 parameters, respectively. On the other hand, the FS-VAE model, designed for functional data representation, leads to a considerably smaller network size, around the order of 2200 parameters.

After transforming the data into a lower-dimensional representation with the respective techniques, we subject them to the K-means clustering algorithm as implemented in the scikit-learn library [35]. Our comparative analysis of the three dimensionality reduction techniques, considering their computational efficiency and the quality of the resultant clusters, is shown in

Table 6. Performance evaluated by K-means clustering on the representation extracted from functional data using different representation learning methods. The evaluated methods include convolutional variational autoencoder (VC), principal component analysis (PCA), and functional subspace variational autoencoder (FS-VAE). FS-VAE demonstrates improved performance across all metrics compared to the other methods. Additionally, the impact of not pre-processing the real-world data (as indicated by Real-world *) is examined. A clear deterioration in performance is observed when phase-variation removal is not applied.

Method	Metric
	Accuracy	Normalized Mutual Information	Purity	Relative Speedup
Synthetic
CVAE	0.79	0.60	0.75	1.50
PCA	0.75	0.58	0.70	1.60
FS-VAE	0.95	0.85	0.90	1.00
Real-world *
CVAE	0.68	0.55	0.68	1.50
PCA	0.65	0.50	0.62	1.60
FS-VAE	0.82	0.65	0.80	1.00
Real-world
CVAE	0.72	0.58	0.71	1.50
PCA	0.70	0.62	0.68	1.60
FS-VAE	0.89	0.77	0.85	1.00

. The accuracy in clustering performance shows improvement between $20\%$ to $30\%$. Notably, there is a computational speedup of about $50\%$ for FS-VAE compared to CVAE, and around $60\%$ when compared to PCA. Figure 7 also shows the t-SNE plot of the clustering result of FS-VAE against PCA. By inspection, the FS-VAE model’s clusters are more distinct and well separated, and the t-SNE plot provides additional evidence to support the improved clustering performance of the FS-VAE model.

We can calculate the mean intra-cluster distance (MICD) [35], which is a measure of the average distance between data points within a cluster in the private latent subspace, as an indicator of changes in the underlying signal generation process. For a given cluster C, its MICD can be computed as follows:

$$\mathrm{MICD}\left(C\right)=\frac{1}{\left|C\right|\left(\left|C\right|-1\right)}\sum _{i\ne j}d(x_i,x_j),$$

(17)

where $\left|C\right|$ is the number of points in cluster $d(x_i,x_j)$ is the Euclidean distance between distinct pair of points $x_i$ and $x_j$ in the latent space. By tracking changes in MICD, we can detect alterations in the underlying signal generation process, which may manifest as variations in data distribution within the latent space. An increase in MICD indicates that the average distance between points within a cluster is growing. This can be linked to the signal generating process becoming more random. This randomness is often associated with a degradation in reliability.

Figure 8 showcases MICD plots for a series of datasets, spanning three distinct time periods, each differing by a month. In addition, it includes a magnitude-shape (MS) plot [36] for functional data analysis comparison. Both the MICD and MS plots exhibit a positive trend over the observed time period. This upward trend potentially indicates a deterioration in the robustness of the underlying signal generating process, hinting at possible degradation [37]. This consistency between the plots effectively demonstrates the utility of the private subspace in finding trends in sensor data variations. It provides valuable insights into the stability of the underlying signal generation process. This approach complements temporal feature-based classification in the shared latent subspace for anomaly detection.

6. Conclusions

In this paper, we leverage the benefits provided by functional data representation of time series data to develop a subspace variational autoencoder and an associated domain adaptation method to facilitate the extraction of temporal features and clustering correlations from unlabeled moving sensor data, such as the signals generated by the impact between train wheels and the railway track.

Our method shows great efficiency in extracting temporal features and clustering correlations from this unlabeled moving sensor data. We achieve a notable $30\%$ improvement in cluster identification accuracy and a $15\%$ improvement in classification accuracy when contrasted with traditional techniques. Furthermore, our technique also maintains a high level of domain adaptation accuracy, with a consistent quality value above $85\%$. The combination of improved accuracy in both clustering and domain adaptation, along with the ability to track the stability of the signal generating process, underscores the potential of our method for reliability monitoring and prognostic analysis in transportation systems.

As for future work, we plan to examine the performance of the technique proposed in this paper over conventional methods on functional datasets that exhibit non-convex shapes in latent space.