PFVAE: A Planar Flow-Based Variational Auto-Encoder Prediction Model for Time Series Data

Abstract

Prediction based on time series has a wide range of applications. Due to the complex nonlinear and random distribution of time series data, the performance of learning prediction models can be reduced by the modeling bias or overfitting. This paper proposes a novel planar flow-based variational auto-encoder prediction model (PFVAE), which uses the long- and short-term memory network (LSTM) as the auto-encoder and designs the variational auto-encoder (VAE) as a time series data predictor to overcome the noise effects. In addition, the internal structure of VAE is transformed using planar flow, which enables it to learn and fit the nonlinearity of time series data and improve the dynamic adaptability of the network. The prediction experiments verify that the proposed model is superior to other models regarding prediction accuracy and proves it is effective for predicting time series data.

1. Introduction

Time series data refer to a sequence of sampling arranged in chronological order [1]. With the development of sensors, information communication, and computer storage technology, the time series data collected have become more abundant. Time series data prediction has become one of the key research directions in artificial intelligence, widely found in traffic flow [2], air pollution [3,4,5], anomaly detection [6,7,8], etc. The parameters of time series models can be used for identification methods [9,10,11,12,13], such as hierarchical algorithms [14,15,16,17,18].

At present, the prediction methods of time series data can be roughly divided into three categories: statistical methods, machine-learning methods, and deep learning algorithms [19,20]. Statistical methods include generalized linear prediction model [21], seasonal gray prediction model [22], Markov prediction method [23], Gaussian process model [24], etc. These models have clear mathematical forms and are easy to understand, while they require the data’s prior knowledge. Moreover, it is difficult to describe the complex nonlinearity.

In contrast, machine-learning methods with parameter self-learning and nonlinear adaptation are more suitable for time series data. The most widely used machine-learning methods include artificial neural network (ANN) [25], support vector regression (SVR) [26], integrated moving average autoregressive model (ARIMA) [27], etc. These models are easy to implement, but they cannot fit complex nonlinear relationships due to insufficient parameters [28,29].

The deep learning method has been popular in artificial intelligence in recent years. It can learn effective feature representations from extensive input data and has robust learning and predictive capabilities [30,31,32]. Recurrent neural network (RNN) [33] in deep learning and its variant forms of long short-term memory network (LSTM) [34] and gated recurrent unit (GRU) [35] has been widely used in the field of time series data prediction. In addition, many advanced network structures have been studied, such as bidirectional long short-term memory network (BiLSTM) [36], bidirectional gated recurrent unit (BiGRU) [37], convolutional neural network-long short-term memory network (CNN-LSTM) [38], convolutional long short-term memory network (ConvLSTM) [39], and Bayesian long short-term memory network (BayesLSTM) [40,41]. BiLSTM and BiGRU can improve the model’s learning power by considering the complete past and future contextual information; ConvLSTM improves the LSTM network’s ability by introducing convolution calculations. BayesLSTM improves LSTM based on variational reasoning theory, replacing fixed-value weights and deviations with optimizable and sampled distributions, which enhances the anti-noise ability and robustness of the model.

Although the abovementioned RNN-based methods have been widely used in time series data prediction, its modeling is still an opening problem due to the complex nonlinear relationship and random distribution of time series data. The remaining paper is as follows. Section 2 summarizes the related works and Section 3 introduces the model proposed, named PFVAE, in detail. Section 4 describes the experimental results to support the model’s effectiveness; finally, Section 5 summarizes and suggests future work.

2. Related Works

The machine-learning model is one of the popular prediction methods in time series data, such that Fi-John Chang et al. [42] used the self-organizing mapping method to extract spatiotemporal features of series data; Gholamreza Goudarzi et al. [43] used an artificial neural network, and Wenbo Liu et al. [44] used support vector regression with different kernel functions to predict the daily average concentration of PM2.5 in Beijing; Shihab Ahmad Shahriar et al. [45] evaluated autoregressive integrated moving average (ARIMA)-artificial neural network (ANN), ARIMA-support vector machine (SVM), and principal component regression (PCR) along with decision tree (DT) and CatBoost deep learning model for the prediction. In practice, machine-learning methods are unsuitable for big time series data prediction with complex nonlinearities because they will fail to give enough performance.

Compared with the classical machine-learning model, deep learning networks have powerful modeling capabilities, can extract potential information, and are widely used in time series data prediction. Xue-Bo Jin et al. [46] proposed a deep hybrid model with a serial two-layer decomposition structure to predict the future power load. Agga, Ali, et al. [47] proposed two models, CNN-LSTM and ConvLSTM, to predict the power generation of photovoltaic power plants in four different time ranges from 1 day to 7 days; Luo, Xianglong, et al. [48] proposed a structural deformation prediction model based on the time convolutional network (TCN), which uses one-dimensional expansion causal convolution to reduce model parameters and obtain long-term memory of time series. Although deep learning methods have strong learning capabilities, the prediction performance is not high enough since a large amount of data causes the learning efficiency of neural networks to be low.

Auto-encoder is a data compression algorithm that can compress and decompress data to reduce noise and complex information interference. Therefore, it has attracted more attention in time series prediction. Yi-Wei Lu et al. [49] proposed a model based on an auto-encoder gated recurrent unit (AE-GRU), where an auto-encoder (AE) extracts important features from raw data and a gated recurrent unit (GRU) selects information to predict the remaining service life. Xinghan Xu et al. [50] proposed a LSTM auto-encoder multi-task predictor to predict air quality in multiple locations. Ki-Su Kim et al. [51] proposed a deep learning model combining a denoising auto-encoder and convolutional LSTM to predict global ocean weather. Thanongsak Xayasouk et al. [52] used LSTM and deep auto-encoder (DAE) to predict the concentration of fine particles.

With the increase of research on auto-encoders, many special auto-encoders have appeared. Variational auto-encoder (VAE) can describe an observation in latent space with a probability distribution, which has been widely used in image and classification and has been gradually applied in time series prediction. Xiaosong Zhu et al. [53] proposed a long-term deformation prediction model for arch dams based on VAE and time attention-based long short-term memory (TALSTM) network. To improve the accuracy of equipment health prediction, Yanfang Yang et al. [54] proposed a neural network prediction model combining VAE and time convolutional network (TCN), which uses VAE to reduce the dimensionality, extracts the hidden information in the original data, reconstructs high-quality sample data, and then uses TCN to mine the internal connection in the long sequence of information.

In the above research, VAE only uses Gaussian approximation, which makes it challenging to simulate the complex time series data. On the other hand, normalizing flow is a tool for calculating complicated distribution. Aditya Arie Nugraha et al. [55] combined the normalizing flows with the VAE method to establish a latent variable model called GF-VAE for speech spectrogram modeling and enhancement; Philippe Esling et al. [56] proposed an audio synthesizer control formula by variational auto-encoder (VAE) and normalized stream. Henter, Gustav Eje et al. [57] proposed a model for probability, generation, and controllable motion data based on normalizing flows with MoGlow. The model is used to generate motion data sequences from normalizing flows so that it cannot only describe the distribution with high complexity but can use accurate maximum likelihood for effective training; Ho, YH et al. [58] built an end-to-end learning image compression system based on a novel flow model Augmented Normalizing Flows (ANF) and stacked with multiple Variational Auto-encoders (VAEs) is built.

The research above illustrates the high modeling ability of normalizing flows for nonlinear relationships. This paper uses VAE as the time series data predictor with normalizing flows to model the complex posterior distribution of time series data, which guarantees the robustness of the network to noise and improves its ability to model complex nonlinear relationships. The main contributions of this research are as follows:

(1)

VAE with LSTM is designed as a time series data predictor to learn the long- and short-term dependencies of time series data. Compared with the prediction networks [34,35,36,37,38,39,40,48,53], the proposed VAE model can effectively learn the relation and extract representative information of time series and improve the computational efficiency of the model. Moreover, it has sufficient robustness to noise and prevents overfitting.

(2)

Based on the Planar flow, this paper reforms the internal structure of VAE, improves the modeling ability of complex posterior approximation, enables it to learn and process time series data with more complex characteristics, and improves the nonlinearity and dynamic adaptability of the network.

3. Method

3.1. Variational Auto-Encoder (VAE)

When neural networks are used in time series prediction and image processing, they often need to receive a large amount of input information, usually containing useless or less valuable data, making it difficult for neural networks to learn. Therefore, we must compress a large amount of input data, extract representative information from the data, reduce the amount of input information, and put the reduced information into the neural network for learning so that the neural network will be easier to learn.

An auto-encoder is a data compression algorithm in which the data compression and decompression functions are data-related, lossy, and automatically learned from samples. In most cases where auto-encoders are mentioned, compression and decompression functions are implemented through neural networks. In this research, we used LSTM as the encoder and decoder to realize the compression and reconstruction process of the input data. The proposed prediction auto-encoder is shown in Figure 1.

In Figure 1, $X=\left[X_1,X_2,\cdots ,X_t\right]$ represents the original input data, $h=\left[h_1,h_2,\cdots ,h_t\right]$ and $g=\left[g_1,g_2,\cdots ,g_t\right]$ represents the hidden state passed in the encoder and decoder, respectively, H is the encoding vector output by the encoder, which means the essence of the original input data, and $Y=\left[Y_1,Y_2,\cdots ,Y_t\right]$ represents the reconstructed data through the decoder. The model’s error is the difference between the reference input and the reconstructed data, defined as the reconstruction error. The auto-encoder uses the mean square error (MSE) as the loss function to calculate the reconstruction error. The loss function is defined as follows:

$${\mathit{Loss}}_{recon}=\frac{1}{n}{\displaystyle \sum }_{i=1}^n{({\widehat{y}}_i-y_i)}^2$$

(1)

where $n$ is the total number of samples in the dataset, $y_i$ is the i-th reference value in the test set, and ${\widehat{y}}_i$ is the reconstructed value obtained by the model. Then we develop the auto-encoder in Figure 1 to VAE by variational inference [59], shown in Figure 2.

In Figure 2, $\mu =\left[{\mu }_1,{\mu }_2,\cdots u_t\right]$ and $\sigma =\left[{\sigma }_1,{\sigma }_2,\cdots {\sigma }_t\right]$ are the mean and variance obtained by the encoder after encoding the input data; $\epsilon =\left[{\epsilon }_1,{\epsilon }_2,\cdots {\epsilon }_t\right]$ is the parameter required for reparameterization, $\epsilon ~N\left(0,I\right)$; $z=\left[z_1,z_2,\cdots z_t\right]$ is the latent variable sampled.

It can be seen from Figure 2 that the loss function of VAE described as (2) includes: (1) the reconstruction error between the reference input data and reconstructed data; (2) the Kullback–Leibler divergence error (KL error) caused by the encoder sampling.

$$\begin{array}{l}{\mathit{Loss}}_{VAE}={\mathit{Loss}}_{recon}+Los{s}_{KL}\\ =\frac{1}{n}{\displaystyle \sum _{i=1}^n}{({\widehat{y}}_i-y_i)}^2+\frac{1}{2}{\displaystyle \sum _{i=1}^D}(1+\log ({({\sigma }_j^{\left(i\right)})}^2)+{(u_j^{\left(i\right)})}^2+{({\sigma }_j^{\left(i\right)})}^2)+{\mathbb{E}}_{q_{\theta }(z|x)}[\log P(x^{\left(i\right)}|z^{\left(i,l\right)})]\end{array}$$

(2)

where $z^{\left(i,l\right)}={\mu }^{\left(i\right)}+{\sigma }^{\left(i\right)}\odot {\epsilon }^{\left(l\right)}$ and ${\epsilon }^{\left(l\right)}~N\left(0,I\right)$, $\odot$ stands for the element-wise product.

The optimization mechanism of the loss function of VAE can be understood as an adversarial learning process for ${\mathit{Loss}}_{recon}$ and $Los{s}_{KL}$:

When the reconstructed data of VAE differs from the reference data, the mean and variance will be adjusted to fit the sampled data, thus ${\mathit{Loss}}_{recon}$ is changed to improve the quality of the reconstructed data.

3.2. VAE Based on Normalizing Flows

3.2.1. Planar Flow: One of the Normalizing Flows

To optimize $Los{s}_{KL}$ in Formula (2), we must find a distribution $P_{\theta }(x|z)=P(x|z)$ that is the same with posterior distribution as the input data. Still, the general Gaussian distribution cannot fit a sufficiently complex posterior distribution. Therefore, we embed normalizing flows into the VAE model.

The normalizing flows describe the transformation of probability density through a reversible mapping sequence. The initial density “flow” passes through the reversible mapping sequence, repeatedly applying the variable transformation rules [60].

By optimizing this series of distributions, a simple Gaussian distribution can be changed into a complex actual posterior distribution.

Specifically, given a reversible mapping: $f:{ℝ}^d\to {ℝ}^d$, use it to transform the latent variable $z~q(z)$ into a new variable $z^{\prime }~f(z)$, therefore the distribution of the new variable is:

$$q(z^{\prime })=q(z)\left|\det \frac{\partial f^{-1}}{\partial z^{\prime }}\right|=q(z){\left|\det \frac{\partial f}{\partial z}\right|}^{-1}$$

(3)

Formula (3) is the Jacobian matrix of invertible functions by the chain rule (inverse function theorem). It is possible to construct the complex density by synthesizing several simple mappings and applying Formula (3) in turn. For the random variable $z_0$ with the distribution of $q_0$, the transformation chain $f_K$ is sequentially used, and after $K$ times, the density $q_K(z)$ obtained is:

$$z_K=f_K\times \cdots \times f_2\times f_1(z_0)$$

(4)

and,

$$\ln q_K(z_K)=\ln q_0(z_0)-{\displaystyle \sum _{k=1}^K\ln \left|\det \frac{\partial f_k}{\partial z_{k-1}}\right|}$$

(5)

In this way, it is not necessary to explicitly calculate $q_K$, but only the initial distribution $q_0$ and the mapped Jacobian matrix. Normalizing flows need to find an invertible mapping function that the Jacobian matrix can be operated on efficiently. Here we used one of the normalizing flows called the Planar flow.

Planar flow is defined as a function of the following form:

$$f(z)=vh(w^{T}z+b)$$

(6)

where $v\in {ℝ}^D$ and $w\in {ℝ}^D$ are vectors (called scale and weight here), $b\in ℝ$ is a scalar (deviation) to be set by learning, $h$ is the activation function.

For Planar flow, the determinant of the Jacobian can be calculated in $\mathrm{O}(D)$ time by relying on the matrix determinant lemma:

$$\psi (z)=h^{\prime }(w^{T}z+b)w$$

(7)

$$\left|\det \frac{\delta f}{\delta z}\right|=\left|\det (I+v\psi {(z)}^T)\right|=\left|1+v^T\psi (z)\right|$$

(8)

Figure 3 shows the changes in the data distribution for the Planar flow. From Figure 3 we can see that by the action of the Planar flow, the single-peaked Gaussian distribution is transformed into the complex multi-peaked distribution.

3.2.2. VAE Based on Planar Flow

We used the Planar flow of length $K$ to parameterize the approximate posterior distribution of the encoder in the VAE. The proposed PFVAE prediction model structure is shown in Figure 4. $w=[w_1,w_2,\cdots w_t]$,$v=[v_1,v_2,\cdots v_t]$ and $b=[b_1,b_2,\cdots b_t]$ are the weights, scales, and deviations of Planar flow; $z_k=[z_{k1}, z_{k2}, \cdots z_{kt}]$ is a latent variable with a complex distribution obtained by the Planar flow posterior.

In the case of applying Planar flow $q_{\theta }(z|x)\simeq q_K(z_K)$, Equation (9) can be written as the expectation $q_0(z)$ of the initial distribution:

$$\begin{array}{l}ELBO(x)\simeq \mathbb{E}{q}_{\theta }(z|x)\left[\log q_{\theta }(z|x)-\log p(x,z)\right]\\ =\mathbb{E}{q}_0(z_0)\left[\ln q_K(z_K)-\log p(x,z_K)\right]\\ =\mathbb{E}{q}_0(z_0)\left[\ln q_0(z_0)\right]-\mathbb{E}{q}_0(z_0)\left[\log p(x,z_K)\right]-\mathbb{E}{q}_0(z_0)\left[{\displaystyle \sum _{i=1}^k\log \left|\det \frac{\partial f_i}{\partial z_{i-1}}\right|}\right]\end{array}$$

(9)

Normalizing flows and free energy circles can be used in any variational optimization scheme. For variational inference, we used a deep neural network to build an inference model from the observation $x$ to the initial density $q_0=N(\mu ,\sigma )$ and the parameter $\lambda$ of the flow to create a mapping. The complete optimization goal is written as:

$$ELBO(x)\simeq \mathbb{E}{q}_0\left[\log p_{\theta }(x|z_K)\right]+\mathbb{E}{q}_0\left[\ln q_0(z_0)-\log p(z_K)\right]-\mathbb{E}{q}_0\left[{\displaystyle \sum _{i=1}^k\log \left|\det \frac{\partial f_i}{\partial z_{i-1}}\right|}\right]$$

(10)

The Planar flow is used as the posterior to fit the complicated posterior distribution. It enables the VAE prediction model to learn and process time series data with complex characteristics and improves the network’s degree of nonlinearity and dynamic adaptability. The loss function of PFVAE is shown in Formula (11):

$$\begin{array}{l}\mathit{Loss}={\mathit{Loss}}_{recon}+Los{s}_{KL}\\ =\frac{1}{n}{\displaystyle \sum _{i=1}^n{({\widehat{y}}_i-y_i)}^2}+\mathbb{E}{q}_0\left[\log p_{\theta }(x|z_K)\right]+\mathbb{E}{q}_0\left[\ln q_0(z_0)-\log p(z_K)\right]-\mathbb{E}{q}_0\left[{\displaystyle \sum _{i=1}^k\log \left|\det \frac{\partial f_i}{\partial z_{i-1}}\right|}\right]\end{array}$$

(11)

With the Planar flow method, the output of the encoder is sampled from a more complex distribution than the Gaussian distribution. This makes the reconstructed output of the decoder more robust to noise, improves the reconstruction of data with a high degree of nonlinearity, and improves the dynamic adaptability of the prediction model.

The specific iterative calculation process of our proposed PFVAE prediction model is shown below:

(1)

The input data $X=[X_1,X_2,\cdots ,X_t]$ is encoded by the encoder to obtain $\mu$ and $\sigma$.

(2)

Using the reparameterization technique and sampling the latent variable $Z$, we get $z=\mu +\sigma \odot \epsilon$, $\epsilon ~N(0,I)$.

(3)

Planar flow is applied to obtain latent variables with a complex distribution $z_k$.

(4)

The latent variables are entered into the decoder to obtain reconstructed data $Y=[Y_1,Y_2,\cdots ,Y_t]$.

(5)

Calculate loss function $\mathit{Loss}=\mathit{Loss}{}_{recon}+Los{s}_{KL}$ and optimize the model.

4. Experimental Results and Discussions

4.1. Experiment Setup and Evaluation Indicators

The experiments were performed on a desktop computer equipped with an AMD R7-5800 processor, 4.0 GHz, and 16 GB of RAM by PyTorch to build a VAE-LSTM network model based on the Planar flow. This consists of a LSTM network for the encoder and decoder with hidden neural units set to 24 and used the Adam algorithm to perform supervised learning. Its learning rate was set to 0.01, and 100 epochs were trained. The prediction steps were 24.

We used 80% of the data as the model’s training set and the remaining 20% as the test set. The Z-score method shown in (12) was used for standardizing the input data.

$$x^{\prime }=\frac{x-x_{mean}}{x_{\sigma }}$$

(12)

where $x$ represents the input data, $x_{mean}$ represents the mean value of the observation data, $x_{\sigma }$ represents the variance of the observation data.

In the experiment, we compared the prediction performance with the other seven models, LSTM [34], GRU [35], BiLSTM [36], BiGRU [37], TCN [48], CNN-LSTM [38], ConvLSTM [39], BayesLSTM [40], and VAE [53], under the same parameters and the same dataset.

In order to evaluate the predictive performance of different models, six evaluation indicators were used, including root mean square error (RMSE), mean absolute error (MAE), symmetric mean fundamental percentage error (SMAPE), mean square error (MSE), Pearson correlation coefficient (R), and goodness-of-fit (R²). The calculation formulas of these six evaluation indicators are shown in Formulas (13)–(18):

$$\mathrm{RMSE}=\sqrt{\frac{1}{n}{\displaystyle \sum _{i=1}^n{({\widehat{y}}_i-y_i)}^2}}$$

(13)

$$\mathrm{MAE}=\frac{1}{n}{\displaystyle \sum _{i=1}^n\left|{({\widehat{y}}_i-y_i)}^2\right|}$$

(14)

$$\mathrm{SMAPE}=\frac{100\%}{n}{\displaystyle \sum _{i=1}^n\frac{\left|{({\widehat{y}}_i-y_i)}^2\right|}{{(|{\widehat{y}}_i|+|y_i|)}^2/2}}$$

(15)

$$\mathrm{MSE}=\frac{1}{n}{\displaystyle \sum _{i=1}^n{({\widehat{y}}_i-y_i)}^2}$$

(16)

$$\mathrm{R}=\frac{{\displaystyle \sum _{i=1}^n(y_i-{\overline{y}}_i)({\widehat{y}}_i-{\overline{\widehat{y}}}_i)}}{\sqrt{{\displaystyle \sum _{i=1}^n{(y_i-{\overline{y}}_i)}^2{\displaystyle \sum _{i=1}^n{({\widehat{y}}_i-{\overline{\widehat{y}}}_i)}^2}}}}$$

(17)

$${\mathrm{R}}^2=1-\frac{{\displaystyle \sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2}}{{\displaystyle \sum _{i=1}^n{({\widehat{y}}_i-{\overline{\widehat{y}}}_i)}^2}}$$

(18)

where $n$ is the total number of samples in the dataset, $y_i$ is the actual in the test set, ${\overline{y}}_i$ is the average value of the actual value, ${\widehat{y}}_i$ is the i-th prediction, and ${\overline{\widehat{y}}}_i$ is the average value of the prediction.

4.2. PM2.5 Prediction

This study used the PM2.5 concentration data in Beijing, China to carry out verification experiments. With the continuous development of the global economy, people are paying more and more attention to the ecological environment. Air pollution is one of the most urgent environmental pollution problems. The PM2.5 is a particulate matter with a diameter of less than 2.5 μm, an essential indicator for measuring and controlling the degree of air pollution. As air quality is affected by various factors such as meteorological factors and human factors, the changes of PM 2.5 time series data are highly nonlinear and are disturbed by noise, which is a challenging problem in forecasting.

The sampling time in the experiment was from 1 January 2017 to 31 December 2019, with a sampling frequency of 1 h. In Figure 5, we show part of the prediction results of each model.

To validate the effectiveness of the proposed method, we repeated each model independently with the same dataset 20 times and recorded its RMSE values to ensure the objectivity of the results. The statistical results are shown in Figure 6.

As shown in Figure 6, our proposed model (PFVAE) has the minor error range, the most uniform and concentrated distribution, and the smallest average error of the model, which maintains high accuracy and stability compared to other models.

It can be seen from

Table 1. Accuracy evaluation of prediction results of each model on the PM2.5 dataset.

Model	RMSE	MAE	SMAPE	MSE	R	R2
LSTM [34]	33.45	24.59	67.06	1119.05	0.44	−0.08
BiLSTM [36]	26.00	17.62	53.04	676.05	0.61	0.30
GRU [35]	28.93	21.37	62.09	836.67	0.56	−0.19
BiGRU [37]	26.92	17.80	55.70	724.74	0.59	0.32
CNN-LSTM [38]	28.78	20.52	59.94	828.56	0.52	0.20
ConvLSTM [39]	26.36	18.11	55.74	694.78	0.60	0.33
TCN [48]	26.77	18.46	61.41	716.66	0.61	0.31
BayesLSTM [40]	30.03	23.16	69.47	901.75	0.47	0.13
VAE [53]	25.84	17.72	53.08	667.91	0.61	0.36
The proposed method	24.45	17.07	51.29	598.13	0.65	0.42

that our proposed model has the lowest RMSE, MAE, SMAPE, and MSE and the highest R and R². RMSE, MAE, SMAPE, MSE, R and R² were 24.45, 17.07, 51.29, 598.13, 0.65 and 0.42, respectively. Compared with the other seven models, the RMSE of our PFVAE model improved by 26.9%, 6.0%, 15.5%, 9.2% 15.0%, 7.2%, 8.6%, 18.5%, and 5.3%, respectively; MAE improved by 30.5%, 3.1%, 20.1%, 4.1%, 16.8%, 5.7%, 7.5%, 26.2%, and 3.6%, respectively. As for SMAPE, the prediction performance of the PFVAE model proposed improved by 23.5%, 3.3%, 17.3%, 7.9%, 14.4%, 7.9%, 16.4%, 26.1%, and 3.3%, respectively; and MSE, 46.5%, 11.5%, 28.5%, 17.5%, 27.8%, 13.9%, 16.5%, 33.6%, and 10.4%, respectively. Moreover, the maximum R-value and R²-value of the proposed PFVAE model represent the best fit between the predicted and observed values. The above experimental data show that our proposed model is better than other models in terms of prediction accuracy and result fit, which proves the applicability of this model in the field of time series prediction.

4.3. Temperature Prediction

This study used the temperature data from Haidian District, Beijing, China, to conduct verification experiments. As we know, changes in atmospheric temperature are closely related to human production and life. The atmospheric temperature significantly impacts human travel, social development, and the ecological environment. Therefore, accurate prediction of atmospheric temperature has essential application prospects in people’s daily life.

Figure 7 shows part of the prediction results. The sampling time was from 1 January 2017 to 31 December 2019, with a sampling frequency of 1 h. The statistical results are shown in Figure 8.

Figure 8 shows that the proposed model (PFVAE) obtained the highest accuracy and stability compared to other models. It can be seen from

Table 2. Accuracy evaluation of prediction results of each model on temperature dataset.

Model	RMSE	MAE	SMAPE	MSE	R	R2
LSTM [34]	3.34	2.48	33.31	11.19	0.95	0.91
BiLSTM [36]	3.05	2.37	33.22	9.28	0.97	0.93
GRU [35]	4.22	3.32	37.00	17.83	0.93	0.86
BiGRU [37]	3.23	2.47	33.46	10.42	0.97	0.92
CNN-LSTM [38]	3.35	2.53	35.81	11.25	0.96	0.91
ConvLSTM [39]	3.24	2.53	33.70	10.48	0.96	0.92
TCN [48]	3.05	2.20	32.05	9.29	0.96	0.93
BayesLSTM [40]	3.03	2.30	32.83	9.18	0.96	0.93
VAE [53]	3.42	2.61	36.74	11.72	0.96	0.91
The proposed method	2.93	2.20	31.61	8.58	0.96	0.93

that compared with the other seven models, RMSE improved by 12.2%, 3.9%, 30.5%, 9.3%, 12.5%, 9.5%, 3.9%, 3.3%, and 14.3%, respectively; MAE, though the same as TCN, improved compared to others by 11.2%, 7.2%, 33.7%, 10.9%, 13.0%, 13.0%, 4.3%, and 15.7%, respectively. As with SMAPE, the prediction performance of the PFVAE model improved by 5.1%, 4.8%, 14.5%, 5.5%, 11.7%, 6.2%, 1.3%, 3.7%, and 13.9%, respectively; and MSE, 23.3%, 7.5%, 51.8%, 17.7%, 23.7%, 14.8%, 7.6%, 6.5%, and 26.7%, respectively. Moreover, the maximum R-value and R²-value of the proposed PFVAE model represent the best fit between the predicted and observed values.

4.4. Humidity Prediction

In this study, we used the humidity data of Haidian District, Beijing, China to carry out verification experiments. The sampling time was from 1 January 2017 to 31 December 2019, with a sampling frequency of 1 h. In Figure 9, we show part of the prediction results of each model.

As shown in Figure 10, our proposed model (PFVAE) has a minor error fluctuation range, the most uniform and concentrated distribution, and the smallest average error of the models.

It can be seen from

Table 3. Accuracy evaluation of prediction results of each model on the humidity dataset.

Model	RMSE	MAE	SMAPE	MSE	R	R2
LSTM [34]	20.47	15.74	29.78	419.13	0.62	0.31
BiLSTM [36]	18.12	13.08	26.15	328.19	0.72	0.46
GRU [35]	20.63	15.53	31.70	425.68	0.66	0.30
BiGRU [37]	17.59	12.84	26.18	309.44	0.75	0.49
CNN-LSTM [38]	20.83	16.03	32.47	433.83	0.65	0.29
ConvLSTM [39]	16.70	12.60	23.60	279.02	0.74	0.54
TCN [48]	17.29	12.94	25.03	299.06	0.74	0.51
BayesLSTM [40]	16.94	12.80	24.37	287.02	0.73	0.53
VAE [53]	17.96	13.83	26.06	322.71	0.71	0.47
The proposed method	16.58	12.52	23.61	274.85	0.75	0.55

that compared with the other seven models, our PFVAE model improved RMSE by 19.0%, 8.5%, 19.6%, 5.7%, 20.4%, 0.7%, 4.1%, 2.1%, and 7.6%, respectively; MAE improved by 20.4%, 4.3%, 19.3%, 2.5%, 21.8%, 0.6%, 3.2%, 2.1%, and 9.4%, respectively. As with SMAPE, though it is almost the same as ConvLSTM, it increased by 20.7%, 9.7%, 25.5%, 9.8%, 27.2%, 5.6%, 3.1%, and 9.4%; and MSE, 34.4%, 16.3%, 35.4%, 16.3%,36.6%, 1.4%, 8.0%, 4.2%, and 14.8%, respectively. Moreover, the maximum R-value and R²-value of the proposed PFVAE model represent the best fit between the predicted and observed values. The results show that our proposed model is better than other models in terms of prediction accuracy and result fit.

5. Summary and Future Work

It is challenging to learn and fit the prediction model due to the high nonlinearity and complex random distribution of time series data. This paper proposed a novel VAE prediction model based on Planar flow, which uses LSTM as the encoder and decoder and designs VAE as a time series data predictor, using Planar flow to transform the internal structure of VAE to learn the time series data with complex characteristics. Through verification experiments on a variety of time series datasets and the multiple evaluation indicators such as RMSE, MAE, SMAPE, MSE, R, and R², our proposed model is superior to other models in terms of prediction accuracy. The proposed prediction approaches of time series models in the paper can combine other parameter estimation algorithms [61,62,63,64,65,66,67,68] with studying the parameter identification problems of linear and nonlinear systems with different disturbances [69,70,71,72,73,74,75,76,77]. The proposed prediction approach can build the soft sensor models and prediction models based on the time series data and can be applied to other fields [78,79,80,81,82,83,84,85,86], such as signal processing and engineering application systems [87,88,89,90,91].

In future work, we will further verify the effectiveness of our proposed time series forecasting model. Meanwhile, we will explore other normalizing flows to transform our proposed prediction model.