Automated Model Selection of the Two-Layer Mixtures of Gaussian Process Functional Regressions for Curve Clustering and Prediction

Abstract

As a reasonable statistical learning model for curve clustering analysis, the two-layer mixtures of Gaussian process functional regressions (TMGPFR) model has been developed to fit the data of sample curves from a number of independent information sources or stochastic processes. Since the sample curves from a certain stochastic process naturally form a curve cluster, the model selection of TMGPFRs, i.e., the selection of the number of mixtures of Gaussian process functional regressions (MGPFRs) in the upper layer, corresponds to the discovery of the cluster number and structure of the curve data. In fact, this is rather challenging because the conventional model selection criteria, such as BIC and cross-validation, cannot lead to a stable result in practice even with a heavy burden of repetitive computation. In this paper, we improve the original TMGPFR model and propose a Bayesian Ying-Yang (BYY) annealing learning algorithm for the parameter learning of the improved model with automated model selection. The experimental results of both synthetic and realistic datasets demonstrate that our proposed algorithm can make correct model selection automatically during parameter learning of the model.

1. Introduction

As a powerful statistical learning model, the Gaussian process (GP), along with its variants, has been widely used in various scenarios of data analysis and information processing to capture the intrinsic features of time series in a variety of fields such as regression analysis, pattern recognition, image processing, and computer vision [1,2,3,4]. However, there exist two major drawbacks which strongly limit the capability of the conventional GP model. Firstly, the training iteration involves the covariance matrix inversion, which is significantly time-consuming for a large-scale dataset [5]. Secondly, for the convenience of parameter learning, the mean function of GP is usually assumed to be zero, so that the learned GP is actually a stationary process. Therefore, it is rather difficult for a GP to model the multi-mode time series, in which there are violent fluctuations [6].

In order to overcome these drawbacks, Tresp [5] adopted the mixture of Gaussian processes (MGP) along the time or input region for multi-mode time series. It employs a number of GPs which are located at different intervals of the time region corresponding to the actual modes. In this case, the heavy computation of a large matrix inversion can be greatly reduced to those of some small matrix inversions in the training iteration because the MGP model limits the parameter learning of each component GP to a small piece of time series. To enhance the learning ability and accommodate different task scenarios, these kinds of MGPs have been improved from the structure and mechanism, such as the mixture of robust GPs [7], domain adaptation MGP [8], sequential local-based MGP [9], etc. The corresponding learning algorithms have been developed with three methodologies: Bayesian inference [10,11], Markov chain Monte Carlo [12,13], and EM algorithm [14,15]. Until now, theoretical research and practical applications of these GP mixtures are still active [16,17,18,19].

Nevertheless, the above GP mixtures merely consider the diversity of time series in the input space and try to learn a stochastic process from a given piece of time series, i.e., only one sample curve. In practical applications, it is more important and significant to consider the diversity of sample curves in the output space so that the curves can be clustered and predicted [20]. To this end, we should get rid of the zero-mean assumption. In fact, Shi et al. [6] proposed a model of Gaussian process functional regression (GPFR) by using a linear combination of B-spline functions to learn and approximate the mean function of the output, and then the GPFR mixture along the output space (referred to as mix-GPFR) was established for curve clustering and prediction [21]. However, these GP mixtures limit the curve clusters to be generated by a number of Gaussian processes, not general stochastic processes which can be actual information sources in practice. In order to overcome this limitation, Wu and Ma [22] established a two-layer mixture model of GPFRs (TMGPFR). In the lower layer, there are a number of GPFRs driven by the input variables, whereas in the upper layer, there are actually a number of mixtures of GPFRs (MGPFR) generated by linearly mixing a certain number of GPFRs in the lower layer, which further mix together in the output space and forms the output variables.

In the TMGPFR model, each MGPFR in the upper layer can model a general stochastic process that acts as a general random information source to generate a kind of sample curve as a curve cluster. Thus, the TMGPFR model can fit those curves generated from a number of independent random information sources and further make curve clustering and prediction after its parameters have been learned from the data of those curves. As the number of GPFRs in the lower layer is large enough and fixed, the number of MGPFRs in the upper layer corresponds to the number of actual clusters in the whole curve dataset. Theoretically, it is a model selection problem to determine the number of MGPFRs in the TMGPFR model for a given curve dataset, which has not been deeply investigated in the previous literature. In fact, if this model selection is made correctly during parameter learning, the true number of curve clusters as well as these actual clusters themselves can be found correctly according to the learned TMGPFR model, leading to an effective way of curve clustering and prediction. However, this particular model selection problem is rather challenging because the conventional model selection criteria, such as BIC and cross-validation, cannot yield a stable result in realistic cases. Furthermore, they are even exposed to a heavy load of repetitive computation.

Recently, the Bayesian Ying-Yang (BYY) harmony learning [23], Dirichlet process [24] and reversible-jump Markov chain Monte Carlo (RJMCMC) [25] have been applied to solving the model selection problem of MGP and achieved good effects [15,26,27,28,29]. They are referred to as automated model selection algorithms since they make model selection automatically during parameter learning of a given dataset. That is, the parameter estimation and model selection are made synchronously. However, it can be found by the experiments that the Dirichlet process-based algorithms usually select a larger number of components, whereas the RJMCMC-based algorithms are not very stable. In addition, these two kinds of algorithms require an additional EM step to conduct the parameter learning in order to yield a proper prediction, which impairs the automation of model selection. Motivated by the fact that the BYY harmony learning shows an excellent ability of automated model selection of Gaussian mixtures [30], we adopt it to solve this particular model selection problem in the TMGPFR model.

In this paper, we improve the original TMGPFR model to a tight and powerful structure and propose a BYY annealing learning algorithm for the improved TMGPFR model with automated model selection. Specifically, a fully connected structure from the lower layer to the upper layer is integrated into the TMGPFR model, and the BYY harmony function is derived for this kind of TMGPFR model. Moreover, the BYY annealing learning algorithm is designed for the parameter learning and automated model selection of the TMGPFR model of a curve dataset.

The remainder is organized as follows. We introduce the GP, GPFR, and TMGPFR models in Section 2. Then, we design our BYY annealing learning algorithm in Section 3. The decision strategies of our improved TMGPFR model and learning algorithm for curve clustering and prediction are discussed in Section 4. Section 5 presents our experimental results and analyses. We finally make further discussions and conclusions in Section 6.

2. Mathematical Description of the TMGPFR Model

2.1. The GP Model

We begin with a brief introduction of the Gaussian process (GP) model. Suppose that we have input variables $x={\left\{x_i\right\}}_{i=1}^N$ and output (or response) variables $y={\left\{y_i\right\}}_{i=1}^N$. Then the GP model can be defined by

$$y\sim \mathcal{N}(0,K(x,x\left)\right),$$

(1)

where $K(x,x)$ is the kernel matrix. A common form of the kernel matrix, the radial basis function (RBF) kernel, can be expressed by

$$K_{\theta }{(x,x)}_{ij}:=K_{\theta }(x_i,x_j)={\theta }_1^{2}exp\left(-{\theta }_2^2\frac{\left|\right|x_i-x_j{\left|\right|}_2^2}{2}\right)+{\theta }_3^2{\delta }_{ij},$$

(2)

where ${\delta }_{ij}$ is the Kronecker delta function, and $\theta =({\theta }_1,{\theta }_2,{\theta }_3)$. We adopt this kernel form throughout the paper.

2.2. The GPFR Model

Since the assumption of zero mean in the GP model is too limited, it is possible to use a weighted sum of certain basis functions to learn and approximate the mean function during the parameter learning. In general, a set of B-spline basis functions are adopted. Specifically, we substitute the mean function in the GP model with

$$m\left(x\right)=\sum _{d=1}^D{\varphi }_d\left(x\right)b_d,$$

(3)

where D is the number of preset B-spline basis functions such that $\varphi \left(x\right)=({\varphi }_1\left(x\right),\cdots ,{\varphi }_D\left(x\right))$ is a set of different B-spline basis functions, and $b=(b_1,\cdots ,b_D)$ is a D-dimensional coefficient or weight vector. Obviously, the larger D brings the higher accuracy of the approximation but a heavier computing burden. Furthermore, the overfitting is more likely to emerge as D increases. This is certainly a trade-off problem and D is generally selected by experience.

In such a setting, the GP model is transformed into the so-called GPFR model [6], which is denoted by $y\sim \mathrm{GPFR}(x,\theta ,b,\varphi )$.

2.3. The TMGPFR Model

The TMGPFR model was developed from the GPFR mixture models to fit the data of curves from different independent random information sources or stochastic processes. In the lower layer, there are a large number of GPFRs driven by the input variables, and in the upper layer, a proper number of MGPFRs is employed to describe these stochastic processes and further mixed together to form the output variables.

2.3.1. The Lower Layer: A Fixed Number of GPFRs for Constructing MGPFRs

We assume that there exist G GPFRs in the lower layer and G is fixed and large enough so that the mixture of these G GPFRs can approximate any stochastic process. For each single sample curve ${\left\{(x_{mn},y_{mn})\right\}}_{n=1}^N$, the latent indicators ${\left\{z_{mn}\right\}}_{n=1}^N$ are generated by a multinomial distribution:

$$P(z_{mn}=g)={\eta }_g,g=1,2,\cdots ,G\mathrm{for}n=1,2,\cdots ,N.$$

(4)

Given the indicator $z_{mn}=g$, the input $x_{mn}$ is subject to a Gaussian distribution:

$$x_{mn}|z_{mn}=g\sim \mathcal{N}({\mu }_g,{\sigma }_g^2)\mathrm{i}.\mathrm{i}.\mathrm{d}.\mathrm{for} n=1,2,\cdots ,N.$$

(5)

After specifying ${\{z_{mn},x_{mn}\}}_{n=1}^N$, all the input variables are divided into G sets: ${\left\{x_{1,s}\right|z_{mn}=1\}}_{s=1}^{S_1},\cdots ,{\left\{x_{G,s}\right|z_{mn}=G\}}_{s=1}^{S_G}$ and ${\left\{x_{mn}\right\}}_{n=1}^N={\left\{{\left\{x_{g,s}\right|z_{mn}=g\}}_{s=1}^{S_g}\right\}}_{g=1}^G$. Then, the input variables in $x_g={\left\{x_{g,s}\right\}}_{s=1}^{S_g}$ and the corresponding output variables of $y_g={\left\{y_{g,s}\right\}}_{s=1}^{S_g}$ follow the g-th component GPFR model:

$$y_g\sim \mathrm{GPFR}(x_g,{\theta }_g,b_g,{\varphi }_g).$$

(6)

In this case, ${\left\{(x_{mn},y_{mn})\right\}}_{n=1}^N={\left\{{\left\{(x_{g,s},y_{g,s})\right\}}_{s=1}^{S_g}\right\}}_{g=1}^G$.

Letting $\eta =({\eta }_1,\cdots ,{\eta }_G)$ and taking the similar notations for $\theta ,b,\varphi ,\mu ,\sigma$, we will use $y\sim \mathrm{MGPFR}(x,\eta ,\theta ,b,\varphi ,\mu ,\sigma )$ to denote a component MGPFR model in the upper layer. It should be noted that $\eta ,\mu ,\sigma \in {\mathbb{R}}^G,\theta \in {\mathbb{R}}^{3\times G},b\in {\mathbb{R}}^{D\times G},\varphi \in {\mathrm{B}-\mathrm{spline}}^{D\times G}$.

2.3.2. The Upper Layer: MGPFRs for Constructing the Output Mix-MGPFR

We assume there exist K MGPFRs in the upper layer. For a dataset $\mathcal{D}={\left\{(x_m,y_m)\right\}}_{m=1}^M$ where $(x_m,y_m)={\left\{(x_{mn},y_{mn})\right\}}_{n=1}^{N_m}$ denotes the m-th sample curve, latent indicators ${\left\{{\alpha }_m\right\}}_{m=1}^M$ are also generated by a multinomial distribution:

$$P({\alpha }_m=k)={\pi }_k,k=1,2,\cdots ,K \mathrm{i}.\mathrm{i}.\mathrm{d}.\mathrm{for} m=1,2,\cdots ,M.$$

(7)

Given the indicator ${\alpha }_m=k$, we have

$$y_m|{\alpha }_m=k\sim \mathrm{MGPFR}(x_m,{\eta }_k,\theta ,b,\varphi ,\mu ,\sigma ).$$

(8)

It should be pointed out that the above parameters $\theta ,b,\varphi ,\mu ,\sigma$ do not have their own subscripts because all the G GPFRs in the lower layer are fully connected and contributed to each of the MGPFRs in the upper layer just like a fully connected neural network, so they share the same parameters $\theta ,b,\varphi ,\mu ,\sigma$ for all the MGPFRs. Certainly, the weight parameters ${\eta }_k\in {\mathbb{R}}^G,k=1,\cdots ,K$ are bound to be different because they serve as a unique feature to distinguish these MGPFRs. In fact, ${\eta }_k$ reflects how these GPFRs in the lower layer essentially contribute to the k-th MGPFR in the upper layer.

According to the above description, our assumptions for the TMGPFR model are different from those in Wu and Ma [22]. That is, we adopt Equation (8) instead of $y_m|{\alpha }_m=k\sim \mathrm{MGPFR}(x_m,{\eta }_k,{\theta }_k,b_k,{\varphi }_k,{\mu }_k,{\sigma }_k)$. In fact, the original TMGPFR model in Wu and Ma [22] assumes that the k-th MGPFR model in the upper layer possesses its own $G_k$ GPFRs in the lower layer and ${\sum }_{k=1}^K{G}_k=G$; thus, it is a sparse structure in the lower layer and the fully-connected structure is not actually implemented, which leads to the neglect of possible correlations among different MGPFRs. Therefore, we improve the TMGPFR model to a tight and powerful structure.

The TMGPFR model is actually a generative model which can describe a mixture of multiple information sources or stochastic processes and can be directly applied to the tasks of curve clustering and prediction of a given curve dataset $\mathcal{D}$. The information flow of the improved TMGPFR model is shown in Figure 1, whereas its structure sketch is displayed in Figure 2.

2.3.3. Notations

For clarity, we summarize all of the necessary notations which will be constantly used to describe our models and algorithms in the following.

To be more convenient, we define a 0–1 indicator variable $A_m^k$ to describe the association between the m-th sample curve and the k-th MGPFR. That is, if the m-th sample curve belongs to the k-th MGPFR, $A_m^k=1$; otherwise, $A_m^k=0$. On the other hand, we define an analogous variable $B_{mn}^{g|k}$ to describe the association between the n-th sample point in the m-th sample curve and the g-th GPFR in the k-th MGPFR. That is, if the n-th sample point in the m-th sample curve belongs to the g-th GPFR on the condition that the m-th sample curve belongs to the k-th MGPFR, $B_{mn}^{g|k}=1$; otherwise, $B_{mn}^{g|k}=0$. Similarly, we denote $A=\left\{A_m^k\right|m=1,2,\cdots ,M,k=1,2,\cdots ,K\}$ and $B=\{B_{mn}^{g|k},n=1,2,\cdots ,N_m,m=1,2,\cdots ,M,k=1,2,\cdots ,K,g=1,2,\cdots ,G\}$.

The notations of the other parameters are identical to those described in Section 2.3.2, including the dataset $\mathcal{D}={\left\{{\left\{(x_{mn},y_{mn})\right\}}_{n=1}^{N_m}\right\}}_{m=1}^M$, latent variables, and parameters in the model. We denote the whole parameter set as $\mathsf{\Theta }=(D,K,G,\pi ,\eta ,\theta ,b,\varphi ,\mu ,\sigma )$, where $D,K,G\in {\mathbb{Z}}^{+},\pi =({\pi }_1,\cdots ,{\pi }_K),\eta ={({\eta }_1,\cdots ,{\eta }_K)}^T={\left({\eta }_{kg}\right)}_{K\times G}$. Among all the parameters, $D,K,G,\varphi$ are preset and remain unchangeable, whereas the others are learned by the algorithm and will be updated during iterations.

3. BYY Annealing Learning Algorithm

3.1. BYY Harmony Function

A BYY harmony learning system describes an observation $\mathit{x}$ and its inner representation $\alpha$ through two types of Bayesian decomposition of the joint probability: $p(\mathit{x},\alpha )=p\left(\mathit{x}\right)p\left(\alpha \right|\mathit{x})=q(\alpha \left)q\right(\mathit{x}\left|\alpha \right)$. The former is called the Yang machine, whereas the latter is called the Ying machine. In our improved TMGPFR model, $\mathit{x}$ denotes a sample curve $(x,y)$ and $\alpha$ denotes the index of the corresponding MGPFR in the upper layer. $p\left(\alpha \right|\mathit{x})$ is the probability (density) that $\mathit{x}$ belongs to $\alpha$, whereas $q\left(\mathit{x}\right|\alpha )$ is the probability (density) that $\alpha$ generates $\mathit{x}$. The objective of BYY harmony learning is to maximize the following harmony function:

$$H\left(p\right|\left|q\right)=\int \int p\left(\mathit{x}\right)p\left(\alpha \right|\mathit{x})logq\left(\alpha \right)q\left(\mathit{x}\right|\alpha )d\mathit{x}d\alpha -r_q,$$

(9)

where $r_q$ is the regularizer. The theoretical details can be found in [23,31].

As for a specifically given dataset of the TMGPFR model, it is natural to use the empirical density function $\widehat{p}(x,y)=\frac{1}{M(N_1+N_2+\cdots N_M)}{\sum }_{m=1}^M{\sum }_{n=1}^{N_m}{\delta }_{(x_{mn},y_{mn})}(x,y)$ to estimate the true density function $p(x,y)$. Here, M denotes the total number of curves and $X_m=\{(x_{mn},y_{mn}):n=1,2,\cdots ,N_m\}$ denotes the m-th sample curve in the given dataset. Supposing that $N_1=\cdots =N_M=N$ and that N and M are large enough, we have $\widehat{p}(x,y)\approx p(x,y)$. For simplicity, the regularizer is omitted. In the same way as the BYY annealing harmony learning for Gaussian mixtures in [30], we consider the annealing harmony function $H_{\lambda }\left(\mathsf{\Theta }\right)=H\left(p\right|\left|q\right)+\lambda E\left(p\left(\alpha \right|\mathit{x})\right)$, where $E(·)$ is the entropy function over K MGPFRs:

$$E\left(p\right(\alpha \left|\mathit{x}\right))=-\int \int p(\alpha \left|\mathit{x}\right)\log p\left(\alpha \right|\mathit{x}\left)p\right(\mathit{x})d\mathit{x}d\alpha ,$$

(10)

and $\lambda$ is a tuning parameter dependent on the iteration time t. When $\lambda$ is relatively large, the maximization of $H_{\lambda }\left(\mathsf{\Theta }\right)$ encourages multiple components with a soft classification. When $\lambda$ tends to be zero, it gradually shifts to a hard-cut classification with a great number of local maximums. Thus, it is reasonable to select a large initial value $\lambda \left(0\right)$ and reduce it to zero during the iterations. In this way, the learning parameters can be ergodic to a certain extent and escape from local maximums, thus having a higher probability of reaching the global maximum of $H\left(p\right|\left|q\right)$, which further leads to the automated model selection during the parameter learning [30]. That is, if the initial value $K_{\mathrm{init}}$ is set to be greater than the true value $K_{\mathrm{true}}$, the global maximization of $H\left(p\right|\left|q\right)$ can select $K_{\mathrm{true}}$ MGPFRs to match the actual components in the curve dataset and force the mixing proportions of the other $(K_{\mathrm{init}}-K_{\mathrm{true}})$ extra MGPFRs to vanish, i.e., ${\pi }_k=0$. Therefore, the remaining $K^{\ast }$ components through the global maximization of $H\left(p\right|\left|q\right)$ are the actual components in the dataset so that the model selection is actually made automatically. That is, the automated model selection of the TMGPFRs can be effectively implemented by the global maximization of the annealing harmony function.

We now derive the specific expression of the annealing harmony function of our improved TMGPFR model of a given curve dataset. By substituting $p(x,y)$ with $\widehat{p}(x,y)$ in Equations (9) and (10), we can obtain the empirical harmony and annealing harmony functions as follows:

$$\widehat{H}\left(p\right|\left|q\right)=\frac{1}{M}\sum _{m=1}^M\sum _{k=1}^{K}p({\alpha }_m=k|(x_m,y_m))\times log\left[{\pi }_{k}q\left((x_m,y_m)\right|{\alpha }_m=k)\right];$$

(11)

$${\widehat{H}}_{\lambda }\left(\mathsf{\Theta }\right)=\widehat{H}\left(p\right|\left|q\right)-\lambda \times \frac{1}{M}\sum _{m=1}^M\sum _{k=1}^{K}p({\alpha }_m=k|(x_m,y_m))\times logp({\alpha }_m=k|(x_m,y_m)).$$

(12)

According to the assumptions in our improved TMGPFR model, $q\left((x_m,y_m)\right|{\alpha }_m=k)$ takes the following specific form:

$$q\left((x_m,y_m)\right|{\alpha }_m=k)=\prod _{g=1}^G\left\{\left[\prod _{n=1}^{N_m}{\left({\eta }_{kg}{p}_{\mathcal{N}}\left(x_{mn}\right|{\mu }_g,{\sigma }_g^2)\right)}^{B_{mn}^{g|k}}\right]p_{\mathcal{N}}\left(y_{mn}^{g|k}\right|b_g,{\theta }_g)\right\}.$$

(13)

Here, $\left\{x_{mn}^{g|k}\right\}=\{x_{mn},n=1,2,\cdots ,N_m|B_{mn}^{g|k}=1\}$ and $\left\{y_{mn}^{g|k}\right\}=\{y_{mn},n=1,2,\cdots ,$ $N_m|B_{mn}^{g|k}=1\}$. ${\prod }_{n=1}^{N_m}{\left({\eta }_{kg}{p}_{\mathcal{N}}\left(x_{mn}\right|{\mu }_g,{\sigma }_g^2)\right)}^{B_{mn}^{g|k}},p_{\mathcal{N}}\left(y_{mn}^{g|k}\right|b_g,{\theta }_g)$ are the likelihoods for $x_m$ and $y_m$ conditioned for ${\alpha }_m=k$, respectively. In the following, we will use the notation $x_{mn}^{g|k}$ or $y_{mn}^{g|k}$ to denote the vector composed of the elements in the set $\left\{x_{mn}^{g|k}\right\}$ or $\left\{y_{mn}^{g|k}\right\}$ and define $S_{m,k,g}=\left|\left\{x_{mn}^{g|k}\right\}\right|$.

3.2. Alternative Optimization Algorithm

As the empirical annealing harmony function is clear and specific with Equations (11)–(13), we further design the parameter learning algorithm to maximize it. Actually, the maximization of this annealing harmony function is rather difficult and cannot be solved directly. However, it is lucky that the whole parameters and certain unknown variables can be divided into two parts and we can effectively optimize each of them alternatively. It should be noted that $\lambda$ is the annealing factor that does not need to be optimized. In this way, we can design an alternative optimization algorithm as follows.

We first divide the whole parameters and unknown variables into 2 sets: ${\mathsf{\Theta }}_1=\left\{{\left\{p({\alpha }_m=k|(x_m,y_m))\right\}}_{k,m},B\right\}$ and ${\mathsf{\Theta }}_2=\{\pi ,\eta ,\theta ,b,\mu ,\sigma \}=\mathsf{\Theta }\backslash \{D,K,G,\varphi \}$. It is clear that ${\mathsf{\Theta }}_2$ contains the common parameters of the TMGPFR model, whereas ${\mathsf{\Theta }}_1$ contains the conditional probabilities as well as the latent indicators of the curve data. After initializing $D,K,G,\varphi ,{\mathsf{\Theta }}_1,{\mathsf{\Theta }}_2$, we then iterate ${\mathsf{\Theta }}_1$ and ${\mathsf{\Theta }}_2$ in the following steps until convergence or reaching the maximum iterations T in an alternative optimization manner:

• Step 1:Fix ${\mathsf{\Theta }}_2$, update ${\mathsf{\Theta }}_1={\mathrm{argmax}}_{{\mathsf{\Theta }}_1}{\widehat{H}}_{\lambda }({\mathsf{\Theta }}_1,{\mathsf{\Theta }}_2)$;
• Step 2: Fix ${\mathsf{\Theta }}_1$, update ${\mathsf{\Theta }}_2={\mathrm{argmax}}_{{\mathsf{\Theta }}_2}{\widehat{H}}_{\lambda }({\mathsf{\Theta }}_1,{\mathsf{\Theta }}_2)$;
• Step 3: Update $\lambda ={\lambda }^{\left(t\right)}<{\lambda }^{(t-1)}$, where t denotes the iteration time.

In Step 1, it is key to determine the values of the elements of B. In fact, only when all the relations among the sample data points, GPFRs, and MGPFRs are known, we can compute $q\left((x_m,y_m)\right|{\alpha }_m=k)$ according to Equation (13) and then update $p({\alpha }_m=k|(x_m,y_m))$. To solve this problem, there are two possible methods: Markov chain Monte Carlo (MCMC) simulation [22] and hard-cut classification [14]. Since the time complexity of the MCMC simulation is rather high [22,32], we choose the following hard-cut rule to obtain the values of the elements of B.

According to the Bayes formula, we first have

$$p(B_{mn}^{g|k}=1|(x_{mn},y_{mn}),{\alpha }_m=k)=\frac{{\eta }_{kg}p\left([x_{mn},y_{mn}]\right|B_{mn}^{g|k}=1)}{{\sum }_{g=1}^G{\eta }_{kg}p\left((x_{mn},y_{mn})\right|B_{mn}^{g|k}=1)},$$

(14)

where

$$p\left((x_{mn},y_{mn})\right|B_{mn}^{g|k}=1)=p_{\mathcal{N}}\left(x_{mn}\right|{\mu }_g,{\sigma }_g^2)\times p_{\mathcal{N}}\left(y_{mm}\right|\varphi \left(x_{mn}\right)·b_g,{\theta }_{g1}^2+{\theta }_{g3}^2).$$

(15)

By letting each data point belong to the component with the highest probability, we then have the hard-cut rule:

$$B_{mn}^{g|k}=\left\{\begin{array}{c}1,\mathrm{if}g={\mathrm{argmax}}_{g}p(B_{mn}^{g|k}=1|[x_{mn},y_{mn}],{\alpha }_m=k);\hfill \\ 0, \mathrm{otherwise}.\hfill \end{array}\right.$$

(16)

With the above update of B, we can compute $q\left((x_m,y_m)\right|{\alpha }_m=k)$ according to Equation (13). It should be noted that as ${\mathsf{\Theta }}_1$ is updated, there exist certain restrictions, i.e., ${\sum }_{k=1}^{K}p({\alpha }_m=k|(x_m,y_m))=1,\forall m$. By using the method of Lagrange multipliers, we can obtain the following update rule:

$$p({\alpha }_m=k|(x_m,y_m))=\frac{[{\pi }_{k}q{\left((x_m,y_m)\right|{\alpha }_m=k]}^{1/\lambda }}{{\sum }_{k=1}^K{\left[{\pi }_{k}q\left((x_m,y_m)\right|{\alpha }_m=k)\right]}^{1/\lambda }}.$$

(17)

From Equation (17), the role of $\lambda$ can be further understood. Actually, $p(z_n=k|[x_n,y_n])\to \frac{1}{K}$ (i.e., soft classification) as $\lambda \to +\infty$, and $p(z_n=k|[x_n,y_n])\to$${\delta }_{k,{argmax}_{k}p(z_n=k|[x_n,y_n])}$ (i.e., hard classification) as $\lambda \to 0+$. This gives a theoretical explanation of the action of $\lambda$ in the training process.

In Step 2, there also exist $K+1$ restrictions, i.e., ${\sum }_{k=1}^K{\pi }_k=1,{\sum }_{g=1}^G{\eta }_{kg}=1,\forall k$. In a similar way, we can derive the update rules of $\pi ,\eta ,\mu ,\sigma ,b,\theta$. The details of the derivations are given in Appendix A.

In Step 3, $\lambda \left(t\right)$ plays an important role in the convergence behavior with the annealing mechanism. It is clear that if $\lambda \left(t\right)$ attenuates faster, the algorithm converges faster but the object function is more likely to be trapped into a local maximum. Therefore, $\lambda \left(t\right)$ should be carefully designed to balance the tradeoff between the convergence speed and effect. For this algorithm, we adopt $\lambda \left(t\right)={[u(1-e^{-v(t-1)})+w]}^{-1}$, given in [30].

The pseudo-code of our BYY annealing learning algorithm is given in Algorithm 1.

Algorithm 1: The BYY annealing learning algorithm of the TMGPFR model.

4. Decision Strategies for Curve Clustering and Prediction

In this section, we further discuss the decision strategies of our improved TMGPFR model and learning algorithm for the tasks of curve clustering and prediction of a test set ${\mathcal{D}}^{\prime }={\left\{{\left\{(x_{mn}^{\prime },y_{mn}^{\prime })\right\}}_{n=1}^{N_m^{\prime }}\right\}}_{m=1}^{M^{\prime }}$ after the BYY annealing algorithm has been implemented in the training dataset $\mathcal{D}$ with the desired parameters.

For curve clustering, the decision strategy or rule for a new curve is clear and simple. First, we determine all the values of the elements in the set $B^{\prime }=\left\{B_{mn}^{g|k^{\prime }}\right\}$ for any curves in ${\mathcal{D}}^{\prime }$ according to Equations (14)–(16). Then, we compute $q\left((x_m^{\prime },y_m^{\prime })\right|{\alpha }_m^{\prime }=k)$ and use the Bayes formula to obtain $p({\alpha }_m^{\prime }=k|(x_m^{\prime },y_m^{\prime }))$ for all $m,k$. Finally, we choose $k={\mathrm{argmax}}_{k}p({\alpha }_m^{\prime }=k|\left(x_m^{\prime }{y}_m^{\prime }\right))$ to classify or label $(x_m^{\prime },y_m^{\prime })$. On the whole, we can use the classification accuracy of ${\mathcal{D}}^{\prime }$ to measure the performance of curve clustering by our algorithm.

For the curve prediction, the situation becomes complicated. Given a sample curve $(x_m^{\prime },y_m^{\prime })$ in ${\mathcal{D}}^{\prime }$, we need to predict $y_{mn\ast }^{\prime }$ after giving a new input variable $x_{mn\ast }^{\prime }$. Assume that we have already finished the complete learning process required for clustering except labeling the sample curves. Then, for any pair $(k,g)\in \{1,2,\cdots ,K\}\times \{1,2,\cdots ,G\}$, we can compute $p_{kg}=p(B_{mn\ast }^{g|k^{\prime }}=1|x_{mn\ast }^{\prime },{\alpha }_m=k)$ according to the following Bayes formula:

$$p_{kg}=\frac{{\eta }_{kg}{p}_{\mathcal{N}}\left(x_{mn\ast }^{\prime }\right|{\mu }_g,{\sigma }_g^2)}{{\sum }_{g=1}^G{\eta }_{kg}{p}_{\mathcal{N}}\left(x_{mn\ast }^{\prime }\right|{\mu }_g,{\sigma }_g^2)}.$$

(18)

Then its unique prediction $y_{kg}^{\prime }\left(x_{mn\ast }^{\prime }\right)$ is computed by

$$y_{kg}^{\prime }\left(x_{mn\ast }^{\prime }\right)={\varphi }_g\left(x_{mn\ast }^{\prime }\right)b_g+C_{kg}{\Sigma }_{kg}^{-1}(y_{mn}^{g|k^{\prime }}-{\mathsf{\Phi }}_{kg}{b}_g),$$

(19)

where

$$\begin{array}{cc}\hfill C_{kg}& =K_{{\theta }_g}(x_{mn\ast }^{\prime },x_{mn}^{g|k^{\prime }})\in {\mathbb{R}}^{S_{m,k,g}^{\prime }};\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill {\Sigma }_{kg}& =K_{{\theta }_g}(x_{mn}^{g|k^{\prime }},x_{mn}^{g|k^{\prime }})\in {\mathbb{R}}^{S_{m,k,g}^{\prime }\times S_{m,k,g}^{\prime }};\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill [{\mathsf{\Phi }}_{kg}{]}_{ij}& ={\varphi }_{gj}\left(x_{mni}^{g|k^{\prime }}\right),{\mathsf{\Phi }}_{kg}\in {\mathbb{R}}^{S_{m,k,g}^{\prime }\times D}.\hfill \end{array}$$

(22)

For any $k\in \{1,2,\cdots ,K\}$, we can compute its unique prediction $y_k^{\prime }\left(x_{mn\ast }^{\prime }\right)$ by

$$y_k^{\prime }\left(x_{mn\ast }^{\prime }\right)=\sum _{g=1}^G{p}_{kg}{y}_{kg}^{\prime }\left(x_{mn\ast }^{\prime }\right).$$

(23)

Finally, we fuse all the component predictions and obtain the ultimate prediction by

$$y_{mn\ast }^{\prime }=\sum _{k=1}^{K}p({\alpha }_m=k|(x_m^{\prime },y_m^{\prime }))y_k^{\prime }\left(x_{mn\ast }^{\prime }\right).$$

(24)

Moreover, we can use the root mean square error (RMSE) between this prediction and the true value to measure the prediction accuracy.

5. Experimental Results

5.1. Curve Clustering Analysis

5.1.1. On the Simulation Dataset

We begin with the test of our improved TMGPFR model through the BYY annealing learning algorithm of a simulation dataset. Typically, we generate a set of $M=$ 50 unlabeled curves from three MGPFR models with different weights: ${\pi }_1=0.2,{\pi }_2=0.3,{\pi }_3=0.5$. Each curve has 60 points, i.e., $N=N_1=N_2=\cdots =N_{50}=60$. The true values of the parameters of three MGPFR models are given in

Table 1. True values of the parameters in three MGPFRs.

Parameters in MGPFR1	True Values
${\eta }_1,{\eta }_2$	$0.7,0.3$
${\theta }_1,{\theta }_2$	$(0.32,0.4,0.01),(0.4,0.5,0.01)$
$m_1\left(x\right),m_2\left(x\right)$	$sin\left(x^3\right)-0.3cos\left(x^3\right),cos\left(\sqrt{\left|x\right|}\right)$
${\mu }_1,{\mu }_2$	$-10,10$
${\sigma }_1,{\sigma }_2$	$5,4$
Parameters in MGPFR2	True Values
${\eta }_1,{\eta }_2$	$0.6,0.4$
${\theta }_1,{\theta }_2$	$(0.3,0.9,0.01),(0.2,0.2,0.01)$
$m_1\left(x\right),m_2\left(x\right)$	$exp(-|x\left|\right),0.01x^2-0.1\left|x\right|+1$
${\mu }_1,{\mu }_2$	$-8,6$
${\sigma }_1,{\sigma }_2$	$2,8$
Parameters in MGPFR3	True Values
${\eta }_1,{\eta }_2$	$0.5,0.5$
${\theta }_1,{\theta }_2$	$(0.25,0.1,0.01),(0.45,0.9,0.01)$
$m_1\left(x\right),m_2\left(x\right)$	$sinx+cosx,\frac{1}{\left|x\right|+1}+0.1\left|x\right|-0.5$
${\mu }_1,{\mu }_2$	$-4,10$
${\sigma }_1,{\sigma }_2$	$3,5$

. It should be noted that we use some other continuous functions to be the mean functions of the GPFRs rather than the given B-spline functions themselves, which are denoted by $m_i\left(x\right)$.

We implement our BYY annealing learning algorithm of this dataset with the initial values of the parameters being set in a simple manner, which are specifically described in Appendix B.1. Our experimental results are shown in Figure 3. In the experiments, we actually set $K_{\mathrm{init}}=10$. After the automated model selection during the parameter learning, we finally obtain $K^{\ast }=3$. It can be observed from Figure 3 that even if the structures of three curve clusters in the left subfigure are so complex and interlaced, all the sample curves are clustered correctly and the classification accuracy of our algorithm arrives at 100%.

We also implement our algorithm of the datasets with $M=200,500,1000,5000$, 10,000, and 50,000, i.e., the number of sample curves increases from 200 to 50,000 such that the sample size changes from small to big. In this situation, we set the batch size to 200 and slow down the annealing speed. Our experimental results of these datasets are given in

Table 2. Average classification accuracies of our BYY annealing leaning algorithm of the simulation datasets with different M.

M	200	500	1000	5000	10,000	50,000
Accuracy	$98.2\%$	$96.6\%$	$91.5\%$	$92.6\%$	$90.3\%$	$91.9\%$
$K^{\ast }$	3	4	4	4	5	5

. It can be found from Table 2 that the classification accuracy of our algorithm fluctuates as M increases, because outliers are more likely to appear due to the stronger randomness. Furthermore, in this case, our algorithm tends to choose a larger $K^{\ast }$ with some components only occupying 0.1–5.0% of the whole curves. However, it does not necessarily mean the prediction results will worsen. We will further clarify it in Section 5.2.

In addition, we conduct several parallel experiments to compare our algorithm with other model selection methods. From Figure A1, we can observe that our BYY annealing learning algorithm performs much better. The details of the experiments and comparisons can be found in Appendix B.1.

5.1.2. On the City Temperature Dataset

We further implement our BYY annealing learning algorithm of a real dataset of daily temperatures of different cities. In practice, it is very crucial to determine the type of climate in a certain city according to its daily temperature records. In total, we choose $M=$ 97 unlabeled sample curves from 4 cities: Abidjan, Auckland, Shanghai, and Helsinki. Each curve consists of 87 sample points, reflecting the change in temperatures in one year. Specifically, each sample point is the average of city temperatures in a period of 4 adjacent days. Due to the inaccuracy of measurement, some abnormal sample points are discarded. The experimental results are shown in Figure 4 and the details are described in Appendix B.2. Here, $K_{\mathrm{init}}=10$, and our algorithm finally yields $K^{\ast }=4$ after the automated model selection. It is found that the classification accuracy is 100% for this particular real-world task.

5.1.3. On the Effective Radiation Dataset

We finally implement our BYY annealing learning algorithm in a real dataset of daily surface effective radiation of different cities in China. In total, we choose $M=$ 100 unlabeled sample curves from 20 different cities: Eji’na, Guilin, Chaoyang, Shenyang, Juxian, Tianshui, Hengfeng, Zhongshan, Wulatezhongqi, Changchun, Dalian, Zhenghe, Leshan, Fuzhou, Xi’an, Yichang, Luohe, Emeishan, Yanji, and Shanghai. The meanings and processing methods of this dataset keep the same as those described in Section 5.1.2. However, due to data missing, a sample curve in this dataset has only 75 sample points. The experimental results are shown in Figure 5. Here $K_{\mathrm{init}}=10$ and our algorithm yields $K^{\ast }=4$ after the automated model selection. The details of the experiments are described in Appendix B.3.

Our classification criterion is the winner-take-all (WTA) principle, i.e., a certain station is definitely classified into the category that its five daily radiation curves mostly belong to. Therefore, despite the fact that these curves are divided into four types by our algorithm, for cities themselves there exist only three types. As shown in Figure 5, a purple curve forms the fourth type, possibly due to certain random noise.

According to our experimental results, all the cities are divided into three clusters: {Eji’na, Wulatezhongqi} (pink), {Chaoyang, Shenyang, Juxian, Tianshui, Changchun, Dalian, Xi’an, Luohe, Yanji} (blue), and {Guilin, Hengfeng, Zhongshan, Zhenghe, Leshan, Fuzhou, Yichang, Emeishan, Shanghai} (orange). In fact, Eji’na and Wulatezhongqi lie in the deserts of the Inner Mongolian Plateau, which explains why these two cities are exposed to the highest solar radiation. The cities in the blue set have a higher latitude than those in the orange set; thus, they are exposed to a longer duration of sunshine in the summer and a lower rainfall during the whole year. These factors make them the second highest radiated. In sum, the clustering results of our algorithm are consistent with the meteorological facts, which demonstrate the effectiveness of our model and algorithm.

For comparison, we use two typical model selection criteria through the EM algorithm to make parameter learning and model selection of this dataset. It is found by our experiments that the AIC approach yields $K^{\ast }=2$, whereas the BIC approach even yields $K^{\ast }=1$, which are both too rough. These two experimental results demonstrate that the conventional model selection criteria based on the penalty of the number of authentic parameters often fail to tackle the complicated and noisy dataset.

5.2. Curve Prediction

We now turn to test our BYY annealing learning algorithm on curve prediction. In fact, given a piece of time series (as part of a sample curve) from the past to the present, it is very important to predict what will probably appear in the future. A reasonable prediction model can help us to understand the intrinsic logic of the data and make more reasonable decisions. In this subsection, we test our algorithm on an electrical load dataset issued by the Northwest China Grid Company. Actually, there are $M=$ 100 training sample curves and 50 test sample curves, each of which records the values of the electrical load over the period of a whole day. The electrical load is detected every 15 min and then there are altogether 96 points in one sample curve. We implement the BYY annealing learning algorithm in the training sample curves to obtain the reasonable TMGPFR model and then utilize it to make the curve prediction, i.e, to predict the future output of the curve with a new given input according to Equation (24). The experimental results of the curve clustering are shown in Figure 6.

It can be found from Figure 6 that the curve clustering results of the electrical load dataset are quite reasonable since these clusters effectively represent different intensities of the electrical load over a day. Moreover, the curve clustering results of the test dataset are consistent with those of the training dataset. In fact, these reasonable clustering results lay a solid foundation for curve prediction.

As for the curve prediction in this dataset, we assume that there are U known sample points in the front and $96-U$ unknown sample points in the behind. That is, we use the previous U sample points to predict the following $96-U$ sample points. It is clear that as U becomes smaller, the prediction becomes more difficult. We make curve predictions of the test dataset for different U and the experimental results are listed in

Table 3. Statistics of the RMSEs of curve predictions of our BYY annealing learning algorithm with different U.

U	20	30	40	50	60	70
mean ± SD	$3.38\pm 0.94$	$2.56\pm 0.38$	$2.14\pm 0.29$	$1.99\pm 0.17$	$1.93\pm 0.22$	$1.79\pm 0.14$
maximum	$4.43$	$3.28$	$2.68$	$2.41$	$2.51$	$2.07$

, where the mean and its standard deviation (SD) and the maximum of the RMSEs over 46 prediction trials are given for certain typical values of U. It can be found that the prediction results are rather precise and stable even if U is only 20.

We also test our BYY annealing learning algorithm on curve prediction for some larger sizes of this dataset; that is, we let $M=200,300,400,500,680$. Here $U\equiv 50$ and the batch size is 100. The larger M is, the slower annealing rate we adopt. The prediction results are listed in

Table 4. Statistics of the RMSEs of curve predictions of our BYY annealing learning algorithm with different M and $U\equiv 50$.

M	200	300	400	500	680
mean ± SD	$1.93\pm 0.18$	$1.64\pm 0.19$	$1.93\pm 0.22$	$1.84\pm 0.15$	$1.92\pm 0.35$
maximum	$2.24$	$1.96$	$2.38$	$2.07$	$2.44$
$K^{\ast }$	4	5	5	4	4

. It can be found that they are more precise, stable, and consistent, except the possible difference of $K^{\ast }$.

For comparison, we first implement three conventional model selection methods: AIC, BIC, and k-fold cross-validation (CV), with the EM algorithm of this electrical load dataset. The hyperparameters in the TMGPFR model keep constant and are selected by experience. The curve clustering results are shown in Figure A3 and the curve prediction results are listed in

Table 5. Statistics of the RMSEs of curve predictions of three conventional model selection approaches with different U.

Method	U	20	30	40	50	60	70
AIC & BIC	mean ± SD	$3.30\pm 1.08$	$3.40\pm 0.69$	$3.18\pm 0.45$	$3.09\pm 0.43$	$2.74\pm 0.25$	$2.58\pm 0.42$
	maximum	$4.61$	$4.38$	$3.92$	$4.02$	$3.26$	$3.12$
5-fold CV	mean ± SD	$3.42\pm 0.96$	$2.78\pm 0.34$	$2.15\pm 0.33$	$2.23\pm 0.25$	$2.22\pm 0.28$	$2.04\pm 0.28$
	maximum	$4.76$	$3.17$	$2.60$	$2.62$	$2.61$	$2.50$
10-fold CV	mean ± SD	$3.42\pm 1.02$	$2.45\pm 0.27$	$1.91\pm 0.18$	$1.90\pm 0.16$	$1.96\pm 0.16$	$1.85\pm 0.17$
	maximum	$5.03$	$2.98$	$2.34$	$2.36$	$2.32$	$2.09$

. Actually, the AIC and BIC approaches both yield $K^{\ast }=3$, the 5-fold CV approach yields $K^{\ast }=4$ and the 10-fold CV approach yields $K^{\ast }=5$. By comparing Table 3 and Table 5, we can find out that our BYY annealing learning algorithm outperforms these conventional model selection methods of this dataset. It is clear that the prediction results of the AIC, BIC, and 5-fold CV approaches are worse than our BYY annealing learning algorithm. Although the prediction error of the 10-fold CV approach is relatively smaller than ours, the difference between them is quite small. On the other hand, our algorithm really saves plenty of time and GPU resource owing to the synchronization of parameter learning and model selection.

We further compare our improved TMGPFR model with some state-of-the-art statistical and machine-learning models of curve prediction. The first kinds of comparative models are several previous statistical models, such as GPFR, mix-GPFR, MGPFR, and the original TMGPFR model proposed in Wu and Ma [22]. The EM algorithm is employed for training the GPFR and MGPFR models, whereas the BYY annealing learning algorithm is employed for training the mix-GPFR and the original TMGPFR models. The second kinds of comparative models are state-of-the-art machine learning models, such as long short-term memory (LSTM) [33], fully visible belief network (FVBN) [34,35], and generative adversarial nets (GAN) [36,37]). The experiments are conducted under the same conditions for $U=50$ and the curve prediction results are listed in

Table 6. Statistics of the RMSEs of curve predictions of the GPFR, mix-GPFR, MGPFR, original TMGPFR, and improved TMGPFR models with different $\{D,G)\}$ when $U=50$.

Model	G	D	10	15	20	25	30
GPFR	-	RMSE	$7.03\pm 1.83$	$7.04\pm 1.82$	$7.03\pm 1.82$	$7.03\pm 1.83$	$7.02\pm 1.83$
mix-GPFR ($K_{\mathrm{init}}\equiv 10$)	-	RMSE	$3.02\pm 0.32$	$2.92\pm 0.31$	$2.50\pm 0.21$	$2.49\pm 0.21$	$2.50\pm 0.17$
		$K^{\ast }$	2	2	3	3	3
MGPFR	4	RMSE	$7.08\pm 1.74$	$7.33\pm 2.27$	$7.54\pm 2.00$	$7.10\pm 1.74$	$7.02\pm 1.67$
	6	RMSE	$7.23\pm 1.82$	$7.11\pm 1.78$	$7.00\pm 1.75$	$7.12\pm 1.73$	$7.27\pm 1.73$
	8	RMSE	$7.18\pm 2.06$	$7.11\pm 1.76$	$7.05\pm 1.77$	$7.18\pm 1.76$	$7.04\pm 1.83$
	10	RMSE	$7.16\pm 1.80$	$7.10\pm 1.77$	$7.03\pm 1.82$	$7.03\pm 1.82$	$7.03\pm 1.80$
	12	RMSE	$7.07\pm 1.79$	$7.13\pm 1.73$	$7.04\pm 1.82$	$7.13\pm 1.79$	$7.08\pm 1.78$
The original TMGPFR [22] ($G_k\equiv 2,$ $K_{\mathrm{init}}\equiv G/G_k$)	4	RMSE	$3.16\pm 0.38$	$3.17\pm 0.35$	$3.06\pm 0.28$	$2.92\pm 0.32$	$2.92\pm 0.23$
		$K^{\ast }$	2	2	2	2	2
	$6$	RMSE	$2.37\pm 0.33$	$2.83\pm 0.57$	$2.41\pm 0.24$	$2.50\pm 0.19$	$2.42\pm 0.22$
		$K^{\ast }$	3	3	3	3	3
	$8$	RMSE	$2.96\pm 0.24$	$2.20\pm 0.38$	$2.50\pm 0.20$	$2.54\pm 0.24$	$2.62\pm 0.30$
		$K^{\ast }$	2	3	3	3	3
	$10$	RMSE	$2.43\pm 0.20$	$2.51\pm 0.37$	$2.49\pm 0.17$	$2.63\pm 0.20$	$2.00\pm 0.16$
		$K^{\ast }$	3	3	3	3	4
	$12$	RMSE	$2.87\pm 0.33$	$2.64\pm 0.33$	$2.31\pm 0.22$	$2.11\pm 0.20$	$2.42\pm 0.27$
		$K^{\ast }$	2	3	4	4	4
improved TMGPFR	4	RMSE	$2.79\pm 0.20$	$2.89\pm 0.35$	$2.46\pm 0.28$	$2.45\pm 0.32$	$2.30\pm 0.26$
		$K^{\ast }$	2	3	3	3	3
	$6$	RMSE	$2.41\pm 0.23$	$2.49\pm 0.34$	$2.48\pm 0.22$	$2.24\pm 0.31$	$2.27\pm 0.35$
		$K^{\ast }$	3	3	3	3	3
	$8$	RMSE	$2.11\pm 0.17$	$2.23\pm 0.20$	$1.94\pm 0.18$	$2.16\pm 0.30$	$2.12\pm 0.19$
		$K^{\ast }$	4	4	6	5	5
	$10$	RMSE	$2.32\pm 0.42$	$2.14\pm 0.24$	$2.10\pm 0.32$	$2.00\pm 0.20$	$1.99\pm 0.17$
		$K^{\ast }$	4	5	5	5	5
	$12$	RMSE	$2.20\pm 0.29$	$2.08\pm 0.21$	$1.98\pm 0.26$	$1.89\pm 0.16$	$2.04\pm 0.16$
		$K^{\ast }$	4	5	6	7	7

and

Table 7. Statistics of the RMSEs of curve predictions of LSTM, FVBN, and GAN when $U=50$.

Model	LSTM			FVBN
params	layers $=15$ hid dims = 16	layers $=5$ hid dims = 32	layers $=32$ hid dims = 3	linear lags = 2	linear lags = 10	linear lags = 50
RMSE	$11.73\pm 2.97$	$11.73\pm 2.96$	$11.75\pm 2.98$	$4.23\pm 3.52$	$7.07\pm 2.49$	$7.17\pm 3.76$
model	FVBN			GAN (·/·: generator/discriminator params)
params	LeakyReLU NN layers = 5 hid dims = 256	Tanh NN layers = 4 hid dims = 1024	Sigmoid NN layers = 6 hid dims = 128	vanilla GAN layers = 20/3 hid dims = 512/2	WGAN layers = 25/2 hid dims = 128/2	WGAN layers = 40/2 hid dims = 512/2
RMSE	$4.51\pm 1.97$	$4.50\pm 1.55$	$8.15\pm 2.38$	$6.59\pm 2.61$	$5.68\pm 2.19$	$5.17\pm 2.16$

. It can be seen that our improved TMGPFR model achieves the best performance for the aspects of both prediction precision and stability. Furthermore, it can be found from the experiments that the previous statistical models are weak (see Table 6) mainly due to their simpler assumptions, whereas LSTM and FVBN fail (see Table 7) mainly due to the accumulative errors in the rolling or successive prediction, and GAN fails (see Table 7) mainly due to the mode collapsing.

More analyses and supplementary experiments of this dataset are stated in Appendix B.4.

6. Discussion and Conclusions

6.1. Relation to the EM Algorithms

Interestingly, the updating rules of our BYY annealing learning algorithm take similar forms as those of the EM algorithm and the deterministic annealing EM algorithm [38]. However, our idea is quite different from those of the EM algorithms. For the EM algorithm, the annealing parameter $\lambda$ should be fixed to 1, whereas in the deterministic annealing EM algorithm, $\lambda$ should gradually approach to 1. Their common goal is to reach the global maximum of the likelihood function. However, in our BYY annealing algorithm, $\lambda$ attenuates to 0, which drives the algorithm to reach the global maximum of the harmony function. That is exactly why our BYY annealing algorithm possesses the ability of automated model selection but the EM algorithms usually do not. Actually, the harmony function provides severe penalization for large K, whereas the likelihood function is bound to increase with K. Unlike AIC, BIC, and CV, we integrate this kind of penalization into the parameter learning so that the model selection can be made automatically. Therefore, we do not need to train multiple candidate models for model selection.

6.2. The Annealing Mechanism in the BYY Harmony Learning System

When training the TMGPFR model with the hard-cut EM algorithm, we find out that a small perturbation of the initial values of the parameters might strongly impact the results, especially in model selection. The reason is that there are a large number of local maximums of the BYY harmony function with the hard-cut form so that the hard-cut EM algorithm is easily trapped. On the contrary, as we apply the annealing mechanism to the hard-cut EM algorithm, most of the components are able to attract certain samples due to a higher posterior probability in the early iterations (see Equation (17)). Moreover, the change in the parameters is more smooth because the annealing mechanism can prevent the abrupt change in the sample number of each component. In this way, the BYY harmony learning system gradually turns into a hard-cut classification form to arrive at the optimal model. Hence, we must emphasize that without the annealing mechanism, it is rather difficult for the hard-cut EM algorithm to reach the global maximum of the harmony function, and therefore the automated model selection cannot be guaranteed.

6.3. Conclusions and Remarks

Conclusions. We have improved the TMGPFR model and proposed a BYY annealing learning algorithm for its parameter learning with the automated model selection of a given dataset. Specifically, we fixed the number of GPFRs in the lower layer and let them be fully connected to the MGPFRs in the upper layer such that the structure of this improved TMGPFR model becomes tight and powerful. Under the BYY harmony learning framework, the BYY harmony function of the improved TMGPFR model is derived and the BYY annealing learning algorithm is established with the ability to automatically find out the number of actual clusters in a given curve dataset and further make curve clustering and prediction. The experimental results of both simulation and real datasets have shown the effectiveness and superiority of the improved TMGPFR model with the BYY annealing learning algorithm.

Remarks. First, the differences between the blind source separation (BSS) techniques and the TMGPFR modeling. In the setting of the BSS models, the original independent sources are blind and cannot be detected individually. However, there are some sensors that can obtain linearly mixed signals from these original sources. When the number of mixed signals is equal to that of the sources, the independent component analysis (ICA) algorithms can be applied to decompose these signals of the original independent sources, whereas in the setting of our TMGPFR model, the original independent sources are also blind. As their sample curves are accumulated together without any labels in the dataset, our goal of the TMGPFR modeling is to discover the structure of these sources as a number of general stochastic processes throughout the BYY harmony annealing learning of the TMGPFR model in a set of unlabeled sample curves. Moreover, we can even find out the true number of the original sources in the dataset with the automated model selection mechanism. Second, the hard-cut mechanism in our BYY annealing harmony algorithm. Although we adopt it to determine the set B to speed up the training process, it impairs the convergence of the algorithm to a certain extent, especially when these GPFRs in the lower layer are strongly overlapped [22]. Fortunately, the annealing mechanism can mitigate these negative effects. The MCMC method (e.g., Gibbs sampler) can play the same role with the expense of a slower convergence rate. Third, the influences of $\mathbf{D},\mathbf{G}$ in our TMGPFR model. We only care about how to select K in the TMGPFR model, regardless of $D,G$. When D or G becomes larger (smaller), overfitting (underfitting) is more likely to emerge. It should be pointed out that our core goal is to find out the actual number of curve clusters in a dataset under the assumption that $D,G$ are large enough so that the TMGPFR model can describe a mixture of general stochastic processes. However, it is still a problem to choose the optimal $D,G$ for a given dataset. In our experiments, we apply certain heuristic methods to select $D,G$, which will be described in Appendix C.