Entanglement-Structured LSTM Boosts Chaotic Time Series Forecasting

Abstract

Traditional machine-learning methods are inefficient in capturing chaos in nonlinear dynamical systems, especially when the time difference $\Delta t$ between consecutive steps is so large that the extracted time series looks apparently random. Here, we introduce a new long-short-term-memory (LSTM)-based recurrent architecture by tensorizing the cell-state-to-state propagation therein, maintaining the long-term memory feature of LSTM, while simultaneously enhancing the learning of short-term nonlinear complexity. We stress that the global minima of training can be most efficiently reached by our tensor structure where all nonlinear terms, up to some polynomial order, are treated explicitly and weighted equally. The efficiency and generality of our architecture are systematically investigated and tested through theoretical analysis and experimental examinations. In our design, we have explicitly used two different many-body entanglement structures—matrix product states (MPS) and the multiscale entanglement renormalization ansatz (MERA)—as physics-inspired tensor decomposition techniques, from which we find that MERA generally performs better than MPS, hence conjecturing that the learnability of chaos is determined not only by the number of free parameters but also the tensor complexity—recognized as how entanglement entropy scales with varying matricization of the tensor.

1. Introduction

Time series forecasting [1], despite its undoubtedly tremendous potential in both theoretical issues (e.g., mechanical analysis, ergodicity) and real-world applications [2] (e.g., traffic, weather, and clinical records analysis), has long been known as an intricate field. From classical work on statistics such as auto-regressive moving average (ARMA) families [3] and basic hidden Markov models (HMM) [4,5] to contemporary machine-learning (ML) methods [6,7,8,9] such as gradient boosted trees (GBT) and neural networks (NN), the essential complexity in time series has been more and more frequently recognized. In particular, forecasting models have extended their applicable range from linear, Markovian cases to nonlinear, non-Markovian, and even more general situations [10]. Among all known methods, recurrent NN architectures [11], including plain recurrent neural networks (RNN) [12] and long short-term memory (LSTM) [13], are the most capable of capturing this complexity, as they admit the fundamental recurrent behavior of time series data. LSTM has proved useful in speech recognition and video analysis tasks [14] in which maintaining long-term memory is essential to the complexity. In relation to this objective, novel architectures such as higher-order RNN/LSTM (HO-RNN/LSTM) [15] have been introduced to capture long-term non-Markovianity explicitly, further improving performance and leading to more accurate theoretical analysis.

Still, another domain of complexity—chaos—has been far less understood [16,17]. Even though enormous theory/data-driven studies on forecasting chaotic time series by means of recurrent NN have been conducted [18,19,20,21,22,23,24,25], there is still a lack of consensus on which features play the most important roles in the forecasting methods. The notorious indication of chaos,

$$\begin{array}{c}\hfill \left|\delta x_t\right|\approx e^{\lambda t}\left|\delta x_0\right|\end{array}$$

(1)

(where $\lambda$ denotes the spectrum of Lyapunov exponents), suggests that the difficulty of forecasting chaotic time series is two-fold: first, any small error will propagate exponentially, and thus multi-step-ahead predictions will be exponentially worse than one-step-ahead ones; second, and more subtly, when the actual time difference $\Delta t$ between consecutive steps increases, the minimum redundancy of model capacity needed for smoothly descending to the global minima (or sufficiently good local ones) during NN training also increases exponentially. Most studies only address the first difficulty by improving the prediction accuracy achievable at the global minima. Yet the latter is in fact more crucial, especially when $\Delta t$ is so large that the time series looks apparently random and a trivial local minimum would most likely be reached instead. Recently, tensorization has been introduced in recurrent NN architectures [26,27]. A tensorized version of HO-RNN/LSTM, namely, HOT-RNN/LSTM [28], has claimed an advantage in learning long-term nonlinearity in Lorenz systems of small $\Delta t$. On the one hand, we believe that the global minima of chaos (where the dominance of linear dependence is absent) can be most efficiently reached through tensorization approaches, where all nonlinear terms, up to some polynomial order, are treated explicitly and weighted equally. On the other hand, for simple chaotic dynamical systems, nonlinear complexity is only encoded in the short term, not the long term, which HO/HOT models will not be efficient in capturing when $\Delta t$ is large. Hence, a new tensorization-based recurrent NN architecture is desired so as to foster our understanding of chaos in time series and to meet practical needs, e.g., the modeling of laminar flame fronts and chemical oscillations [29,30,31,32].

In this paper, we introduce a new LSTM-based architecture by tensorizing the cell-state-to-state propagation therein, retaining the long-term memory features of LSTM while simultaneously enhancing the learning of short-term nonlinear complexity. Compared with traditional LSTM architectures, including stacked LSTM [33] and other aforementioned statistics/ML-based forecasting methods, our model is shown to be a general and outperforming approach for capturing chaos in almost every typical chaotic continuous-time dynamical system and discrete-time map with controlled comparable NN training conditions, justified by both our theoretical analysis and experimental results. Our model is also tested on real-world time series datasets, where the improvements range up to $6.3\%$.

During the tensorization, we have explicitly embedded many-body quantum state structures—a way of reducing the exponentially large degree of freedom of a tensor (i.e., tensor decomposition)—popularly studied in condensed matter physics, which is not unseen in NN design [34]. A many-body entangled state living in a tensor-product Hilbert space is hardly separable. The same inseparability behavior also appears in nonlinear multi-variate functions when crossing terms between different variables become too complex. This similarity motivated us to adopt a special measure of tensor complexity, namely, the entanglement entropy (EE) [35], which is commonly used in quantum physics and quantum information [34]. For one-dimensional many-body states, two thoroughly studied, popular but different structures exist—multiscale entanglement renormalization ansatz (MERA) [36] and matrix product states (MPS) [37], of which the EE scales with the subsystem size or not at all, respectively [35]. For most pertinent studies, MPS has been proven to be efficient enough to be applicable to a variety of tasks [38,39,40,41]. However, our experiments show that, regarding our entanglement-structured design of the new tensorized LSTM architecture, LSTM-MERA performs even better than LSTM-MPS in general without increasing the number of parameters. This finding leads to another interesting result. We conjecture that not only should tensorization be introduced, but the tensor’s EE has to scale with the system size as well; hence, MERA is more efficient than MPS in learning chaos.

2. Recurrent Architecture and Tensorization

2.1. Formalism of LSTM Architecture

The formalism starts from an operator-theoretical perspective by defining two general types of real operators, $\mathcal{W}$ and $\sigma$, through which most NN architectures can be represented. $\mathcal{W}:\mathbb{X}\to \mathbb{G}$ is simply a linear operator, but $\sigma :\mathbb{G}\to \mathbb{G}$ is a nonlinear operator that $\sigma \left(G\right)=(\sigma \circ G)\in \mathbb{G}$ given $G\in \mathbb{G}$ where ∘ stands for the entry-wise operator product. All double-struck symbols ($\mathbb{X},\mathbb{G},\cdots$) used in this context are general real vector spaces considered to be of covariant type, as $\mathcal{W}$ can be interpreted as a linear-map-induced 2-contravariant bilinear form. Next, a state propagation function (i.e,. a gate) $g(x,y,\cdots ;\mathcal{W})=\sigma \left(\mathcal{W}\right(x\oplus y\oplus \cdots \left)\right)$ is introduced, where $x\oplus y\oplus \cdots$ stands for the tensor direct sum of real vectors $x,y,\cdots$. Following the formalism, an LSTM architecture can be expressed as follows:

$$\begin{array}{ccc}\hfill s_t& =& g(1,x_{t-1},s_{t-1};{\mathcal{W}}_o)\circ \sigma \left(c_t\right),x_t=g(1,s_t;{\mathcal{W}}_x),\hfill \\ \hfill c_t& =& g(1,x_{t-1},s_{t-1};{\mathcal{W}}_f)\circ c_{t-1}\hfill \\ & +& g(1,x_{t-1},s_{t-1};{\mathcal{W}}_i)\circ g(1,x_{t-1},s_{t-1};{\mathcal{W}}_m),\hfill \end{array}$$

(2)

where the four gates controlled by ${\mathcal{W}}_i$, ${\mathcal{W}}_m$, ${\mathcal{W}}_f$, and ${\mathcal{W}}_o$ are the input, memory, forget, and output gates. The state $s_t$ and the cell state $c_t$ are h-dimensional covectors, whereas the input $x_t$ is a d-dimensional covector (Figure 1). Therefore, ${\mathcal{W}}_i$ (as well as ${\mathcal{W}}_m$, ${\mathcal{W}}_f$, and ${\mathcal{W}}_o$) has a direct-sum contravariant realization as a matrix $W_i\in \mathbf{M}(h,1)\oplus \mathbf{M}(h,d)\oplus \mathbf{M}(h,h)$ that contains $h(1+d+h)$ free real parameters at most. During NN training, only these free parameters of the linear operators are learnable, whereas all $\sigma$ (i.e., activation functions) are fixed to be tanh, sigmoid, or other nonlinear functions. The cell state $c_t$ is designed to suffer less from the vanishing gradient problem and thus to capture long-term memory better, whereas $s_t$ tends to capture short-term dependence.

2.2. Tensorized State Propagation

Our tensorized LSTM architecture (Figure 1) is exactly based on Equation (2), from which the only change is:

$$s_t=g(1,x_{t-1},s_{t-1};{\mathcal{W}}_o)\circ g(\mathcal{T}\left(\sigma \left(c_t\right)\right);{\mathcal{W}}_{\mathcal{T}}).$$

(3)

$g(\mathcal{T}\left(\sigma \left(c_t\right)\right);{\mathcal{W}}_{\mathcal{T}})$ is coined a tensorized state propagation function, for which ${\mathcal{W}}_{\mathcal{T}}:\mathbb{T}\to \mathbb{G}$ acts on as a covariant tensor

$$\mathcal{T}\left(\sigma \left(c_t\right)\right)=\underset{l}{⨂}\left(1\oplus q_{t,l}\right)=\underset{l}{⨂}\left(1\oplus {\mathcal{W}}_l\left(\sigma \left(c_t\right)\right)\right).$$

(4)

Each ${\mathcal{W}}_l$ in Equation (4) maps from $\sigma \left(c_t\right)$ to a new covector $q_{t,l}\in \mathbb{Q}$. Here, $\mathbb{Q}$ is named a local q-space, as, by way of analogy, considered as encoding the local degree of freedom in quantum mechanics. $\mathbb{Q}$ can be extended to the complex number field if necessary. Mathematically, Equation (4) offers the possibility of directly constructing orthogonal polynomials up to order L from $\sigma \left(c_t\right)$ to build up nonlinear complexity. In fact, when L goes to infinity, $\mathbb{T}=\underset{L\to \infty }{lim}{(1\oplus \mathbb{Q})}^{\otimes L}=1\oplus \mathbb{Q}\oplus \mathbb{Q}\otimes \mathbb{Q}\oplus \cdots$ becomes a tensor algebra (up to a multiplicative coefficient), and $\mathcal{T}\left(\sigma \left(c_t\right)\right)$ admits any nonlinear smooth function of $\sigma \left(c_t\right)$.

We now realize Equation (4) by choosing L independent realizations, $W_l\in \mathbf{M}(P-1,h)$, $l=1,2,\cdots ,L$, which in total contain $L(P-1)h$ learnable parameters at most, each mapping $\sigma \left(c_t\right)\equiv tanh{c}_t$ to a ($P-1$)-dimensional covector $q_{t,l}$,

$$\left(\begin{array}{c}{\left[tanh{c}_t\right]}_1\\ {\left[tanh{c}_t\right]}_2\\ ⋮\\ {\left[tanh{c}_t\right]}_h\end{array}\right)\to \left(\begin{array}{cccc}\left(\begin{array}{c}1\\ {\left[q_t\right]}_{21}\\ {\left[q_t\right]}_{31}\\ ⋮\\ {\left[q_t\right]}_{P1}\end{array}\right)& \left(\begin{array}{c}1\\ {\left[q_t\right]}_{22}\\ {\left[q_t\right]}_{32}\\ ⋮\\ {\left[q_t\right]}_{P2}\end{array}\right)& \begin{array}{c}\cdots \\ \cdots \\ \cdots \\ \ddots \\ \cdots \end{array}& \left(\begin{array}{c}1\\ {\left[q_t\right]}_{2L}\\ {\left[q_t\right]}_{3L}\\ ⋮\\ {\left[q_t\right]}_{PL}\end{array}\right)\end{array}\right).$$

(5)

Following Equation (4), $\mathcal{T}(tanh{c}_t)$ is simply the tensor product of all column vectors on the right hand side of Equation (5).

From the realization of Equation (3), $W_{\mathcal{T}}\in \mathbf{M}(h,P^L)$ [Figure 2a], however, a problem of exponential explosion (a.k.a. the “curse of dimensionality”) arises. Treating $W_{\mathcal{T}}$ maximally by training all $h{P}^L$ learnable parameters is very computationally expensive, especially as L cannot be small because it governs the nonlinear complexity. To overcome this “curse of dimensionality”, tensor decomposition techniques have to be exploited [39] for the purpose of finding a much smaller subset $\mathbf{T}\subset \mathbf{M}(h,P^L)$ to which all possible $W_{\mathcal{T}}$ belong, without sacrificing any expressive power.

2.3. Many-Body Entanglement Structures

Below, we introduce the two many-body quantum state structures (MPS and MERA) as efficient low-order tensor decomposition techniques for representing $W_{\mathcal{T}}$.

2.3.1. MPS

As one of the most commonly used tensor decomposition techniques, MPS is also widely known as tensor-train decomposition [42] and takes the following form [Figure 2b]

$${\left[W_{\mathcal{T}}\right]}_{{\mu }_1\cdots {\mu }_L}^h=\sum _{\left\{\alpha \right\}}^{D_{\mathrm{II}}}{\left[w_0\right]}_{{\alpha }_1{\alpha }_{L+1}}^h\left({\left[w_1^{†}\right]}_{{\mu }_1}^{{\alpha }_1{\alpha }_2}{\left[w_2^{†}\right]}_{{\mu }_2}^{{\alpha }_2{\alpha }_3}\cdots {\left[w_L^{†}\right]}_{{\mu }_L}^{{\alpha }_L{\alpha }_{L+1}}\right)$$

in our model, where $w_1^{†},w_2^{†},\cdots ,w_L^{†}$ are learnable 3-tensors (the symbol ${}^{†}$ denoting that they are inverse isometries [35]). $D_{\mathrm{II}}$ is an artificial dimension (the same for all $\alpha$). $w_0$ is no more than a linear transformation that collects the boundary terms and maintains symmetry. The above notations are used for consistency with quantum theory [35] and the following MERA representation.

2.3.2. MERA

The best way to explain MERA is using graphical tools, e.g., tensor networks [35]. MERA differs from MPS in its hierarchical tree structure: within each level $\{\mathrm{I},\mathrm{II},\cdots \}$, the structure contains a layer of 4-tensor disentanglers of dimensions $\{D_{\mathrm{I}}^4,D_{\mathrm{II}}^4,\cdots \}$, and then a layer of 3-tensor isometries of dimensions $\{D_{\mathrm{I}}^2\times D_{\mathrm{II}},D_{\mathrm{II}}^2\times D_{\mathrm{III}},\cdots \}$, of which details can be found in [36]. MERA is similar to the Tucker decomposition [43] but fundamentally different because of the existence of disentanglers, which smear the inhomogeneity of different tensor entries [36].

Figure 2c shows the reorganized version of MERA used in our model, where the storage of independent tensors is maximally compressed before they are multiplied with each other by tensor products, which allows more GPU acceleration during NN training.

2.3.3. Scaling Behavior of EE

Now we take advantage of an important measure of tensor complexity: the entanglement entropy (EE). Given an arbitrary tensor $W_{{\mu }_1\cdots {\mu }_L}$ of dimension $P^L$ and a cut l so that $1\le l\ll L$, the EE is defined in terms of the $\alpha$-Rényi entropy [35],

$$S_{\alpha }\left(l\right)\equiv S_{\alpha }\left(W\left(l\right)\right)=\frac{1}{1-\alpha }log\frac{{\sum }_{i=1}^{P^l}{\sigma }_i^{\alpha }\left(W\left(l\right)\right)}{{\left({\sum }_{i=1}^{P^l}{\sigma }_i\left(W\left(l\right)\right)\right)}^{\alpha }},$$

(6)

assuming $\alpha \ge 1$. The Shannon entropy is recovered under $\alpha \to 1$. ${\sigma }_i\left(W\left(l\right)\right)$ in Equation (6) is the i-th singular value of matrix $W\left(l\right)=W_{({\mu }_1\times \cdots \times {\mu }_l),({\mu }_{l+1}\times \cdots \times {\mu }_L)}$, matricized from $W_{{\mu }_1\cdots {\mu }_L}$. How $S\left(l\right)$ scales with l determines how much redundancy exists in $W_{{\mu }_1\cdots {\mu }_L}$, which in turn reveals how efficient at most a tensor decomposition technique can be. For one-dimensional gapped low-energy quantum states (ground states), their EE is saturated even as l increases, i.e., $S_{\alpha }\left(l\right)=\mathsf{\Theta }\left(1\right)$. Thus, their low-entanglement characteristics can be efficiently represented via MPS, of which the EE does not scale with l either and is bounded by $S_{\alpha }\left(l\right)\le S_1\left(l\right)\le 2log{D}_{\mathrm{II}}$ [35]. By contrast, a non-trivial scaling behavior $S_{\alpha }\left(l\right)=\mathsf{\Theta }(logl)$ corresponds to gapless low-energy states and can only be efficiently represented by MERA, of which $S_{\alpha }\left(l\right)\le S_1\left(l\right)\le C+{\sum }_{\mathrm{level}=1}^{{log}_{2}l}log{D}_{\mathrm{level}}\approx C+C^{\prime }logl$ scales logarithmically [36]. The bounds of both MPS and MERA have also been proven to be tight [35,36].

The different EE scaling behaviors of MERA and MPS have hence provided an apparent geometric advantage of MERA, i.e., its quasi-two-dimensional structure [Figure 2c], enlarging which will increase not only the width but also the depth of NN as the number of applicable levels scales logarithmically with L, offering even more power for model generalization on the already-inherited LSTM architecture [11]. Such an advantage is further confirmed by Equation (9) and then in Section 4.1, in which tensorized LSTMs with the two different representations LSTM-MPS and LSTM-MERA are tested.

3. Theoretical Analysis

3.1. Expressive Power

First, we prove the following theorem that links the variations of $c_t$ and $s_t$:

Theorem 1.

Given an LSTM architecture [Equation (2)], to which the input is a chaotic dynamical system $x_t$, characterized by a matrix λ of which the spectrum is the Lyapunov exponent(s), so that any variation $\delta x_t$ propagates exponentially [Equation (1)], then, up to the first order (i.e., $\delta x_{t-1}$),

$$\begin{array}{c}\hfill \left|\delta s_t\right|\ge C{e}^{\lambda }\left|\delta c_t\right|,\end{array}$$

(7)

where $C\propto 1/{\Vert \mathcal{W}\Vert }_{\infty }^2$ and ${\Vert ·\Vert }_{p=\infty }$ is the operator norm.

Proof. 

From Equation (2), one has $\delta x_t=(\partial g(1,s_t;{\mathcal{W}}_x)/\partial s_t)\delta s_t$ where the first-order derivative is bounded by $\Vert \partial g(1,s_t;{\mathcal{W}}_x)/\partial s_t{\Vert }_{\infty }\le \Vert {\mathcal{W}}_x{\Vert }_{\infty }\Vert {\sigma }^{\prime }{\Vert }_{L_{\mu }^{\infty }}\le {\Vert {\mathcal{W}}_x\Vert }_{\infty }$, since the derivative of the active function supported on $(-\infty ,\infty )$ satisfies $\Vert {\sigma }^{\prime }{\Vert }_{L_{\mu }^{\infty }}\equiv {\Vert 1/{cosh}^2\Vert }_{L_{\mu }^{\infty }}\le 1$. On the other hand, one has

$$\begin{array}{c}\hfill \delta c_t=\left[c_{t-1}\circ \partial g(1,x_{t-1},s_{t-1};{\mathcal{W}}_f)/\partial x_{t-1}+\partial \left(g(1,x_{t-1},s_{t-1};{\mathcal{W}}_i)\right.\right.\\ \hfill \left.\left.\circ g(1,x_{t-1},s_{t-1};{\mathcal{W}}_m)\right)/\partial x_{t-1}\right]\delta x_{t-1}+O\left(\delta x_{t-2}\right)+\cdots ,\end{array}$$

and thus

$$|\delta c_t|\le \Vert {\mathcal{W}}_f{\Vert }_{\infty }\left|c_{t-1}\right|\circ \delta |x_{t-1}|+\left(\Vert {\mathcal{W}}_i{\Vert }_{\infty }+{\Vert {\mathcal{W}}_m\Vert }_{\infty }\right)\delta \left|x_{t-1}\right|,$$

which yields Equation (7), where w.l.o.g. all linear maps are assumed to be around the same magnitude $\sim {\Vert \mathcal{W}\Vert }_{\infty }$. Note that $\left|c_{t-1}\right|$ is also bounded because $\left|g\right(1,x_{t-1},s_{t-1};{\mathcal{W}}_f\left)\right|\le 1$, which means $c_t$ is stationary. □

Equation (7) suggests that the state propagation from $c_t$ to $s_t$ carries the chaotic behavior. In fact, to preserve the long-term memorization in LSTM, $c_t$ has to depend on $c_{t-1}$ with a linear behavior and thus cannot carry chaos itself. This is further experimentally verified in Section 4.3. Nevertheless, note that Equation (1) is a necessary condition of chaos, not a sufficient condition.

Now, we look into the expressive power of our introduced tensorized state propagation function, ${\mathcal{W}}_{\mathcal{T}}\mathcal{T}\left(\sigma \left(c_t\right)\right)$. One of the advantages of tensorizing the state propagation function in the form of Equation (4) is the well-behaved polynomial space constructed by the tensor product, by virtue of which the approximation of ${\mathcal{W}}_{\mathcal{T}}\mathcal{T}$ to any (k-Sobolev) function f can always be bounded, as proven by the following theorem:

Theorem 2.

Let $f\in H_{\mu }^k\left(\mathsf{\Lambda }\right)$ be a target function living in the k-Sobolev space, $H_{\mu }^k\left(\mathsf{\Lambda }\right)=\left\{f\in L_{\mu }^2\left(\mathsf{\Lambda }\right)\right|\left.{\sum }_{\left|\mathbf{i}\right|\le k}\Vert {\partial }^{\left(\mathbf{i}\right)}{f\Vert }_{L_{\mu }^2\left(\mathsf{\Lambda }\right)}<\infty \right\}$, where ${\partial }^{\left(\mathbf{i}\right)}f\in L_{\mu }^2\left(\mathsf{\Lambda }\right)$ is the $\mathbf{i}$-th weak derivative of f, up to order $k\ge 0$, square-integrable on support $\mathsf{\Lambda }={(-1,1)}^h$ with measure μ. ${\mathcal{W}}_{\mathcal{T}}\mathcal{T}\left(\sigma \left(c_t\right)\right)$ can approximate $f\left(\sigma \left(c_t\right)\right)$ with an $L_{\mu }^2\left(\mathsf{\Lambda }\right)$ error at most

$$\Vert f-{\mathcal{W}}_{\mathcal{T}}{\mathcal{T}\Vert }_{L_{\mu }^2\left(\mathsf{\Lambda }\right)}\le Cmin{(L,\lfloor L(P-1)/h\rfloor )}^{-k}{\Vert f\Vert }_{H_{\mu }^k\left(\mathsf{\Lambda }\right)}$$

(8)

provided that $(h-1)h{P}^L\ge (h^{1+min(L,\lfloor L(P-1)/h\rfloor )}-1)$. ${\Vert f\Vert }_{H_{\mu }^k\left(\mathsf{\Lambda }\right)}={\sum }_{\left|\mathbf{i}\right|\le k}\Vert {\partial }^{\left(\mathbf{i}\right)}{f\Vert }_{L_{\mu }^2\left(\mathsf{\Lambda }\right)}$ is the Sobolev norm and C is a finite constant.

Proof. 

The Hölder-continuous spectral convergence theorem [44] states that $\Vert f-{\mathrm{P}}_N{f\Vert }_{L_{\mu }^2\left(\mathsf{\Lambda }\right)}\le C{N}^{-k}{\Vert f\Vert }_{H_{\mu }^k\left(\mathsf{\Lambda }\right)}$, in which ${\mathrm{P}}_N:L_{\mu }^2\left(\mathsf{\Lambda }\right)\to {\mathbb{P}}_N$ is an orthogonal projection that maps f to ${\mathrm{P}}_{N}f$. $\sigma \left(c\right(t\left)\right)\in \mathsf{\Lambda }$ is guaranteed as $\sigma \equiv tanh$. The Sobolev space ${\mathbb{P}}_N\subset L_{\mu }^2\left(\mathsf{\Lambda }\right)$ is spanned by polynomials of a degree of at most N. Next, note that in the realization of $\mathcal{T}\left(\sigma \left(c_t\right)\right)$ each $W_l$ is independent [Equation (4)], and thus ${\mathbb{P}}_N=\mathrm{span}\left(\mathcal{T}\left(\sigma \left(c_t\right)\right)\right)$ is possible, where N is determined by L, P, and h. When $P-1\ge h$, the maximum polynomial order is guaranteed L; when $P-1<h$, $dim\left\{\mathbb{Q}\right\}<dim\left\{\mathbb{G}\right\}$, and hence $\mathcal{T}\left(\sigma \left(c_t\right)\right)$ can only fully cover a polynomial order of up to $\lfloor L(P-1)/h\rfloor$. Finally, Equation (8) is proven based on the fact that ${\mathcal{W}}_{\mathcal{T}}\mathcal{T}$ maximally admits ${\mathrm{P}}_{N}f$ as long as $h{P}^L\ge {\sum }_{i=0}^N{h}^i=(h^{N+1}-1)/(h-1)$, the latter of which is the size of the maximum orthogonal polynomial basis admitted by ${\mathbb{P}}_N$. □

Equation (8) can be used to estimate how L scales with the chaos the dynamical system possesses. In particular, Equation (7) suggests that ${\partial }^{\left(1\right)}f\sim e^{\lambda \Delta t}$, where $\Delta t$ is the actual time difference between consecutive steps. Therefore, to persevere the error bound [Equation (8)] one at least expects that $L^{-1}\sim e^{-\lambda \Delta t}$, i.e., L has to increase exponentially with respect to $\lambda \Delta t$. To achieve this, tensorization is undoubtedly the most efficient approach, especially when $\Delta t$ is large.

3.2. Worst-Case Bound by EE

The above analysis offers an intuitive bound on the expressive power of ${\mathcal{W}}_{\mathcal{T}}\mathcal{T}$. Unfortunately, Equation (8) is valid only when all $h{P}^L$ degrees of freedom of ${\mathcal{W}}_{\mathcal{T}}$ are independent. A low-order tensor decomposition may therefore impact the expressive power.

Below, we compare the two different entanglement structures, MPS and MERA, which have major differences in their EE scaling behaviors. We proceed via the following theorem, relating the tensor approximation error to entanglement scaling:

Theorem 3.

Given a tensor ${\left[W_{\mathcal{T}}\right]}_{{\mu }_1\cdots {\mu }_L}$ and its tensor decomposition ${\overline{W}}_{\mathcal{T}}$, the worst-case p-norm ($p\ge 1$) approximation error is bounded from below by

$$\begin{array}{cc}\hfill & \underset{\left\{{\overline{W}}_{\mathcal{T}}\right\}}{min}\Vert W_{\mathcal{T}}-{\overline{W}}_{\mathcal{T}}{\Vert }_p=\underset{\left\{{\overline{W}}_{\mathcal{T}}\right\}}{min}\underset{l\ge 1}{max}{\Vert W_{\mathcal{T}}\left(l\right)-{\overline{W}}_{\mathcal{T}}\left(l\right)\Vert }_p\hfill \\ \hfill \ge & \underset{\left\{{\overline{W}}_{\mathcal{T}}\right\}}{min}\underset{l\ge 1}{max}\left|e^{\frac{1-p}{p}{S}_p\left(W_{\mathcal{T}}\left(l\right)\right)}\Vert W_{\mathcal{T}}{\Vert }_1-e^{\frac{1-p}{p}{S}_p\left({\overline{W}}_{\mathcal{T}}\left(l\right)\right)}{\Vert {\overline{W}}_{\mathcal{T}}\Vert }_1\right|,\hfill \end{array}$$

(9)

where $S_{\alpha \equiv p}\left(W\left(l\right)\right)$ is the α-Rényi entropy [Equation (6)].

Proof. 

Equation (9) is easily proven by noting the Minkowski inequality ${\Vert A+B\Vert }_p\le {\Vert A\Vert }_p+{\Vert B\Vert }_p$ and that $(1-\alpha )S_{\alpha }\left(l\right)=\alpha log\Vert W_{\mathcal{T}}{\left(l\right)\Vert }_{\alpha }-\alpha log{\Vert W_{\mathcal{T}}\left(l\right)\Vert }_1$ when $\alpha \equiv p\ge 1$ [Equation (6)]. □

The worst-case bound [Equation (9)] is optimized whenever $S_p\left({\overline{W}}_{\mathcal{T}}\left(l\right)\right)$ scales the same way as $S_p\left(W_{\mathcal{T}}\left(l\right)\right)$ does. Assuming $S_p\left(W_{\mathcal{T}}\left(l\right)\right)=C+C^{\prime }logl$, then an MPS-type ${\overline{W}}_{\mathcal{T}}$ cannot efficiently approximate $W_{\mathcal{T}}$ unless $D_{\mathrm{II}}$ increases with $logl$ too, from which the total number of free parameters $\sim PL{D}_{\mathrm{II}}^2$ [Figure 2b], however, becomes unbounded. By contrast, a MERA-type ${\overline{W}}_{\mathcal{T}}$ matches the scaling, by which the total number of free parameters $\sim (D^4+D^3)L$ (where $D\equiv D_{\mathrm{II}},\cdots =exp{C}^{\prime }$) is efficient enough for any worst-case l.

It is unknown how quantitatively the failure to approximate $W_{\mathcal{T}}$ may impact the expressive power given in Equation (8). The disappearance of the worst-case bound in Equation (9) is a necessary condition for Equation (8) to be valid.

4. Results

We investigated the accuracy of LSTM-MERA and its generalization ability on different chaotic time series datasets by evaluating the root mean squared error (RMSE) of its one-step-ahead predictions against target values. The benchmark for comparison was chosen to be a vanilla LSTM, of which the hidden dimension h was arbitrarily chosen in advance. LSTM-MERA (and other architectures if present) was built upon the benchmark.

Each time series dataset for training/testing consisted of a set of $N_X$ time series, $\left\{X^i\right|i=1,2,\cdots ,N_X\}$. Each time series $X^i=\left\{x_t^i\right|t\in T^i\}$ is of fixed length $|T^i|=\mathrm{input}\mathrm{steps}+1$ so that all but the last step of $X^i$ were inputs, whereas the last step was the one-step-ahead target to be predicted. The dataset $\left\{X^i\right\}$ was divided into two subsets—one for testing and one for training, which was further randomly split into a plain training set and a validation set by $80\%:20\%$. Complete details are given in Appendix C.

All models were trained using Mathematica 12.0 on its NN infrastructure, Apache MXNet, using an ADAM optimizer with ${\beta }_1=0.9$, ${\beta }_2=0.999$, and $ϵ={10}^{-5}$. The learning rate $={10}^{-2}$ and batch size $=64$ were chosen a priori. The NN parameters producing the lowest validation loss during the entire training process were accepted.

4.1. Comparison of LSTM-Based Architectures

When evaluating the advantages of LSTM-MERA, a controlled comparison is essential to confirm that the architecture of LSTM-MERA is inherently better than that of other architectures, not just because the increase of the number of free and learnable parameters (even though more parameters do not necessarily mean more learning power). Here, we studied different architectures (Figure 3) that were all built upon the LSTM benchmark and shared nearly the same number of parameters (param. #). A “wider” LSTM was simply built by increasing h. A “deeper” LSTM was built by stacking two LSTM units as one unit. In particular, LSTM-MPS and LSTM-MERA were built and compared.

4.1.1. Lorenz System

Figure 3a describes the forecasting task on the Lorenz system and shows the training results of the LSTM-based models. $\Delta t=0.5$ was chosen for discretization, which was large enough that the resultant time series hardly exhibited any pattern without the help of a phase line [Figure 3a, input 1–8].

In general, non-tensorized LSTM models performed worse than tensorized LSTM models. After the number of free parameters increased from 332 (benchmark) to $668\pm 28$, both the “wider” and “deeper” LSTMs showed signs of overfitting. The “deeper” LSTM yielded lower RMSE than the “wider” LSTM, confirming the common sense that a deep NN is more suitable for generalization than a wide NN [45]. Both LSTM-MPS and LSTM-MERA yielded better RMSE and showed no sign of overfitting. However, LSTM-MERA was more powerful, showing an improvement of ∼25% over LSTM-MPS in RMSE [Figure 3a].

4.1.2. Logistic Map

Figure 3b describes a specific forecasting task on the simplest one-dimensional discrete-time map—the logistic map: predicting the target given only a three-step-behind input. Different LSTM models yielded very different results when learning this complex task. After the number of free parameters increased from 35 (benchmark) to $1142\pm 89$, all LSTM models yielded lower RMSE than the benchmark. Only LSTM-MERA was able to reach a much lower RMSE (presumably a global minimum) with a remarkable improvement of ∼94% over LSTM-MPS [Figure 3b]. We infer that the local minima reached by the other LSTM models might correspond to the infinite numbers of unstable quasi-periodic cycles in the chaotic phases. In fact, as shown in Figure 3b, Prediction 3, the benchmark fit the target better than LSTM-MERA for this specific example of a quasi-period-2 cycle. However, LSTM-MERA learned the full chaotic behavior and thus performed much better on general examples.

The learning process for the logistic map task was indeed very random, and different realizations yielded very different results. In many realizations, non-tensorized LSTM models did not even learn any patterns at all. By contrast, tensorized LSTM models were more stable in learning.

4.2. Comparison with Statistical/ML Models

We compared LSTM-MERA with more general models, including traditional statisical and ML models, including RNN-based architectures (Figure 4). Specifically, we looked into HOT-RNN/LSTM, which is also claimed to be able to learn chaotic dynamics (e.g., the Lorenz system) through tensorization [28]. Furthermore, for each model we fed its one-step-ahead predictions back so as to make predictions for the second step, and kept feeding back and so on. In theory, the prediction error at the t-th step should increase exponentially with t for chaotic dynamics [Equation (1)].

Gauss Iterated Map

We tested the one-step-ahead learning task on the Gauss “cubed” map on plain HO-RNN/LSTM [15] and its tensorized version HOT-RNN/LSTM [28]. The explicit “history” length was chosen to be equal to our physical Length L. The tensor-train ranks were all chosen to be equal to $D_{\mathrm{II}}$, as when we built the MPS structure in LSTM-MPS.

Figure 4 shows that neither HO-RNN nor HO-LSTM performed better than the benchmark, suggesting that introducing explicit non-Markovian dependence (Appendix B.1) is not helpful for capturing chaotic dynamics where the existing nonlinear complexity is never long-term. HOT-LSTM was better than the benchmark because of its MPS structure, suggesting that tensorization, on the other hand, is indeed helpful for forecasting chaos. LSTM-MERA was still the best, with an improvement of $\sim 88\%$ over the benchmark. Interestingly, the benchmark itself as a vanilla LSTM was already much better than plain RNN architectures (HO-/HOT-RNN).

The learning task was next tested on fully connected deep NN architectures of depth $\le 8$ (equal to the input steps). At each depth three units were connected in series: a linear layer, a scaled exponential linear unit, and a dropout layer. Hyperparameters were determined by means of an optimal search. The best model having the lowest validation loss consisted of $\mathrm{17,950}$ free parameters. The task was also tested on GBT of maximum depth $=8$, as well as on the ARMA family (ARMA, ARIMA, FARIMA, and SARIMA), among which the best statistical model selected by Kalman filtering was $\mathrm{ARMA}(3,4)$.

With enough parameters, the deep NN became the second best (Figure 4). All RMSE values increased when making longer-step-ahead predictions, and for the four-step-ahead task the deep NN and LSTM-MERA were the only models that did not overfit and still performed better than the statistical model, ARMA, which made no learning progress but only trivial predictions.

4.3. Comparison with LSTM-MERA Alternatives

Here we tested the ability of LSTM-MERA in the learning of short-term nonlinear complexity by changing its NN topology (Figure 5). We expected to see that, to achieve the best performance, our tensorization (dashed rectangle in Figure 1) should indeed act on the state propagation path $c_t\to s_t$, not on $s_{t-1}\to s_t$ or $c_{t-1}\to c_t$.

Thomas’ Cyclically Symmetric System

We investigated different LSTM-MERA alternatives on Thomas’ cyclically symmetric system (Figure 5) in order to see if the short-term complexity could still be efficiently learned. The embedded layers, in addition to being located at Site A (the proper NN topology of LSTM-MERA), were also located alternatively at Site B, C or D for comparison. The benchmark was a vanilla LSTM with no embedded layers.

As expected, the lowest RMSE was produced by the proper LSTM-MERA but not its alternatives (Figure 5). The improvement of the proper LSTM-MERA over the benchmark was ∼60%. Interestingly, two alternatives (Site B, Site C) performed barely better than the benchmark even with more free learnable parameters. In fact, in the case in which the state propagation path $c_{t-1}\to c_t$ is tensorized (Site B), the long-term gradient propagation along cell states is interfered and the performance of LSTM is deterred; when the path $s_{t-1}\to s_t$ is tensorized (Site C), the improvement is the same as just on a plain RNN and is thus also limited. Hence, proper LSTM-MERA NN topology is critical for improving the performance of learning short-term complexity.

4.4. Generalization and Parameter Dependence of LSTM-MERA

The inherent advantage of LSTM-MERA and its ability to learn chaos have been shown. Hereafter investigated are its parameter dependence, as well as its generalization ability (Figure 6). Each following model (benchmark versus LSTM-MERA) was sufficiently trained through the same number of epochs so that it could reach the lowest stable RMSE. In-between check points were chosen during training, in which models were tested a posteriori on the test data to confirm that an RMSE minimum had eventually been reached.

4.4.1. Rössler System

In theory, a chaotic time series of larger $\Delta t$ should be harder to learn [Equation (1)]. This is confirmed in Figure 6a, in which a larger $\Delta t$ corresponds to a larger RMSE for both models. The greatest improvement of LSTM-MERA over the benchmark was ∼76%, observed at $\Delta t=5$. The improvement was less when $\Delta t$ increased, possibly because the time series became too random to preserve any feasible pattern even for LSTM-MERA. The improvement was also less when $\Delta t$ was small, as the time series was smooth enough and the first-order (linear) time-dependence predominated, which a vanilla LSTM could also learn.

4.4.2. Hénon Map

In view of the fact that the time-dependence is second-order [Figure 6b], there was no explicit and exact dependence between the input and target in the time series dataset. Different input steps were chosen for comparison. When input steps $=1$, there was no sufficient information to be learned other than a linear dependence between the input and target, and thus both the benchmark and LSTM-MERA performed the same [Figure 6b]. When input steps $>1$, however, the time-dependence could be learned implicitly and “bidirectionally” given enough history in length. LSTM-MERA constantly exhibited an average improvement of $45.3\%$, the fluctuation of which was mostly due to the learning instability not of LSTM-MERA but of the benchmark.

4.4.3. Duffing Oscillator System

Based on Figure 6d, it was clearly observed that larger L yielded better RMSE values. The improvement related to L was significant. This result is not unexpected, since L determines the depth of the MERA structure, with a larger depth corresponding to better generalization ability.

4.4.4. Chirikov Standard Map

As Figure 6d shows, by choosing different P, the greatest improvement of LSTM-MERA over the benchmark was ∼56%, observed at $P=8$. In general, there was no strong dependence on P.

4.4.5. Real-World Data: Weather Forecasting

The advantage of LSTM-MERA was also tested on real-world weather forecasting tasks [Figure 6e,f]. Unlike for the synthetic time series, here we removed the first-layer translational symmetry [Equation (A1)] previously imposed on LSTM-MERA so that presumed non-stationarity in real-world time series could be better addressed. To perform practical multi-step forecasting, we kept the one-step-ahead prediction architecture of LSTM, yet regrouped the original time series by choosing different prediction window lengths (Appendix C.3).

The improvement of LSTM-MERA over the benchmark was less significant. The average improvement was ∼3.0%, whereas the greatest improvement was ∼6.3%, considering that the prediction window length was small and reflecting that LSTM-MERA is better at capturing short-term nonlinear complexity rather than long-term non-Markovianity. Note that, in the second dataset [Figure 6f], we deliberately used a very small number (=128) of training data to test the overfitting resistibility of the models. Interestingly, LSTM-MERA did not generally perform worse than vanilla LSTM even with more parameters, probably due to the deep architecture of LSTM-MERA.

5. Discussion and Conclusions

The limitations of our model mostly come from the fact that it is only better than traditional LSTM at capturing short-term nonlinearity but not long-term non-Markovianity, and thus its improvement on long-term tasks such as sequence prediction would be limited. That being said, the advantages of tensorizing state propagation in LSTM are evident, including: (1) Tensorization is the most suitable method for the forecasting of nonlinear chaos since nonlinear terms are treated explicitly and weighted equally by polynomials. (2) Theoretical analysis is conductible since an orthogonal polynomial basis on k-Sobolev space is always available. (3) Tensor decomposition techniques (in particular, from quantum physics) are applicable, which in turn can identify chaos from a different perspective, i.e., tensor complexity (tensor ranks, entropies, etc.).

Our tensorized LSTM model not only offers a general and efficient approach for capturing chaos—as demonstrated by both theoretical analysis and experimental results, showing great potential in unraveling real-world time series—but also brings out a fundamental question of how tensor complexity is related to the learnability of chaos. Our conjecture that a tensor complexity of $S_{\alpha }\left(l\right)=\mathsf{\Theta }(logl)$ in terms of $\alpha$-Rényi entropy [Equation (6)] generally performs better than $S_{\alpha }\left(l\right)=\mathsf{\Theta }\left(1\right)$ in chaotic time series forecasting will be further investigated and formalized in the near future.