Information Geometric Theory in the Prediction of Abrupt Changes in System Dynamics

Abstract

Detection and measurement of abrupt changes in a process can provide us with important tools for decision making in systems management. In particular, it can be utilised to predict the onset of a sudden event such as a rare, extreme event which causes the abrupt dynamical change in the system. Here, we investigate the prediction capability of information theory by focusing on how sensitive information-geometric theory (information length diagnostics) and entropy-based information theoretical method (information flow) are to abrupt changes. To this end, we utilise a non-autonomous Kramer equation by including a sudden perturbation to the system to mimic the onset of a sudden event and calculate time-dependent probability density functions (PDFs) and various statistical quantities with the help of numerical simulations. We show that information length diagnostics predict the onset of a sudden event better than the information flow. Furthermore, it is explicitly shown that the information flow like any other entropy-based measures has limitations in measuring perturbations which do not affect entropy.

1. Introduction

Even if occurring very infrequently, rare or extreme events can mediate large transport with significant impact. Examples would include the sudden outbreak of devastating infectious diseases, solar flares, extreme weather conditions, flood, forest fire, sudden stock market crash, flow sensor failure, bursty gene expression and protein productions. The resulting large transports can be either beneficial (e.g., promoting mixing and air circulations by atmospheric jets or removing toxins) or harmful. For instances, tornadoes cause a lot of damage; in magnetic fusion, plasma confinement is hampered by intermittent transport of particles and energy from hot plasma core to the colder plasma boundaries.

Given the damage that these events can cause, finding good statistical methods to predict their sudden onset, or abrupt changes in the system dynamics is a critical issue. For instance, there are different types of plasma disruptions in fusion plasmas [1] and the current guidance for the minimum required warning time for successful disruption mitigation on ITER is about 30 ms [2]. Increasing the warning time by the early detection of a sudden event will greatly help ensuring a sufficient time for a control strategy to minimise harmful effects.

Obviously, the whole mark of the onset of a sudden event is an abrupt dynamical change in the system or data over time—time-variability/large fluctuation, whose proper description requires non-stationary statistical measures such as time-dependent probability density functions (PDFs). By using time-dependent PDFs, we can quantify how the “information” unfolds in time through information geometry. The latter refers to the application of the techniques of differential geometry in probability and statistics by using differential geometry to define the metric [3,4,5,6] (a notion of length). The main purpose of this paper is to examine the capability of the information-geometric theory proposed in a series of recent works [7,8,9,10,11,12] in predicting the onset of a sudden event and compare it with one of the entropy-based information theoretical measures [13,14,15].

In nutshell, the information length [7,8] measures the evolution of a system in terms of a dimensionless distance which represents the total number of different statistical states that are accessed by the system (see Section 2.2). The larger time-variability, the more abrupt change in the information length; in a statistically stationary state, the information length does not change in time. In fact, the recent work [6] has demonstrated the capability of the information length in the early prediction of transitions in fusion plasmas.

In this paper, we mimic the onset of a sudden event by including a sudden perturbation to the system and calculate time-dependent PDFs and various statistical quantities including information length and one of the entropy-based information-theoretical measure (information flow) [16,17]. The latter measures the directional information flow between two variables. This is more sensitive than mutual information which measures the correlation between the variables. The point we want to make is that this information flow like any other entropy-based measures depends solely on entropy, and thus it cannot pick up the onset of a sudden event which does not affect entropy, for instance, such as the mean value (recall, the entropy is independent of the local arrangement of the probability [3] as well as the mean value).

We should note that there are many other information theoretical measures [3,13,14,15,17,18,19,20,21,22,23,24,25,26] that have been used to understand different aspects of complexity, emergent behaviours, etc in non-equilibrium systems. However, the main purpose of this paper is not to provide an exhaustive exploration of these methods, but to point out the possible limitation of the entropy-based information measurements in predicting sudden events. Additionally, our intention is not on modelling the appearance of rare, extreme events (that are nonlinear, non-Gaussian) themselves, but on testing the predictability of information theoretical measures on the onset of such sudden events.

Specifically, to gain a key insight, we utilise an analytically solvable model—a non-autonomous Kramers equation (for the two variables, $x_1$ and $x_2$)—which enables us to derive exact PDFs and analytical expressions for various statistical measures including entropy, information length and information flow, which are then simulated for a wide range of different parameters. This model is the generalisation of the Kramers equation in [27] where non-autonomy is introduced by an impulse. The latter is included either in the strength of stochastic noise or by an external impulse input which models a sudden perturbation to the system. Examples are shown in Figure 1; panel (a) shows the phase portrait of $x_1$ and $x_2$ without any impulse, where blue dots are generated by sample stochastic simulations using the Cholesky decomposition [28]. Panel (b) shows the case where an impulse causes the perturbation in the covariance matrix $\mathsf{\Sigma }$ while panel (c) is the case where the sudden perturbations affect both covariance matrix $\mathsf{\Sigma }$ and the mean value $\langle \mathbf{x}\rangle$.

The paper is organised as follows: Section 2 introduces a non-autonomous linear system of equations and provides key statistical properties including the information length and information flow. In Section 3, we present the analysis of the non-autonomous Kramers equation and our main theoretical results, referring readers to Appendix A and Appendix B for the detailed steps involved in the derivations. In Section 4 (and also Appendix C), we present simulation results; Section 5 contains our concluding remarks.

To help readers, in the following, we summarise our notations. $\mathbb{R}$ is the set of real numbers. $\mathbf{x}\in {\mathbb{R}}^n$ represents a column vector $\mathbf{x}$ of real numbers of dimension n, $\mathbf{A}\in {\mathbb{R}}^{n\times n}$ represents a real matrix of dimension $n\times n$ (bold-face letters are used to represent vectors and matrices), $\mathbf{tr}(\mathbf{A})$ corresponds to the trace of the matrix $\mathbf{A}$. $\left|\mathbf{A}\right|$, ${\mathbf{A}}^T$ and ${\mathbf{A}}^{-1}$ are the determinant, transpose and inverse of matrix $\mathbf{A}$, respectively. ${\partial }_t$ is used for the partial derivative with respect to the variable t. Finally, the average of a random vector $\mathbf{x}$ is denoted by $\langle \mathbf{x}\rangle$, the angular brackets representing the average.

2. Preliminaries

In this section we introduce a non-autonomous linear system of equations and provide useful statistical properties including the information length and information flow.

2.1. Statistical Properties of Linear Non-Autonomous Stochastic Processes

A linear non-autonomous process is given by

$$\stackrel{·}{\mathbf{x}}(t)=\mathbf{A}\mathbf{x}(t)+\mathbf{B}u(t)+\mathsf{\Gamma }(t),$$

(1)

where $\mathbf{A}$ and $\mathbf{B}$ are $n\times n$ and $n\times 1$ constant real matrices, respectively; $u(t)$ is a (bounded smooth) external input, $\mathsf{\Gamma }\in {\mathbb{R}}^n$ is a Gaussian stochastic noise given by a n dimensional vector of $\delta$-correlated Gaussian noises ${\mathsf{\Gamma }}_i$ ($i=1,2,\dots n$), with the following statistical property

$$\langle {\mathsf{\Gamma }}_i(t)\rangle =0,\langle {\mathsf{\Gamma }}_i(t){\mathsf{\Gamma }}_j(t_1)\rangle =2D_{ij}(t)\delta (t-t_1),D_{ij}(t)=D_{ji}(t),\forall i,j=1,\dots ,n.$$

(2)

Here the angular brackets denote the average over ${\mathsf{\Gamma }}_i$. By assuming an initial Gaussian probability density function (PDF), the PDF remains Gaussian for all time. Thus, the following holds.

Proposition 1

(Joint probability). The value of the joint PDF of system (1) and (2) at any time t is given by

$$p(\mathbf{x};t)=\frac{1}{\sqrt{\det (2\pi \mathsf{\Sigma })}}{e}^{-\frac{1}{2}{\left(\mathbf{x}-\langle \mathbf{x}(t)\rangle \right)}^T{\mathsf{\Sigma }}^{-1}\left(\mathbf{x}-\langle \mathbf{x}(t)\rangle \right)},$$

(3)

where

$$\langle \mathbf{x}(t)\rangle =e^{\mathbf{A}t}\langle \mathbf{x}(0)\rangle +{\int }_0^t{e}^{\mathbf{A}(t-\tau )}\mathbf{B}u(\tau )d\tau ,$$

(4)

$$\mathsf{\Sigma }(t)=e^{\mathbf{A}t}〈\delta \mathbf{x}(0)\delta \mathbf{x}{(0)}^T〉e^{{\mathbf{A}}^{T}t}+2{\int }_0^t{e}^{\mathbf{A}(t-\tau )}\mathbf{D}{e}^{{\mathbf{A}}^T(t-\tau )}d\tau ,$$

(5)

and $\mathbf{D}\in {\mathbb{R}}^{n\times n}$ is the matrix with its elements $D_{ij}(t)$. Here, $\langle \mathbf{x}(t)\rangle$ is the mean value of $\mathbf{x}(t)$ while Σ is the covariance matrix.

We recall that in Proposition 1, the computation of the exponential matrix $e^{\mathbf{A}t}$ can be done by using the following result [29]

$$e^{\mathbf{A}t}={\mathcal{L}}^{-1}\left[{(s\mathbf{I}-\mathbf{A})}^{-1}\right].$$

(6)

Here, ${\mathcal{L}}^{-1}$ stands for the inverse Laplace transform of the complex variable s.

2.2. Information Length (IL)

Given its joint PDF $p(\mathbf{x};t)$, we define the information length (IL) $\mathcal{L}$ of system (1) as follows

$$\mathcal{L}(t)={\int }_0^{t}d{t}_1\sqrt{{\int }_{-\infty }^{\infty }d\mathbf{x}\frac{{\left[{\partial }_{t_1}p(\mathbf{x};t_1)\right]}^2}{p(\mathbf{x};t_1)}}={\int }_0^{t}d{t}_1\sqrt{\mathcal{E}},$$

(7)

where $\mathcal{E}={\int }_{-\infty }^{\infty }d\mathbf{x}\frac{{\left[{\partial }_{t_1}p(\mathbf{x};t_1)\right]}^2}{p(\mathbf{x};t_1)}$ is the square of the information velocity.

It is important to note that the dimension of $1/\sqrt{\mathcal{E}}\equiv \tau$ is time which gives a dynamical time unit for information change. Therefore, integrating $\sqrt{\mathcal{E}}$ between time 0 and t gives the total information change in that time interval. In other words, $\mathcal{L}$ quantifies the number of statistical different states that the system passes through in time from an initial $p(\mathbf{x};0)$ to a final $p(\mathbf{x};t)$ [7]. Note that $\tau$ was shown to provide a universal bound on the timescale of transient dynamical fluctuations, independent of the physical constraints on the stochastic dynamics or their function [30].

For the case of a linear stochastic process like (1), the following results can be used to obtain the value of IL.

Theorem 1

(Information Length [27]). The information length of the joint PDF of system (1) and (2) is given by

$$\mathcal{L}(t)\hspace{1em}=\hspace{1em}{\int }_0^{t}d{t}_1\sqrt{\mathcal{E}(t_1)},$$

(8)

$$\mathcal{E}(t_1)\hspace{1em}=\hspace{1em}\left({\partial }_{t_1}{\langle \mathbf{x}(t_1)\rangle }^T\right){\mathsf{\Sigma }}^{-1}\left({\partial }_{t_1}\langle \mathbf{x}(t_1)\rangle \right)+\frac{1}{2}\mathbf{tr}\left({({\mathsf{\Sigma }}^{-1}{\partial }_{t_1}\mathsf{\Sigma })}^2\right).$$

(9)

To calculate Equation (9), we recall that $\langle \mathbf{x}(t)\rangle$ and $\mathsf{\Sigma }(t)$ can be found from Equations (4) and (5), respectively. Specifically for ${\partial }_t\langle \mathbf{x}(t)\rangle$ we have

$${\partial }_t\langle \mathbf{x}(t)\rangle =\mathbf{A}\langle \mathbf{x}(t)\rangle +\mathbf{B}u(t).$$

(10)

Definition 1

(${\mathcal{E}}_m$ from marginal PDFs). For a n-th order linear process (1) with n random variables $\mathbf{x}\in {\mathbb{R}}^n={[x_1,x_2,\dots ,x_n]}^T$, it is useful to introduce ${\mathcal{E}}_m(t)$ as follows

$${\mathcal{E}}_m(t)=\sum _{i=1}^n{\mathcal{E}}_i(t)=\sum _{i=1}^n\frac{({\partial }_t\langle x_i\rangle {)}^2}{{\mathsf{\Sigma }}_{x_i{x}_i}}+\sum _{i=1}^n\frac{{({\partial }_t{\mathsf{\Sigma }}_{x_i{x}_i})}^2}{2{\mathsf{\Sigma }}_{x_i{x}_i}^2},$$

(11)

where ${\mathcal{E}}_i$ is calculated from a marginal PDF $p(x_i;t)$ of $x_i$. Note that $\mathcal{E}$ in Equation (9) is identical to ${\mathcal{E}}_m$ in Equation (11) when the n random variables are independent.

By utilising $\mathcal{E}={\mathcal{E}}_m$ for independent variables, we can introduce

$$\mathcal{E}(t)-{\mathcal{E}}_m(t),$$

(12)

as a measure of correlation (see Section 4.2.5).

2.3. Information Flow (IF)

Information flow (IF), or also usually called information transfer, is one of the useful information-theory measure that has been studied for causality (causation), uncertainty propagation and predictability transfer [22,23]. It also give us insight into the degree of interconnection among states of the system [16,17]. [16] considered a system of two Brownian particles with coordinates $\mathbf{x}=(x_1,x_2)$ interacting with two independent thermal baths at temperatures $T_1$ and $T_2$, respectively, subject to a potential $H(\mathbf{x})$, which are described by the Langevin equations

$$\begin{array}{ccc}\hfill 0& =& -{\partial }_{x_i}H(\mathbf{x})-{\mathsf{\Gamma }}_i{\stackrel{·}{x}}_i(t)+u_i(t)+{\eta }_i(t),\hfill \\ \hfill \langle {\eta }_i(t){\eta }_j(t_1)\rangle & =& 2{\mathsf{\Gamma }}_i{T}_i{\delta }_{ij}\delta (t-t_1),\hspace{1em}i,j=1,2,\hfill \end{array}$$

(13)

where ${\mathsf{\Gamma }}_i$ are the damping constants, which characterise the coupling of the particles to their baths/environments (with the temperature $T_i$), ${\delta }_{ij}$ is the Kronecker symbol and $u_i(t)$ is a bounded input. The information flows T from $2\to 1$ and $1\to 2$ are then given by (see [16]):

$$T_{2\to 1}=\frac{1}{{\mathsf{\Gamma }}_1}\int d\mathbf{x}P(\mathbf{x};t)\left[{\partial }_{x_1}H(\mathbf{x})+T_1{\partial }_{x_1}\ln P(\mathbf{x};t)\right]{\partial }_{x_1}\ln \frac{P_{x_1}(x_1;t)}{P(\mathbf{x};t)},$$

(14)

$$T_{1\to 2}=\frac{1}{{\mathsf{\Gamma }}_2}\int d\mathbf{x}P(\mathbf{x};t)\left[{\partial }_{x_2}H(\mathbf{x})+T_2{\partial }_{x_2}\ln P(\mathbf{x};t)\right]{\partial }_{x_2}\ln \frac{P_{x_2}(x_2;t)}{P(\mathbf{x};t)}.$$

(15)

To appreciate the physical meaning of IF, it is useful to recall that Equations (14) and (15)) can also be expressed in terms of entropy S or mutual information I (see Equations (17) and (23) in [16]), for instance, as follows:

$$T_{2\to 1}={\partial }_{t}S\left[x_1(t)\right]-{\partial }_{t_1}S\left[x_1(t+t_1)\right|x_2(t)]{|}_{t_1\to 0},$$

(16)

where $S\left[x_1(t+t_1)\right|x_2(t)]$ denotes the entropy of $x_1(t+t_1)$ at time $t+t_1$ conditioned by $x_2(t)$ at the earlier time t. From (16), we can see that IF represents the rate of change in the marginal entropy of $x_1$ minus that of the conditional entropy of $x_1$, $x_2$ being frozen between the time $(t,t+t_1)$. In other words, $T_{2\to 1}$ is that part of the entropy change of $x_1$ (between t and $t+t_1$), which exists due to fluctuations of $x_2$ [16].

Several important remarks are in order. First, IF $T_{2\to 1}$ and $T_{1\to 2}$ can be both negative and positive; a negative $T_{2\to 1}$ means that $x_2$ acts to reduce the marginal entropy of $x_1$ ($S_1$). This is different from the case of transfer entropy which is non-negative [31]. Second, the causality is inferred only from the absolute value of IF [23]. Third, the advantage of Equation (14) over Equation (16) would be that Equation (14) can be calculated using the equal-time joint/marginal PDFs without needing two-point time PDFs, which will be especially useful in the analysis of actual (experimental or observational) data. Finally, although it is not immediately clear from either Equations (15) or (16), we will show in Section 3 that IF depends only on the (equal-time) covariance matrix. This is similar to other causality measures such as the classical Granger causality [32] and transfer entropy [31] which quantify the improvement of the predictability of one variable by the knowledge of the value of another variable in the past and at present. This means these entropy-based measures do not pick up the onset of a sudden event which does not affect the covariance matrix (variance), for instance, such as the mean value.

3. Non-Autonomous Kramers Equation

To demonstrate how IF and IL can be used in the prediction of abrupt changes in system dynamics, we focus on the non-autonomous Kramers equation, as noted in Section 1. Recall that the original (autonomous) Kramers equation describes the Brownian motion in a potential, for instance, as a model for reaction kinetics [33]. By including a time-dependent external input $u(t)$, we generalise this to the following non-autonomous model for the two stochastic variables $\mathbf{x}={[x_1,x_2]}^T$

$$\stackrel{·}{\mathbf{x}}(t)=\left[\begin{array}{cc}0& 1\\ -{\omega }^2& -\gamma \end{array}\right]\mathbf{x}(t)+\left[\begin{array}{c}0\\ 1\end{array}\right]u(t)+\left[\begin{array}{c}0\\ \xi (t)\end{array}\right].$$

(17)

Here, $\xi$ is a short correlated Gaussian noise with a zero mean $\langle \xi \rangle =0$ and the strength D with the following property

$$\langle \xi (t)\xi (t^{\prime })\rangle =2D(t)\delta (t-t^{\prime }).$$

(18)

In this paper, we consider a time-dependent $D(t)$ to incorporate a sudden perturbation in D as follows

$$D(t)=D_0+\frac{b}{\left|a\right|\sqrt{\pi }}{e}^{-{\left(\frac{t-t_{1,0}}{a}\right)}^2}.$$

(19)

Here, the second term on RHS is an impulse function which takes a non-zero value for a short time interval a around $t=t_{1,0}$; $b=\{0,1\}$ is used to cover the two cases without and with the impulse.

Furthermore, we are interested in the case where $u(t)$ is as well an impulse like function given by

$$u(t)=\frac{d}{\left|c\right|\sqrt{\pi }}{e}^{-{\left(\frac{t-t_{2,0}}{c}\right)}^2}.$$

(20)

Here, the impulse is localised around $t=t_{2,0}$ with the width c; again $d=\{0,1\}$ is used to cover the two cases without and with the impulse. To find IL and IF for system (17) and (18), we use Proposition 1 and calculate the expressions for

$$\mathsf{\Sigma }(t)=\left[\begin{array}{cc}{\mathsf{\Sigma }}_{x_1{x}_1}& {\mathsf{\Sigma }}_{x_1{x}_2}\\ {\mathsf{\Sigma }}_{x_2{x}_1}& {\mathsf{\Sigma }}_{x_2{x}_2}\end{array}\right]\mathrm{and}\langle \mathbf{x}(t)\rangle ={[\langle x_1(t)\rangle ,\langle x_2(t)\rangle ]}^T,$$

(21)

using Equations (19) and (20), as shown in Appendix A.

Equation (21) then determines the form of the joint PDF $p(\mathbf{x};t)$ in Equation (3) for the two variables $i=1,2$. On the other hand, the marginal PDFs of $x_1$ and $x_2$ for Equations (17) and (18) are given by

$$P_{x_1}(x_1;t)=\frac{1}{\sqrt{2\pi {\mathsf{\Sigma }}_{x_1{x}_1}}}{e}^{-\frac{{(x-\langle x\rangle )}^2}{2{\mathsf{\Sigma }}_{x_1{x}_1}}},P_{x_2}(x_2;t)=\frac{1}{\sqrt{2\pi {\mathsf{\Sigma }}_{x_2{x}_2}}}{e}^{-\frac{{(x_2-\langle x_2\rangle )}^2}{2{\mathsf{\Sigma }}_{x_2{x}_2}}}.$$

(22)

From these PDFs, we can easily obtain the entropy based on the joint and marginal PDFs, respectively, as follows

$$\begin{array}{ccc}\hfill S(t)& =& -\int d\mathbf{x}p(\mathbf{x};t)\ln p(\mathbf{x};t)=\frac{1}{2}\left[1+\ln ({(2\pi )}^2|\mathsf{\Sigma }|)\right],\hfill \end{array}$$

(23)

$$\begin{array}{ccc}\hfill S_{x_1}(t)& =& -\int d{x}_{1}p(x_1;t)\ln p(x_1;t)=\frac{1}{2}\left[1+\ln (2\pi {\mathsf{\Sigma }}_{x_1{x}_1})\right],\hfill \end{array}$$

(24)

$$\begin{array}{ccc}\hfill S_{x_2}(t)& =& -\int d{x}_{2}p(x_2;t)\ln p(x_2;t)=\frac{1}{2}\left[1+\ln (2\pi {\mathsf{\Sigma }}_{x_2{x}_2})\right].\hfill \end{array}$$

(25)

3.1. Information Length for Equation (17)

We now use Proposition 1 (Equations (3) for (17)) and Theorem 1. Since the covariance matrix $\mathsf{\Sigma }$ as well as the mean values $\langle \mathbf{x}(t)\rangle$ (see Appendix A) for the joint PDF involve many terms including special (error) functions, it requires a long algebra and numerical simulations (integrations) to calculate Equations (8) and (9), respectively. The following thus summarise the main steps only. First, we can show that $\mathcal{E}(t)$ for the linear non-autonomous stochastic process (1) can be rewritten as

$$\mathcal{E}(t) = {\langle \mathbf{x}\rangle }^T{\mathbf{A}}^T{\mathsf{\Sigma }}^{-1}\mathbf{A}\langle \mathbf{x}\rangle + u{\mathbf{B}}^T{\mathsf{\Sigma }}^{-1}\mathbf{B}u + {\langle \mathbf{x}\rangle }^T{\mathbf{A}}^T{\mathsf{\Sigma }}^{-1}\mathbf{B}u + u{\mathbf{B}}^T{\mathsf{\Sigma }}^{-1}\mathbf{A}\langle \mathbf{x}\rangle + \frac{1}{2}\mathbf{tr}\left({({\mathsf{\Sigma }}^{-1}{\partial }_{t_1}\mathsf{\Sigma })}^2\right).$$

(26)

We can then show that for Equation (17), Equation (26) becomes

$$\begin{array}{ccc}\hfill \mathcal{E}(t)& =& \frac{1}{\left|\mathsf{\Sigma }\right|}\left({\langle x_2\rangle }^2{\mathsf{\Sigma }}_{x_2{x}_2}+\left(\gamma \langle x_2\rangle +{\omega }^2\langle x_1\rangle +u\right)\left(2\langle x_2\rangle {\mathsf{\Sigma }}_{x_1{x}_2}+{\mathsf{\Sigma }}_{x_1{x}_1}\left(\gamma \langle x_2\rangle +{\omega }^2\langle x_1\rangle +u\right)\right)\right)\hfill \\ & & +\frac{1}{{\left|\mathsf{\Sigma }\right|}^2}\left(2{\mathsf{\Sigma }}_{x_1{x}_2}^2\left(({\partial }_t{\mathsf{\Sigma }}_{x_2{x}_2})({\partial }_t{\mathsf{\Sigma }}_{x_1{x}_1})+{({\partial }_t{\mathsf{\Sigma }}_{x_1{x}_2})}^2\right)+2{\mathsf{\Sigma }}_{x_1{x}_1}({\partial }_t{\mathsf{\Sigma }}_{x_1{x}_2})\left({\mathsf{\Sigma }}_{x_2{x}_2}({\partial }_t{\mathsf{\Sigma }}_{x_1{x}_2})\right.\right.\hfill \\ & & \left.\left.-2{\mathsf{\Sigma }}_{x_1{x}_2}({\partial }_t{\mathsf{\Sigma }}_{x_2{x}_2})\right)+{\mathsf{\Sigma }}_{x_1{x}_1}^2{({\partial }_t{\mathsf{\Sigma }}_{x_2{x}_2})}^2+4{\mathsf{\Sigma }}_{x_2{x}_2}{\mathsf{\Sigma }}_{x_1{x}_2}({\partial }_t{\mathsf{\Sigma }}_{x_1{x}_2})({\partial }_t{\mathsf{\Sigma }}_{x_1{x}_1})+{\mathsf{\Sigma }}_{x_2{x}_2}^2{({\partial }_t{\mathsf{\Sigma }}_{x_1{x}_1})}^2\right).\hfill \end{array}$$

(27)

By using $\langle x_1\rangle ,\langle x_2\rangle ,{\mathsf{\Sigma }}_{x_1{x}_1},{\mathsf{\Sigma }}_{x_1{x}_2}$ and ${\mathsf{\Sigma }}_{x_2{x}_2}$ given in Appendix A, we calculate (28). Finally, to calculate IL in Equation (8), we perform the numerical integration of $\sqrt{\mathcal{E}(t)}$ over time for the chosen parameters and initial conditions. Results are presented in Section 4.

3.2. Information Flow for Equation (17)

To find the information flow for Equation (17), we compare it with Equation (13)

$$\begin{array}{ccc}\hfill \frac{{\partial }_{x_1}H(\mathbf{x})}{{\mathsf{\Gamma }}_1}=-x_2(t),\frac{{\partial }_{x_2}H(\mathbf{x})}{{\mathsf{\Gamma }}_2}& =& \gamma x_2(t)+{\omega }^2{x}_1(t)-u(t),T_1=0,\frac{T_2}{{\mathsf{\Gamma }}_2}=D(t).\hfill \end{array}$$

(28)

After some algebra using Equation (28) in Equations (14) and (15), we can show (see Appendix B for derivation)

$$\begin{array}{ccc}\hfill T_{1\to 2}& =& -{\omega }^2\frac{{\mathsf{\Sigma }}_{x_1{x}_2}}{{\mathsf{\Sigma }}_{x_2{x}_2}}-D\frac{{\mathsf{\Sigma }}_{x_1{x}_2}^2}{\left|\mathsf{\Sigma }\right|{\mathsf{\Sigma }}_{x_2{x}_2}},\hfill \end{array}$$

(29)

$$\begin{array}{ccc}\hfill T_{2\to 1}& =& \frac{1}{2}\frac{d}{dt}\ln {\mathsf{\Sigma }}_{x_1{x}_1}.\hfill \end{array}$$

(30)

It is important to note that unlike (28), Equations (29) and (30) depend only on the covariance matrix $\mathsf{\Sigma }$, being independent of the mean values, as noted in Section 1.

4. Simulations

In this section, we present simulation results that show how IF and IL capture abrupt changes in the system dynamics of the Kramers equation. To this end, we designed four simulation experimental scenarios, which are summarised in Figure 2. The different scenarios were chosen depending on whether $D(t)$ and $u(t)$ (defined in Equations (19) and/or (20), respectively) include(s) an impulse function (that is, whether $b=0$ or 1 and $d=0$ or 1), which caused the abrupt changes in the values of $\mathsf{\Sigma }(t)$ and $\langle \mathbf{x}\rangle$, respectively. Specifically, Case 1 was without any impulse ($b=d=0$); Cases 2 and 3 were when the impulse was included in D and $u(t)$ ($b=1,d=0$ and $b=0,d=1$), respectively; Case 4 was with both impulses ($b=d=1$). As noted at the end of Section 4, IL and IF in Equation (28) and Equations (29) and (30) clearly reveal that IF was not affected by the change in the mean values. This means, IF took the same value in both Cases 1 and 3; it also took the same value in both Cases 2 and 4. This is highlighted in Figure 2 by the purple colour.

For Cases 1–4 in Figure 2, we fixed the value of $\omega$ to be $\omega =1$ and varied $\gamma$ to explore different scenarios of no damping $\gamma =0$, underdamping $\gamma <2\omega$, critically damping $\gamma =2\omega$ and over damping $\gamma >2\omega$. Furthermore, we fixed the values of the initial covariance matrix as follows

$$\mathsf{\Sigma }(0)=\left[\begin{array}{cc}0.01& 0\\ 0& 0.01\end{array}\right].$$

(31)

The initial mean values were fixed as $\langle \mathbf{x}(0)\rangle ={[-0.5,0.7]}^T$ for all Cases.

In addition, we performed the stochastic simulations for Cases 1–4 by using a Cholesky decomposition to generate random numbers [28] according to the Gaussian statistics $\mathbf{x}\sim N(\langle \mathbf{x}\rangle ,\mathsf{\Sigma })$, specified by the values of $\mathsf{\Sigma }$ and $\langle x_i\rangle$ ($i=1,2$) given in Appendix A. Simulated random trajectories are shown in blue dots in the phase portrait of $x_1$ and $x_2$ in Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8 of the following subsections.

4.1. Information Flow Simulation Results

As noted in Section 2.3, we recall that IF is used to measure a directional information flow in terms of its entropy and that IF is either positive or negative unlike transfer entropy. In our experimental simulations, we were interested in how sensitive IF was to abrupt changes. The time-evolutions of IF $T_{1\to 2}$, $T_{2\to 1}$, joint $S(t)$ and marginal $S_{x_1}(t)$, $S_{x_2}(t)$ entropies in Equations (23)–(25), and the phase portrait of $x_1$ vs. $x_2$ are shown in Figure 3 and Figure 4. We used the same initial condition $\mathsf{\Sigma }(0)$ given by Equation (31) and $\omega =1$ while varying the value of $\gamma$. As noted above, random trajectories from stochastic simulations (using a Cholesky decomposition to generate the random number [28]) were overplotted in blue dots in the phase portraits. Specifically, Figure 3 and Figure 4 are for Case 1 and Case 2, respectively (with $b=0$ and $b=1$ in (19), respectively). The exact value of $D(t)$ is shown in Figure 2 and as a blue dotted line in all panels of Figure 3 and Figure 4 (using the y-axis on the right of each panel).

4.1.1. Case 1—Constant D(t) and u(t) = 0

We started with Case 1 which had no perturbation (constant $D(t)=D_0=0.001$ and $u(t)=0$) and examined the effects of the system parameters $\gamma$ on IF. First, with no damping $\gamma =0$ (Figure 3a), $S_{x_1},S_{x_2}$ and S all increased monotonically in time from a negative value (a less disordered state) to a positive value (more disordered state) due to the stochastic noise. On the other hand, $T_{1\to 2}$ and $T_{2\to 1}$ showed similar behaviours but with opposite sign, making $T_{2\to 1}+T_{1\to 2}\approx 0$. The opposite sign of $T_{1\to 2}$ and $T_{2\to 1}$ suggests that $x_2$ acted to increase the marginal entropy of $x_1$ (by transferring the stochasticity fed into $x_2$ by $\xi$) while $x_1$ decreased the marginal entropy of $x_2$ (by providing a restoring/inertial force causing the harmonic oscillations). The fact that $T_{2\to 1}+T_{1\to 2}\approx 0$ can be corroborated by the similarity between the marginal entropies $S_{x_1}$ and $S_{x_2}$.

Second, in the underdamped case with $0<\gamma <2\omega$ shown in Figure 3b, the phase portrait exhibited the behaviour of an underdamped harmonic oscillator. The role of the damping $\gamma \ne 0$ was to bring the system to an equilibrium in the long time limit where PDFs were stationary and $S_{x_1},S_{x_2}$ and S took constant values

$$\underset{t\to \infty }{\lim }{S}_{x_1}(t)=\frac{1}{2}\ln \left(\frac{2D\pi }{\gamma {\omega }^2}\right),\underset{t\to \infty }{\lim }{S}_{x_2}(t)=\frac{1}{2}\ln \left(\frac{2D\pi }{\gamma }\right),\underset{t\to \infty }{\lim }S(t)=\ln \left(\frac{2D\pi }{\gamma \omega }\right),$$

as can be shown by using (A7) in (23)–(25). Specifically, in Equation (5), the first term in RHS (which depended on $\mathsf{\Sigma }(0)$) vanisheed as $t\to \infty$ while the second term in RHS (which depended on $D(t)$) determined the value of ${\lim }_{t\to \infty }\mathsf{\Sigma }(t)$ which for $\gamma =1$ was as follows (see Equation (A7))

$$\mathsf{\Sigma }(t\to \infty )=\left[\begin{array}{cc}0.001& 0\\ 0& 0.001\end{array}\right].$$

(32)

The reason why $S_{x_1},S_{x_2}$ and S overall decreased in time is because the equilibrium had a narrower PDF (${\mathsf{\Sigma }}_{x_1{x}_2}(t\to \infty )=0.001,{\mathsf{\Sigma }}_{x_2{x}_2}(t\to \infty )=0.001$) (see Equation (32)) than the initial PDF (${\mathsf{\Sigma }}_{x_1{x}_1}(0)={\mathsf{\Sigma }}_{x_2{x}_2}(0)=0.01$). Consequently,

$$\underset{t\to \infty }{\lim }{T}_{1\to 2}(t)=\underset{t\to \infty }{\lim }{T}_{2\to 1}(t)=0.$$

Third, in the critical/overdamped case $\gamma \ge 2\omega$ in Figure 3c,d, we observed a much faster decrease in $S_{x_2}$ than $S_{x_1}$ as $\gamma$ damps $x_2$ quickly (recall that $\frac{d{x}_1}{dt}=x_2$ and see (17)). Consequently, there was a faster and higher transient in $T_{1\to 2}$ compared with $T_{2\to 1}$ for larger $\gamma$, fluctuations in $x_1$ having a greater effect on the rate of change in the marginal entropy $S_{x_2}$. It is worth emphasising that our results for $\gamma \ne 0$ above (e.g., the decrease in entropies) involved the narrowing of a PDF over time. In particular, $T_{1\to 2}$ and $T_{2\to 1}$ for a constant $D(t)=0.001$ were caused by the change in $\mathsf{\Sigma }(t)$ from its initial value $\mathsf{\Sigma }(0)$ to the equilibrium value in Equation (32) due to $D(t)=0.001$. For a much larger $D(t)$, Equation (32) took a larger value than ${\mathsf{\Sigma }}_{x_1{x}_1}(0)={\mathsf{\Sigma }}_{x_2{x}_2}(0)$, and PDFs became broaden over time, entropies increasing in time, for instance. As a result, $T_{2\to 1}\le 0$ while $T_{1\to 2}>0$. Appendix C explores how different values of the constant $D(t)$ affect IF. Finally, we note that in the phase portrait plots, the stochastic trajectories shown in blue dots generated by $\mathbf{x}\sim N(\langle \mathbf{x}\rangle ,\mathsf{\Sigma })$ remained near the trajectories of the mean values.

4.1.2. Case 2—Perturbation in D(t) and u(t) = 0

To study how sensitive IF was to a sudden perturbation in $D(t)$ (therefore in $\mathsf{\Sigma }(t)$), we included an impulse function localised around $t=4$ (see Figure 2) in $D(t)$, which is shown in blue dotted line using the right y axis on Figure 4. As before, Figure 4 shows results for the undamped, underdamped, critically damped and over damped cases, respectively.

First, in Figure 4a for $\gamma =0$, we observed that in a sharp contract to Figure 3a, the impulse rendered large fluctuations in the simulated trajectory $\mathbf{x}\sim N(\langle \mathbf{x}\rangle ,\mathsf{\Sigma })$, with significant deviation from the mean trajectory $\langle \mathbf{x}(t)\rangle$. On the other hand, such an abrupt change in $\mathsf{\Sigma }(t)$ led to a rapid increase in $S_{x_1},S_{x_2},S$, $T_{1\to 2}$ and $T_{2\to 1}$ followed by oscillations. The amplitude of these oscillations slowly decreased in time, the oscillation frequency set by $\omega$ (as expected for no-damping).

Second, in the underdamped case $0<\gamma <2\omega$ shown in Figure 4b, $T_{1\to 2}$ and $T_{2\to 1}$ exhibited some oscillations before reaching the equilibrium, as can also be seen from the phase portrait behaviour. Since the damping was still small, there was rather a long transient. It is interesting to notice that $T_{1\to 2}$ and $T_{2\to 1}$ flipped their signs (e.g., $T_{2\to 1}<0$ to $T_{2\to 1}>0$ around $t=4$ as t increased) due to a sudden increase in D ($\mathsf{\Sigma }$). This can be understood since the perturbation applied to $x_2$ increased marginal entropy $S_{x_1}$ while $x_1$ decreased the marginal entropy $S_{x_2}$. As a result, around the time $t=4$ where D was maximum, the sign of IF became opposite to that without the perturbation shown in Figure 3b. Third, for the case $\gamma \ge 2\omega$ shown in Figure 4c,d, the sign of $T_{1\to 2}$ and $T_{2\to 1}$ behaved similarly to the underdamped case Figure 4b). Overall, Figure 4 shows that $|T_{1\to 2|}$ and $|T_{2\to 1}|$ exhibited their peaks around $t=4$. However, a close examination of the cases with $\gamma \ne 0$ revealed that the peak of $|T_{1\to 2}|$ and $|T_{2\to 1}|$ appeared after the peak of the impulse (in blue dotted line). That is, the peaks of $|T_{1\to 2|}$ and $|T_{2\to 1}|$ proceeded (not preceded) the actual impulse peak. This will be compared with the case of IL in the next section where the peak of the information length diagnostics $\mathcal{E}$ tended to precede the impulse peak, predicting the abrupt changes earlier than IF. Furthermore, IF was independent of external perturbations in $\langle \mathbf{x}\rangle$.

4.2. Information Length Diagnostics Simulation Results

In this subsection, we investigated how sensitive information length diagnostics ($\mathcal{L}$, $\mathcal{E}$) were to the abrupt changes in the system dynamics. In contrast to IF, IL was capable of detecting changes in both mean values ($u(t)$) and $\mathsf{\Sigma }$ ($D(t)$), as can be inferred from Equation (9). We considered the four Cases 1–4 in Figure 2 in Figure 5, Figure 6, Figure 7 and Figure 8, respectively. In each case, we present the results of $\mathcal{L}$, $\mathcal{E}$, ${\mathcal{E}}_{x_1}$, ${\mathcal{E}}_{x_2}$, $\mathcal{E}-{\mathcal{E}}_m$ and the phase portrait of $x_1$ vs. $x_2$ (where the stochastic simulations are shown in blue dots). As before, we used the same initial conditions $\mathsf{\Sigma }(0)$ in Equation (31) and the same parameter values ($\omega =1$) while varying $\gamma$ for undamped, underdamped, critically damped and overdamped cases. The initial mean values are fixed as $\langle \mathbf{x}(0)\rangle ={[-0.5,0.7]}^T$ for all Cases.

It is worth noting that (the unperturbed) Case 1 in Figure 2 corresponded to the usual Kramers equation, previously studied in [27]. We nevertheless show results for Case 1 below to be able to compare with Cases 2–4 as well as show new results such as ${\mathcal{E}}_{x_1}$, ${\mathcal{E}}_{x_2}$, and $\mathcal{E}-{\mathcal{E}}_m$ that might be useful for understanding the correlation between variables. Note that in the following, $\mathcal{E}-{\mathcal{E}}_m$ plots are not discussed in each Case, but instead discussed separately in Section 4.2.5.

4.2.1. Case 1—Constant D(t) and u(t) = 0

In this unperturbed case, our main focus here was on the effects of $\gamma$ on $\mathcal{L}$, $\mathcal{E}$ and the marginal information velocities ${\mathcal{E}}_{x_1}$ and ${\mathcal{E}}_{x_2}$.

First, for the undamped case $\gamma =0$ shown in Figure 5a, harmonic oscillations (e.g., seen in the phase portrait) appeared in ${\mathcal{E}}_{x_1}$ and ${\mathcal{E}}_{x_2}$, their oscillation frequency determined by $\omega$. We recall that ${\mathcal{E}}_{x_1}$ and ${\mathcal{E}}_{x_2}$ are calculated from the marginal PDF of $x_1$ and $x_2$, respectively. Because of the absence of damping, $\mathcal{E}(t)$ decreased but never reached 0. The finite value of $\mathcal{E}(t)$ is due to ${\partial }_t\mathsf{\Sigma }(t)\ne 0$ and ${\partial }_t\langle \mathbf{x}\rangle \ne 0$ as the PDF $p(\mathbf{x};t)$ evolved according to (3).

When $0<\gamma <2\omega$ in Figure 5b, a non-zero damping led to ${\lim }_{t\to \infty }\mathcal{E}(t)=0,$ as the PDF reached its equilibrium value while $\mathcal{L}$ converged to a finite value. It is worth highlighting that non-zero $\mathcal{E},{\mathcal{E}}_{x_1}$ and ${\mathcal{E}}_{x_2}$ signified transient behaviour far from equilibrium. Finally, in Figure 5c,d for $\gamma \ge 2\omega$, we observed that a higher value of $\gamma$ led to the shorter duration of transients and larger fluctuations in $\mathcal{E}$.

4.2.2. Case 2—Perturbation in D(t) and u(t) = 0

Figure 6 shows the effect of an impulse like function in $D(t)$ (see (19)), which then led to an abrupt change in the covariance of the system PDF $p(\mathbf{x};t)$ given by (3). Since IL depended on the value of $\frac{1}{2}\mathbf{tr}\left({({\mathsf{\Sigma }}^{-1}{\partial }_{t_1}\mathsf{\Sigma })}^2\right)$ (see Equation (9)), this abrupt change in $\mathsf{\Sigma }$ had a considerable impact on $\mathcal{E}(t)$.

For the case $\gamma =0$ shown in Figure 6a, the amplitude of $\mathcal{E}$ and $\mathcal{L}$ was seen to be increased around the time of the impulse peak. The phase portrait clearly shows the increase in the uncertainty (more scattered data). The values of ${\mathcal{E}}_{x_1}$ and ${\mathcal{E}}_{x_2}$ were also seen to increase due to the perturbation.

For $0<\gamma <2\omega$, the oscillations in ${\mathcal{E}}_{x_1}$ and ${\mathcal{E}}_{x_2}$ were much less pronounced due to damping (see Figure 6b). This behaviour prevailed also for $\gamma \ge 2\omega$ shown in Figure 6c,d. Interestingly, a close examination revealed that the maxima in $\mathcal{E}$ and ${\mathcal{E}}_{x_2}$ proceeded the peaks of the impulse (in blue dotted line), as alluded at the end of Section 4.1.2. This was seen more clearly for larger $\gamma$ in Figure 6c,d where the maxima in $\mathcal{E}$, ${\mathcal{E}}_{x_1}$ and ${\mathcal{E}}_{x_2}$ all preceded the impulse peaks. These results demonstrate that the information diagnostics predicted the onset of a sudden event earlier than the information flow.

4.2.3. Case 3—Constant D(t) and Perturbation in u(t)

Figure 7 shows results for a constant $D(t)$ and an impulse-like external input $u(t)$ (see (20)) which caused an abrupt change in $\langle \mathbf{x}(t)\rangle$. $u(t)$ is shown in a red dotted line using the right y axis.

When $\gamma =0$, Figure 7a shows how the perturbation changed the dynamics of $\langle \mathbf{x}(t)\rangle$ while $\mathsf{\Sigma }(t)$ remained unchanged in the phase portrait plot. When a non-zero damping was included in Figure 7b–d, $\mathcal{E}$, ${\mathcal{E}}_{x_1}$ and ${\mathcal{E}}_{x_2}$ approached zero as $t\to \infty$. The phase portrait in Figure 7b–d shows how the perturbation changed the trajectory temporarily.

Overall, we observed a very large increase in $\mathcal{E}$, ${\mathcal{E}}_{x_1}$ and ${\mathcal{E}}_{x_2}$ (larger increase in ${\mathcal{E}}_{x_2}$ than in ${\mathcal{E}}_{x_1}$), their peaks forming a little before or around the impulse peak (shown in red dotted line). Besides, the value of $\mathcal{L}$ was higher when we had a perturbation on $u(t)$ and a constant $D(t)$ than when $D(t)$ was perturbed and $u(t)=0$ for $\gamma >0$ (see it by comparing Figure 6 to Figure 7). Furthermore, ${\mathcal{E}}_{x_2}$ was the most affected by the changes in $u(t)$ since $x_2$ directly depends on $u(t)$.

Finally, it is important to highlight that our result of a high sensitivity of IL to abrupt changes in $u(t)$ was not shared with IF which was insensitive to $u(t)$.

4.2.4. Case 4—Perturbations in Both D(t) and u(t)

Case 4 in Figure 2 is when we added impulse like functions to both $D(t)$ and $u(t)$ ($b=1$ and $d=1$ in Equations (19) and (20), respectively.). Again, note that $u(t)$ is shown in a red dotted line using the right y axis. Overall, the phase portraits in Figure 8 for the undamped, underdamped, critically damped and overdamped scenarios show that the perturbations momentarily broadened the width of PDF (3) while causing a large deviation of the trajectory of $\langle \mathbf{x}(t)\rangle$.

Figure 8a for the undamped case $\gamma =0$ shows that the perturbations increased the value of $\mathcal{L}$ in comparison to Case 3 with $\gamma =0$ (See Figure 7a). This is due to the increase in $\mathsf{\Sigma }$ in Case 4 by the impulse in $D(t)$, which increased the uncertainty against which the information was measured.

For non-zero damping in Figure 8b–d, we saw a substantial increment in the amplitude of ${\mathcal{E}}_{x_2}$ (similar to Case 2 but smaller than in Case 3). In fact, in all cases of the underdamped, critically damped and overdamped scenarios, the overall behaviour was close to that observed in Case 2 (see Figure 6) than that in Case 4. It is because the increase in mean values due to the impulse $u(t)$ was somewhat compensated by the uncertainty increase due to the impulse in $D(t)$. This is a consequence of both impulses that had the same form, e.g., taking their maximum values at the same time $t=4$ (see Figure 2). For instance, if Case 4 were considered with the two impulses that were timed differently, much larger values of $\mathcal{E},{\mathcal{E}}_{x_1},{\mathcal{E}}_{x_2}$ were expected for Case 4 compared with Case 2. There were obviously differences between Case 2 and Case 4, for instance, in the long time limit $t\to \infty$, $\mathcal{L}$ in Case 4 was always bigger than that in Case 3. Finally, similar comments as before could be made in regards to the prediction capabilities of the information length diagnostics $\mathcal{E}$.

4.2.5. Interpretation of the $\mathcal{E}-{\mathcal{E}}_m$ Plots

We now discuss the plot of $\mathcal{E}-{\mathcal{E}}_m$ for all Cases 1–4 collectively to point out its usefulness.

First, according to (9), it is clear that $\mathcal{E}$ considered the contribution from the non-independent random variables $\langle x_1\rangle$, $\langle x_2\rangle$, and its covariance matrix $\mathsf{\Sigma }(t)$ to the information changes in time, while ${\mathcal{E}}_m$ was based on the sum of ${\mathcal{E}}_i$ from a marginal PDF of $x_i$ (see Definition 1). Thus plotting $\mathcal{E}-{\mathcal{E}}_m$ gave an approximation of the contribution from the cross-correlation ${\mathsf{\Sigma }}_{x_i{x}_j}\forall i\ne j$ to $\mathcal{E}$.

As an example, Figure 9 shows the simulation of a non-perturbed scenario ($u(t)=0$ and $D(t)=0.001$) using $\langle \mathbf{x}(0)\rangle ={[-0.5,0.7]}^T$, ${\mathsf{\Sigma }}_{x_1{x}_1}(0)={\mathsf{\Sigma }}_{x_2{x}_2}(0)=0.01$, ${\mathsf{\Sigma }}_{x_1{x}_2}(0)={\mathsf{\Sigma }}_{x_2{x}_1}(0)=0$, $\gamma =1$ and $\omega =2$ (underdamped). This example permitted us to compare the evolution/deformation of the width of $p(\mathbf{x};t)$ (given by Equation (3)) in the $x_1$-$x_2$ plane with the value of $\mathcal{E}-{\mathcal{E}}_m$ over time shown in the right panel of Figure 9.

Figure 9 when $\mathcal{E}-{\mathcal{E}}_m=0$ (at $t=0$, for instance), shows that the shape of $p(\mathbf{x};t)$ was a perfect circle (this because ${\mathsf{\Sigma }}_{x_1{x}_2}(t\to 0)=0$). For $\mathcal{E}-{\mathcal{E}}_m\ne 0$, the shape of $p(\mathbf{x};t)$ was deformed according to the value of $\mathcal{E}-{\mathcal{E}}_m$. The simulations suggest that the bigger the value of $|\mathcal{E}-{\mathcal{E}}_m|$ the higher the correlation between the random variables $x_1$ and $x_2$ ($p(\mathbf{x};t)$ was highly deformed).

In summary, in regard to Cases 1–4, we can remark two characteristics on the behaviour of $\mathcal{E}-{\mathcal{E}}_m$ in Figure 5, Figure 6, Figure 7 and Figure 8. First, the value presented more variations when we had a perturbation on $D(t)$, for instance when $\gamma =0$ there were high oscillations not presented when there was a perturbation on $u(t)$ but not on $D(t)$. Second, the higher the value of $\gamma$ the less the deformations through time of $p(\mathbf{x};t)$’s width since $\mathcal{E}-{\mathcal{E}}_m$ showed less changes through time.

5. Concluding Remarks

We have investigated the prediction capability of information theory by focusing on how sensitive information-geometric theory (information length diagnostics) [7,8,9,10,11,12] and one of the entropy-based information theoretical methods (information flow) [16,17] are to abrupt changes. Specifically, we proposed a non-autonomous Kramers equation by including sudden perturbations to the system as impulses to mimic the onset of a sudden event and calculate time-dependent probability density functions (PDFs) and various statistical quantities with the help of numerical simulations. It was explicitly shown that the information flow like any other entropy-based measures is insensitive to to perturbations which do not affect entropy (such as the mean values). Specifically, the information length diagnostics are very sensitive to both perturbations in the covariance $\mathsf{\Sigma }(t)$ and mean $\langle \mathbf{x}(t)\rangle$ of the process while the information flow only detects perturbations in its covariance. Furthermore, we demonstrated that information length diagnostics predict the onset of a sudden event earlier than the information flow; the peaks of $T_{1\to 2}$ (or $T_{2\to 1}$) tend to proceed the impulse peak while the peak of information length diagnostics $\mathcal{E}$ tends to precede the impulse peak.

We expect that some of the results presented in this work would be useful in different engineering applications [34,35] since linear approximations are often useful [36] for control engineering applications. For instance, one can develop an information-geometric cost function for control design to achieve a guided self-organisation [37,38], instead of using entropy as a cost function [39]. Given high variabilities involved in complexity and emergent behaviour [13,14,15], it will be interesting to further extend this work to investigate interconnection of the components in a complex system, or causality and also to non-linear, non-Gaussian models or real data.