An Excess Entropy Approach to Classify Long-Term and Short-Term Memory Stationary Time Series

Abstract

Long-term memory behavior is one of the most important phenomena that has appeared in the time series analysis. Different from most definitions of second-order properties, an excess entropy approach is developed for stationary time series to classify long-term and short-term memory. A stationary sequence with finite block entropy is long-term memory if its excess entropy is infinite. The simulation results are graphically demonstrated after some theoretical results are simply presented by various stochastic sequences. Such an approach has advantages over the traditional ways that the excess entropy of stationary sequence with finite block entropy is invariant under instantaneous one-to-one transformation, and that it only requires very weak moment conditions rather than second-order moment conditions and thus can be applied to distinguish the LTM behavior of stationary sequences with unbounded second moment (e.g., heavy tail distribution). Finally, several applications on real data are exhibited.

1. Introduction

Long-term memory (abbreviated as LTM) or long-range dependency behavior is one of the most important phenomena that appeared in the time series analysis, which plays a very important role in almost all fields such as telecommunications, video traffic modeling, biology, econometrics, finance, hydrology, medicine, linguistics, etc. [1,2,3,4,5,6,7,8,9,10].

A stationary (stochastic) sequence is a class of sequences of random variables with time-invariant probability law. For a stationary sequence, the LTM behavior is related to the rate of decay of its statistical dependence.

Different definitions of LTM are applied to different real-world applications. An econometric survey [11] introduces nine different definitions such as Parzen’s concept, the concept of rate of convergence, the behavior in Allan’s sense, the behavior in the prediction error variance sense and the behavior in distribution. The second-order properties including the asymptotic properties of covariances, variances of partial sums, and spectral density, are mostly applied to define the LTM of a stationary sequence in literature. The popularity of different definitions based on second-order properties has both historical and practical reasons: they are relatively easy to understand and estimate from the experimental data. The notion of LTM is discussed from several perspectives, and is comprehensively reviewed by Samorodnitsky [8].

One classical way of characterizing the LTM is by lack of summability of autocovariance functions (ACF) [12]. In short-term memory (STM) time series, it decays rapidly as the time-difference grows and so the summability holds. Either it drops rapidly to zero after a short time-lag, or it eventually decreases exponentially. In LTM time series, it has a much stronger statistical dependence and lacks summability. The ACF decreases more slowly than exponentially, or decreases typically power-like [13].

In information theory, block entropy was developed to capture the theoretic measurements of a sequence’s uncertainty (randomness) and its memory. The entropy rate measures the (irreducible) uncertainty in (binary) sequences that remains after the structures and correlations in larger and larger blocks of sequences are considered. For a stationary series, the higher the randomness, the larger entropy rate. In contrast to a series with a low entropy rate, there are many correlations between symbols. The calculations of the entropy of (LTM) stochastic sequences have been widely proposed in [14,15,16,17]. For a stationary time series $X=\{X_n:n\in \mathbf{Z}\}$, the excess entropy $EE\left(X\right)$ was widely applied and also developed to measure the mutual information between the infinite future $\overrightarrow{X}=X_0{X}_1{X}_2\cdots$ and the infinite past $\overleftarrow{X}=\cdots X_{-3}{X}_{-2}{X}_{-1}$ [18,19].

In the current letter, a new approach is developed to characterize LTM in the sense of excess entropy. It is concluded that a stationary sequence with finite block entropy is LTM if its excess entropy is infinite.

For some classical sequences, such an approach is consistent with the traditional methods. Apart from characterizing those classical sequences, such an approach has advantages over the traditional ways that the excess entropy of stationary sequence with finite block entropy is invariant under instantaneous one-to-one transformation, and that it only requires very weak moment conditions rather than second-order moment conditions and thus can be applied to distinguish the LTM behavior of stationary sequences with unbounded second moment (e.g., heavy tail distribution).

In particular, for a stationary Gaussian sequence, the LTM in the sense of excess entropy is stricter than that in the sense of ACF. For fractional Gaussian noise, not only excess entropy can be applied to identify whether or not a sequence is LTM, but the entropy growth curve can be graphically applied to roughly discriminate the Hurst parameter from two different sequences. Roughly speaking, the larger the entropy rate, the smaller the Hurst parameter (i.e., larger fractal dimension). In addition, for any real data with a not large sample size, the entropy growth curve can also be graphically applied to identify whether it is LTM or not.

This paper is organized as follows. In Section 2, we recall some background on information theory such as excess entropy. In Section 3, we first show the relationship between the excess entropy and the mutual information, and the invariance of a one-to-one transformation. Then, we present an excess entropy criterion for LTM series and some main theoretical results. In Section 4, we demonstrate graphically the simulation results on various stochastic sequences. Finally, we exhibit its applications on some real data.

2. Background on Information Theory

2.1. Shannon Entropy

Let X be a discrete random variable taking values $x\in S$. The probability that X takes on the particular value x is written $p\left(x\right):=P(X=x)$.

The Shannon entropy of X is well defined as [20]

$$H\left[X\right]=-E\left[{\log }_{2}P\left(X\right)\right]=-\sum _{x\in S}p\left(x\right){\log }_{2}p\left(x\right),$$

(1)

with the usual convention that $0\log 0=0$. Note that $0⩽H\left[X\right]⩽{\log }_2\left|S\right|$. The units of $H\left[X\right]$ are bits. The entropy $H\left[X\right]$ measures the uncertainty (randomness) of the variable X. In other words, it measures the average memory of X.

2.2. Block Entropy and Entropy Rate

We assume that a stochastic system is described by a stationary time series $X=\{X_n:n\in \mathbf{Z}\}$, or $X=\{\cdots ,X_{-1},X_0,X_1,X_2,\cdots \}$ that range over a finite set S.

The stationarity natural means the identical joint probability distribution when shifted in time, i.e., for all $L>0$ and $t=0,\pm 1,\pm 2\cdots$,

$$P(x_{t+1},\cdots ,x_{t+L})=P(x_1,\cdots ,x_L),$$

where $P\left(x_t\right)$ denotes the probability that at time t the random variable $X_t$ takes on the particular value $x_t\in S$ and $P(x_{t+1},\cdots ,x_{t+L})$ denotes the joint probability over blocks of L consecutive symbols $X^L:=X_1\cdots X_L$.

One can now explore the Shannon entropy $H\left(L\right)$ of the distribution $P\left(X^L\right)$ over blocks of L consecutive variables, that captures the theoretic measures of a sequence’s uncertainty (randomness) and its memory.

The total Shannon entropy of L$(L>0)$ consecutive symbols is well defined as [21]

$$\begin{array}{cc}H\left(L\right)\hfill & :=H(X_1,\cdots ,X_L):=-{\displaystyle \sum _{x^L}}P\left(X^L\right){\log }_{2}P\left(X^L\right)\hfill \\ \hfill & =-{\displaystyle \sum _{x_1}}\cdots {\displaystyle \sum _{x_L}}p(x_1,\cdots ,x_L){\log }_{2}p(x_1,\cdots ,x_L)\hfill \end{array}$$

(2)

with the convention that $H\left(0\right)=0$. Such $H\left(L\right)$ is called the $L-$block entropy, which is the sum over all possible $L-$blocks. It is easy to show that $H\left(L\right)$ is a non-decreasing and concave function of L.

To determine the (Shannon) entropy of the entire system X, one could take the limit when $L\to \infty$. It is not difficult to find, however, that $H\left(L\right)$ will be infinite as L goes to infinity.

The entropy rate of $X=\{X_n:n\in \mathbf{Z}\}$ can be defined as [22]

$$h_{\mu }:=h_{\mu }\left(X\right):=\underset{L\to \infty }{\lim }{h}_{\mu }\left(L\right)=\underset{L\to \infty }{\lim }H\left(L\right)/L=\underset{L\to \infty }{\lim }\Delta H\left(L\right),$$

(3)

where $\mu$ denotes the measure over the infinite symbols that induces the joint distribution $P\left(X^L\right)$ of L-block.

Notice that the entropy rate always exists for stationary processes with finite block entropy in [20].

The entropy rate of X can also be viewed as the limit of a conditional entropy [22,23]:

$$h_{\mu }=\underset{L\to \infty }{\lim }H\left(X_L\right|X_1,\cdots ,X_{L-1}).$$

(4)

2.3. Excess Entropy and Mutual Information

The finite L-block approximation to the entropy rate representation $\Delta H\left(L\right)$ typically overestimates another $h_{\mu }$ by a quantity $\Delta H\left(L\right)-h_{\mu }$ that suggests how much more randomness in the finite L-blocks appears than the infinite system. In other words, such excess randomness quantizes how much redundant information must be gained about the systems in order to reveal exactly the actual per-symbol uncertainty $h_{\mu }$. Summing over the overestimates, one can obtain the total excess entropy (see [18,21] and references therein for further information).

$$EE\left(X\right)=\sum _{L=1}^{\infty }(\Delta H\left(L\right)-h_{\mu }).$$

(5)

Plugging $\Delta H\left(L\right)=H\left(L\right)-H(L-1)$ into Equation (5), one can arrive at an alternate definition for the excess entropy:

$$EE\left(X\right)=\underset{L\to \infty }{\lim }(H\left(L\right)-h_{\mu }L).$$

(6)

It means that the excess entropy $EE\left(X\right)$ is the subextensive part of (big enough) block entropy $H\left(L\right)$ [21].

Let $X=\{X_n:n\in \mathbf{Z}\}$ be a stationary sequence with finite entropy $H\left(L\right)$ for each $L\in {\mathbf{Z}}^{+}$, and $I(X,L):=I(X_{-(L-1)}\cdots X_0;X_1\cdots X_L)$ be the mutual information between past and future with the length L. The mutual information between infinite past and infinite future $I\left(X\right)$ is defined as [22]

$$I\left(X\right):=\underset{L\to \infty }{\lim }I(X,L).$$

Thus, the excess entropy can also be viewed as the mutual information between the left and right semi-infinite halves (i.e., the infinite past and infinite future) of the sequence X (see Lemma 1). The excess entropy has a large number of different definitions or interpretations, which has been discussed in [18].

3. Theoretical Results

The following result shows that the excess entropy and the mutual information are identical for stationary time series with finite block entropy, which does not imply the block entropy $H\left(L\right)$ is bounded.

The following lemma is a straightforward conclusion of Theorem 4.6 of [22].

Lemma 1.

Let $X=\{X_n:n\in \mathbf{Z}\}$ be a stationary sequence with finite block entropy $H\left(L\right)$. Then, the excess entropy $EE\left(X\right)$ of X and the mutual information $I\left(X\right)$ are identical:

$$EE\left(X\right)=I\left(X\right).$$

The following theorem is also an immediate conclusion of Theorem 3.8 of [22] and Lemma 1.

Theorem 1.

Let $X=\{X_n:n\in \mathbf{Z}\}$ be a stationary sequence with finite block entropy $H\left(L\right)$, and f be a one-to-one transformation, then the excess entropy $EE\left(Y\right)$ of a transformational sequence $Y:=f\left(X\right)=\{f\left(X_n\right):n\in \mathbf{Z}\}$ equals to that of X, i.e.,

$$EE\left(Y\right)=EE\left[f\right(X\left)\right]=EE\left(X\right).$$

A new definition of LTM is derived by the Lemma 1 and Theorem 1.

Definition 1.

A stationary sequence with finite block entropy is called LTM if its excess entropy is infinite; otherwise, it is called STM.

The theoretical results of the LTM behaviors of various stationary sequences are provided by the Definition 1.

Theorem 2.

Let $X=\{X_n:n\in \mathbf{Z}\}$ be an independent and identical distribution sequence, then it is STM with $EE\left(X\right)=0.$

In this case, the entropy rate $h_{\mu }=H\left(1\right)$ since $H\left(L\right)=LH\left(1\right)$, thus the excess entropy $EE\left(X\right)=0$.

Theorem 3.

Let $X=\{X_n:n\in \mathbf{Z}\}$ be an irreducible positive recurrent Markov chain with a generator $P=\left(p_{ij}\right)$ on a countable state space, then it is STM.

In this case, the entropy rate of Markov chain X is as follows:

$${\displaystyle h_{\mu }\left(X\right)=-\sum _{ij}{\pi }_i{p}_{ij}\log p_{ij}}$$

where ${\pi }_i$ is its equilibrium distribution of X. Therefore, by the Markovian property, its excess entropy is given as follows:

$$\begin{array}{c}\hfill EE\left(X\right)=[H\left(1\right)-h_{\mu }]+[H\left(2\right)-H\left(1\right)-h_{\mu }]=H\left(2\right)-2h_{\mu }\\ \hfill =-\sum _i{\pi }_i\log {\pi }_i+\sum _{ij}{\pi }_i{p}_{ij}\log p_{ij}.\end{array}$$

For an order-R Markovian sequence, a more general argument for the excess entropy is given as (Ref. [18])

$$E=H\left(R\right)-h_{\mu }R.$$

(7)

3.1. Fractional Gaussian Noise

In recent years, the fractional Brownian motion model has been widely applied to various natural shapes and random phenomena [24,25]. A fractional Brownian motion of Hurst (exponent) parameter $H\in (0,1)$ is a zero mean Guassian process $W^H=\{W_t^H:t\ge 0\}$ on continuous-time, with the following covariance function:

$$r(s,t):=\mathbf{E}\left(W_t^H{W}_s^H\right)=\frac{1}{2}{\left(\right|t|}^{2H}+{\left|s\right|}^{2H}{-|t-s|}^{2H})$$

(8)

for $s,t\ge 0$, where $W^H$ reduces to a standard Brownian motion when $H=0.5$.

Let $X_k^H=W_k^H-W_{k-1}^H$, then $X^H=\{X_k^H:k=0,1,\dots \}$ is the stationary incremental process of a FBM and has a probability density function with joint Gaussian, which is called fractional Gaussian noise (FGN). The ACF of a FGN $X^H=\{X_k^H:k=0,1,\dots \}$ can also be defined as

$$r_n:=\mathbf{E}\left(X_{k+n}^H{X}_k^H\right)=\frac{1}{2}{\left(\right|n+1|}^{2H}-{2\left|n\right|}^{2H}{+|n-1|}^{2H}),n=1,2,\dots .$$

It is not difficult to see that

$$r_n\sim H(2H-1){\left|n\right|}^{2(H-1)}$$

(9)

as $\left|n\right|\to \infty$.

In particular, if $H=0.5$, then $r_n=0$ for all $n\ge 1$. In other words, a standard Brownian motion with $H=0.5$ has independent increments.

Recall a definition of LTM where the ACF lacks summability. Apparently, the summability of correlations (i.e., ${\sum }_{n=1}^{\infty }\left|r_n\right|<+\infty$) holds when $H\in (0,0.5)$ and does not hold when $H\in (0.5,1)$. Following Equation (9), a FGN with $H\in (0.5,1)$ is lack of the summability of correlations and is widely accepted as a LTM sequence.

According to discussions in [22], the following theorem remains.

Theorem 4.

Let $X=\{X_n:n\in \mathbf{Z}\}$ be a FGN with Hurst parameter $H\in (0,0.5)\cup (0.5,1)$. Then, it is LTM if and only if $H\in (0.5,1)$, i.e.,

$$EE\left(X\right)=+\infty ⟺H\in (0.5,1).$$

Remark 1.

This theorem implies that for a FGN, the excess entropy approach to LTM is exactly the same as the second moment approach.

3.2. Stationary Gaussian Sequence

In this subsection, the LTM of stationary Gaussian sequence (SGS) is investigated, where the subtle distinction between definitions of LTM in the sense of excess entropy and ACF is detected.

Let $X=\{X_n:n\in \mathbf{Z}\}$ be a zero-mean SGS. X can be completely characterized by its ACF $r_n:=\mathbf{E}\left(X_{k+n}{X}_k\right)$, or by its power spectral density $f\left(\lambda \right)$ equivalently, which is the Fourier transform of ACF:

$$f\left(\lambda \right)=\sum _{n=-\infty }^{\infty }{r}_n\exp (-in\lambda ),r_n=\frac{1}{2\pi }{\int }_0^{2\pi }f\left(\lambda \right)\exp \left(in\lambda \right)d\lambda .$$

Put

$$b_n=\frac{1}{2\pi }{\int }_{-\pi }^{\pi }\log f\left(\lambda \right)e^{-in\lambda }d\lambda ,$$

for any integer n if it is well defined, and $\left\{b_n\right\}$ are referred as cepstrum coefficients [26].

The following Lemma 2 results from [22,26].

Lemma 2.

Let $X=\{X_n:n\in \mathbf{Z}\}$ be a SGS. The following claims remain.

(1)

The excess entropy $EE\left(X\right)$ of X is finite if and only if the cepstrum coefficients $\left\{b_n\right\}$ satisfy the condition: ${\sum }_{n=-\infty }^{\infty }\left|n\right|b_n^2<\infty$. Meanwhile,

$$EE\left(X\right)=\frac{1}{2}\sum _{n=1}^{\infty }n{b}_n^2.$$

(2)

If the spectral density is continuous and positive, then the excess entropy $EE\left(X\right)$ of X is finite if and only if the ACFs $\left\{r_n\right\}$ satisfy the condition ${\sum }_{n=1}^{\infty }n{r}_n^2<\infty .$

Lemma 3.

Let $\left\{c\right(n\left)\right\}$ be a decreasing positive series. The following relationship remains valid [26]

$$\sum _{n}c\left(n\right)<\infty \Rightarrow \sum _{n}n{c}^2\left(n\right)<\infty .$$

Remark 2.

Put $a\left(n\right)=\frac{1}{n\log n}$, then ${\sum }_{n}a\left(n\right)=\infty$, but ${\sum }_{n}n{a}^2\left(n\right)<\infty$. It suggests that the converse implication of Lemma 3 fails.

Remember that for a SGS X with autocorrelation $\left\{r_n\right\}$, X is LTM if $\sum _n\left|r_n\right|=\infty$. The following result shows the relationship between the finiteness of excess entropy $EE\left(X\right)$ and the summability of autocovariances ($\sum _n\left|r_n\right|<+\infty$) for a SGS (followed by [22,26]).

Theorem 5.

Let $X=\{X_n:n\in \mathbf{Z}\}$ be a SGS with decreasing auto-covariance $\left\{\right|r_n\left|\right\}$, and continuous and positive spectral density. Then,

$$\sum _n\left|r_n\right|<+\infty ⟹EE\left(X\right)<+\infty .$$

Equivalently, X is STM in the sense of ACF suggests that it is also STM in the sense of excess entropy.

The proof of Theorem 5 immediately follows from Lemmas 2 and 3.

Remark 3.

The converse implication of Theorem 5 fails. The result suggests that for a SGS with continuous and positive spectral density as well as decreasing auto-covariances, LTM in the sense of excess entropy is stricter than that in the sense of covariance. In other words, the excess entropy can capture more dependent information than the second order properties of the above SGS.

4. Simulation Results

4.1. Computation on Block Entropy

To compute the block entropy $H\left(L\right)$, we first normalize the original data (or state space) S into m stationary subset $S_1,\cdots ,S_m$ and denote simply by i the subset $S_i$. Thus, the translational state space is $S=\{1,2,\cdots ,m\}$, that is to say, it can be presented on a sequence of 1 s, 2 s, ⋯ m s. According to the notation in information theory, the sequence with two states can be presented on a binary sequence of 0 s and 1 s, for example, 01111010111100. It is natural to ask how many states one should divide the original time series data into and how to normalize it. There are currently two possible methods taken into account: m-quantiles and the Non-stationary Measurement method; see [27]. Although the two different methods lead to different scales of their entropy and excess entropy, the corresponding shapes cannot be changed. Therefore, the normalization methods cannot have effects on the finiteness or infiniteness of the excess entropy. Here, a m-quantiles method is preferred to the later simulations.

Once one has finished the normalization of the state space, one can compute the block entropy. For a sequence with m states, it consists of (overlapping) blocks of L digits each. For a given length-L, there are totally $m^L$ such blocks, $B_k,k=0,1,\cdots ,m^L-1$, and they appear with very different frequencies $f\left(B_k\right)$ in the m-ary sequence. If the length of the sequence is large enough, then counting all these frequencies provides a good empirical estimate of the probability distribution $P_L\left(B_k\right)$ of the length-L blocks. Hence, it is not very difficult to calculate the block entropy $H\left(L\right)$ by the following formula [21]:

$$H\left(L\right)=-\sum _{k=0}^{m^L-1}{P}_L\left(B_k\right){\log }_2{P}_L\left(B_k\right).$$

(10)

To keep in accordance with the information theory, a 2-quantiles method is applied to normalize the sequence into a binary one in the latter simulations. Note that a 2-quantiles method can really qualify the shape and then the finiteness or infiniteness of the excess entropy.

4.2. Independent, Identically Distributed Sequences

The simplest stochastic sequence to begin with is binary variables (or 0–1 distributed variables) as an independently and identically distributed (i.i.d.) sequence. As examples, two i.i.d. sequences, a fair sequence with $p=0.5$ and a biased sequence with $p=0.8$, are simulated. (Unless otherwise specified, the length of sequences in all simulations is always no less than ${10}^7$, which corresponds to a length of about $2^{24}$.)

The left panel of Figure 1 shows their entropy growth curves $H\left(L\right)$. For both sequences, the entropy $H\left(L\right)$ grows linearly. The right panel of Figure 1 shows their entropy rate growth curves, which demonstrates that each entropy rate $\Delta H\left(L\right)$ or $h_{\mu }$ is constant. Notice, however, that the two sequences have different entropy rates $h_{\mu }$. The fair one (blue asterisks) has an entropy rate $h_{\mu }=1$, while the biased one with $p=0.8$ (red asterisks), being more predictable or less unpredictable, has an entropy rate $h_{\mu }=-p{\log }_{2}p-(1-p){\log }_2(1-p)=0.7219$. Hence, the excess entropy is zero and memoryless for all i.i.d. sequences.

4.3. Periodic Sequences

Since periodic sequences are now investigated, a binary period-5 sequence, which consists of repetitions of a template block $s^5$ = 10101, should be first considered. The entropy curve $H\left(L\right)$, which is shown in the left panel (red dots) of Figure 2, becomes a constant as L goes to a small number. Figure 2 (red, right panel) shows that the entropy rate converges asymptotically to a value, $h_{\mu }=0$. Thus, the excess entropy $EE$ for the period-5 sequence is a constant equal to ${\log }_{2}5\approx 2.3219$ bits.

Furthermore a period-8 sequence with a template block $s^8$ = 13422222 is considered. The entropy $H\left(L\right)$ and the entropy rate $h_{\mu }\left(L\right)$ curves are shown in Figure 2 (blue asterisks) that the laws are as an analogue of a period-5 sequence. Thus, the excess entropy $EE$ for such period-8 sequence is a constant equal to ${\log }_{2}8=3$ bits.

In summary, for any periodic sequence of period-p, no matter what it consists of, the entropy rate is $h_{\mu }=0$ and the excess entropy is $EE={\log }_{2}p$. It is consistent with the usual argument that a periodic sequence is STM.

4.4. Markov Sequences

Now, a simple Markov sequence with a non-zero entropy rate $h_{\mu }>0$ is considered. (The periodic sequences in the previous subsection are also Markovian, but with a zero entropy rate $h_{\mu }=0$.) It is well known that the Markov sequence is memoryless. Thus, it will be shown that the excess entropy is finite.

In particular, we shall consider two simple homogeneous Markov chains (of first order): a Markov chain $\left\{X_n\right\}$ on two state space $\{0,1\}$ with the transition probability $P(X_1=1|X_0=0)=0.95$ and $P(X_1=0|X_0=1)=0.9$; and a birth–death chain with the birth rate of $0.5$ and death rate of $0.8$.

The left panel of Figure 3 shows their entropy growth curves $H\left(L\right)$. For both sequences, the entropy $H\left(L\right)$ grows linearly when $L\ge 1$. The right panel of Figure 3 shows their entropy rate growth curves. For both sequences, the convergences of $\Delta H\left(L\right)$ to $h_{\mu }$ occur at $L=2$. Equivalently, once the statistics over all possible $2-$blocks are completed, one can gain no redundancy by tracking all possible blocks of larger length. Thus, the excess entropy for these two are constants, $EE=0.5579$ and $EE=1.403$, respectively. It is consistent with the theoretical results calculated from $EE=H\left(1\right)-\Delta H\left(2\right)$ in Equation (7).

4.5. Aperiodic Sequence: TM Sequence

Next a special aperiodic sequence: Thue–Morse (TM) sequence is investigated. The Thue–Morse (TM) sequence is a binary sequence that begins:

$$01101001100101101001011001101001$$

There are several equivalent ways of defining the TM sequence (see Wikipedia and the references therein for further information).

Figure 4 shows that it has zero entropy rate (left panel) and its excess entropy is an asymptotically logarithmic curve ($\propto 0.84899{\log }_{2}L$, in right panel). Thus, it is a LTM sequence. In fact, the logical structure of the TM sequence does not depend on the symbols that are used to represent it. The entropy growth $H\left(L\right)$ can be scaled as $H\left(L\right)\sim c_1+c_2{\log }_{2}L$.

4.6. Ising Model

The Ising model is a stochastic process model that describes the phase transitions of substances of the ferromagnetic system. The one-dimensional Ising model consists of discrete random variables with two states (pointing upwards and downwards). In this subsection, the excess entropy and LTM of a one-dimensional Ising model are investigated.

An one-dimensional Ising sequence can also be described as a binary sequence of 1 s and 0 s (pointing upwards and downwards respectively) with the Hamiltonian

$$H\left(\sigma \right)=-\sum _{s,t}{J}_{st}{\sigma }_s{\sigma }_t,$$

(11)

where $J_{st}$ denotes the interaction between spins s and t. The configuration $\sigma =\left({\sigma }_s\right)$ probability is given by the Boltzmann distribution with inverse temperature $\beta ⩾0$:

$$P_{\beta }\left(\sigma \right)=\exp (-\beta H\left(\sigma \right))/\sum _{\sigma }\exp (-\beta H\left(\sigma \right)).$$

(12)

Now, three different sequences of ${10}^9$ symbols each are investigated for different scales $J_{st}$, with an initial setup $\beta =1$. For the first sequence, only the nearest-neighbors are allowed to have nonzero interactions $J_{s,s+1}=1$, for all s. The second one also follows the nearest-neighbor interactions above, but the value is reset every $5\times {10}^5$ symbols by generating a random number from a standard normal distribution. The third one follows the long-ranged interactions at the same frequency as the second one, where the variances of interactions decrease with the distance between the symbols as $J_{st}^2=1/{(s-t)}^2$.

The entropy and excess entropy curves for all these cases are plotted in Figure 5. The left panel shows that all entropy for different interactions have asymptotically linear growth. However their excess entropy curves are very different from the varying interactions (in right panel). More vigorously, it is a constant for the first one; as for the second one, it is asymptotically logarithmic, $\propto c_1+c_2\log L$; and for the last one, it is asymptotically power-like, $\propto c_3+c_4{L}^{0.5}$ ($c_i$ are different constant). Thus, it follows that the first excess entropy is finite and the last two are infinite, and consequently that the last two Ising models are LTM. The readers are referred to [28,29] for other information.

4.7. Stationary Gaussian Sequence

Now, a stationary Gaussian sequence with a decreasing ACF $r_n=4\exp (-2n)$ is investigated by simulation. In terms of ACF, it is STM because of

$$\sum _n{r}_n=\sum _{n}4\exp (-2n)<+\infty .$$

The left panel of Figure 6 shows its entropy growth curve $H\left(L\right)$. For such a sequence, the entropy $H\left(L\right)$ grows linearly when L goes to a small number. The right panel of Figure 6 shows its excess entropy growth curve that it is nearly a constant. It is concluded that such a stationary Gaussian sequence with the decreasing $r_n=4\exp (-2n)$ is STM in the sense of excess entropy, which is consistent with the result of ACF.

4.8. Fractal Gaussian Noise

It is well known that a FGN with the parameter $H\in (0.5,1)$ is LTM while it is STM when $H\in (0,0.5)$ (cf. [30] and Equation (8)). Here, it will show that the corresponding excess entropy is infinite for $H\in (0.5,1)$, and finite for $H\in (0,0.5)$.

In simulations, without loss of generality, the Hurst parameters are varied from $H=0.1$ to $H=0.9$, by a step of $0.1$.

Figure 7 indicates that their entropy grows linearly. Figure 8 shows their excess entropies for $H⩽0.5$ (left panel) and $H>0.5$ (right panel), respectively. The left panel shows that their excess entropies for $H⩽0.5$ are constant, and right panel shows that each of them for $H>0.5$ can be fitted by a logarithmic curve ($\sim c_1+c_2\log L$). All those show that excess entropies are infinite for $H>0.5$ and finite for $H⩽0.5$, respectively.

It is concluded that a FGN with $H\in (0.5,1)$ is LTM and one with $H\in (0,0.5)$ is STM, in the sense of excess entropy.

Furthermore, it can be seen from Figure 8 that excess entropy from different Hurst parameters $H\in (0,0.5)$ are different constants, and those from $H\in (0.5,1)$ are asymptotically logarithmic curves with different coefficients. More precisely, for $H⩽0.5$, the larger the Hurst parameter, the smaller the excess entropy, i.e., the larger entropy rate (In particular, a standard Brownian motion for $H=0.5$ has an excess entropy of zero, i.e., an entropy rate of unitary); for $H>0.5$, the larger the Hurst parameter, the larger the coefficient of the logarithmic part of excess entropy; in other words, the larger the Hurst parameter, the faster the divergence.

It is also concluded that excess entropy can be applied to not only identify whether or not a FGN is LTM, but roughly discriminate the relative size of a Hurst parameter from two sequences. This also seems to provide another way to distinguish the Hurst parameter of FGN sequence. However, it requires further rigorous exploration.

Note that, viewed from the point of entropy growth, one can neither identify whether or not a FGN is LTM, nor straightforwardly discriminate the Hurst parameters (unless one has known that they are all LTM or STM, i.e., $H>0.5$ or $H⩽0.5$; a standard Brownian motion case of $H=0.5$ has the biggest entropy rate of unitary; when $H<0.5$, the bigger the Hurst parameter, the bigger the entropy rate; while $H>0.5$, the bigger the Hurst parameter, the smaller the entropy rate).

4.9. AR Model

The AR (autoregressive) model is applied in modeling and forecasting for time series, which shows the features with STM. In this subsection, the STM will be exhibited by way of excess entropy.

In terms of the classical ACF, however, it is impossible for an AR model with heavy tail noise, such as Levy noise, to characterize whether or not it is LTM, due to its infinite variance. Fortunately, an excess entropy approach can be applied to characterize it; see examples.

Figure 9 first presents the entropy and excess entropy curves for a stationary AR(2) model (two parameters of $0.7$ and $-0.12$) with a classical Gaussian noise (blue diamonds) and a standard Levy noise (red dots), which both show the features with STM.

Then, Figure 10 presents the entropy and excess entropy curves for another stationary AR(2) model (two parameters of $0.1$ and $0.3$) with a classical Gaussian noise (blue diamonds) and a standard Levy noise (red dots). The right panel of Figure 10 clearly shows that the excess entropy for Levy noise can be fitted by a logarithmic curve ($\propto 0.14471\log L$) and that such a model can exhibit a LTM.

It can be seen from above that the AR model with Levy noise may be LTM or STM. The issue of whether or not it is LMT seems to be associated with the parameters, i.e., the roots of the characteristic equation of the AR model (so does for the stationarity). What the criterion is for the roots of the characteristic equation, however, still remains an open problem.

Note that the above AR(2) models with standard Levy noise are stationary according to the Non-stationary Measurement method [27]. These two examples show the significance that such an approach can be applied to explore the LTM behavior of stationary sequences with an unbounded second moment and further it is possible to explore its stationary parameter domain and LTM parameter domain.

4.10. Summary of Simulations

First, one consequence is that the entropy growth for a stationary sequence can be scaled as

$$H\left(L\right)\sim h_{\mu }L+E\left(L\right)$$

(13)

with $E\left(L\right)=c_1+c_2\log L$ or $E\left(L\right)=c_1+c_2{L}^{\alpha }(\alpha <1)$, in the $L\to \infty$ limit, where $c_1$ and $c_2$ are constants, the entropy rate $h_{\mu }$ might be equal to zero. In particular, $c_2=0$ and $c_1=EE$ for STM sequence.

Specifically, the entropy growth for STM sequence scales as $H\left(L\right)\sim EE+h_{\mu }L$, where $EE$ is a constant (i.e., its excess entropy), independent of L. In contrast, a LTM sequence might be scaled as

$$H\left(L\right)\sim h_{\mu }L+c_1+c_2\log L,$$

(14)

or

$$H\left(L\right)\sim h_{\mu }L+c_1+c_2{L}^{\alpha },\alpha <1.$$

(15)

Thus, it follows that the entropy growth curve grows linearly for most of sequences, unless its entropy rate is zero such as periodic sequence (trivial one) and TM sequence ($H\left(L\right)\sim c_1+c_2\log L$).

In addition to maintaining the consistency with the traditional methods on the classical sequences, such an approach has some advantages over the traditional second-moment ways: one is that the excess entropy of a stationary sequence with finite block entropy is invariant under instantaneous one-to-one transformation. The second one is that it only requires very weak moment conditions rather than second-order moment conditions and thus can be applied to investigate whether or not a stationary sequence with unbounded second moment is LTM; for example, such an approach can investigate the LTM behavior of AR models with Levy noise, while it is impossible to be implemented in the sense of ACF.

In general, for real data with a not large sample size (most of sample data, about ${10}^3$–${10}^4$), the entropy growth curve can also be graphically applied to identify whether or not a sequence is LTM.

This argument can be interpreted by discussing the relationships between the entropy growth curve and the sample size from the sequence with the LTM or STM behavior.

As stated earlier, most entropy growth curves remain linear. A natural question is whether this linearity still holds as the length L of the block increases for a not very large sample size. It is explored by a simulation with $2^{10}=1024$ samples corresponding to one in Section 4.9. Figure 11 presents the entropy for the second AR(2) model with Gaussian noise (blue diamonds) and standard Levy noise (red dots).

The red curve shows that the entropy with LTM approximately keeps growing linearly. In contrast, the blue one shows that a STM case grows linearly up to $L=10$, while the linearity quickly vanishes to a larger length of $L>10$ (reaches to a threshold value). This phenomenon is a consequence of the limited samples whenever counting those frequencies provides a bad empirical estimate of the probability distribution of different blocks of length L. For example, the theoretical probability of each block of large L for a fair coin sequence, $0.5^L$, is very different from the frequency, at least $1/n$, from n samples; for a practical sequence with LTM behavior, however, either the probability of a block of length L is 0 or far bigger than $0.5^L$ and then it can be estimated by the frequency.

In fact, the phenomenon also implies another auxiliary approach to visually identify by the entropy curve whether or not a sequence is LTM. For a STM sequence, in general, the entropy curve grows linearly up to $L_0$ (depending on the sample size n and the number of partitioning) and then vanishes gradually with $L>L_0$; in contrast to a LTM sequence, it still keeps the linearity. There are exceptions if the entropy converges rapidly to a constant for STM sequence such as periodic case, and zero entropy rate ($h_{\mu }=0$) sequence for LTM, which it grows logarithmically such as a TM sequence. However, the entropy growth curve is only an auxiliary graphical approach, and the final conclusion still needs to be determined by excess entropy.

5. Application

Now some real data are applied to show the implementation of the excess entropy approach.

We started by utilizing hydrological data from a river for about 10 months (with a sample size of 63,537). River level trends can be a hot issue. Often the original data can be normalized into a new sequence of three states (1 s, $-1$ s and 0 s, pointing to rising stage, falling stage and unchanging stage, respectively). Further, the entropy and excess entropy growth curves are shown in Figure 12. The left panel shows that the block entropy is an asymptotically linear growth curve and it might be LTM. The right panel further shows that the excess entropy growth appears to be a LTM trend, which implies that its historical data play a very important role in the forecasting stage of this river.

Two sets of data are then explored. One is from the peak gamma-ray intensity of solar flares between 1980 and 1989. Another is from the Richter magnitudes of earthquakes occurring in California between 1910 and 1992. Those are adapted from [31] and they follow a power law. According to Non-stationary Measurement method [27], the peak gamma-ray intensity (resp. the earthquake magnitude) data are finally normalized into a new sequence of four states (resp. two states). The excess entropy growth curves (blue asterisks) and the logarithmic estimates (red plus) for those two sets of data are shown in Figure 13, where they both exhibit the LTM behaviors in the sense of excess entropy. The results are basically coincident with the practical data. Actually, the so-called Richter magnitude was defined as the logarithm of the maximum amplitude of motion of the earthquake, and hence it is a logarithmic scale of amplitude [27].

Several applications on real data show that the proposed approach is powerful and effective.

6. Discussion

In the current paper, an excess entropy approach is proposed to characterize LTM behavior. It is concluded that a stationary sequence with finite block entropy is LTM if its excess entropy is infinite. By presenting a range of theoretical and simulation results, such an approach has advantages over the traditional ways that the excess entropy of stationary sequence with finite block entropy is invariant under instantaneous one-to-one transformation, and that it only requires very weak moment conditions rather than second-order moment conditions and thus can be applied to distinguish the LTM behavior of stationary sequences with an unbounded second moment (e.g., heavy tail distribution). In particular, for a stationary Gaussian sequence, the LTM in the sense of excess entropy is stricter than that in the sense of ACF. For this reason, stationary Gaussian sequence prediction with LTM in the sense of ACF might be simplified and improved, provided that it has been detected as a STM by the proposed approach.

As stated in Section 4.9, it is possible to characterize the LTM behavior of the AR model with heavy tail noise. It is well known that for AR(2) model with the classical Gaussian noise, it is stationary if the characteristic equation has a unit root. So, what about the AR model with heavy tail noise? In the near future, we might shed light on the AR model with heavy tail noise (e.g., Levy noise and fractional Gaussian noise), where the corresponding stationary parameter domain and LTM parameter domain are explored. On the other hand, the prediction model based on excess entropy might be developed to improve the accuracy of time series prediction.