Kernel Methods for Nonlinear Connectivity Detection

Abstract

In this paper, we show that the presence of nonlinear coupling between time series may be detected using kernel feature space $\mathbb{F}$ representations while dispensing with the need to go back to solve the pre-image problem to gauge model adequacy. This is done by showing that the kernelized auto/cross sequences in $\mathbb{F}$ can be computed from the model rather than from prediction residuals in the original data space $\mathbb{X}$. Furthermore, this allows for reducing the connectivity inference problem to that of fitting a consistent linear model in $\mathbb{F}$ that works even in the case of nonlinear interactions in the $\mathbb{X}$-space which ordinary linear models may fail to capture. We further illustrate the fact that the resulting $\mathbb{F}$-space parameter asymptotics provide reliable means of space model diagnostics in this space, and provide straightforward Granger connectivity inference tools even for relatively short time series records as opposed to other kernel based methods available in the literature.

1. Introduction

Describing ‘connectivity’ has become of paramount interest in many areas of investigation that involve interacting systems. Physiology, climatology, and economics are three good examples where dynamical evolution modelling is often hindered as system manipulation may be difficult or unethical. Consequently, interaction inference is frequently constrained to using time observations alone.

A number of investigation approaches have been put forward [1,2,3,4,5]. However, the most popular and traditional one still is the nonparametric computation of cross-correlation (CC) between pairs of time series, and variants thereof, like coherence analysis [6], even despite their many shortcomings [7].

When it comes to connectivity analysis, recent times have seen the rise of Granger Causality (GC) as a unifying concept. This is mostly due to GC’s unreciprocal character [8] (as opposed to CC) which allows for establishing the direction of information flow between component subsystems.

Most GC approaches rest on fitting parametric models to time series data and, again as opposed to CC, under appropriate conceptualization, also holds for more than just pairs of time series, giving rise to the ideas of (a) Granger connectivity and (b) Granger influentiability [9].

GC inferential methodology is dominated by the use of linear multivariate time series models [10]. This is so because linear models have statistical properties (and shortcomings) that are well understood besides having the advantage of sufficing when the data are Gaussian. As an added advantage GC characterization allows immediate frequency domain connectivity characterization via concepts like ‘directed coherence’ (DC) and ‘partial directed coherence’ (PDC) [11].

It is often the case, however that data Gaussianity does not hold. Whereas nonparametric approaches do exist [1,4,5], parametric nonlinear modelling offers little relief from the need for long observation data sets for reliable estimation in sharp contrast to linear models that perform well under the typical scenario of fairly short datasets over which natural phenomena can be considered stable. A case in point is neural data where animal behaviour changes are associated with relatively short-lived episodic signal modifications.

The motivation for the present development is that reproducing kernel transformations applied to data, as in the support vector machine learning classification case [12], can effectively produce estimates that inherit many of the good convergence properties of linear methods. Because these properties carry over under proper kernelization, it is possible to show that nonlinear links between subsystems can be rigorously detected.

Before proceeding, it is important to have in mind that the present developments focus on addressing the connectivity detection issue exclusively, in which we clearly show that solving the so-called pre-image reconstruction problem is unnecessary as has been until now assumed essential. This leads to a considerably simpler approach.

In Section 2, we formulate the problem and review some background about reproducing kernel theory together with the main results which are backed up by extensive numerical Monte Carlo illustrations in Section 3. Conclusions and current problem status and challenges end the paper (Section 4).

2. Problem Formulation

The most popular approach to investigating GC connectivity is through modeling multivariate time series via linear vector autoregressive models [10], where the central idea is to compare prediction effectiveness for a time series $x_i\left(n\right)$ when the past of other time series is taken into account in addition to its own past. Namely,

$$\mathbf{x}\left(n\right)=\sum _{k=1}^p{\mathbf{A}}_k\mathbf{x}(n-k)+\mathbf{w}\left(n\right).$$

(1)

Under mild conditions, Equation (1) constitutes a valid representation of a linear stationary stochastic process where the evolution of $\mathbf{x}\left(n\right)={[x_1\left(n\right),\cdots ,x_D\left(n\right)]}^{\top }$ is obtained by filtering suitable $\mathbf{w}\left(n\right)={[w_1\left(n\right),\cdots ,w_D\left(n\right)]}^{\top }$ purely stochastic innovation processes, i.e., where $w_i\left(n\right)$ and $w_j\left(m\right)$ are independent provided $n\ne m$ [13]. If $w_i\left(n\right)$ are jointly Gaussian, so are $x_i\left(n\right)$ and the problem of characterizing connectivity reduces to well known procedures to estimate the ${\mathbf{A}}_k$ parameters in Equation (1) via least squares, which is the applicable maximum likelihood procedure. Nongaussian $w_i\left(n\right)$ translate into nongaussian $x_i\left(n\right)$ even if some actual (1) linear generation mechanism holds. Linearity among nongaussian $x_i\left(n\right)$ time series may be tested with help of cross-polyspectra [14,15], which, if unrejected, still allows for a representation like (1) whose optimal estimation requires a suitable likelihood function to accommodate the observed non-Gaussianity.

If linearity is rejected, $x_i\left(n\right)$ non-Gaussianity is a sign of nonlinear mechanisms of generation modelled by

$$\mathbf{x}\left(n\right)=\mathbf{g}(\mathbf{x}\left(n_{-}\right),\mathbf{w}\left(n\right)),$$

(2)

which generalizes (1) where $\mathbf{x}\left(n_{-}\right)$ stands for $\mathbf{x}\left(n\right)$’s past under some suitable dynamical law $\mathbf{g}(·)$.

The distinction between (a) nonlinear $x_i\left(n\right)$ that are nonetheless linearly coupled as in (1) under nongaussian $\mathbf{w}\left(n\right)$ and (b) fully nonlinearly coupled processes is often overlooked. In the former case, linear methods suffice for connectivity detection [16] but fail in the latter case [17] calling for the adoption of alternative approaches. In some cases, however, linear approximations are inadequate in so far as to preclude connectivity detection [17].

In the present context, the solution to the connectivity problem entails a suitable data driven approximation of $\mathbf{g}(·)$ whilst singling out the $x_i\left(n\right)$ and $x_j\left(n\right)$ of interest. To do so, we examine the employment of kernel methods [18] where functional characterization is carried out with the help of a high dimensional space representation

$$\mathbf{\varphi }:\mathbb{X}\to \mathbb{F},$$

(3)

for $F=\dim \left(\mathbb{F}\right)\gg D=\dim (\mathbb{X}$), where $\mathit{\varphi }\left(\mathbf{x}\right(n\left)\right)$ is a mapping from the input space $\mathbb{X}$ into the feature space $\mathbb{F}$ whose role is to properly unwrap the data and yet ensure that the inner product $\langle \mathit{\varphi }\left(\mathbf{x}\right)\left|\mathit{\varphi }\right(\mathbf{y})\rangle$ can be written as a simple function of $\mathbf{x}$ and $\mathbf{y}$ dispensing with the need for computations in $\mathbb{F}$. This possibility is granted by chosing $\mathit{\varphi }\left(\mathbf{x}\right)$ to satisfy the so-called Mercer condition [19].

A simple example of (3) is the mapping

$$\varphi :x\mapsto \langle \varphi \left(x\right)|={[c,\sqrt{2c}x,x^2]}^{\top },$$

(4)

for $x\in \mathbb{R}$ and $\langle \varphi \left(x\right)|\in \mathbb{F}$ using Dirac’s bra-ket notation. In this case, the Mercer kernel is given by

$$\kappa (x,y)=\langle \mathbf{\varphi }\left(x\right)|\mathbf{\varphi }\left(y\right)\rangle ={(c+xy)}^2,$$

(5)

which is the simplest example of a polynomial kernel [18].

In the multivariate time series case, we consider

$$\mathbf{\varphi }:\mathbf{x}\left(n\right)\mapsto [\langle {\varphi }_1\left(x_1\left(n\right)\right)|,\cdots ,\langle {\varphi }_i\left(x_i\left(n\right)\right)|,\cdots ,{\langle {\varphi }_D\left(x_D\left(n\right)\right)\left|\right]}^{\top },$$

(6)

where, for simplicity, we adopt the same transformation $\varphi (·)={\varphi }_i(·)={\varphi }_j(·)$ for each $x_i\left(n\right)\in \mathbb{R}$ time series component so that the

$$\langle \mathbf{\varphi }\left(\mathbf{x}\right(n\left)\right)\left|\mathbf{\varphi }\right(\mathbf{x}\left(m\right))\rangle =\mathbf{K}(\mathbf{x}\left(n\right),\mathbf{x}\left(m\right))$$

(7)

is a matrix whose elements are given by $K_{ij}(n,m)=\langle \varphi \left(x_i\left(n\right)\right)|\varphi \left(x_j\left(m\right)\right)\rangle$. In the development below, we follow the standard practice of denoting the $\mathbf{K}\left(\mathbf{x}\right(n),\mathbf{x}(m\left)\right)$ quantities as $\mathbf{K}(m-n)$ in view of the assumed stationarity of the processes under study.

Rather than go straight into the statement of the general theory, a simple example is more enlightening. In this sense, consider a bivariate stationary time series

$$\begin{array}{ccc}\hfill x_1\left(n\right)& =& g_1(x_1(n-1),w_1\left(n\right)),\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill x_2\left(n\right)& =& g_2(x_1(n-1),x_2(n-1),w_2\left(n\right)),\hfill \end{array}$$

(9)

where $g_i(·)$ are nonlinear functions and only the previous instant is relevant in producing the present behaviour. An additional feature, thru (9), is that $x_1\left(n\right)$ is connected to (Granger causes) $x_2\left(n\right)$ but not conversely. Application of the kernel transformation leads to

$$\begin{array}{ccc}\hfill \langle \varphi \left(x_1\left(n\right)\right)|& =& \langle \varphi \left(g_1(x_1(n-1),w_1\left(n\right))\right)|,\hfill \end{array}$$

(10)

$$\begin{array}{ccc}\hfill \langle \varphi \left(x_2\left(n\right)\right)|& =& \langle \varphi \left(g_2(x_1(n-1),x_2(n-1),w_2\left(n\right))\right)|.\hfill \end{array}$$

(11)

However, if one assumes the possibility of a linear approximation in $\mathbb{F}$, one may write

$$\left[\begin{array}{c}\langle \varphi \left(x_1\left(n\right)\right)|\\ \langle \varphi \left(x_2\left(n\right)\right)|\end{array}\right]=\left[\begin{array}{cc}{\alpha }_{11}& {\alpha }_{12}\\ {\alpha }_{21}& {\alpha }_{22}\end{array}\right]\left[\begin{array}{c}\langle \varphi \left(x_1(n-1)\right)|\\ \langle \varphi \left(x_2(n-1)\right)|\end{array}\right]+\left[\begin{array}{c}\langle {\tilde{w}}_1\left(n\right)|\\ \langle {\tilde{w}}_2\left(n\right)|\end{array}\right],$$

(12)

where $[\langle {\tilde{w}}_1\left(n\right)|{\langle {\tilde{w}}_2\left(n\right)\left|\right]}^{\top }$ stands for approximation errors in the form of innovations. Mercer kernel theory allows for taking the external product with respect to $\left[\right|\varphi \left(x_1(n-1)\right)\rangle |\varphi \left(x_2(n-1)\right){\rangle ]}^{\top }$ on both sides of (12) leading to

$$\mathbf{K}\left(\mathbf{x}\right(n),\mathbf{x}(n-1\left)\right)=\mathbf{A}\mathbf{K}\left(\mathbf{x}\right(n-1),\mathbf{x}(n-1\left)\right),$$

(13)

after taking expectations on both sides where

$$\mathbf{A}=\left[\begin{array}{cc}{\alpha }_{11}& {\alpha }_{12}\\ {\alpha }_{21}& {\alpha }_{22}\end{array}\right]$$

(14)

and

$$\mathbf{K}(\mathbf{x}\left(n\right),\mathbf{x}\left(m\right))=\left[\begin{array}{cc}\mathbb{E}\left[\langle \varphi \left(x_1\left(n\right)\right)|\varphi \left(x_1\left(m\right)\right)\rangle \right]& \mathbb{E}\left[\langle \varphi \left(x_1\left(n\right)\right)|\varphi \left(x_2\left(m\right)\right)\rangle \right]\\ \mathbb{E}\left[\langle \varphi \left(x_2\left(n\right)\right)|\varphi \left(x_1\left(m\right)\right)\rangle \right]& \mathbb{E}\left[\langle \varphi \left(x_2\left(n\right)\right)|\varphi \left(x_2\left(m\right)\right)\rangle \right]\end{array}\right],$$

(15)

since $\mathbb{E}\left[\langle {\tilde{w}}_i\left(n\right)|\varphi \left(x_j\left(m\right)\right)\rangle \right]=0$ for $n>m$ given that $\langle {\tilde{w}}_i\left(n\right)|$ plays a zero mean innovations role.

It is easy to obtain $\mathbf{A}$ from sample kernel estimates. Furthermore, it is clear that (8) holds if and only if ${\alpha }_{12}=0$.

To (13), which plays the role of Yule–Walker equations and which can be written more simply as

$$\mathbf{K}(-1)=\mathbf{A}\mathbf{K}\left(0\right),$$

(16)

and one may add the following equation to compute the innovations covariance

$${\mathsf{\Sigma }}_{\langle \tilde{\mathbf{w}}\left(n\right)|}=\mathbf{K}(\mathbf{x}\left(n\right),\mathbf{x}\left(n\right))-\mathbf{A}\mathbf{K}(\mathbf{x}\left(n\right),\mathbf{x}\left(n\right)){\mathbf{A}}^{\top },$$

(17)

where only reference to the $m-n$ difference is explicitly denoted assuming signal stationarity so that (17) simplifies to

$${\mathsf{\Sigma }}_{\langle \tilde{\mathbf{w}}\left(n\right)|}=\mathbf{K}\left(0\right)-\mathbf{A}\mathbf{K}\left(0\right){\mathbf{A}}^{\top }=\mathbf{K}\left(0\right)-\mathbf{K}(-1){\mathbf{A}}^{\top }.$$

(18)

This formulation is easy to generalize to model orders $p>1$ and to more time series via

$$\langle \mathbf{\varphi }\left(\mathbf{x}\left(n\right)\right)|=\sum _{k=1}^p{\mathbf{A}}_k\langle \mathbf{\varphi }\left(\mathbf{x}(n-k)\right)|+\langle \tilde{\mathbf{w}}\left(n\right)|,$$

(19)

where

$$\langle \mathbf{\varphi }\left(\mathbf{x}\left(n\right)\right)|=[\langle \varphi \left(x_1\left(n\right)\right)|,\cdots ,{\langle \varphi \left(x_D\left(n\right)\right)\left|\right]}^{\top },$$

(20)

which is assumed as due to filtering appropriately modelled innovations $\langle \tilde{\mathbf{w}}\left(n\right)|$. For the present formulation, one must also consider the associated ‘ket’-vector

$$|\mathbf{\varphi }\left(\mathbf{x}\left(m\right)\right)\rangle =\left[\right|\varphi \left(x_1\left(m\right)\right)\rangle ,\cdots ,|\varphi \left(x_D\left(m\right)\right){\rangle ]}^{\top },$$

(21)

that when applied to (19) for $n>m$ after taking expectations $\mathbb{E}[·]$ under the zero mean innovations nature of $\langle {\mathbf{w}}^{\varphi }\left(n\right)|$ leads to

$${\mathbf{K}}_{\mathbf{x}}^{\varphi }\left(l\right)=\sum _{k=1}^p{\mathbf{A}}_k{\mathbf{K}}_{\mathbf{x}}^{\varphi }(l+k),$$

(22)

where $l=m-n$ and ${\mathbf{K}}_{\mathbf{x}}^{\varphi }\left(m\right)$’s elements are given by $\mathbb{E}\left[\langle \varphi \left(x_i(l-m)\right)|\varphi \left(x_j\left(l\right)\right)\rangle \right]$ so that (22) constitutes a generalization of the Yule–Walker equations. By making $l=m-n=-1,\cdots ,-p$ one may reframe (22) in matrix form as

$${\overline{\mathbf{\kappa }}}_p=\left[\begin{array}{c}{\mathbf{K}}_{\mathbf{x}}^{\varphi }(-1)\\ ⋮\\ {\mathbf{K}}_{\mathbf{x}}^{\varphi }(-p)\end{array}\right]=\left[{\mathbf{A}}_1\cdots {\mathbf{A}}_p\right]\left[\begin{array}{cccc}{\mathbf{K}}_{\mathbf{x}}^{\varphi }\left(0\right)& {\mathbf{K}}_{\mathbf{x}}^{\varphi }(-1)& \cdots & {\mathbf{K}}_{\mathbf{x}}^{\varphi }(-p+1)\\ {\mathbf{K}}_{\mathbf{x}}^{\varphi }\left(1\right)& {\mathbf{K}}_{\mathbf{x}}^{\varphi }\left(0\right)& \cdots & {\mathbf{K}}_{\mathbf{x}}^{\varphi }(-p+2)\\ ⋮& \ddots & \ddots & ⋮\\ {\mathbf{K}}_{\mathbf{x}}^{\varphi }(p-1)& {\mathbf{K}}_{\mathbf{x}}^{\varphi }(p-2)& \cdots & {\mathbf{K}}_{\mathbf{x}}^{\varphi }\left(0\right)\end{array}\right]=\mathcal{A}{\mathcal{K}}_p\left(0\right),$$

(23)

where ${\mathcal{K}}_p\left(0\right)$ is block Toeplitz matrix containing p Toeplitz blocks. Equation (23) provides $p{D}^2$ equations for the same number of unknown parameters in $\mathcal{A}$.

The high model order counterpart to (17) is given by

$${\mathsf{\Sigma }}_{\langle \tilde{\mathbf{w}}\left(n\right)|}={\mathbf{K}}_{\mathbf{x}}^{\varphi }\left(0\right)-\sum _{k=1}^p\sum _{l=1}^p{\mathbf{A}}_k{\mathbf{K}}_{\mathbf{x}}^{\varphi }(k-l){\mathbf{A}}_l^{\top }={\mathbf{K}}_{\mathbf{x}}^{\varphi }\left(0\right)-\mathcal{A}{\mathcal{K}}_p\left(0\right){\mathcal{A}}^{\top }.$$

(24)

It is not difficult to see that the more usual Yule–Walker complete equation form becomes

$$[\mathbf{I}-\mathcal{A}]{\mathcal{K}}_{p+1}\left(0\right)=\left[\begin{array}{c}{\mathsf{\Sigma }}_{\langle \tilde{\mathbf{w}}\left(n\right)|}\\ \mathbf{0}\end{array}\right].$$

(25)

There are a variety of ways for solving for the parameters. A simple one is to define $\mathbf{a}=\mathrm{vec}\left(\mathcal{A}\right)$ leading to

$$\mathrm{vec}\left({\overline{\mathbf{\kappa }}}_p\right)=({\mathcal{K}}_p^{\top }\left(0\right)\otimes \mathbf{I})\mathbf{a}.$$

(26)

Even though one may employ least-squares solution methods to solve either (26) or (23), a Total-Least-Squares (TLS) approach [20] has proven a better solution since both members of the equations are affected by estimation inaccuracies that are better dealt with using TLS.

Likewise, (24) can be used in conjunction with generalizations of model order criteria of Akaike’s AIC type

$${}_{\mathrm{g}}\mathrm{AIC}\left(k\right)=\ln (det\left({\mathsf{\Sigma }}_{\langle \tilde{\mathbf{w}}\left(n\right)|}\right))+{\displaystyle \frac{c_{n_s}}{n_s}}k{D}^2,$$

(27)

where $n_s$ stands for the number of available time observations. In generalizing Akaike’s criterion to the multivariate case $c_{n_s}=2$, whereas $c_{n_s}=\ln \left(\ln \left(n_s\right)\right)$ for the Hannan–Quinn criterion, our choice in this paper.

Thus far, we have described procedures for choosing model order in the $\mathbb{F}$ space. In ordinary time series analysis, in addition to model order identification, one must also perform proper model diagnostics. This entails checking for residual whiteness among other things. This is usually done by checking the residual auto/crosscorrelation functions for their conformity to a white noise hypothesis.

In the present formulation because, we do not explicitly compute the $\mathbb{F}$ space series, we must resort to means other than computing the latter correlation functions from the residual data as usual. However, using the same ideas for computing (24), one may obtain estimates of the innovation cross-correlation in the feature space at various lags as

$${\mathsf{\Sigma }}_{\langle \tilde{\mathbf{w}}\left(n\right)|\tilde{\mathbf{w}}\left(m\right)\rangle }={\mathsf{\Sigma }}_{\tilde{\mathbf{w}}}(m-n)={\mathbf{K}}_{\mathbf{x}}^{\varphi }(m-n)-\sum _{k=1}^p\sum _{l=1}^p{\mathbf{A}}_k{\mathbf{K}}_{\mathbf{x}}^{\varphi }(m-n+k-l){\mathbf{A}}_l^{\top },$$

(28)

by replacing ${\mathbf{K}}_{\mathbf{x}}^{\varphi }(m-n+k-l)$ by their estimates and using ${\mathbf{A}}_k$ obtained by solving (22) for $m-n$ between a minimum $-L$ to a $+L$ maximum lag. The usefulness of (28) is to provide means to test model accuracy and quality as a function of $\varphi$ choice under the best model order provided by the model order criterion.

By defining a suitable normalized estimated lagged kernel correlation function (KCF)

$${\mathrm{KCF}}_{ij}\left(\tau \right)={\displaystyle \frac{K_{ij}\left(\tau \right)}{\sqrt{K_{ii}\left(0\right)K_{jj}\left(0\right)}}},$$

(29)

which, given the inner product nature of kernel definition, satisfies the condition

$$|{\mathrm{KCF}}_{ij}\left(\tau \right)|\le 1,$$

(30)

as easily proved using the Cauchy–Schwarz inequality.

The notion of $\mathrm{KFC}\left(\tau \right)$ applies not only to the original kernels but also in connection with the residual kernel values given by (28) where, for explicitness, we write it as

$${\mathrm{KCF}}_{ij}^{\left(r\right)}\left(\tau \right)={\displaystyle \frac{{\mathsf{\Sigma }}_{ij}\left(\tau \right)}{\sqrt{{\mathsf{\Sigma }}_{ii}\left(0\right){\mathsf{\Sigma }}_{jj}\left(0\right)}}},$$

(31)

where ${\mathsf{\Sigma }}_{ij}\left(\tau \right)$ are the matrix entries in (28).

In the numerical illustrations that follow, we have assumed that ${\mathrm{KCF}}_{ij}^{\left(r\right)}\left(\tau \right)\sim \mathcal{N}(0,1/n_s)$ asymptotically under the white residual hypothesis

$${\mathcal{H}}_0:{\mathrm{KCF}}_{ij}^{\left(r\right)}\left(\tau \right)=0.$$

(32)

This choice turned out to be reasonably consistent in practice. Along the same line of reasoning, other familiar tests over residuals, such as the Portmanteau test [10] were also carried out and consistently allowed verifying residual nonwhiteness.

One may say that the present theory follows closely the developments of ordinary second order moment theory with the added advantage that now nonlinear connections can be effectively captured by replacing second order moments by their respective lagged kernel estimates.

2.1. Estimation and Asymptotic Considerations

The essential problem then becomes that of estimating the entries of ${\mathbf{K}}_{\mathbf{x}}^{\varphi }(n,m)$, entries. They can be obtained by averaging kernel values computed over the available data

$$K_{ij}(n,m)={\displaystyle \frac{1}{n_s}}\sum _s\langle \varphi \left(x_i(n-s)\right)|\varphi \left(x_j(m-s)\right)\rangle ,$$

(33)

for nonzero terms in the $s\in [1,n_s]$ range.

Under these conditions, for an appropriately defined kernel function, the feature space becomes linearized and, following [21], it is fair to assume that the estimated vector stacked representation of the model coefficient matrices

$$\mathbf{a}=\mathrm{vec}\left([{\mathbf{A}}_1\cdots {\mathbf{A}}_p]\right)$$

(34)

is asymptotically Gaussian, i.e.,

$$\sqrt{n_s}(\widehat{\mathbf{a}}-\mathbf{a})\sim \mathcal{N}(\mathbf{0},{\mathsf{\Gamma }}^{-1}\otimes {\mathsf{\Sigma }}_{\langle \tilde{\mathbf{w}}\left(n\right)|}),$$

(35)

where ${\mathsf{\Sigma }}_{\langle \tilde{\mathbf{w}}\left(n\right)|}$ is the feature space residual matrix given by (28) and where

$$\mathsf{\Gamma }=\mathbb{E}\left[{\mathbf{y}}_L{\mathbf{y}}_R^{\top }\right]$$

(36)

for the ‘bra’-vector

$${\mathbf{y}}_L^{\top }={[\langle \mathbf{\varphi }\left(\mathbf{x}\left(n\right)\right)|,\cdots ,\langle \mathbf{\varphi }\left(\mathbf{x}(n-p+1)\right)\left|\right]}^{\top }$$

(37)

and the ‘ket’-vector

$${\mathbf{y}}_R^{\top }={\left[\right|\mathbf{\varphi }\left(\mathbf{x}\left(n\right)\right)\rangle ,\cdots ,|\mathbf{\varphi }\left(\mathbf{x}(n-p+1)\right)\rangle ]}^{\top },$$

(38)

which are used to construct the kernel scalar products. It is immediate to note that (36) is a Toeplitz matrix composed of suitably displaced ${\mathbf{K}}_{\mathbf{x}}^{\varphi }(·)$ blocks.

An immediate consequence of (35) is that one may test for model coefficient nullity and thereby provide a kernel Granger Causality test. This is equivalent to testing for $a_{ij}\left(k\right)=0$ so that the statistic

$${}_g{\lambda }_W={\widehat{\mathbf{a}}}^{\top }{\mathbf{C}}^{\top }{\left[\mathbf{C}\left({\mathsf{\Gamma }}^{-1}\otimes {\mathsf{\Sigma }}_{\langle \tilde{\mathbf{w}}\left(n\right)|}\right){\mathbf{C}}^{\top }\right]}^{-1}\mathbf{C}\widehat{\mathbf{a}},$$

(39)

where $\mathbf{C}$ is a contrast matrix (or structure selection matrix) so that the null hypothesis becomes

$${\mathcal{H}}_0:\mathbf{C}\mathbf{a}=\mathbf{0}.$$

(40)

Hence, under (35),

$${}_g{\lambda }_W\stackrel{d}{\to }{\chi }_{\nu }^2,$$

(41)

where $\nu =\mathrm{rank}\left(\mathbf{C}\right)$ corresponds to the number of the explicitly imposed constraints on $a_{ij}\left(k\right)$.

Data Workflow

Given $x_i\left(n\right)$, analysis proceeds by

• Computing the kernel values (33) to obtain the kernel Yule–Walker equations (25) or equivalently (26) for a given value of p (starting from $p=1$);
• After solving the latter for the parameters in $\mathbf{a}$ via Total-Least-Squares (TLS), one computes (18) wherefrom the generalized model order choice criterion (27) can be computed;
• With the help of the computed (33) values, one can obtain the residual ${\mathrm{KCF}}_{ij}^{\left(r\right)}\left(\tau \right)$ functions in (31) which can be used to check model adequacy via (32). Additionally, Portmanteau tests may be also used;
• If ${\mathrm{KCF}}_{ij}^{\left(r\right)}\left(\tau \right)$ analysis does not suggest feature space model residual whiteness, p is increased by 1, and the procedure from step 1 is repeated until feature space model residual whiteness is obtained and ${}_g\mathrm{AIC}\left(k\right)$ attains its first local minimum meaning that the ideal model order has been reached;
• Once the best model is attained, one employs the (39) to infer connectivity.

These steps closely mirror those of ordinary time series model fitting and analysis.

3. Numerical Illustrations

The following examples consist of nonlinearly coupled systems that are simulated with the help of zero mean unit variance normal uncorrelated innovations $w_i\left(n\right)$. All simulations (10,000 realizations each) were preceded by an initial a burn-in period of 10,000 data points to avoid transient phenomena. Estimation results are examined as a function of $n_s=\{32,64,128,256,512,1024,2048\}$ with $\alpha =1\%$ significance.

For brevity, Example 1 is carried out in full detail, whereas approach performance for the other ones is gauged mostly through the computation of observed detection rates except for Examples 4 and 5 which also portray model order choice criterion behaviour.

Simulation results are displayed in terms of how true and false detection rates depend on realization length $n_s$.

3.1. Example 1

Consider the simplest possible system whose connectivity cannot be captured by linear methods [17] as there is a unidirectional quadratic coupling from $x_2\left(n\right)$ to $x_1\left(n\right)$

$$\left\{\begin{array}{cc}\hfill x_1\left(n\right)& =a{x}_1(n-1)+c{x}_2^2(n-1)+w_1\left(n\right),\hfill \\ \hfill x_2\left(n\right)& =b{x}_2(n-1)+w_2\left(n\right),\hfill \end{array}\right.$$

(42)

with $a=0.2$, $b=0.6$ and $c=0.7$.

An interesting aspect of this simple system is the possibility of easily relating its coefficients a, b and c to those in (14) that describe its $\mathbb{F}$ space evolution. This may be carried out explicitly after substituting (42) into the computed kernels of Equation (13). After a little algebra, this leads to

$$\left[\left[\begin{array}{cc}a& 0\\ 0& b\end{array}\right]-\left[\begin{array}{cc}{\alpha }_{11}& {\alpha }_{12}\\ {\alpha }_{21}& {\alpha }_{22}\end{array}\right]\right]=\left[\begin{array}{cc}c& 0\\ 0& 0\end{array}\right]\left[\begin{array}{cc}{\theta }_{11}& {\theta }_{12}\\ 0& 0\end{array}\right],$$

(43)

where ${\theta }_{11}$ and ${\theta }_{12}$ depend on the computed kernel values. From (43), it immediately follows for example that $b={\alpha }_{22}$ and more importantly that ${\alpha }_{21}=0$ as expected. Vindication of the observation of these theoretically determined values also gives the means for testing estimation accuracy.

For illustrations’ sake, we write the kernel Yule–Walker Equations (22) with their respective solutions ($n_s=512$) for one given (typical) realization

$$\left[\begin{array}{cc}210.7583& 23.5416\\ 23.5416& 8.6450\end{array}\right]{\mathbf{A}}^{\left(2\right)}=\left[\begin{array}{cc}125.7501& 37.7803\\ 17.7389& 5.3788\end{array}\right]\to {\mathbf{A}}^{\left(2\right)}=\left[\begin{array}{cc}0.1559& 3.9456\\ 0.0211& 0.5648\end{array}\right],$$

(44)

for the quadratic kernels ($\kappa (x,y)={\left(xy\right)}^2$) and

$${10}^5\times \left[\begin{array}{cc}8.0302& 0.0868\\ 0.0868& 0.0052\end{array}\right]{\mathbf{A}}^{\left(4\right)}={10}^5\times \left[\begin{array}{cc}4.1597& 0.1755\\ 0.0594& 0.0025\end{array}\right]\to {\mathbf{A}}^{\left(4\right)}=\left[\begin{array}{cc}0.1843& 30.8922\\ 0.0027& 0.4386\end{array}\right],$$

(45)

for the quartic kernels ($\kappa (x,y)={\left(xy\right)}^4$). Superscripts point to kernel order. One may readily notice approximate compliance to the expected ${\alpha }_{ij}$ coefficients.

Further appreciation of this example may be obtained via a plot of the normalized estimated $KCF\left(\tau \right)$ (29) shown in Figure 1.

The residual normalized kernel sequences (31) computed using (28) are depicted in Figure 2 for each kernel and show effective decrease below the null hypothesis decision threshold line vindicating adequate modelling.

Moreover, for this realization, one may show that the Hannan–Quinn Information Criterion (27) points to the correct order of $p=1$. In addition, Portmanteau tests do not reject whiteness in the $\mathbb{F}$ space for either kernel further confirming successful modelling in both cases.

To illustrate and confirm the Gaussian asymptotic behaviour discussed in Section 2.1, normal probability plots for ${\widehat{a}}_{21}$ are presented in Figure 3. Further objective quantification of the convergence speed towards normality is provided by the evolution towards 1 of the Filliben squared-correlation coefficient [22,23,24] as a function of $n_s$ (Figure 4).

Convergence to normality justifies using (39) to test for null connectivity hypotheses. Test perfomance is depicted in Figure 5.

3.2. Example 2

Consider $x_1\left(n\right)$, a highly resonant ($R=0.99$) linear oscillator (at a normalized frequency of $f=0.1$) to be unidirectionally coupled to a low pass system $x_2\left(n\right)$ through a delayed squared term

$$\left\{\begin{array}{cc}\hfill x_1\left(n\right)& =2R\cos \left(2\pi f\right)x_1(n-1)-R^2{x}_1(n-2)+w_1\left(n\right),\hfill \\ \hfill x_2\left(n\right)& =-0.9x_2(n-1)+c{x}_1^2(n-1)+w_2\left(n\right),\hfill \end{array}\right.$$

(46)

where $c=0.1$ [17].

This system was already investigated elsewhere [17,25,26] under a different estimation algorithm and with fewer Monte Carlo replications. The null hypothesis connectivity results are presented in Figure 6 showing adequate asymptotic decision success. A quadratic kernel was used in all cases.

3.3. Example 3

The present example comes from a model in [27],

$$\left\{\begin{array}{cc}\hfill x_1\left(n\right)& =3.4x_1(n-1){[1-x_1^2(n-1)]\mathrm{e}}^{-x_1^2(n-1)}+w_1\left(n\right),\hfill \\ \hfill x_2\left(n\right)& =3.4x_2(n-1){[1-x_2^2(n-1)]\mathrm{e}}^{-x_2^2(n-1)}+c_1{x}_1^2(n-1)+w_2\left(n\right),\hfill \\ \hfill x_3\left(n\right)& =3.4x_3(n-1){[1-x_3^2(n-1)]\mathrm{e}}^{-x_3^2(n-1)}+c_2{x}_2^4(n-1)+w_3\left(n\right).\hfill \end{array}\right.$$

(47)

This choice was dictated by the nonlinear wideband character of its signals. The values $c_1=0.7$ and $c_2=0.9$ were adopted.

Figure 7 shows that connection detectability improves as signal duration $n_s$ increases except for the nonexisting $x_3\left(n\right)\leftarrow x_1\left(n\right)$ connection whose performance stays more or less constant with a false positive rate slightly above $\alpha =1\%$. All computations used quadratic kernels.

3.4. Example 4

For this numerical illustration, consider the model presented in [28]

$$\left\{\begin{array}{cc}\hfill x_1\left(n\right)& =3.4x_1(n-1){[1-x_1^2(n-1)]\mathrm{e}}^{-x_1^2(n-1)}+0.8x_1(n-2)+w_1\left(n\right),\hfill \\ \hfill x_2\left(n\right)& =3.4x_2(n-1){[1-x_2^2(n-1)]\mathrm{e}}^{-x_2^2(n-1)}+0.5x_2(n-2)+c{x}_1^2(n-2)+w_2\left(n\right).\hfill \end{array}\right.$$

(48)

System (48) produces nonlinear wideband signals with a quadratic ($1\to 2$) coupling factor whose intensity is given by c taken here as $0.5$.

It is worth noting that, kernelized Granger causality true positive rate improves as sample size ($n_s$) increases (Figure 8) and using the generalized Hannan–Quinn criterion, the order of kernelized autoregressive vector models identified for a typical realization was correctly identified and equals 2 as expected (see Figure 9).

3.5. Example 5

As a last numerical illustration, consider data generated by

$$\left\{\begin{array}{cc}\hfill x_1\left(n\right)& =3.4x_1(n-3){[1-x_1^2(n-3)]\mathrm{e}}^{-x_1^2(n-3)}+0.4x_1(n-4)+w_1\left(n\right),\hfill \\ \hfill x_2\left(n\right)& =3.4x_2(n-1){[1-x_2^2(n-1)]\mathrm{e}}^{-x_2^2(n-1)}+c_1{x}_1^2(n-2)+w_2\left(n\right),\hfill \\ \hfill x_3\left(n\right)& =3.4x_3(n-2){[1-x_3^2(n-2)]\mathrm{e}}^{-x_3^2(n-2)}+c_2{x}_2^2(n-3)+w_3\left(n\right),\hfill \end{array}\right.$$

(49)

with $c_1=0.9$ and $c_2=0.4$.

Under the quadratic kernel and employing kernelized Hannan–Quinn information criterion (27) (see Figure 10), one can see that the estimated model order is $p=3$ as expected judging from the $x_2^2(n-3)$ term in (49). In addition, kernelized Granger causality detectability improves with record length $n_s$ increase (Figure 11).

4. Conclusions and Future Work

After a brief theoretical presentation (Section 2), we have shown that canonical model fitting procedures that involve (a) model specification with order determination and (b) explicit model diagnostic testing can be successfully carried out in the feature space $\mathbb{F}$ to detect connectivity via reproducing kernels. In dealing with Granger causality detection using kernels as in [29,30], this stands in sharp contrast as the latter depend on solving the reconstruction/pre-image problem to provide prediction error estimates in the original data space $\mathbb{X}$. In fact, part of the challenge in pre-image determination lies in its frequently associated numerical ill-condition [31].

The key result behind doing model diagnostics and inference in $\mathbb{F}$ is (28) by realizing that kernel quantities may be normalized much as correlation coefficients. It should be noted that (28) holds even in the case of (nonkernel) linear modelling by replacing the $\mathbf{K}$ matrices by auto/crosscorrelation matrices, something that, in practice, is never adopted in classical linear time series modelling because the necessary auto/crosscorrelations are more efficiently computed from model residuals that are easy to obtain as no pre-imaging problem is involved there.

Thus, what importantly sets the present approach apart from previous work is the lack of need for returning to the original input space $\mathbb{X}$ to gauge model quality as the reconstruction/pre-image problem can be fully circumvented bypassing unnecessary uncertainties.

As such, we showed that, because model adequacy testing can be performed directly in the feature space $\mathbb{F}$, directional Granger type connectivity can be detected for a variety of multivariate nonlinear coupling scenarios, thereby totally dispensing with the need for detailed ‘a priori’ model knowledge.

We observed that successful connectivity detection is achievable at the expense of a relatively short time series. A systematic comparison with other approaches [4,5,32,33,34,35] is planned for future work, but, at least for the cases we tested so far, savings of at least one order magnitude in record lengths are feasible.

One of the basic tenants of the present work is that model coefficients in the feature space are asymptotically normal, something whose consistency was successfully illustrated though the need for a more formal proof remains, especially in connection to explicit kernel estimates under the total-least-squares solution to (23). Our choice of TLS was dictated by its apparent superiority when compared to the ‘kernel trick’ [32] whose multivariate version we employed in [26,36,37].

In this context, it is important to note that, contrary to other methods that require time-consuming resampling procedures for adequate inference, the present approach relies on asymptotic statistics and is thus less susceptible to eventual problems derived from data shuffling.

One of the advantages of the present development is that the procedure allows for determining how far in the past to look via the model order criteria we employed (27).

Even though order estimation and model testing were successful upon borrowing from the usual linear modelling practices, further systematic examination is still needed and is under way.

One may rightfully argue that the kernels we chose for illustrating the present work are equivalent to modelling the original time series after the application of a suitable $\varphi (·)$ transformation to the data and that they look for the causality evidence present in higher order momenta. This, in fact, explains why quadratic kernels converge much faster than quartic ones in Example 1. The merit of framing the time series transformation discussion for connectivity detection in terms of kernels produces a simple workflow and paves the way to developing future data-driven criteria towards optimum data transformation choice for a given problem. Other kernel choices are being investigated.

The signal model used in the present development does not contemplate additive measurement noise whose impact on connectivity detection we also leave for future examination.

One thing the present type of analysis cannot do is expose details of how the nonlinearity takes place. For example, coupling may be quadratic or involve higher exponential powers or some other function. What the present approach can do, however, is to expose existing connections, so that modelling efforts can be concentrated on them, thereby avoiding modelling parameter waste on non relevant links.

Finally, the present systematic empirical investigation sets the proposal of using feature space-frequency domain descriptions of connectivity like kernel partial directed coherence [26,36], and kernel directed transfer function [37] on sound footing, especially with respect to their asymptotic connectivity behaviour.