Transversality Conditions for Geodesics on the Statistical Manifold of Multivariate Gaussian Distributions

Abstract

We consider the problem of finding the closest multivariate Gaussian distribution on a constraint surface of all Gaussian distributions to a given distribution. Previous research regarding geodesics on the multivariate Gaussian manifold has focused on finding closed-form, shortest-path distances between two fixed distributions on the manifold, often restricting the parameters to obtain the desired solution. We demonstrate how to employ the techniques of the calculus of variations with a variable endpoint to search for the closest distribution from a family of distributions generated via a constraint set on the parameter manifold. Furthermore, we examine the intermediate distributions along the learned geodesics which provide insight into uncertainty evolution along the paths. Empirical results elucidate our formulations, with visual illustrations concretely exhibiting dynamics of 1D and 2D Gaussian distributions.

1. Introduction

The importance of a metric to measure the similarity between two distributions has weaved itself into a plethora of applications. Fields concerning statistical inference [1,2,3], model selection [4,5,6] and machine learning have found it necessary to quantify the likeness of two distributions. A common approach to measure this similarity is to define a distance or divergence between distributions using the tenets of information geometry, e.g., the Fisher–Rao distance or the f-divergence [7], respectively. To the best of our knowledge, research and results in information geometry have predominantly focused on establishing similarities between two given distributions. Here, we consider an important class of problems where one or both endpoint distributions are not fixed, but instead, constrained to live on subset of the parameter manifold.

When one relaxes the fixed endpoint requirements, the development of finding the shortest path between a given distribution and constraint surface (not single distribution) must be reconsidered using transversality conditions [8,9] for the standard length-minimizing functional. This is precisely the focus of the present work, where we derive the transversality conditions for working in the Riemannian space of multivariate Gaussian distributions. This approach opens new avenues for research and application areas where one no longer needs to provide the ending distribution but rather a description of the constraint set where the most similar model must be discovered. For example, applications such as domain adaptation [10,11,12] can be reformulated such that the optimal target domain distribution is discovered among a constraint family starting from a known source distribution estimated from training data, or even model selection [4,5,6,13,14], where a search in a constrained family of distributions may be better aligned with the desired objective versus evaluating the MLE fit penalized by parameter cardinality.

In this work, we have purposefully chosen to work in the natural parameter space of multivariate Gaussians (mean vector, $\mu$, and covariance matrix, Σ) and address the problem formulation in a completely Riemannian context. We are aware of the usual dually flat constructions [2,15,16] afforded by information geometry using divergence measures and the Legendre transformation. Though elegant in their algebraic constructions, these alternate parameterizations have yet to be employed in most statistical and machine-learning applications. Hence, we develop all geometric motivations and consequential mathematical derivations using Riemannian geometry and the natural parameter space, i.e., we employ the Fisher information matrix metric tensor and find the length-minimizing curve to the constraint set.

The remainder of this paper is organized as follows. In Section 2, we give a summary of important results concerning geodesics on manifolds. In Section 3, we provide a brief introduction to the techniques of calculus of variations with the goal of developing the Euler-Lagrange equations necessary to find the shortest path between two multivariate Gaussian distributions. In this section, we restrict ourselves to problems in which both the initial and final distributions are known exactly. Following this, in Section 3.3, we develop the conditions required to satisfy transversality constraints, or constraints where either the initial and/or final distribution are determined from those residing on a defined subsurface of the manifold rather than being exactly known. The results from these sections are employed in Section 4, where we explore various constraint surfaces and numerical experiments to demonstrate the utility of our variable-endpoint framework. Finally, some closing perspectives and remarks are given in Section 5.

2. Related Works

Most efforts towards measuring distances on statistical manifolds build on the foundation started by Fisher in [17], in which he introduces the idea of the information matrix. In [18] Kullback and Leibler published a pioneering effort to describe this distance. Works such as [19,20], endowed statistical distributions with geometrical properties. However, it was Rao [21] that expanded on the ideas of Fisher that defined a metric for statistical models based on the information matrix. The information matrix satisfies the conditions of a metric on a Riemannian statistical manifold, and is widely used because of its invariance [22]. This connection between distance and distributions encouraged others to explore the distance between specific families of distributions [23]. Among these families include special cases of the multivariate normal model [24], the negative binomial distribution [25], the gamma distribution [26,27], Poisson distribution [28], among others.

In [29], the authors offer a detailed exploration of geodesics on a multivariate Gaussian manifold. They show that there exists a geodesic connecting any two distributions on a Gaussian manifold. However, a closed-form solution for the most general case remains an open problem.

In [30] and expanded on in [31], the authors offer a very detailed discussion, focusing primarily on the univariate normal distribution for which a closed-form solution for the Fisher–Rao distance is known. Here, the authors focus on a geometrical approach, abandoning the “proposition-proof” format offered in previous research. With this geometric approach, closed-form solutions to various special cases are derived: univariate Gaussian distributions, isotropic Gaussian distributions, and Gaussian distributions with diagonal covariance matrix.

Another novel application of geodesics on a Gaussian statistical manifold is explored in [32], where the authors use information geometry for shape analysis. Shapes were represented using a K-component Gaussian Mixture Model, with the number of components being the same for each shape. With this, each shape occupied a unique point on a common statistical manifold. Upon mapping two shapes to their points on this manifold, the authors use an iterative approach to calculate the geodesic between these two points, with the length of the geodesic offering a measure of similarity of the shapes. Furthermore, because of the iterative approach to solving for the geodesic, all intermediate points along path are revealed. These points can be mapped to their own unique shapes, essentially showing the evolution from one shape to another. This shape deformation exhibits the benefit of analyzing more than just the distance between points on a manifold and that “walk” along the path has real substance.

In [33], the authors explore the complexity of Gaussian geodesic paths, with the ultimate goal of relating the complexity of a geodesic path on a manifold to the correlation of the variables labeling its macroscopic state. Specifically, the authors show that, if there is a dependence between the variables, the complexity of the geodesic path decreases. Complexity, for these purposes is defined as the volume of the manifold traversed by the geodesic connecting a known initial state to a future state, which is well defined. It is shown that this volume decays by a power law at a rate that is determined by the correlation between the variables on the Gaussian manifold.

In [34], the authors use the geometry of statistical manifolds to study how the quantum characteristics of a system are affected by its statistical properties. Similar to our work, the authors prescribe an initial distribution on the manifold of Gaussians and examine the geodesics emanating from it, without dictating a specific terminating distribution. The authors show that these paths tend to terminate at distributions that minimize Shannon entropy. However, unlike our work, these paths are free to roam on the manifold and are not required to terminate on a specific surface on the manifold. Furthermore, the most relevant part of the author’s work considers only univariate Gaussians with a two-dimensional parameter manifold, without ever considering higher dimensions.

Though we have chosen to work with Riemannian geometry, it is worth mentioning that information geometry often employs dualistic geometries that can be established using divergence measures. In [35], the authors detail the use of divergence measures to obtain the dual coordinates for space of multivariate Gaussians. However, they point out that the choice of divergence measure is not unique and resulting geometries lack the same interpretative power of the natural parameterization.

Though these previous works operate in the space of multivariate Gaussians and deriving geodesics therein, they all require defining the initial and terminal distributions on the manifold. In this work, we address a novel problem of finding the geodesics when the terminal conditions are hypersurface constraints rather than a single point. Technically, these transversality conditions are variable boundary conditions placed on the initial and final distributions requiring them to reside on a parametric surface typically defined by constraining the coordinates. The usefulness of these variable boundary conditions has emerged in many areas including physics [36] in which the author studied wetting phenomenon on rough surfaces and in [37], where the authors studied the elasticity of materials. Additionally, in [8,38,39], transversality conditions were employed in economic optimal control problems with a free-time terminal condition. However, as practical as transversality conditions have been in the above fields, their application in information geometry literature is deficient.

3. Geodesics for Fixed-Endpoint Problems

We begin by briefly developing standard calculus of variation results for discovering the shortest path between fixed points on a differentiable manifold. Then, the result is applied to the space of multivariate Gaussians, with detailed derivations for the case of bivariate Gaussians. We finally extend the formulation to show how to incorporate variable-endpoint boundary conditions.

3.1. General Euler-Lagrange Equations

Among differential calculus’ many applications are problems regarding finding the maxima and minima of functions. Analogously, techniques of calculus of variation operates on functionals, which are mappings from a space of functions to its underlying field of scalars. considering the functional $L\left[y\right]$, the typical formulation of a calculation of variations problem is

$$\begin{array}{cc}\hfill min& L\left[y\right]={\int }_{x_0}^{x_1}F(x,y,\dot{y})dx\hfill \\ & y\left(x_0\right)=y_{0}y\left(x_1\right)=y_1\hfill \end{array}$$

(1)

where initial and terminal values are defined as ($x_0,y_0$) and ($x_1,y_1$), respectively and $\dot{y}$ is notation for the derivative. In general, y can be a vector of functions dependent on x, a vector of independent variables. The theory behind finding the extremum to problems such as these are analogous to single variable calculus, in which we a vanishing first derivative to locate critical points. Here, we locate the extremal functions using functional derivatives, leading to solving the Euler-Lagrange equations outlined in Section 3.2.

Though the formulation of finding the shortest path as a calculus of variations problem is rather elementary, its solution is involved. Obviously, in Euclidean geometry, this path is a straight line, and this distance is easily found. However, moving these ideas onto statistical manifolds complicates both the geometry and the calculus of this seemingly elementary problem. Analogous to the Euclidean setting, in a Riemannian manifold such as our space of Gaussians, solving for the shortest path L, involves the summation of many infinitely small arc lengths, $ds$

$$d{s}^2={\dot{\theta }}^{T}g\left(\theta \right)\dot{\theta },$$

(2)

where $\theta$ is a parameter vector, $g\left(\theta \right)$ is a metric tensor dependent on the parameter vector and (${·)}^T$ represents the transpose of a vector. The metric tensor for Euclidean space is the identity matrix but on the multivariate Gaussian manifold, this metric tensor is the Fisher information matrix, discussed later in Section 3.2.

This makes the functional we wish to minimize

$$P={\int }_{x_0}^{x_1}\sqrt{{\dot{\theta }}^{T}g\left(\theta \right)\dot{\theta }}dx$$

(3)

or, because the square root is a monotonically increasing function, we can conveniently use

$$\mathcal{F}={\int }_{x_0}^{x_1}{\dot{\theta }}^{T}g\left(\theta \right)\dot{\theta }dx.$$

(4)

With this, the calculus of variation problem that solves for the minimum distance on a manifold is

$$\begin{array}{cc}\hfill min& \mathcal{F}\left[\theta \right]={\int }_{x_0}^{x_1}{\dot{\theta }}^{T}g\left(\theta \right)\dot{\theta }dx\hfill \\ & \theta \left(x_0\right)=[{\theta }_{01},{\theta }_{02},\dots ,{\theta }_{0n}]\theta \left(x_1\right)=[{\theta }_{11},{\theta }_{12},\dots ,{\theta }_{1n}]\hfill \end{array}$$

(5)

In the present context, the Fisher information metric tensor $g\left(\theta \right)=g(\mu ,\mathsf{\Sigma })$, the natural parameterization for multivariate Gaussians. Moreover, ${\theta }_0$ and ${\theta }_1$ are the parameters of the initial and final distributions.

The minimizer to Equation (5) is the well-known Euler-Lagrange equation, a system of second-order differential equations. These equations operate on the function in Equation (4). Accordingly, we define

$$K={\dot{\theta }}^{T}g\left(\theta \right)\dot{\theta }$$

(6)

With this, the Euler-Lagrange equations are

$$K_{\theta }-\frac{d}{dx}{K}_{\dot{\theta }}=0.$$

(7)

where $K_{\theta }$ and $K_{\dot{\theta }}$ are the functional derivatives with respect the curve $\theta \left(x\right)$.

3.2. Euler-Lagrange Equation for Gaussian Distributions

The Fisher information matrix is a measure of how much information about the parameter of interest from a multivariate distribution is revealed from random data space. Intuitively, it can be considered an indication of how “peaked” a distribution is around a parameter. If the distribution is sharply peaked, very few data points are required to locate it. As such, each data point carries a lot of information.

For a multivariate probability distribution, the Fisher information matrix is given by

$$g_{i,j}\left(\theta \right)=\int f(x;\theta )\frac{\partial }{\partial {\theta }_i}logf(x;\theta )\frac{\partial }{\partial {\theta }_j}logf(x;\theta )dx,$$

(8)

where the index $(i,j)$ represents the appropriate parameter pair of the multivariate parameter vector $\theta$.

Alternatively, there are additional useful forms of the Fisher information, provided that certain regularity conditions are satisfied. First, the Fisher information matrix is the expectation of the Hessian of the log likelihood of the density function. Specifically,

$$g_{i,j}\left(\theta \right)=-\mathcal{E}\left[\frac{{\partial }^2}{\partial {\theta }_i\partial {\theta }_j}logf(x;\theta )\right]=-\mathcal{E}\left[H\right],$$

(9)

where H is the Hessian matrix of the log-likelihood.

Second, the Fisher information can be calculated from the variance of the score function

$$g\left(\theta \right)=\mathrm{Var}\left(S_f(x;\theta )\right),$$

(10)

where

$$S_f(x;\theta )=\nabla logf(x;\theta ).$$

(11)

Most importantly and for our purposes, the Fisher information matrix is the metric tensor that will define distances on Riemannian Gaussian manifold. Given a distribution on a manifold, by use of this metric tensor, we can minimize an appropriate functional to find a closest second distribution residing on a constrained subset the manifold. A class of problems covered by variable-endpoint conditions in the calculus of variations.

Consider the n-dimensional multivariate Gaussian with density given by

$$f(x_n:{\mu }_n,\mathsf{\Sigma })=2{\pi }^{-\frac{n}{2}}det{\left(\mathsf{\Sigma }\right)}^{-\frac{1}{2}}exp-\frac{{(X-\mu )}^T{\mathsf{\Sigma }}^{-1}(X-\mu )}{2}$$

(12)

where X is the random variable vector, $\mu ={\mu }_1,{\mu }_2,...,{\mu }_n$ is the n-dimensional mean vector of the distribution and $\mathsf{\Sigma }$ is the $n\times n$ covariance matrix.

Since the covariance matrix is symmetric, it contains $\frac{(n+1)\left(n\right)}{2}$ number of unique parameters, i.e., the number of diagonal and the upper (or lower) triangular elements. With the n-dimensional mean vector, the total number of scalar parameters in an n-dimensional multivariate Gaussian is $\frac{(n+3)n}{2}$, which will be the size of the Fisher information matrix. For all further developments in the parameter space, these parameters are collected in a single vector, $\theta$ such that

$$\theta =\{\underset{{\theta }_1,{\theta }_2,...,{\theta }_n}{{\mu }_1,{\mu }_2,...{\mu }_n},\underset{{\theta }_{n+1},...,{\theta }_{\frac{(n+3)n}{2}}}{{\sigma }_{1,1}^2,{\sigma }_{1,2}^2,...,{\sigma }_{n,n}^2\}}$$

(13)

To clarify, this new parameter $\theta$ has the mean vector $\mu$ as its first n components and the resulting components are made up of the unique elements of the covariance matrix, starting with the first row, followed, by the second row but without the first entry, since ${\mathsf{\Sigma }}_{1,2}={\mathsf{\Sigma }}_{2,1}$ and ${\mathsf{\Sigma }}_{1,2}$ is already included in $\theta$. We capture all the parameters of the multivariate Gaussian distribution in this non-traditional vector form because it is more conceptually in line with the calculation of the Fisher information matrix defined in Equation (8).

Therefore, using Equations (8) and (12), the Fisher information for the general multivariate Gaussian distribution is

$$g_{ij}(\mu ,\mathsf{\Sigma })=\frac{1}{2}tr\left[\left({\mathsf{\Sigma }}^{-1}\frac{\partial \mathsf{\Sigma }}{\partial {\theta }_i}\right)\left({\mathsf{\Sigma }}^{-1}\frac{\partial \mathsf{\Sigma }}{\partial {\theta }_j}\right)\right]+{\frac{\partial \mu }{\partial {\theta }_i}}^T{\mathsf{\Sigma }}^{-1}\frac{\partial \mu }{\partial {\theta }_j}$$

(14)

for which a very detailed proof can be found in the Appendix A of this paper. In the case of the bivariate Gaussian distribution, this $5\times 5$ matrix has only 15 unique elements, because of its symmetry. Once again, the detailed derivation of each of the elements is provided in the Appendix A. The resulting metric tensor elements are

$$\begin{array}{cc}\hfill g_{11}& =\frac{{\sigma }_2^2}{{\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2}\hfill \\ \hfill g_{22}& =\frac{{\sigma }_1^2}{{\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2}\hfill \\ \hfill g_{33}& =\frac{1}{2}{\left(\frac{{\sigma }_2^2}{{\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2}\right)}^2\hfill \\ \hfill g_{44}& =\frac{1}{2}{\left(\frac{{\sigma }_1^2}{{\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2}\right)}^2\hfill \\ \hfill g_{55}& =\frac{{\sigma }_1^2{\sigma }_2^2+{\sigma }_{12}^2}{{\left({\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2\right)}^2}\hfill \\ \hfill g_{12}& =-\frac{{\sigma }_{12}}{{\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2}=g_{21}\hfill \\ \hfill g_{34}& =\frac{1}{2}{\left(\frac{{\sigma }_{12}}{{\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2}\right)}^2=g_{43}\hfill \\ \hfill g_{35}& =-\frac{{\sigma }_{12}{\sigma }_2^2}{{\left({\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2\right)}^2}=g_{53}\hfill \\ \hfill g_{45}& =-\frac{{\sigma }_{12}{\sigma }_1^2}{{\left({\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2\right)}^2}=g_{54}\hfill \end{array}$$

(15)

All elements dealing with the information between a component of $\mu$ and an element of $\mathsf{\Sigma }$ vanish, which is a property extended to every multivariate Gaussian distribution of higher dimensions as well.

3.3. Variable-Endpoint Formulation: Transversality Boundary Conditions

Our development so far has focused on summarizing the usual situation of finding the length-minimizing path between two fixed points on a Riemannian manifold. We now shift to the situation of allowing one or both of the initial or final endpoints (distributions in this context) to be variable. This changes both the scope and the mathematics of the problem and requires the use of transversality boundary conditions.

Transversality conditions have been useful in other applications. In economics, the conditions are needed to solve the infinite horizon [40,41] problems. Other applications are found in biology [42] and physics [43]. However, to the best of our knowledge, there is little to no prior work investigating variable-endpoint formulations in the domain of information geometry. As we will demonstrate, the crux of employing these methods revolves around appropriately defining the parameter constraint surface. Though our results here provide concrete examples of interesting constraint surfaces, guidance on prescribing these subsets or developing techniques for automatically learning them will be important areas for future research.

The transversality conditions take into account the constrained hypersurface, $\varphi \left(\theta \right)$, from the coordinates parameterizing the distributions of the statistical manifold. If, for example, we are given an initial distribution and are asked to find which distribution on $\varphi \left(\theta \right)$ is closest, the usual geodesic problem formulated in Equation (5) now becomes

$$\begin{array}{cc}\hfill min& \mathcal{F}\left[\theta \right]=\frac{1}{2}{\int }_{x_0}^{x_1}{\dot{\theta }}^{T}g\left(\theta \right)\dot{\theta }dx\hfill \\ & \theta \left(x_0\right)=[{\theta }_{01},{\theta }_{02},\dots ,{\theta }_{0n}]\theta \left(x_1\right)=\varphi \left(\theta \right)\hfill \end{array}$$

(16)

Therefore, in addition to satisfying the Euler-Lagrange equation, now with transversality conditions, the optimal solution must also satisfy

$$\left[\begin{array}{c}{K}_{\dot{{\theta }_1}}\\ K_{\dot{{\theta }_2}}\\ ⋮\\ K_{\dot{{\theta }_n}}\end{array}\right]=\alpha \left[\begin{array}{c}{\varphi }_{{\theta }_1}\\ {\varphi }_{{\theta }_2}\\ ⋮\\ {\varphi }_{{\theta }_n}\end{array}\right]$$

(17)

In Equation (17), the left-hand side is a vector tangent to the optimal path and the vector on the right right-hand side is the gradient of the terminal surface, which is orthogonal to the surface. Considering that the optimal path and the constraint surface intersect at the terminal distribution, this view of the transversality requirement implies that the tangent vector to the optimal path and the gradient of the constraint surface be collinear at the intersecting distribution. The scalar multiple $\alpha$ affects that magnitude of the vector and, from a geometric perspective, there is no loss of generality by setting $\alpha =1$.

4. Results: Transversal Euler-Lagrange Equations for Bivariate Gaussian Distributions

We now turn our attention to various use cases of the transversal boundary conditions on the Gaussian manifold. We limit our derivations to bivariate Gaussians to make the calculations tractable and for visualization purposes. However, the same development is applicable to higher-dimensional Gaussians. It is worth mentioning that even in the fixed-endpoint scenario, there are no closed-form solutions for the geodesic when manifold coordinates include $\mu$ and $\mathsf{\Sigma }$, with analytical solutions existing only for special cases such zero-mean distributions.

Using the Fisher information matrix defined in Equation (14), more specifically for the bivariate Gaussian distribution discussed in the Appendix, and employing the general form of the geodesic functional in Equation (4), we can define the integrand of the arc-length-minimizing function on the space of bivariate Gaussian distribution as

$$\begin{array}{cc}\hfill K\left(\theta \right)=& \frac{{\sigma }_2^2{\dot{{\mu }_1}}^2}{k}+\frac{{\sigma }_1^2{\dot{{\mu }_2}}^2}{k}+\frac{{\left({\sigma }_2^2\right)}^2{\left({\dot{\sigma }}_1^2\right)}^2}{2k^2}+\frac{{\left({\sigma }_1^2\right)}^2{\left({\dot{\sigma }}_2^2\right)}^2}{2k^2}+\frac{{\sigma }_1^2{\sigma }_2^2{\dot{\sigma }}_{12}^2}{k^2}+\frac{{\sigma }_{12}^2{\dot{\sigma }}_{12}^2}{k^2}\hfill \\ & -\frac{2{\sigma }_{12}{\dot{\mu }}_1{\dot{\mu }}_2}{k}+\frac{{\sigma }_{12}^2{\dot{\sigma }}_1^2{\dot{{\sigma }_2}}^2}{k^2}-\frac{2{\sigma }_{12}{\sigma }_2^2{\dot{\sigma }}_1^2{\dot{\sigma }}_{12}}{k^2}-\frac{2{\sigma }_{12}{\sigma }_1^2{\dot{\sigma }}_2^2{\dot{\sigma }}_{12}}{k^2}\hfill \end{array}$$

(18)

where $k={\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2$.

We can use Equation (18) to derive the system of second-order differential equations, solutions to which yield the shortest path between two distributions.

$${\ddot{\mu }}_1=\frac{{\dot{\mu }}_1{\dot{\sigma }}_1^2{\sigma }_2^2+{\dot{\mu }}_2{\sigma }_1^2{\dot{\sigma }}_{12}-{\dot{\mu }}_2{\dot{\sigma }}_1^2{\sigma }_{12}-{\dot{\mu }}_1{\sigma }_{12}{\dot{\sigma }}_{12}}{{\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2}$$

(19)

$${\ddot{\mu }}_2=\frac{{\dot{\mu }}_2{\sigma }_1^2{\dot{\sigma }}_2^2+{\dot{\mu }}_1{\sigma }_2^2{\dot{\sigma }}_{12}-{\dot{\mu }}_1{\dot{\sigma }}_2^2{\sigma }_{12}-{\dot{\mu }}_2{\sigma }_{12}{\dot{\sigma }}_{12}}{{\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2}$$

(20)

$${\ddot{\sigma }}_1^2=\frac{{\dot{\mu }}_1^2{\sigma }_{12}^2+{\dot{\sigma }}_1^2{\sigma }_2^2+{\sigma }_1^2{\dot{\sigma }}_{12}^2-{\dot{\mu }}_1^2{\sigma }_1^2{\sigma }_2^2-2{\dot{\sigma }}_1^2{\sigma }_{12}{\dot{\sigma }}_{12}}{{\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2}$$

(21)

$${\ddot{\sigma }}_2^2=\frac{{\dot{{\mu }_2}}^2{\sigma }_{12}^2+{\dot{\sigma }}_2^2{\sigma }_1^2+{\sigma }_2^2{\dot{\sigma }}_{12}^2-{\dot{\mu }}_2^2{\sigma }_1^2{\sigma }_2^2-2{\dot{\sigma }}_2^2{\sigma }_{12}{\dot{\sigma }}_{12}}{{\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2}$$

(22)

$${\ddot{\sigma }}_{12}=-\frac{{\sigma }_{12}{\dot{\sigma }}_{12}^2-{\dot{\mu }}_2\dot{{\mu }_2}{\sigma }_{12}^2-{\sigma }_1^2{\dot{\sigma }}_2^2{\dot{\sigma }}_{12}^2-{\dot{\sigma }}_1^2{\sigma }_2^2{\dot{\sigma }}_{12}^2+{\dot{\sigma }}_1^2{\dot{\sigma }}_2^2{\sigma }_{12}^2+{\dot{\mu }}_2\dot{{\mu }_2}{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2{\sigma }_2^2-{\sigma }_{12}^2}$$

(23)

Along with satisfying this system of equations, the solutions presented here must satisfy transversality conditions at one or both the terminal and initial boundaries. In what follows, those conditions will be prescribed accordingly, considering various applications of interest.

4.1. Isotropic Terminal Distribution

Working with the full parameterization of a multivariate Gaussian distribution can be computationally expensive, especially when calculating their Fisher information matrix. For example, the number of parameters grows quadratically with dimension. However, isotropic Gaussian distributions, defined below in Equation (24) grow only linearly with the mean vector’s size, orders of magnitude more favorable. In addition to being computationally efficient, isotropic Gaussian distributions provide a submanifold prescribable by a tractable transversality condition. Accordingly, we formulate the following variable-endpoint problem: Given a general multivariate Gaussian distribution, what is the closest (in a geodesic sense) isotropic distribution?

Let ${\mathsf{\Sigma }}_i$ be the covariance matrix of a multivariate Gaussian distribution with n mean components. This distribution is isotropic if

$${\mathsf{\Sigma }}_i={\sigma }^2{I}_n$$

(24)

where $I_n$ is the n-dimensional identity matrix.

Formally, let $\theta$ capture all the parameters of a multivariate Gaussian distribution according to Equation (13). The functional to minimize is given be

$$\begin{array}{cc}\hfill min& \mathcal{F}\left[\theta \right]=\frac{1}{2}{\int }_{x_0}^{x_1}{\dot{\theta }}^{T}g\left(\theta \right)\dot{\theta }dx\hfill \\ & {\theta }_0=[{\mu }_0,{\mathsf{\Sigma }}_0]{\theta }_1=\varphi ({\mu }_1,{\mathsf{\Sigma }}_1)\hfill \end{array}$$

(25)

where ${\mu }_0$ and ${\mathsf{\Sigma }}_0$ are the known parameters of the starting distribution, but ${\mu }_1$ and ${\mathsf{\Sigma }}_1$ are identified by solving the Euler-Lagrange equations while satisfying the transversal condition in Equation (24).

For the bivariate Gaussian distribution, the terminal surface described in Equation (24) can be defined by

$$\mathsf{\Phi }({\sigma }_1^2,{\sigma }_2^2)={\sigma }_1^2-{\sigma }_2^2=0,$$

(26)

with ${\mu }_1$ free and ${\sigma }_{12}=0$. Appling Equation (17) to this surface, we obtain the condition

$${\left({\sigma }_2^2\right)}^2{\dot{{\sigma }_1}}^2+{\left({\sigma }_1^2\right)}^2{\dot{{\sigma }_2}}^2+{\sigma }_{12}^2{\dot{{\sigma }_2}}^2+{\sigma }_{12}^2{\dot{{\sigma }_1}}^2-2{\sigma }_{12}{\sigma }_1^2\dot{{\sigma }_{12}}-2{\sigma }_{12}{\sigma }_2^2\dot{{\sigma }_{12}}=0.$$

(27)

Therefore, in addition to the Euler-Lagrange equations in Equation (19) through Equation (23), requiring the final distribution to be isotropic implies the geodesic must also satisfy Equation (27). Additionally, the terminal distribution must satisfy the conditions of constraint surface in Equation (26).

4.1.1. Constant Mean Vector with Initial Isotropic Covariance

In this use case, we introduce a slight modification of the previous scenario, where now we set the mean vector of the initial and final distributions to be ${\mu }_0={\mu }_1=[0,0]$. We are still interested in finding the closed isotropic Gaussian, but now not allowing the curve evolution to move the distribution’s mode.

For example, let us assume an initial distribution with

$${\mu }_0=[0,0],{\mathsf{\Sigma }}_0=\left[\begin{array}{cc}7& 0\\ 0& 2\end{array}\right]$$

(28)

and the final distribution lie on the surface defined in Equation (26).

Applying the Euler-Lagrange equation with the transversality conditions to the problem above results in a final isotropic distribution with ${\sigma }^2=3.74$. Figure 1a shows the information path (dashed and curved) from the initial distribution to the chosen distribution on the isotropic constraint. A Euclidean path would end with a distribution with an isotropic variance equivalent to the average of the original variances. The geodesic path calculated under the Fisher information matrix is an indication of the curvature of the manifold in this region of the parameter space. It is possible in this special zero-mean case to analytically calculate the ending variance on the isotropic constraint surface. Given initial variances of ${\sigma }_1^2,{\sigma }_2^2$, it can be shown that the final variance, ${\sigma }_f^2$ is given by

$${\sigma }_f^2=\sqrt{{\sigma }_1^2{\sigma }_2^2}.$$

(29)

In Figure 1b, we can see the evolution of all the parameters, starting from the initial distribution to end. Noteworthy is that, even though ${\sigma }_{12}$ is not required to stay at 0, there is no benefit for it deviating from 0, as seen in Figure 1b. The Fisher information matrix is independent of the mean vector and, since the values of the mean vector are also not part of our isotropic constraint on the final distribution, the mean vector is not compelled to change from the original distribution, justifying the exclusion of the mean vector’s path in Figure 1.

4.1.2. Constant Mean Vector with Initial Full Covariance

In the example above, the distributions move along a 2-dimensional manifold parameterized just by the individual variances of the variables. Neither the mean vector, nor the off-diagonal values of the covariance matrices are considered by the Euler-Lagrange equation and therefore remain static. However, starting with a full covariance matrix changes the problem appreciably. For comparison, we will start with same diagonal elements of the covariance matrix but now incorporate the off-diagonal element.

The problem formulation is analogous to Equation (25) subject to the isotropic constraint in Equation (26). Furthermore, we define the initial distribution as

$${\mu }_0=[0,0],{\mathsf{\Sigma }}_0=\left[\begin{array}{cc}7& -3\\ -3& 2\end{array}\right]$$

(30)

As seen in Figure 2, including ${\sigma }_{12}=-3$ alters where the geodesic terminates on the transversality constraint, with the final covariance matrix having diagonal elements of ${\sigma }_1^2={\sigma }_2^2=2.24$. In Figure 2a, the effects of including the off-diagonal value ${\sigma }_{12}$ are clearly visible on the path of the geodesic causing it to deviate significantly. Figure 2b shows all values of the parameters along the geodesic. As mentioned before, the mean vector remains constant at all intermediate values of the distribution. Unlike before, the value for ${\sigma }_{12}$ must evolve to satisfy the terminal constraint surface, illustrated in Figure 2b and emphasized in Figure 2c.

4.1.3. Starting on the Constraint Surface

When searching for the isotropic boundary condition, interesting paths occur if we start with an initial isotropic distribution, but require the mean vector to change. That is, if the initial distribution already resides on the terminal constraint surface, it would seem counter-intuitive if the geodesic is compelled to leave this constraint. However, as seen in Figure 3, this is exactly what happens.

Employing the same modeling equations as the previous example, we define the initial distribution as

$${\mu }_0=[-3,3],{\mathsf{\Sigma }}_0=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$$

(31)

which is already isotropic. We search for the closest final distribution that is isotropic but with a mean vector ${\mu }_1=[3,-4]$. If we track evolution of just the mean vector, it starts in Quadrant II and moves to a distribution in Quadrant IV. Qualitatively, the initial and final distributions are geometrically symmetric. However, the isotropic uncertainty is considerably different.

The insights gleaned from this example shed new light into information evolution. Naively, one would think the shortest path would be one that maintains its current shape and just moves along the constraint surface to reach the desired $\mu$. However, as shown in Figure 3, this is not the case. In fact, intermediate distributions obtain covariance matrices with ${\sigma }_{12}<0$, as demonstrated in Figure 3c. Instead of just staying on the constraint surface, the distributions stretch in the direction of the desired mean, which explains negative values of the covariance between the variables. This elongation of the covariance in the direction of final mean vector suggests that the information metric prefers uncertainty reduction in regions with few plausible solutions. Instead, the distributions along the geodesic “reach” for their destination, i.e., the initial mean vector in Quadrant II and final in Quadrant IV. This behavior is illustrated by the intermediate (red) ellipse in Figure 4.

To reiterate this insightful behavior of information flow, a second example was conducted with a mean vector that starts in Quadrant III, ${\mu }_0=[-3,-3]$, and seeks out a final mean vector in Quadrant I, ${\mu }_1=[3,4]$, as shown in Figure 5. The initial distribution is still isotropic and the requirement to end isotropic remains. However, as seen in Figure 6, the values of ${\sigma }_{12}$ acquire positive values along the geodesic. The path of the main diagonal variances remains unchanged.

4.2. Initial and Terminal Variable-Endpoint Conditions

It is possible to place transversality conditions on both the initial and final boundaries, with each being entirely independent of the other. Essentially, we are searching for a geodesic between two almost unknown distributions, with only minimal knowledge about the constraint set characterizing the initial and final hypersurfaces.

We consider the example where the final distribution is prescribed to be isotropic as before, but now the initial distribution must have a mean vector with equal components. This enforcement has the practical benefit of reducing the parameter dimensionality of allowable distributions. The problem is formulated as

$$\begin{array}{cc}\hfill min& \mathcal{F}\left[\theta \right]=\frac{1}{2}{\int }_{x_0}^{x_1}{\dot{\theta }}^{T}g\left(\theta \right)\dot{\theta }dx\hfill \\ & {\theta }_0={\varphi }_0({\mu }_0,{\mathsf{\Sigma }}_0),{\theta }_1={\varphi }_1({\mu }_1,{\mathsf{\Sigma }}_1)\hfill \end{array}$$

(32)

where ${\varphi }_0$ and ${\varphi }_1$ represent the initial and final transversality constraint surfaces, such that

$${\varphi }_0({\mu }_{01},{\mu }_{02})={\mu }_{01}-{\mu }_{02}=0\mathrm{and}{\varphi }_1({\sigma }_1^2,{\sigma }_2^2)={\sigma }_1^2-{\sigma }_2^2=0.$$

(33)

To demonstrate this concretely, the initial distribution with unknown mean vector is prescribed with the following covariance matrix

$${\mathsf{\Sigma }}_0=\left[\begin{array}{cc}10& 0\\ 0& 2\end{array}\right].$$

(34)

Similarly, the unknown isotropic terminal distribution is given the mean vector ${\mu }_1=[-3,13]$.

Using Equation (17), the constraint requiring the initial distribution to reside on ${\varphi }_0$ imposes that the geodesic satisfy

$$({\sigma }_2^2-{\sigma }_{12}){\dot{\mu }}_1+({\sigma }_1^2-{\sigma }_{12}){\dot{\mu }}_2=0.$$

(35)

As shown in Figure 7, the unknown initial mean vector satisfying the ${\varphi }_1$ is ${\mu }_0=[7.4,7.4]$ and the final isotropic distribution has ${\sigma }_1^2={\sigma }_2^2=26.4$. The behavior of the geodesic under these variable-endpoint conditions is shown in Figure 8.

5. Conclusions

In this work, we have explored new formulations for working on information geometric manifolds. Previous and contemporary work using the Riemannian geometry of statistical manifolds has focused on establishing geodesics between two fixed-endpoint distributions. Here, by employing techniques from the calculus of variations, we have developed constructions that allow variable endpoints that are prescribed by a constraint set rather than fixed points.

These transversality conditions on initial and final distributions, enable new insights into how information evolves under different constraint use cases. Though this present effort focused on just a small variety of constraints on Gaussian manifolds, this approach can be readily extended to other statistical families. This novel approach of relaxing fixed endpoints and moving constraint sets has the potential to impact several application domains that employ information geometric models. In future research, we plan to recast the problem of optimal distribution discovery, in areas such as model selection and domain adaptation, using the presented framework, which allows for greater expressive power. We also plan to investigate observational data scenarios where parameters must be estimated and impacts uncertainty modeling of the manifold parameters.