Entropy-Regularized Optimal Transport on Multivariate Normal and q-normal Distributions

Abstract

The distance and divergence of the probability measures play a central role in statistics, machine learning, and many other related fields. The Wasserstein distance has received much attention in recent years because of its distinctions from other distances or divergences. Although computing the Wasserstein distance is costly, entropy-regularized optimal transport was proposed to computationally efficiently approximate the Wasserstein distance. The purpose of this study is to understand the theoretical aspect of entropy-regularized optimal transport. In this paper, we focus on entropy-regularized optimal transport on multivariate normal distributions and q-normal distributions. We obtain the explicit form of the entropy-regularized optimal transport cost on multivariate normal and q-normal distributions; this provides a perspective to understand the effect of entropy regularization, which was previously known only experimentally. Furthermore, we obtain the entropy-regularized Kantorovich estimator for the probability measure that satisfies certain conditions. We also demonstrate how the Wasserstein distance, optimal coupling, geometric structure, and statistical efficiency are affected by entropy regularization in some experiments. In particular, our results about the explicit form of the optimal coupling of the Tsallis entropy-regularized optimal transport on multivariate q-normal distributions and the entropy-regularized Kantorovich estimator are novel and will become the first step towards the understanding of a more general setting.

1. Introduction

Comparing probability measures is a fundamental problem in statistics and machine learning. A classical way to compare probability measures is the Kullback–Leibler divergence. Let M be a measurable space and $\mu ,\nu$ be the probability measure on M; then, the Kullback–Leibler divergence is defined as:

$$\mathrm{KL}\left(\mu \right|\nu )={\int }_{M}d\mu log\frac{d\mu }{d\nu }.$$

(1)

The Wasserstein distance [1], also known as the earth mover distance [2], is another way of comparing probability measures. It is a metric on the space of probability measures derived by the mass transportation theory of two probability measures. Informally, optimal transport theory considers an optimal transport plan between two probability measures under a cost function, and the Wasserstein distance is defined by the minimum total transport cost. A significant difference between the Wasserstein distance and the Kullback–Leibler divergence is that the former can reflect the metric structure, whereas the latter cannot. The Wasserstein distance can be written as:

$$W_p(\mu ,\nu ):={\left\{\underset{\pi \in \Pi (\mu ,\nu )}{inf}{\int }_{M\times M}d{(x,y)}^{p}d\pi (x,y)\right\}}^{\frac{1}{p}},$$

(2)

where $d(·,·)$ is a distance function on a measurable metric space M and $\Pi (\mu ,\nu )$ denotes the set of probability measures on $M\times M$, whose marginal measures correspond to $\mu$ and $\nu$. In recent years, the application of optimal transport and the Wasserstein distance has been studied in many fields such as statistics, machine learning, and image processing. For example, Reference [3] generated the interpolation of various three-dimensional (3D) objects using the Wasserstein barycenter. In the field of word embedding in natural language processing, Reference [4] embedded each word as an elliptical distribution, and the Wasserstein distance was applied between the elliptical distributions. There are many studies on the applications of optimal transport to deep learning, including [5,6,7]. Moreover, Reference [8] analyzed the denoising autoencoder [9] with gradient flow in the Wasserstein space.

In the application of the Wasserstein distance, it is often considered in a discrete setting where $\mu$ and $\nu$ are discrete probability measures. Then, obtaining the Wasserstein distance between $\mu$ and $\nu$ can be formulated as a linear programming problem. In general, however, it is computationally intensive to solve such linear problems and obtain the optimal coupling of two probability measures. For such a situation, a novel numerical method, entropy regularization, was proposed by [10],

$$C_{\lambda }(\mu ,\nu ):=\underset{\pi \in \Pi (\mu ,\nu )}{inf}{\int }_{{\mathbb{R}}^n\times {\mathbb{R}}^n}c(x,y)\pi (x,y)dxdy-\lambda \mathrm{Ent}\left(\pi \right).$$

(3)

This is a relaxed formulation of the original optimal transport of a cost function $c(·,·)$, in which the negative Shannon entropy $-\mathrm{Ent}(·)$ is used as a regularizer. For a small $\lambda$, $C_{\lambda }(\mu ,\nu )$ can approximate the p-th power of the Wasserstein distance between two discrete probability measures, and it can be computed efficiently by using Sinkhorn’s algorithm [11].

More recently, many studies have been published on improving the computational efficiency. According to [12], the most computationally efficient algorithm at this moment to solve the linear problem for the Wasserstein distance is Lee–Sidford linear solver [13], which runs in $O\left(n^{2.5}\right)$. Reference [14] proved that a complexity bound for the Sinkhorn algorithm is $\tilde{O}\left(n^2{\epsilon }^{-2}\right)$, where $\epsilon$ is the desired absolute performance guarantee. After [10] appeared, various algorithms have been proposed. For example, Reference [15] adopted stochastic optimization schemes for solving the optimal transport. The Greenkhorn algorithm [16] is the greedy variant of the Sinkhorn algorithm, and Reference [12] proposed its acceleration. Many other approaches such as adapting a variety of standard optimization algorithms to approximate the optimal transport problem can be found in [12,17,18,19]. Several specialized Newton-type algorithms [20,21] achieve complexity bound $\tilde{O}\left(n^2{\epsilon }^{-1}\right)$  [22,23], which are the best ones in terms of computational complexity at the present moment.

Moreover, entropy-regularized optimal transport has another advantage. Because of the differentiability of the entropy-regularized optimal transport and the simple structure of Sinkhorn’s algorithm, we can easily compute the gradient of the entropy-regularized optimal transport cost and optimize the parameter of a parametrized probability distribution by using numerical differentiation or automatic differentiation. Then, we can define a differentiable loss function that can be applied to various supervised learning methods [24]. Entropy-regularized optimal transport can be used to approximate not only the Wasserstein distance, but also its optimal coupling as a mapping function. Reference [25] adopted the optimal coupling of the entropy-regularized optimal transport as a mapping function from one domain to another.

Despite the empirical success of the entropy-regularized optimal transport, its theoretical aspect is less understood. Reference [26] studied the expected Wasserstein distance between a probability measure and its empirical version. Similarly, Reference [27] showed the consistency of the entropy-regularized optimal transport cost between two empirical distributions. Reference [28] showed that minimizing the entropy-regularized optimal transport cost between empirical distributions is equivalent to a type of maximum likelihood estimator. Reference [29] considered Wasserstein generative adversarial networks with an entropy regularization. Reference [30] constructed information geometry from the convexity of the entropy-regularized optimal transport cost.

Our intrinsic motivation of this study is to produce an analytical solution about the entropy-regularized optimal transport problem between continuous probability measures so that we can gain insight into the effects of entropy regularization in a theoretical, as  well as an experimental way. In our study, we generalized the Wasserstein distance between two multivariate normal distributions by entropy regularization. We derived the explicit form of the entropy-regularized optimal transport cost and its optimal coupling, which can be used to analyze the effect of entropy regularization directly. In general, the nonregularized Wasserstein distance between two probability measures and its optimal coupling cannot be expressed in a closed form; however, Reference [31] proved the explicit formula for multivariate normal distributions. Theorem 1 is a generalized form of [31]. We obtain an explicit form of the entropy-regularized optimal transport between two multivariate normal distributions. Furthermore, by adopting the Tsallis entropy [32] as the entropy regularization instead of the Shannon entropy, our theorem can be generalized to multivariate q-normal distributions.

Some readers may find it strange to study the entropy-regularized optimal transport for multivariate normal distributions, where the exact (nonregularized) optimal transport has been obtained explicitly. However, we think it is worth studying from several perspectives:

• Normal distributions are the simplest and best-studied probability distributions, and thus, it is useful to examine the regularization theoretically in order to infer results for other distributions. In particular, we will partly answer the questions “How much do entropy constraints affect the results?” and “What does it mean to constrain by the entropy?’’ for the simplest cases. Furthermore, as a first step in constructing a theory for more general probability distributions, in Section 4, we propose a generalization to multivariate q-normal distributions.
• Because normal distributions are the limit distributions in asymptotic theories using the central limit theorem, studying normal distributions is necessary for the asymptotic theory of regularized Wasserstein distances and estimators computed by them. Moreover, it was proposed to use the entropy-regularized Wasserstein distance to compute a lower bound of the generalization error for a variational autoencoder [29]. The study of the asymptotic behavior of such bounds is one of the expected applications of our results.
• Though this has not yet been proven theoretically, we suspect that entropy regularization is efficient not only for computational reasons, such as the use of the Sinkhorn algorithm, but also in the sense of efficiency in statistical inference. Such a phenomenon can be found in some existing studies, including [33]. Such statistical efficiency is confirmed by some experiments in Section 6.

The remainder of this paper is organized as follows. First, we review some definitions of optimal transport and entropy regularization in Section 2. Then, in Section 3, we provide an explicit form of the entropy-regularized optimal transport cost and its optimal coupling between two multivariate normal distributions. We also extend this result to q-normal distributions for Tsallis entropy regularization in Section 4. In Section 5, we obtain the entropy-regularized Kantorovich estimator of probability measures on ${\mathbb{R}}^n$ with a finite second moment that are absolutely continuous with respect to the Lebesgue measure in Theorem 3. We emphasize that Theorem 3 is not limited to the case of multivariate normal distribution, but can handle a wider range of probability measures. We analyze how entropy regularization affects the optimal result experimentally in certain sections.

We note that after publishing the preprint version of the paper, we found closely related results [34,35] reported within half a year. In Janati et al. [34], they proved the same result as Theorem 1 based on solving the fixed-point equation behind Sinkhorn’s algorithm. Their results include the unbalanced optimal transport between unbalanced multivariate normal distributions. They also studied the convexity and differentiability of the objective function of the entropy-regularized optimal transport. In [35], the same closed-form as Theorem 1 was proven by ingeniously using the Schrödinger system. Although there are some overlaps, our paper has significant novelty in the following respects. Our proof is more direct than theirs and can be extended directly to the proof for the Tsallis entropy-regularized optimal transport between multivariate q-normal distributions provided in Section 4. Furthermore, Corollaries 1 and 2 are novel and important results to evaluate how much the entropy regularization affects the estimation results or not at all. We also obtain the entropy-regularized Kantorovich estimator in Theorem 3.

2. Preliminary

In this section, we review some definitions of optimal transport and entropy-regularized optimal transport. These definitions were referred to in [1,36]. In this section, we use a tuple $(M,\mathsf{\Sigma })$ for a set M and $\sigma$-algebra on M and $\mathcal{P}\left(X\right)$ for the set of all probability measures on a measurable space X.

Definition 1

(Pushforward measure). Given measurable spaces $(M_1,{\mathsf{\Sigma }}_1)$ and $(M_2,{\mathsf{\Sigma }}_2)$, a measure $\mu :{\mathsf{\Sigma }}_1\to [0,+\infty ]$, and a measurable mapping $\phi :M_1\to M_2$, the pushforward measure of μ by φ is defined by:

$$\forall B\in {\mathsf{\Sigma }}_2,{\phi }_{\#}\mu \left(B\right):=\mu \left({\phi }^{-1}\left(B\right)\right).$$

(4)

Definition 2

(Optimal transport map). Consider a measurable space $(M,\mathsf{\Sigma })$, and let $c:M\times M\to {\mathbb{R}}_{+}$ denote a cost function. Given $\mu ,\nu \in \mathcal{P}\left(M\right)$, we call $\phi :M\to M$ the optimal transport map if φ realizes the infimum of:

$$\underset{{\phi }_{\#}\mu =\nu }{inf}{\int }_{M}c(x,\phi \left(x\right))d\mu \left(x\right).$$

(5)

This problem was originally formalized by [37]. However, the optimal transport map does not always exist. Then, Kantorovich considered a relaxation of this problem in [38].

Definition 3

(Coupling). Given $\mu ,\nu \in \mathcal{P}\left(M\right)$, the coupling of μ and ν is a probability measure on $M\times M$ that satisfies:

$$\forall A\in \mathsf{\Sigma },\pi (A\times M)=\mu \left(A\right),\pi (M\times A)=\nu \left(A\right).$$

(6)

Definition 4

(Kantorovich problem). The Kantorovich problem is defined as finding a coupling π of μ and ν that realizes the infimum of:

$${\int }_{M\times M}c(x,y)d\pi (x,y).$$

(7)

Hereafter, let $\Pi (\mu ,\nu )$ be the set of all couplings of $\mu$ and $\nu$. When we adopt a distance function as the cost function, we can define the Wasserstein distance.

Definition 5

(Wasserstein distance). Given $p\ge 1$, a measurable metric space $(M,\mathsf{\Sigma },d)$, and $\mu ,\nu \in \mathcal{P}\left(M\right)$ with a finite p-th moment, the p-Wasserstein distance between μ and ν is defined as:

$$W_p(\mu ,\nu ):=\underset{\pi \in \Pi (\mu ,\nu )}{inf}{\left({\int }_{M\times M}d{(x,y)}^{p}d\pi (x,y)\right)}^{\frac{1}{p}}.$$

(8)

Now, we review the definition of entropy-regularized optimal transport on ${\mathbb{R}}^n$.

Definition 6

(Entropy-regularized optimal transport). Let $\mu ,\nu \in \mathcal{P}\left({\mathbb{R}}^n\right)$, $\lambda >0$, and let $\pi (x,y)$ be the density function of the coupling of μ and ν, whose reference measure is the Lebesgue measure. We define the entropy-regularized optimal transport cost as:

$$C_{\lambda }(\mu ,\nu ):=\underset{\pi \in \Pi (\mu ,\nu )}{inf}{\int }_{{\mathbb{R}}^n\times {\mathbb{R}}^n}c(x,y)\pi (x,y)dxdy-\lambda \mathrm{Ent}(\pi ),$$

(9)

where $\mathrm{Ent}(·)$ denotes the Shannon entropy of a probability measure:

$$\mathrm{Ent}\left(\pi \right)=-{\int }_{{\mathbb{R}}^n\times {\mathbb{R}}^n}\pi (x,y)log\pi (x,y)dxdy.$$

(10)

There is another variation in entropy-regularized optimal transport defined by the relative entropy instead of the Shannon entropy:

$${\tilde{C}}_{\lambda }(\mu ,\nu ):=\underset{\pi \in \Pi (\mu ,\nu )}{inf}{\int }_{{\mathbb{R}}^n\times {\mathbb{R}}^n}c(x,y)\pi (x,y)dxdy+\lambda \mathrm{KL}\left(\pi \right|d\mu \otimes d\nu ).$$

(11)

This is definable even when $\Pi (\mu ,\nu )$ includes a coupling that is not absolutely continuous with respect to the Lebesgue measure. We note that when both $\mu$ and $\nu$ are absolutely continuous, the infimum is attained by the same $\pi$ for $C_{\lambda }$ and ${\tilde{C}}_{\lambda }$, and it depends only on $\mu$ and $\nu$. In the following part of the paper, we assume the absolute continuity of $\mu ,\nu$, and $\pi$ with respect to the Lebesgue measure for well-defined entropy regularization.

3. Entropy-Regularized Optimal Transport between Multivariate Normal Distributions

In this section, we provide a rigorous solution of entropy-regularized optimal transport between two multivariate normal distributions. Throughout this section, we adopt the squared Euclidean distance ${\parallel x-y\parallel }^2$ as the cost function. To prove our theorem, we start by expressing $C_{\lambda }$ using mean vectors and covariance matrices. The following lemma is a known result; for example, see [31].

Lemma 1.

Let $X\sim P,Y\sim Q$ be two random variables on ${\mathbb{R}}^n$ with means ${\mu }_1,{\mu }_2$ and covariance matrices ${\mathsf{\Sigma }}_1,{\mathsf{\Sigma }}_2$, respectively. If $\pi (x,y)$ is a coupling of P and Q, we have:

$${\int }_{{\mathbb{R}}^n\times {\mathbb{R}}^n}{\parallel x-y\parallel }^2\pi (x,y)dxdy={\parallel {\mu }_1-{\mu }_2\parallel }^2+\mathrm{tr}\left\{{\mathsf{\Sigma }}_1+{\mathsf{\Sigma }}_2-2\mathrm{Cov}(X,Y)\right\}.$$

(12)

Proof. 

Without loss of generality, we can assume X and Y are centralized, because:

$$\int \parallel (x-{\mu }_1)-(y-{\mu }_2){\parallel }^2\pi (x,y)dxdy={\int \parallel x-y\parallel }^2\pi (x,y)dxdy-{\parallel {\mu }_1-{\mu }_2\parallel }^2.$$

(13)

Therefore, we have:

$$\begin{array}{cc}\hfill {\int \parallel x-y\parallel }^2\pi (x,y)dxdy& ={E[\parallel X-Y\parallel }^2]=E\left[\mathrm{tr}\left\{(X-Y){(X-Y)}^{\mathrm{T}}\right\}\right]\hfill \\ \hfill & =\mathrm{tr}\left\{{\mathsf{\Sigma }}_1+{\mathsf{\Sigma }}_2-2\mathrm{Cov}(X,Y)\right\}.\hfill \end{array}$$

(14)

By adding $\parallel {\mu }_1-{\mu }_2{\parallel }^2$, we obtain (12).  □

Lemma 1 shows that ${\int }_{{\mathbb{R}}^n\times {\mathbb{R}}^n}{\parallel x-y\parallel }^2\pi (x,y)dxdy$ can be parameterized by the covariance matrices ${\mathsf{\Sigma }}_1,{\mathsf{\Sigma }}_2,\mathrm{Cov}(X,Y)$. Because ${\mathsf{\Sigma }}_1$ and ${\mathsf{\Sigma }}_2$ are fixed, the infinite-dimensional optimization of the coupling $\pi$ is a finite-dimensional optimization of covariance matrix $\mathrm{Cov}(X,Y)$.

We prepare the following lemma to prove Theorem 1.

Lemma 2.

Under a fixed mean and covariance matrix, the probability measure that maximizes the entropy is a multivariate normal distribution.

Lemma 2 is a particular case of the principle of maximum entropy [39], and the proof can be found in [40] Theorem 3.1.

Theorem 1.

Let $P\sim \mathcal{N}({\mu }_1,{\mathsf{\Sigma }}_1),Q\sim \mathcal{N}({\mu }_2,{\mathsf{\Sigma }}_2)$ be two multivariate normal distributions. The optimal coupling π of P and Q of the entropy-regularized optimal transport:

$$C_{\lambda }(P,Q)=\underset{\pi \in \Pi (P,Q)}{inf}{\int }_{{\mathbb{R}}^n\times {\mathbb{R}}^n}{\parallel x-y\parallel }^2\pi (x,y)dxdy-4\lambda \mathrm{Ent}\left(\pi \right).(⁎)$$

(15)

is expressed as:

$$\pi \sim \mathcal{N}\left(\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right),\left(\begin{array}{cc}{\mathsf{\Sigma }}_1& {\mathsf{\Sigma }}_{\lambda }\\ {\mathsf{\Sigma }}_{\lambda }^{\mathrm{T}}& {\mathsf{\Sigma }}_2\end{array}\right)\right)$$

(16)

where:

$${\mathsf{\Sigma }}_{\lambda }:={\mathsf{\Sigma }}_1^{1/2}{({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\lambda }^{2}I)}^{1/2}{\mathsf{\Sigma }}_1^{-1/2}-\lambda I.$$

(17)

Furthermore, $C_{\lambda }(P,Q)$ can be written as:

$$\begin{array}{cc}\hfill C_{\lambda }(P,Q)=& \parallel {\mu }_1-{\mu }_2{\parallel }^2+\mathrm{tr}({\mathsf{\Sigma }}_1+{\mathsf{\Sigma }}_2-2{({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\lambda }^{2}I)}^{1/2})\hfill \\ \hfill -& 2\lambda log|{({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\lambda }^{2}I)}^{1/2}-\lambda I|-2\lambda nlog\left(2\pi \lambda \right)-4\lambda nlog\left(2\pi \right)-2\lambda n\hfill \end{array}$$

(18)

and the relative entropy version can be written as:

$${\tilde{C}}_{\lambda }(P,Q)=C_{\lambda }(P,Q)+2\lambda log|{\mathsf{\Sigma }}_1\left|\right|{\mathsf{\Sigma }}_2|+4\lambda n\{log\left(2\pi \right)+1\}.$$

(19)

We note that we use the regularization parameter $4\lambda$ in $(⁎)$ for the sake of simplicity.

Proof. 

Although the first half of the proof can be derived directly from Lemma 2, we provide a proof of this theorem by Lagrange calculus, which will be used later for the extension to q-normal distributions. Now, we define an optimization problem that is equivalent to the entropy-regularized optimal transport as follows:

$$\begin{array}{cc}\hfill \mathrm{minimize}& {\int \parallel x-y\parallel }^2\pi (x,y)dxdy-4\lambda \mathrm{Ent}\left(\pi \right)\hfill \\ \hfill \mathrm{subject}\mathrm{to}& \int \pi (x,y)dx=q\left(y\right)\mathrm{for}\forall y\in {\mathbb{R}}^n,\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & \int \pi (x,y)dy=p\left(x\right)\mathrm{for}\forall x\in {\mathbb{R}}^n.\hfill \end{array}$$

(21)

Here, $p\left(x\right)$ and $q\left(y\right)$ are probability density functions of P and Q, respectively. Let $\alpha \left(x\right)$, $\beta \left(y\right)$ be Lagrange multipliers that correspond to the above two constraints. The Lagrangian function of (20) is defined as:

$$\begin{array}{cc}\hfill L(\pi ,\alpha ,\beta ):& ={\int \parallel x-y\parallel }^2\pi (x,y)dxdy+4\lambda \int \pi (x,y)log\pi (x,y)dxdy\hfill \\ \hfill & -\int \alpha (x)\pi (x,y)dxdy+\int \alpha (x)p(x)dx\hfill \\ \hfill & -\int \beta (y)\pi (x,y)dxdy+\int \beta (y)q(y)dy.\hfill \end{array}$$

(22)

Taking the functional derivative of (22) with respect to $\pi$, we obtain:

$$\delta L(\pi ,\alpha ,\beta )=\int \left({\parallel x-y\parallel }^2+4\lambda log\pi (x,y)-\alpha \left(x\right)-\beta \left(y\right)\right)\delta \pi (x,y)dxdy.$$

(23)

By the fundamental lemma of the calculus of variations, we have:

$$\pi (x,y)\propto exp\left(\alpha \left(x\right)+\beta \left(y\right)-\frac{{\parallel x-y\parallel }^2}{4\lambda }\right).$$

(24)

Here, $\alpha \left(x\right),\beta \left(y\right)$ are determined from the constraints (21). We can assume that $\pi$ is a $2n$-variate normal distribution, because for a fixed covariance matrix $\mathrm{Cov}(X,Y)$, $-\mathrm{Ent}\left(\pi \right)$ takes the infimum when the coupling $\pi$ is a multivariate normal distribution by Lemma 2. Therefore, we can express $\pi$ by using $z={(x^{\mathrm{T}},y^{\mathrm{T}})}^{\mathrm{T}}$ and a covariance matrix $\mathsf{\Sigma }:=\mathrm{Cov}(X,Y)$ as:

$$\pi (x,y)\propto exp\left\{-\frac{1}{2}{z}^{\mathrm{T}}{\left(\begin{array}{cc}{\mathsf{\Sigma }}_1& {\mathsf{\Sigma }}_{\phantom{2}}\\ {\mathsf{\Sigma }}^{\mathrm{T}}& {\mathsf{\Sigma }}_2\end{array}\right)}^{-1}z\right\}.$$

(25)

Putting:

$$\left(\begin{array}{cc}{\tilde{\mathsf{\Sigma }}}_1& \tilde{\mathsf{\Sigma }}\\ {\tilde{\mathsf{\Sigma }}}^{\mathrm{T}}& {\tilde{\mathsf{\Sigma }}}_2\end{array}\right):={\left(\begin{array}{cc}{\mathsf{\Sigma }}_1& {\mathsf{\Sigma }}_{\phantom{2}}\\ {\mathsf{\Sigma }}^{\mathrm{T}}& {\mathsf{\Sigma }}_2\end{array}\right)}^{-1},$$

(26)

we write:

$$\begin{array}{cc}\hfill -\frac{1}{2}{z}^{\mathrm{T}}{\left(\begin{array}{cc}{\mathsf{\Sigma }}_1& {\mathsf{\Sigma }}_{\phantom{2}}\\ {\mathsf{\Sigma }}^{\mathrm{T}}& {\mathsf{\Sigma }}_2\end{array}\right)}^{-1}z& =-\frac{1}{2}\left(\begin{array}{cc}{x}^{\mathrm{T}}& y^{\mathrm{T}}\end{array}\right)\left(\begin{array}{cc}{\tilde{\mathsf{\Sigma }}}_1& \tilde{\mathsf{\Sigma }}\\ {\tilde{\mathsf{\Sigma }}}^{\mathrm{T}}& {\tilde{\mathsf{\Sigma }}}_2\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill & =-\frac{1}{2}{x}^{\mathrm{T}}{\tilde{\mathsf{\Sigma }}}_{1}x-\frac{1}{2}{y}^{\mathrm{T}}{\tilde{\mathsf{\Sigma }}}_{2}y-x^{\mathrm{T}}\tilde{\mathsf{\Sigma }}y.\hfill \end{array}$$

(28)

According to block matrix inversion formula [41], $\tilde{\mathsf{\Sigma }}=-{\mathsf{\Sigma }}_1^{-1}\mathsf{\Sigma }{A}^{-1}$ holds, where $A:={\mathsf{\Sigma }}_2-{\mathsf{\Sigma }}^{\mathrm{T}}{\mathsf{\Sigma }}_1^{-1}\mathsf{\Sigma }$ is positive definite. Then, comparing the term $x^{\mathrm{T}}y$ between (24) and (28), we obtain ${\mathsf{\Sigma }}_1^{-1}\mathsf{\Sigma }{A}^{-1}=\frac{1}{2\lambda }I$ and:

$$2\lambda {\mathsf{\Sigma }}_1^{-1}\mathsf{\Sigma }=A={\mathsf{\Sigma }}_2-{\mathsf{\Sigma }}^{\mathrm{T}}{\mathsf{\Sigma }}_1^{-1}\mathsf{\Sigma }.$$

(29)

Here, ${\mathsf{\Sigma }}_1^{-1}\mathsf{\Sigma }={\mathsf{\Sigma }}^{\mathrm{T}}{\mathsf{\Sigma }}_1^{-1}$ holds, because A is a symmetric matrix, and thus, we obtain:

$$\lambda {\mathsf{\Sigma }}_1^{-1}\mathsf{\Sigma }+\lambda {\mathsf{\Sigma }}^{\mathrm{T}}{\mathsf{\Sigma }}_1^{-1}={\mathsf{\Sigma }}_2-{\mathsf{\Sigma }}^{\mathrm{T}}{\mathsf{\Sigma }}_1^{-1}\mathsf{\Sigma }.$$

(30)

Completing the square of the above equation, we obtain:

$${\left({\mathsf{\Sigma }}_1^{-1/2}(\mathsf{\Sigma }+\lambda I){\mathsf{\Sigma }}_1^{1/2}\right)}^{\mathrm{T}}\left({\mathsf{\Sigma }}_1^{-1/2}(\mathsf{\Sigma }+\lambda I){\mathsf{\Sigma }}_1^{1/2}\right)={\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\lambda }^{2}I$$

(31)

Let Q be an orthogonal matrix; then, (31) can be solved as:

$${\mathsf{\Sigma }}_1^{-1/2}(\mathsf{\Sigma }+\lambda I){\mathsf{\Sigma }}_1^{1/2}=Q{({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\lambda }^{2}I)}^{1/2}.$$

(32)

We rearrange the above equation as follows:

$${\mathsf{\Sigma }}_1^{1/2}\left({\mathsf{\Sigma }}_1^{-1}\mathsf{\Sigma }\right){\mathsf{\Sigma }}_1^{1/2}+\lambda I=Q{({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\lambda }^{2}I)}^{1/2}.$$

(33)

Because the left terms and ${({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\lambda }^{2}I)}^{1/2}$ are all symmetric positive definite, we can conclude that Q is the identity matrix by the uniqueness of the polar decomposition. Finally, we obtain:

$$\mathsf{\Sigma }={\mathsf{\Sigma }}_1^{1/2}{({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\lambda }^{2}I)}^{1/2}{\mathsf{\Sigma }}_1^{-1/2}-\lambda I=:{\mathsf{\Sigma }}_{\lambda }.$$

(34)

We obtain (18) by the direct calculation of $C_{\lambda }$ using Lemma 1 with this ${\mathsf{\Sigma }}_{\lambda }$.  □

The following corollary helps us to understand the properties of ${\mathsf{\Sigma }}_{\lambda }$.

Corollary 1.

Let ${\nu }_{\lambda ,1}\le {\nu }_{\lambda ,2}\le \dot{\le }{\nu }_{\lambda ,n}$ be the eigenvalues of ${\mathsf{\Sigma }}_{\lambda }$; then, ${\nu }_{\lambda ,i}$ monotonically decreases with λ for any $i\in \{1,2,\dot{,}n\}$.

Proof. 

Because ${\mathsf{\Sigma }}_1^{-1/2}{\mathsf{\Sigma }}_{\lambda }{\mathsf{\Sigma }}_1^{1/2}={({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\lambda }^{2}I)}^{1/2}-\lambda I$ has the same eigenvalues as ${\mathsf{\Sigma }}_{\lambda }$, if we let $\left\{{\nu }_{0,i}\right\}$ be the eigenvalues of ${\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}$, ${\nu }_{\lambda ,i}=\sqrt{{\nu }_{0,i}+{\lambda }^2}-\lambda$, which is a monotonically decreasing function of the regularization parameter $\lambda$.  □

By the proof, for large $\lambda$, we can prove ${\mathsf{\Sigma }}_1^{-1/2}{\mathsf{\Sigma }}_{\lambda }{\mathsf{\Sigma }}_1^{1/2}\simeq \frac{1}{2\lambda }{\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}$ by diagonalization and ${\nu }_{\lambda ,i}\simeq \frac{1}{2\lambda }{\nu }_{0,i}$. Thus, ${\mathsf{\Sigma }}_{\lambda }\simeq \frac{1}{2\lambda }{\mathsf{\Sigma }}_1{\mathsf{\Sigma }}_2$, and each element of ${\mathsf{\Sigma }}_{\lambda }$ converges to zero as $\lambda \to \infty$.

We show how entropy regularization behaves in two simple experiments. We calculate the entropy-regularized optimal transport cost $\mathcal{N}\left(\left(\begin{array}{c}0\\ 0\end{array}\right),\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)\right)$ and $\mathcal{N}\left(\left(\begin{array}{c}0\\ 0\end{array}\right),\left(\begin{array}{cc}2& -1\\ -1& 2\end{array}\right)\right)$ in the original version and the relative entropy version in Figure 1. We separate the entropy-regularized optimal transport cost into the transport cost term and regularization term and display both of them.

It is reasonable that as $\lambda \downarrow 0$, ${\mathsf{\Sigma }}_{\lambda }$ converges to ${\mathsf{\Sigma }}_1^{1/2}{\left({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}\right)}^{1/2}{\mathsf{\Sigma }}_1^{-1/2}$, which is equal to the original optimal coupling of nonregularized optimal transport and as $\lambda \to \infty$, ${\mathsf{\Sigma }}_{\lambda }$ converges to 0. This is a special case of Corollary 1.The larger $\lambda$ becomes, the less correlated the optimal coupling is. We visualize this behavior by computing the optimal couplings of two one-dimensional normal distributions in Figure 2.

The left panel shows the original version. The transport cost is always positive, and the entropy regularization term can take both signs in general; then, the sign and total cost depend on their balance. We note that the transport cost as a function of $\lambda$ is bounded, whereas the entropy regularization is not. The boundedness of the optimal cost is deduced from (1) and Corollary 1, and the unboundedness of the entropy regularization is due to the regularization parameter $\lambda$ multiplied by the entropy. The right panel shows the relative entropy version. It always takes a non-negative value. Furthermore, because the total cost is bounded by the value for the independent joint distribution (which is always a feasible coupling), both the transport cost and the relative entropy regularization regularization term are also bounded. Nevertheless, the larger the regularization parameter $\lambda$, the greater the influence of entropy regularization over the total cost.

It is known that a specific Riemannian metric can be defined in the space of multivariate normal distributions, which induces the Wasserstein distance [42]. To understand the effect of entropy regularization, we illustrate how entropy regularization deforms this geometric structure in Figure 3. Here, we generate 100 two-variate normal distributions ${\left\{\mathcal{N}(0,{\mathsf{\Sigma }}_{r,k})\right\}}_{r,k\in \{1,2,\dot{,}10\}}$, where $\left\{{\mathsf{\Sigma }}_{r,k}\right\}$ is defined as:    

$${\mathsf{\Sigma }}_{r,k}={\left(\begin{array}{cc}cos\left(2\pi ·\frac{k}{10}\right)& -sin\left(2\pi ·\frac{k}{10}\right)\\ sin\left(2\pi ·\frac{k}{10}\right)& cos\left(2\pi ·\frac{k}{10}\right)\end{array}\right)}^{\mathrm{T}}\left(\begin{array}{cc}1& 0\\ 0& \sqrt{\frac{r}{10}}\end{array}\right)\left(\begin{array}{cc}cos\left(2\pi ·\frac{k}{10}\right)& -sin\left(2\pi ·\frac{k}{10}\right)\\ sin\left(2\pi ·\frac{k}{10}\right)& cos\left(2\pi ·\frac{k}{10}\right)\end{array}\right).$$

(35)

To visualize the geometric structure of these two-variate normal distributions, we compute the relative entropy-regularized optimal transport cost ${\tilde{C}}_{\lambda }$ between each pairwise two-variate normal distributions. Then, we apply multidimensional scaling [43] to embed them into a plane (see Figure 3). We can see entropy regularization deforming the geometric structure of the space of multivariate normal distributions. The deformation for distributions close to the isotopic normal distribution is more sensitive to the change in $\lambda$.

The following corollary states that if we allow orthogonal transformations of two multivariate normal distributions with fixed covariance matrices, then the minimum and maximum of $C_{\lambda }$ are attained when ${\mathsf{\Sigma }}_1$ and ${\mathsf{\Sigma }}_2$ are diagonalizable by the same orthogonal matrix or, equivalently, when the ellipsoidal contours of the two density functions are aligned with the same orthogonal axes.

Corollary 2.

With the same settings as in Theorem 1, fix ${\mu }_1$, ${\mu }_2$, ${\mathsf{\Sigma }}_1$, and all eigenvalues of ${\mathsf{\Sigma }}_2$. When ${\mathsf{\Sigma }}_1$ is diagonalized as ${\mathsf{\Sigma }}_1={\mathsf{\Gamma }}^{\mathrm{T}}{\mathsf{\Lambda }}_1^{\downarrow }\mathsf{\Gamma }$, where ${\mathsf{\Lambda }}_1^{\downarrow }$ is the diagonal matrix of the eigenvalues of ${\mathsf{\Sigma }}_1$ in descending order and Γ is an orthogonal matrix,

(i) 

$C_{\lambda }(P,Q)$ is minimized by ${\mathsf{\Sigma }}_2={\mathsf{\Gamma }}^{\mathrm{T}}{\mathsf{\Lambda }}_2^{\downarrow }\mathsf{\Gamma }$ and

(ii) 

$C_{\lambda }(P,Q)$ is maximized by ${\mathsf{\Sigma }}_2={\mathsf{\Gamma }}^{\mathrm{T}}{\mathsf{\Lambda }}_2^{\uparrow }\mathsf{\Gamma }$,

where ${\mathsf{\Lambda }}_2^{\downarrow }$ and ${\mathsf{\Lambda }}_2^{\uparrow }$ are the diagonal matrices of the eigenvalues of ${\mathsf{\Sigma }}_2$ in descending and ascending order, respectively. Therefore, neither the minimizer, nor the maximizer depend on the choice of λ.

Proof. 

Because ${\mu }_1$, ${\mu }_2$, ${\mathsf{\Sigma }}_1$, and all eigenvalues of ${\mathsf{\Sigma }}_2$ are fixed,

$$\begin{array}{cc}\hfill C_{\lambda }(P,Q)& =-2\mathrm{tr}\left({({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\lambda }^{2}I)}^{1/2}\right)-\frac{\lambda }{2}log|{({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\lambda }^{2}I)}^{1/2}-\lambda I|+\left(\mathrm{constant}\right)\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill & =\sum _{i=1}^n-2{({\nu }_i+{\lambda }^2)}^{1/2}-\frac{\lambda }{2}log\{{({\nu }_i+{\lambda }^2)}^{1/2}-\lambda \}+\left(\mathrm{constant}\right)\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill & =\sum _{i=1}^n{g}_{\lambda }(log\left({\nu }_i\right))+\left(\mathrm{constant}\right)\hfill \end{array}$$

(38)

where ${\nu }_1\le \cdots \le {\nu }_n$ are the eigenvalues of ${\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}$ and:

$$g_{\lambda }\left(x\right):=-2{(e^x+{\lambda }^2)}^{1/2}-\frac{\lambda }{2}log\{{(e^x+{\lambda }^2)}^{1/2}-\lambda \}.$$

(39)

Note that $g_{\lambda }\left(x\right)$ is a concave function, because:

$$g_{\lambda }^{\prime \prime }\left(x\right)=-\frac{e^x(4e^x+7{\lambda }^2)}{8{(e^x+{\lambda }^2)}^{3/2}}<0.$$

(40)

Let ${\nu }_1^{\downarrow \downarrow }\le \cdots \le {\nu }_n^{\downarrow \downarrow }$ and ${\nu }_1^{\downarrow \uparrow }\le \cdots \le {\nu }_n^{\downarrow \uparrow }$ be the eigenvalues of ${\mathsf{\Lambda }}_1^{\downarrow }{\mathsf{\Lambda }}_2^{\downarrow }$ and ${\mathsf{\Lambda }}_1^{\downarrow }{\mathsf{\Lambda }}_2^{\uparrow }$, respectively. By Exercise 6.5.3 of [44] or Theorem 6.13 and Corollary 6.14 of [45],

$$\begin{array}{c}\hfill (log\left({\nu }_i^{\downarrow \uparrow }\right))\prec (log\left({\nu }_i\right))\prec (log\left({\nu }_i^{\downarrow \downarrow }\right)),\end{array}$$

(41)

Here, for $\left(a_i\right),\left(b_i\right)\in {\mathbb{R}}^n$ such that $a_1\ge \cdots \ge a_n$ and $b_1\ge \cdots \ge b_n$, $\left(a_i\right)\prec \left(b_i\right)$ means:

$$\sum _{i=1}^k{a}_i\le \sum _{i=1}^k{b}_i\mathrm{for}k=1,\cdots ,n-1,\mathrm{and}\sum _{i=1}^n{a}_i=\sum _{i=1}^n{b}_i$$

(42)

and $\left(a_i\right)$ is said to be majorized by $\left(b_i\right)$. Because $g_{\lambda }\left(x\right)$ is concave,

$$g_{\lambda }(log\left({\nu }_i^{\downarrow \uparrow }\right)){\prec }^{\mathrm{w}}{g}_{\lambda }(log\left({\nu }_i\right)){\prec }^{\mathrm{w}}{g}_{\lambda }(log\left({\nu }_i^{\downarrow \downarrow }\right)),$$

(43)

where ${\prec }^{\mathrm{w}}$ represents weak supermajorization, i.e., $\left(a_i\right){\prec }^{\mathrm{w}}\left(b_i\right)$ means:

$$\sum _{i=k}^n{a}_i\ge \sum _{i=k}^n{b}_i\mathrm{for}k=1,\cdots ,n$$

(44)

(see Theorem 5.A.1 of [46], for example). Therefore,

$$\sum _{i=1}^n{g}_{\lambda }(log\left({\nu }_i^{\downarrow \uparrow }\right))\ge \sum _{i=1}^n{g}_{\lambda }(log\left({\nu }_i\right))\ge \sum _{i=1}^n{g}_{\lambda }(log\left({\nu }_i^{\downarrow \downarrow }\right)).$$

(45)

As in Case (i) (or (ii)), the eigenvalues of ${\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}$ correspond to the eigenvalues of ${\mathsf{\Lambda }}_1^{\downarrow }{\mathsf{\Lambda }}_2^{\downarrow }$ (or ${\mathsf{\Lambda }}_1^{\downarrow }{\mathsf{\Lambda }}_2^{\uparrow }$, respectively), the corollary follows.  □

Note that a special case of Corollary 2 for the ordinary Wasserstein metric ($\lambda =0$) has been studied in the context of fidelity and the Bures distance in quantum information theory. See Lemma 3 of [47]. Their proof is not directly applicable to our generalized result; thus, we used another approach to prove it.

4. Extension to Tsallis Entropy Regularization

In this section, we consider a generalization of entropy-regularized optimal transport. We now focus on the Tsallis entropy [32], which is a generalization of the Shannon entropy and appears in nonequilibrium statistical mechanics. We show that the optimal coupling of Tsallis entropy-regularized optimal transport between two q-normal distributions is also a q-normal distribution. We start by recalling the definition of the q-exponential function and q-logarithmic function based on [32].

Definition 7.

Let q be a real parameter, and let $u>0$. The q-logarithmic function is defined as:

$${log}_q\left(u\right):=\left\{\begin{array}{c}\frac{1}{1-q}(u^{1-q}-1)ifq\ne 1,\hfill \\ log\left(u\right)ifq=1\hfill \end{array}\right.$$

(46)

and the q-exponential function is defined as:

$${exp}_q\left(u\right):=\left\{\begin{array}{c}{[1+(1-q)u]}_{+}^{\frac{1}{1-q}}ifq\ne 1,\hfill \\ exp\left(u\right)ifq=1\hfill \end{array}\right.$$

(47)

Definition 8.

Let $q<1$ or $1<q<1+\frac{2}{n}$; an n-variate q-normal distribution is defined by two parameters: $\mu \in {\mathbb{R}}^n$ and a positive definite matrix Σ, and its density function is:

$$f\left(x\right):=\frac{1}{C_q\left(\mathsf{\Sigma }\right)}{exp}_q\left(-{(x-\mu )}^{\mathrm{T}}{\mathsf{\Sigma }}^{-1}(x-\mu )\right),$$

(48)

where $C_q\left(\mathsf{\Sigma }\right)$ is a normalizing constant. μ and Σ are called the location vector and scale matrix, respectively.

In the following, we write the multivariate q-normal distribution ${\mathcal{N}}_q(\mu ,\mathsf{\Sigma })$. We note that the property of the q-normal distribution changes in accordance with q. The q-normal distribution has an unbounded support for $1<q<\frac{2}{n}$ and a bounded support for $q<1$. The second moment exists for $q<1+\frac{2}{n+2}$, and the covariance becomes $\frac{1}{2+(n+2)(1-q)}\mathsf{\Sigma }$. We remark that each n-variate $\left(1+\frac{2}{\nu +n}\right)$-normal distribution is equivalent to an n-variate t-distribution with $\nu$ degrees of freedom,

$$\frac{\mathsf{\Gamma }\left[\right(\nu +n)/2]}{\mathsf{\Gamma }(\nu /2){\nu }^{n/2}{\pi }^{n/2}{\left|\mathsf{\Sigma }\right|}^{1/2}}{\left[1+\frac{1}{\nu }{(x-\mu )}^T{\mathsf{\Sigma }}^{-1}(x-\mu )\right]}^{-(\nu +n)/2},$$

(49)

for $1<q<1+\frac{2}{n+2}$ and an n-variate normal distribution for $q\downarrow 1$.

Definition 9.

Let p be a probability density function. The Tsallis entropy is defined as:

$$S_q\left(p\right):=\int p\left(x\right){log}_q\frac{1}{p\left(x\right)}dx=\frac{1}{q-1}\left(1-\int p{\left(x\right)}^{q}dx\right).$$

(50)

Then, the Tsallis entropy-regularized optimal transport is defined as:

$$\begin{array}{cc}\hfill \mathrm{minimize}& {\int \parallel x-y\parallel }^2\pi (x,y)dxdy-2\lambda S_q\left(\pi \right)\hfill \\ \hfill \mathrm{subject}\mathrm{to}& \int \pi (x,y)dx=q\left(y\right)\mathrm{for}\forall y\in {\mathbb{R}}^n,\hfill \end{array}$$

(51)

$$\begin{array}{cc}\hfill & \int \pi (x,y)dy=p\left(x\right)\mathrm{for}\forall x\in {\mathbb{R}}^n.\hfill \end{array}$$

(52)

The following lemma is a generalization of the maximum entropy principle for the Shannon entropy shown in Section 2 of [48].

Lemma 3.

Let P be a centered n-dimensional probability measure with a fixed covariance matrix Σ; the maximizer of the Renyi α-entropy:

$$\frac{1}{1-\alpha }log\int f{\left(x\right)}^{\alpha }\mathrm{d}x$$

(53)

under the constraint is ${\mathcal{N}}_{2-\alpha }(\mathbf{0},((n+2)\alpha -n)\mathsf{\Sigma })$ for $\frac{n}{n+2}<\alpha <1$.

We note that the maximizers of the Renyi $\alpha$-entropy and the Tsallis entropy with $q=\alpha$ coincide; thus, the above lemma also holds for the Tsallis entropy. This is mentioned, for example, in Section 9 of [49].

To prove Theorem 2, we use the following property of multivariate t-distributions, which is summarized in Chapter 1 of [50].

Lemma 4.

Let X be a random vector following an n-variate t-distribution with degree of freedom ν. Considering a partition of the mean vector μ and scale matrix Σ, such as:

$$X=\left(\begin{array}{c}{X}_1\hfill \\ X_2\hfill \end{array}\right),\mu =\left(\begin{array}{c}{\mu }_1\hfill \\ {\mu }_2\hfill \end{array}\right),\mathsf{\Sigma }=\left(\begin{array}{cc}{\mathsf{\Sigma }}_{11}\hfill & {\mathsf{\Sigma }}_{12}\hfill \\ {\mathsf{\Sigma }}_{21}\hfill & {\mathsf{\Sigma }}_{22}\hfill \end{array}\right),$$

(54)

$X_1$ follows a p-variate t-distribution with degree of freedom ν, mean vector ${\mu }_1$, and scale matrix ${\mathsf{\Sigma }}_{11}$, where p is the dimension of $X_1$.

Recalling the correspondence of the parameter of the multivariate q-normal distribution and the degree of freedom of the multivariate t-distribution $q=1+\frac{2}{\nu +n}$, we can obtain the following corollary.

Corollary 3.

Let X be a random vector following an n-variate q-normal distribution for $1<q<1+\frac{2}{n+2}$. Consider a partition of the mean vector μ and scale matrix Σ in the same way as in (54). Then, $X_1$ follows a p-variate $\left(1+\frac{2(q-1)}{2-(n-p)(q-1)}\right)$-normal distribution with mean vector ${\mu }_1$ and scale matrix ${\mathsf{\Sigma }}_{11}$, where p is the dimension of $X_1$.

Theorem 2.

Let $P\sim {\mathcal{N}}_q({\mu }_1,{\mathsf{\Sigma }}_1),Q\sim {\mathcal{N}}_q({\mu }_2,{\mathsf{\Sigma }}_2)$ be n-variate q-normal distributions for $1<q<1+\frac{2}{n+2}$ and $\tilde{q}=-\frac{2(q-1)}{2-n(q-1)}$; consider the Tsallis entropy-regularized optimal transport:

$$C_{\lambda }(P,Q)=\underset{\pi \in \Pi (P,Q)}{inf}{\int }_{{\mathbb{R}}^n\times {\mathbb{R}}^n}{\parallel x-y\parallel }^2\pi (x,y)dxdy-2\lambda S_{1+\tilde{q}}\left(\pi \right).$$

(55)

Then, there exists a unique $\tilde{\lambda }=\tilde{\lambda }(q,{\mathsf{\Sigma }}_1,{\mathsf{\Sigma }}_2,\lambda )\in {\mathbb{R}}_{+}$ such that the optimal coupling π of the entropy-regularized optimal transport is expressed as:

$$\pi \sim {\mathcal{N}}_{1-\tilde{q}}\left(\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right),\left(\begin{array}{cc}{\mathsf{\Sigma }}_1& {\mathsf{\Sigma }}_{\tilde{\lambda }}\\ {\mathsf{\Sigma }}_{\tilde{\lambda }}^{\mathrm{T}}& {\mathsf{\Sigma }}_2\end{array}\right)\right),$$

(56)

where:

$${\mathsf{\Sigma }}_{\tilde{\lambda }}:={\mathsf{\Sigma }}_1^{1/2}{({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\tilde{\lambda }}^{2}I)}^{1/2}{\mathsf{\Sigma }}_1^{-1/2}-\tilde{\lambda }I.$$

(57)

Proof. 

The proof proceeds in a similar way as in Theorem 1. Let $\alpha \in L\left(P\right)$ and $\beta \in L\left(Q\right)$ be the Lagrangian multipliers. Then, the Lagrangian function $L(\pi ,\alpha ,\beta )$ of (52) is defined as:

$$\begin{array}{cc}\hfill L(\pi ,\alpha ,\beta ):=& {\int \parallel x-y\parallel }^2\pi (x,y)dxdy-2\lambda \left\{\frac{1}{\tilde{q}}\left(1-\int \pi {(x,y)}^{1+\tilde{q}}dxdy\right)\right\}\hfill \\ \hfill & -\int \alpha (x)\pi (x,y)dxdy+\int \alpha (x)p(x)dx\hfill \\ \hfill & -\int \beta (y)\pi (x,y)dxdy+\int \beta (y)q(y)dy\hfill \end{array}$$

(58)

and the extremum of the Tsallis entropy-regularized optimal transport is obtained by the functional derivative with respect to $\pi$,

$$\pi (x,y)={\left(\frac{\tilde{q}}{2(\tilde{q}+1)\lambda }\left({-\alpha \left(x\right)-\beta \left(y\right)+\parallel x-y\parallel }^2\right)\right)}^{\frac{1}{\tilde{q}}}.$$

(59)

Here, $\alpha$ and $\beta$ are quadratic polynomials by Lemma 3. To separate the normalizing constant, we introduce a constant $c\in {\mathbb{R}}_{+}$, and $\pi$ can be written as:

$$\pi (x,y)=c^{\frac{1}{\tilde{q}}}{\left(\tilde{\alpha }\left(x\right)+\tilde{\beta }\left(y\right)+\frac{\tilde{q}{\parallel x-y\parallel }^2}{2c(\tilde{q}+1)\lambda }\right)}^{\frac{1}{\tilde{q}}},$$

(60)

with quadratic functions $\tilde{\alpha }\left(x\right)$ and $\tilde{\beta }\left(y\right)$.

Let $\tilde{\lambda }=\frac{c(\tilde{q}+1)\lambda }{\tilde{q}}>0$. Then, by the same argument as in the proof of Theorem 1 and using Corollary 3, we obtain the scale matrix of $\pi$ as:

$$\mathsf{\Sigma }=\left(\begin{array}{cc}{\mathsf{\Sigma }}_1& {\mathsf{\Sigma }}_{\tilde{\lambda }}\\ {\mathsf{\Sigma }}_{\tilde{\lambda }}^{\mathrm{T}}& {\mathsf{\Sigma }}_2\end{array}\right),$$

(61)

where:

$${\mathsf{\Sigma }}_{\tilde{\lambda }}={\mathsf{\Sigma }}_1^{1/2}{({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}+{\tilde{\lambda }}^{2}I)}^{1/2}{\mathsf{\Sigma }}_1^{-1/2}-\tilde{\lambda }I.$$

(62)

Let $z={(x^{\mathrm{T}},y^{\mathrm{T}})}^{\mathrm{T}}$ and $K_{\tilde{q}}=\int {(1+z^{\mathrm{T}}z)}^{\frac{1}{\tilde{q}}}dz$; $\pi$ can be written as:

$$\pi (x,y)=\frac{1}{K_{\tilde{q}}\left|\mathsf{\Sigma }\right|}{(1+z^{\mathrm{T}}{\mathsf{\Sigma }}^{-1}z)}^{\frac{1}{\tilde{q}}}.$$

(63)

The constant c is determined by:

$$\frac{1}{K_{\tilde{q}}\left|\mathsf{\Sigma }\right|}=c^{\frac{1}{\tilde{q}}}.$$

(64)

We will show that the above equation has a unique solution. Let ${\left\{\tau \right\}}_{i=1}^n$ be the eigenvalues of ${\left({\mathsf{\Sigma }}_1^{1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{1/2}\right)}^{1/2}$; $\left|\mathsf{\Sigma }\right|$ can be expressed as ${\prod }_{i=1}^{2n}2\tilde{\lambda }(\sqrt{{\tau }_i^2+{\tilde{\lambda }}^2}-\tilde{\lambda })$. We consider:

$$\begin{array}{cc}\hfill f\left(c\right)& =log(c^{\frac{1}{\tilde{q}}}{K}_{\tilde{q}}\left|\mathsf{\Sigma }\right|)\hfill \end{array}$$

(65)

$$\begin{array}{cc}\hfill & =\frac{1}{\tilde{q}}logc+\sum _{i=1}^{2n}log(\sqrt{{\tau }_i^2+{\tilde{\lambda }}^2}-\tilde{\lambda })+2nlog\left(2\tilde{\lambda }\right)+log{K}_{\tilde{q}}.\hfill \end{array}$$

(66)

Because $\tilde{q}<0$, $f\left(c\right)$ is a monotonic decreasing function, and ${lim}_{c\downarrow 0}f\left(c\right)=\infty$, ${lim}_{c\to \infty }f\left(c\right)=-\infty$, (64) has a unique positive solution, and $\tilde{\lambda }$ is determined uniquely.  □

5. Entropy-Regularized Kantorovich Estimator

Many estimators are defined by minimizing the divergence or distance $\rho$ between probability measures, that is $arg{min}_{\mu }\rho (\mu ,\nu )$ for a fixed $\nu$. When $\rho$ is the Kullback–Leibler divergence, the estimator corresponds to the maximum likelihood estimator. When $\rho$ is the Wasserstein distance, the following estimator is called the minimum Kantorovich estimator, according to [36]. In this section, we consider a probability measure $Q^{*}$ that minimizes $C_{\lambda }(P,Q)$ for a fixed P over ${\mathcal{P}}_2\left({\mathbb{R}}^n\right)$, the set of all probability measures on ${\mathbb{R}}^n$ with finite second moment that are absolutely continuous with respect to the Lebesgue measure. In other words, we define the entropy-regularized Kantorovich estimator $arg{min}_{Q\in {\mathcal{P}}_2\left({\mathbb{R}}^n\right)}{C}_{\lambda }(P,Q).$ The entropy-regularized Kantorovich estimator for discrete probability measures was studied in [33], Theorem 2. We obtain the entropy-regularized Kantorovich estimator for continuous probability measures in the following theorem:

Theorem 3.

For a fixed $P\in {\mathcal{P}}_2\left({\mathbb{R}}^n\right)$,

$$Q^{*}=arg\underset{Q\in {\mathcal{P}}_2\left({\mathbb{R}}^n\right)}{min}{C}_{\lambda }(P,Q)$$

(67)

exists, and its density function can be written as:

$$d{Q}^{*}=dP🟉{\varphi }_{\lambda },$$

(68)

where ${\varphi }_{\lambda }\left(x\right)$ is a density function of $\mathcal{N}(0,\frac{\lambda }{2}I)$, and 🟉 denotes the convolution operator.

We use the dual problem of the entropy-regularized optimal transport to prove Theorem 3 (for details, see Proposition 2.1 of [15] or Section 3 of [51]).

Lemma 5.

The dual problem of entropy-regularized optimal transport can be written as:

$$\begin{array}{cc}\hfill {\mathcal{A}}_{\lambda }(P,Q)=\underset{\begin{array}{c}\alpha \in L_1\left(P\right)\\ \beta \in L_1\left(Q\right)\end{array}}{sup}& \int \alpha (x)p(x)dx+\int \beta (y)q(y)dy\hfill \\ \hfill & -\lambda \int exp\left\{\frac{{\alpha \left(x\right)+\beta \left(y\right)-\parallel x-y\parallel }^2}{\lambda }\right\}dxdy.\hfill \end{array}$$

(69)

Moreover, ${\mathcal{A}}_{\lambda }(P,Q)=C_{\lambda }(P,Q)$ holds.

Now, we prove Theorem 3.

Proof. 

Let $Q^{*}$ be the minimizer of ${min}_Q{C}_{\lambda }(P,Q)$. Applying Lemma 5, there exist ${\alpha }^{*}\in L_1\left(P\right)$ and ${\beta }^{*}\in L_1\left(Q^{*}\right)$ such that:

$$\begin{array}{cc}\hfill C_{\lambda }(P,Q^{*})={\mathcal{A}}_{\lambda }(P,Q^{*})& =\int {\alpha }^{*}\left(x\right)p\left(x\right)dx+\int {\beta }^{*}\left(y\right)q^{*}\left(y\right)dy\hfill \\ \hfill & -\lambda \int exp\left\{\frac{{\alpha }^{*}\left(x\right)+{\beta }^{*}\left(y\right)-{\parallel x-y\parallel }^2}{\lambda }\right\}dxdy.\hfill \end{array}$$

(70)

Now, ${\mathcal{A}}_{\lambda }(P,Q^{*})$ is the minimum value of ${\mathcal{A}}_{\lambda }$, such that the variation $\delta {\mathcal{A}}_{\lambda }(P,Q^{*})$ is always zero. Then,

$$\delta {\mathcal{A}}_{\lambda }(P,Q^{*})=\int {\beta }^{*}\left(y\right)\delta q^{*}\left(y\right)dy=0\Rightarrow {\beta }^{*}\equiv 0$$

(71)

holds, and the optimal coupling of $P,Q$ can be written as:

$$\begin{array}{cc}\hfill {\pi }^{*}(x,y)& =exp\left\{\frac{{\alpha }^{*}\left(x\right)+{\beta }^{*}\left(y\right)}{\lambda }-\frac{{\parallel x-y\parallel }^2}{\lambda }\right\}\hfill \end{array}$$

(72)

$$\begin{array}{cc}\hfill & =exp\left\{\frac{{\alpha }^{*}\left(x\right)}{\lambda }\right\}exp\left\{-\frac{{\parallel x-y\parallel }^2}{\lambda }\right\}.\hfill \end{array}$$

(73)

Moreover, we can obtain a closed-form of ${\alpha }^{*}\left(x\right)$ as follows from the equation $\int \pi (x,y)dy$$=p\left(x\right)$:

$$\frac{{\alpha }^{*}\left(x\right)}{\lambda }=logp\left(x\right)-log\int exp\left\{-\frac{{\parallel x-y\parallel }^2}{\lambda }\right\}dy=logp\left(x\right)-\frac{n}{2}log\left(\pi \lambda \right).$$

(74)

Then, by calculating the marginal distribution of $\pi (x,y)$ with respect to x, we can obtain:

$$q^{*}\left(y\right)=\int \frac{1}{{\left(\pi \lambda \right)}^{\frac{n}{2}}}exp\left\{-\frac{{\parallel x-y\parallel }^2}{\lambda }\right\}p\left(x\right)dx=(p🟉{\varphi }_{\lambda })\left(y\right).$$

(75)

Therefore, we conclude that a probability measure Q that minimizes $C_{\lambda }(P,Q)$ is expressed as (75).  □

It should be noted that when P in Theorem 3 are multivariate normal distributions, $Q^{*}$ and P are simultaneously diagonalizable by a direct consequence of the theorem. This is consistent with the result of Corollary 2(1) for minimization when all eigenvalues are fixed.

We can determine that the entropy-regularized Kantorovich estimator is a measure convolved with an isotropic multivariate normal distribution scaled by the regularization parameter $\lambda$. This is similar to the idea of prior distributions in the context of Bayesian inference. Applying Theorem 3, the entropy-regularized Kantorovich estimator of the multivariate normal distribution $\mathcal{N}(\mu ,\mathsf{\Sigma })$ is $\mathcal{N}(\mu ,\mathsf{\Sigma }+\frac{\lambda }{2}I)$.

6. Numerical Experiments

In this section, we introduce experiments that show the statistical efficiency of entropy regularization in Gaussian settings. We consider two different setups, estimating covariance matrices (Section 6.1) and the entropy-regularized Wasserstein barycenter (Section 6.2). To obtain the entropy-regularized Wasserstein barycenter, we adopt the Newton–Schulz method and a manifold optimization method, which are explained in Section 6.3 and Section 6.4, respectively.

6.1. Estimation of Covariance Matrices

We provide a covariance estimation method based on entropy-regularized optimal transport. Let $P=\mathcal{N}(\mu ,\mathsf{\Sigma })$ be an n-variate normal distribution. We define an entropy-regularized Kantorovich estimator ${\widehat{P}}_{\lambda }$, that is,

$${\widehat{P}}_{\lambda }=arg\underset{Q}{min}{C}_{\lambda }(P,Q).$$

(76)

We generate some samples from $\mathcal{N}(\mu ,\mathsf{\Sigma })$ and estimate the mean and covariance matrix. We compare the maximum likelihood estimator ${\widehat{P}}_{\mathrm{MLE}}=\mathcal{N}({\widehat{\mu }}_{\mathrm{MLE}},{\widehat{\mathsf{\Sigma }}}_{\mathrm{MLE}})$ and ${\widehat{P}}_{\lambda }$ with respect to the prediction error:

$$\mathrm{KL}(P,{\widehat{P}}_{\mathrm{MLE}}),\mathrm{KL}(P,{\widehat{P}}_{\lambda }).$$

(77)

In our experiment, the dimension n is set to $5,15,30$, and the sample size is set to $60,120$. The experiment proceeds as follows.

1.

Obtain a random sample of size 60 (or 120) from $\mathcal{N}(0,\mathsf{\Sigma })$ and its sample covariance matrix $\widehat{\mathsf{\Sigma }}$.

2.

Obtain the entropy-regularized minimum Kantorovich estimator of $\widehat{\mathsf{\Sigma }}$ obtained in Step 1.

3.

Compute the prediction error between $\mathsf{\Sigma }$ and the entropy-regularized minimum Kantorovich estimator of $\widehat{\mathsf{\Sigma }}$

4.

Repeat the above steps 1000 times and obtain a confidence interval of the prediction error.

Table 1. Average prediction error of the MLE and entropy-regularized Kantorovich estimator of covariance matrices from 60 samples from an n-variate normal distribution with the 95% confidential interval.

$\mathit{\lambda }$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=5$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=15$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=30$
0(MLE)	0.062 $\pm 0.005$	1.346 $\pm 0.022$	10.69 $\pm 0.112$
0.01	0.051 $\pm 0.005$	1.242 $\pm 0.021$	8.973 $\pm 0.087$
0.1	0.104 $\pm 0.004$	0.841 $\pm 0.013$	4.180 $\pm 0.033$
0.5	0.647 $\pm 0.003$	0.931 $\pm 0.007$	3.093 $\pm 0.010$
1.0	1.166 $\pm 0.003$	1.670 $\pm 0.006$	5.075 $\pm 0.009$

shows the average prediction error of the MLE and entropy-regularized Kantorovich estimator of covariance matrices from 60 samples from an n-variate normal distribution with the 95% confidential interval. We can see that the prediction error is smaller than the maximum likelihood estimator under adequately small $\lambda$ for $n=15,30$, but not for $n=5$. Moreover, the decrease in the prediction error is larger for $n=30$ than for $n=15$, which indicates that the entropy regularization is effective in a high dimension. On the other hand,

Table 2. Average prediction error of the MLE and entropy-regularized Kantorovich estimator of covariance matrices from 120 samples from an n-variate normal distribution with the 95% confidential interval.

$\mathit{\lambda }$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=5$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=15$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=30$
0(MLE)	0.024 $\pm 0.002$	0.490 $\pm 0.007$	2.810 $\pm 0.021$
0.01	0.020 $\pm 0.002$	0.459 $\pm 0.006$	2.528 $\pm 0.018$
0.1	0.101 $\pm 0.002$	0.397 $\pm 0.005$	1.700 $\pm 0.001$
0.5	0.659 $\pm 0.002$	0.875 $\pm 0.004$	2.833 $\pm 0.005$
1.0	1.180 $\pm 0.002$	1.730 $\pm 0.004$	5.124 $\pm 0.005$

shows in all cases that the decreases in the prediction error are more moderate than Table 1. We can see that this is due to the increase in the sample size. Then, we can conclude that the entropy regularization is effective in a high-dimensional setting with a small sample size.

6.2. Estimation of the Wasserstein Barycenter

A barycenter with respect to the Wasserstein distance is definable [52] and is widely used for image interpolation and 3D object interpolation tasks with entropy regularization [3,33].

Definition 10.

Let ${\left\{Q_i\right\}}_{i=1}^m$ be a set of probability measures in $\mathcal{P}\left({\mathbb{R}}^n\right)$. The barycenter with respect to $C_{\lambda }$ (entropy-regularized Wasserstein barycenter) is defined as:

$$arg\underset{P\in \mathcal{P}\left({\mathbb{R}}^n\right)}{min}\sum _{i=1}^m{C}_{\lambda }(P,Q_i).$$

(78)

Now, we restrict P and ${\left\{Q_i\right\}}_{i=1}^m$ to be multivariate normal distributions and apply our theorem to illustrate the effect of entropy regularization.

The experiment proceeds as follows. The dimensionality and the sample size were set the same as in the experiments in Section 6.1.

1.

Obtain a random sample of size 60 (or 120) from $\mathcal{N}(0,\mathsf{\Sigma })$ and its sample covariance matrix $\widehat{\mathsf{\Sigma }}$.

2.

Repeat Step 1 three times, and obtain ${\left\{\widehat{\mathsf{\Sigma }}\right\}}_{i=1}^3$.

3.

Obtain the barycenter of ${\left\{{\widehat{\mathsf{\Sigma }}}_i\right\}}_{i=1}^3$.

4.

Compute the prediction error between $\mathsf{\Sigma }$ and the barycenter obtained in step 3.

5.

Repeat the above steps 100 times and obtain a confidence interval of the prediction error.

We show the results for several values of the regularization parameter $\lambda$ in

Table 3. Average prediction error of the entropy-regularized barycenter with the 95% confidential interval (random sample of size 60).

$\mathit{\lambda }$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=5$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=15$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=30$
0	0.455 $\pm 0.395$	1.318 $\pm 0.006$	4.875 $\pm 0.035$
0.001	0.429 $\pm 0.396$	1.318 $\pm 0.004$	4.887 $\pm 0.036$
0.01	0.434 $\pm 0.270$	1.344 $\pm 0.006$	4.551 $\pm 0.164$
0.025	0.780 $\pm 0.223$	1.456 $\pm 0.064$	5.710 $\pm 0.536$
0.005	1.047 $\pm 0.029$	1.537 $\pm 0.064$	7.570 $\pm 0.772$

and

Table 4. Average prediction error of the entropy-regularized barycenter with the 95% confidential interval (random sample of size 120).

$\mathit{\lambda }$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=5$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=15$	$\mathbf{KL}(\mathit{P},{\widehat{\mathit{P}}}_{\mathbf{W}}),\mathit{n}=30$
0	0.154 $\pm 0.600$	1.303 $\pm 0.010$	5.091 $\pm 0.035$
0.001	0.212 $\pm 0.070$	1.305 $\pm 0.010$	5.072 $\pm 0.037$
0.01	0.306 $\pm 0.046$	1.328 $\pm 0.008$	5.274 $\pm 0.252$
0.025	0.671 $\pm 0.028$	1.337 $\pm 0.073$	5.851 $\pm 0.424$
0.005	1.109 $\pm 0.063$	1.603 $\pm 0.184$	8.072 $\pm 0.725$

. A decrease in the prediction error can be seen in Table 3 for $n=30$, as well as Table 1 and Table 2. However, because the computation of the entropy-regularized Wasserstein barycenter uses more data than that of the minimum Kantorovich estimator, the decrease in the prediction error is mild. The entropy-regularized Kantorovich estimator is a special case of the entropy-regularized Wasserstein barycenter (78) for $m=1$. Our experiments show that the appropriate range of $\lambda$ to decrease the prediction error depends on m and becomes narrow as m increases. In addition, we note that there is a small decrease in the prediction error in Table 4 for $n=30$.

6.3. Gradient Descent on ${\mathrm{Sym}}_{+}\left(n\right)$

We use a gradient descent method to compute the entropy-regularized barycenter. Applying the gradient descent method to the loss function defined by the Wasserstein distance was proposed in [4]. This idea is extendable to entropy-regularized optimal transport. The detailed algorithm is shown below. Because $C_{\lambda }(P,Q)$ is a function of a positive definite matrix, we used a manifold gradient descent algorithm on the manifold of positive definite matrices.

We review the manifold gradient descent algorithm used in our numerical experiment. Let ${\mathrm{Sym}}_{+}\left(n\right)$ be the manifold of n-dimensional positive definite matrices. We require a formula for a gradient operator and the inner product of ${\mathrm{Sym}}_{+}\left(n\right)$ in the gradient descent algorithm. In this paper, we use the following inner product from [44], Chapter 6. For a fixed $X\in \mathrm{int}\left({\mathrm{Sym}}_{+}\left(n\right)\right)$, we define an inner product of ${\mathrm{Sym}}_{+}\left(n\right)$ as:

$$g_X(Y,Z)=tr\left(Y{X}^{-1}Z{X}^{-1}\right),Y,Z\in {\mathrm{Sym}}_{+}\left(n\right),$$

(79)

Equation (79) is the best choice in terms of the convergence speed according to [53]. Let $f:{\mathrm{Sym}}_{+}\left(n\right)\to \mathbb{R}$ be a differential matrix function. Then, the induced gradient of f under (79) is:

$$gradf\left(X\right)=X\left(\frac{\partial f\left(X\right)}{\partial X}\right)X.$$

(80)

We consider the updating step after obtaining the gradient of f. $\mathrm{grad}f\left(X\right)$ is an element of the tangent space, and we have to project it to ${\mathrm{Sym}}_{+}\left(n\right)$. This projection map is called a retraction. It is known that the Riemannian metric $g_X$ leads to the following retraction:

${exp}_{X}x=XExp\left(X^{-1}x\right)$, where $\mathrm{Exp}$ is the matrix exponential. Then, the corresponding gradient descent method becomes as shown in Algorithm 1.

6.4. Approximate the Matrix Square Root

To compute the gradient of the square root of a matrix in the objective function, we approximate it using the Newton–Schulz method [54], which can be implemented by matrix operations as shown in Algorithm 2. It is amenable to automatic differentiation, such that we can easily apply the gradient descent method to our algorithm.

Algorithm 1 Gradient descent on the manifold of positive definite matrices.

Input:$f\left(X\right)$    initialize  X    while no convergence do       $\eta :$ step size       $\mathrm{grad}\leftarrow X\left(\frac{\partial f\left(X\right)}{\partial X}\right)X$       $X\leftarrow {exp}_X\left(\eta \mathrm{grad}\right)=X\mathrm{Exp}\left(\eta X^{-1}\mathrm{grad}\right)$    end while Output:X

Algorithm 2: Newton–Schulz method.

Input:$A\in {\mathrm{Sym}}_{+}\left(n\right),ϵ>0$    $Y\leftarrow \frac{A}{(1+ϵ)\parallel A\parallel },Z\leftarrow I$    while no convergence do       $T\leftarrow (3I-ZY)/2$       $Y\leftarrow YT,Z\leftarrow TZ$    end while Output:$\sqrt{(1+ϵ)\parallel A\parallel }Y$

7. Conclusions and Future Work

In this paper, we studied entropy-regularized optimal transport and derived several result. We summarize these as follows and add notes on future work.

• We obtain the explicit form of entropy-regularized optimal transport between two multivariate normal distributions and derived Corollaries 1 and 2, which clarified the properties of optimal coupling. Furthermore, we demonstrate experimentally how entropy regularization affects the Wasserstein distance, the optimal coupling, and the geometric structure of multivariate normal distributions. Overall, the properties of optimal coupling were revealed both theoretically and experimentally. We expect that the explicit formula can be a replacement for the existing methodology using the (nonregularized) Wasserstein distance between normal distributions (for example, [4,5]).
• Theorem 2 derives the explicit form of the optimal coupling of the Tsallis entropy-regularized optimal transport between multivariate q-normal distributions. The optimal coupling of the Tsallis entropy-regularized optimal transport between multivariate q-normal distributions is also a multivariate q-normal distribution, and the obtained result has an analogy to that of the normal distribution. We believe that this result can be extended to other elliptical distribution families.
• The entropy-regularized Kantorovich estimator of a probability measure in ${\mathcal{P}}_2\left(\mathbb{R}\right)$ is the convolution of a multivariate normal distribution and its own density function. Our experiments show that both the entropy-regularized Kantorovich estimator and the Wasserstein barycenter of multivariate normal distributions outperform the maximum likelihood estimator in the prediction error for adequately selected $\lambda$ in a high dimensionality and small sample setting. As future work, we want to show the efficiency of entropy regularization using real data.