α-Geodesical Skew Divergence

Abstract

The asymmetric skew divergence smooths one of the distributions by mixing it, to a degree determined by the parameter $\lambda$, with the other distribution. Such divergence is an approximation of the KL divergence that does not require the target distribution to be absolutely continuous with respect to the source distribution. In this paper, an information geometric generalization of the skew divergence called the $\alpha$-geodesical skew divergence is proposed, and its properties are studied.

1. Introduction

Let $(\mathcal{X},\mathcal{F},\mu )$ be a measure space where $\mathcal{X}$ denotes the sample space, $\mathcal{F}$ the $\sigma$-algebra of measurable events, and $\mu$ a positive measure. The set of the strictly positive probability measure $\mathcal{P}$ is defined as

$$\mathcal{P}\left\{f\left(x\right)>0(\forall x\in \mathcal{X}),\mathrm{and}{\int }_{\mathcal{X}}f\left(x\right)d\mu \left(x\right)=1\right\},$$

(1)

and the set of nonnegative probability measure ${\mathcal{P}}_{+}$ is defined as

$${\mathcal{P}}_{+}\left\{f\left(x\right)\ge 0(\forall x\in \mathcal{X}),\mathrm{and}{\int }_{\mathcal{X}}f\left(x\right)d\mu \left(x\right)=1\right\}.$$

(2)

Then a number of divergences that appear in statistics and information theory [1,2] are introduced.

Definition 1

(Kullback–Leibler divergence [3]). The Kullback–Leibler divergence or KL-divergence $D_{KL}:{\mathcal{P}}_{+}\times \mathcal{P}\to [0,\infty ]$ is defined between two Radon–Nikodym densities p and q of μ-absolutely continuous probability measures by

$$D_{KL}[p\parallel q]{\int }_{\mathcal{X}}pln\frac{p}{q}d\mu .$$

(3)

KL-divergence is a measure of the difference between two probability distributions in statistics and information theory [4,5,6,7]. This is also called the relative entropy and is known not to satisfy the axiom of distance. Because the KL-divergence is asymmetric, several symmetrizations have been proposed in the literature [8,9,10].

Definition 2

(Jensen–Shannon divergence [8]). The Jensen–Shannon divergence or JS-divergence $D_{JS}:\mathcal{P}\times \mathcal{P}\to [0,\infty )$ is defined between two Radon–Nikodym densities p and q of μ-absolutely continuous probability measures by

$$\begin{array}{cc}\hfill D_{JS}[p\parallel q]& \mathrm{:}=\frac{1}{2}\left(D_{KL}\left[p\parallel \frac{p+q}{2}\right]+D_{KL}\left[q\parallel \frac{p+q}{2}\right]\right)\hfill \\ \hfill & =\frac{1}{2}{\int }_{\mathcal{X}}\left(pln\frac{2p}{p+q}+qln\frac{2q}{p+q}\right)d\mu \hfill \\ \hfill & =D_{JS}[q\parallel p].\hfill \end{array}$$

(4)

The JS-divergence is a symmetrized and smoothed version of the KL-divergence, and it is bounded as

$$0\le D_{JS}[p\parallel q]\le ln2.$$

(5)

This property contrasts with the fact that KL-divergence is unbounded.

Definition 3

(Jeffreys divergence [11]). The Jeffreys divergence $D_J[p\parallel q]:\mathcal{P}\times \mathcal{P}\to [0,\infty ]$ is defined between two Radon–Nikodym densities p and q of μ-absolutely continuous probability measures by

$$D_J[p\parallel q]\mathrm{:}=D_{KL}[p\parallel q]+D_{KL}[q\parallel p].$$

(6)

Such symmetrized KL-divergences have appeared in various pieces of literature [12,13,14,15,16,17,18].

For continuous distributions, the KL-divergence is known to have computational difficulty. To be more specific, if q takes a small value relative to p, the value of $D_{KL}[p\parallel q]$ may diverge to infinity. The simplest idea to avoid this is to use very small $ϵ>0$ and modify $D_{KL}[p\parallel q]$ as follows:

$$D_{KL}^{+}[p\parallel q]\mathrm{:}={\int }_{\mathcal{X}}pln\frac{p}{q+ϵ}d\mu .$$

However, such an extension is unnatural in the sense that $q+ϵ$ no longer satisfies the condition for a probability measure: ${\int }_{\mathcal{X}}ϵ+q\left(x\right)d\mu \left(x\right)\ne 1$. As a more natural way to stabilize KL-divergence, the following skew divergences have been proposed:

Definition 4

(Skew divergence [8,19]). The skew divergence $D_S^{\left(\lambda \right)}[p\parallel q]:\mathcal{P}\times \mathcal{P}\to [0,\infty ]$ is defined between two Radon–Nikodym densities p and q of μ-absolutely continuous probability measures by

$$\begin{array}{cc}\hfill D_S^{\left(\lambda \right)}[p\parallel q]& \mathrm{:}=D_{KL}[p\parallel (1-\lambda )p+\lambda q]\hfill \\ \hfill & ={\int }_{\mathcal{X}}pln\frac{p}{(1-\lambda )p+\lambda q}d\mu ,\hfill \end{array}$$

(7)

where $\lambda \in [0,1]$.

Skew divergences have been experimentally shown to perform better in applications such as natural language processing [20,21], image recognition [22,23] and graph analysis [24,25]. In addition, there is research on quantum generalization of skew divergence [26].

The main contributions of this paper are summarized as follows:

• Several symmetrized divergences or skew divergences are generalized from an information geometry perspective.
• It is proved that the natural skew divergence for the exponential family is equivalent to the scaled KL-divergence.
• Several properties of geometrically generalized skew divergence are proved. Specifically, the functional space associated with the proposed divergence is shown to be a Banach space.

Implementation of the proposed divergence is available on GitHub (https://github.com/nocotan/geodesical_skew_divergence (accessed on 3 April 2021)).

2. α-Geodesical Skew Divergence

The skew divergence is generalized based on the following function.

Definition 5

(f-interpolation). For any $a,b,\in \mathbb{R}$, $\lambda \in [0,1]$ and $\alpha \in \mathbb{R}$, f-interpolation is defined as

$$m_f^{(\lambda ,\alpha )}(a,b)=f_{\alpha }^{-1}\left((1-\lambda )f_{\alpha }\left(a\right)+\lambda f_{\alpha }\left(b\right)\right),$$

(8)

where

$$f_{\alpha }\left(x\right)=\left\{\begin{array}{cc}{x}^{\frac{1-\alpha }{2}}\hfill & (\alpha \ne 1)\hfill \\ lnx\hfill & (\alpha =1)\hfill \end{array}\right.$$

(9)

is the function that defines the f-mean [27].

The f-mean function satisfies

$$\begin{array}{cc}\hfill \underset{\alpha \to \infty }{lim}{f}_{\alpha }\left(x\right)& =\left\{\begin{array}{cc}\infty \hfill & \left(\right|x|<1),\hfill \\ 1\hfill & \left(\right|x|=1),\hfill \\ 0\hfill & \left(\right|x|>1),\hfill \end{array}\right.\hfill \\ \hfill \underset{\alpha \to -\infty }{lim}{f}_{\alpha }\left(x\right)& =\left\{\begin{array}{cc}0\hfill & \left(\right|x|<1),\hfill \\ 1\hfill & \left(\right|x|=1),\hfill \\ \infty \hfill & \left(\right|x|>1).\hfill \end{array}\right.\hfill \end{array}$$

It is easy to see that this family includes various known weighted means including the e-mixture and m-mixture for $\alpha =\pm 1$ in the literature of information geometry [28]:

$$\begin{array}{cc}\hfill (\alpha =1)& m_f^{(\lambda ,1)}(a,b)=exp\{(1-\lambda )lna+\lambda lnb\}\hfill \\ \hfill (\alpha =-1)& m_f^{(\lambda ,-1)}(a,b)=(1-\lambda )a+\lambda b\hfill \\ \hfill (\alpha =0)& m_f^{(\lambda ,0)}(a,b)={\left((1-\lambda )\sqrt{a}+\lambda \sqrt{b}\right)}^2\hfill \\ \hfill (\alpha =3)& m_f^{(\lambda ,3)}(a,b)=\frac{1}{(1-\lambda )\frac{1}{a}+\lambda \frac{1}{b}}\hfill \\ \hfill (\alpha =\infty )& m_f^{(\lambda ,\infty )}(a,b)=min\{a,b\}\hfill \\ \hfill (\alpha =-\infty )& m_f^{(\lambda ,-\infty )}(a,b)=max\{a,b\}\hfill \end{array}$$

The inverse function $f_{\alpha }^{-1}$ is convex when $\alpha \in [-1,1]$, and concave when $\alpha \in (-\infty ,-1]\cup (1,\infty )$. It is worth noting that the f-interpolation is a special case of the Kolmogorov–Nagumo average [29,30,31] when $\alpha$ is restricted in the interval $[-1,1]$.

In order to consider the geometric meaning of this function, the notion of the statistical manifold is introduced.

2.1. Statistical Manifold

Let

$$\mathcal{S}=\{p_{\mathit{\xi }}=p(\mathit{x};\mathit{\xi })\in \mathcal{P}|\mathit{\xi }=({\xi }^1,\dots ,{\xi }^n)\in \Xi \}$$

(10)

be a family of probability distribution on $\mathcal{X}$, where each element $p_{\mathit{\xi }}$ is parameterized by n real-valued variables $\mathit{\xi }=({\xi }^1,\dots ,{\xi }^n)\in \Xi \subset {\mathbb{R}}^n$. The set $\mathcal{S}$ is called a statistical model and is a subset of $\mathcal{P}$. We also denote $(\mathcal{S},g_{ij})$ as a statistical model equipped with the Riemannian metric $g_{ij}$. In particular, let $g_{ij}$ be the Fisher–Rao metric, which is the Riemannian metric induced from the Fisher information matrix [32].

In the rest of this paper, the abbreviations

$$\begin{array}{cc}\hfill {\partial }_i& ={\partial }_{{\xi }^i}=\frac{\partial }{\partial {\xi }^i},\hfill \\ \hfill \ell & ={\ell }_{\mathit{x}}\left(\mathit{\xi }\right)=ln{p}_{\mathit{\xi }}\left(\mathit{x}\right)\hfill \end{array}$$

are used.

Definition 6

(Christoffel symbols). Let $g_{ij}$ be a Riemannian metric, particularly the Fisher information matrix, then the Christoffel symbols are given by

$${\Gamma }_{ij,k}=\frac{1}{2}\left({\partial }_i{g}_{jk}+{\partial }_j{g}_{ik}-{\partial }_k{g}_{ij}\right),i,j,k=1,\dots ,n.$$

(11)

Definition 7

(Levi-Civita connection). Let g be a Fisher–Riemannian metric on $\mathcal{S}$ which is a 2-covariant tensor defined locally by

$$g(X_{\mathit{\xi }},Y_{\mathit{\xi }})=\sum _{i,j=1}^n{g}_{ij}\left(\mathit{\xi }\right)a^i\left(\mathit{\xi }\right)b^j\left(\mathit{\xi }\right),$$

where $X_{\mathit{\xi }}={\sum }_{i=1}^n{a}^i\left(\mathit{\xi }\right){\partial }_i{p}_{\mathit{\xi }}$ and $Y_{\mathit{\xi }}={\sum }_{i=1}^n{b}^i\left(\mathit{\xi }\right){\partial }_i{p}_{\mathit{\xi }}$ are vector fields in the 0-representation on $\mathcal{S}$. Then, its associated Levi-Civita connection ${\nabla }^{\left(0\right)}$ is defined by

$$g({\nabla }_{{\partial }_i}^{\left(0\right)}{\partial }_j,{\partial }_k)={\Gamma }_{ij,k}.$$

(12)

The fact that ${\nabla }^{\left(0\right)}$ is a metrical connection can be written locally as

$${\partial }_k{g}_{ij}={\Gamma }_{ki,j}+{\Gamma }_{kj,i}.$$

(13)

It is worth noting that the superscript $\alpha$ of ${\nabla }^{\left(\alpha \right)}$ corresponds to a parameter of the connection. Based on the above definitions, several connections parameterized by the parameter $\alpha$ are introduced. The case $\alpha =0$ corresponds to the Levi-Civita connection induced by the Fisher metric.

Definition 8

(${\nabla }^{\left(1\right)}$-connection). Let g be the Fisher-Riemannian metric on $\mathcal{S}$, which is a 2-covariant tensor. Then, the ${\nabla }^{\left(1\right)}$-connection is defined by

$$g({\nabla }_{{\partial }_i}^{\left(1\right)}{\partial }_j,{\partial }_k)={\mathbb{E}}_{\mathit{\xi }}[{\partial }_i{\partial }_j\ell {\partial }_k\ell ].$$

(14)

It can also be expressed equivalently by explicitly writing as the Christoffel coefficients

$${\Gamma }_{ij,k}^{\left(1\right)}\left(\mathit{\xi }\right)={\mathbb{E}}_{\mathit{\xi }}[{\partial }_i{\partial }_j\ell {\partial }_k\ell ].$$

(15)

Definition 9

(${\nabla }^{(-1)}$-connection). Let g be the Fisher–Riemannian metric on $\mathcal{S}$, which is a 2-covariant tensor. Then, the ${\nabla }^{(-1)}$-connection is defined by

$$g({\nabla }_{{\partial }_i}^{(-1)}{\partial }_j,{\partial }_k)={\Gamma }_{ij,k}^{(-1)}\left(\mathit{\xi }\right)={\mathbb{E}}_{\mathit{\xi }}[({\partial }_i{\partial }_j\ell +{\partial }_i\ell {\partial }_j\ell ){\partial }_k\ell ].$$

(16)

In the following, the ∇-flatness is considered with respect to the corresponding coordinates system. More details can be found in [28].

Proposition 1.

The exponential family is ${\nabla }^{\left(1\right)}$-flat.

Proposition 2.

The exponential family is ${\nabla }^{(-1)}$-flat if and only if it is ${\nabla }^{\left(0\right)}$-flat.

Proposition 3.

The mixture family is ${\nabla }^{(-1)}$-flat.

Proposition 4.

The mixture family is ${\nabla }^{\left(1\right)}$-flat if and only if it is ${\nabla }^{\left(0\right)}$-flat.

Proposition 5.

The relation between the foregoing three connections is given by

$${\nabla }^{\left(0\right)}=\frac{1}{2}\left({\nabla }^{(-1)}+{\nabla }^{\left(1\right)}\right).$$

(17)

Proof. 

It suffices to show

$${\Gamma }_{ij,k}^{\left(0\right)}=\frac{1}{2}\left({\Gamma }_{ij,k}^{(-1)}+{\Gamma }_{ij,k}^{\left(1\right)}\right).$$

From the definitions of ${\Gamma }^{(-1)}$ and ${\Gamma }^{\left(1\right)}$,

$$\begin{array}{cc}\hfill {\Gamma }_{ij,k}^{(-1)}+{\Gamma }_{ij,k}^{\left(1\right)}& ={\mathbb{E}}_{\mathit{\xi }}[({\partial }_i{\partial }_j\ell +{\partial }_i\ell {\partial }_j\ell ){\partial }_k\ell ]+{\mathbb{E}}_{\mathit{\xi }}[{\partial }_i{\partial }_j\ell {\partial }_k\ell ]\hfill \\ \hfill & ={\mathbb{E}}_{\mathit{\xi }}[(2{\partial }_i{\partial }_j\ell +{\partial }_i\ell {\partial }_j\ell ){\partial }_k\ell ]\hfill \\ \hfill & =2{\mathbb{E}}_{\mathit{\xi }}\left[({\partial }_i{\partial }_j\ell +\frac{1}{2}{\partial }_i\ell {\partial }_j\ell ){\partial }_k\ell \right]\hfill \\ \hfill & =2{\Gamma }_{ij,k}^{\left(0\right)},\hfill \end{array}$$

which proves the proposition. □

The connections ${\nabla }^{(-1)}$ and ${\nabla }^{\left(1\right)}$ are two special connections on $\mathcal{S}$ with respect to the mixture family and the exponential family, respectively. Moreover, they are related by the duality condition, and the following 1-parameter family of connections is defined.

Definition 10

(${\nabla }^{\left(\alpha \right)}$-connection). For $\alpha \in \mathbb{R}$, the ${\nabla }^{\left(\alpha \right)}$-connection on the statistical model $\mathcal{S}$ is defined as

$${\nabla }^{\left(\alpha \right)}=\frac{1+\alpha }{2}{\nabla }^{\left(1\right)}+\frac{1-\alpha }{2}{\nabla }^{(-1)}.$$

(18)

Proposition 6.

The components ${\Gamma }_{ij,k}^{\left(\alpha \right)}$ can be written as

$${\Gamma }_{ij,k}^{\left(\alpha \right)}={\mathbb{E}}_{\mathit{\xi }}\left[\left({\partial }_i{\partial }_j\ell +\frac{1-\alpha }{2}{\partial }_i\ell {\partial }_j\ell \right){\partial }_k\ell \right].$$

(19)

The $\alpha$-coordinate system associated with the ${\nabla }^{\left(\alpha \right)}$-connection is endowed with the $\alpha$-geodesic, which is a straight line on the corresponding coordinates system. Then, we introduce some relevant notions.

Definition 11

(α-divergence [33]). Let α be a real parameter. The α-divergence between two probability vectors $\mathit{p}$ and $\mathit{q}$ is defined as

$$D_{\alpha }[\mathit{p}\parallel \mathit{q}]=\frac{4}{1-{\alpha }^2}\left(1-\sum _i{p}_i^{\frac{1-\alpha }{2}}{q}_i^{\frac{1+\alpha }{2}}\right).$$

(20)

The KL-divergence, which is a special case with $\alpha =1$, induces the linear connection ${\nabla }^{\left(1\right)}$ as follows.

Proposition 7.

The diagonal part of the third mixed derivatives of the KL-divergence is the negative of the Christoffel symbol:

$$-{\partial }_{{\mathit{\xi }}^i}{\partial }_{{\mathit{\xi }}^j}{\partial }_{{\mathit{\xi }}_0^k}{D}_{KL}[p_{{\mathit{\xi }}_0}\parallel p_{\mathit{\xi }}]{|}_{\mathit{\xi }={\mathit{\xi }}_0}={\Gamma }_{ij,k}^{\left(1\right)}\left({\mathit{\xi }}_0\right).$$

(21)

Proof. 

The second derivative in the argument $\mathit{\xi }$ is given by

$${\partial }_{{\mathit{\xi }}^i}{\partial }_{{\mathit{\xi }}^j}{D}_{KL}[p_{{\mathit{\xi }}_0}\parallel p_{\mathit{\xi }}]=-{\int }_{\mathcal{X}}{p}_{{\mathit{\xi }}_0}\left(\mathit{x}\right){\partial }_{{\mathit{\xi }}^i}{\partial }_{{\mathit{\xi }}^j}{\ell }_{\mathit{x}}\left(\mathit{\xi }\right)d\mathit{x},$$

and differentiating it with respect to ${\mathit{\xi }}_0^k$ yields

$$\begin{array}{cc}\hfill -{\partial }_{{\mathit{\xi }}^i}{\partial }_{{\mathit{\xi }}^j}{\partial }_{{\mathit{\xi }}_0^k}{D}_{KL}[p_{{\mathit{\xi }}_0}\parallel p_{\mathit{\xi }}]& ={\partial }_{{\mathit{\xi }}_0^k}{\int }_{\mathcal{X}}{p}_{{\mathit{\xi }}_0}\left(\mathit{x}\right){\partial }_{{\mathit{\xi }}^i}{\partial }_{{\mathit{\xi }}^j}{\ell }_{\mathit{x}}\left(\mathit{\xi }\right)d\mathit{x}\hfill \\ \hfill & ={\int }_{\mathcal{X}}{p}_{{\mathit{\xi }}_0}\left(\mathit{x}\right){\partial }_{{\mathit{\xi }}^i}{\partial }_{{\mathit{\xi }}^j}{\ell }_{\mathit{x}}\left(\mathit{\xi }\right){\partial }_{{\mathit{\xi }}_0^k}{\ell }_{\mathit{x}}\left(\mathit{\xi }\right)d\mathit{x}.\hfill \end{array}$$

Then, considering the diagonal part, one yields

$$\begin{array}{cc}\hfill -{\partial }_{{\mathit{\xi }}^i}{\partial }_{{\mathit{\xi }}^j}{\partial }_{{\mathit{\xi }}_0^k}{D}_{KL}[p_{{\mathit{\xi }}_0}\parallel p_{\mathit{\xi }}]{|}_{\mathit{\xi }={\mathit{\xi }}_0}& ={\mathbb{E}}_{{\mathit{\xi }}_0}[{\partial }_i{\partial }_j\ell \left(\mathit{\xi }\right){\partial }_k\ell \left(\mathit{\xi }\right)]\hfill \\ \hfill & ={\Gamma }_{ij,k}^{\left(1\right)}\left({\mathit{\xi }}_0\right).\hfill \end{array}$$

□

More generally, the $\alpha$-divergence with $\alpha \in \mathbb{R}$ induces the ${\nabla }^{\left(\alpha \right)}$-connection.

Definition 12

(α-representation [34]). For some positive measure $m_i^{\frac{1-\alpha }{2}}$, the coordinate system $\mathit{\theta }=\left({\theta }^i\right)$ derived from the α-divergence is

$${\theta }^i=m_i^{\frac{1-\alpha }{2}}=f_{\alpha }\left(m_i\right)$$

(22)

and ${\theta }^i$ is called the α-representation of a positive measure $m_i^{\frac{1-\alpha }{2}}$.

Definition 13

(α-geodesic [28]). The α-geodesic connecting two probability vectors $p\left(\mathit{x}\right)$ and $q\left(\mathit{x}\right)$ is defined as

$$r_i\left(t\right)=c\left(t\right)f_{\alpha }^{-1}\left\{(1-t)f_{\alpha }\left(p\left(x_i\right)\right)+t{f}_{\alpha }\left(q\left(x_i\right)\right)\right\},t\in [0,1]$$

(23)

where $c\left(t\right)$ is determined as

$$c\left(t\right)=\frac{1}{{\sum }_{i=1}^n{r}_i\left(t\right)}.$$

(24)

It is known that the appropriate reparameterizations for the parameter t are necessary for a rigorous discussion in the space of probability measures [35,36]. However, as mentioned in the literature [35], an explicit expression for the reparametrizations ${\tau }_{p,a}$ and ${\tau }_{p,q}$ is unknown. A similar discussion has been made in the derivation of the ${\varphi }_{\beta }$-path [37], where it is mentioned that the normalizing factor is unknown in general. Furthermore, the f-mean is not convex depending on the $\alpha$. For these reasons, it is generally difficult to discuss $\alpha$-geodesics in probability measures by normalization or reparameterization, and to avoid unnecessary complexity, the parameter t is assumed to be appropriately reparameterized.

Let ${\psi }_{\alpha }\left(\mathit{\theta }\right)=\frac{1-\alpha }{2}{\sum }_{i=1}^n{m}_i$. Then, the dual coordinate system $\mathit{\eta }$ is given by $\mathit{\eta }=\nabla {\psi }_{\alpha }\left(\mathit{\theta }\right)$ as

$${\eta }_i={\left({\theta }^i\right)}^{\frac{1+\alpha }{1-\alpha }}=f_{-\alpha }\left(m_i\right).$$

(25)

Hence, it is the ($-\alpha$)-representation of $m_i$.

2.2. Generalization of Skew Divergences

From Definition 13, the f-interpoloation is considered as an unnormalized version of the $\alpha$-geodesic. Using the notion of geodesics, skew divergence is generalized in terms of information geometry as follows.

Definition 14

(α-Geodesical Skew Divergence). The α-geodesical skew divergence $D_{GS}^{(\alpha ,\lambda )}:\mathcal{P}\times \mathcal{P}\to [0,\infty ]$ is defined between two Radon–Nikodym densities p and q of μ-absolutely continuous probability measures by:

$$\begin{array}{cc}\hfill D_{GS}^{(\alpha ,\lambda )}\left[p\parallel q\right]& \mathrm{:}=D_{KL}\left[p\parallel m_f^{(\lambda ,\alpha )}(p,q)\right]\hfill \\ \hfill & ={\int }_{\mathcal{X}}pln\frac{p}{m_f^{(\lambda ,\alpha )}(p,q)}d\mu ,\hfill \end{array}$$

(26)

where $\alpha \in \mathbb{R}$ and $\lambda \in [0,1]$.

Some special cases of $\alpha$-geodesical skew divergence are listed below:

$$\begin{array}{cc}\hfill (\forall \alpha \in \mathbb{R},\lambda =1)& D_{GS}^{(\alpha ,1)}[p\parallel q]=D_{KL}[p\parallel q]\hfill \\ \hfill (\forall \alpha \in \mathbb{R},\lambda =0)& D_{GS}^{(\alpha ,0)}[p\parallel q]=D_{KL}[p\parallel p]=0\hfill \\ \hfill (\alpha =1,\forall \lambda \in [0,1\left]\right)& D_{GS}^{(1,\lambda )}[p\parallel q]=\lambda D_{KL}[p\parallel q]\left(\mathrm{scaled}\text{KL-divergence}\right)\hfill \\ \hfill (\alpha =-1,\forall \lambda \in [0,1\left]\right)& D_{GS}^{(-1,\lambda )}[p\parallel q]=D_S^{\left(\lambda \right)}[p\parallel q]\left(\mathrm{skew}\mathrm{divergence}\right)\hfill \\ \hfill (\alpha =0,\forall \lambda \in [0,1\left]\right)& D_{GS}^{(0,\lambda )}[p\parallel q]={\int }_{\mathcal{X}}pln\frac{p}{{\{(1-\lambda )\sqrt{p}+\lambda \sqrt{q}\}}^2}d\mu \hfill \\ \hfill (\alpha =3,\forall \lambda \in [0,1\left]\right)& D_{GS}^{(3,\lambda )}[p\parallel q]=D_S^{\left(\lambda \right)}[p\parallel q]+H\left(p\right)+H\left(q\right)\hfill \\ \hfill (\alpha =\infty ,\forall \lambda \in [0,1\left]\right)& D_{GS}^{(\infty ,\lambda )}[p\parallel q]={\int }_{\mathcal{X}}pln\frac{p}{min\{p,q\}}d\mu \hfill \\ \hfill (\alpha =-\infty ,\forall \lambda \in [0,1\left]\right)& D_{GS}^{(-\infty ,\lambda )}[p\parallel q]={\int }_{\mathcal{X}}pln\frac{p}{max\{p,q\}}d\mu \hfill \end{array}$$

Furthermore, $\alpha$-geodesical skew divergence is a special form of the generalized skew K-divergence [10,38], which is a family of abstract means-based divergences. In this paper, the skew K-divergence touched upon in [10] is characterized in terms of $\alpha$-geodesic on positive measures, and its geometric and functional analytic properties are investigated. When the Kolmogorov–Nagumo average (i.e., when the function $f^{-1}$ in Equation (8) is a strictly monotone convex function) the geodesic has been shown to be well-defined [37].

2.3. Symmetrization of $\alpha$-Geodesical Skew Divergence

It is easy to symmetrize the $\alpha$-geodesical skew divergence as follows.

Definition 15

(Symmetrized α-Geodesical Skew Divergence). The symmetrized α-geodesical skew divergence ${\overline{D}}_{GS}^{(\alpha ,\lambda )}:\mathcal{P}\times \mathcal{P}\to [0,\infty ]$ is defined between two Radon–Nikodym densities p and q of μ-absolutely continuous probability measures by:

$$\begin{array}{cc}\hfill {\overline{D}}_{GS}^{(\alpha ,\lambda )}[p\parallel q]& \mathrm{:}=\frac{1}{2}\left(D_{GS}^{(\alpha ,\lambda )}[p\parallel q]+D_{GS}^{(\alpha ,\lambda )}[q\parallel p]\right),\hfill \end{array}$$

(27)

where $\alpha \in \mathbb{R}$ and $\lambda \in [0,1]$.

It is seen that ${\overline{D}}_{GS}^{(\alpha ,\lambda )}[p\parallel q]$ includes several symmetrized divergences.

$$\begin{array}{cc}\hfill {\overline{D}}_{GS}^{(\alpha ,1)}[p\parallel q]& =\frac{1}{2}\left(D_{KL}[p\parallel q]+D_{KL}[q\parallel p]\right),\left(\mathrm{half}\mathrm{of}\mathrm{Jeffreys}\mathrm{divergence}\right)\hfill \\ \hfill {\overline{D}}_{GS}^{(-1,\frac{1}{2})}[p\parallel q]& =\frac{1}{2}\left(D_{KL}\left[p\parallel \frac{p+q}{2}\right]+D_{KL}\left[q\parallel \frac{p+q}{2}\right]\right),\left(\text{JS-divergence}\right)\hfill \\ \hfill {\overline{D}}_{GS}^{(-1,\lambda )}[p\parallel q]& =\frac{1}{2}\left(D_{KL}\left[p\parallel (1-\lambda )p+\lambda q\right]+D_{KL}\left[q\parallel (1-\lambda )q+\lambda p\right]\right).\hfill \end{array}$$

The last one is the $\lambda$-JS-divergence [39], which is a generalization of the JS-divergence.

3. Properties of α-Geodesical Skew Divergence

In this section, the properties of the $\alpha$-geodesical skew divergence are studied.

Proposition 8

(Non-negativity of the α-geodesical skew divergence). For $\alpha \ge -1$ and $\lambda \in [0,1]$, the α-geodesical skew divergence $D_{GS}^{(\alpha ,\lambda )}[p\parallel q]$ satisfies the following inequality:

$$D_{GS}^{(\alpha ,\lambda )}[p\parallel q]\ge 0.$$

(28)

Proof. 

When λ is fixed, the f-interpolation has the following inverse monotonicity with respect to α:

$$m_f^{(\lambda ,\alpha )}(p,q)\ge m_f^{(\lambda ,{\alpha }^{\prime })}(p,q),(\alpha \le {\alpha }^{\prime }).$$

(29)

From Gibbs’ inequality [40] and Equation (29), one obtains

$$\begin{array}{cc}\hfill D_{GS}^{(\alpha ,\lambda )}[p\parallel q]& ={\int }_{\mathcal{X}}pln\frac{p}{m_f^{(\alpha ,\lambda )}(p,q)}d\mu \hfill \\ \hfill & \ge \left({\int }_{\mathcal{X}}pd\mu \right)ln\frac{p}{m_f^{(\alpha ,\lambda )}(p,q)}\hfill \\ \hfill & \ge 1·ln1=0.\hfill \end{array}$$

□

Proposition 9

(Asymmetry of the α-geodesical skew divergence). α-Geodesical skew divergence is not symmetric in general:

$$D_{GS}^{(\alpha ,\lambda )}\left[p\right|q]\ne D_{GS}^{(\alpha ,\lambda )}[q\parallel p].$$

(30)

Proof. 

For example, if $\lambda =1$, then $\forall \alpha \in \mathbb{R}$, it holds that

$$\begin{array}{cc}\hfill D_{GS}^{(\alpha ,1)}[p\parallel q]-D_{GS}^{(\alpha ,1)}[q\parallel p]& =D_{KL}[p\parallel q]-D_{KL}[q\parallel p],\hfill \end{array}$$

and the asymmetry of the KL-divergence results in an asymmetry of the geodesic skew divergence. □

When a function $f\left(x\right)$ of $x\in [0,1]$ satisfies $f\left(x\right)=f(1-x)$, it is referred to as centrosymmetric.

Proposition 10

(Non-centrosymmetricity of the α-geodesical skew divergence with respect to λ). α-Geodesical skew divergence is not centrosymmetric in general with respect to the parameter $\lambda \in [0,1]$:

$$D_{GS}^{(\alpha ,\lambda )}[p\parallel q]\ne D_{GS}^{(\alpha ,1-\lambda )}[p\parallel q].$$

(31)

Proof. 

For example, if $\lambda =1$, then $\forall \alpha \in \mathbb{R}$, we have

$$\begin{array}{cc}\hfill D_{GS}^{(\alpha ,\lambda )}[p\parallel q]-D_{GS}^{(\alpha ,1-\lambda )}[p\parallel q]& =D_{GS}^{(\alpha ,1)}[p\parallel q]-D_{GS}^{(\alpha ,0)}[p\parallel q]\hfill \\ \hfill & ={\int }_{\mathcal{X}}pln\frac{p}{q}-{\int }_{\mathcal{X}}pln\frac{p}{p}\hfill \\ \hfill & ={\int }_{\mathcal{X}}pln\frac{p}{q}\ge 0.\hfill \end{array}$$

(32)

□

Proposition 11

(Monotonicity of the α-geodesical skew divergence with respect to α). α-Geodesical skew divergence satisfies the following inequality for all $\alpha \in \mathbb{R},\lambda \in [0,1]$.

$$D_{GS}^{(\alpha ,\lambda )}[p\parallel q]\ge D_{GS}^{({\alpha }^{\prime },\lambda )}[p\parallel q],(\alpha \ge {\alpha }^{\prime }).$$

Proof. 

Obvious from the inverse monotonicity of the f-interpolation (29) and the monotonicity of the logarithmic function. □

Figure 1 shows the monotonicity of the geodesic skew divergence with respect to $\alpha$. In this figure, divergence is calculated between two binomial distributions.

Proposition 12

(Subadditivity of the α-geodesical skew divergence with respect to α). α-Geodesical skew divergence satisfies the following inequality for all $\alpha ,\beta \in \mathbb{R},\lambda \in [0,1]$

$$D_{GS}^{(\alpha +\beta ,\lambda )}[p\parallel q]\le D_{GS}^{(\alpha ,\lambda )}[p\parallel q]+D_{GS}^{(\beta ,\lambda )}[p\parallel q].$$

Proof. 

For some $\alpha$ and $\lambda$, $m_f^{(\lambda ,\alpha )}$ takes the form of the Kolmogorov mean [29], which is obvious from its continuity, monotonicity and self-distributivity. □

Proposition 13

(Continuity of the α-geodesical skew divergence with respect to α and λ). α-Geodesical skew divergence has the continuity property.

Proof. 

We can prove from the continuity of the KL-divergence and the Kolmogorov mean. □

Figure 2 shows the continuity of the geodesic skew divergence with respect to $\alpha$ and $\lambda$. Both source and target distributions are binomial distributions. From this figure, it can be seen that the divergence changes smoothly as the parameters change.

Lemma 1.

Suppose $\alpha \to \infty$. Then,

$$\underset{\alpha \to \infty }{lim}{D}_{GS}^{(\alpha ,\lambda )}[p\parallel q]={\int }_{\mathcal{X}}pln\frac{p}{min\{p,q\}}d\mu$$

(33)

holds for all $\lambda \in [0,1]$.

Proof. 

Let $u=\frac{1-\alpha }{2}$. Then ${lim}_{\alpha \to \infty }u=-\infty$. Assuming $p_0\le p_1$, it holds that

$$\begin{array}{cc}\hfill \underset{\alpha \to \infty }{lim}{m}_f^{(\lambda ,\alpha )}(p_0,p_1)& =\underset{u\to -\infty }{lim}{\left((1-\lambda )p_0^u+\lambda p_1^u\right)}^{\frac{1}{u}}\hfill \\ \hfill & =p_0\underset{u\to -\infty }{lim}{\left((1-\lambda )+\lambda {\left(\frac{p_1}{p_0}\right)}^u\right)}^{\frac{1}{u}}\hfill \\ \hfill & =p_0=min\{p_0,p_1\}.\hfill \end{array}$$

Then, the following equality

$$\begin{array}{cc}\hfill \underset{\alpha \to \infty }{lim}{D}_{GS}^{(\alpha ,\lambda )}[p\parallel q]& ={\int }_{\mathcal{X}}pln\frac{p}{{lim}_{\alpha \to \infty }{m}_f^{(\lambda ,\alpha )}(p_0,p_1)}d\mu \hfill \\ \hfill & ={\int }_{\mathcal{X}}pln\frac{p}{min\{p,q\}}d\mu \hfill \end{array}$$

holds. □

Lemma 2.

Suppose $\alpha \to -\infty$. Then,

$$\underset{\alpha \to \infty }{lim}{D}_{GS}^{(\alpha ,\lambda )}[p\parallel q]={\int }_{\mathcal{X}}pln\frac{p}{max\{p,q\}}d\mu$$

(34)

holds for all $\lambda \in [0,1]$.

Proof. 

Let $u=\frac{1-\alpha }{2}$. Then ${lim}_{\alpha \to -\infty }u=\infty$. Assuming $p_0\le p_1$, it holds that

$$\begin{array}{cc}\hfill \underset{\alpha \to \infty }{lim}{m}_f^{(\lambda ,\alpha )}(p_0,p_1)& =\underset{u\to -\infty }{lim}{\left((1-\lambda )p_0^u+\lambda p_1^u\right)}^{\frac{1}{u}}\hfill \\ \hfill & =p_1\underset{u\to -\infty }{lim}{\left((1-\lambda ){\left(\frac{p_0}{p_1}\right)}^u+\lambda \right)}^{\frac{1}{u}}\hfill \\ \hfill & =p_1=max\{p_0,p_1\}.\hfill \end{array}$$

Then, the following equality

$$\begin{array}{cc}\hfill \underset{\alpha \to -\infty }{lim}{D}_{GS}^{(\alpha ,\lambda )}[p\parallel q]& ={\int }_{\mathcal{X}}pln\frac{p}{{lim}_{\alpha \to -\infty }{m}_f^{(\lambda ,\alpha )}(p_0,p_1)}d\mu \hfill \\ \hfill & ={\int }_{\mathcal{X}}pln\frac{p}{max\{p,q\}}d\mu \hfill \end{array}$$

holds. □

Proposition 14

(Lower bound of the α-geodesical skew divergence). α-Geodesical skew divergence satisfies the following inequality for all $\alpha \in \mathbb{R},\lambda \in [0,1]$.

$$D_{GS}^{(\alpha ,\lambda )}[p\parallel q]\ge {\int }_{\mathcal{X}}pln\frac{p}{max\{p,q\}}d\mu .$$

(35)

Proof. 

It follows from the definition of the inverse monotonicity of f-interpolation (29) and Lemma 2. □

Proposition 15

(Upper bound of the α-geodesical skew divergence). α-Geodesical skew divergence satisfies the following inequality for all $\alpha \in \mathbb{R},\lambda \in [0,1]$.

$$D_{GS}^{(\alpha ,\lambda )}[p\parallel q]\le {\int }_{\mathcal{X}}pln\frac{p}{min\{p,q\}}d\mu .$$

(36)

Proof. 

It follows from the definition of the f-interpolation (29) and Lemma 1. □

Theorem 1

(Strong convexity of the α-geodesical skew divergence). α-Geodesical skew divergence $D_{GS}^{(\alpha ,\lambda )}[p\parallel q]$ is strongly convex in p with respect to the total variation norm.

Proof. 

Let $r:=m_f^{(\alpha ,\lambda )}(p,q)$ and $f_j:=\frac{p_j}{r}(j=0,1)$, so that $f_t=\frac{p_t}{r}(t\in (0,1))$. From Taylor’s theorem, for $g\left(x\right):=xlnx$ and $j=0,1$, it holds that

$$\begin{array}{c}\hfill g\left(f_j\right)=g\left(f_t\right)+g^{\prime }\left(f_t\right)(f_j-f_t)+{(f_j-f_t)}^2{\int }_0^1{g}^{″}((1-s)f_t+s{f}_j)(1-s)ds.\end{array}$$

Let

$$\begin{array}{cc}\hfill \delta & :=(1-t)g\left(f_0\right)+tg\left(f_1\right)-g\left(f_t\right)\hfill \\ \hfill & =(1-t)t{(f_1-f_0)}^2{\int }_0^1\left(\frac{t}{(1-s)f_t+s{f}_0}+\frac{1-t}{(1-s)f_t+s{f}_1}\right)(1-s)ds\hfill \\ \hfill & =(1-t)t{(f_1-f_0)}^2{\int }_0^1\left(\frac{t}{f_{u_0}(t,s)}+\frac{1-t}{f_{u_1}(t,s)}\right)(1-s)ds,\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill u_j(t,s):=& (1-s)t+jt,\hfill \\ \hfill f_{{\mu }_j}(t,s):=& (1-s)f_t+s{f}_j.\hfill \end{array}$$

Then,

$$\begin{array}{cc}\hfill \Delta & :=(1-t)H\left(p_0\right)+tH\left(p_1\right)-H\left(p_t\right)\hfill \\ \hfill & =\int \delta dr\hfill \\ \hfill & =(1-t)t{\int }_0^1(1-s)ds\left[tI\left(u_0(t,s)\right)+(1-t)I\left(u_1(t,s)\right)\right],\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill \parallel p_1-p_0\parallel :=& \int |d{p}_1-d{p}_0|d\mu ,\hfill \\ \hfill H\left(p\right):=& D_{GS}^{(\alpha ,\lambda )}[p\parallel r]=\int pln\frac{p}{r}d\mu ,\hfill \\ \hfill I\left(u\right):=& \int \frac{{(f_1-f_0)}^2}{f_u}dr.\hfill \end{array}$$

Now, it is suffice to prove that $\Delta \ge \frac{t(1-t)}{2}{\parallel p_1-p_0\parallel }^2$. For all $u\in (0,1)$, it is seen that $p_1$ is absolutely continuous with respect to $p_u$. Let $g_u:=\frac{p_1}{p_u}=\frac{f_1}{f_u}$. One obtains

$$\begin{array}{cc}\hfill I\left(u\right)& =\frac{1}{{(1-u)}^2}\int \frac{{(f_1-f_u)}^2}{f_u}dr\hfill \\ \hfill & =\frac{1}{{(1-u)}^2}\int {(g_u-1)}^{2}d{p}_u\hfill \\ \hfill & \ge \frac{1}{{(1-u)}^2}{\left(\int |g_u-1|d{p}_u\right)}^2\hfill \\ \hfill & =\frac{1}{{(1-u)}^2}\parallel p_1-p_u{\parallel }^2={\parallel p_1-p_0\parallel }^2,\hfill \end{array}$$

and hence, for $j=0,1$,

$$\Delta \ge \frac{t(1-t)}{2}{\parallel p_1-p_0\parallel }^2.$$

□

4. Natural α-Geodesical Skew Divergence for Exponential Family

In this section, the exponential family is considered in which the probability density function is given by

$$p(\mathit{x};\mathit{\theta })=exp\left\{\mathit{\theta }·\mathit{x}+k\left(\mathit{x}\right)-\psi \left(\mathit{\theta }\right)\right\},$$

(37)

where $\mathit{x}$ is a random variable. In the above equation, $\mathit{\theta }=({\theta }^1,\cdots ,{\theta }^n)$ is an n-dimensional vector parameter to specify distribution, $k\left(\mathit{x}\right)$ is a function of $\mathit{x}$ and $\psi$ corresponds to the normalization factor.

In skew divergence, the probability distribution of the target is a weighted average of the two distributions. This implicitly assumes that interpolation of the two probability distributions is properly given by linear interpolation. Here, in the exponential family, the interpolation between natural parameters rather than interpolation between probability distributions themselves is considered. Namely, the geodesic connecting two distributions $p(\mathit{x};{\mathit{\theta }}_p)$ and $q(\mathit{x};{\mathit{\theta }}_q)$ on the $\mathit{\theta }$-coordinate system is considered:

$$\mathit{\theta }\left(\lambda \right)=(1-\lambda ){\mathit{\theta }}_p+\lambda {\mathit{\theta }}_q,$$

(38)

where $\lambda \in [0,1]$ is the parameter. The probability distributions on the geodesic $\mathit{\theta }\left(\lambda \right)$ are

$$\begin{array}{cc}\hfill p(\mathit{x};\lambda )& =p(\mathit{x};\mathit{\theta }(\lambda \left)\right)\hfill \\ \hfill & =exp\left\{\lambda ({\mathit{\theta }}_q-{\mathit{\theta }}_p)·\mathit{x}+{\mathit{\theta }}_p·\mathit{x}-\psi \left(\lambda \right)\right\}.\hfill \end{array}$$

(39)

Hence, a geodesic itself is a one-dimensional exponential family, where $\lambda$ is the natural parameter. A geodesic consists of a linear interpolation of the two distributions in the logarithmic scale because

$$lnp(\mathit{x};\lambda )=(1-\lambda )lnp(\mathit{x};{\mathit{\theta }}_p)+\lambda lnp(\mathit{x};{\mathit{\theta }}_q)-\psi \left(\lambda \right).$$

(40)

This corresponds to the case $\alpha =1$ on the f-interpolation with normalization factor $c\left(\lambda \right)=exp\{-\psi (\lambda \left)\right\}$,

$$p(\mathit{x};\mathit{\theta }\left(\lambda \right))=m_f^{(\lambda ,1)}(p(\mathit{x};{\mathit{\theta }}_p),p(\mathit{x};{\mathit{\theta }}_q)).$$

(41)

This induces the natural geodesic skew divergence with $\alpha =1$ as

$$\begin{array}{cc}\hfill D_{GS}^{(1,\lambda )}[p\parallel q]& ={\int }_{\mathcal{X}}pln\left(\frac{p}{m_f^{(\lambda ,1)}(p,q)}\right)d\mu \hfill \\ \hfill & ={\int }_{\mathcal{X}}plnp-pln\left(m_f^{(\lambda ,1)}(p,q)\right)d\mu \hfill \\ \hfill & ={\int }_{\mathcal{X}}plnp-pln\left(exp\left\{\right(1-\lambda )lnp+\lambda lnq\}\right)d\mu \hfill \\ \hfill & ={\int }_{\mathcal{X}}\left(plnp-(1-\lambda )plnp-\lambda plnq\right)d\mu \hfill \\ \hfill & ={\int }_{\mathcal{X}}\left(\lambda plnp-\lambda plnq\right)d\mu \hfill \\ \hfill & =\lambda {\int }_{\mathcal{X}}pln\frac{p}{q}d\mu \hfill \\ \hfill & =\lambda D_{KL}[p\parallel q],\hfill \end{array}$$

and this is equal to the scaled KL divergence.

More generally, let ${\theta }_P^{\left(\alpha \right)}$ and ${\theta }_Q^{\left(\alpha \right)}$ be the parameter representations on the $\alpha$-coordinate system of probability distributions P and Q. Then, the geodesics between them are represented as in Figure 3, and it induces the $\alpha$-geodesical skew divergence.

5. Function Space Associated with the α-Geodesical Skew Divergence

To discuss the functional nature of the $\alpha$-geodesical skew divergence in more depth, the function space it constitutes is considered. For an $\alpha$-geodesical skew divergence $f_q^{(\alpha ,\lambda )}\left(p\right)=D_{GS}^{(\alpha ,\lambda )}[p\parallel q]$ with one side of the distribution fixed, let the entire set be

$${\mathcal{F}}_q=\left\{f_q^{(\alpha ,\lambda )}\mid \alpha \in \mathbb{R},\lambda \in [0,1]\right\}.$$

(42)

For $f_q^{(\alpha ,\lambda )}\in {\mathcal{F}}_q$, its semi-norm is defined by

$$\parallel f_q^{(\alpha ,\lambda )}{\parallel }_p:={\int }_{\mathcal{X}}{\left(|f_q^{(\alpha ,\lambda )}{|}^{p}d\mu \right)}^{\frac{1}{p}}.$$

(43)

By defining addition and scalar multiplication for $f_q^{(\alpha ,\lambda )},g_q^{(\alpha ,\lambda )}\in {\mathcal{F}}_q$, $c\in \mathbb{R}$ as follows, ${\mathcal{F}}_q$ becomes a semi-norm vector space:

$$\begin{array}{cc}\hfill (f_q^{(\alpha ,\lambda )}+g_q^{(\alpha ,\lambda )})\left(u\right):=& f_q^{(\alpha ,\lambda )}\left(u\right)+g_q^{(\alpha ,\lambda )}\left(u\right)=D_{GS}^{(\alpha ,\lambda )}[u\parallel q]+D_{GS}^{({\alpha }^{\prime },{\lambda }^{\prime })}[u\parallel q],\hfill \end{array}$$

(44)

$$\begin{array}{cc}\hfill \left(cf\right)\left(u\right):=& c{f}_q^{(\alpha ,\lambda )}\left(u\right)=c·D_{GS}^{(\alpha ,\lambda )}[u\parallel q].\hfill \end{array}$$

(45)

Theorem 2.

Let $\mathcal{N}$ be the kernel of ${\parallel ·\parallel }_p$ as follows:

$${\mathcal{N}ker(\parallel ·\parallel }_p)=\left\{f_q^{(\alpha ,\lambda )}\mid f_q^{(\alpha ,\lambda )}=0\right\}.$$

(46)

Then the quotient space $\mathcal{V}({\mathcal{F}}_q,\parallel ·{\parallel }_p)/N$ is a Banach space.

Proof. 

It is sufficient to prove that $f_q^{(\alpha ,\lambda )}$ is integrable to the power of p and that $\mathcal{V}$ is complete. From Proposition 15, the $\alpha$-geodesical skew divergence is bounded from above for all $\alpha \in \mathbb{R}$ and $\lambda \in [0,1]$. Since $f_q^{(\alpha ,\lambda )}$ is continuous, we know that it is p-power integrable.

Let $\left\{f_n\right\}$ be a Cauchy sequence of $\mathcal{V}$:

$$\underset{n,m\to \infty }{lim}{\parallel f_n-f_m\parallel }_p=0.$$

As $n\left(k\right),k=1,2,\cdots ,$ can be taken to be monotonically increasing and

$$\parallel f_n-f_{n\left(k\right)}{\parallel }_p<2^{-k}$$

with respect to $n>n\left(k\right)$, let

$$\parallel f_{n(k+1)}-f_{n\left(k\right)}{\parallel }_p<2^{-k}.$$

If $g_n=|f_{n\left(1\right)}|+{\sum }_{j=1}^{n-1}|f_{n(j+1)}-f_{n\left(j\right)}|\in \mathcal{V}$, it is non-negatively monotonically increasing at each point, and from the subadditivity of the norm, $\parallel g_n{\parallel }_p\le {\parallel f_{n\left(1\right)}\parallel }_p+{\sum }_{j=1}^{n-1}{2}^{-j}$. From the monotonic convergence theorem, we have

$$\parallel \underset{n\to \infty }{lim}{g}_n{\parallel }_p=\underset{n\to \infty }{lim}\parallel g_n{\parallel }_p\le {\parallel f_{n\left(1\right)}\parallel }_p+1<\infty .$$

That is, ${lim}_{n\to \infty }{g}_n$ exists almost everywhere, and ${lim}_{n\to \infty }{g}_n\in \mathcal{V}$. From ${lim}_{n\to \infty }{g}_n<\infty$, we have

$$f_{n\left(1\right)}+\sum _{j=1}^{n-1}(f_{n(j+1)}-f_{n\left(j\right)})=\underset{n\to \infty }{lim}{f}_{n\left(1\right)}$$

converges absolutely almost everywhere to $|{lim}_{n\to \infty }{f}_{n\left(n\right)}|\le {lim}_{n\to \infty }{g}_n,a.e.$ That is, ${lim}_{n\to \infty }{f}_{n\left(n\right)}\in \mathcal{V}$. Then

$$|\underset{n\to \infty }{lim}{f}_n-f_{n\left(n\right)}|\le \underset{n\to \infty }{lim}{g}_n$$

and from the superior convergence theorem, we can obtain

$$\underset{n\to \infty }{lim}\parallel \underset{n\to \infty }{lim}{f}_n-f_{n\left(n\right)}{\parallel }_p=0$$

We have now confirmed the completeness of $\mathcal{V}$. □

Corollary 1.

Let

$${\mathcal{F}}_{+}=\left\{f_q^{(\alpha ,\lambda )}\mid \alpha \in \mathbb{R},\lambda \in (0,1],q\in \mathcal{P}\right\}.$$

(47)

Then the space ${\mathcal{V}}_{+}:=({\mathcal{F}}_{+}{,\parallel ·\parallel }_p)$ is a Banach space.

Proof. 

If we restrict $\lambda \in (0,1]$, $D_{GS}^{(\alpha ,\lambda )}[u\parallel q]=0$ if and only if $u=q$. Then, ${\mathcal{V}}_{+}$ has the unique identity element, and then ${\mathcal{V}}_{+}$ is a complete norm space. □

Consider the second argument Q of $D_{GS}^{(\alpha ,\lambda )}\left(P\right|\left|Q\right)$ is fixed, which is referred to as the reference distribution. Figure 4 shows values of the $\alpha$-geodesical skew divergence for a fixed reference Q, where both P and Q are restricted to be Gaussian. In this figure, the reference distribution is $\mathcal{N}(0,0.5)$ and the parameters of input distributions are varied in $\mu \in [0,4.5]$ and ${\sigma }^2\in [0.5,2.3]$. From this figure, one can see that a larger value of $\alpha$ emphasizes the discrepancy between distributions P and Q. Figure 5 illustrates a coordinate system associated with the $\alpha$-geodesical skew divergence for different $\alpha$. As seen from the figure, for the same pair of distributions P and Q, the value of divergence with $\alpha =3$ is larger than that with $\alpha =-1$.

6. Conclusions and Discussion

In this paper, a new family of divergence is proposed to address the computational difficulty of KL-divergence. The proposed $\alpha$-geodesical skew divergence is a natural derivation from the concept of $\alpha$-geodesics in information geometry and generalizes many existing divergences.

Furthermore, $\alpha$-geodesical skew divergence leads to several applications. For example, the new divergence can be applied to the annealed importance sampling by the same analogy as in previous studies using q-paths [41]. It could also be applied to linguistics, a field in which skew divergence was originally used [19].