Local Intrinsic Dimensionality, Entropy and Statistical Divergences

Abstract

Properties of data distributions can be assessed at both global and local scales. At a highly localized scale, a fundamental measure is the local intrinsic dimensionality (LID), which assesses growth rates of the cumulative distribution function within a restricted neighborhood and characterizes properties of the geometry of a local neighborhood. In this paper, we explore the connection of LID to other well known measures for complexity assessment and comparison, namely, entropy and statistical distances or divergences. In an asymptotic context, we develop analytical new expressions for these quantities in terms of LID. This reveals the fundamental nature of LID as a building block for characterizing and comparing data distributions, opening the door to new methods for distributional analysis at a local scale.

1. Introduction

Fundamental activities for analyzing data include both an ability to characterize data complexity and an ability to make comparisons between distributions. Widely used measures for these activities include entropy (for assessing uncertainty) and statistical divergences or distances (to compare distributions) [1]. Such analysis can be performed at either a global scale across the entire data distribution or at a local scale, in the vicinity of a given location in the distribution.

An important measure of global complexity is intrinsic dimensionality, which captures the effective number of degrees of freedom needed to describe the entire dataset. On the other hand, local intrinsic dimensionality (LID) [2] is capable of characterizing the complexity of the data distribution around a specified query location, thus capturing the number of degrees of freedom present at a local scale. LID is a unitless quantity that can also be interpreted as a relative growth rate of probability measure within an expanding neighborhood around the specified query location, or the intrinsic dimension of the space immediately around the query point.

Our focus in this paper is to characterize entropy and statistical divergences at a highly local scale, for an asymptotically small vicinity around a specified location. We show that it is possible to leverage properties that arise from LID based characterizations of lower tail distributions [3], to develop analytical expressions for a wide selection of entropy variants and statistical divergences, in both univariate and multivariate settings. This yields expressions for tail entropies and tail divergences.

Analytical characterizations for tail divergences and tail entropies are appealing from a number of perspectives. These are as follows:

• For univariate scenarios, if working with the tail of a distribution that has a single variable, we can conduct:

  –

  Temporal analysis: when a distribution models some property varying over time (e.g., survival analysis), we can analyze the entropy of a univariate distribution within an asymptotically short window of time, or the divergence between two univariate distributions within an asymptotically short window of time.

  –

  Distance-based analysis: when a distribution models distances from a query location to its nearest neighbors and the distances are induced by a global data distribution. Here, our results can be used for analysis of tail entropy or divergence between distributions within an asymptotically small distance interval. In the case of the latter, this can provide insight into multivariate properties, since under minimal assumptions the divergences between univariate distance distributions provide lower bounds for distances between multivariate distributions [4,5]. This is applicable for models such as generative adversarial networks (GANs), where it is important to test correspondence between synthetic and true distributions at a local level [6].

• For multivariate scenarios where we are analyzing distributions with multiple variables:

  –

  If an assumption of locally spherical symmetry of the distribution holds, then we can directly compute the tail entropy of a distribution or the divergence between two tail distributions in the vicinity of a single point. Such an assumption is suitable for analyzing data distributions for many types of physical systems such as fluids, glasses, metals and polymers, where local isotropy holds.

A key challenge in developing analytical characterizations for tail entropies and tail divergences is how to avoid or minimize assumptions about the form of the local distribution in the vicinity of the query (for example, assumptions such as a local normal distribution or a local uniform distribution). As we will see, analytical results are in fact possible—as the neighborhood radius asymptotically tends to zero, the tail distribution (a truncated distribution induced from the global distribution) is guaranteed to converge to a generalized pareto distribution (GPD), with the GPD parameter determined by the LID value of the tail distribution. The technical challenge is to rigorously delineate under what circumstances it is possible to leverage this relationship to achieve a dramatic simplification of the integrals that are required to compute varieties of tail entropy or distribution divergences. Our results in this paper show that such simplifications are in fact possible, for a wide range of tail entropies and divergences. This allows us to characterize and analyze fundamental properties of local neighborhood geometry, with results holding asymptotically for essentially all smooth data distributions.

In summary, our key contributions are the development of substantial new theory that asymptotically relates tail entropy, divergences and LID. It builds on and extends an earlier work by Bailey et al. [3], which focused solely on univariate entropies, without reference to divergences or multivariate settings. Specifically in this paper, we:

• Formulate technical lemmas which delineate when it is possible to substitute certain types of tail distributions by simple formulations that depend only on their associated LID values.
• Use these lemmas to compute univariate tail formulations of entropy, cross entropy, cumulative entropy, entropy power and generalized q-entropies, all in terms of the LID values of the original tail distributions.
• Use these lemmas to compute tail formulations of univariate statistical divergences and distances (Kullback–Leibler divergence, Jensen–Shannon divergence, Hellinger distance, ${\chi }^2$ divergence, $\alpha$-divergence, Wasserstein distance and $L_2$ distance).
• Extend the univariate results to a multivariate context, when local spherical symmetry of the distribution holds.

2. Related Work

The core of our study involves intrinsic dimensionality (ID) and we begin by reviewing previous work on this topic.

There is a long history of work on ID, and this can be assessed either globally (for every data point) or locally (with respect to a chosen query point). Surveys of the field provide a good overview [7,8,9]. In the global case, a range of previous works have focused on topological models and appropriate estimation methods [10,11,12]. Such examples encompass techniques such as PCA and its variants [13], graph based methods [14] and fractal models [7,15]. Other approaches such as IDEA [16,17], DANCo [18] or 2-NN estimate the (global) intrinsic dimension based on concentration of norms and angles, or 2-nearest neighbors [19].

Local intrinsic dimensionality focuses on the intrinsic dimension of a particular query point and has been used in a range of applications. These include modeling deformation in granular materials [20,21], climate science [22,23], dimension reduction via local PCA [24], similarity search [25], clustering [26], outlier detection [27], statistical manifold learning [28], adversarial example detection [29], adversarial nearest neighbor characterization [30,31] and deep learning understanding [32,33]. In deep learning, it has been shown that adversarial examples are associated with high LID estimates, a characteristic that can be leveraged to build accurate adversarial example detectors [29]. It has also been found that the LID of deep representations [33] learned by Deep Neural Networks (DNNs) or the raw input data [34,35] is correlated with the generalization performance of DNNs. A ‘dimensionality expansion’ phenomenon has been observed when DNNs overfit to noisy class labels [32] and this can be leveraged to develop improved loss functions. The use of a “cross-LID” measure to evaluate the quality of synthetic examples generated by GANs has been proposed in [36]. Connections between local intrinsic dimensionality and global intrinsic dimensionality were explored by Romano et al in [37]. In the area of climate science and dynamical systems, a formulation similar to local intrinsic dimensionality has been developed and referred to as local dimension or instantaneous dimension [22,23,38], using links to extreme value theoretic methods. It has proved useful as measure to characterize predictability of states and explain system dynamics.

For local intrinsic dimensionality, a popular estimator is the maximum likelihood estimator, studied in the Euclidean setting by Levina and Bickel [39] and later formulated under the more general assumptions of extreme value theory by Houle [2] and Amsaleg et al. [40], who showed it to be equivalent to the classic Hill estimator [41]. Other local estimators include expected simplex skewness [42], the tight locality estimator [43], the MiND framework [17], manifold adaptive dimension [44], statistical distance [45] and angle-based approaches [46]. Smoothing approaches for estimation have also been used with success [47,48].

Local intrinsic dimensionality is closely related to (univariate) distance distributions. Fundamental relations for interpoint distances, connecting multivariate distributions and univariate distributions have been explored by both [4,5]. The former showed that two multivariate distributions are equal whenever the interpoint distances both within and between samples have the same univariate distribution, while the latter showed that two multivariate distributions F and G are different if their univariate distance distributions from some randomly chosen point z are different. This can form the basis of a two sample test for comparing F and G. These studies have implications for our work in this paper, since they characterize the role that comparison between univariate distributions can play as a necessary condition for comparing equality of multivariate distributions.

Our work in this paper formulates results for different varieties of entropy and different types of divergences. Entropy is a fundamental notion used across many scientific disciplines. A good overview of its role in information theory is presented in [49]. Entropy power (the exponential of entropy) is commonly used in signal processing and information theory, and is a building block for the well-known Shannon entropy power inequality which can be used to analyze the convolution of two independent random variables [50]. Entropy power goes under the name of perplexity in the field of natural language processing [51] and true diversity in the field of ecology [52]. It also corresponds to the volume of the smallest set that contains most of the probability measure [49], and it can be interpreted as a measure of statistical dispersion [53]. It is also related to Fisher information via Stam’s inequality [54].

Cumulative entropy was formulated in [55] and is a modification of cumulative residual entropy [56]. It is popular in reliability theory where it is used to characterize uncertainty over time intervals. Apart from reliability theory analysis, it has been used in data mining tasks such as dependency analysis [57] and subspace cluster analysis [58], where it has proved more effective due to good estimation properties. These data mining investigations have used cumulative entropy at a global level (over the entire data domain), rather than at the local (tail) level, as in our study. Generalized variants based on Tsallis q-statistics have been developed for both entropy [59] and cumulative entropy [60]. Inclusion of the extra q parameter can assist with higher robustness to anomalies and better fitting to characteristics of data distributions. Tail entropy has been used in financial applications for measuring the expected shortfall [61] in the upper tail using quantization. This is different from our context, where the our exclusive focus is on lower tails and we develop exact results for an asymptotic regime where lower tail size approaches zero.

Divergences between probability distributions are a fundamental building block in statistics and are used to assess the degree to which one probability distribution is different from another probability distribution. They have a wide range of formulations [1] and applications, which range from use as objective functions in supervised and unsupervised machine learning [62], to hypothesis and two sample or goodness of fit testing in statistics [63], as well as generative modeling in deep learning, particularly using the Wasserstein distance [64]. Asymptotic forms of KL divergence have been investigated by Contreras-Reyes [65], for comparison of multivariate asymmetric heavy-tailed distributions.

Finally, we note that this work considerably expands a recent study by Bailey et al. [3], which established relationships between tail entropies and LID. This current paper extends and generalizes that work in several directions: (i) We establish general lemmas that provide sufficient conditions for when it is possible to substitute a tail distribution with components such as a power law, inside an integral. The techniques of [3] were specially crafted for specific integrals. (ii) We provide results for statistical divergences and distances (the work of [3] only considers entropy). (iii) We show how to formulate results for the multivariate context (as [3] only considers univariate scenarios).

3. Local Intrinsic Dimensionality

In this section, we summarize the LID model using the presentation of [2]. LID can be regarded as a continuous extension of the expansion dimension [66,67]. Like earlier expansion-based models of intrinsic dimension, its motivation comes from the relationship between volume and radius in an expanding ball, where (as originally stated in [68]) the volume of the ball is taken to be the probability measure associated with the region it encloses. The probability as a function of radius—denoted by $F\left(r\right)$—has the form of a univariate cumulative distribution function (CDF). The model formulation (as stated in [2]) generalizes this notion to real-valued functions F for which $F\left(0\right)=0$, under appropriate assumptions of smoothness.

Definition 1

([2]). Let F be a real-valued function that is non-zero over some open interval containing $r\in \mathbb{R}$, $r\ne 0$. The intrinsic dimensionality of F at r is defined as follows whenever the limit exists:

$$\begin{array}{ccc}\hfill {\mathrm{IntrDim}}_F\left(r\right)& \triangleq & \underset{ϵ\to 0}{lim}\frac{ln\left(F\left(\right(1+ϵ\left)r\right)/F\left(r\right)\right)}{ln(1+ϵ)}.\hfill \end{array}$$

When F satisfies certain smoothness conditions in the vicinity of r, its intrinsic dimensionality has a convenient known form:

Theorem 1

([2]). Let F be a real-valued function that is non-zero over some open interval containing $r\in \mathbb{R}$, $r\ne 0$. If F is continuously differentiable at r and using $F^{\prime }\left(r\right)$ to denote the derivative $\frac{dF\left(r\right)}{dr}$, then

$${ID}_F\left(r\right)\triangleq \frac{r·F^{\prime }\left(r\right)}{F\left(r\right)}={\mathrm{IntrDim}}_F\left(r\right).$$

Let $\mathbf{x}$ be a location of interest within a data domain $\mathcal{S}$ for which the distance measure $d:\mathcal{S}\times \mathcal{S}\to {\mathbb{R}}_{+}\cup 0$ has been defined. To any generated sample $\mathbf{s}\in \mathcal{S}$ we associate the distance $d(\mathbf{x},\mathbf{s})$; in this way, a global distribution that produces the sample $\mathbf{s}$ can be said to induce the random value $d(\mathbf{x},\mathbf{s})$ from a local distribution of distances taken with respect to $\mathbf{x}$. The CDF $F\left(r\right)$ of the local distance distribution is simply the probability of the sample distance lying within a threshold r—that is, $F\left(r\right)\triangleq \mathsf{Pr}\left[d\right(\mathbf{x},\mathbf{s})\le r]$. In characterizing the local intrinsic dimensionality in the vicinity of location $\mathbf{x}$, we are interested in the limit of ${ID}_F\left(r\right)$ as the distance r tends to 0, which we denote by

$${ID}_F^{\ast }\triangleq \underset{r\to 0}{lim}{ID}_F\left(r\right).$$

Henceforth, when we refer to the local intrinsic dimensionality (LID) of a function F, or of a point $\mathbf{x}$ whose induced distance distribution has F as its CDF, we will take ‘LID’ to mean the quantity ${ID}_F^{\ast }$. In general, ${ID}_F^{\ast }$ is not necessarily an integer. In practice, estimation of the LID at $\mathbf{x}$ would give an indication of the dimension of the submanifold containing $\mathbf{x}$ that best fits the distribution.

The function ${ID}_F$ can be seen to fully characterize its associated function F. This result is analogous to a foundational result from the statistical theory of extreme values (EVT), in that it corresponds under an inversion transformation to the Karamata representation theorem [69] for the upper tails of regularly varying functions. For more information on EVT and how the LID model relates to the extreme-value theoretic generalized pareto distribution, we refer the reader to [2,70,71].

Theorem 2 (LID Representation Theorem

[2]).Let $F:\mathbb{R}\to \mathbb{R}$ be a real-valued function, and assume that ${ID}_F^{\ast }$ exists. Let x and w be values for which $x/w$ and $F\left(x\right)/F\left(w\right)$ are both positive. If F is non-zero and continuously differentiable everywhere in the interval $[min\{x,w\},max\{x,w\left\}\right]$, then

$$\begin{array}{ccc}\hfill \frac{F\left(x\right)}{F\left(w\right)}={\left(\frac{x}{w}\right)}^{{ID}_F^{\ast }}·A_F(x,w),& \mathit{where}& A_F(x,w)\triangleq exp\left({\int }_x^w\frac{{ID}_F^{\ast }-{ID}_F\left(t\right)}{t}\mathrm{d}t\right),\hfill \end{array}$$

whenever the integral exists.

In [2], conditions on x and w are provided for which the factor $A_F(x,w)$ can be seen to tend to 1 as $x,w\to 0$. The convergence characteristics of F to its asymptotic form are expressed by the factor $A_F(x,w)$, which is related to the slowly varying component of functions as studied in EVT [70]. As we will shown in the next section, we make use of the LID Representation Theorem in our analysis of the limits of tail entropy variants under a form of normalization.

4. Definitions of Tail Entropies and Tail Dissimilarity Measures

In this section, we present the formulations of entropy, divergences and distances that will be studied in the later sections, in the light of the model of local intrinsic dimensionality outlined in Section 3. These entropies and dissimilarity measures will all be conditioned on the lower tails of smooth functions on domains bounded from below at zero. In each case, the formulations involve one or more non-negative real-valued functions whose restriction to $[0,w]$ satisfies certain smooth growth properties:

Definition 2.

Let$F:{\mathbb{R}}_{+}\cup 0\to {\mathbb{R}}_{+}\cup 0$be a function that is positive except at$F\left(0\right)=0$. We say that F is a smooth growth function if

• There exists a value $r>0$ such that F is monotonically increasing over $(0,r)$;
• F is continuous over $[0,r)$;
• F is differentiable over $(0,r)$; and
• The local intrinsic dimensionality ${ID}_F^{\ast }$ exists and is positive.

Given a smooth growth function F and a value $w>0$, we define $F_w\left(t\right)\triangleq \frac{F\left(t\right)}{F\left(w\right)}$. If F is the CDF of some random variable $X\ge 0$, then $F_w\left(t\right)=\mathsf{Pr}[X\le t|X\le w]$, which can in turn be interpreted as the CDF of the distribution of X conditioned to the lower tail $[0,w]$. It is easy to see that for a sufficiently small choice of w, $F_w$ must also be a smooth growth function. Its derivative $F_w^{\prime }\left(t\right)=\frac{F^{\prime }\left(t\right)}{F\left(w\right)}$ exists since $F^{\prime }\left(t\right)$ exists, and thus can be regarded as the probability density function (PDF) of the restriction of F to $[0,w]$. In addition, it can easily be shown (using Theorem 1) that the LID of $F_w$ is identical to that of F.

If the monotonicity of the function F is strict over the domain of interest $[0,r)$, its inverse function $F^{-1}$ exists and satisfies the smooth growth conditions within some neighborhood of the origin. Moreover, $F_w^{-1}$ is also a smooth growth function over $[0,1]$, with $F_w^{-1}\left(0\right)=0$ and $F_w^{-1}\left(1\right)=w$.

The following tail entropy, tail divergence and tail distance formulations all apply to any functions F and G satisfying the conditions stated above; in particular, they involve one or more of $F_w$, $F_w^{\prime }$, $G_w$, $G_w^{\prime }$, and (if the monotonicity of the functions is strict) $F_w^{-1}$ and $G_w^{-1}$. In their definitions, the only difference between the tail variants and the original versions is that the distributions are conditioned on the lower tail $[0,w]$. In the tail measures involving one or more of $F_w$, $F_w^{\prime }$, $G_w$ and $G_w^{\prime }$, integration is performed over the lower tail and not the entire distributional range $[0,+\infty )$; for the variant involving $F_w^{-1}$ and $G_w^{-1}$, integration is performed over $[0,1]$ for values of w constrained to the lower tail.

We begin with (differential) tail entropy. Entropy is perhaps the most fundamental and widely used model of data complexity and can be regarded as a measure of the uncertainty of a distribution. Differential entropy assesses the expected surprisal of a random variable and can take negative values.

Definition 3

(Tail Entropy).The entropy of F conditioned on $[0,w]$ is

$$\mathrm{H}(F,w)\triangleq -{\int }_0^w{F}_w^{\prime }\left(t\right)ln{F}_w^{\prime }\left(t\right)\mathrm{d}t.$$

The tail entropy is equal to $E(-log{F}_w^{\prime })$, the expected value of the (tail) log-likelihood. It is also possible to define the variance of the (tail) log-likelihood. This is known as the varentropy. Understanding this further, note that one may define the information content of a random variable X with density function f, to be $-logf\left(X\right)$. The entropy (uncertainty) then corresponds to the expected value of the information content of X and the varentropy corresponds to the variance of the information content of X. The varentropy was introduced by Song [72] as an intrinsic measure of the shape of a distribution and has been explored in a range of studies [73,74,75].

Definition 4

(Tail varentropy).The varentropy of F conditioned on $[0,w]$ is

$$\mathrm{VarH}(F,w)\triangleq {\int }_0^w{F}_w^{\prime }\left(t\right){ln}^2{F}_w^{\prime }\left(t\right)\mathrm{d}t-{\left({\int }_0^w{F}_w^{\prime }\left(t\right)ln{F}_w^{\prime }\left(t\right)\mathrm{d}t\right)}^2$$

The cumulative entropy is a variant of entropy proposed in [55,56] due to its attractive theoretical properties. Tail conditioning on the cumulative entropy has the same general form as that of the tail entropy. Cumulative entropy [55,56] is an information-theoretic measure popular in reliability theory, where it is used to model uncertainty over time intervals. It corresponds to the expected value of the mean inactivity time. Compared to ordinary Shannon differential entropy, cumulative entropy has certain attractive properties, such as non-negativity and ease of estimation.

Definition 5

(Cumulative Tail Entropy).The cumulative entropy of F conditioned on $[0,w]$ is

$$\mathrm{cH}(F,w)\triangleq -{\int }_0^w{F}_w\left(t\right)ln{F}_w\left(t\right)\mathrm{d}t.$$

The entropy power is the exponential of the entropy, and is also known as perplexity in the natural language processing community. It corresponds to the volume of the smallest set that contains most of the probability measure [49], and can be interpreted as a measure of statistical dispersion [53]. There are several standard definitions of entropy power in the research literature. For our purposes, we adopt the simplest—the exponential of Shannon entropy—for our definition conditioned to the tail.

Definition 6

(Tail Entropy Power).The entropy power of F conditioned on $[0,w]$ is defined to be

$$\mathrm{HP}(F,w)\triangleq exp\left(\mathrm{H}\right(F,w\left)\right).$$

In the introduction, we briefly mentioned some motivation for the entropy power $\mathrm{HP}(F,w)$. We can add to this as follows:

• It can be interpreted as a diversity. Observe that when F is a (univariate) uniform distance distribution ranging over the interval $[0,w]$, we have ${ID}_F^{\ast }=1$ and $\mathrm{HP}(F,w)=w$. In other words, the entropy power is equal to the ‘effective diversity’ of the distribution (the number of neighbor distance possibilities).
• Given two different queries, each with its own neighborhood, one query with tail entropy power equal to 2 and the other with tail entropy power equal to 4, we can say that the distance distribution of the second query is twice as diverse as that of the first query.

For each of the tail entropy variants introduced above, we also propose analogous variants based on the q-entropy formulation due to Tsallis [59]. Generalized Tsallis entropies [59,60] are a family of entropies characterized via an exponent parameter q applied to the probabilities, in which the traditional (Shannon) entropy variants are obtained as the special case when q is allowed to tend to 1. The use of such a parameter q can often facilitate more accurate fitting of data characteristics and robustness to outliers.

Definition 7

(Tail q-Entropy).For any $q>0$$(q\ne 1)$, the q-entropy of F conditioned on $[0,w]$ is defined to be

$$\begin{array}{ccc}\hfill {\mathrm{H}}_q(F,w)& \triangleq & \frac{1}{q-1}\left(1-{\int }_0^w{\left(F_w^{\prime }\left(t\right)\right)}^q\mathrm{d}t\right)=\frac{1}{q-1}{\int }_0^w{F}_w^{\prime }\left(t\right)-{\left(F_w^{\prime }\left(t\right)\right)}^q\mathrm{d}t.\hfill \end{array}$$

Definition 8

(Cumulative Tail q-Entropy).For any $q>0$$(q\ne 1)$, the cumulative q-entropy of F conditioned on $[0,w]$ is defined to be

$$\begin{array}{c}\hfill {\mathrm{cH}}_q(F,w)\triangleq \frac{1}{q-1}{\int }_0^w{F}_w\left(t\right)-{\left(F_w\left(t\right)\right)}^q\mathrm{d}t.\end{array}$$

We define the tail q-entropy power using the q-exponential function from Tsallis statistics [59], ${exp}_q\left(x\right)\triangleq {[1+(1-q)x]}^{\frac{1}{1-q}}$. Note that L’Hôpital’s rule can be used to show that ${exp}_q\left(x\right)\to e^x$ as $q\to 1$.

Definition 9

(Tail q-Entropy Power).For any $q>0$$(q\ne 1)$, the q-entropy power of F conditioned on $[0,w]$ is defined to be

$${\mathrm{HP}}_q(F,w)\triangleq {\left[1+(1-q){\mathrm{H}}_q(F,w)\right]}^{\frac{1}{1-q}}.$$

We next define the tail cross entropy. Cross entropy can be used to compare two probability distributions and is often employed as a loss function in machine learning, comparing a true distribution and a learned distribution. From an information theoretic perspective, cross entropy corresponds to the expected coding length when a wrong distribution G is assumed while the data actually follows a distribution F.

Definition 10

(Tail Cross Entropy).The cross entropy from F to G, conditioned on $[0,w]$, is defined to be

$$\mathrm{XH}(F;G,w)\triangleq {-\int }_0^w{F}_w^{\prime }\left(t\right)ln{G}_w^{\prime }\left(t\right)\mathrm{d}t.$$

Similar to entropy power, we can also define the cross entropy power, which is the exponential of the cross entropy.

Definition 11

(Tail Cross Entropy Power).The cross entropy power from F to G, conditioned on $[0,w]$, is defined to be

$$\mathrm{XHP}(F;G,w)\triangleq exp\left({-\int }_0^w{F}_w^{\prime }\left(t\right)ln{G}_w^{\prime }\left(t\right)\mathrm{d}t\right).$$

A classic and fundamental method for comparing two probability distributions is the Kullback–Leibler divergence (KL Divergence) [76]. $KL(F,G)$ measures the degree to which a probability distribution G is different from a reference probability distribution F. It is a member of both the family of f-divergences and Bregman divergences. It is widely used in statistics, machine learning and information theory.

Definition 12

(Tail KL Divergence).The Kullback–Leibler divergence from F to G, conditioned on $[0,w]$, is defined to be

$$\mathrm{KL}(F;G,w)\triangleq {\int }_0^w{F}_w^{\prime }\left(t\right)ln\frac{F_w^{\prime }\left(t\right)}{G_w^{\prime }\left(t\right)}\mathrm{d}t.$$

The tail KL divergence can be connected to the tail entropy and the tail cross entropy according to the relationship $\mathrm{KL}(F;G,w)=\mathrm{XH}(F;G;w)-\mathrm{H}(F,w)$.

The Jensen–Shannon divergence (JS divergence) [77] is another popular measure of distance between probability distributions. It is based on the KL divergence, but unlike the KL, the square root of the JS divergence is a true metric.

Definition 13

(Tail JS Divergence).The Jensen–Shannon divergence between F and G, conditioned on $[0,w]$, is defined to be

$$\mathrm{JS}(F;G,w)\triangleq \frac{\mathrm{KL}(F;M,w)+\mathrm{KL}(G;M,w)}{2},\mathrm{where}M\left(t\right)=\frac{F\left(t\right)+G\left(t\right)}{2}.$$

The tail JS divergence can also be written in terms of the tail entropies $\mathrm{JS}(F;G,w)=\mathrm{H}(\frac{F+G}{2},w)-\frac{\mathrm{H}(F,w)+\mathrm{H}(G,w)}{2}$

The L2 distance is the squared Euclidean distance when comparing two probability distributions. It is part of the family of $\beta$ divergences when setting $\beta =2$ [78].

Definition 14

(Tail L2 Distance).The L2 distance between F and G, conditioned on $[0,w]$, is defined to be

$$\mathrm{L}2\mathrm{D}(F;G,w)\triangleq {\int }_0^w{\left(F_w^{\prime }\left(t\right)-G_w^{\prime }\left(t\right)\right)}^2\mathrm{d}t.$$

The Hellinger distance [79] is a true metric for comparing two probability distributions. The squared Hellinger distance a member of the family of f-divergences and is part of the family of $\alpha$ divergences when setting $\alpha =\frac{1}{2}$ [80].

Definition 15

(Tail Hellinger Distance).The Hellinger distance between F and G, conditioned on $[0,w]$, is defined to be

$$\mathrm{HD}(F;G,w)\triangleq \sqrt{\frac{1}{2}{\int }_0^w{\left(\sqrt{F_w^{\prime }\left(t\right)}-\sqrt{G_w^{\prime }\left(t\right)}\right)}^2\mathrm{d}t}.$$

The ${\chi }^2$ divergence between two probability distributions [81] is a member of the family of f divergences and is part of the family of $\alpha$ divergences when setting $\alpha =2$ [80].

Definition 16

(Tail${\chi }^{\mathbf{2}}$-Divergence).The ${\chi }^2$ divergence between F and G, conditioned on $[0,w]$, is defined to be

$${\chi }^2\mathrm{D}(F;G,w)\triangleq {\int }_0^w\frac{{\left(F_w^{\prime }\left(t\right)-G_w^{\prime }\left(t\right)\right)}^2}{G_w^{\prime }\left(t\right)}\mathrm{d}t.$$

The asymmetric $\alpha$-divergence [80] is another member of the family of f divergences. When $\alpha =2$ it is proportional to the ${\chi }^2$ divergence. When $\alpha =0.5$ it is proportional to the squared Hellinger distance. When $\alpha \to 1$ it corresponds to the KL-divergence.

Definition 17

(Tail$\mathbf{\alpha }$-Divergence).The α-divergence from F to G, conditioned on $[0,w]$, is defined to be

$$\alpha \mathrm{D}(F;G,w)\triangleq \frac{1}{\alpha (1-\alpha )}{\int }_0^w\alpha F_w^{\prime }\left(t\right)+(1-\alpha )G_w^{\prime }\left(t\right)-{\left(F_w^{\prime }\left(t\right)\right)}^{\alpha }{\left(G_w^{\prime }\left(t\right)\right)}^{1-\alpha }\mathrm{d}t.$$

The Wasserstein distance between two probability distributions is also known as the Kantorovich–Rubinstein metric [82] or the earth mover’s distance. It has become very popular as part of the loss function used in generative adversarial networks [83]. In the univariate case it can be expressed in a simple analytic form.

Definition 18

(Tail Wasserstein Distance).The p-th Wasserstein distance between F and G, conditioned on $[0,w]$, is defined to be

$${\mathrm{WD}}_p(F;G,w)\triangleq {\left({\int }_0^1{\left|F_w^{-1}\left(u\right)-G_w^{-1}\left(u\right)\right|}^p\mathrm{d}u\right)}^{\frac{1}{p}}.$$

For some of the aforementioned tail measures, we will also consider a normalization of the entropy, divergence or distance (as the case may be) with respect to w, the length of the tail. In Section 5 and Section 6, we will show that as w tends to zero, the limits of these (possibly normalized) tail entropies and tail divergences can be expressed in terms of the local intrinsic dimensionalities of F and G. The notation for these variants, and our results for their limits in terms of ${ID}_F^{\ast }$ and ${ID}_G^{\ast }$, are summarized in

Table 1. Asymptotic equivalences between LID formulations and tail measures of entropy or divergence. In each case, the functions F and G are assumed to be smooth growth functions. In addition, for the Normalized Wasserstein Distance, F and G must be strictly monotonically increasing, thereby guaranteeing that the inverses of $F_w$ and $G_w$ exist near zero. In some cases, for the asymptotic limit to exist non-trivially (that is, to be both finite and non-zero), the tail entropy or tail divergence must be normalized by the multiplicative factor $\frac{1}{w}$, w. For the Tail Entropy and Tail Cross Entropy, no reweighting by powers of w can lead to a non-trivial asymptotic limit as w tends to zero.

Tail Measure	Formulation	Limit as $\mathit{w}\to 0^{+}$
Entropy	$\mathrm{H}(F,w)$ = $-{\int }_0^w{F}_w^{\prime }\left(t\right)ln{F}_w^{\prime }\left(t\right)\mathrm{d}t$	Diverges (no reweighting possible)
Varentropy	$\mathrm{VarH}(F,w)$ = ${\int }_0^w{F}_w^{\prime }\left(t\right){ln}^2{F}_w^{\prime }\left(t\right)\mathrm{d}t-{\left({\int }_0^w{F}_w^{\prime }\left(t\right)ln{F}_w^{\prime }\left(t\right)\mathrm{d}t\right)}^2$	${\left(1-\frac{1}{{ID}_F^{\ast }}\right)}^2$
q-Entropy	${\mathrm{H}}_q(F,w)$ = $\frac{1}{q-1}{\int }_0^w{F}_w^{\prime }\left(t\right)-{\left(F_w^{\prime }\left(t\right)\right)}^q\mathrm{d}t$	$\frac{1}{q-1}$  if  $q<1$, diverges if $q>1$
Normalized Cumulative Entropy	$\frac{1}{w}\mathrm{cH}(F,w)$ = $-\frac{1}{w}{\int }_0^w{F}_w\left(t\right)ln{F}_w\left(t\right)\mathrm{d}t$	$\frac{{ID}_F^{\ast }}{{({ID}_F^{\ast }+1)}^2}$
Normalized Cumulative q-Entropy	$\frac{1}{w}{\mathrm{cH}}_q(F,w)$ = $\frac{1}{w(q-1)}{\int }_0^w{F}_w\left(t\right)-{\left(F_w\left(t\right)\right)}^q\mathrm{d}t$	$\frac{{ID}_F^{\ast }}{({ID}_F^{\ast }+1)(q{ID}_F^{\ast }+1)}$   if   $q\ne 1$
Normalized Entropy Power	$\frac{1}{w}\mathrm{HP}(F,w)$ = $\frac{1}{w}exp\left(\mathrm{H}(F,w)\right)$	$\frac{1}{{ID}_F^{\ast }}exp\left(1-\frac{1}{{ID}_F^{\ast }}\right)$
Normalized q-Entropy Power	$\frac{1}{w}{\mathrm{HP}}_q(F,w)$ = $\frac{1}{w}{\left[1+(1-q){\mathrm{H}}_q(F,w)\right]}^{\frac{1}{1-q}}$	${\left(\frac{{\left({ID}_F^{\ast }\right)}^q}{q{ID}_F^{\ast }-q+1}\right)}^{\frac{1}{1-q}}$
		if   $q\ne 1$   and   $q{ID}_F^{\ast }-q+1>0$
Cross Entropy	$\mathrm{XH}(F;G,w)$ = $-{\int }_0^w{F}_w^{\prime }\left(t\right)ln{G}_w^{\prime }\left(t\right)\mathrm{d}t$	Diverges (no reweighting possible)
Normalized Cross Entropy Power	$\frac{1}{w}\mathrm{XHP}(F;G,w)$ = $\frac{1}{w}exp\left({-\int }_0^w{F}_w^{\prime }\left(t\right)ln{G}_w^{\prime }\left(t\right)\mathrm{d}t\right)$	$\frac{1}{{ID}_G^{\ast }}exp\left(\frac{{ID}_G^{\ast }-1}{{ID}_F^{\ast }}\right)$
KL Divergence	$\mathrm{KL}(F;G,w)$ = ${\int }_0^w{F}_w^{\prime }\left(t\right)ln\frac{F_w^{\prime }\left(t\right)}{G_w^{\prime }\left(t\right)}\mathrm{d}t$	$\rho -ln\rho -1;\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$
JS Divergence	$\mathrm{JS}(F;G,w)$ = $\frac{1}{2}\left(\mathrm{KL}\left(F;\frac{F+G}{2},w\right)+\mathrm{KL}\left(G;\frac{F+G}{2},w\right)\right)$	$\frac{1}{2}\left(\tau -ln\tau -1\right);\tau =min\{\rho ,\frac{1}{\rho }\};\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$
Weighted L2 Distance	$w\mathrm{L}2\mathrm{D}(F;G,w)$ = $w{\int }_0^w{\left(F_w^{\prime }\left(t\right)-G_w^{\prime }\left(t\right)\right)}^2\mathrm{d}t$	$\frac{{\left({ID}_F^{\ast }-{ID}_G^{\ast }\right)}^2}{2({ID}_F^{\ast }+{ID}_G^{\ast }-1)}\left[1+\frac{1}{(2{ID}_F^{\ast }-1)(2{ID}_G^{\ast }-1)}\right]$
		${ID}_F^{\ast }>\frac{1}{2}$; ${ID}_G^{\ast }>\frac{1}{2}$
Hellinger Distance	$\mathrm{HD}(F;G,w)$ = $\sqrt{\frac{1}{2}{\int }_0^w{\left(\sqrt{F_w^{\prime }\left(t\right)}-\sqrt{G_w^{\prime }\left(t\right)}\right)}^2\mathrm{d}t}$	$\frac{\left|1-\sqrt{\rho }\right|}{\sqrt{1+\rho }};\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$
${\chi }^2$-Divergence	${\chi }^2\mathrm{D}(F;G,w)$ = ${\int }_0^w\frac{{\left(F_w^{\prime }\left(t\right)-G_w^{\prime }\left(t\right)\right)}^2}{G_w^{\prime }\left(t\right)}\mathrm{d}t$	$\frac{{\left(1-\rho \right)}^2}{\rho (2-\rho )};\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }};\rho <2$
$\alpha$-Divergence	$\alpha \mathrm{D}(F;G,w)$ = $\frac{1}{\alpha (1-\alpha )}{\int }_0^w\alpha F_w^{\prime }\left(t\right)+(1-\alpha )G_w^{\prime }\left(t\right)$	$\frac{1}{\alpha (1-\alpha )}\left(1-\frac{1}{\alpha {\rho }^{\alpha -1}+(1-\alpha ){\rho }^{\alpha }}\right)$
	$-{\left(F_w^{\prime }\left(t\right)\right)}^{\alpha }{\left(G_w^{\prime }\left(t\right)\right)}^{1-\alpha }\mathrm{d}t$	$\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }};\alpha +\rho (1-\alpha )>0$
Normalized Wasserstein Distance	$\frac{1}{w}{\mathrm{WD}}_p(F;G,w)$ = $\frac{1}{w}{\left({\int }_0^1{\left|F_w^{-1}\left(u\right)-G_w^{-1}\left(u\right)\right|}^p\mathrm{d}u\right)}^{\frac{1}{p}}$	$p=2$:  $\sqrt{\frac{1}{\frac{2}{{ID}_F^{\ast }}+1}-\frac{2}{\frac{1}{{ID}_F^{\ast }}+\frac{1}{{ID}_G^{\ast }}+1}+\frac{1}{\frac{2}{{ID}_G^{\ast }}+1}}$
		p even:  ${\left({\sum }_{j=0}^p\frac{{(-1)}^j\left(\stackrel{p}{j}\right)}{(p-j)·{\left({ID}_F^{\ast }\right)}^{-1}+j·{\left({ID}_G^{\ast }\right)}^{-1}+1}\right)}^{\frac{1}{p}}$

.

5. Simplification of Tail Measures

Next, we present the main theoretical contributions of the paper: three technical lemmas that will later be used to establish relationships between local intrinsic dimensionality and a variety of tail measures based on entropy, divergences or distances. The results presented in this section all apply asymptotically, as the tail boundary tends toward zero.

Each of the three lemmas allow, under certain conditions, the simplification of limits of integrals involving smooth growth functions of the form $F_w$ (as defined in Section 4), or its associated first derivative $F_w^{\prime }$ or inverse function $F_w^{-1}$. The limit integral simplifications allow for the substitution of the function (or derivative or inverse) by expressions that involve one or more of the following: the LID value of the function, the variable of integration or the tail boundary w. Moreover, the lemmas require that the integrand be monotone with respect to small variations in the targeted function.

The first lemma allows terms of the form $F_w$ (resembling the CDF of a tail-conditioned distribution) to be converted into a term that depends only on the variable of integration, the tail length w, and the local intrinsic dimension ${ID}_F^{\ast }$.

Lemma 1.

Let F be a smooth growth function over the interval $[0,r)$. Consider the function $\varphi :{\mathbb{R}}_{+}^2\to \mathbb{R}$ admitting a representation of the form

$$\varphi (t,w)\equiv \psi \left(t,w,z(t,w)\right),$$

where:

• $\psi :{\mathbb{R}}_{+}^3\to \mathbb{R}$;
• $z(t,w)=F_w\left(t\right)=\frac{F\left(t\right)}{F\left(w\right)}$; and
• for all fixed choices of t and w satisfying $0<t\le w<r$, $\psi (t,w,z)$ is monotone and continuously partially differentiable with respect to z over the interval $z\in (0,1]$.

Then

$$\begin{array}{ccc}\hfill \underset{w\to 0^{+}}{lim}{\int }_0^w\varphi (t,w)\mathrm{d}t& \equiv & \underset{w\to 0^{+}}{lim}{\int }_0^w\psi \left(t,w,F_w\left(t\right)\right)\mathrm{d}t\hfill \\ & =& \underset{w\to 0^{+}}{lim}{\int }_0^w\psi \left(t,w,{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}\right)\mathrm{d}t\hfill \end{array}$$

whenever the latter limit exists or diverges to $+\infty$ or $-\infty$.

Proof. 

Since F is assumed to be a smooth growth function, the limit ${ID}_F^{\ast }={lim}_{v\to 0^{+}}{ID}_F\left(v\right)$ exists and is positive. We present an ‘epsilon-delta’ argument based on this limit. For any real value $ϵ>0$ satisfying $ϵ<min\{r,{ID}_F^{\ast }\}$, there must exist a value $0<\delta <ϵ$ such that $v<\delta$ implies that $|{ID}_F\left(v\right)-{ID}_F^{\ast }|<ϵ$. Therefore, when $0<t\le w<\delta$,

$$\begin{array}{ccc}\hfill |ln{A}_F\left(t,w\right)|& =& \left|{\int }_t^w\frac{{ID}_F^{\ast }-{ID}_F\left(v\right)}{v}\mathrm{d}v\right|<ϵ·\left|{\int }_t^w\frac{1}{v}\mathrm{d}v\right|=ϵ·ln\frac{w}{t}.\hfill \end{array}$$

Exponentiating, we obtain the bounds

$${\left(\frac{w}{t}\right)}^{-ϵ}<A_F\left(t,w\right)<{\left(\frac{w}{t}\right)}^{ϵ}.$$

Applying this bound together with Theorem 2, the ratio $F_w\left(t\right)=\frac{F\left(t\right)}{F\left(w\right)}$ can be seen to satisfy

$${\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }+ϵ}<\frac{F\left(t\right)}{F\left(w\right)}=A_F\left(t,w\right)·{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}<{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }-ϵ}.$$

(1)

Over the domain of interest $0<t\le w<\delta$, the assumption that $0<ϵ<min\{r,{ID}_F^{\ast }\}$ ensures that $0<\frac{t}{w}\le 1$, and that the upper and lower bounds of Inequality (1) lie in the interval $(0,1]$. Since $\psi (t,w,z)$ has been assumed to be monotone with respect to $z\in (0,1]$, the maximum and minimum attained by $\psi$ over choices of z restricted to any (closed) subinterval of $(0,1]$ must occur at opposite endpoints of the subinterval. With this in mind, for any choice of $ϵ\in (0,min\{r,{ID}_F^{\ast }\})$, Inequality (1) implies that

$$\begin{array}{ccc}\hfill B_{min}(t,w,ϵ)\le & {\displaystyle \psi \left(t,w,F_w\left(t\right)\right)}& \le B_{max}(t,w,ϵ)\hfill \\ \hfill \mathrm{and}{\int }_0^w{B}_{min}(t,w,ϵ)\mathrm{d}t\le & {\displaystyle {\int }_0^w\psi \left(t,w,F_w\left(t\right)\right)\mathrm{d}t}& \le {\int }_0^w{B}_{max}(t,w,ϵ)\mathrm{d}t,\hfill \end{array}$$

where

$$\begin{array}{ccc}\hfill B_{min}(t,w,ϵ)& \triangleq & min\left\{\psi \left(t,w,{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }-ϵ}\right),\psi \left(t,w,{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }+ϵ}\right)\right\},\hfill \\ \hfill B_{max}(t,w,ϵ)& \triangleq & max\left\{\psi \left(t,w,{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }-ϵ}\right),\psi \left(t,w,{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }+ϵ}\right)\right\}.\hfill \end{array}$$

Since $\psi (t,w,z)$ and ${\int }_0^w\psi (t,w,z)\mathrm{d}t$ are also continuously partially differentiable with respect to z over $z\in (0,1]$,

$$\begin{array}{ccc}\hfill \underset{ϵ\to 0^{+}}{lim}{B}_{min}(t,w,ϵ)=& {\displaystyle \underset{ϵ\to 0^{+}}{lim}{B}_{max}(t,w,ϵ)}& =\psi \left(t,w,{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}\right)\mathrm{and}\hfill \\ \hfill \underset{\stackrel{ϵ\to 0^{+}}{w<ϵ}}{lim}{\int }_0^w{B}_{min}(t,w,ϵ)\mathrm{d}t=& {\displaystyle \underset{\stackrel{ϵ\to 0^{+}}{w<ϵ}}{lim}{\int }_0^w{B}_{max}(t,w,ϵ)\mathrm{d}t}& =\underset{w\to 0^{+}}{lim}{\int }_0^w\psi \left(t,w,{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}\right)\mathrm{d}t.\hfill \end{array}$$

It therefore follows from the squeeze theorem for integrals that

$$\begin{array}{ccc}\hfill \underset{w\to 0^{+}}{lim}{\int }_0^w\psi \left(t,w,F_w\left(t\right)\right)\mathrm{d}t& =& \underset{w\to 0^{+}}{lim}{\int }_0^w\psi \left(t,w,{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}\right)\mathrm{d}t,\hfill \end{array}$$

whenever the right-hand limit exists or diverges. □

In a manner similar to that of the preceding lemma, the following result allows terms of the form $F_w^{-1}$ (the inverse of $F_w$) to be converted into a term that depends only on the variable of integration, the tail length w and the local intrinsic dimension ${ID}_F^{\ast }$. Here, in order to ensure the existence of the inverse function, F (and by extension $F_w$ and $F_w^{-1}$) must be strictly monotonically increasing over the tail.

Lemma 2.

Let F be a smooth growth function over the interval $[0,r)$. Let us also assume that, over the interval, the monotonicity of F is strict. Consider the function $\varphi :{\mathbb{R}}_{+}^2\to \mathbb{R}$ admitting a representation of the form

$$\varphi (u,w)\equiv \psi \left(u,w,z(u,w)\right),$$

where:

• $\psi :{\mathbb{R}}_{+}^3\to \mathbb{R}$;
• $z(u,w)=F_w^{-1}\left(u\right)$ for all $w\in (0,r)$, where $F_w\left(t\right)\triangleq F\left(t\right)/F\left(w\right)$ is restricted to values of t in $[0,w]$; and
• for all fixed choices of u and w satisfying $u\in [0,1]$ and $0<w<r$, $\psi (u,w,z)$ is monotone and continuously partially differentiable with respect to z over the interval $z\in (0,r)$.

Then

$$\begin{array}{ccc}\hfill \underset{w\to 0^{+}}{lim}{\int }_0^1\varphi (u,w)\mathrm{d}u& \equiv & \underset{w\to 0^{+}}{lim}{\int }_0^1\psi \left(u,w,F_w^{-1}\left(u\right)\right)\mathrm{d}u\hfill \\ & =& \underset{w\to 0^{+}}{lim}{\int }_0^1\psi \left(u,w,w{u}^{\frac{1}{{ID}_F^{\ast }}}\right)\mathrm{d}u\hfill \end{array}$$

whenever the latter limit exists or diverges to $+\infty$ or $-\infty$.

Proof. 

First, we note that the strict monotonicity of F implies that for all $u\in [0,1]$ and $w\in (0,r)$, the function $F_w^{-1}\left(u\right)$ is uniquely defined when $F_w$ is restricted to $[0,w]$.

As in the proof of Lemma 1, an ‘epsilon-delta’ argument based on the existence of the limit ${ID}_F^{\ast }={lim}_{v\to 0^{+}}{ID}_F\left(v\right)$ yields the following: for any real value $ϵ>0$ satisfying $ϵ<min\{r,{ID}_F^{\ast }\}$, there exists a value $\delta \in (0,ϵ)$ such that

$${\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }+ϵ}<F_w\left(t\right)=\frac{F\left(t\right)}{F\left(w\right)}<{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }-ϵ}$$

holds for all $0<t\le w<\delta$. Solving for t through exponentiation of the bounds, and then setting $t=F_w^{-1}\left(u\right)$, we obtain

$$\begin{array}{ccc}\hfill w·{\left(F_w\left(t\right)\right)}^{\frac{1}{{ID}_F^{\ast }-ϵ}}<& t& <w·{\left(F_w\left(t\right)\right)}^{\frac{1}{{ID}_F^{\ast }+ϵ}}\hfill \\ \hfill w·{\left(F_w\left(F_w^{-1}\left(u\right)\right)\right)}^{\frac{1}{{ID}_F^{\ast }-ϵ}}<& F_w^{-1}\left(u\right)& <w·{\left(F_w\left(F_w^{-1}\left(u\right)\right)\right)}^{\frac{1}{{ID}_F^{\ast }+ϵ}}\hfill \\ \hfill w{u}^{\frac{1}{{ID}_F^{\ast }-ϵ}}<& F_w^{-1}\left(u\right)& <w{u}^{\frac{1}{{ID}_F^{\ast }+ϵ}}.\hfill \end{array}$$

The remainder of the proof follows essentially the same path as that of Lemma 1. Over the domain of interest $0<t\le w<\delta$, the assumption that $0<ϵ<min\{r,{ID}_F^{\ast }\}$ ensures that $0<\frac{t}{w}\le 1$, and that u lies in the interval $(0,w]$. Since $\psi (u,w,z)$ has been assumed to be monotone with respect to $z\in (0,r)$, the maximum and minimum attained by $\psi$ over choices of z restricted to any (closed) subinterval of $(0,r)$ must occur at opposite endpoints. Therefore, for any choice of $ϵ\in (0,min\{r,{ID}_F^{\ast }\})$,

$$\begin{array}{ccc}\hfill C_{min}(u,w,ϵ)\le & {\displaystyle \psi \left(u,w,F_w^{-1}\left(u\right)\right)}& \le C_{max}(u,w,ϵ)\hfill \\ \hfill \mathrm{and}{\int }_0^1{C}_{min}(u,w,ϵ)\mathrm{d}u\le & {\displaystyle {\int }_0^1\psi \left(u,w,F_w^{-1}\left(u\right)\right)\mathrm{d}u}& \le {\int }_0^1{C}_{max}(u,w,ϵ)\mathrm{d}u,\hfill \end{array}$$

where

$$\begin{array}{ccc}\hfill C_{min}(u,w,ϵ)& \triangleq & min\left\{\psi \left(u,w,w{u}^{\frac{1}{{ID}_F^{\ast }-ϵ}}\right),\psi \left(u,w,w{u}^{\frac{1}{{ID}_F^{\ast }+ϵ}}\right)\right\},\hfill \\ \hfill C_{max}(u,w,ϵ)& \triangleq & max\left\{\psi \left(u,w,w{u}^{\frac{1}{{ID}_F^{\ast }-ϵ}}\right),\psi \left(u,w,w{u}^{\frac{1}{{ID}_F^{\ast }+ϵ}}\right)\right\}.\hfill \end{array}$$

Since $\psi (u,w,z)$ is also continuously partially differentiable with respect to z over $z\in (0,r)$,

$$\begin{array}{ccc}\hfill \underset{ϵ\to 0^{+}}{lim}{C}_{min}(u,w,ϵ)=& {\displaystyle \underset{ϵ\to 0^{+}}{lim}{C}_{max}(u,w,ϵ)}& =\psi \left(u,w,w{u}^{\frac{1}{{ID}_F^{\ast }}}\right)\mathrm{and}\hfill \\ \hfill \underset{\stackrel{ϵ\to 0^{+}}{w<ϵ}}{lim}{\int }_0^1{C}_{min}(u,w,ϵ)\mathrm{d}u=& {\displaystyle \underset{\stackrel{ϵ\to 0^{+}}{w<ϵ}}{lim}{\int }_0^1{C}_{max}(u,w,ϵ)\mathrm{d}u}& =\underset{w\to 0^{+}}{lim}{\int }_0^1\psi \left(u,w,w{u}^{\frac{1}{{ID}_F^{\ast }}}\right)\mathrm{d}u.\hfill \end{array}$$

It therefore follows from the squeeze theorem for integrals that

$$\begin{array}{ccc}\hfill \underset{w\to 0^{+}}{lim}{\int }_0^1\psi \left(u,w,F_w^{-1}\left(u\right)\right)\mathrm{d}u& =& \underset{w\to 0^{+}}{lim}{\int }_0^1\psi \left(u,w,w{u}^{\frac{1}{{ID}_F^{\ast }}}\right)\mathrm{d}u,\hfill \end{array}$$

whenever the right-hand limit exists or diverges. □

The third lemma facilitates the conversion of a term of the form $F_w^{\prime }$ to $F_w$, together with a factor that depends only on the variable of integration and ${ID}_F^{\ast }$. Since F is assumed to be a smooth growth function, $F_w$ must be smooth as well, and therefore $F_w$ satisfies the conditions of Theorem 1 over $[0,w)$. Hence, $F_w^{\prime }$ can be substituted by an expression involving $F_w$:

$$F_w^{\prime }\left(t\right)=\frac{{ID}_{F_w}\left(t\right)}{t}·F_w\left(t\right)=\frac{{ID}_F\left(t\right)}{t}·F_w\left(t\right).$$

The substitution comes at the cost of introducing a non-constant factor ${ID}_F\left(t\right)$. The following lemma shows that ${ID}_F\left(t\right)$ can in turn be substituted by the constant ${ID}_F^{\ast }$, provided that certain monotonicity assumptions are satisfied.

Lemma 3.

Let F be a smooth growth function over the interval $[0,r)$. Consider the function $\varphi :{\mathbb{R}}_{+}^2\to \mathbb{R}$ admitting a representation of the form

$$\varphi (t,w)\equiv \psi \left(t,w,z(t,w)\right),$$

where:

• $\psi :{\mathbb{R}}_{+}^3\to \mathbb{R}$;
• $z(t,w)={ID}_F\left(t\right)$, and
• there exists a value $\gamma \in (0,{ID}_F^{\ast })$ such that for all fixed choices of t satisfying $0<t\le w<r$, $\psi (t,w,z)$ is monotone with respect to z over the interval $z\in ({ID}_F^{\ast }-\gamma ,{ID}_F^{\ast }+\gamma )$.

Then

$$\begin{array}{ccc}\hfill \underset{w\to 0^{+}}{lim}{\int }_0^w\varphi (t,w)\mathrm{d}t& \equiv & \underset{w\to 0^{+}}{lim}{\int }_0^w\psi \left(t,w,{ID}_F\left(t\right)\right)\mathrm{d}t=\underset{w\to 0^{+}}{lim}{\int }_0^w\psi \left(t,w,{ID}_F^{\ast }\right)\mathrm{d}t\hfill \end{array}$$

whenever the latter limit exists or diverges to $+\infty$ or $-\infty$.

Proof. 

Since F is assumed to be a smooth growth function, the limit ${ID}_F^{\ast }={lim}_{v\to 0^{+}}{ID}_F\left(v\right)$ exists and is positive. We present an ‘epsilon-delta’ argument based on this limit. For any real value $ϵ>0$ satisfying $ϵ<min\{r,\gamma \}$, there must exist a value $0<\delta <ϵ$ such that $v<\delta$ implies that $|{ID}_F\left(v\right)-{ID}_F^{\ast }|<ϵ$.

Since $\psi (t,w,z)$ has been assumed to be monotone with respect to z over the interval $z\in ({ID}_F^{\ast }-\gamma ,{ID}_F^{\ast }+\gamma )$, the restriction $v<\delta <ϵ<min\{r,\gamma \}$ ensures that $\psi (t,w,z)$ is monotone over the entire domain of interest $0<t\le w<\delta$. Therefore, the maximum and minimum attained by $\psi$ over choices of z restricted to any (closed) subinterval of $({ID}_F^{\ast }-\gamma ,{ID}_F^{\ast }+\gamma )$ must occur at opposite endpoints of the subinterval. As in the proof of Lemma 1,

$$\begin{array}{ccc}\hfill D_{min}(t,w,ϵ)\le & {\displaystyle \psi \left(t,w,{ID}_F\left(t\right)\right)}& \le D_{max}(t,w,ϵ)\hfill \\ \hfill \mathrm{and}{\int }_0^w{D}_{min}(t,w,ϵ)\mathrm{d}t\le & {\displaystyle {\int }_0^w\psi \left(t,w,{ID}_F\left(t\right)\right)\mathrm{d}t}& \le {\int }_0^w{D}_{max}(t,w,ϵ)\mathrm{d}t,\hfill \end{array}$$

where

$$\begin{array}{ccc}\hfill D_{min}(t,w,ϵ)& \triangleq & min\left\{\psi \left(t,w,{ID}_F^{\ast }-ϵ\right),\psi \left(t,w,{ID}_F^{\ast }+ϵ\right)\right\},\hfill \\ \hfill D_{max}(t,w,ϵ)& \triangleq & max\left\{\psi \left(t,w,{ID}_F^{\ast }-ϵ\right),\psi \left(t,w,{ID}_F^{\ast }+ϵ\right)\right\}.\hfill \end{array}$$

Since $\psi (t,w,z)$ is also continuously partially differentiable with respect to z over the range $({ID}_F^{\ast }-\gamma ,{ID}_F^{\ast }+\gamma )$,

$$\begin{array}{ccc}\hfill \underset{ϵ\to 0^{+}}{lim}{D}_{min}(t,w,ϵ)=& {\displaystyle \underset{ϵ\to 0^{+}}{lim}{D}_{max}(t,w,ϵ)}& =\psi \left(t,w,{ID}_F^{\ast }\right)\mathrm{and}\hfill \\ \hfill \underset{\stackrel{ϵ\to 0^{+}}{w<ϵ}}{lim}{\int }_0^w{D}_{min}(t,w,ϵ)\mathrm{d}t=& {\displaystyle \underset{\stackrel{ϵ\to 0^{+}}{w<ϵ}}{lim}{\int }_0^w{D}_{max}(t,w,ϵ)\mathrm{d}t}& =\underset{w\to 0^{+}}{lim}{\int }_0^w\psi \left(t,w,{ID}_F^{\ast }\right)\mathrm{d}t.\hfill \end{array}$$

It therefore follows from the squeeze theorem for integrals that

$$\begin{array}{c}\hfill \underset{w\to 0^{+}}{lim}{\int }_0^w\psi \left(t,w,{ID}_F\left(t\right)\right)\mathrm{d}t=\underset{w\to 0^{+}}{lim}{\int }_0^w\psi \left(t,w,{ID}_F^{\ast }\right)\mathrm{d}t,\end{array}$$

whenever the right-hand limit exists or diverges. □

6. Derivation of the Limits of Tail Measures

In this section, we will see how the three substitution lemmas can be applied to the limits of tail measures of entropy, divergence or distance, so as to produce formulations that depend only on the local intrinsic dimensions of the functions involved. All three lemmas require that the integral function be monotone with respect to small variations in the term that is targeted for substitution. In the discussion, we choose two tail measures as running examples: the tail KL divergence and the second tail Wasserstein distance ($p=2$).

6.1. Handling Derivatives of Smooth Growth Functions

In the case of the tail KL divergence, Theorem 1 allows us to substitute out the first derivatives $F_w^{\prime }$ and $G_w^{\prime }$ for the functions $F_w$ and $G_w$:

$$\mathrm{KL}(F;G,w)={\int }_0^w{F}_w^{\prime }\left(t\right)ln\frac{F_w^{\prime }\left(t\right)}{G_w^{\prime }\left(t\right)}\mathrm{d}t={\int }_0^w\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}ln\frac{{ID}_F\left(t\right)F_w\left(t\right)}{{ID}_G\left(t\right)G_w\left(t\right)}\mathrm{d}t.$$

6.2. Substitution of LID Functions by Constants

In the limit of the tail KL divergence, the functions ${ID}_F\left(t\right)$ and ${ID}_G\left(t\right)$ can be replaced by the constants ${ID}_F^{\ast }$ and ${ID}_G^{\ast }$, respectively, through three successive applications of Lemma 3. To verify that the monotonicity condition of the Lemma is satisfied, we choose one of the terms and replace it by a new variable, z:

$$\underset{w\to 0^{+}}{lim}\mathrm{KL}(F;G,w)=\underset{w\to 0^{+}}{lim}{\int }_0^w\frac{z{F}_w\left(t\right)}{t}ln\frac{{ID}_F\left(t\right)F_w\left(t\right)}{{ID}_G\left(t\right)G_w\left(t\right)}\mathrm{d}t.$$

For any fixed values of t and w, it is easy to see that the integrand is locally monotone in the vicinity of $z={ID}_F\left(t\right)$—here, if $ln\frac{{ID}_F\left(t\right)F_w\left(t\right)}{{ID}_G\left(t\right)G_w\left(t\right)}$ is positive, a small increase in z (above the value ${ID}_F\left(t\right)$) would result in an increase in the value of the integrand, and a small decrease would cause the integrand to decrease. If instead the logarithmic factor were negative, an increase in z would result in a decrease in the value of the integrand. Either way, the integrand would be monotone in the vicinity of $z={ID}_F\left(t\right)$ at each fixed value of t and w. Its monotonicity condition thus being satisfied, Lemma 3 allows the targeted instance of ${ID}_F\left(t\right)$ to be substituted by ${ID}_F^{\ast }$:

$$\underset{w\to 0^{+}}{lim}\mathrm{KL}(F;G,w)=\underset{w\to 0^{+}}{lim}{\int }_0^w\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}ln\frac{{ID}_F\left(t\right)F_w\left(t\right)}{{ID}_G\left(t\right)G_w\left(t\right)}\mathrm{d}t.$$

Similarly, it can be verified that the new integrand is monotone in each of the remaining two factors ${ID}_F\left(t\right)$ and ${ID}_G\left(t\right)$; consequently, they too can be substituted by ${ID}_F^{\ast }$ and ${ID}_G^{\ast }$, one at a time, to yield

$$\underset{w\to 0^{+}}{lim}\mathrm{KL}(F;G,w)=\underset{w\to 0^{+}}{lim}{\int }_0^w\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}ln\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{{ID}_G^{\ast }{G}_w\left(t\right)}\mathrm{d}t.$$

6.3. Elimination of Tail-Conditioned Smooth Growth Functions

Now that the tail KL divergence has been reformulated in terms of the tail-conditioned smooth growth functions $F_w$ and $G_w$, these two functions can be substituted out via three successive applications of Lemma 1, so as to obtain the limit of an integral involving only the variable of integration t, and the constants w, ${ID}_F$ and ${ID}_G$:

$$\underset{w\to 0^{+}}{lim}\mathrm{KL}(F;G,w)=\underset{w\to 0^{+}}{lim}{\int }_0^w\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}ln\left[\frac{{ID}_F^{\ast }}{{ID}_G^{\ast }}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }-{ID}_G^{\ast }}\right]\mathrm{d}t.$$

As in the previous step in which ${ID}_F\left(t\right)$ and ${ID}_G\left(t\right)$ were substituted out, the monotonicity conditions of Lemma 1 can easily be verified.

Now that the integral involves only constants and the variable t, it can be solved straightforwardly using the integration-by-parts technique, yielding

$$\underset{w\to 0^{+}}{lim}\mathrm{KL}(F;G,w)=\underset{w\to 0^{+}}{lim}\left(\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}-ln\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}-1\right)=\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}-ln\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}-1.$$

6.4. Elimination of the Inverses of Tail-Conditioned Smooth Growth Functions

We now turn our attention to the limit of the tail Wasserstein distance for the case $p=2$. Using Lemma 2, the inverse functions $F_w^{-1}$ and $G_w^{-1}$ can be substituted out, provided that the monotonicity requirements are satisfied. However, immediate application of the lemma to $F_w^{-1}\left(u\right)$ or $G_w^{-1}\left(u\right)$ does not necessarily work—to see this, consider substituting $F_w^{-1}\left(u\right)$ by the new variable z.

$${\mathrm{WD}}_2(F;G,w)=\sqrt{{\int }_0^1{\left(F_w^{-1}\left(u\right)-G_w^{-1}\left(u\right)\right)}^2\mathrm{d}u}=\sqrt{{\int }_0^1{\left(z-G_w^{-1}\left(u\right)\right)}^2\mathrm{d}u}.$$

Clearly, the integrand is not necessarily monotone in z in the vicinity of those values of the integration variable u where $G_w^{-1}\left(u\right)=z$.

Instead, we expand the squared difference and apply Lemma 3 to each of the resulting four occurrences of $F_w^{-1}$ and $G_w^{-1}$, one by one. By way of illustration, we consider substitution by z for the factor of $F_w^{-1}\left(u\right)$ in the cross term:

$$\begin{array}{ccc}\hfill \underset{w\to 0^{+}}{lim}{\left({\mathrm{WD}}_2(F;G,w)\right)}^2& =& \underset{w\to 0^{+}}{lim}{\int }_0^1{\left(F_w^{-1}\left(u\right)-G_w^{-1}\left(u\right)\right)}^2\mathrm{d}u\hfill \\ & =& \underset{w\to 0^{+}}{lim}{\int }_0^1{\left(F_w^{-1}\left(u\right)\right)}^2-2F_w^{-1}\left(u\right)G_w^{-1}\left(u\right)+{\left(G_w^{-1}\left(u\right)\right)}^2\mathrm{d}u\hfill \\ & =& \underset{w\to 0^{+}}{lim}{\int }_0^1{\left(F_w^{-1}\left(u\right)\right)}^2-2z·G_w^{-1}\left(u\right)+{\left(G_w^{-1}\left(u\right)\right)}^2\mathrm{d}u.\hfill \end{array}$$

With respect to small variations in the variable z about the value $F_w^{-1}\left(u\right)$, noting that $G_w^{-1}$ is always non-negative, the integrand is easily seen to be monotone in z when $G_w^{-1}\left(u\right)$ is non-zero: for any increase in z, the value of the integrand decreases, and for any decrease in z, the value of the integrand increases. Lemma 2 can therefore be applied, producing

$$\begin{array}{ccc}\hfill \underset{w\to 0^{+}}{lim}{\left({\mathrm{WD}}_2(F;G,w)\right)}^2& =& \underset{w\to 0^{+}}{lim}{\int }_0^1{\left(F_w^{-1}\left(u\right)\right)}^2-2w{u}^{\frac{1}{{ID}_F^{\ast }}}·G_w^{-1}\left(u\right)+{\left(G_w^{-1}\left(u\right)\right)}^2\mathrm{d}u.\hfill \end{array}$$

After three more applications of Lemma 2, followed by taking the square root of the integral, we obtain

$$\begin{array}{ccc}\hfill \underset{w\to 0^{+}}{lim}{\mathrm{WD}}_2(F;G,w)& =& \underset{w\to 0^{+}}{lim}{\left({\int }_0^1{w}^2{u}^{\frac{2}{{ID}_F^{\ast }}}-2w^2{u}^{\frac{1}{{ID}_F^{\ast }}+\frac{1}{{ID}_G^{\ast }}}+w^2{u}^{\frac{2}{{ID}_G^{\ast }}}\mathrm{d}u\right)}^{\frac{1}{2}}\hfill \\ & =& \underset{w\to 0^{+}}{lim}w·{\left(\frac{1}{\frac{2}{{ID}_F^{\ast }}+1}-\frac{2}{\frac{1}{{ID}_F^{\ast }}+\frac{1}{{ID}_G^{\ast }}+1}+\frac{1}{\frac{2}{{ID}_G^{\ast }}+1}\right)}^{\frac{1}{2}}=0.\hfill \end{array}$$

6.5. Normalization

Even though the limit of the second tail Wasserstein distance is zero and therefore uninteresting, we observe that by normalizing it by the tail length w, we arrive at a more useful result:

$$\begin{array}{ccc}\hfill \underset{w\to 0^{+}}{lim}\frac{1}{w}{\mathrm{WD}}_2(F;G,w)& =& {\left(\frac{1}{\frac{2}{{ID}_F^{\ast }}+1}-\frac{2}{\frac{1}{{ID}_F^{\ast }}+\frac{1}{{ID}_G^{\ast }}+1}+\frac{1}{\frac{2}{{ID}_G^{\ast }}+1}\right)}^{\frac{1}{2}}.\hfill \end{array}$$

In general, reweighting by a power of w may be required to expose a relationship between the tail limit of an entropy measure or divergence and an expression in terms of the local intrinsic dimensions of the functions involved. Since local intrinsic dimension is a unitless quantity, in order to establish a non-trivial formulation solely in terms of LID values, any tail measure whose values are not unitless will generally require some form of normalization.

6.6. Summary of Results

Table 1 provides a summary of results. All the results stated in Table 1 can be derived either using the techniques outlined earlier in this section, or through direct substitution of another result in the table. The derivations are outlined in

Table 2. Derivations of asymptotic relationships between tail entropy variants and local intrinsic dimensionality. Each step shows the equivalences between the formulations when w is allowed to tend to zero. In the comments column, for each step of the derivation, the lemmas invoked are stated, as well as any additional assumptions made. If a normalization other weighting is needed to avoid divergence, or convergence to a constant (independent of F), the details are shown in a comment in the final step. In all cases, F is assumed to be a smooth growth function.

Tail Measure	Derivation Steps	Comments
Entropy	$\mathrm{H}(F,w)$ → $-{\int }_0^w{F}_w^{\prime }\left(t\right)ln{F}_w^{\prime }\left(t\right)\mathrm{d}t$
	→ $-{\int }_0^w\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}ln\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}\mathrm{d}t$	using Theorem 1
	→ $-{\int }_0^w\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}ln\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}\mathrm{d}t$	using Lemma 3
	→ $-{\int }_0^w\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}ln\left[\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}\right]\mathrm{d}t$	using Lemma 1
	→ $1-\frac{1}{{ID}_F^{\ast }}-ln\frac{{ID}_F^{\ast }}{w}$	no reweighting
Varentropy	$\mathrm{VarH}(F,w)$ → ${\int }_0^w{F}_w^{\prime }\left(t\right){ln}^2{F}_w^{\prime }\left(t\right)\mathrm{d}t-{\left({\int }_0^w{F}_w^{\prime }\left(t\right)ln{F}_w^{\prime }\left(t\right)\mathrm{d}t\right)}^2$
	→ ${\int }_0^w\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}{ln}^2\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}\mathrm{d}t-{\left({\int }_0^w\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}ln\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}\mathrm{d}t\right)}^2$	using Theorem 1
	→ ${\int }_0^w\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}{ln}^2\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}\mathrm{d}t-{\left({\int }_0^w\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}ln\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}\mathrm{d}t\right)}^2$	using Lemma 3
	→ ${\int }_0^w\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}{ln}^2\left[\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}\right]\mathrm{d}t-{\left({\int }_0^w\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}ln\left[\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}\right]\mathrm{d}t\right)}^2$	using Lemma 1
	→ ${\left(1-\frac{1}{{ID}_F^{\ast }}\right)}^2$
q-Entropy	${\mathrm{H}}_q(F,w)$ → $\frac{1}{q-1}{\int }_0^w{F}_w^{\prime }\left(t\right)-{\left(F_w^{\prime }\left(t\right)\right)}^q\mathrm{d}t$	$q>1$
	→ $\frac{1}{q-1}{\int }_0^w\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}-{\left(\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}\right)}^q\mathrm{d}t$	using Theorem 1
	→ $\frac{1}{q-1}{\int }_0^w\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}-{\left(\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}\right)}^q\mathrm{d}t$	using Lemma 3
	→ $\frac{1}{q-1}{\int }_0^w\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}-{\left(\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}\right)}^q\mathrm{d}t$	using Lemma 1
	→ $\frac{1}{q-1}\left(1-\frac{1}{w^{q-1}}·\frac{{\left({ID}_F^{\ast }\right)}^q}{q{ID}_F^{\ast }-q+1}\right)$
Cumulative Entropy	$\mathrm{cH}(F,w)$ → $-{\int }_0^w{F}_w\left(t\right)ln{F}_w\left(t\right)\mathrm{d}t$
	→ $-{\int }_0^w{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}ln{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}\mathrm{d}t$	using Lemma 1
	→ $w\frac{{ID}_F^{\ast }}{{({ID}_F^{\ast }+1)}^2}$	weight by $\frac{1}{w}$
Cumulative q-Entropy	${\mathrm{cH}}_q(F,w)$ → $\frac{1}{q-1}{\int }_0^w{F}_w\left(t\right)-{\left(F_w\left(t\right)\right)}^q\mathrm{d}t$	$q\ne 1$
	→ $\frac{1}{q-1}{\int }_0^w{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}-{\left(\frac{t}{w}\right)}^{q{ID}_F^{\ast }}\mathrm{d}t$	using Lemma 1
	→ $w\frac{{ID}_F^{\ast }}{({ID}_F^{\ast }+1)(q{ID}_F^{\ast }+1)}$	weight by $\frac{1}{w}$
Entropy Power	$\mathrm{HP}(F,w)$ → $exp\left(\mathrm{H}(F,w)\right)$
	→ $exp\left(1-\frac{1}{{ID}_F^{\ast }}-ln\frac{{ID}_F^{\ast }}{w}\right)$	by substitution
	→ $w\frac{1}{{ID}_F^{\ast }}exp\left(1-\frac{1}{{ID}_F^{\ast }}\right)$	weight by $\frac{1}{w}$
q-Entropy Power	${\mathrm{HP}}_q(F,w)$ → ${\left[1+(1-q){\mathrm{H}}_q(F,w)\right]}^{\frac{1}{1-q}}$	$q\ne 1$
	→ ${\left(1+(1-q)·\frac{1}{q-1}\left[1-\frac{1}{w^{q-1}}·\frac{{\left({ID}_F^{\ast }\right)}^q}{q{ID}_F^{\ast }-q+1}\right]\right)}^{\frac{1}{1-q}}$	by substitution
	→ $w{\left(\frac{{\left({ID}_F^{\ast }\right)}^q}{q{ID}_F^{\ast }-q+1}\right)}^{\frac{1}{1-q}}$	weight by $\frac{1}{w}$

(tail entropy variants),

Table 3. Derivations of asymptotic relationships between tail divergences and local intrinsic dimensionality. Each step shows the equivalences between the formulations when w is allowed to tend to zero. In the comments column, for each step of the derivation, the lemmas invoked are stated, as well as any additional assumptions made. If a normalization or weighting is needed, the details are shown in a comment in the final step. In all cases, F and G are assumed to be smooth growth functions.

Tail Measure	Derivation Steps	Comments
Cross Entropy	$\mathrm{XH}(F;G,w)$ → $-{\int }_0^w{F}_w^{\prime }\left(t\right)ln{G}_w^{\prime }\left(t\right)\mathrm{d}t$
	→ $-{\int }_0^w\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}ln\frac{{ID}_G\left(t\right)G_w\left(t\right)}{t}\mathrm{d}t$	using Theorem 1
	→ $-{\int }_0^w\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}ln\frac{{ID}_G^{\ast }{G}_w\left(t\right)}{t}\mathrm{d}t$	using Lemma 3
	→ $-{\int }_0^w\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}ln\left[\frac{{ID}_G^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_G^{\ast }}\right]\mathrm{d}t$	using Lemma 1
	→ $\frac{{ID}_G^{\ast }-1}{{ID}_F^{\ast }}-ln\frac{{ID}_G^{\ast }}{w}$	no reweighting
Cross Entropy Power	$\mathrm{XHP}(F;G,w)$ → $exp\left(\mathrm{XH}(F;G,w)\right)$
	→ $exp\left(\frac{{ID}_G^{\ast }-1}{{ID}_F^{\ast }}-ln\frac{{ID}_G^{\ast }}{w}\right)$	by substitution
	→ $w\frac{1}{{ID}_G^{\ast }}exp\left(\frac{{ID}_G^{\ast }-1}{{ID}_F^{\ast }}\right)$	weight by $\frac{1}{w}$
KL Divergence	$\mathrm{KL}(F;G,w)$ → ${\int }_0^w{F}_w^{\prime }\left(t\right)ln\frac{F_w^{\prime }\left(t\right)}{G_w^{\prime }\left(t\right)}\mathrm{d}t$
	→ ${\int }_0^w\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}ln\frac{{ID}_F\left(t\right)F_w\left(t\right)}{{ID}_G\left(t\right)G_w\left(t\right)}\mathrm{d}t$	using Theorem 1
	→ ${\int }_0^w\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}ln\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{{ID}_G^{\ast }{G}_w\left(t\right)}\mathrm{d}t$	using Lemma 3
	→ ${\int }_0^w\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}ln\left[\frac{{ID}_F^{\ast }}{{ID}_G^{\ast }}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }-{ID}_G^{\ast }}\right]\mathrm{d}t$	using Lemma 1
	→ $\rho -ln\rho -1$	$\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$
JS Divergence	$\mathrm{JS}(F;G,w)$ → $\frac{1}{2}\left(\mathrm{KL}(F;M,w)+\mathrm{KL}(G;M,w)\right)$	$M\left(t\right)=\frac{1}{2}\left(F\left(t\right)+G\left(t\right)\right)$
	→ $\frac{1}{2}\left(\frac{{ID}_M^{\ast }}{{ID}_F^{\ast }}-ln\frac{{ID}_M^{\ast }}{{ID}_F^{\ast }}-1+\frac{{ID}_M^{\ast }}{{ID}_G^{\ast }}-ln\frac{{ID}_M^{\ast }}{{ID}_G^{\ast }}-1\right)$	${ID}_M^{\ast }=min\{{ID}_F^{\ast },{ID}_G^{\ast }\}$
	→ $\frac{1}{2}\left(\frac{{ID}_M^{\ast }}{B}+\frac{{ID}_M^{\ast }}{{ID}_M^{\ast }}-ln\frac{{ID}_M^{\ast }}{B}-ln\frac{{ID}_M^{\ast }}{{ID}_M^{\ast }}-2\right)$	let $B=max\{{ID}_F^{\ast },{ID}_G^{\ast }\}$
	→ $\frac{1}{2}\left(\tau -ln\tau -1\right)$	$\tau =min\left\{\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }},\frac{{ID}_F^{\ast }}{{ID}_G^{\ast }}\right\}$

(tail divergence variants),

Table 4. Derivations of asymptotic relationships between tail distances and local intrinsic dimensionality. Each step shows the equivalences between the formulations when w is allowed to tend to zero. In the comments column, for each step of the derivation, the lemmas invoked are stated, as well as any additional assumptions made. For each tail distance, the first step of the derivations shows an expansion by which the monotonicity of each factor can be verified. If a normalization or weighting is needed, the details are shown in a comment in the final step. In all cases, F and G are assumed to be smooth growth functions.

Tail Measure	Derivation Steps	Comments
L2 Distance	$\mathrm{L}2\mathrm{D}(F;G,w)$ → ${\int }_0^w{\left(F_w^{\prime }\left(t\right)-G_w^{\prime }\left(t\right)\right)}^2\mathrm{d}t$
	→ ${\int }_0^w{\left(\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}-\frac{{ID}_G\left(t\right)G_w\left(t\right)}{t}\right)}^2\mathrm{d}t$	using Theorem 1
	→ ${\int }_0^w{\left(\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}\right)}^2-2\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}·\frac{{ID}_G^{\ast }{G}_w\left(t\right)}{t}+{\left(\frac{{ID}_G^{\ast }{G}_w\left(t\right)}{t}\right)}^2\mathrm{d}t$	using Lemma 3
	→ ${\int }_0^w\frac{{\left({ID}_F^{\ast }\right)}^2}{t^2}{\left(\frac{t}{w}\right)}^{2{ID}_F^{\ast }}-\frac{2{ID}_F^{\ast }{ID}_G^{\ast }}{t^2}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }+{ID}_G^{\ast }}+\frac{{\left({ID}_G^{\ast }\right)}^2}{t^2}{\left(\frac{t}{w}\right)}^{2{ID}_G^{\ast }}\mathrm{d}t$	using Lemma 1
	→ $\frac{1}{w}·\frac{{\left({ID}_F^{\ast }-{ID}_G^{\ast }\right)}^2}{2({ID}_F^{\ast }+{ID}_G^{\ast }-1)}\left[1+\frac{1}{(2{ID}_F^{\ast }-1)(2{ID}_G^{\ast }-1)}\right]$	weight by w
Hellinger Distance	$\mathrm{HD}(F;G,w)$ → $\sqrt{\frac{1}{2}{\int }_0^w{\left(\sqrt{F_w^{\prime }\left(t\right)}-\sqrt{G_w^{\prime }\left(t\right)}\right)}^2\mathrm{d}t}$
	→ $\sqrt{\frac{1}{2}{\int }_0^w{\left(\sqrt{\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}}-\sqrt{\frac{{ID}_G\left(t\right)G_w\left(t\right)}{t}}\right)}^2\mathrm{d}t}$	using Theorem 1
	→ $\sqrt{\frac{1}{2}{\int }_0^w\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}-2\frac{\sqrt{{ID}_F^{\ast }{F}_w\left(t\right)·{ID}_G^{\ast }{G}_w\left(t\right)}}{t}+\frac{{ID}_G^{\ast }{G}_w\left(t\right)}{t}\mathrm{d}t}$	using Lemma 3
	→ $\sqrt{{\int }_0^w\frac{1}{2t}\left({ID}_F^{\ast }{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}-2\sqrt{{ID}_F^{\ast }{ID}_G^{\ast }}{\left(\frac{t}{w}\right)}^{({ID}_F^{\ast }+{ID}_G^{\ast })/2}+{ID}_G^{\ast }{\left(\frac{t}{w}\right)}^{{ID}_G^{\ast }}\right)\mathrm{d}t}$	using Lemma 1
	→ $\frac{\left|1-\sqrt{\rho }\right|}{\sqrt{1+\rho }}$	$\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$
${\chi }^2$-Divergence	${\chi }^2\mathrm{D}(F;G,w)$ → ${\int }_0^w\frac{{\left(F_w^{\prime }\left(t\right)-G_w^{\prime }\left(t\right)\right)}^2}{G_w^{\prime }\left(t\right)}\mathrm{d}t$
	→ ${\int }_0^w{\left(\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}-\frac{{ID}_G\left(t\right)G_w\left(t\right)}{t}\right)}^2\frac{t}{{ID}_G\left(t\right)G_w\left(t\right)}\mathrm{d}t$	using Theorem 1
	→ ${\int }_0^w\left[{\left(\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}\right)}^2-2\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}·\frac{{ID}_G^{\ast }{G}_w\left(t\right)}{t}+{\left(\frac{{ID}_G^{\ast }{G}_w\left(t\right)}{t}\right)}^2\right]\frac{t}{{ID}_G^{\ast }{G}_w\left(t\right)}\mathrm{d}t$	using Lemma 3
	→ ${\int }_0^w\left[\frac{{\left({ID}_F^{\ast }\right)}^2}{t^2}{\left(\frac{t}{w}\right)}^{2{ID}_F^{\ast }}-\frac{2{ID}_F^{\ast }{ID}_G^{\ast }}{t^2}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }+{ID}_G^{\ast }}+\frac{{\left({ID}_G^{\ast }\right)}^2}{t^2}{\left(\frac{t}{w}\right)}^{2{ID}_G^{\ast }}\right]\frac{t}{{ID}_G^{\ast }}{\left(\frac{w}{t}\right)}^{{ID}_G^{\ast }}\mathrm{d}t$	using Lemma 1
	→ $\frac{{\left(1-\rho \right)}^2}{\rho (2-\rho )}$	$\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$
$\alpha$-Divergence	$\alpha \mathrm{D}(F;G,w)$ → $\frac{1}{\alpha (1-\alpha )}{\int }_0^w\alpha F_w^{\prime }\left(t\right)+(1-\alpha )G_w^{\prime }\left(t\right)-{\left(F_w^{\prime }\left(t\right)\right)}^{\alpha }{\left(G_w^{\prime }\left(t\right)\right)}^{1-\alpha }\mathrm{d}t$
	→ $\frac{1}{\alpha (1-\alpha )}{\int }_0^w\frac{\alpha {ID}_F\left(t\right)F_w\left(t\right)}{t}+\frac{(1-\alpha ){ID}_G\left(t\right)G_w\left(t\right)}{t}-{\left(\frac{{ID}_F\left(t\right)F_w\left(t\right)}{t}\right)}^{\alpha }{\left(\frac{{ID}_G\left(t\right)G_w\left(t\right)}{t}\right)}^{1-\alpha }\mathrm{d}t$	using Theorem 1
	→ $\frac{1}{\alpha (1-\alpha )}{\int }_0^w\frac{\alpha {ID}_F^{\ast }{F}_w\left(t\right)}{t}+\frac{(1-\alpha ){ID}_G^{\ast }{G}_w\left(t\right)}{t}-{\left(\frac{{ID}_F^{\ast }{F}_w\left(t\right)}{t}\right)}^{\alpha }{\left(\frac{{ID}_G^{\ast }{G}_w\left(t\right)}{t}\right)}^{1-\alpha }\mathrm{d}t$	using Lemma 3
	→ $\frac{1}{\alpha (1-\alpha )}{\int }_0^w\frac{\alpha {ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}+\frac{(1-\alpha ){ID}_G^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_G^{\ast }}-\frac{{\left({ID}_F^{\ast }\right)}^{\alpha }{\left({ID}_G^{\ast }\right)}^{1-\alpha }}{t}{\left(\frac{t}{w}\right)}^{\alpha {ID}_F^{\ast }+(1-\alpha ){ID}_G^{\ast }}\mathrm{d}t$	using Lemma 1
	→ $\frac{1}{\alpha (1-\alpha )}\left(1-\frac{{\left({ID}_F^{\ast }\right)}^{\alpha }{\left({ID}_G^{\ast }\right)}^{1-\alpha }}{\alpha {ID}_F^{\ast }+(1-\alpha ){ID}_G^{\ast }}\right)$
	→ $\frac{1}{\alpha (1-\alpha )}\left(1-\frac{1}{\alpha {\rho }^{\alpha -1}+(1-\alpha ){\rho }^{\alpha }}\right)$	$\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$

(tail distance variants) and

Table 5. Derivations of asymptotic relationships between tail Wasserstein distances and local intrinsic dimensionality. Each step shows the equivalences between the formulations when w is allowed to tend to zero. In the comments column, for each step of the derivation, the lemmas invoked are stated, as well as any additional assumptions made. Normalization details are shown in a comment in the final step. In all cases, F and G are assumed to be invertible smooth growth functions.

Tail Measure	Derivation Steps	Comments
Wasserstein Distance	${\mathrm{WD}}_2(F;G,w)$ → $\sqrt{{\int }_0^1{\left(F_w^{-1}\left(u\right)-G_w^{-1}\left(u\right)\right)}^2\mathrm{d}u}$
	→ $\sqrt{{\int }_0^1{\left(F_w^{-1}\left(u\right)\right)}^2-2F_w^{-1}\left(u\right)·G_w^{-1}\left(u\right)+{\left(G_w^{-1}\left(u\right)\right)}^2\mathrm{d}u}$
$p=2$	→ $\sqrt{{\int }_0^1{w}^2{u}^{\frac{2}{{ID}_F^{\ast }}}-2w^2{u}^{\frac{1}{{ID}_F^{\ast }}+\frac{1}{{ID}_G^{\ast }}}+w^2{u}^{\frac{2}{{ID}_G^{\ast }}}\mathrm{d}u}$	using Lemma 2
	→ $w\sqrt{\frac{1}{\frac{2}{{ID}_F^{\ast }}+1}-\frac{2}{\frac{1}{{ID}_F^{\ast }}+\frac{1}{{ID}_G^{\ast }}+1}+\frac{1}{\frac{2}{{ID}_G^{\ast }}+1}}$	weight by $\frac{1}{w}$
Wasserstein Distance	${\mathrm{WD}}_p(F;G,w)$ → ${\left({\int }_0^1{\left(F_w^{-1}\left(u\right)-G_w^{-1}\left(u\right)\right)}^p\mathrm{d}u\right)}^{\frac{1}{p}}$
$p\in \mathbb{N}$,   p even	→ ${\left({\int }_0^1{\sum }_{j=0}^p{(-1)}^j\left(\stackrel{p}{j}\right){\left(F_w^{-1}\left(u\right)\right)}^{p-j}{\left(G_w^{-1}\left(u\right)\right)}^j\mathrm{d}u\right)}^{\frac{1}{p}}$
	→ ${\left({\int }_0^1{\sum }_{j=0}^p{(-1)}^j\left(\stackrel{p}{j}\right){\left(w{u}^{\frac{1}{{ID}_F^{\ast }}}\right)}^{p-j}{\left(w{u}^{\frac{1}{{ID}_G^{\ast }}}\right)}^j\mathrm{d}u\right)}^{\frac{1}{p}}$	using Lemma 2
	→ $w{\left({\sum }_{j=0}^p\frac{{(-1)}^j\left(\stackrel{p}{j}\right)}{(p-j)·{\left({ID}_F^{\ast }\right)}^{-1}+j·{\left({ID}_G^{\ast }\right)}^{-1}+1}\right)}^{\frac{1}{p}}$	weight by $\frac{1}{w}$

(tail Wasserstein distances). Most of these derivations are straightforward; however, for two of the tail measures, some clarifications are required.

Generally speaking, for the normalized tail Wasserstein distances with p non-integer or p odd (Table 5), Lemma 2 cannot be applied, due to the absolute value operation in the integrand. It may happen that the functions $F^{-1}\left(u\right)$ and $G^{-1}\left(u\right)$ may have crossing points for many (possibly even infinitely many) values of u between 0 and 1. At these values of u, $\left|F^{-1}\left(u\right)-G^{-1}\left(u\right)\right|=0$, and neither $\left|z-G^{-1}\left(u\right)\right|$ nor $\left|F^{-1}\left(u\right)-y\right|$ would be monotone in the vicinity of $z=F^{-1}\left(u\right)$ or $y=G^{-1}\left(u\right)$, as the case may be.

For the tail JS divergence (Table 3), the derivation relies on the fact that the LID of the sum (or average) of two non-negative smooth growth functions is the smaller of the two individual LID values. This is an implication of the fact that ${lim}_{t\to 0^{+}}\frac{V\left(t\right)}{W\left(t\right)}=0$ whenever the smooth growth functions $V\left(t\right)$ and $W\left(t\right)$ have $0<{ID}_W^{\ast }<{ID}_V^{\ast }$ (see [84] for more details). Accordingly, if ${ID}_F^{\ast }\ne {ID}_G^{\ast }$, then the function (F or G) with smaller LID value must have the same LID value as the average function $M\left(t\right)=\frac{F\left(t\right)+G\left(t\right)}{2}$, and the other function (G or F) must have LID value equal to the maximum of the two. From these observations, the derivation can be seen to hold.

The result for the limit of the tail KL divergence has an interesting interpretation in light of the so-called Itakura–Saito (IS) divergence (or distance) [85]:

$$d_{\mathrm{IS}}\left(\mathbf{x}\right|\mathbf{y})=\sum _{i=1}^n\left(\frac{x_i}{y_i}-ln\frac{x_i}{y_i}-1\right).$$

As the tail boundary w tends to 0, the tail KL divergence between smooth functions F and G tends to the (univariate) IS divergence between their associated LID values ${ID}_G^{\ast }$ and ${ID}_F^{\ast }$:

$$\underset{w\to 0^{+}}{lim}\mathrm{KL}(F;G,w)=\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}-ln\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}-1=d_{\mathrm{IS}}\left({ID}_G^{\ast }\right|{ID}_F^{\ast }).$$

When F and G are interpreted as the CDFs of distance distributions, the shape parameters of the extreme-value-theoretic generalized pareto distributions (GPDs) that asymptotically characterize their lower tails are known to equal $-\frac{1}{{ID}_F^{\ast }}$ and $-\frac{1}{{ID}_G^{\ast }}$, respectively [40]. Since the ratio of these parameters is equal to (the reciprocal of) the ratio of LID values, the tail KL divergence between F and G can also be interpreted as tending to the IS divergence between GPD parameters.

The IS divergence is popular as an objective for matrix factorization of audio spectra [86], for assessing the loss of using entry $y_{i,j}$ to approximate a true entry $x_{i,j}$; more precisely, to approximate a matrix $\mathbf{V}$ by factorization $\mathbf{WH}$, the loss is ${\sum }_i{\sum }_j{d}_{\mathrm{IS}}\left({\left[\mathbf{V}\right]}_{ij}\right|{\left[\mathbf{WH}\right]}_{ij})$. The IS divergence is a convenient choice for this scenario due to its scale-free property ($d_{\mathrm{IS}}\left(\mathbf{x}\right|\mathbf{y})=d_{\mathrm{IS}}\left(\alpha \mathbf{x}\right|\alpha \mathbf{y})$ for any $\alpha \ne 0$), thus giving the same relative weight to both small and large values of $x_i$ and $y_i$, since they only appear as the ratio $\frac{x_i}{y_i}$. This is important for scenarios such as audio spectra, where the magnitudes of $x_i$ and $y_i$ can vary greatly.

The Itakura–Saito divergence falls into the family of so-called Bregman divergences (or distances) [87], which have a geometric interpretation as the difference between the value of a convex generator function at $\mathbf{x}$ on the one hand, and the value at $\mathbf{x}$ of a hyperplane function that is tangent to the generator curve at $\mathbf{y}$. Bregman divergences are a highly expressive family of distances with a wide range of applications [88]. For the IS divergence, the convex generator function is the negative logarithm $-{\sum }_{i=1}^{n}ln{x}_i$. Interestingly, the KL divergence is also a Bregman divergence, with its convex generator being the negative entropy function ${\sum }_{i=1}^n{x}_{i}ln{x}_i$ [89].

7. Extension to Multivariate Distributions

Thus far, our results have focused on a univariate scenario, wherein entropy and divergence variants were shown to be asymptotically equivalent to formulations involving the local intrinsic dimensionalities of smooth distributions of a single random variable. As discussed in Section 3, these results can be applied to distance-based analysis, through characterizations involving the LIDs of local (univariate) distance distributions induced by the overall (global) multivariate distribution. These characterizations are indirect, in that they do not explicitly involve (nor do they require) any knowledge of the underlying global distribution and its parameters. However, characterizations in terms of induced distance distributions may not be entirely satisfying when the nature of the global multivariate distribution is either known or assumed. In this section, we will assume that our domain $\mathcal{S}$ is the n-dimensional space ${\mathbb{R}}^n$ equipped with the Euclidean distance, $d(\mathbf{x},\mathbf{y})=\parallel \mathbf{x}-\mathbf{y}\parallel$. Within $\mathcal{S}$, we will also assume that we are given a data distribution $\mathcal{D}$ with probability density function $p:{\mathbb{R}}^n\to {\mathbb{R}}_{+}\cup 0$.

7.1. Multivariate Tail Distributions with Local Spherical Symmetry

Within the Euclidean domain, the challenge is to analyze distributions in terms of the probability measure captured within volumes associated with a distributional tail. However, unlike in univariate distributions, there is no universally accepted notion of ‘distributional tail’ for multivariate distributions. For our purposes, given a distance $r>0$, we define the tail of $\mathcal{D}$ of length r to be the region enclosed by the ball of radius r centered at the origin; that is, $\mathcal{B}\left(r\right)\triangleq \{\mathbf{x}\in {\mathbb{R}}^n:\parallel \mathbf{x}\parallel \le r\}$. The boundary of the tail is the $(n-1)$-dimensional surface area of $\mathcal{B}\left(r\right)$, which we denote by ${\mathcal{B}}^{\prime }\left(r\right)\triangleq \{\mathbf{x}\in {\mathbb{R}}^n:\parallel \mathbf{x}\parallel =r\}$.

To enable tractable analysis, we will assume that the PDF can be expressed in terms of a locally spherically symmetrical function. One example of where local spherical symmetry can be expected to hold is for a locally isotropic context. This is a common assumption for physical systems, including metals, glasses, fluids and polymers, for which the distribution locally surrounding a particle in the system does not have a directional preference.

Formally, we say that a density function f is locally spherically symmetrical within radius w if for all $\parallel \mathbf{x}\parallel \le w$, we have $f\left(\mathbf{x}\right)=f_{\ast }\left(r\right)$ for some univariate function $f_{\ast }$ where $r=\parallel \mathbf{x}\parallel$. For f to be locally spherically symmetrical, it suffices that $f\left(\mathbf{x}\right)$ be equal to $f\left(\mathbf{y}\right)$ whenever $0\le \parallel \mathbf{x}\parallel =\parallel \mathbf{y}\parallel \le r$. The assumption also implies the existence of a function $f_{\ast }$ for which $f\left(\mathbf{x}\right)=f_{\ast }\left(r\right)$, and therefore that f must be constant over all points of the sphere ${\mathcal{B}}^{\prime }\left(r\right)$.

The probability measure captured by $\mathcal{B}\left(r\right)$, which we denote by $F\left(r\right)$, is obtained through the integration of f over this ball:

$$F\left(r\right)\triangleq {\int }_{\mathcal{B}\left(r\right)}f\mathrm{d}\mathcal{B}\left(r\right).$$

It is not difficult to see that the univariate function F is simply the CDF of the distribution of distances to the origin induced by the global distribution $\mathcal{D}$. If F is differentiable over the tail interval $(0,r]$, then the integral of $F^{\prime }$ over this interval exists, and equals F:

$$\begin{array}{ccc}\hfill {\int }_{\mathcal{B}\left(r\right)}f\mathrm{d}\mathcal{B}\left(r\right)& =F\left(r\right)=& {\int }_0^r{F}^{\prime }\left(t\right)\mathrm{d}t.\hfill \end{array}$$

(2)

The derivative $F^{\prime }\left(r\right)$ can therefore be interpreted as the PDF of the radial distance distribution as measured from the origin.

For spherically symmetric distributions in Euclidean spaces, the multivariate density and radial density is related through a factor that depends on the surface area of spherical volumes. The formulae for the volume of an n-sphere and its $(n-1)$-dimensional surface area are given by

$$\begin{array}{ccc}\hfill V_n\left(r\right)& \triangleq & \frac{{\pi }^{n/2}}{\Gamma \left(\right(n/2)+1)}{r}^n\mathrm{and}\hfill \\ \hfill S_{n-1}\left(r\right)& \triangleq & \frac{2{\pi }^{n/2}}{\Gamma (n/2)}{r}^{n-1},\hfill \end{array}$$

respectively. $\Gamma$ is the common gamma function $\Gamma \left(n\right)=(n-1)!$ if n is a positive integer and $\Gamma (n+\frac{1}{2})=(n-\frac{1}{2})(n-\frac{3}{2})\dots \frac{1}{2}\sqrt{\pi }$ if n is a non negative integer. Furthermore, the volume and surface area have a simple relationship that allows for easy conversion between the two:

$$r·S_{n-1}\left(r\right)=n·V_n\left(r\right).$$

(3)

Lemma 4

([90]). Let $\mathbf{X}$ be an n-dimensional random vector that is spherically symmetric with a radial distribution $\mathcal{R}$. Then $\mathbf{X}$ has a density $f\left(\mathbf{x}\right)$ if and only if $\mathcal{R}$ has a density s and

$$s\left(r\right)=f\left(\mathbf{x}\right)·S_{n-1}\left(r\right).$$

If F is a smooth growth function that is locally spherically symmetric over $[0,r]$, Equation (2) and Lemma 4 together give us the following relationship between the radial density $F^{\prime }$ and the multivariate density f:

$$f\left(\mathbf{x}\right)=\frac{F^{\prime }(\parallel \mathbf{x}\parallel )}{S_{n-1}(\parallel \mathbf{x}\parallel )}$$

whenever $\parallel \mathbf{x}\parallel \le r$. Conditioning the distribution to the ball $\mathcal{B}\left(r\right)$, the tail distribution PDF becomes

$$f_r\left(\mathbf{x}\right)\triangleq \frac{f\left(\mathbf{x}\right)}{{\int }_{\mathcal{B}\left(w\right)}f\mathrm{d}\mathcal{B}\left(w\right)}=\frac{F^{\prime }(\parallel \mathbf{x}\parallel )}{S_{n-1}(\parallel \mathbf{x}\parallel )·F\left(r\right)}=\frac{F_r^{\prime }(\parallel \mathbf{x}\parallel )}{S_{n-1}(\parallel \mathbf{x}\parallel )}.$$

7.2. Multivariate Tail Entropy Variants

The aforementioned relationships between multivariate and radial densities can be immediately used to compute the various tail entropies for the locally spherically symmetric multivariate case. Useful background on evaluating radial integrals can be found in Baker [91]. For example, the multivariate Tail Entropy is

$$\begin{array}{ccc}\hfill \mathrm{H}(f,w)& \triangleq & -{\int }_{\mathcal{B}\left(w\right)}{f}_{w}ln{f}_w\mathrm{d}\mathcal{B}\left(w\right)\hfill \\ & =& -{\int }_0^w\left(\frac{F_w^{\prime }\left(t\right)}{S_{n-1}\left(t\right)}ln\frac{F_w^{\prime }\left(t\right)}{S_{n-1}\left(t\right)}\right)·S_{n-1}\left(t\right)\mathrm{d}t\hfill \\ & =& -{\int }_0^w{F}_w^{\prime }\left(t\right)ln\frac{F_w^{\prime }\left(t\right)}{S_{n-1}\left(t\right)}\mathrm{d}t.\hfill \end{array}$$

Although the multivariate formulation of Tail Entropy $\mathrm{H}(f,w)$ resembles that of the univariate formulation $\mathrm{H}(F,w)$, the two are not identical. Nevertheless, the multivariate formulation can still be simplified using the technical lemmas introduced in Section 5. In much the same way as for the univariate Tail Entropy Power, we can use Theorem 1 together with Lemmas 1 and 3 to determine the limit of $\mathrm{H}(f,w)$ as w tends to 0. Replacing $F_w^{\prime }\left(t\right)$ by $\frac{1}{t}{ID}_F\left(t\right)F_w\left(t\right)$, then ${ID}_F\left(t\right)$ by ${ID}_F^{\ast }$, and finally $F_w\left(t\right)$ by ${\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}$, we obtain

$$\begin{array}{ccc}\hfill \underset{w\to 0}{lim}\mathrm{H}(f,w)& =& \underset{w\to 0}{lim}-{\int }_0^w{F}_w^{\prime }\left(t\right)ln\frac{F_w^{\prime }\left(t\right)}{t^{n-1}{S}_{n-1}\left(1\right)}\mathrm{d}t\hfill \\ & =& \underset{w\to 0}{lim}-{\int }_0^w\frac{{ID}_F^{\ast }}{t}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}ln\left[\frac{{ID}_F^{\ast }}{t^n{S}_{n-1}\left(1\right)}{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}\right]\mathrm{d}t\hfill \\ & =& \underset{w\to 0}{lim}-{\int }_0^w\frac{{ID}_F^{\ast }}{w^{{ID}_F^{\ast }}}{t}^{{ID}_F^{\ast }-1}ln\left[\frac{{ID}_F^{\ast }}{w^{{ID}_F^{\ast }}{S}_{n-1}\left(1\right)}{t}^{{ID}_F^{\ast }-n}\right]\mathrm{d}t\hfill \\ & =& \underset{w\to 0}{lim}-{\int }_0^w\frac{{ID}_F^{\ast }}{w^{{ID}_F^{\ast }}}{t}^{{ID}_F^{\ast }-1}\left[ln\frac{{ID}_F^{\ast }}{w^{{ID}_F^{\ast }}{S}_{n-1}\left(1\right)}+({ID}_F^{\ast }-n)lnt\right]\mathrm{d}t.\hfill \end{array}$$

Solving the integral, and then using Equation (3) to convert the surface area factor $S_{n-1}$ to an expression involving the volume $V_n$, we eventually arrive at

$$\begin{array}{ccc}\hfill \underset{w\to 0}{lim}\mathrm{H}(f,w)& =& \underset{w\to 0}{lim}\left(1-\frac{n}{{ID}_F^{\ast }}-ln\frac{{ID}_F^{\ast }}{w^n{S}_{n-1}\left(1\right)}\right)\hfill \\ & =& \underset{w\to 0}{lim}\left(1-\frac{n}{{ID}_F^{\ast }}-ln\frac{{ID}_F^{\ast }}{w{S}_{n-1}\left(w\right)}\right)\hfill \\ & =& \underset{w\to 0}{lim}\left(1-\frac{n}{{ID}_F^{\ast }}-ln\frac{{ID}_F^{\ast }}{n}+ln{V}_n\left(w\right)\right),\hfill \end{array}$$

which diverges even when the Tail Entropy is reweighted by $V_n\left(w\right)$ (or indeed, by any other polynomial in w). However, the Tail Entropy Power, when normalized by $V_n\left(w\right)$, does converge to a strictly positive value:

$$\begin{array}{ccc}\hfill \underset{w\to 0}{lim}\frac{1}{V_n\left(w\right)}\mathrm{HP}(f,w)& \triangleq & \underset{w\to 0}{lim}\frac{1}{V_n\left(w\right)}exp\left(\mathrm{H}(f,w)\right)\hfill \\ & =& \underset{w\to 0}{lim}\frac{1}{V_n\left(w\right)}exp\left(1-\frac{n}{{ID}_F^{\ast }}-ln\frac{{ID}_F^{\ast }}{n}+ln{V}_n\left(w\right)\right)\hfill \\ & =& \frac{1}{\phi }exp\left(1-\frac{1}{\phi }\right),\mathrm{where}\phi =\frac{{ID}_F^{\ast }}{n}.\hfill \end{array}$$

As one might expect in the n-dimensional Euclidean setting, the (normalized asymptotic) multivariate Tail Entropy Power is maximized whenever ${ID}_F^{\ast }$, the local intrinsic dimensionality of the associated radial CDF F, is equal to n.

7.3. Multivariate Cumulative Tail Entropy

In the multivariate setting, cumulative entropy is defined in terms of the distributional tail, according to the notion laid out in Section 7.1. In place of the usual probability density $f\left(\mathbf{x}\right)$, the entropy function is applied to the probability measure associated with the ball centered at the origin with radius $\parallel \mathbf{x}\parallel$; that is, with

$$Pr[X\le \parallel \mathbf{x}\parallel ]\triangleq {\int }_{\mathcal{B}(\parallel \mathbf{x}\parallel )}f\mathrm{d}\mathcal{B}(\parallel \mathbf{x}\parallel )=F(\parallel \mathbf{x}\parallel ).$$

Note that since F takes the same value at $\mathbf{x}$ and $\mathbf{y}$ whenever $\parallel \mathbf{x}\parallel =\parallel \mathbf{y}\parallel$, the quantity $F(\parallel \mathbf{x}\parallel )$ is locally spherically symmetric even when the underlying density function f is not.

We can adapt the multivariate formulation of cumulative residual entropy that was originally proposed by Rao [56]. The multivariate Cumulative Tail Entropy, conditioned to a distributional tail of radius w, is expressed as a multivariate integral involving $F_w(\parallel \mathbf{x}\parallel )$, or as a radial integral involving $F_w$, as follows:

$$\begin{array}{ccc}\hfill \mathrm{cH}(f,w)& \triangleq & -{\int }_{\mathbf{x}\in \mathcal{B}\left(w\right)}{F}_w(\parallel \mathbf{x}\parallel )ln{F}_w(\parallel \mathbf{x}\parallel )\mathrm{d}\mathcal{B}\left(w\right)\hfill \\ & =& -{\int }_0^w\left(F_w\left(t\right)ln{F}_w\left(t\right)\right)·S_{n-1}\left(t\right)\mathrm{d}t.\hfill \end{array}$$

As in the treatment of the univariate tail entropies, we can use Lemma 1 to determine the limit of $\mathrm{cH}(f,w)$ as w tends to 0. Replacing $F_w\left(t\right)$ by ${\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}$,

$$\begin{array}{ccc}\hfill \underset{w\to 0}{lim}\mathrm{cH}(f,w)& =& \underset{w\to 0}{lim}-{\int }_0^w{t}^{n-1}{S}_{n-1}\left(1\right)F_w\left(t\right)ln{F}_w\left(t\right)\mathrm{d}t\hfill \\ & =& \underset{w\to 0}{lim}-{\int }_0^w{t}^{n-1}{S}_{n-1}\left(1\right){\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}ln{\left(\frac{t}{w}\right)}^{{ID}_F^{\ast }}\mathrm{d}t\hfill \\ & =& \underset{w\to 0}{lim}-{\int }_0^w\frac{S_{n-1}\left(1\right){ID}_F^{\ast }}{w^{{ID}_F^{\ast }}}·t^{{ID}_F^{\ast }+n-1}\left(lnt-lnw\right)\mathrm{d}t.\hfill \end{array}$$

Solving the integral, and then converting the surface area factor $S_{n-1}$ to a volume factor $V_n$ using Equation (3), we obtain

$$\begin{array}{ccc}\hfill \underset{w\to 0}{lim}\mathrm{cH}(f,w)& =& \underset{w\to 0}{lim}w{S}_{n-1}\left(w\right)·\frac{{ID}_F^{\ast }}{{({ID}_F^{\ast }+n)}^2}\hfill \\ & =& \underset{w\to 0}{lim}{V}_n\left(w\right)·\frac{\phi }{{(\phi +1)}^2},\mathrm{where}\phi =\frac{{ID}_F^{\ast }}{n}.\hfill \end{array}$$

Although the multivariate Cumulative Tail Entropy vanishes as the tail boundary w tends to zero, when normalized by the tail volume $V_n\left(w\right)$ it converges to a strictly positive value:

$$\begin{array}{ccc}\hfill \underset{w\to 0}{lim}\frac{1}{V_n\left(w\right)}\mathrm{cH}(f,w)& =& \frac{\phi }{{(\phi +1)}^2}.\hfill \end{array}$$

Again, as with the Normalized Tail Entropy Power, the (asymptotic) multivariate Tail Cumulative Entropy is maximized whenever $\phi =1$. That is, when ${ID}_F^{\ast }=n$.

7.4. Multivariate Tail Divergences

Several of the tail divergence measures, when considered in the multivariate setting under the assumptions of locally spherical symmetry, turn out to be identical to those of the radial (univariate) setting. As an example, consider the multivariate Tail KL Divergence, defined as

$$\begin{array}{ccc}\hfill \mathrm{KL}(f;g,w)& \triangleq & {\int }_{\mathcal{B}\left(w\right)}{f}_{w}ln\frac{f_w}{g_w}\mathrm{d}\mathcal{B}\left(w\right).\hfill \end{array}$$

Applying Lemma 4 and integrating radially over the tail, we see that

$$\begin{array}{ccc}\hfill \mathrm{KL}(f;g,w)& =& {\int }_0^w\left(\frac{F_w^{\prime }\left(t\right)}{S_{n-1}\left(t\right)}ln\frac{F_w^{\prime }\left(t\right)/S_{n-1}\left(t\right)}{G_w^{\prime }\left(t\right)/S_{n-1}\left(t\right)}\right)·S_{n-1}\left(t\right)\mathrm{d}t\hfill \\ & =& {\int }_0^w{F}_w^{\prime }\left(t\right)ln\frac{F_w^{\prime }\left(t\right)}{G_w^{\prime }\left(t\right)}\mathrm{d}t\hfill \\ & =& \mathrm{KL}(F;G,w),\hfill \end{array}$$

the Tail KL Divergence of F and G, which (as stated in Table 1) has the limit $\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}-ln\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}-1$ as the tail length w tends to zero.

Similarly, it can easily be seen that the multivariate versions of the JS Divergence, the Hellinger Distance, the ${\chi }^2$-Divergence and the $\alpha$-Divergence all have radial integral formulations identical to their corresponding univariate versions.

7.5. Observations

The general strategy for deriving these results is essentially the same as for the multivariate Tail Entropy: first use Lemma 4 to convert the multidimensional integral to an integral in one dimension, then use the technical lemmas of Section 5 to simplify the univariate integral as before.

Our results for the locally spherically symmetric multivariate case are shown in

Table 6. Asymptotic equivalences between LID formulations and tail measures of entropy or divergence for locally spherically symmetric distributions in the n-dimensional Euclidean setting. In each case, the density functions are assumed to be f and g, and the CDFs F and G of their induced distance distributions are assumed to be smooth growth functions. In the results, $V_n\left(r\right)$ and $S_{n-1}\left(r\right)$ denote the volume and surface area of the n-dimensional ball with radius r (respectively). In some cases, for the asymptotic limit to exist non-trivially (that is, to be both finite and non-zero), the tail entropy or tail divergence must be normalized by some multiplicative factor dependent on the tail volume $V_n\left(w\right)$.

Tail Measure	Formulation	Limit as $\mathit{w}\to 0^{+}$
Entropy	$\mathrm{H}(f,w)$ = $-{\int }_{\mathcal{B}\left(w\right)}{f}_{w}ln{f}_w\mathrm{d}\mathcal{B}\left(w\right)=-{\int }_0^w{F}_w^{\prime }\left(t\right)ln\frac{F_w^{\prime }\left(t\right)}{S_{n-1}\left(t\right)}\mathrm{d}t$	Diverges (no reweighting possible)
Varentropy	$\mathrm{VarH}(f,w)$ = ${\int }_{\mathcal{B}\left(w\right)}{f}_w{ln}^2{f}_w\mathrm{d}\mathcal{B}\left(w\right)-{\left({\int }_{\mathcal{B}\left(w\right)}{f}_{w}ln{f}_w\mathrm{d}\mathcal{B}\left(w\right)\right)}^2$	${\left(1-\frac{1}{\phi }\right)}^2$
	= ${\int }_0^w{F}_w^{\prime }\left(t\right){ln}^2\frac{F_w^{\prime }\left(t\right)}{S_{n-1}\left(t\right)}\mathrm{d}t-{\left({\int }_0^w{F}_w^{\prime }\left(t\right)ln\frac{F_w^{\prime }\left(t\right)}{S_{n-1}\left(t\right)}\mathrm{d}t\right)}^2$	$\phi =\frac{{ID}_F^{\ast }}{n}$
q-Entropy	${\mathrm{H}}_q(f,w)$ = $\frac{1}{q-1}{\int }_{\mathcal{B}\left(w\right)}{f}_w-f_w^q\mathrm{d}\mathcal{B}\left(w\right)$	$\frac{1}{q-1}$  if  $q<1$
	= $\frac{1}{q-1}{\int }_0^w{F}_w^{\prime }\left(t\right)-\frac{{\left(F_w^{\prime }\left(t\right)\right)}^q}{{\left(S_{n-1}\left(t\right)\right)}^{q-1}}\mathrm{d}t$	diverges if $q>1$
Normalized	$\frac{1}{V_n\left(w\right)}\mathrm{cH}(f,w)$ = $-\frac{1}{V_n\left(w\right)}{\int }_{\mathbf{x}\in \mathcal{B}\left(w\right)}{F}_w(\parallel \mathbf{x}\parallel )ln{F}_w(\parallel \mathbf{x}\parallel )\mathrm{d}\mathcal{B}\left(w\right)$	$\frac{\phi }{{(\phi +1)}^2}$
Cumulative Entropy	= $-\frac{1}{V_n\left(w\right)}{\int }_0^w\left(F_w\left(t\right)ln{F}_w\left(t\right)\right)·S_{n-1}\left(t\right)\mathrm{d}t$	$\phi =\frac{{ID}_F^{\ast }}{n}$
Normalized	$\frac{1}{V_n\left(w\right)}{\mathrm{cH}}_q(f,w)$ = $-\frac{1}{V_n\left(w\right)}·\frac{1}{q-1}{\int }_{\mathbf{x}\in \mathcal{B}\left(w\right)}{F}_w(\parallel \mathbf{x}\parallel )-{\left(F_w(\parallel \mathbf{x}\parallel )\right)}^q\mathrm{d}\mathcal{B}\left(w\right)$	$\frac{\phi }{(q\phi +1)(\phi +1)}$   if   $q\ne 1$
Cumulative q-Entropy	= $\frac{1}{V_n\left(w\right)}·\frac{1}{q-1}{\int }_0^w\left(F_w\left(t\right)-{\left(F_w\left(t\right)\right)}^q\right)·S_{n-1}\left(t\right)\mathrm{d}t$	$\phi =\frac{{ID}_F^{\ast }}{n}$
Normalized Entropy Power	$\frac{1}{V_n\left(w\right)}\mathrm{HP}(f,w)$ = $\frac{1}{V_n\left(w\right)}exp\left(\mathrm{H}(f,w)\right)$	$\frac{1}{\phi }exp\left(1-\frac{1}{\phi }\right);\phi =\frac{{ID}_F^{\ast }}{n}$
Normalized q-Entropy Power	$\frac{1}{V_n\left(w\right)}{\mathrm{HP}}_q(f,w)$ = $\frac{1}{V_n\left(w\right)}{\left[1+(1-q){\mathrm{H}}_q(f,w)\right]}^{\frac{1}{1-q}}$	${\left(\frac{{\phi }^q}{q\phi -q+1}\right)}^{\frac{1}{1-q}}$ ;    $\phi =\frac{{ID}_F^{\ast }}{n}$
		if   $q\ne 1$   and   $q\phi -q+1>0$
Cross Entropy	$\mathrm{XH}(f;g,w)$ = $-{\int }_{\mathcal{B}\left(w\right)}{f}_{w}ln{g}_w\mathrm{d}\mathcal{B}\left(w\right)=-{\int }_0^w{F}_w^{\prime }\left(t\right)ln\frac{G_w^{\prime }\left(t\right)}{S_{n-1}\left(t\right)}\mathrm{d}t$	Diverges (no reweighting possible)
Normalized Cross Entropy Power	$\frac{1}{V_n\left(w\right)}\mathrm{XHP}(f;g,w)$ = $\frac{1}{V_n\left(w\right)}exp\left(\mathrm{XH}(f;g,w)\right)$	$\frac{1}{\gamma }exp\left(\frac{\gamma -1}{\phi }\right);\phi =\frac{{ID}_F^{\ast }}{n};\gamma =\frac{{ID}_G^{\ast }}{n}$
Weighted	$V_n\left(w\right)·\mathrm{L}2\mathrm{D}(f;g,w)$ = $V_n\left(w\right){\int }_{\mathcal{B}\left(w\right)}{\left(f_w-g_w\right)}^2\mathrm{d}\mathcal{B}\left(w\right)$	$\frac{{\left(\phi -\gamma \right)}^2}{2(\phi +\gamma -1)}\left[1+\frac{1}{(2\phi -1)(2\gamma -1)}\right]$
L2 Distance	= $V_n\left(w\right){\int }_0^w\frac{1}{S_{n-1}\left(t\right)}{\left(F_w^{\prime }\left(t\right)-G_w^{\prime }\left(t\right)\right)}^2\mathrm{d}t$	$\phi =\frac{{ID}_F^{\ast }}{n};\gamma =\frac{{ID}_G^{\ast }}{n}$
		${ID}_F^{\ast }>\frac{1}{2}$; ${ID}_G^{\ast }>\frac{1}{2}$
Hellinger Distance	$\mathrm{HD}(f;g,w)$ = $\sqrt{\frac{1}{2}{\int }_{\mathcal{B}\left(w\right)}{\left(\sqrt{f_w}-\sqrt{g_w}\right)}^2\mathrm{d}\mathcal{B}\left(w\right)}$	$\frac{\left|1-\sqrt{\rho }\right|}{\sqrt{1+\rho }}$
	= $\sqrt{\frac{1}{2}{\int }_0^w{\left(\sqrt{F_w^{\prime }\left(t\right)}-\sqrt{G_w^{\prime }\left(t\right)}\right)}^2\mathrm{d}t}$	$\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$
${\chi }^2$-Divergence	${\chi }^2\mathrm{D}(f;g,w)$ = ${\int }_{\mathcal{B}\left(w\right)}\frac{{\left(f_w-g_w\right)}^2}{g_w}\mathrm{d}\mathcal{B}\left(w\right)$	$\frac{{\left(1-\rho \right)}^2}{\rho (2-\rho )}$
	= ${\int }_0^w\frac{{\left(F_w^{\prime }\left(t\right)-G_w^{\prime }\left(t\right)\right)}^2}{G_w^{\prime }\left(t\right)}\mathrm{d}t$	$\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }};\rho <2$
$\alpha$-Divergence	$\alpha \mathrm{D}(f;g,w)$ = $\frac{1}{\alpha (1-\alpha )}{\int }_{\mathcal{B}\left(w\right)}\alpha f_w+(1-\alpha )g_w-f_w^{\alpha }{g}_w^{1-\alpha }\mathrm{d}\mathcal{B}\left(w\right)$	$\frac{1}{\alpha (1-\alpha )}\left(1-\frac{1}{\alpha {\rho }^{\alpha -1}+(1-\alpha ){\rho }^{\alpha }}\right)$
	= $\frac{1}{\alpha (1-\alpha )}{\int }_0^w\alpha F_w^{\prime }\left(t\right)+(1-\alpha )G_w^{\prime }\left(t\right)$	$\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$
	$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-{\left(F_w^{\prime }\left(t\right)\right)}^{\alpha }{\left(G_w^{\prime }\left(t\right)\right)}^{1-\alpha }\mathrm{d}t$	Require $\alpha +\rho (1-\alpha )>0$
KL Divergence	$\mathrm{KL}(f;g,w)$ = ${\int }_{\mathcal{B}\left(w\right)}{f}_{w}ln\frac{f_w}{g_w}\mathrm{d}\mathcal{B}\left(w\right)={\int }_0^w{F}_w^{\prime }\left(t\right)ln\frac{F_w^{\prime }\left(t\right)}{G_w^{\prime }\left(t\right)}\mathrm{d}t$	$\rho -ln\rho -1;\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$
JS Divergence	$\mathrm{JS}(f;g,w)$ = $\frac{1}{2}\left(\mathrm{KL}\left(f;\frac{f+g}{2},w\right)+\mathrm{KL}\left(g;\frac{f+g}{2},w\right)\right)$	$\frac{\tau -ln\tau -1}{2};\tau =min\{\rho ,\frac{1}{\rho }\};\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$

; however, since their derivations greatly resemble those of the analogous univariate cases, we omit the details. Some remarks:

• A result for the Wasserstein Distance is not included, since its formulation does not generalize straightforwardly to higher dimensions, unlike the other divergence measures.
• The normalizations and weightings used depend only on the tail volume $V_n\left(w\right)$ and (for the Tsallis entropy variants) the parameter q. This generalizes our earlier univariate results where normalization was performed with regard to the tail length w.
• All the multivariate tail variants considered Table 6 are elegant generalizations of their corresponding univariate formulations, and all explicitly depend on the ratios between the LIDs and the dimension of the space n ($\phi =\frac{{ID}_F^{\ast }}{n}$ and $\gamma =\frac{{ID}_G^{\ast }}{n}$), or on the ratio of two LID values ($\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}=\frac{\gamma }{\phi }$). Among these, the Normalized Entropy Power and the Normalized Cumulative Entropy are maximized when ${ID}_F^{\ast }=n$, which can occur when the tail distribution is uniform. The Varentropy is minimized when ${ID}_F^{\ast }=n$, which can occur when the variance of the log-likelihood for a uniform distribution is equal to zero.
• As mentioned in Related Work, a number of previous studies in deep learning have found that the local intrinsic dimension in learned representations is lower than the dimension of the full space [32,33,34,35] (i.e., ${ID}_F^{\ast }<n$) and that the learning process progressively reduces local intrinsic dimension. Consider a concrete example where $n=100$ and ${ID}_F^{\ast }=12$ and the learning process is reducing ${ID}_F^{\ast }$ at a point from 12 to 11. The consequent effect on entropy can be interpreted from two different perspectives, either as an increase in tail distance entropy or a decrease in tail location entropy:
  • Considering univariate normalized entropy power or normalized cumulative entropy (Table 1), reduction of ${ID}_F^{\ast }$ corresponds to an increase in entropy. Here, the entropy is measuring the uncertainty of the univariate random variable modeling distances to nearest neighbors. Thus, reduction of ${ID}_F^{\ast }$ corresponds to an increase in “distance entropy”.
  • Considering multivariate normalized entropy power or multivariate normalized cumulative entropy (Table 6), reduction of ${ID}_F^{\ast }$ corresponds to an decrease in entropy. Here, the entropy is measuring the uncertainty of the multivariate random variable modeling locations of nearest neighbors, assuming local spherical symmetry. So reduction of ${ID}_F^{\ast }$ corresponds to a decrease in “location entropy”.

We will see a visualization of these scenarios in Section 7.6.

5.

All four of the multivariate tail divergences listed in Table 6, as well as the Hellinger Distance, have radial integral formulations that are identical to their univariate counterparts. All the divergences and distances (including the Weighted L2 Distance) are minimized when ${ID}_F^{\ast }={ID}_G^{\ast }$.

6.

By setting $n=1$, we can recover the univariate results from Table 1. However, note that the range of integration used in Table 6 is a hypersphere of radius w, where for $n=1$ it is the interval $[-w,w]$. In contrast, the integral formulations listed in Table 1 were taken over the interval $[0,w]$. For some results, this means a minor (constant factor of 2) difference between Table 1 and the result from Table 6 when $n=1$.

7.6. Visualization of Behavior

Our results in Table 6 relate local intrinsic dimensionality to entropies and divergences. If analyzing an n dimensional global distribution such as the standard normal distribution or uniform distribution, then the dimension of every sub-manifold (i.e., the local intrinsic dimensionality ${ID}_F^{\ast }$) will be n. However, our interest is in situations where the local intrinsic dimensionality differs from the representation dimension n. To provide further intuition on this aspect, two plots are shown in Figure 1.

Figure 1a compares the behavior of the normalized entropy power and the normalized cumulative entropy (multiplied by a constant factor of 4) in n-dimensional space, as the ratio $\varphi =\frac{{ID}_F^{\ast }}{n}$ is varied. We see that these measures have similar trends and they are maximized when ${ID}_F^{\ast }=n$. We also see that when $1\ll {ID}_F^{\ast }<n$, these entropic measures will decrease if ${ID}_F^{\ast }$ is decreased (for a fixed n). On the other hand, if $n=1$ and $1\ll {ID}_F^{\ast }$, then these entropic measures will increase if ${ID}_F^{\ast }$ is decreased, where $n=1$ corresponds to the scenario where we are modeling the uncertainty of a distance distribution. This illustrates remark number 4 from Section 7.5 above.

Figure 1b compares the behavior of different tail divergences as the ratio $\rho =\frac{{ID}_G^{\ast }}{{ID}_F^{\ast }}$ varies. The divergences shown are the KL divergence, the Jensen–Shannon divergence and the Hellinger distance. These measures have similar trends as $\rho$ varies and are minimized and equal to zero when ${ID}_F^{\ast }={ID}_G^{\ast }$. Also, the Hellinger distance is bounded above by 1.

8. Conclusions

In this theoretical investigation, we have established asymptotic relationships between tail entropy variants, tail divergences and the theory of local intrinsic dimensionality. Our results are derived under the assumption that the distribution(s) under consideration are being analyzed in a highly local context, within the distribution tail(s), an asymptotically small neighborhood whose radius approaches zero. These results show that tail entropies and tail divergences depend in a fundamental way on local intrinsic dimensionality and help form a theoretical foundation for cross-fertilization between intrinsic dimensionality research and entropy research. As future work, we plan to investigate the potential of these new characterizations in a range of application settings. For example, for use as a basis in machine learning to characterize and improve representations and representation learning, as well as use in understanding behavior of physical systems such as fluids and helping characterize their critical transitions in time and space.

Our results from both univariate and multivariate cases, show that the tail entropies and divergences considered in this paper depend only on (i) the embedding (representation) dimension in which the distribution is situated, and (ii) the local intrinsic dimension(s) of the distribution(s). Furthermore, in many cases there is dependence involving the ratio between the intrinsic dimension and the embedding dimension.

Consider the context of distance based analysis, when a distribution models distances from a central query location to its nearest neighbors, and the distances are induced by global data. In this situation, our characterization of entropy might be termed as ‘personalized’, in that entropy expresses the uncertainty (or complexity) from the perspective of the query, in regard to the distances to samples within an asymptotically small neighborhood. Phrased another way, these local entropies are ‘observer-dependent’, since they are tied to the choice of query (the observer). This can be contrasted with the more common notion of entropy, where one analyzes a global distribution, and there is no requirement of a query point or its local neighborhood.

As alluded to in the introduction, divergences between tail distributions could be used for comparison of real and synthetic distributions, as is commonly required for generative adversarial networks (GANs). Given a particular query location we may either: (i) compute the divergence between the univariate tail distance distributions of synthetic and real examples, as measured from a query point; or (ii) compute the divergence between the multivariate tail distance distributions of synthetic and real examples, again as measured from the query, under an assumption of local isotropy. Our results show that under the assumption of local spherical symmetry, the use of divergences (such as KL) between tail distance distributions is asymptotically equivalent to the standard multivariate formulations with the same divergences, when restricted to the neighborhoods around locations of interest. For future work it will be interesting to consider whether it is possible to further extend our multivariate results to elliptically symmetric distributions or skew-elliptical distributions, such as those studied by Contreras-Reyes [65].

Lastly, our results in Table 1 and Table 6 show theoretical relationships for entropies and divergences, but in practice one must estimate the measures using samples of data. A natural approach here is to first estimate local intrinsic dimensional values such as ${ID}_F^{\ast }$ and ${ID}_G^{\ast }$ using any desired estimator (such as the maximum likelihood estimator [39,40,41]), and then plug in the estimated LID value into the desired tail entropy or tail divergence formula. For example, an estimator of the (univariate) Normalized Cumulative Entropy could be obtained by computing $\frac{\widehat{{ID}_F^{\ast }}}{{(\widehat{{ID}_F^{\ast }}+1)}^2}$, where $\widehat{{ID}_F^{\ast }}$ is the estimated LID of the distance distribution F.