Towards More Efficient Rényi Entropy Estimation

Abstract

Estimation of Rényi entropy is of fundamental importance to many applications in cryptography, statistical inference, and machine learning. This paper aims to improve the existing estimators with regard to: (a) the sample size, (b) the estimator adaptiveness, and (c) the simplicity of the analyses. The contribution is a novel analysis of the generalized “birthday paradox” collision estimator. The analysis is simpler than in prior works, gives clear formulas, and strengthens existing bounds. The improved bounds are used to develop an adaptive estimation technique that outperforms previous methods, particularly in regimes of low or moderate entropy. Last but not least, to demonstrate that the developed techniques are of broader interest, a number of applications concerning theoretical and practical properties of “birthday estimators” are discussed.

1. Introduction

1.1. Motivation and Background

The aim of entropy estimation is to approximately compute the Rényi entropy of an unknown probability distribution using only observed samples. Since Rényi entropy is the most established and popular uncertainty measure, the problem is not only one of fundamental interest to information theory [1], but also one of importance to a number of applied research areas. These applications of Rényi entropy include, in particular, quantifying diversity in ecology [2,3,4], statistical mechanics [5,6], thermodynamics [7], characterizing properties of probability distributions [8,9], DNA sequencing [10,11], network anomaly detection [12,13], clustering [14,15], data mining [16,17], predictive modelling [18], as well as security and cryptography [19,20,21,22,23,24,25,26,27,28].

To state the problem formally, if we consider a fixed discrete distribution with probability mass function $p_X$, the Rényi entropy [1] of some fixed positive order d is defined as

$$\begin{array}{c}\hfill {\mathsf{H}}_d\left(p_X\right)\triangleq \frac{1}{1-d}{log}_2\left(\sum _x{p}_X{\left(x\right)}^d\right).\end{array}$$

(1)

The challenge is to estimate this quantity from n independent samples $X_1,\dots ,X_n{\sim }^{iid}{p}_X$. More precisely, we seek an explicit function of samples $\widehat{\mathsf{H}}$ such that the approximation

$$\begin{array}{c}\hfill \widehat{\mathsf{H}}(X_1,\dots ,X_n)\approx {\mathsf{H}}_d\left(p_X\right)\end{array}$$

(2)

holds with a small error, high probability, and possibly minimal sample size n. We are interested in non-parametric estimation, as the distribution X remains unknown.

As in prior work on Rényi entropy estimation, we focus on integer orders. This is not limiting, because Rényi entropies of positive integer orders: (a) encode the complete information about the distribution [29], (b) are sufficient for practical applications due to known smoothing and interpolation properties [30,31], and finally (c) are more efficient to estimate from the algorithmic perspective [32].

1.2. Related Work

The most natural, albeit not most efficient, are so-called plugin estimators, which insert a non-parametric estimate of the probability mass function into the entropy formula [33]. As opposed to that, sample-optimal estimators for Rényi entropy are more involved, as shown in a relatively recent line of work [32,34,35]. Loosely speaking, these estimators relate the entropy to collision probabilities and then take advantage of the birthday paradox. More specifically, base estimations are obtained by counting collisions among tuples in the observed sample, and then are optionally run in parallel to boost the statistical confidence. The birthday paradox intuitively explains why the resulting algorithms are sublinear in the alphabet size. Specifically, for the alphabet of size K, the sample-optimal estimation takes $O_d\left(K^{1-1/d}\right)$ samples (the asymptotic notation $O_d\left(\right)$ hides dependencies on d.) to achieve an additive error of at most 1 and a level of confidence of at least $2/3$ [32,34,35]. The aforesaid state-of-the-art estimators are asymptotically minimax optimal; that is, they achieve the asymptotically minimal sample size over the worst choice of the sampling distribution. However, their known analyses leave room for improvement with regard to simplicity, numerical precision, estimation adaptiveness, and techniques used. We elaborate on these issues below:

1.

Lack of simplicity and numerical precision. The analyses of the state-of-the-art estimators [32,34,35] struggle with analysing the variance of collision estimators, which is tackled either by poissonization approximations [32,34] (which carry their own overhead) or by using involved combinatorics [35]. As a consequence, the variance bounds are available in asymptotic “big-Oh” notation hiding constants and higher-order dependencies (such as relations to the order d), and are not suitable for applications in statistics or cryptography, which demanding precise formulas. This point was already raised in the context of applied works on physically unclonable functions [36].

2.

Adaptiveness gap. The focus of prior research was on establishing bounds under the worst-case choice over all distributions [32,34]. This is overly pessimistic because distributions that arise in practical applications are of a different structure than those occurring in this worst-case scenario analysis (the worst-case choice is known to be a mixture of uniform and Dirac distributions). The bounds were somewhat improved in the follow-up work [35] where a prior entropy bound is assumed. Still, there is a gap as the entropy bound is usually not known prior to the actual experiment. In fact, getting an entropy bound might be more costly than its application.

3.

Lack of established techniques. Prior work focused on delivering asymptotic formulas and did not elaborate much on techniques that could help obtain simpler and tighter bounds. The difficulty of analyzing collision estimators is a recurring issue, well-known to the researchers working on property testing [37]. Do we have a systematic method of handling it?

1.3. Our Contribution

This work fills the aforementioned gaps with the following contributions:

1.

Simpler and more accurate analysis of collision estimators, the main building blocks of the state-of-the-art Rényi entropy estimators. We analyze the collision estimators as kernel averages with the technique of Hoeffding’s decomposition. This novelty brings the promised simplicity and improvement in accuracy.

2.

Adaptive estimation of Rényi entropy, using no prior knowledge of the sampling distribution. The sample size cost of the presented algorithm is essentially optimal (up to a poly-logarithmic factor).

3

Modular approach using the established methods of U-statistics and Robust Mean Estimation. Specifically, we point out that the moment estimation problem, which Rényi entropy estimation reduces to, can be seen as the estimation of certain U-statistics. While the dedicated statistical theory provides the bias–variance analysis, the confidence can be independently boosted by techniques of Robust Mean Estimation.

This paper aims to solve the two mentioned bottlenecks and, in this way, to close the gap between the theory-oriented state-of-the-art and the demand coming from applied researchers and their practical use cases, such as [36].

1.4. Organization

The notation and preliminary concepts are discussed in Section 2. The technical results and applications are presented in Section 3. The proofs are discussed in Appendix A, and the work is concluded in Section 4.

2. Preliminaries

2.1. Basic Notation

Throughout the paper, $p_X$ is the probability mass function of a fixed discrete distribution over an alphabet of size K and $X_1,\dots ,X_n$ are observed independent samples from $p_X$. We denote $\left[n\right]=\{1,\dots ,n\}$ and let $\left(\genfrac{}{}{0pt}{}{\left[n\right]}{d}\right)$ denote the collection of all d-element subsets of n.

2.2. Estimation of Entropy, Moments, and Collisions

We leverage the following observation from prior work: the task of Rényi entropy estimation is equivalent to the task of moment estimation. More precisely, the d-th moment of the probability mass function $p_X$ is defined as

$$\begin{array}{c}\hfill P_d\triangleq \sum _x{p}_X{\left(x\right)}^d,\end{array}$$

(3)

and then—immediately from definition—we have the following result:

Proposition 1.

An estimate $\widehat{H}$ of the Rényi entropy of order d (defined in (1)) has an additive error of ϵ if and only if $2^{\widehat{H}}$ is an estimate of the d-th moment (defined in (3)) with a relative error of ${ϵ}^{\prime }=2^{ϵ(d-1)}-1$.

Solving the (equivalent) problem of moment estimation is more convenient due to the beautiful representation of moments as collision probabilities. More precisely, we have

$$\begin{array}{c}\hfill P_d=\mathbf{P}(X_1=X_2=\dots =X_d).\end{array}$$

(4)

2.3. U-Statistics

For a symmetric real function h of d arguments, the U-statistic with kernel h of the sample $X_1,\dots ,X_n$ is defined as:

$$\begin{array}{c}\hfill U_h(X_1,\dots ,X_n)\triangleq {\left(\genfrac{}{}{0pt}{}{n}{d}\right)}^{-1}\sum _{1⩽i_1<\dots <i_d⩽n}h(X_{i_1},\dots ,X_{i_d}).\end{array}$$

(5)

The U-statistic gives an unbiased estimate of the function expectation, hence its name. U-statistics were invented by Hoeffding [38] to extend certain results, such as concentration bounds, to sums of partly dependent terms. Many statistical quantities can be related to U-statistics; for example, moments or sample variances [39]. In the same spirit, we will see estimators of the collision probability in Equation (3) as U-statistics and use those to establish their desired properties.

2.4. Robust Mean Estimation

It is difficult to directly obtain high confidence bounds (such as those of the Chernoff–Hoeffding type) for moment estimators. Instead, we will boost weaker bounds obtained from bias–variance analyses. To this end, we combine independent runs of estimators into high-confidence bounds using the technique of Robust Mean Estimation.

Estimating the mean of a distribution from i.i.d. samples is not trivial: the “obvious” use of the empirical mean is inaccurate for heavy detailed distributions. Following the recent survey [40], we mention here two solutions:

• the median-of-means approach organizes data (such as independent algorithm outputs) into blocks and computes the median of means within blocks.
• the trimmed mean approach takes the mean of independent runs, excluding a certain fraction of smallest and biggest outcomes (removing outliers).

We note that any robust mean estimation can be used to achieve confidence boosting. In this work, we stick to the median-of-means. The following result discusses its performance.

Proposition 2

(Performance of Median-of-Means [40]). Let $Z_1,\dots ,Z_n$ be i.i.d. random variables with mean μ and variance ${\sigma }^2$. For $k=\lceil 8log(1/\delta )\rceil$, split $Z_1,\dots ,Z_n$ into k blocks and let $\widehat{\mu }$ be the median of the means within blocks. Then, with probability $1-\delta$,

$$\begin{array}{c}\hfill |\widehat{\mu }-\mu |⩽\sigma \sqrt{\frac{4}{\lfloor n/k\rfloor }}.\end{array}$$

(6)

2.5. Moment Bounds

We will need some bounds on moments of probability distributions, in order to simplify formulas that arise from variance analysis. Specifically, we will use these auxiliary results to express higher-order moments in terms of moments of small order ($d=2$ and $d=3$).

Proposition 3.

For any probability distribution $p=\left(p_i\right)$, we have

$$\begin{array}{c}\hfill \sum _i{p}_i^{2d-k}⩾{\left(\sum _i{p}_i^d\right)}^2\end{array}$$

for any integer $k,d$ such that $1⩽k⩽d$. Moreover, with $p_{max}\triangleq {max}_i{p}_i$ it holds that

$$\begin{array}{c}\hfill \sum _i{p}_i^{2d-k}⩾{\left(\sum _i{p}_i^d\right)}^2/p_{max}^{k-1}.\end{array}$$

Proposition 4.

For any non-negative sequence $\left(p_i\right)$, it holds that the quantity ${\parallel p\parallel }_d\triangleq {\left({\sum }_i{p}_i^d\right)}^{1/d}$ decreases in $d⩾1$.

3. Results

Following the convention from prior work, our results are stated for moment estimation, which is equivalent to entropy estimation as discussed in Proposition 1. Throughout the rest of the paper, we keep this reduction in mind.

3.1. Simpler & More Accurate Moment Estimation

The first novelty offered in the current work is a simplified and strengthened variance analysis of the state-of-the-art moment estimator, presented in Algorithm 1. Differently than in prior work, we write the estimator output as a kernel average of the function $h(x_{i_1},\dots ,x_{i_d})\triangleq \mathbf{I}(x_{i_1}=\dots x_{i_d})$ over the d-element subsets of the sample. This approach is not only more readable but ultimately also more accurate, as it links the task of moment estimation to the established theory of U-statistics [38].

On top of that comes the refined high-confidence moment estimator in Algorithm 2, which we build on robust mean estimators [40]. Due to this modularity, it uses fewer samples than the direct approach from prior work [34].

Algorithm 1: BIRTHDAY MOMENT ESTIMATOR

Data:

entropy order d samples $X_1,\dots ,X_n$, $n⩾d$, from an unknown distribution with p.m.f. $p_X$

Result: an estimate ${\widehat{P}}_d$ of $P_d={\sum }_x{p}_X{\left(x\right)}^d$, the d-th moment of $p_X$

$C\leftarrow \#\left\{\{i_1\dots i_d\}\in \left(\genfrac{}{}{0pt}{}{\left[n\right]}{d}\right):X_{i_1}=X_{i_2}=\dots =X_{i_d}\right\}$

${\widehat{P}}_d\leftarrow C/\left(\genfrac{}{}{0pt}{}{n}{d}\right)$

return  ${\widehat{P}}_d$

Theorem 1

(Bias–Variance Analysis of Moment Estimator). With the notation as above, the output of Algorithm 1 is unbiased:

$$\begin{array}{c}\hfill \mathbf{E}[{\widehat{P}}_d-P_d]=0,\end{array}$$

(7)

and its variance equals

$$\begin{array}{c}\hfill \mathbf{Var}\left[{\widehat{P}}_d\right]=\frac{{\sum }_{k=1}^d\left(\genfrac{}{}{0pt}{}{d}{k}\right)\left(\genfrac{}{}{0pt}{}{n-d}{d-k}\right)(P_{2d-k}-P_d^2)}{\left(\genfrac{}{}{0pt}{}{n}{d}\right)}.\end{array}$$

(8)

In particular, for any $n⩾d$ and $ϵ>0$ we can upper-bound the variance as

$$\begin{array}{c}\hfill \mathbf{Var}\left[{\widehat{P}}_d\right]⩽\frac{2d^2{P}_d^{2-1/d}}{n},\end{array}$$

(9)

and the relative error confidence as

$$\begin{array}{c}\hfill \mathbf{P}\left[\right|{\widehat{P}}_d-P_d|>ϵP_d]⩽\frac{2d^2}{n{P}_d^{1/d}{ϵ}^2}.\end{array}$$

(10)

Remark 1

(Efficient Implementation). Birthday estimators can be efficiently computed, in memory $O\left(n\right)$ and one-pass over samples, by using a dictionary to count empirical frequencies of observed elements. Such an implementation is given in the supplementary material [41].

Remark 2

(Structural Assumptions). The bounds from Theorem 1 use the statistic $P_d$ of the sampling distribution. This explicit dependency is beneficial, as further discussion clarifies. Lacking any prior knowledge, it can be estimated by the worst-case behavior, in terms of the alphabet size.

Algorithm 2: HIGH-CONFIDENCE BIRTHDAY MOMENT ESTIMATOR

Data:

a moment order d samples $X_1,\dots ,X_n$, $n⩾d$, from an unknown distribution with p.m.f. $p_X$ a confidence parameter $\delta$

Result: an estimate $\widehat{P_d}$ of $P_d={\sum }_x{p}_X{\left(x\right)}^d$, the d-th moment of $p_X$

$k\leftarrow \lceil 8log(1/\delta )\rceil$

$\ell \leftarrow \lfloor n/k\rfloor$

for  $j=1,\dots ,k\rfloor$  do

$|{\widehat{P}}_d^{\left(j\right)}\leftarrow$ output of Algorithm 1 $X_{j·\ell },\dots ,X_{(j+1)·\ell -1}$ on (j-th input block of length ℓ)

end

$\phantom{(}$$\widehat{P_d}\leftarrow \mathrm{MEDIAN}({\widehat{P}}_d^{\left(j\right)},j=1,\dots ,k)$

We see that the choice $n>6d^2{ϵ}^{-2}/P_d^{1/d}$ guarantees $\mathbf{P}\left[\right|{\widehat{P}}_d-P_d|>ϵP_d]⩽\delta$ for $\delta =\frac{1}{3}$. Higher confidence (smaller $\delta$) can be handled with the method of Robust Mean Estimation.

Theorem 2

(High-Confidence Moment Estimator). For any $ϵ>0$, Algorithm 2 approximates $P_d$ with probability $1-\delta$ and a relative error of ϵ provided that

$$\begin{array}{c}\hfill n⩾⌈\frac{8d^2}{{ϵ}^2{P}_d^{1/d}}⌉·\lceil 8log(1/\delta )\rceil .\end{array}$$

(11)

Remark 3.

The constant can be refined a little based on the methods from [42].

When comparing these results with prior work, we review the following aspects:

1.

Novel techniques of broader interest. We recall that analyzing variance formulas has been challenging for prior works on entropy estimation ([32,34] resorted to Poisson approximations, while [35] gave an involved combinatorial argument), even for the case $d=2$ (the lack of sharp analysis caused lots of difficulties in property testing [37]). As opposed to these ad hoc approaches, we establish the formula in a simple yet direct manner, pointing out that such formulas can be obtained by the techniques of U-statistics. When discussing applications, we will further benefit from these tools.

2.

Clean and improved formulas. Our confidence bound does not involve any implicit constants, while prior works in their main statements have unspecified dependencies on d (essentially, hiding more than absolute constants). We compare the accuracy bounds from this and prior works in

Table 1. The performance of the “birthday estimator” of moments. In the formulas, $ϵ\in (0,1)$ is the relative error, n is the sample size, and $1-\delta$ is the confidence (the prob. that $|{\widehat{P}}_d-P_d|⩽ϵP_d$).

Confidence $1-\mathit{\delta }$ in Algorithm 1	Author	Assumption
$\delta ⩽4d^2{n}^{-1}{ϵ}^{-2}{P}_d^{-1/d}$	this paper	$n⩾d$
$\delta ⩽O_d\left(n^{-1}{ϵ}^{-2}{P}_d^{-1/d}\right)$	[35]	$n⩾d$
$\delta ⩽O_d\left(n^{-1}{ϵ}^{-2}{K}^{1-1/d}\right)$	[32]	$n⩾d$

below. Our bound is strictly better given that $P_d$ is minimized at the uniform distribution and thus $P_d^{-1/d}⩽K^{-1+1/d}$ (with a large gap when the distribution is far from uniform), which establishes that the dependency on d is $4d^2$. Leveraging the theory of U-statistics, we will show that the factor $O\left(d^2\right)$ is optimal, which is also a novel contribution.

3.2. Adaptive Estimation

As per our variance analysis, the performance actually depends not on the alphabet size K (that is ultimately the pessimistic bound) but rather on a more fine-grained statistic of X, namely $P_d$. Following the result in Theorem 2, we could hope for a moment estimation of

$$\begin{array}{c}\hfill n{=}^{?}\mathrm{\Theta }(log(1/\delta )){ϵ}^{-2}{d}^2{P}_d^{-\frac{1}{d}}.\end{array}$$

(12)

The obvious obstacle is that, in general, we do not know $P_d$ in advance. We solve this problem by developing an adaptive algorithm. It does not assume the right number of samples in advance but tries gradually and eventually terminates with high probability, giving the answer within the desired margin of error and using only a few more samples than the ideal bound conjectured above. Its core is a subroutine that guesses the moment value, gradually changing the candidate.

3.2.1. Lower-Bounding Moments

The key ingredients of our approach are the following two subroutines: Algorithm 3 tests, based on samples, whether the moment is smaller or bigger than a proposed candidate; subsequently, Algorithm 4 loops the tester over a grid of candidate values.

Algorithm 3: MOMENTLESSTHAN

Data:

independent samples $X_1,\dots ,X_n{\sim }^{iid}{p}_X$ tested threshold Q access to an estimator $\widehat{P}$ of $P_d$

Result: tests if $P_d⩽\frac{Q}{2}$ or $P_d⩾2Q$

$\widehat{P}\leftarrow \widehat{P}(X_1,\dots ,X_n)$

if  $\widehat{P}<Q$  then

$|$  return True

else if  $\widehat{P}⩾Q$  then

$|$  return False

Algorithm 4: MOMENTBOUND

Data:

independent samples $X_1,X_2,\dots {\sim }^{iid}{p}_X$ (online access) access to Algorithm 3 the sample size $n(Q,\delta )$ for Algorithm 3

Result: finds bound Q such that $\frac{Q}{2}⩽P_d⩽Q$

$Q\leftarrow 1$

while$Q⩾1/K^{d-1}\&b=\mathtt{True}$  do

end

$\phantom{(}$return Q

The correctness of the approach is guaranteed by the lemmas stated below.

Lemma 1

(Moment Testing). Let $\widehat{P}$ be any estimator of $P_d$ which, given

$$n⩾C\left(\delta \right)P_d^{-1/d}{ϵ}^{-2}$$

samples, for some function $C(·)$ and any $ϵ>0,1>\delta >0$, achieves a relative error of ϵ with probability $1-\delta$. Then Algorithm 3, when given at least

$$\begin{array}{c}\hfill n(Q,\delta )=\lceil 4C\left(\delta \right)Q^{-1/d}\rceil \end{array}$$

samples of X, with probability $1-\delta$, outputs True when $P_d⩽Q/2$ and False when $P_d⩾2Q$.

Lemma 2

(Moment Bounding). With same $\widehat{P}$ as in Lemma 1, with probability $1-\delta$, Algorithm 4 terminates after using at most

$$\begin{array}{c}\hfill n=\lceil 4C(\delta /(d-1)logK)P_d^{-1/d}\rceil \end{array}$$

samples of X, and its output Q satisfies $\frac{Q}{2}⩽P_d⩽4Q$.

3.2.2. Construction of Adaptive Estimator

Armed with Lemma 2, we are ready to analyze adaptive estimation. The algorithm is the same in the non-adaptive case, and we need Lemma 2 only to adjust the sample size.

Theorem 3

(Adaptive Moment Estimation). With online access to samples from X, with probability $1-\delta$, one can estimate $P_d$ within a relative error of ϵ, terminating at the number of samples

$$\begin{array}{c}\hfill n⩽O\left((log(1/\delta )+loglogK)P_d^{-\frac{1}{d}}{ϵ}^{-2}\right).\end{array}$$

(13)

Remark 4

(Adaptive Overhead). The sample size for adaptive estimation differs from the “dream bound” in (12) by a (very small) factor of $loglogK$.

Remark 5

(Sample Complexity Guarantees). Observe that the algorithm is not guaranteed to achieve the “good” sample complexity every time, but rather with high probability. This is a minor issue inherently related to the concept of adaptive estimation and does not affect much the performance in practical applications. Namely, we can always clip the total number of samples available at the pessimistic level from prior work and fall back to the fixed-size sample estimation should the adaptive estimation exceed the limit. This, however, happens with small probability δ, which can be further decreased with little overhead.

Remark 6

(Comparison to Prior Work). We give a clear comparison of our adaptive estimation and prior work in

Table 2. The performance of moment (Rényi entropy) estimators. In the formulas, K is the alphabet size, $ϵ$ is the relative error, and the confidence is $1-\delta$.

Sample Size n	Author	Assumptions
$O\left((log(1/\delta )+loglogK){ϵ}^{-2}{P}_d^{-1/d}\right)$	this paper
$O(log(1/\delta ){ϵ}^{-2}{Q}^{-\frac{1}{d}})$	[35]	prior bound $Q⩽P_d$
$O(log(1/\delta ){ϵ}^{-2}{K}^{1-\frac{1}{d}})$	[32]

below. We always have $P_d⩾Q$ and $P_d⩾K^{-1+1/d}$. Furthermore, usually $P_d/Q$ is much bigger than 1 (because in practice we do not know a prior lower bound in advance) and $P_d$ is much bigger than $K^{-1+1/d}$ (this gap can be as big as $K^{\mathrm{\Omega }\left(1\right)}$, which is a huge factor for some applications; for example, in cryptography, we consider alphabets as big as $K=2^{256}$). Thus, our bound outperforms the previous approaches for typical use cases.

3.3. Novel Applications

3.3.1. One-Sided Estimation: Random Sources for Cryptography

The collision probability $P_d$ for $d=2$ plays an important role in cryptography: it quantifies the amount of randomness that can be extracted from a distribution [28]. For that extraction to work, $P_2$ should be small enough. Specifically, if $P_2⩽2^{-k}$ allows for extraction of nearly k bits of cryptographic quality, how could we check whether $P_2⩽2^{-k}$?

To solve the problem, we adapt Algorithm 4 by adding early stopping; namely, we quit the loop if $Q⩾2^{-k}$. We take $n=O(log(k/\delta )2^{k/2})$ samples.

It remains for us to show that the algorithm behaves as desired. By the guarantees in Lemma 1 and the union bound over at most k steps, with probability $1-\delta$, we have the following: when $P_2<2^{-k-1}$, the algorithm finishes with $Q⩽2^{-k}$; and when $P_2>2^{-k+1}$, the algorithm finishes with $Q⩾2^{-k}$. This can be generalized to one-sided estimation for any d, where the goal is to decide whether $P_d>\mathrm{\Omega }\left(2^{-k}\right)$ or $P_d<O\left(2^{-k}\right)$.

This one-sided estimation allows for saving samples and testing only up to a necessary extent. In cryptography, we do not have to estimate the whole entropy (which may be more costly, even with adaptive estimation) but only what suffices for the chosen application.

3.3.2. Birthday Estimators Are UMVUE

The shortcut UMVUE stands for uniformly minimum unbiased variance estimators. We prove this conceptually strong and interesting characterization, which essentially shows that the birthday estimator in Theorem 1 is variance-optimal among unbiased estimators. The argument is inspired by our variance analysis, seeing the estimator as a U-statistic.

Corollary 1

(Birthday estimators are UMVUE). Let ${\widehat{P}}_d$ be as in Algorithm 1 and ${\tilde{P}}_d$ be another unbiased (for any X) estimator of $P_d$. Then we have $\mathbf{Var}\left[{\widehat{P}}_d\right]⩽\mathbf{Var}\left[{\tilde{P}}_d\right]$.

Proof. 

We will use the known result due to Lehmann and Sheff, which states that if an unbiased estimator is a function of a complete and sufficient data statistic, then it has the smallest possible variance [43,44].

To apply this result, without losing generality (as it is a matter of encoding the alphabet), we assume that X takes values in the set $\{1,\dots ,K\}$. Consider the sample $X_1,\dots ,X_n$, and let $\sigma$ be the rearrangement such that $X_{\sigma \left(1\right)}⩽X_{\sigma \left(2\right)}⩽\dots X_{\sigma \left(n\right)}$ (this is called the order statistic). The estimator ${\widehat{P}}_d$ can be seen as the average of the symmetric function $h(X_{i_1},\dots ,X_{i_d})=\mathbf{I}(X_{i_1}=x_{i_2}=\dots =X_{i_d})$ over tuples $i_1<i_2<\dots <i_d$, and thus is also the function of $T=(X_{\sigma \left(1\right)},X_{\sigma \left(2\right)},\dots ,X_{\sigma \left(n\right)})$. The claim follows if we prove that T is sufficient and complete (as a sample statistic).

Order statistics are sufficient for univariate distributions. This is because we have:

$$\begin{array}{c}\hfill P_{X_1,\dots ,X_n|X_{\sigma \left(1\right)},X_{\sigma \left(2\right)},\dots ,X_{\sigma \left(n\right)}}=\frac{1}{n!},\end{array}$$

which does not depend on $X_i$. Thus, $X_{\sigma \left(1\right)},X_{\sigma \left(2\right)},\dots ,X_{\sigma \left(n\right)}$ carries the same information about the data as $X_i$.

The completeness of T means that there are no non-trivial unbiased estimators of zero; equivalently, if $\mathbf{E}f\left(T\right)=0$ for all sampling distributions and some function f, then $\mathbf{P}\left[f\right(T)=0]=1$. To this end, observe that from the sufficiency proved above we have

$$\begin{array}{c}\hfill \mathbf{E}f(X_{\sigma \left(1\right)},\dots ,X_{\sigma \left(n\right)})=0⟹\mathbf{E}f(X_1,\dots ,X_n)=0\end{array}$$

Suppose that the above holds for any finitely supported sampling distribution X. Let X take values $i\in I$ with probability $p_i$. Then the above implies

$$\begin{array}{c}\hfill \sum _{i_1,\dots ,i_n\in I\times \dots \times I}{p}_{i_1}\dots p_{i_n}f(i_1,\dots ,i_n)=0\end{array}$$

for every distribution $\left(p_i\right)$. The left-hand side represents a multivariate polynomial in variable $p_i$, which evaluates to zero on the entire simplex of dimension $n-1$. Thus, its coefficients must be zero, which implies $f(i_1,\dots ,i_n)=0$ for each tuple $i_1,\dots ,i_n$ and proves that T is sufficient. □

3.3.3. Central Limit Theorem for Birthday Estimators

We again represent the estimator as the average sum over tuples:

$$\begin{array}{cc}\hfill {\widehat{P}}_d& ={\left(\genfrac{}{}{0pt}{}{n}{d}\right)}^{-1}\sum _{i_1<\dots <i_d}h(X_{i_1},\dots ,X_{i_d}),\hfill \end{array}$$

where

$$\begin{array}{c}\hfill h(x_{i_1},\dots ,x_{i_d})\triangleq \mathbf{I}(x_{i_1}=x_{i_2}=\dots =x_{i_d}).\end{array}$$

We view the whole expression as the U-statistic with the kernel function h. Then we show the following strong result (below, $\mathcal{N}(0,{\sigma }^2)$ denotes the normal distribution with zero-mean and variance ${\sigma }^2$).

Corollary 2

(Asymptotic Normality). For $n\to +\infty$ it holds that $\sqrt{n}·({\widehat{P}}_d-P_d)\to \mathcal{N}(0,{\sigma }^2)$ where ${\sigma }^2=d^2(P_{2d-1}-P_d^2)$, with $P_d$ as in (3).

The proof utilizes the classical convergence results for U-statistics [38] and the derivation of our variance formula. Note that the result says that the central limit theorem applies, despite the fact that the sum components are correlated. Clearly, the result is interesting on its own, particularly because (a) it proves that our constant $O\left(d^2\right)$ is sharp, and (b) can be used more generally to benchmark other proposed bounds, by means of comparing with the asymptotic gaussian tail.

However, we would like to point out an application to applied statistical research. In [36], Rényi entropy of order $d=2$ has been estimated for the distribution of physically unclonable functions (PUFs), which are important in the field of cryptography. However, their methodology lacks statistical rigor. Particularly, for the authors’ needs, prior work on Rényi entropy estimation was insufficient in terms of clarity on constants; thus, they resorted to the naive application of the central limit theorem, which can give very biased results.

A more solid alternative would be to use the above corollary to (a) justify the soundness, at least in the regime of large n, and (b) establish a more robust estimation of the variance.

Proof of Corollary 2.

The limiting variance equals $d^2{\sigma }_1^2$ [39] with

$$\begin{array}{c}\hfill {\sigma }_1^2=\mathbf{Cov}[h(X_{i_1},\dots ,X_{i_d}),h(X_{j_1},\dots ,X_{j_d})],\end{array}$$

where the tuples $(i_1,\dots ,i_d)$ and $(j_1,\dots ,j_d)$ have only $k=1$ element in common. We analyze this expression when proving Theorem 1 and know that it equals $P_{2d-1}-P_d^2$. The claimed formula now follows. □

3.3.4. Adaptive Testing in Evaluation of PUFs

Here we discuss again an application to [36], but from a different perspective. As explained by the authors, the problem with estimating Rényi entropy of PUFs is a serious bottleneck: for this problem, the alphabet is huge, which limits the experiment scope, even on computational clusters [36]. In this note, we would like to point out that parts of these difficulties can be solved by our adaptive estimation. In fact, PUFs provide an excellent use case when entropy is quite low; therefore, the moment $P_d$ term in Theorem 3 is much bigger than the pessimistic bound based on the alphabet size. We discuss this application in full detail in a follow-up work.

3.3.5. Applications to Property Testing

The estimator from Algorithm 1 was first studied in [45], but the variance bounds obtained were not sharp. Quite oddly, in the ongoing research on closeness testing, the birthday-like collision estimators (being subroutines for uniformity checking) seemed to be suboptimal [46] until, very recently, the work of [37] re-examined the variance formula for $d=2$ and shows that it achieves (in our notation) optimal dependence on K and $ϵ$. Thus, a breakthrough was possible just because of a specialized version of (8). In this discussion, we would like to (a) point out that the general variance formula can likely have similar applications and impact for $d>2$ and should be of broader interest, and (b) comment on a minor gap in an early version of the proof of the central result in [37]. Lemma 2.3 in [37], which establishes the variance bound for $d=2$, is the key ingredient of the main results. The authors derive an expression bounding, in our notation, the variance in Theorem 1 for the case $d=2$. When doing so, they face up the term $n(n-1)(n-2)(P_3-P_2^2)$ (in our notation) and bound it as $n^3(P_3-P_2^2)$ (the last line of derivation claims the upper bound, the following remark claims the lower bound). The reasoning, however, is valid when $P_3-P_2^2$ is non-negative. This is true by Proposition 3.

3.4. Application to Statistical Inference

We will use Algorithm 1 to efficiently test whether a given distribution is close or far from a uniform one. The procedure described below is asymptotically equivalent but numerically superior to the one from [37].

Denote by K the alphabet size and let $\gamma$ be such that $P_2=\frac{1}{K}+\gamma$. We see that

$$\begin{array}{c}\hfill \gamma =\sum _x{\left(p_X\left(x\right)-\frac{1}{K}\right)}^2,\end{array}$$

(14)

which shows that $\gamma$ measures the squared ${\ell }_2$-distance between $p_X$ and the uniform distribution. For convenience, we will refer to $\gamma$ as the collision gap. Define

$$\begin{array}{c}\hfill \widehat{\gamma }=\widehat{P_2}-\frac{1}{K}.\end{array}$$

(15)

This estimator gives an unbiased approximation of $\gamma$, because $\mathbf{E}\left[\widehat{\gamma }\right]=\mathbf{E}\left[\widehat{P_2}\right]-\frac{1}{K}=P_2-\frac{1}{K}=\gamma$. Furthermore, its variance equals the variance of $\widehat{P_2}$ because $\frac{1}{K}$ is a deterministic constant. By Chebyshev’s inequality, with probability $1-\delta$, we have

$$\begin{array}{c}\hfill |\widehat{\gamma }-\gamma |⩽\sqrt{\mathbf{Var}\left[\widehat{P_2}\right]/\delta }.\end{array}$$

(16)

We now analyze the variance in more detail. Define $\delta \left(x\right)=p_X\left(x\right)-\frac{1}{K}$. Then $P_2=\frac{1}{K}+{\sum }_x\delta {\left(x\right)}^2$ and $P_3-P_2^2={\sum }_x\delta {\left(x\right)}^3+\frac{1}{K}{\sum }_x\delta {\left(x\right)}^2-{\left({\sum }_x\delta {\left(x\right)}^2\right)}^2$. In particular, we have $P_2=\frac{1}{K}+\gamma$ and $0⩽P_3-P_2^2⩽{\gamma }^{\frac{3}{2}}+\frac{\gamma }{K}$ by Propositions 3 and 4. Therefore, by Theorem 1 we obtain

$$\begin{array}{c}\hfill \mathbf{Var}\left[\widehat{P_d}-\frac{1}{K}\right]⩽\frac{4(\frac{\gamma }{K}+{\gamma }^{\frac{3}{2}})}{n}+\frac{\gamma +\frac{1}{K}}{\left(\genfrac{}{}{0pt}{}{n}{2}\right)}.\end{array}$$

(17)

Thus, with probability $1-\delta$, we have

$$\begin{array}{c}\hfill \gamma -\sqrt{\frac{4(\frac{\gamma }{K}+{\gamma }^{\frac{3}{2}})}{n\delta }+\frac{\gamma +\frac{1}{K}}{\left(\genfrac{}{}{0pt}{}{n}{2}\right)\delta }}⩽\widehat{\gamma }⩽\gamma +\sqrt{\frac{4(\frac{\gamma }{K}+{\gamma }^{\frac{3}{2}})}{n\delta }+\frac{\gamma +\frac{1}{K}}{\left(\genfrac{}{}{0pt}{}{n}{2}\right)\delta }}.\end{array}$$

(18)

The above two-sided inequality allows us to estimate a range of possible values $\gamma$ with respect to the (observed) statistics $\widehat{\gamma }$, which yields high-confidence bounds for $\gamma$.

We illustrate the procedure numerically on real-world datasets. Data and results are summarized in Figure 1a,b. Our method confirms non-uniformity in both cases and provides confidence intervals. The details of the experiment are shared in the supplementary material [41].

3.5. Application to Entropy Estimation

The following experiment illustrates advantages of adaptive entropy estimation for distributions with large support and relatively low entropy, such as Zipf’s law.

Let X follow Zipf’s law with parameter $s=1.1$ and the support of $K={10}^4$ elements. By numerical calculations, we find that $P_2\approx 0.40005$. Consider now the task of estimating entropy of X from samples. Theorem 3 allows us to save a large factor of about $K^{1/2}={10}^2$ in the number of samples. Calculations show that on a sample of size about $n=10,000$, the algorithm from Algorithm 2 finds an approximation ${\widehat{P}}_d=0.39898$ with a relative error $ϵ=\frac{1}{2}$ and confidence $1-\delta =0.95$. The details appear in the supplementary material [41].

4. Conclusions

This work simplifies the variance analysis of collision estimators, establishing the closed-form exact formulas and improving upon prior data-oblivious bounds by making them dependent on certain data statistics. In particular, we use the derived formulas to estimate Rényi entropy adaptively, asymptotically, and give other applications.

Numerical experiments highlight the importance of the dependency of sample size on confidence. The constants involved exponentially affect the confidence, so that further improvements are of significance for many real-world inference problems. This problem is left for future research.