Tight Bounds on the Rényi Entropy via Majorization with Applications to Guessing and Compression

Abstract

This paper provides tight bounds on the Rényi entropy of a function of a discrete random variable with a finite number of possible values, where the considered function is not one to one. To that end, a tight lower bound on the Rényi entropy of a discrete random variable with a finite support is derived as a function of the size of the support, and the ratio of the maximal to minimal probability masses. This work was inspired by the recently published paper by Cicalese et al., which is focused on the Shannon entropy, and it strengthens and generalizes the results of that paper to Rényi entropies of arbitrary positive orders. In view of these generalized bounds and the works by Arikan and Campbell, non-asymptotic bounds are derived for guessing moments and lossless data compression of discrete memoryless sources.

1. Introduction

Majorization theory is a simple and productive concept in the theory of inequalities, which also unifies a variety of familiar bounds [1,2]. These mathematical tools find various applications in diverse fields (see, e.g., [3]) such as economics [2,4,5], combinatorial analysis [2,6], geometric inequalities [2], matrix theory [2,6,7,8], Shannon theory [5,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25], and wireless communications [26,27,28,29,30,31,32,33].

This work, which relies on the majorization theory, has been greatly inspired by the recent insightful paper by Cicalese et al. [12] (the research work in the present paper has been initialized while the author handled [12] as an associate editor). The work in [12] provides tight bounds on the Shannon entropy of a function of a discrete random variable with a finite number of possible values, where the considered function is not one to one. For that purpose, and while being of interest by its own right (see [12], Section 6), a tight lower bound on the Shannon entropy of a discrete random variable with a finite support was derived in [12] as a function of the size of the support, and the ratio of the maximal to minimal probability masses. The present paper aims to extend the bounds in [12] to Rényi entropies of arbitrary positive orders (note that the Shannon entropy is equal to the Rényi entropy of order 1), and to study the information-theoretic applications of these (non-trivial) generalizations in the context of non-asymptotic analysis of guessing moments and lossless data compression.

The motivation for this work is rooted in the diverse information-theoretic applications of Rényi measures [34]. These include (but are not limited to) asymptotically tight bounds on guessing moments [35], information-theoretic applications such as guessing subject to distortion [36], joint source-channel coding and guessing with application to sequential decoding [37], guessing with a prior access to a malicious oracle [38], guessing while allowing the guesser to give up and declare an error [39], guessing in secrecy problems [40,41], guessing with limited memory [42], and guessing under source uncertainty [43]; encoding tasks [44,45]; Bayesian hypothesis testing [9,22,23], and composite hypothesis testing [46,47]; Rényi generalizations of the rejection sampling problem in [48], motivated by the communication complexity in distributed channel simulation, where these generalizations distinguish between causal and noncausal sampler scenarios [49]; Wyner’s common information in distributed source simulation under Rényi divergence measures [50]; various other source coding theorems [23,39,51,52,53,54,55,56,57,58], channel coding theorems [23,58,59,60,61,62,63,64], including coding theorems in quantum information theory [65,66,67].

The presentation in this paper is structured as follows: Section 2 provides notation and essential preliminaries for the analysis in this paper. Section 3 and Section 4 strengthen and generalize, in a non-trivial way, the bounds on the Shannon entropy in [12] to Rényi entropies of arbitrary positive orders (see Theorems 1 and 2). Section 5 relies on the generalized bound from Section 4 and the work by Arikan [35] to derive non-asymptotic bounds for guessing moments (see Theorem 3); Section 5 also relies on the generalized bound in Section 4 and the source coding theorem by Campbell [51] (see Theorem 4) for the derivation of non-asymptotic bounds for lossless compression of discrete memoryless sources (see Theorem 5).

2. Notation and Preliminaries

Let

• P be a probability mass function defined on a finite set $\mathcal{X}$;
• $p_{\max }$ and $p_{\min }$ be, respectively, the maximal and minimal positive masses of P;
• $G_P\left(k\right)$ be the sum of the k largest masses of P for $k\in \{1,\dots ,|\mathcal{X}\left|\right\}$ (note that $G_P\left(1\right)=p_{\max }$ and $G_P\left(\right|\mathcal{X}\left|\right)=1$);
• ${\mathcal{P}}_n$, for an integer $n\ge 2$, be the set of all probability mass functions defined on $\mathcal{X}$ with $\left|\mathcal{X}\right|=n$; without any loss of generality, let $\mathcal{X}=\{1,\dots ,n\}$;
• ${\mathcal{P}}_n\left(\rho \right)$, for $\rho \ge 1$ and an integer $n\ge 2$, be the subset of all probability measures $P\in {\mathcal{P}}_n$ such that

  $$\begin{array}{c}\hfill \frac{p_{\max }}{p_{\min }}\le \rho .\end{array}$$

  (1)

Definition 1 (Majorization).

Consider discrete probability mass functions P and Q defined on the same (finite or countably infinite) set $\mathcal{X}$. It is said that P is majorized by Q (or Q majorizes P), and it is denoted by $P\prec Q$, if $G_P\left(k\right)\le G_Q\left(k\right)$ for all $k\in \{1,\dots ,|\mathcal{X}|-1\}$ (recall that $G_P\left(\right|\mathcal{X}\left|\right)=G_Q\left(\right|\mathcal{X}\left|\right)=1$). If P and Q are defined on finite sets of different cardinalities, then the probability mass function which is defined over the smaller set is first padded by zeros for making the cardinalities of these sets be equal.

By Definition 1, a unit mass majorizes any other distribution; on the other hand, the equiprobable distribution on a finite set is majorized by any other distribution defined on the same set.

Definition 2 (Schur-convexity/concavity).

A function $f:{\mathcal{P}}_n\to \mathbb{R}$ is said to be Schur-convex if for every $P,Q\in {\mathcal{P}}_n$ such that $P\prec Q$, we have $f\left(P\right)\le f\left(Q\right)$. Likewise, f is said to be Schur-concave if $-f$ is Schur-convex, i.e., $P,Q\in {\mathcal{P}}_n$ and $P\prec Q$ imply that $f\left(P\right)\ge f\left(Q\right)$.

Definition 3 

(Rényi entropy [34]). Let X be a random variable taking values on a finite or countably infinite set $\mathcal{X}$, and let $P_X$ be its probability mass function. The Rényi entropy of order $\alpha \in (0,1)\cup (1,\infty )$ is given by

$$\begin{array}{cc}\hfill H_{\alpha }\left(X\right)& =H_{\alpha }\left(P_X\right)=\frac{1}{1-\alpha }\log \left(\sum _{x\in \mathcal{X}}{P}_X^{\alpha }\left(x\right)\right).\hfill \end{array}$$

(2)

Unless explicitly stated, the logarithm base can be chosen by the reader, with exp indicating the inverse function of log.

By its continuous extension,

$$\begin{array}{c}{H}_0\left(X\right)=\log \left|\{x\in \mathcal{X}:P_X\left(x\right)>0\}\right|,\hfill \end{array}$$

(3)

$$\begin{array}{c}{H}_1\left(X\right)=H\left(X\right),\hfill \end{array}$$

(4)

$$\begin{array}{c}{H}_{\infty }\left(X\right)=\log \frac{1}{p_{\max }}\hfill \end{array}$$

(5)

where $H\left(X\right)$ is the (Shannon) entropy of X.

Proposition 1

(Schur-concavity of the Rényi entropy (Appendix F.3.a (p. 562) of [2])). The Rényi entropy of an arbitrary order $\alpha >0$ is Schur-concave; in particular, for $\alpha =1$, the Shannon entropy is Schur-concave.

Remark 1.

[17] (Theorem 2) strengthens Proposition 1, though it is not needed for our analysis.

Definition 4 

(Rényi divergence [34]). Let P and Q be probability mass functions defined on a finite or countably infinite set $\mathcal{X}$. The Rényi divergence of order $\alpha \in [0,\infty ]$ is defined as follows:

• If $\alpha \in (0,1)\cup (1,\infty )$, then

  $$\begin{array}{c}\hfill D_{\alpha }(P\Vert Q)=\frac{1}{\alpha -1}\log \sum _{x\in \mathcal{X}}{P}^{\alpha }\left(x\right)Q^{1-\alpha }\left(x\right).\end{array}$$

  (6)
• By the continuous extension of $D_{\alpha }(P\Vert Q)$,

  $$\begin{array}{cc}\hfill D_0(P\Vert Q)& =\underset{\mathcal{A}:P\left(\mathcal{A}\right)=1}{\max }\log \frac{1}{Q\left(\mathcal{A}\right)},\hfill \end{array}$$

  (7)

  $$\begin{array}{cc}\hfill D_1(P\Vert Q)& =D(P\Vert Q),\hfill \end{array}$$

  (8)

  $$\begin{array}{cc}\hfill D_{\infty }(P\Vert Q)& =\log \underset{x\in \mathcal{X}}{\sup }\frac{P\left(x\right)}{Q\left(x\right)},\hfill \end{array}$$

  (9)

  where $D(P\Vert Q)$ in the right side of (8) is the relative entropy (a.k.a. Kullback-Leibler divergence).

Throughout this paper, for $a\in \mathbb{R}$, $\lceil a\rceil$ denotes the ceiling of a (i.e., the smallest integer not smaller than the real number a), and $\lfloor a\rfloor$ denotes the flooring of a (i.e., the greatest integer not greater than a).

3. A Tight Lower Bound on the Rényi Entropy

We provide in this section a tight lower bound on the Rényi entropy, of an arbitrary order $\alpha >0$, when the probability mass function of the discrete random variable is defined on a finite set of cardinality n, and the ratio of the maximal to minimal probability masses is upper bounded by an arbitrary fixed value $\rho \in [1,\infty )$. In other words, we derive the largest possible gap between the order-$\alpha$ Rényi entropies of an equiprobable distribution and a non-equiprobable distribution (defined on a finite set of the same cardinality) with a given value for the ratio of the maximal to minimal probability masses. The basic tool used for the development of our result in this section relies on the majorization theory. Our result strengthens the result in [12] (Theorem 2) for the Shannon entropy, and it further provides a generalization for the Rényi entropy of an arbitrary order $\alpha >0$ (recall that the Shannon entropy is equal to the Rényi entropy of order $\alpha =1$, see (4)). Furthermore, the approach for proving the main result in this section differs significantly from the proof in [12] for the Shannon entropy. The main result in this section is a key result for all what follows in this paper.

The following lemma is a restatement of [12] (Lemma 6).

Lemma 1.

Let $P\in {\mathcal{P}}_n\left(\rho \right)$ with $\rho \ge 1$ and an integer $n\ge 2$, and assume without any loss of generality that the probability mass function P is defined on the set $\mathcal{X}=\{1,\dots ,n\}$. Let $Q\in {\mathcal{P}}_n$ be defined on $\mathcal{X}$ as follows:

$$Q\left(j\right)=\{\begin{array}{cc}\rho p_{\min },\hfill & j\in \{1,\dots ,i\},\hfill \\ 1-(n+i\rho -i-1)p_{\min },\hfill & j=i+1,\hfill \\ p_{\min },\hfill & j\in \{i+2,\dots ,n\}\hfill \end{array}$$

(10)

where

$$\begin{array}{c}\hfill i:=⌊\frac{1-n{p}_{\min }}{(\rho -1)p_{\min }}⌋.\end{array}$$

(11)

Then,

(1) 

$Q\in {\mathcal{P}}_n\left(\rho \right)$, and $Q\left(1\right)\ge Q\left(2\right)\ge \dots \ge Q\left(n\right)>0$;

(2) 

$P\prec Q$.

Proof. 

See [12] (p. 2236) (top of the second column). ☐

Lemma 2.

Let $\rho >1$, $\alpha >0$, and $n\ge 2$ be an integer. For

$$\begin{array}{c}\hfill \beta \in \left[\frac{1}{1+(n-1)\rho },\frac{1}{n}\right]:={\Gamma }_{\rho }^{\left(n\right)},\end{array}$$

(12)

let $Q_{\beta }\in {\mathcal{P}}_n\left(\rho \right)$ be defined on $\mathcal{X}=\{1,\dots ,n\}$ as follows:

$$Q_{\beta }\left(j\right)=\{\begin{array}{cc}\rho \beta ,\hfill & j\in \{1,\dots ,i_{\beta }\},\hfill \\ 1-(n+i_{\beta }\rho -i_{\beta }-1)\beta ,\hfill & j=i_{\beta }+1,\hfill \\ \beta ,\hfill & j\in \{i_{\beta }+2,\dots ,n\}\hfill \end{array}$$

(13)

where

$$\begin{array}{c}\hfill i_{\beta }:=⌊\frac{1-n\beta }{(\rho -1)\beta }⌋.\end{array}$$

(14)

Then, for every $\alpha >0$,

$$\begin{array}{c}\hfill \underset{P\in {\mathcal{P}}_n\left(\rho \right)}{\min }{H}_{\alpha }\left(P\right)=\underset{\beta \in {\Gamma }_{\rho }^{\left(n\right)}}{\min }{H}_{\alpha }\left(Q_{\beta }\right).\end{array}$$

(15)

Proof. 

See Appendix A. ☐

Lemma 3.

For $\rho >1$ and $\alpha >0$, let

$$\begin{array}{c}\hfill c_{\alpha }^{\left(n\right)}\left(\rho \right):=\log n-\underset{P\in {\mathcal{P}}_n\left(\rho \right)}{\min }{H}_{\alpha }\left(P\right),n=2,3,\dots \end{array}$$

(16)

with $c_{\alpha }^{\left(1\right)}\left(\rho \right):=0$. Then, for every $n\in \mathbb{N}$,

$$\begin{array}{c}0\le c_{\alpha }^{\left(n\right)}\left(\rho \right)\le \log \rho ,\hfill \end{array}$$

(17)

$$\begin{array}{c}{c}_{\alpha }^{\left(n\right)}\left(\rho \right)\le c_{\alpha }^{\left(2n\right)}\left(\rho \right),\hfill \end{array}$$

(18)

and $c_{\alpha }^{\left(n\right)}\left(\rho \right)$ is monotonically increasing in $\alpha \in [0,\infty ]$.

Proof. 

See Appendix B. ☐

Lemma 4.

For $\alpha >0$ and $\rho >1$, the limit

$$\begin{array}{cc}\hfill c_{\alpha }^{(\infty )}\left(\rho \right)& :=\underset{n\to \infty }{\lim }{c}_{\alpha }^{\left(n\right)}\left(\rho \right)\hfill \end{array}$$

(19)

exists, having the following properties:

(a) 

If $\alpha \in (0,1)\cup (1,\infty )$, then

$$\begin{array}{c}\hfill c_{\alpha }^{(\infty )}\left(\rho \right)=\frac{1}{\alpha -1}\log \left(1+\frac{1+\alpha (\rho -1)-{\rho }^{\alpha }}{(1-\alpha )(\rho -1)}\right)-\frac{\alpha }{\alpha -1}\log \left(1+\frac{1+\alpha (\rho -1)-{\rho }^{\alpha }}{(1-\alpha )({\rho }^{\alpha }-1)}\right),\end{array}$$

(20)

and

$$\begin{array}{c}\hfill \underset{\alpha \to \infty }{\lim }{c}_{\alpha }^{(\infty )}\left(\rho \right)=\log \rho .\end{array}$$

(21)

(b) 

If $\alpha =1$, then

$$\begin{array}{c}\hfill c_1^{(\infty )}\left(\rho \right)=\underset{\alpha \to 1}{\lim }{c}_{\alpha }^{(\infty )}\left(\rho \right)=\frac{\rho \log \rho }{\rho -1}-\log \left(\frac{\mathrm{e}\rho {\log }_{\mathrm{e}}\rho }{\rho -1}\right).\end{array}$$

(22)

(c) 

For all $\alpha >0$,

$$\begin{array}{c}\hfill \underset{\rho \downarrow 1}{\lim }{c}_{\alpha }^{(\infty )}\left(\rho \right)=0.\end{array}$$

(23)

• For every $n\in \mathbb{N}$, $\alpha >0$ and $\rho \ge 1$,

  $$\begin{array}{c}\hfill 0\le c_{\alpha }^{\left(n\right)}\left(\rho \right)\le c_{\alpha }^{\left(2n\right)}\left(\rho \right)\le c_{\alpha }^{(\infty )}\left(\rho \right)\le \log \rho .\end{array}$$

  (24)

Proof. 

See Appendix C. ☐

In view of Lemmata 1–4, we obtain the following main result in this section:

Theorem 1.

Let $\alpha >0$, $\rho >1$, $n\ge 2$, and let $c_{\alpha }^{\left(n\right)}\left(\rho \right)$ in (16) designate the maximal gap between the order-α Rényi entropies of equiprobable and arbitrary distributions in ${\mathcal{P}}_n\left(\rho \right)$. Then,

(a) 

The non-negative sequence ${\{c_{\alpha }^{\left(n\right)}\left(\rho \right)\}}_{n=2}^{\infty }$ can be calculated by the real-valued single-parameter optimization in the right side of (15).

(b) 

The asymptotic limit as $n\to \infty$, denoted by $c_{\alpha }^{(\infty )}\left(\rho \right)$, admits the closed-form expressions in (20) and (22), and it satisfies the properties in (21), (23) and (24).

Remark 2.

Setting $\alpha =2$ in Theorem 1 gives that, for all $P\in {\mathcal{P}}_n\left(\rho \right)$ (with $\rho >1$, and an integer $n\ge 2$),

$$\begin{array}{cc}\hfill H_2\left(P\right)& \ge \log n-c_2^{\left(n\right)}\left(\rho \right)\hfill \end{array}$$

(25)

$$\begin{array}{cc}& \ge \log n-c_2^{(\infty )}\left(\rho \right)\hfill \end{array}$$

(26)

$$\begin{array}{cc}& =\log \frac{4\rho n}{{(1+\rho )}^2}\hfill \end{array}$$

(27)

where (25)–(27) hold, respectively, due to (16), (24) and (20). This strengthens the result in [68] (Proposition 2) which gives the same lower bound as in the right side of (27) for $H\left(P\right)$ rather than for $H_2\left(P\right)$ (recall that $H\left(P\right)\ge H_2\left(P\right)$).

For a numerical illustration of Theorem 1, Figure 1 provides a plot of $c_{\alpha }^{(\infty )}\left(\rho \right)$ in (20) and (22) as a function of $\rho \ge 1$, confirming numerically the properties in (21) and (23). Furthermore, Figure 2 provides plots of $c_{\alpha }^{\left(n\right)}\left(\rho \right)$ in (16) as a function of $\alpha >0$, for $\rho =2$ (left plot) and $\rho =256$ (right plot), with several values of $n\ge 2$; the calculation of the curves in these plots relies on (15), (20) and (22), and they illustrate the monotonicity and boundedness properties in (24).

Remark 3.

Theorem 1 strengthens the result in [12] (Theorem 2) for the Shannon entropy (i.e., for $\alpha =1$), in addition to its generalization to Rényi entropies of arbitrary orders $\alpha >0$. This is because our lower bound on the Shannon entropy is given by

$$\begin{array}{c}\hfill H\left(P\right)\ge \log n-c_1^{\left(n\right)}\left(\rho \right),\forall P\in {\mathcal{P}}_n\left(\rho \right),\end{array}$$

(28)

whereas the looser bound in [12] is given by (see [12] ((7)) and (22) here)

$$\begin{array}{c}\hfill H\left(P\right)\ge \log n-c_1^{(\infty )}\left(\rho \right),\forall P\in {\mathcal{P}}_n\left(\rho \right),\end{array}$$

(29)

and we recall that $0\le c_1^{\left(n\right)}\left(\rho \right)\le c_1^{(\infty )}\left(\rho \right)$ (see (24)). Figure 3 shows the improvement in the new lower bound (28) over (29) by comparing $c_1^{(\infty )}\left(\rho \right)$ versus $c_1^{\left(n\right)}\left(\rho \right)$ for $\rho \in [1,{10}^5]$ and with several values of n. It is reflected from Figure 3 that there is a very marginal improvement in the lower bound on the Shannon entropy (28) over the bound in (29) if $\rho \le 30$ (even for small values of n), whereas there is a significant improvement over the bound in (29) for large values of ρ; by increasing the value of n, also the value of ρ needs to be increased for observing an improvement of the lower bound in (28) over (29) (see Figure 3).

An improvement of the bound in (28) over (29) leads to a tightening of the upper bound in [12] (Theorem 4) on the compression rate of Tunstall codes for discrete memoryless sources, which further tightens the bound by Jelinek and Schneider in [69] (Equation (9)). More explicitly, in view of [12] (Section 6), an improved upper bound on the compression rate of these variable-to-fixed lossless source codes is obtained by combining [12] (Equations (36) and (38)) with a tightened lower bound on the entropy $H\left(W\right)$ of the leaves of the tree graph for Tunstall codes. From (28), the latter lower bound is given by $H\left(W\right)\ge {\log }_{2}n-c_1^{\left(n\right)}\left(\rho \right)$ where $c_1^{\left(n\right)}\left(\rho \right)$ is expressed in bits, $\rho :=\frac{1}{p_{\min }}$ is the reciprocal of the minimal positive probability of the source symbols, and n is the number of codewords (so, all codewords are of length $\lceil {\log }_{2}n\rceil$ bits). This yields a reduction in the upper bound on the non-asymptotic compression rate R of Tunstall codes from $\frac{\lceil {\log }_{2}n\rceil H\left(X\right)}{{\log }_{2}n-c_1^{(\infty )}\left(\rho \right)}$ (see [12] (Equation (40)) and (22)) to $\frac{\lceil {\log }_{2}n\rceil H\left(X\right)}{{\log }_{2}n-c_1^{\left(n\right)}\left(\rho \right)}$ bits per source symbol where $H\left(X\right)$ denotes the source entropy (converging, in view of (17), to $H\left(X\right)$ as we let $n\to \infty$).

Remark 4.

Equality (15) with the minimizing probability mass function of the form (13) holds, in general, by replacing the Rényi entropy with an arbitrary Schur-concave function (as it can be easily verified from the proof of Lemma 2 in Appendix A). However, the analysis leading to Lemmata 3–4 and Theorem 1 applies particularly to the Rényi entropy.

4. Bounds on the Rényi Entropy of a Function of a Discrete Random Variable

This section relies on Theorem 1 and majorization for extending [12] (Theorem 1), which applies to the Shannon entropy, to Rényi entropies of any positive order. More explicitly, let $\alpha \in (0,\infty )$ and

• $\mathcal{X}$ and $\mathcal{Y}$ be finite sets of cardinalities $\left|\mathcal{X}\right|=n$ and $\left|\mathcal{Y}\right|=m$ with $n>m\ge 2$; without any loss of generality, let $\mathcal{X}=\{1,\dots ,n\}$ and $\mathcal{Y}=\{1,\dots ,m\}$;
• X be a random variable taking values on $\mathcal{X}$ with a probability mass function $P_X\in {\mathcal{P}}_n$;
• ${\mathcal{F}}_{n,m}$ be the set of deterministic functions $f:\mathcal{X}\to \mathcal{Y}$; note that $f\in {\mathcal{F}}_{n,m}$ is not one to one since $m<n$.

The main result in this section sharpens the inequality $H_{\alpha }\left(f\left(X\right)\right)\le H_{\alpha }\left(X\right)$, for every deterministic function $f\in {\mathcal{F}}_{n,m}$ with $n>m\ge 2$ and $\alpha >0$, by obtaining non-trivial upper and lower bounds on $\underset{f\in {\mathcal{F}}_{n,m}}{\max }{H}_{\alpha }\left(f\left(X\right)\right)$. The calculation of the exact value of $\underset{f\in {\mathcal{F}}_{n,m}}{\min }{H}_{\alpha }\left(f\left(X\right)\right)$ is much easier, and it is expressed in closed form by capitalizing on the Schur-concavity of the Rényi entropy.

The following main result extends [12] (Theorem 1) to Rényi entropies of arbitrary positive orders.

Theorem 2.

Let $X\in \{1,\dots ,n\}$ be a random variable which satisfies $P_X\left(1\right)\ge P_X\left(2\right)\ge \dots \ge P_X\left(n\right)$.

(a) 

For $m\in \{2,\dots ,n-1\}$, if $P_X\left(1\right)<\frac{1}{m}$, let ${\tilde{X}}_m$ be the equiprobable random variable on $\{1,\dots ,m\}$; otherwise, if $P_X\left(1\right)\ge \frac{1}{m}$, let ${\tilde{X}}_m\in \{1,\dots ,m\}$ be a random variable with the probability mass function

$$P_{{\tilde{X}}_m}\left(i\right)=\{\begin{array}{cc}{P}_X\left(i\right),\hfill & i\in \{1,\dots ,n^{\ast }\},\hfill \\ \frac{1}{m-n^{\ast }}{\displaystyle \sum _{j=n^{\ast }+1}^n}{P}_X\left(j\right),\hfill & i\in \{n^{\ast }+1,\dots ,m\},\hfill \end{array}$$

(30)

where $n^{\ast }$ is the maximal integer $i\in \{1,\dots ,m-1\}$ such that

$$\begin{array}{c}\hfill P_X\left(i\right)\ge \frac{1}{m-i}\sum _{j=i+1}^n{P}_X\left(j\right).\end{array}$$

(31)

Then, for every $\alpha >0$,

$$\begin{array}{c}\hfill \underset{f\in {\mathcal{F}}_{n,m}}{\max }{H}_{\alpha }\left(f\left(X\right)\right)\in \left[H_{\alpha }\left({\tilde{X}}_m\right)-v\left(\alpha \right),H_{\alpha }\left({\tilde{X}}_m\right)\right],\end{array}$$

(32)

where

$$v\left(\alpha \right):=c_{\alpha }^{(\infty )}\left(2\right)=\{\begin{array}{cc}\log \left(\frac{\alpha -1}{2^{\alpha }-2}\right)-\frac{\alpha }{\alpha -1}\log \left(\frac{\alpha }{2^{\alpha }-1}\right),\hfill & \alpha \ne 1,\hfill \\ \log \left(\frac{2}{e\ln 2}\right)\approx 0.08607bits,\hfill & \alpha =1.\hfill \end{array}$$

(33)

(b) 

There exists an explicit construction of a deterministic function $f^{\ast }\in {\mathcal{F}}_{n,m}$ such that

$$\begin{array}{c}\hfill H_{\alpha }\left(f^{\ast }\left(X\right)\right)\in \left[H_{\alpha }\left({\tilde{X}}_m\right)-v\left(\alpha \right),H_{\alpha }\left({\tilde{X}}_m\right)\right]\end{array}$$

(34)

where $f^{\ast }$ is independent of α, and it is obtained by using Huffman coding (as in [12] for $\alpha =1$).

(c) 

Let ${\tilde{Y}}_m\in \{1,\dots ,m\}$ be a random variable with the probability mass function

$$P_{{\tilde{Y}}_m}\left(i\right)=\{\begin{array}{cc}{\displaystyle \sum _{k=1}^{n-m+1}}{P}_X\left(k\right),\hfill & i=1,\hfill \\ P_X(n-m+i),\hfill & i\in \{2,\dots ,m\}.\hfill \end{array}$$

(35)

Then, for every $\alpha >0$,

$$\begin{array}{c}\hfill \underset{f\in {\mathcal{F}}_{n,m}}{\min }{H}_{\alpha }\left(f\left(X\right)\right)=H_{\alpha }\left({\tilde{Y}}_m\right).\end{array}$$

(36)

Remark 5.

Setting $\alpha =1$ specializes Theorem 2 to [12] (Theorem 1) (regarding the Shannon entropy). This point is further elaborated in Remark 8, after the proof of Theorem 2.

Remark 6.

Similarly to [12] (Lemma 1), an exact solution of the maximization problem in the left side of (32) is strongly NP-hard [70]; this means that, unless $P=NP$, there is no polynomial time algorithm which, for an arbitrarily small $\epsilon >0$, computes an admissible deterministic function $f_{\epsilon }\in {\mathcal{F}}_{n,m}$ such that

$$\begin{array}{c}\hfill H_{\alpha }\left(f_{\epsilon }\left(X\right)\right)\ge (1-\epsilon )\underset{f\in {\mathcal{F}}_{n,m}}{\max }{H}_{\alpha }\left(f\left(X\right)\right).\end{array}$$

(37)

This motivates the derivation of the bounds in (32), and the simple construction of a deterministic function $f^{\ast }\in {\mathcal{F}}_{n,m}$ achieving (34).

A proof of Theorem 2 relies on the following lemmata.

Lemma 5.

Let $X\in \{1,\dots ,n\}$, $m<n$ and $\alpha >0$. Then,

$$\begin{array}{c}\hfill \underset{Q\in {\mathcal{P}}_m:P_X\prec Q}{\max }{H}_{\alpha }\left(Q\right)=H_{\alpha }\left({\tilde{X}}_m\right)\end{array}$$

(38)

where the probability mass function of ${\tilde{X}}_m$ is given in (30).

Proof. 

Since $P_X\prec P_{{\tilde{X}}_m}$ (see [12] (Lemma 2)) with $P_{{\tilde{X}}_m}\in {\mathcal{P}}_m$, and $P_{{\tilde{X}}_m}\prec Q$ for all $Q\in {\mathcal{P}}_m$ such that $P_X\prec Q$ (see [12] (Lemma 4)), the result follows from the Schur-concavity of the Rényi entropy. ☐

Lemma 6.

Let $X\in \{1,\dots ,n\}$, $\alpha >0$, and $f\in {\mathcal{F}}_{n,m}$ with $m<n$. Then,

$$\begin{array}{c}\hfill H_{\alpha }\left(f\left(X\right)\right)\le H_{\alpha }\left({\tilde{X}}_m\right).\end{array}$$

(39)

Proof. 

Since f is a deterministic function in ${\mathcal{F}}_{n,m}$ with $m<n$, the probability mass function of $f\left(X\right)$ is an element in ${\mathcal{P}}_m$ which majorizes $P_X$ (see [12] (Lemma 3)). Inequality (39) then follows from Lemma 5. ☐

We are now ready to prove Theorem 2.

Proof. 

In view of (39),

$$\begin{array}{c}\hfill \underset{f\in {\mathcal{F}}_{n,m}}{\max }{H}_{\alpha }\left(f\left(X\right)\right)\le H_{\alpha }\left({\tilde{X}}_m\right).\end{array}$$

(40)

We next construct a function $f^{\ast }\in {\mathcal{F}}_{n,m}$ such that, for all $\alpha >0$,

$$\begin{array}{cc}\hfill H_{\alpha }\left(f^{\ast }\left(X\right)\right)& \ge \underset{Q\in {\mathcal{P}}_m:P_X\prec Q}{\max }{H}_{\alpha }\left(Q\right)-v\left(\alpha \right)\hfill \end{array}$$

(41)

$$\begin{array}{c}\ge \underset{f\in {\mathcal{F}}_{n,m}}{\max }{H}_{\alpha }\left(f\left(X\right)\right)-v\left(\alpha \right)\hfill \end{array}$$

(42)

where the function $v:(0,\infty )\to (0,\infty )$ in the right side of (41) is given in (33), and (42) holds due to (38) and (40). The function $f^{\ast }$ in our proof coincides with the construction in [12], and it is, therefore, independent of $\alpha$.

We first review and follow the concept of the proof of [12] (Lemma 5), and we then deviate from the analysis there for proving our result. The idea behind the proof of [12] (Lemma 5) relies on the following algorithm:

(1)

Start from the probability mass function $P_X\in {\mathcal{P}}_n$ with $P_X\left(1\right)\ge \dots \ge P_X\left(n\right)$;

(2)

Merge successively pairs of probability masses by applying the Huffman algorithm;

(3)

Stop the merging process in Step 2 when a probability mass function $Q\in {\mathcal{P}}_m$ is obtained (with $Q\left(1\right)\ge \dots \ge Q\left(m\right)$);

(4)

Construct the deterministic function $f^{\ast }\in {\mathcal{F}}_{n,m}$ by setting $f^{\ast }\left(k\right)=j\in \{1,\dots ,m\}$ for all probability masses $P_X\left(k\right)$, with $k\in \{1,\dots ,n\}$, being merged in Steps 2–3 into the node of $Q\left(j\right)$.

Let $i\in \{0,\dots ,m-1\}$ be the largest index such that $P_X\left(1\right)=Q\left(1\right),\dots ,P_X\left(i\right)=Q\left(i\right)$ (note that $i=0$ corresponds to the case where each node $Q\left(j\right)$, with $j\in \{1,\dots ,m\}$, is constructed by merging at least two masses of the probability mass function $P_X$). Then, according to [12] (p. 2225),

$$\begin{array}{c}\hfill Q(i+1)\le 2Q\left(m\right).\end{array}$$

(43)

Let

$$\begin{array}{c}\hfill S:=\sum _{j=i+1}^{m}Q\left(j\right)\end{array}$$

(44)

be the sum of the $m-i$ smallest masses of the probability mass function Q. In view of (43), the vector

$$\begin{array}{c}\hfill \overline{Q}:=\left(\frac{Q(i+1)}{S},\dots ,\frac{Q\left(m\right)}{S}\right)\end{array}$$

(45)

represents a probability mass function where the ratio of its maximal to minimal masses is upper bounded by 2.

At this point, our analysis deviates from [12] (p. 2225). Applying Theorem 1 to $\overline{Q}$ with $\rho =2$ gives

$$\begin{array}{c}\hfill H_{\alpha }\left(\overline{Q}\right)\ge \log (m-i)-c_{\alpha }^{(\infty )}\left(2\right)\end{array}$$

(46)

with

$$\begin{array}{cc}\hfill c_{\alpha }^{(\infty )}\left(2\right)& =\frac{1}{\alpha -1}\log \left(1+\frac{1+\alpha -2^{\alpha }}{1-\alpha }\right)-\frac{\alpha }{\alpha -1}\log \left(1+\frac{1+\alpha -2^{\alpha }}{(1-\alpha )(2^{\alpha }-1)}\right)\hfill \end{array}$$

(47)

$$\begin{array}{c}=\log \left(\frac{\alpha -1}{2^{\alpha }-2}\right)-\frac{\alpha }{\alpha -1}\log \left(\frac{\alpha }{2^{\alpha }-1}\right)\hfill \end{array}$$

(48)

$$\begin{array}{c}=v\left(\alpha \right)\hfill \end{array}$$

(49)

where (47) follows from (20); (48) is straightforward algebra, and (49) is the definition in (33).

For $\alpha \in (0,1)\cup (1,\infty )$, we get

$$\begin{array}{cc}\hfill H_{\alpha }\left(Q\right)& =\frac{1}{1-\alpha }\log \left(\sum _{j=1}^m{Q}^{\alpha }\left(j\right)\right)\hfill \end{array}$$

(50)

$$\begin{array}{cc}& =\frac{1}{1-\alpha }\log \left(\sum _{j=1}^i{Q}^{\alpha }\left(j\right)+\sum _{j=i+1}^m{Q}^{\alpha }\left(j\right)\right)\hfill \end{array}$$

(51)

$$\begin{array}{cc}& =\frac{1}{1-\alpha }\log \left(\sum _{j=1}^i{Q}^{\alpha }\left(j\right)+S^{\alpha }\exp \left((1-\alpha )H_{\alpha }\left(\overline{Q}\right)\right)\right)\hfill \end{array}$$

(52)

$$\begin{array}{cc}& \ge \frac{1}{1-\alpha }\log \left(\sum _{j=1}^i{Q}^{\alpha }\left(j\right)+S^{\alpha }\exp \left((1-\alpha )\left(\log (m-i)-v\left(\alpha \right)\right)\right)\right)\hfill \end{array}$$

(53)

$$\begin{array}{cc}& =\frac{1}{1-\alpha }\log \left(\sum _{j=1}^i{Q}^{\alpha }\left(j\right)+S^{\alpha }{(m-i)}^{1-\alpha }\exp \left((\alpha -1)v\left(\alpha \right)\right)\right)\hfill \end{array}$$

(54)

where (51) holds since $i\in \{0,\dots ,m-1\}$; (52) follows from (2) and (45); (53) holds by (46)–(49).

In view of (44), let $Q^{\ast }\in {\mathcal{P}}_m$ be the probability mass function which is given by

$$Q^{\ast }\left(j\right)=\{\begin{array}{cc}Q\left(j\right),\hfill & j=1,\dots ,i\hfill \\ \frac{S}{m-i},\hfill & j=i+1,\dots ,m.\hfill \end{array}$$

(55)

From (50)–(55), we get

$$\begin{array}{cc}\hfill H_{\alpha }\left(Q\right)& \ge \frac{1}{1-\alpha }\log \left(\sum _{j=1}^i{\left(Q^{\ast }\left(j\right)\right)}^{\alpha }+\sum _{j=i+1}^m{\left(Q^{\ast }\left(j\right)\right)}^{\alpha }\exp \left((\alpha -1)v\left(\alpha \right)\right)\right)\hfill \end{array}$$

(56)

$$\begin{array}{cc}& =\frac{1}{1-\alpha }\log \left(\sum _{j=1}^m{\left(Q^{\ast }\left(j\right)\right)}^{\alpha }+\sum _{j=i+1}^m{\left(Q^{\ast }\left(j\right)\right)}^{\alpha }\left(\exp \left((\alpha -1)v\left(\alpha \right)\right)-1\right)\right)\hfill \end{array}$$

(57)

$$\begin{array}{cc}& =H_{\alpha }\left(Q^{\ast }\right)+\frac{1}{1-\alpha }\log \left(1+T\left(\exp \left((\alpha -1)v\left(\alpha \right)\right)-1\right)\right)\hfill \end{array}$$

(58)

with

$$\begin{array}{c}\hfill T:=\frac{{\displaystyle \sum _{j=i+1}^m}{\left(Q^{\ast }\left(j\right)\right)}^{\alpha }}{{\displaystyle \sum _{j=1}^m}{\left(Q^{\ast }\left(j\right)\right)}^{\alpha }}\in [0,1].\end{array}$$

(59)

Since $T\in [0,1]$ and $v\left(\alpha \right)>0$ for $\alpha >0$, it can be verified from (56)–(58) that for $\alpha \in (0,1)\cup (1,\infty )$

$$\begin{array}{c}\hfill H_{\alpha }\left(Q\right)\ge H_{\alpha }\left(Q^{\ast }\right)-v\left(\alpha \right).\end{array}$$

(60)

The validity of (60) is extended to $\alpha =1$ by taking the limit $\alpha \to 1$ on both sides of this inequality, and due to the continuity of $v(·)$ in (33) at $\alpha =1$. Applying the majorization result $Q^{\ast }\prec P_{{\tilde{X}}_m}$ in [12] ((31)), it follows from (60) and the Schur-concavity of the Rényi entropy that, for all $\alpha >0$,

$$\begin{array}{c}\hfill H_{\alpha }\left(Q\right)\ge H_{\alpha }\left(Q^{\ast }\right)-v\left(\alpha \right)\ge H_{\alpha }\left({\tilde{X}}_m\right)-v\left(\alpha \right),\end{array}$$

(61)

which together with (40), prove Items a) and b) of Theorem 2 (note that, in view of the construction of the deterministic function $f^{\ast }\in {\mathcal{F}}_{n,m}$ in Step 4 of the above algorithm, we get $H_{\alpha }\left(f^{\ast }\left(X\right)\right)=H_{\alpha }\left(Q\right)$).

We next prove Item c). Equality (36) is due to the Schur-concavity of the Rényi entropy, and since we have

• $f\left(X\right)$ is an aggregation of X, i.e., the probability mass function $Q\in {\mathcal{P}}_m$ of $f\left(X\right)$ satisfies $Q\left(j\right)={\displaystyle {\displaystyle \sum _{i\in I_j}}}{P}_X\left(i\right)$ ($1\le j\le m$) where $I_1,\dots ,I_m$ partition $\{1,\dots ,n\}$ into m disjoint subsets as follows:

  $$\begin{array}{c}\hfill I_j:=\{i\in \{1,\dots ,n\}:f\left(i\right)=j\},j=1,\dots ,m;\end{array}$$

  (62)
• By the assumption $P_X\left(1\right)\ge P_X\left(2\right)\ge \dots \ge P_X\left(n\right)$, it follows that $Q\prec P_{{\tilde{Y}}_m}$ for every such $Q\in {\mathcal{P}}_m$;
• From (35), ${\tilde{Y}}_m=\tilde{f}\left(X\right)$ where the function $\tilde{f}\in {\mathcal{F}}_{n,m}$ is given by $\tilde{f}\left(k\right):=1$ for all $k\in \{1,\dots ,n-m+1\}$, and $\tilde{f}(n-m+i):=i$ for all $i\in \{2,\dots ,m\}$. Hence, $P_{{\tilde{Y}}_m}$ is an element in the set of the probability mass functions of $f\left(X\right)$ with $f\in {\mathcal{F}}_{n,m}$ which majorizes every other element from this set.

 ☐

Remark 7.

The solid line in the left plot of Figure 2 depicts $v\left(\alpha \right):=c_{\alpha }^{(\infty )}\left(2\right)$ in (33) for $\alpha >0$. In view of Lemma 4, and by the definition in (33), the function $v:(0,\infty )\to (0,\infty )$ is indeed monotonically increasing and continuous.

Remark 8.

Inequality (43) leads to the application of Theorem 1 with $\rho =2$ (see (46)). In the derivation of Theorem 2, we refer to $v\left(\alpha \right):=c_{\alpha }^{(\infty )}\left(2\right)$ (see (47)–(49)) rather than referring to $c_{\alpha }^{\left(n\right)}\left(2\right)$ (although, from (24), we have $0\le c_{\alpha }^{\left(n\right)}\left(2\right)\le v\left(\alpha \right)$ for all $\alpha >0$). We do so since, for $n\ge 16$, the difference between the curves of $c_{\alpha }^{\left(n\right)}\left(2\right)$ (as a function of $\alpha >0$) and the curve of $c_{\alpha }^{(\infty )}\left(2\right)$ is marginal (see the dashed and solid lines in the left plot of Figure 2), and also because the function v in (33) is expressed in a closed form whereas $c_{\alpha }^{\left(n\right)}\left(2\right)$ is subject to numerical optimization for finite n (see (15) and (16)). For this reason, Theorem 2 coincides with the result in [12] (Theorem 1) for the Shannon entropy (i.e., for $\alpha =1$) while providing a generalization of the latter result for Rényi entropies of arbitrary positive orders α. Theorem 1, however, both strengthens the bounds in [12] (Theorem 2) for the Shannon entropy with finite cardinality n (see Remark 3), and it also generalizes these bounds to Rényi entropies of all positive orders.

Remark 9.

The minimizing probability mass function in (35) to the optimization problem (36), and the maximizing probability mass function in (30) to the optimization problem (38) are in general valid when the Rényi entropy of a positive order is replaced by an arbitrary Schur-concave function. However, the main results in (32)–(34) hold particularly for the Rényi entropy.

Remark 10.

Theorem 2 makes use of the random variables denoted by ${\tilde{X}}_m$ and ${\tilde{Y}}_m$, rather than (more simply) $X_m$ and $Y_m$ respectively, because Section 5 considers i.i.d. samples ${\left\{X_i\right\}}_{i=1}^k$ and ${\left\{Y_i\right\}}_{i=1}^k$ with $X_i\sim P_X$ and $Y_i\sim P_Y$; note, however, that the probability mass functions of ${\tilde{X}}_m$ and ${\tilde{Y}}_m$ are different from $P_X$ and $P_Y$, respectively, and for that reason we make use of tilted symbols in the left sides of (30) and (35).

5. Information-Theoretic Applications: Non-Asymptotic Bounds for Lossless Compression and Guessing

Theorem 2 is applied in this section to derive non-asymptotic bounds for lossless compression of discrete memoryless sources and guessing moments. Each of the two subsections starts with a short background for making the presentation self-contained.

5.1. Guessing

5.1.1. Background

The problem of guessing discrete random variables has various theoretical and operational aspects in information theory (see [35,36,37,38,40,41,43,56,71,72,73,74,75,76,77,78,79,80,81]). The central object of interest is the distribution of the number of guesses required to identify a realization of a random variable X, taking values on a finite or countably infinite set $\mathcal{X}=\{1,\dots ,|\mathcal{X}\left|\right\}$, by successively asking questions of the form “Is X equal to x?” until the value of X is guessed correctly. A guessing function is a one-to-one function $g:\mathcal{X}\to \mathcal{X}$, which can be viewed as a permutation of the elements of $\mathcal{X}$ in the order in which they are guessed. The required number of guesses is therefore equal to $g\left(x\right)$ when $X=x$ with $x\in \mathcal{X}$.

Lower and upper bounds on the minimal expected number of required guesses for correctly identifying the realization of X, expressed as a function of the Shannon entropy $H\left(X\right)$, have been respectively derived by Massey [77] and by McEliece and Yu [78], followed by a derivation of improved upper and lower bounds by De Santis et al. [80]. More generally, given a probability mass function $P_X$ on $\mathcal{X}$, it is of interest to minimize the generalized guessing moment $\mathbb{E}\left[g^{\rho }\left(X\right)\right]={\displaystyle \sum _{x\in \mathcal{X}}}{P}_X\left(x\right)g^{\rho }\left(x\right)$ for $\rho >0$. For an arbitrary positive $\rho$, the $\rho$-th moment of the number of guesses is minimized by selecting the guessing function to be a ranking function $g_X$, for which $g_X\left(x\right)=\ell$ if $P_X\left(x\right)$ is the ℓ-th largest mass [77]. Although the tie breaking affects the choice of $g_X$, the distribution of $g_X\left(X\right)$ does not depend on how ties are resolved. Not only does this strategy minimize the average number of guesses, but it also minimizes the $\rho$-th moment of the number of guesses for every $\rho >0$. Upper and lower bounds on the $\rho$-th moment of ranking functions, expressed in terms of the Rényi entropies, were derived by Arikan [35], Boztaş [71], followed by recent improvements in the non-asymptotic regime by Sason and Verdú [56]. Although if $\left|\mathcal{X}\right|$ is small, it is straightforward to evaluate numerically the guessing moments, the benefit of bounds expressed in terms of Rényi entropies is particularly relevant when dealing with a random vector $X^k=(X_1,\dots ,X_k)$ whose letters belong to a finite alphabet $\mathcal{X}$; computing all the probabilities of the mass function $P_{X^k}$ over the set ${\mathcal{X}}^k$, and then sorting them in decreasing order for the calculation of the $\rho$-th moment of the optimal guessing function for the elements of ${\mathcal{X}}^k$ becomes infeasible even for moderate values of k. In contrast, regardless of the value of k, bounds on guessing moments which depend on the Rényi entropy are readily computable if for example, ${\left\{X_i\right\}}_{i=1}^k$ are independent; in which case, the Rényi entropy of the vector is equal to the sum of the Rényi entropies of its components. Arikan’s bounds in [35] are asymptotically tight for random vectors of length k as $k\to \infty$, thus providing the correct exponential growth rate of the guessing moments for sufficiently large k.

5.1.2. Analysis

We next analyze the following setup of guessing. Let ${\left\{X_i\right\}}_{i=1}^k$ be i.i.d. random variables where $X_1\sim P_X$ takes values on a finite set $\mathcal{X}$ with $\left|\mathcal{X}\right|=n$. To cluster the data [82] (see also [12] (Section 3.A) and references therein), suppose that each $X_i$ is mapped to $Y_i=f\left(X_i\right)$ where $f\in {\mathcal{F}}_{n,m}$ is an arbitrary deterministic function (independent of the index i) with $m<n$. Consequently, ${\left\{Y_i\right\}}_{i=1}^k$ are i.i.d., and each $Y_i$ takes values on a finite set $\mathcal{Y}$ with $\left|\mathcal{Y}\right|=m<\left|\mathcal{X}\right|$.

Let $g_{X^k}:{\mathcal{X}}^k\to \{1,\dots ,n^k\}$ and $g_{Y^k}:{\mathcal{Y}}^k\to \{1,\dots ,m^k\}$ be, respectively, the ranking functions of the random vectors $X^k=(X_1,\dots ,X_k)$ and $Y^k=(Y_1,\dots ,Y_k)$ by sorting in separate decreasing orders the probabilities $P_{X^k}\left(x^k\right)={\prod }_{i=1}^k{P}_X\left(x_i\right)$ for $x^k\in {\mathcal{X}}^k$, and $P_{Y^k}\left(y^k\right)={\prod }_{i=1}^k{P}_Y\left(y_i\right)$ for $y^k\in {\mathcal{Y}}^k$ where ties in both cases are resolved arbitrarily. In view of Arikan’s bounds on the $\rho$-th moment of ranking functions (see [35] (Theorem 1) for the lower bound, and [35] (Proposition 4) for the upper bound), since $|{\mathcal{X}}^k|=n^k$ and $|{\mathcal{Y}}^k|=m^k$, the following bounds hold for all $\rho >0$:

$$\begin{array}{c}\hfill \rho H_{\frac{1}{1+\rho }}\left(X\right)-\frac{\rho \log (1+k\ln n)}{k}\le \frac{1}{k}\log \mathbb{E}\left[g_{X^k}^{\rho }\left(X^k\right)\right]\le \rho H_{\frac{1}{1+\rho }}\left(X\right),\end{array}$$

(63)

$$\begin{array}{c}\hfill \rho H_{\frac{1}{1+\rho }}\left(Y\right)-\frac{\rho \log (1+k\ln m)}{k}\le \frac{1}{k}\log \mathbb{E}\left[g_{Y^k}^{\rho }\left(Y^k\right)\right]\le \rho H_{\frac{1}{1+\rho }}\left(Y\right).\end{array}$$

(64)

In the following, we rely on Theorem 2 and the bounds in (63) and (64) to obtain bounds on the exponential reduction of the $\rho$-th moment of the ranking function of $X^k$ as a result of its mapping to $Y^k$. First, the combination of (63) and (64) yields

$$\begin{array}{c}\rho \left[H_{\frac{1}{1+\rho }}\left(X\right)-H_{\frac{1}{1+\rho }}\left(Y\right)\right]-\frac{\rho \log (1+k\ln n)}{k}\hfill \\ \le \frac{1}{k}\log \frac{\mathbb{E}\left[g_{X^k}^{\rho }\left(X^k\right)\right]}{\mathbb{E}\left[g_{Y^k}^{\rho }\left(Y^k\right)\right]}\hfill \end{array}$$

(65)

$$\begin{array}{c}\le \rho \left[H_{\frac{1}{1+\rho }}\left(X\right)-H_{\frac{1}{1+\rho }}\left(Y\right)\right]+\frac{\rho \log (1+k\ln m)}{k}.\hfill \end{array}$$

(66)

In view of Theorem 2-(a) and (65), it follows that for an arbitrary $f\in {\mathcal{F}}_{n,m}$ and $\rho >0$

$$\begin{array}{c}\hfill \frac{1}{k}\log \frac{\mathbb{E}\left[g_{X^k}^{\rho }\left(X^k\right)\right]}{\mathbb{E}\left[g_{Y^k}^{\rho }\left(Y^k\right)\right]}\ge \rho \left[H_{\frac{1}{1+\rho }}\left(X\right)-H_{\frac{1}{1+\rho }}\left({\tilde{X}}_m\right)\right]-\frac{\rho \log (1+k\ln n)}{k}\end{array}$$

(67)

where ${\tilde{X}}_m$ is a random variable whose probability mass function is given in (30). Please note that

$$\begin{array}{c}\hfill H_{\frac{1}{1+\rho }}\left({\tilde{X}}_m\right)\le H_{\frac{1}{1+\rho }}\left(X\right),\frac{\rho \log (1+k\ln n)}{k}\underset{k\to \infty }{⟶}0\end{array}$$

(68)

where the first inequality in (68) holds since $P_X\prec P_{{\tilde{X}}_m}$ (see Lemma 5) and the Rényi entropy is Schur-concave.

By the explicit construction of the function $f^{\ast }\in {\mathcal{F}}_{n,m}$ according to the algorithm in Steps 1–4 in the proof of Theorem 2 (based on the Huffman procedure), by setting $Y_i:=f^{\ast }\left(X_i\right)$ for every $i\in \{1,\dots ,k\}$, it follows from (34) and (66) that for all $\rho >0$

$$\begin{array}{c}\hfill \frac{1}{k}\log \frac{\mathbb{E}\left[g_{X^k}^{\rho }\left(X^k\right)\right]}{\mathbb{E}\left[g_{Y^k}^{\rho }\left(Y^k\right)\right]}\le \rho \left[H_{\frac{1}{1+\rho }}\left(X\right)-H_{\frac{1}{1+\rho }}\left({\tilde{X}}_m\right)+v\left(\frac{1}{1+\rho }\right)\right]+\frac{\rho \log (1+k\ln m)}{k}\end{array}$$

(69)

where the monotonically increasing function $v:(0,\infty )\to (0,\infty )$ is given in (33), and it is depicted by the solid line in the left plot of Figure 2. In view of (33), it can be shown that the linear approximation $v\left(\alpha \right)\approx v\left(1\right)\alpha$ is excellent for all $\alpha \in [0,1]$, and therefore for all $\rho >0$

$$\begin{array}{c}\hfill v\left(\frac{1}{1+\rho }\right)\approx \frac{0.08607}{1+\rho }\mathrm{bits}.\end{array}$$

(70)

Hence, for sufficiently large value of k, the gap between the lower and upper bounds in (67) and (69) is marginal, being approximately equal to $\frac{0.08607\rho }{1+\rho }\mathrm{bits}$ for all $\rho >0$.

The following theorem summarizes our result in this section.

Theorem 3.

Let

• ${\left\{X_i\right\}}_{i=1}^k$ be i.i.d. with $X_1\sim P_X$ taking values on a set $\mathcal{X}$ with $\left|\mathcal{X}\right|=n$;
• $Y_i=f\left(X_i\right)$, for every $i\in \{1,\dots ,k\}$, where $f\in {\mathcal{F}}_{n,m}$ is a deterministic function with $m<n$;
• $g_{X^k}:{\mathcal{X}}^k\to \{1,\dots ,n^k\}$ and $g_{Y^k}:{\mathcal{Y}}^k\to \{1,\dots ,m^k\}$ be, respectively, ranking functions of the random vectors $X^k=(X_1,\dots ,X_k)$ and $Y^k=(Y_1,\dots ,Y_k)$.

Then, for every $\rho >0$,

(a) 

The lower bound in (67) holds for every deterministic function $f\in {\mathcal{F}}_{n,m}$;

(b) 

The upper bound in (69) holds for the specific $f^{\ast }\in {\mathcal{F}}_{n,m}$, whose construction relies on the Huffman algorithm (see Steps 1–4 of the procedure in the proof of Theorem 2);

(c) 

The gap between these bounds, for $f=f^{\ast }$ and sufficiently large k, is at most $\rho v\left(\frac{1}{1+\rho }\right)\approx \frac{0.08607\rho }{1+\rho }bits.$

5.1.3. Numerical Result

The following simple example illustrates the tightness of the achievable upper bound and the universal lower bound in Theorem 3, especially for sufficiently long sequences.

Example 1.

Let X be geometrically distributed restricted to $\{1,\dots ,n\}$ with the probability mass function

$$\begin{array}{c}\hfill P_X\left(j\right)=\frac{(1-a)a^{j-1}}{1-a^n},j\in \{1,\dots ,n\}\end{array}$$

(71)

where $a=\frac{24}{25}$ and $n=128$. Assume that $X_1,\dots ,X_k$ are i.i.d. with $X_1\sim P_X$, and let $Y_i=f\left(X_i\right)$ for a deterministic function $f\in {\mathcal{F}}_{n,m}$ with $n=128$ and $m=16$. We compare the upper and lower bounds in Theorem 3 for the two cases where the sequence $X^k=(X_1,\dots ,X_k)$ is of length $k=100$ or $k=1000$. The lower bound in (67) holds for an arbitrary deterministic $f\in {\mathcal{F}}_{n,m}$, and the achievable upper bound in (69) holds for the construction of the deterministic function $f=f^{\ast }\in {\mathcal{F}}_{n,m}$ (based on the Huffman algorithm) in Theorem 3. Numerical results are shown in Figure 4, providing plots of the upper and lower bounds on $\frac{1}{k}{\log }_2\frac{\mathbb{E}\left[g_{X^k}^{\rho }\left(X^k\right)\right]}{\mathbb{E}\left[g_{Y^k}^{\rho }\left(Y^k\right)\right]}$ in Theorem 3, and illustrating the improved tightness of these bounds when the value of k is increased from 100 (left plot) to 1000 (right plot). From Theorem 3-(c), for sufficiently large k, the gap between the upper and lower bounds is less than 0.08607 bits (for all $\rho >0$); this is consistent with the right plot of Figure 4 where $k=1000$.

5.2. Lossless Source Coding

5.2.1. Background

For uniquely decodable (UD) lossless source coding, Campbell [51,83] proposed the cumulant generating function of the codeword lengths as a generalization to the frequently used design criterion of average code length. Campbell’s motivation in [51] was to control the contribution of the longer codewords via a free parameter in the cumulant generating function: if the value of this parameter tends to zero, then the resulting design criterion becomes the average code length per source symbol; on the other hand, by increasing the value of the free parameter, the penalty for longer codewords is more severe, and the resulting code optimization yields a reduction in the fluctuations of the codeword lengths.

We introduce the coding theorem by Campbell [51] for lossless compression of a discrete memoryless source (DMS) with UD codes, which serves for our analysis jointly with Theorem 2.

Theorem 4 

(Campbell 1965, [51]). Consider a DMS which emits symbols with a probability mass function $P_X$ defined on a (finite or countably infinite) set $\mathcal{X}$. Consider a UD fixed-to-variable source code operating on source sequences of k symbols with an alphabet of the codewords of size D. Let $\ell \left(x^k\right)$ be the length of the codeword which corresponds to the source sequence $x^k:=(x_1,\dots ,x_k)\in {\mathcal{X}}^k$. Consider the scaled cumulant generating function of the codeword lengths

$$\begin{array}{c}\hfill {\Lambda }_k\left(\rho \right):=\frac{1}{k}{\log }_D\left(\sum _{x^k\in {\mathcal{X}}^k}{P}_{X^k}\left(x^k\right)D^{\rho \ell \left(x^k\right)}\right),\rho >0\end{array}$$

(72)

where

$$\begin{array}{c}\hfill P_{X^k}\left(x^k\right)=\prod _{i=1}^k{P}_X\left(x_i\right),\forall x^k\in {\mathcal{X}}^k.\end{array}$$

(73)

Then, for every $\rho >0$, the following hold:

(a) 

Converse result:

$$\begin{array}{c}\hfill \frac{{\Lambda }_k\left(\rho \right)}{\rho }\ge \frac{1}{\log D}{H}_{\frac{1}{1+\rho }}\left(X\right).\end{array}$$

(74)

(b) 

Achievability result: there exists a UD source code, for which

$$\begin{array}{c}\hfill \frac{{\Lambda }_k\left(\rho \right)}{\rho }\le \frac{1}{\log D}{H}_{\frac{1}{1+\rho }}\left(X\right)+\frac{1}{k}.\end{array}$$

(75)

The term scaled cumulant generating function is used in view of [56] (Remark 20). The bounds in Theorem 4, expressed in terms of the Rényi entropy, imply that for sufficiently long source sequences, it is possible to make the scaled cumulant generating function of the codeword lengths approach the Rényi entropy as closely as desired by a proper fixed-to-variable UD source code; moreover, the converse result shows that there is no UD source code for which the scaled cumulant generating function of its codeword lengths lies below the Rényi entropy. By invoking L’Hôpital’s rule, one gets from (72)

$$\begin{array}{c}\hfill \underset{\rho \downarrow 0}{\lim }\frac{{\Lambda }_k\left(\rho \right)}{\rho }=\frac{1}{k}\sum _{x^k\in {\mathcal{X}}^k}{P}_{X^k}\left(x^k\right)\ell \left(x^k\right)=\frac{1}{k}\mathbb{E}\left[\ell \left(X^k\right)\right].\end{array}$$

(76)

Hence, by letting $\rho$ tend to zero in (74) and (75), it follows from (4) that Campbell’s result in Theorem 4 generalizes the well-known bounds on the optimal average length of UD fixed-to-variable source codes (see, e.g., [84] ((5.33) and (5.37))):

$$\begin{array}{c}\hfill \frac{1}{\log D}H\left(X\right)\le \frac{1}{k}\mathbb{E}\left[\ell \left(X^k\right)\right]\le \frac{1}{\log D}H\left(X\right)+\frac{1}{k},\end{array}$$

(77)

and (77) is satisfied by Huffman coding (see, e.g., [84] (Theorem 5.8.1)). Campbell’s result therefore generalizes Shannon’s fundamental result in [85] for the average codeword lengths of lossless compression codes, expressed in terms of the Shannon entropy.

Following the work by Campbell [51], Courtade and Verdú derived in [52] non-asymptotic bounds for the scaled cumulant generating function of the codeword lengths for $P_X$-optimal variable-length lossless codes [23,86]. These bounds were used in [52] to obtain simple proofs of the asymptotic normality of the distribution of codeword lengths, and the reliability function of memoryless sources allowing countably infinite alphabets. Sason and Verdú recently derived in [56] improved non-asymptotic bounds on the cumulant generating function of the codeword lengths for fixed-to-variable optimal lossless source coding without prefix constraints, and non-asymptotic bounds on the reliability function of a DMS, tightening the bounds in [52].

5.2.2. Analysis

The following analysis for lossless source compression with UD codes relies on a combination of Theorems 2 and 4.

Let $X_1,\dots ,X_k$ be i.i.d. symbols which are emitted from a DMS according to a probability mass function $P_X$ whose support is a finite set $\mathcal{X}$ with $\left|\mathcal{X}\right|=n$. Similarly to Section 5.1, to cluster the data, suppose that each symbol $X_i$ is mapped to $Y_i=f\left(X_i\right)$ where $f\in {\mathcal{F}}_{n,m}$ is an arbitrary deterministic function (independent of the index i) with $m<n$. Consequently, the i.i.d. symbols $Y_1,\dots ,Y_k$ take values on a set $\mathcal{Y}$ with $\left|\mathcal{Y}\right|=m<\left|\mathcal{X}\right|$. Consider two UD fixed-to-variable source codes: one operating on the sequences $x^k\in {\mathcal{X}}^k$, and the other one operates on the sequences $y^k\in {\mathcal{Y}}^k$; let D be the size of the alphabets of both source codes. Let $\ell \left(x^k\right)$ and $\overline{\ell }\left(y^k\right)$ denote the length of the codewords for the source sequences $x^k$ and $y^k$, respectively, and let ${\Lambda }_k(·)$ and ${\overline{\Lambda }}_k(·)$ denote their corresponding scaled cumulant generating functions (see (72)).

In view of Theorem 4-(b), for every $\rho >0$, there exists a UD source code for the sequences in ${\mathcal{X}}^k$ such that the scaled cumulant generating function of its codeword lengths satisfies (75). Furthermore, from Theorem 4-(a), we get

$$\begin{array}{c}\hfill \frac{{\overline{\Lambda }}_k\left(\rho \right)}{\rho }\ge \frac{1}{\log D}{H}_{\frac{1}{1+\rho }}\left(Y\right).\end{array}$$

(78)

From (75), (78) and Theorem 2 (a) and (b), for every $\rho >0$, there exist a UD source code for the sequences in ${\mathcal{X}}^k$, and a construction of a deterministic function $f\in {\mathcal{F}}_{n,m}$ (as specified by Steps 1–4 in the proof of Theorem 2, borrowed from [12]) such that the difference between the two scaled cumulant generating functions satisfies

$$\begin{array}{c}\hfill {\Lambda }_k\left(\rho \right)-{\overline{\Lambda }}_k\left(\rho \right)\le \frac{\rho }{\log D}\left[H_{\frac{1}{1+\rho }}\left(X\right)-H_{\frac{1}{1+\rho }}\left({\tilde{X}}_m\right)+v\left(\frac{1}{1+\rho }\right)\right]+\frac{\rho }{k},\end{array}$$

(79)

where (79) holds for every UD source code operating on the sequences in ${\mathcal{Y}}^k$ with $Y_i=f\left(X_i\right)$ (for $i=1,\dots ,k$) and the specific construction of $f\in {\mathcal{F}}_{n,m}$ as above, and ${\tilde{X}}_m$ in the right side of (79) is a random variable whose probability mass function is given in (30). The right side of (79) can be very well approximated (for all $\rho >0$) by using (70).

We proceed with a derivation of a lower bound on the left side of (79). In view of Theorem 4, it follows that (74) is satisfied for every UD source code which operates on the sequences in ${\mathcal{X}}^k$; furthermore, Theorems 2 and 4 imply that, for every $f\in {\mathcal{F}}_{n,m}$, there exists a UD source code which operates on the sequences in ${\mathcal{Y}}^k$ such that

$$\begin{array}{cc}\hfill \frac{{\overline{\Lambda }}_k\left(\rho \right)}{\rho }& \le \frac{1}{\log D}{H}_{\frac{1}{1+\rho }}\left(Y\right)+\frac{1}{k},\hfill \end{array}$$

(80)

$$\begin{array}{cc}& \le \frac{1}{\log D}{H}_{\frac{1}{1+\rho }}\left({\tilde{X}}_m\right)+\frac{1}{k},\hfill \end{array}$$

(81)

where (81) is due to (39) since $Y_i=f\left(X_i\right)$ (for $i=1,\dots ,k$) with an arbitrary deterministic function $f\in {\mathcal{F}}_{n,m}$, and $Y_i\sim P_Y$ for every i; hence, from (74), (80) and (81),

$$\begin{array}{c}\hfill {\Lambda }_k\left(\rho \right)-{\overline{\Lambda }}_k\left(\rho \right)\ge \frac{\rho }{\log D}\left(H_{\frac{1}{1+\rho }}\left(X\right)-H_{\frac{1}{1+\rho }}\left({\tilde{X}}_m\right)\right)-\frac{\rho }{k}.\end{array}$$

(82)

We summarize our result as follows.

Theorem 5.

Let

• $X_1,\dots ,X_k$ be i.i.d. symbols which are emitted from a DMS according to a probability mass function $P_X$ whose support is a finite set $\mathcal{X}$ with $\left|\mathcal{X}\right|=n$;
• Each symbol $X_i$ be mapped to $Y_i=f\left(X_i\right)$ where $f\in {\mathcal{F}}_{n,m}$ is the deterministic function (independent of the index i) with $m<n$, as specified by Steps 1–4 in the proof of Theorem 2 (borrowed from [12]);
• Two UD fixed-to-variable source codes be used: one code encodes the sequences $x^k\in {\mathcal{X}}^k$, and the other code encodes their mappings $y^k\in {\mathcal{Y}}^k$; let the common size of the alphabets of both codes be D;
• ${\Lambda }_k(·)$ and ${\overline{\Lambda }}_k(·)$ be, respectively, the scaled cumulant generating functions of the codeword lengths of the k-length sequences in ${\mathcal{X}}^k$ (see (72)) and their mapping to ${\mathcal{Y}}^k$.

Then, for every $\rho >0$, the following holds for the difference between the scaled cumulant generating functions ${\Lambda }_k(·)$ and ${\overline{\Lambda }}_k(·)$:

(a) 

There exists a UD source code for the sequences in ${\mathcal{X}}^k$ such that the upper bound in (79) is satisfied for every UD source code which operates on the sequences in ${\mathcal{Y}}^k$;

(b) 

There exists a UD source code for the sequences in ${\mathcal{Y}}^k$ such that the lower bound in (82) holds for every UD source code for the sequences in ${\mathcal{X}}^k$; furthermore, the lower bound in (82) holds in general for every deterministic function $f\in {\mathcal{F}}_{n,m}$;

(c) 

The gap between the upper and lower bounds in (79) and (82), respectively, is at most $\frac{\rho }{\log D}v\left(\frac{1}{1+\rho }\right)+\frac{2\rho }{k}$ (the function $v:(0,\infty )\to (0,\infty )$ is introduced in (33)), which is approximately $\frac{0.08607\rho {\log }_{D}2}{1+\rho }+\frac{2\rho }{k}$;

(d) 

The UD source codes in Items (a) and (b) for the sequences in ${\mathcal{X}}^k$ and ${\mathcal{Y}}^k$, respectively, can be constructed to be prefix codes by the algorithm in Remark 11.

Remark 11 (An Algorithm for Theorem 5 (d)).

A construction of the UD source codes for the sequences in ${\mathcal{X}}^k$ and ${\mathcal{Y}}^k$, whose existence is assured by Theorem 5 (a) and (b) respectively, is obtained by the following algorithm (of three steps) which also constructs them as prefix codes:

(1) 

As a preparatory step, we first calculate the probability mass function $P_Y$ from the given probability mass function $P_X$ and the deterministic function $f\in {\mathcal{F}}_{n,m}$ which is obtained by Steps 1–4 in the proof of Theorem 2; accordingly, $P_Y\left(y\right)={\displaystyle \sum _{x\in \mathcal{X}:f\left(x\right)=y}}{P}_X\left(x\right)$ for all $y\in \mathcal{Y}$. We then further calculate the probability mass functions for the i.i.d. sequences in ${\mathcal{X}}^k$ and ${\mathcal{Y}}^k$ (see (73)); recall that the number of types in ${\mathcal{X}}^k$ and ${\mathcal{Y}}^k$ is polynomial in k (being upper bounded by ${(k+1)}^{n-1}$ and ${(k+1)}^{m-1}$, respectively), and the values of these probability mass functions are fixed over each type;

(2) 

The sets of codeword lengths of the two UD source codes, for the sequences in ${\mathcal{X}}^k$ and ${\mathcal{Y}}^k$, can (separately) be designed according to the achievability proof in Campbell’s paper (see [51] (p. 428)). More explicitly, let $\alpha :=\frac{1}{1+\rho }$; for all $x^k\in {\mathcal{X}}^k$, let $\ell \left(x^k\right)\in \mathbb{N}$ be given by

$$\begin{array}{c}\hfill \ell \left(x^k\right)=⌈-\alpha {\log }_D{P}_{X^k}\left(x^k\right)+{\log }_D{Q}_k⌉\end{array}$$

(83)

with

$$\begin{array}{c}\hfill Q_k:=\sum _{x^k\in {\mathcal{X}}^k}{P}_{X^k}^{\alpha }\left(x^k\right)={\left(\sum _{x\in \mathcal{X}}{P}_X^{\alpha }\left(x\right)\right)}^k,\end{array}$$

(84)

and let $\overline{\ell }\left(y^k\right)\in \mathbb{N}$, for all $y^k\in {\mathcal{Y}}^k$, be given similarly to (83) and (84) by replacing $P_X$ with $P_Y$, and $P_{X^k}$ with $P_{Y^k}$. This suggests codeword lengths for the two codes which fulfil (75) and (80), and also, both satisfy Kraft’s inequality;

(3) 

The separate construction of two prefix codes (a.k.a. instantaneous codes) based on their given sets of codeword lengths ${\left\{\ell \left(x^k\right)\right\}}_{x^k\in {\mathcal{X}}^k}$ and ${\left\{\overline{\ell }\left(y^k\right)\right\}}_{y^k\in {\mathcal{Y}}^k}$, as determined in Step 2, is standard (see, e.g., the construction in the proof of [84] (Theorem 5.2.1)).

Theorem 5 is of interest since it provides upper and lower bounds on the reduction in the cumulant generating function of close-to-optimal UD source codes because of clustering data, and Remark 11 suggests an algorithm to construct such UD codes which are also prefix codes. For long enough sequences (as $k\to \infty$), the upper and lower bounds on the difference between the scaled cumulant generating functions of the suggested source codes for the original and clustered data almost match (see (79) and (82)), being roughly equal to $\rho \left(H_{\frac{1}{1+\rho }}\left(X\right)-H_{\frac{1}{1+\rho }}({\tilde{X}}_m)\right)$ (with logarithms on base D, which is the alphabet size of the source codes); as $k\to \infty$, the gap between these upper and lower bounds is less than $0.08607{\log }_{D}2$. Furthermore, in view of (76),

$$\begin{array}{c}\hfill \underset{\rho \downarrow 0}{\lim }\frac{{\Lambda }_k\left(\rho \right)-{\overline{\Lambda }}_k\left(\rho \right)}{\rho }=\frac{1}{k}\left(\mathbb{E}\left[\ell \left(X^k\right)\right]-\mathbb{E}\left[\overline{\ell }\left(Y^k\right)\right]\right),\end{array}$$

(85)

so, it follows from (4), (33), (79) and (82) that the difference between the average code lengths (normalized by k) of the original and clustered data satisfies

$$\begin{array}{c}\hfill \frac{H\left(X\right)-H\left({\tilde{X}}_m\right)}{\log D}-\frac{1}{k}\le \frac{\mathbb{E}\left[\ell \left(X^k\right)\right]-\mathbb{E}\left[\overline{\ell }\left(Y^k\right)\right]}{k}\le \frac{H\left(X\right)-H\left({\tilde{X}}_m\right)+0.08607\log 2}{\log D},\end{array}$$

(86)

where the gap between the upper and lower bounds is equal to $0.08607{\log }_{D}2+\frac{1}{k}$.