Some Families of Jensen-like Inequalities with Application to Information Theory

Abstract

It is well known that the traditional Jensen inequality is proved by lower bounding the given convex function, $f\left(x\right)$, by the tangential affine function that passes through the point $\left(\mathbb{E}\right\{X\},f(\mathbb{E}\left\{X\right\}\left)\right)$, where $\mathbb{E}\left\{X\right\}$ is the expectation of the random variable X. While this tangential affine function yields the tightest lower bound among all lower bounds induced by affine functions that are tangential to f, it turns out that when the function f is just part of a more complicated expression whose expectation is to be bounded, the tightest lower bound might belong to a tangential affine function that passes through a point different than $\left(\mathbb{E}\right\{X\},f(\mathbb{E}\left\{X\right\}\left)\right)$. In this paper, we take advantage of this observation by optimizing the point of tangency with regard to the specific given expression in a variety of cases and thereby derive several families of inequalities, henceforth referred to as “Jensen-like” inequalities, which are new to the best knowledge of the author. The degree of tightness and the potential usefulness of these inequalities is demonstrated in several application examples related to information theory.

In memory of Jacob Ziv, a shining star in the sky of information theory, whose legacy as a researcher will continue to inspire me and many others for years to come.

1. Introduction

As is well known, the Jensen inequality is one of the most fundamental and useful mathematical tools in a variety of fields, including information theory. Interestingly, it includes many other very well-known inequalities, which are important on their own, as special cases. Among many examples, we mention the Shwartz–Cauchy inequality (which in turn supports uncertainty principles and the Cramér–Rao bound), the Lyapunov inequality, the Hölder inequality, and the inequalities among the harmonic, geometric and arithmetic means. In the field of information theory, the Jensen inequality stands at the basis of the information inequality (i.e., the non-negativity of the relative entropy), the data processing inequality (which in turn leads to the Fano inequality), and the inequality between conditional and unconditional entropies. Moreover, it plays a central role in support of the derivation of single-letter formulas in Shannon theory and in the theory of maximum entropy under moment constraints (see, for example, Chapter 12 of [1]).

During the last two decades, there have been many research efforts around Jensen’s inequality, which included refinements [2,3,4,5], variations [6,7,8], improvements [9,10,11], and extensions [12], just to name a few. There have also many derivations of reversed versions of the Jensen inequality. For a non-exhaustive list of works, see, e.g., ref. [13] for mixtures of exponential families, refs. [14,15,16,17] for global bounds on the difference between the two sides of Jensen’s inequality, ref. [18] for functions of self-adjoint operators in Hibert spaces, refs. [19,20] for inequalities via Green functions, refs. [21,22] for inequalities via Chebychev and Chernoff bounds, ref. [23] for quantum Simpson’s and quantum Newton’s inequalities, and ref. [24] for new quantum Hermite–Hadamard-like inequalities. In most of them, the derived inequalities are exemplified in many applications, for instance, useful relationships between arithmetic and geometric means, converse bounds on the entropy, the relative entropy, as well as the more general f-divergence, converse forms of the Hölder inequality, and so on. In many of these works, the main results are given in the form of an upper bound on the difference, $\mathbb{E}\left\{f\right(X\left)\right\}-f\left(\mathbb{E}\right\{X\left\}\right)$, where f is a convex function, $\mathbb{E}\{·\}$ is the expectation operator, and X is the random variable. However, those bounds depend mostly on global parameters associated with f, for example, its range and domain, but not particularly on the underlying probability function (probability density function in the continuous case, or probability mass function in the discrete case), of X. For one thing, a desirable property of a reverse Jensen inequality would be that it is tight when X is well concentrated in the vicinity of its mean, just like the same well-known property of the ordinary Jensen inequality. In [22], there is an attempt to address this issue.

This paper revisits the Jensen inequality from a completely different angle. It is not meant to be another improvement of earlier bounds in an existing line of work. It is meant to propose a different approach for generating useful inequalities in the spirit of Jensen’s inequality. It is based on the following simple observation, which is rooted in the proof of Jensen’s inequality: The given convex function, $f\left(x\right)$, is lower bounded by the tangential affine function, $\ell \left(x\right)=f\left(a\right)+f^{\prime }\left(a\right)(x-a)$, where a is an arbitrary number in the domain of x and $f^{\prime }\left(a\right)$ is the derivative of f at $x=a$ (provided that f is differentiable at $x=a$). By selecting $a=\mathbb{E}\left\{X\right\}$ and taking expectations of both sides of the inequality, $f\left(X\right)\ge \ell \left(X\right)$, the Jensen inequality is readily proved. The point to be remembered is that here, $a_{\ast }=\mathbb{E}\left\{X\right\}$ is the optimal choice of a in the sense of maximizing $\mathbb{E}\{\ell (X\left)\right\}$ over all possible values of a, thus yielding the tightest lower bound within this class of lower bounds on $\mathbb{E}\left\{f\right(X\left)\right\}$. The optimal choice of a, however, might be different than $\mathbb{E}\left\{X\right\}$ when the function $f\left(X\right)$ is only a part of a more complicated expression whose expectation is to be lower bounded. For example, one might be interested in lower bounding $\mathbb{E}\left\{g\right[f\left(X\right)\left]\right\}$, where g is a monotonically non-decreasing function, or $\mathbb{E}\left\{f\right(X\left)g\right(X\left)\right\}$, where g is a non-negative and/or convex function, or a combination of both, etc.

To demonstrate this fact, consider the example (to be treated in detail in Section 2) of lower bounding $\mathbb{E}\left\{f\right(X\left)g\right(X\left)\right\}$, where g is a non-negative function. In this case,

$$\mathbb{E}\left\{f\left(X\right)g\left(X\right)\right\}\ge \mathbb{E}\left\{[f\left(a\right)+f^{\prime }\left(a\right)(X-a)]g\left(X\right)\right\},$$

(1)

and by maximizing the right-hand side (r.h.s.) over a, we easily obtain that the optimal choice of a here is $a_{\ast }=\mathbb{E}\left\{Xg\left(X\right)\right\}/\mathbb{E}\left\{g\left(X\right)\right\}$, yielding the inequality,

$$\mathbb{E}\left\{f\left(X\right)g\left(X\right)\right\}\ge f\left(\frac{\mathbb{E}\left\{Xg\right(X\left)\right\}}{\mathbb{E}\left\{g\right(X\left)\right\}}\right)·\mathbb{E}\left\{g\left(X\right)\right\},$$

(2)

which is useful as long as g is such that we can easily calculate both $\mathbb{E}\left\{g\right(X\left)\right\}$ and $\mathbb{E}\left\{Xg\right(X\left)\right\}$. While this particular inequality could have been obtained also by applying the (ordinary) Jensen inequality, $\mathbb{E}\left\{f\right(X\left)\right\}\ge f\left(\mathbb{E}\right\{X\left\}\right)$, with respect to (with respect to) the density, $\tilde{p}\left(x\right)=p\left(x\right)g\left(x\right)/{\int }_{-\infty }^{\infty }p\left(x^{\prime }\right)g\left(x^{\prime }\right)d{x}^{\prime }$, we will see in the sequel also various examples of inequalities with no apparent simple interpretations such as this. We henceforth refer to these classes of inequalities as Jensen-like inequalities, since they are derived using the same general idea that underlies the proof the classical Jensen inequality. We will also demonstrate the usefulness of these inequalities in information theory.

Our contributions, in this work, have the following features:

• In many cases (such as the one above), the optimal value of the parameter(s) (e.g., the parameter a in the above discussion) can be found in closed form. In other cases, the resulting expressions may not lend themselves to closed-form optimization, and then we have two possibilities: (i) carry out the optimization numerically, and (ii) select an arbitrary choice of a and obtain a valid lower bound, bearing in mind that an educated guess can potentially result in a good bound.
• Our inequalities provide two types of bounds: (i) bounds that require the calculation of the first two moments (or equivalently, the first two cumulants) of X, and (ii) bounds that require the calculation of the moment-generating function (MGF) of X and its derivative, or equivalently, the cumulant-generating function (CGF) of X and its derivative. All these types of moments are often easily calculable in closed form, especially in situations where X is given by the sum of independent and identically distributed (i.i.d.) random variables, which is frequently encountered in information–theoretic applications.
• Most of our derivations extend to convex functions of more than one variable.
• The classes of Jensen-like inequalities that we consider allow enough flexibility to obtain derivations of lower bounds on functions that are not necessarily convex, and even for some concave functions, and thereby open the door for another route to reverse Jensen inequalities. This can be accomplished by representing the given function in one of the categories discussed (e.g., a product of a convex function and a non-negaive function, a product of two non-negative convex functions, a composition of a monotone function and a convex function, etc.).
• We demonstrate the utility of the Jensen-like inequalities in several examples of information–theoretic relevance. We also display numerical results that exemplify the degree of tightness of these bounds.
• Our Jensen-like inequalities have the desirable property of becoming tighter as X becomes more and more concentrated around its mean, just like the ordinary Jensen inequality.
• Throughout the paper, we confine ourselves to lower bounds on expectations of expressions that include a convex function f, but it should be understood that they all continue to apply also if f is concave and the inequalities are reversed.
• It should be understood that the classes of Jensen-like inequalities that we derive in this work are just examples that demonstrate the basic underlying idea of optimizing the point of tangency to the given convex function for the specific expression at hand. It is conceivable that the same idea can be applied to many more situations of theoretical and practical interest.

In all forthcoming derivations, it will be assumed that the convex functions involved are weakly convex and differentiable. In other words, we will rely on the well-known fact that a differentiable convex function, $f\left(x\right)$, is nowhere below the supporting line, $\ell \left(x\right)=f\left(a\right)+f^{\prime }\left(a\right)(x-a)$, for every value of the parameter a in the domain of the independent variable, x [25] (p. 69, eq. (3.2)). In order to show that the point of zero-derivative of the lower bound (w.r.t. a) indeed yields a maximum (and not a minimum, etc.) of the lower bound, we will need to further assume that f is twice differentiable, but such an assumption will not limit the applicability of the claimed lower bound, because the lower bound applies to any value of a, including the point of zero-derivative, even if this point cannot be proved to yield the maximum of the lower bound using the standard methods. Similar comments apply when the lower bound will depend on more than one parameter.

In the remaining part of this article, each section is devoted to a different class of Jensen-like inequalities, which corresponds to a different form of an expression that includes the convex function, f.

2. A Product of a Convex Function and a Non-Negative Function

In this section, we focus on lower bounding expressions of the form $\mathbb{E}\left\{f\right(X\left)g\right(X\left)\right\}$, where f is convex and g is non-negative. Indeed, let $f:\mathbb{R}\to \mathbb{R}$ be a convex function and let $g:\mathbb{R}\to {\mathbb{R}}^{+}$ be a non-negative function. Then, for any $a\in \mathbb{R}$,

$$\begin{array}{}\mathrm{(3)}& \mathbb{E}\left\{f\right(X\left)g\right(X\left)\right\}& \ge \mathbb{E}\left\{[f\left(a\right)+f^{\prime }\left(a\right)(X-a)]g\left(X\right)\right\}\hfill \mathrm{(4)}& & =[f\left(a\right)-a{f}^{\prime }\left(a\right)]\mathbb{E}\left\{g\left(X\right)\right\}+f^{\prime }\left(a\right)\mathbb{E}\left\{Xg\left(X\right)\right\}.\hfill \end{array}$$

To find the value of a that maximizes the r.h.s., we equate the derivative to zero and obtain:

$$[f^{\prime }\left(a\right)-f^{\prime }\left(a\right)-a{f}^{″}\left(a\right)]\mathbb{E}\left\{g\left(X\right)\right\}+f^{″}\left(a\right)\mathbb{E}\left\{Xg\left(X\right)\right\}=0$$

(5)

or equivalently,

$$f^{″}\left(a\right)[\mathbb{E}\left\{Xg\left(X\right)\right\}-a\mathbb{E}\left\{g\left(X\right)\right\}]=0,$$

(6)

whose solution is readily obtained as

$$a=a_{\ast }\stackrel{▵}{=}\frac{\mathbb{E}\left\{Xg\right(X\left)\right\}}{\mathbb{E}\left\{g\right(X\left)\right\}},$$

(7)

and it is easy to verify that the second derivative at $a=a_{\ast }$ is $-f^{″}\left(a_{\ast }\right)\mathbb{E}\left\{g\left(X\right)\right\}<0$, which means that it is a maximum (at least a local one). The resulting lower bound on $\mathbb{E}\left\{f\right(X\left)g\right(X\left)\right\}$ is then given by

$$\mathbb{E}\left\{f\left(X\right)g\left(X\right)\right\}\ge f\left(\frac{\mathbb{E}\left\{Xg\right(X\left)\right\}}{\mathbb{E}\left\{g\right(X\left)\right\}}\right)·\mathbb{E}\left\{g\left(X\right)\right\}.$$

(8)

This result extends straightforwardly to the case where X is a vector provided that f is jointly convex and differentiable in all components of X. In particular, it extends to the case where f and g act as different random variables, X and Y, with a joint distribution:

$$\mathbb{E}\left\{f\left(X\right)g\left(Y\right)\right\}\ge f\left(\frac{\mathbb{E}\left\{Xg\right(Y\left)\right\}}{\mathbb{E}\left\{g\right(Y\left)\right\}}\right)·\mathbb{E}\left\{g\left(Y\right)\right\}.$$

(9)

We next consider several examples.

Example 1. 

Let $f\left(x\right)=-lnx$ and $g\left(x\right)=x$, $x>0$. Applying Inequality (8),

$$\mathbb{E}\{-XlnX\}\ge -\mathbb{E}\left\{X\right\}·ln\frac{\mathbb{E}\left\{X^2\right\}}{\mathbb{E}\left\{X\right\}}=-\mathbb{E}\left\{X\right\}·ln\left(\mathbb{E}\left\{X\right\}\right)-\mathbb{E}\left\{X\right\}·ln\left(1+\frac{Var\left\{X\right\}}{{\left[\mathbb{E}\left\{X\right\}\right]}^2}\right).$$

(10)

Note that the function $-xlnx$ is concave, rather than convex, yet we have here a lower bound (rather than an upper bound) to its expectation, namely, a reversed Jensen inequality. The first term on the right-most side is the (ordinary) Jensen upper bound on $\mathbb{E}\{-XlnX\}$, and the second term is the gap, which depends not only on the expectation of X but also on its variance, which manifests the fluctuations around $\mathbb{E}\left\{X\right\}$. Clearly, if $Var\left\{X\right\}=0$, the second term vanishes, which makes sense, because when X is a degenerated random variable, Jensen’s inequality is achieved with equality and there is no gap. This inequality has an immediate application for obtaining a lower bound to the expectation of the empirical entropy of a sequence drawn by a memoryless source, which is relevant in the context of universal source coding [26]. Each term of the empirical entropy is of the form $-XlnX$, where $X=N\left(u\right)/N$, $N\left(u\right)$ is the number of occurrences of a letter u in a randomly drawn N-tuple from a memoryless source, P, with a finite alphabet, $\mathcal{U}$. Clearly, each $N\left(u\right)$ is a binomial random variable with N trials and probability of success, $P\left(u\right)$. In this case, $\mathbb{E}\left\{X\right\}=P\left(u\right)$ and $Var\left\{X\right\}=P\left(u\right)[1-P(u\left)\right]/N$. Thus, denoting the entropy and the empirical entropy, respectively, by

$$\begin{array}{ccc}\hfill H& =& -\sum _{u\in \mathcal{U}}P\left(u\right)lnP\left(u\right)\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill \hat{H}& =& -\sum _{u\in \mathcal{U}}\frac{N\left(u\right)}{N}ln\left(\frac{N\left(u\right)}{N}\right),\hfill \end{array}$$

(12)

with the convention that $0ln0\stackrel{▵}{=}0$, we have:

$$\begin{array}{ccc}\hfill \mathbb{E}\{\hat{H}\}& \ge & -\sum _{u\in \mathcal{U}}P\left(u\right)lnP\left(u\right)-\sum _{u\in \mathcal{U}}P\left(u\right)ln\left[1+\frac{P\left(u\right)[1-P(u\left)\right]/N}{P^2\left(u\right)}\right]\hfill \\ & =& H-\sum _{u\in \mathcal{U}}P\left(u\right)ln\left(1+\frac{1-P\left(u\right)}{NP\left(u\right)}\right)\hfill \\ & \ge & H-\sum _{u\in \mathcal{U}}P\left(u\right)·\frac{1-P\left(u\right)}{NP\left(u\right)}\hfill \\ & =& H-\frac{1}{N}\sum _{u\in \mathcal{U}}[1-P\left(u\right)]\hfill \\ \mathrm{(13)}& & =& H-\frac{\left|\mathcal{U}\right|-1}{N},\hfill \end{array}$$

where $\left|\mathcal{U}\right|$ is the cardinality of $\mathcal{U}$. The use of the ordinary Jensen inequality yields an upper bound rather than a lower bound, $\mathbb{E}\{\hat{H}\}\le H$. We conclude that the expected empirical entropy, $\mathbb{E}\{\hat{H}\}$, is sandwiched between H and $H-\left(\right|\mathcal{U}|-1)/N$, which is reasonable because the variance of the empirical probabilities, $N\left(u\right)/N$, decays at the rate of $1/N$.

Example 2. 

Let s and t be two real numbers whose difference, $s-t$, is either negative or larger than unity. Now, let $g\left(x\right)=x^t$, and $f\left(x\right)=x^{s-t}$. Then,

$$\begin{array}{ccc}\hfill \mathbb{E}\left\{X^s\right\}& =& \mathbb{E}\left\{X^t{X}^{s-t}\right\}\hfill \\ & \ge & {\left(\frac{\mathbb{E}\left\{X^{t+1}\right\}}{\mathbb{E}\left\{X^t\right\}}\right)}^{s-t}·\mathbb{E}\left\{X^t\right\}\hfill \\ \mathrm{(14)}& & =& \frac{{\left(\mathbb{E}\left\{X^{t+1}\right\}\right)}^{s-t}}{{\left(\mathbb{E}\left\{X^t\right\}\right)}^{s-t-1}}.\hfill \end{array}$$

In particular, for $t=1$ and $s\notin (1,2)$, this becomes

$$\mathbb{E}\left\{X^s\right\}\ge \frac{{\left(\mathbb{E}\left\{X^2\right\}\right)}^{s-1}}{{\left(\mathbb{E}\left\{X\right\}\right)}^{s-2}}={\left[\mathbb{E}\left\{X\right\}\right]}^s·{\left(1+\frac{Var\left\{X\right\}}{{\left[\mathbb{E}\left\{X\right\}\right]}^2}\right)}^{s-1}$$

(15)

which is, once again, a bound that depends only on the first two moments of X. For $s\in (0,1)$, the function $x^s$ is concave, and so, this is a reversed version of the Jensen inequality. For $s\le 0$ and $s\ge 2$, the function $x^s$ is convex, and so, this is an improved version of the Jensen inequality: While the first factor, ${\left[\mathbb{E}\left\{X\right\}\right]}^s$, corresponds to the ordinary Jensen inequality, the second factor expresses the improvement, which depends on the relative fluctuation term, $Var\left\{X\right\}/{\left[\mathbb{E}\left\{X\right\}\right]}^2$. The degree of improvement depends, of course, on the variance of X. If the variance vanishes, there is nothing to improve because the ordinary Jensen inequality becomes an equality. On the other hand, the larger the variance, the larger the gap between the ordinary Jensen bound, ${\left[\mathbb{E}\left\{X\right\}\right]}^s$, and the improved one. Accordingly, this also demonstrates the role of the optimization of the parameter a as opposed to the default choice of $a=\mathbb{E}\left\{X\right\}$ of the ordinary Jensen inequality.

To particularize this example even further, consider the problem of randomized guessing under a distribution Q (see, e.g., [27] and many references therein). Then, the probability of a single success in guessing a discrete alphabet random variable, X, given that we know that $X=x$ (but not the guesser), is $Q\left(x\right)$. In sequential guessing until the first success, the number of guesses, G, is a geometric RV with parameter $p=Q\left(x\right)$, whose mean and variance are $1/p$ and $(1-p)/p^2$, respectively. For $s\in (1,2)$,

$$\mathbb{E}\left\{G^s\right\}\ge {\left(\frac{1}{p}\right)}^s·{\left(1+\frac{(1-p)/p^2}{1/p^2}\right)}^{s-1}=\frac{{(2-p)}^{s-1}}{p^s}=\frac{{[2-Q\left(x\right)]}^{s-1}}{{\left[Q\left(x\right)\right]}^s}.$$

(16)

Example 3. 

Let f be an arbitrary convex function and let $g\left(x\right)=e^{sx}$, where s is a given real number. Then, Inequality (8) becomes:

$$\mathbb{E}\left\{f\left(X\right)e^{sX}\right\}\ge f\left({\psi }^{\prime }\left(s\right)\right)·e^{\psi \left(s\right)}$$

(17)

where

$$\psi \left(s\right)=ln\mathbb{E}\left\{e^{sX}\right\}$$

(18)

is the CGF of X and ${\psi }^{\prime }\left(s\right)$ is its derivative. This gives a lower bound in terms of the CGF of X and its derivative. The ordinary Jensen inequality is obtained as the special case of $s=0$, where $\psi \left(0\right)=0$ and ${\psi }^{\prime }\left(0\right)=\mathbb{E}\left\{X\right\}$.

3. A Composition of a Monotone Function and a Convex Function

Another family of Jensen-like inequalities corresponds to the need to lower bound an expression of the form $\mathbb{E}\left\{g\right[f\left(X\right)\left]\right\}$, where f is convex as before and g is a monotonically non-decreasing function. The general idea is to carry out the optimization of the r.h.s. of the following inequality.

$$\mathbb{E}\left\{g\left[f\left(X\right)\right]\right\}\ge \underset{a}{sup}\mathbb{E}\left\{g[f\left(a\right)+f^{\prime }\left(a\right)(X-a)]\right\}.$$

(19)

In the important special case where $g\left(x\right)=e^x$, we have:

$$\begin{array}{ccc}\hfill \mathbb{E}\{e^{f\left(X\right)}\}& \ge & \underset{a}{sup}\mathbb{E}\left\{e^{f\left(a\right)+f^{\prime }\left(a\right)(X-a)}\right\}\hfill \\ & =& \underset{a}{sup}{e}^{f\left(a\right)-a{f}^{\prime }\left(a\right)}\mathbb{E}\left\{e^{X{f}^{\prime }\left(a\right)}\right\}\hfill \\ \mathrm{(20)}& & =& exp\left\{\underset{a}{sup}\{f\left(a\right)-a{f}^{\prime }\left(a\right)+\psi \left[f^{\prime }\left(a\right)\right]\}\right\},\hfill \end{array}$$

where $\psi (·)$ is again the CGF of X. The optimal value, $a_{\ast }$, of a, is the solution to the equation obtained by equating the derivative of the exponent to zero, i.e.,

$${\psi }^{\prime }\left[f^{\prime }\left(a_{\ast }\right)\right]=a_{\ast },\mathrm{provided}\mathrm{that}{f}^{″}\left(a_{\ast }\right){\psi }^{″}\left[f^{\prime }\left(a_{\ast }\right)\right]<1,$$

(21)

where ${\psi }^{\prime }(·)$ and ${\psi }^{″}(·)$ are the first and the second derivatives of $\psi (·)$, respectively.

Example 4. 

Consider the case where $f\left(x\right)=s{x}^2/2$ and $X\sim \mathcal{N}(\mu ,{\sigma }^2)$, where ${\sigma }^2<1/s$, as otherwise, $\mathbb{E}\{e^{s{X}^2}\}=\infty$. In this case, the condition $f^{″}\left(a_{\ast }\right){\psi }^{″}\left[f^{\prime }\left(a_{\ast }\right)\right]<1$ is equivalent to ${\sigma }^2<1/s$, and we have $f^{\prime }\left(a\right)=sa$, $\psi \left(t\right)=\mu t+{\sigma }^2{t}^2/2$, and so, ${\psi }^{\prime }\left(t\right)=\mu +{\sigma }^{2}t$, which means that ${\psi }^{\prime }\left[f^{\prime }\left(a\right)\right]=\mu +{\sigma }^{2}sa$. The equation for the optimal a becomes then

$$\mu +{\sigma }^{2}sa=a,$$

(22)

whose solution is

$$a=a_{\ast }\stackrel{▵}{=}\frac{\mu }{1-{\sigma }^{2}s},$$

(23)

which yields

$$\mathbb{E}\left\{e^{s{X}^2/2}\right\}\ge exp\left\{s{a}_{\ast }^2/2-s{a}_{\ast }^2+\mu s{a}_{\ast }+{\sigma }^2{s}^2{a}_{\ast }^2/2\right\}=exp\left\{\frac{{\mu }^{2}s}{2(1-{\sigma }^{2}s)}\right\}.$$

(24)

The ordinary Jensen inequality yields

$$\mathbb{E}\left\{e^{s{X}^2/2}\right\}\ge exp\left\{s\mathbb{E}\left\{X^2\right\}/2\right\}=e^{s({\mu }^2+{\sigma }^2)/2},$$

(25)

which does not capture the singularity at $s=1/{\sigma }^2$. The exact calculation yields

$$\mathbb{E}\left\{e^{s{X}^2/2}\right\}=\frac{1}{\sqrt{1-{\sigma }^{2}s}}·exp\left\{\frac{{\mu }^{2}s}{2(1-{\sigma }^{2}s)}\right\},$$

(26)

namely, the Jensen-like bound (24) gives the correct exponential term (along with the singularity at $s=1/{\sigma }^2$) and differs from the exact quantity only in the pre-exponential factor. Once again, this demonstrates the fact that optimizing the point of tangency, a, rather than using the default value, $a=\mathbb{E}\left\{X\right\}$, can make a significant difference.

4. A Product of a Convex Function and a Monotone-Convex Composition

Yet another class of Jensen-like inequalities corresponds to lower bounding the expectation of the product of two functions, where one is convex and the other is a composition of a non-negative monotonically non-decreasing function and a convex function, i.e.,

$$\mathbb{E}\left\{h\left[f\left(X\right)\right]g\left(X\right)\right\}\ge \underset{a,b}{sup}\mathbb{E}\{h[f\left(a\right)+f^{\prime }\left(a\right)(X-a)]·[g\left(b\right)+g^{\prime }\left(b\right)(X-b)]\},$$

(27)

where f and g are convex and h is monotonically non-decreasing and non-negative. For the case where $h\left(x\right)=e^x$, we end up with a bound that depends on the CGF of X and its derivative:

$$\begin{array}{}\mathrm{(28)}& \hfill \mathbb{E}\left\{e^{f\left(X\right)}g\left(X\right)\right\}& \ge & \mathbb{E}\left\{e^{f\left(a\right)+f^{\prime }\left(a\right)(X-a)}[g\left(b\right)+g^{\prime }\left(b\right)(X-b)]\right\}\hfill \mathrm{(29)}& & =& e^{f\left(a\right)-a{f}^{\prime }\left(a\right)}\mathbb{E}\left\{e^{X{f}^{\prime }\left(a\right)}[g\left(b\right)-b{g}^{\prime }\left(b\right)+g^{\prime }\left(b\right)X]\right\}\hfill \mathrm{(30)}& & =& exp\{f\left(a\right)-a{f}^{\prime }\left(a\right)+\psi \left[f^{\prime }\left(a\right)\right]\}\{g\left(b\right)+g^{\prime }\left(b\right)({\psi }^{\prime }\left[f^{\prime }\left(a\right)\right]-b)\}.\hfill \end{array}$$

Maximizing with respect to b while a is kept fixed yields $b_{\ast }={\psi }^{\prime }\left[f^{\prime }\left(a\right)\right]$, and we obtain:

$$\mathbb{E}\{e^{f\left(X\right)}g\left(X\right)\}\ge \underset{a}{sup}exp\{f\left(a\right)-a{f}^{\prime }\left(a\right)+\psi \left[f^{\prime }\left(a\right)\right]\}·g\left({\psi }^{\prime }\left[f^{\prime }\left(a\right)\right]\right).$$

(31)

Example 5. 

Considering the case where $f\left(x\right)=-lnx$ and $g\left(x\right)=xlnx$, we may obtain a reversed Jensen-like inequality, namely, a lower bound to the expectation of the concave function $lnX$:

$$\begin{array}{}\mathrm{(32)}& \hfill \mathbb{E}\{lnX\}& =& \mathbb{E}\left\{e^{-lnX}·XlnX\right\}\hfill \mathrm{(33)}& & \ge & \underset{a\ge 0}{sup}exp\{-lna+1+\psi (-1/a)\}·{\psi }^{\prime }(-1/a)ln{\psi }^{\prime }(-1/a)\hfill \mathrm{(34)}& & =& \underset{\alpha \ge 0}{sup}exp\{ln\alpha +1+\psi (-\alpha )\}{\psi }^{\prime }(-\alpha )ln{\psi }^{\prime }(-\alpha )\hfill \mathrm{(35)}& & =& e·\underset{\alpha \ge 0}{sup}\alpha e^{\psi (-\alpha )}{\psi }^{\prime }(-\alpha )ln{\psi }^{\prime }(-\alpha )\hfill \mathrm{(36)}& & =& e·\underset{\alpha \ge 0}{sup}\alpha \mathbb{E}\left\{X{e}^{-\alpha X}\right\}ln\frac{\mathbb{E}\left\{X{e}^{-\alpha X}\right\}}{\mathbb{E}\left\{e^{-\alpha X}\right\}}.\hfill \end{array}$$

Defining the MGF $\varphi \left(s\right)=\mathbb{E}\left\{e^{sX}\right\}=e^{\psi \left(s\right)}$, we have:

$$\begin{array}{}\mathrm{(37)}& \hfill \mathbb{E}\{lnX\}& \ge & e·\underset{\alpha \ge 0}{sup}\alpha {\varphi }^{\prime }(-\alpha )ln{\psi }^{\prime }(-\alpha )\hfill \mathrm{(38)}& & =& e·\underset{\alpha \ge 0}{sup}\alpha \varphi (-\alpha ){\psi }^{\prime }(-\alpha )ln{\psi }^{\prime }(-\alpha )\hfill \mathrm{(39)}& & =& e·\underset{\alpha \ge 0}{sup}\alpha {\varphi }^{\prime }(-\alpha )ln\frac{{\varphi }^{\prime }(-\alpha )}{\varphi (-\alpha )}.\hfill \end{array}$$

We obtained a lower bound in terms of the MGF and its derivative (or, equivalently, the CGF and its derivative), which is appealing in cases where X is the sum of i.i.d. random variables.

Accordingly, we now particularize this example further by examining the case where $X=1+{\sum }_{i=1}^k{Y}_i^2$, with $Y_i\sim \mathcal{N}(0,{\sigma }^2)$, $i=1,\dots ,k$, being independent random variables. The motivation of assessing an expression of the form, $\mathbb{E}\left\{ln\left(1+{\sum }_{i=1}^k{Y}_i^2\right)\right\}$, is two-fold. The first is that it is useful for bounding the ergodic capacity of the single-input, multiple-output (SIMO) channel, where $\left\{Y_i\right\}$ designates random channel transfer coefficients (see, e.g., [22,28,29] and references therein). The second is that it is relevant for bounding the joint differential entropy associated with the multivariate Cauchy density. Here, $(Y_1,\dots ,Y_k)$ are not Gaussian as defined above, but their multivariate Cauchy density can be represented as a continuous mixture of i.i.d. zero-mean Gaussian random variables, where the mixture is taken over all possible variances—see [22] (Example 6) for the details. In this case,

$$\begin{array}{}\mathrm{(40)}& \hfill \varphi \left(s\right)& =& \mathbb{E}\left\{exp\left(s\left[1+\sum _{i=1}^k{Y}_i^2\right]\right)\right\}\hfill \mathrm{(41)}& & =& e^s{\left(\mathbb{E}\left\{e^{s{Y}^2}\right\}\right)}^k\hfill \mathrm{(42)}& & =& \frac{e^s}{{(1-2s{\sigma }^2)}^{k/2}},s<\frac{1}{2{\sigma }^2}.\hfill \end{array}$$

Thus,

$$\psi \left(s\right)=s-\frac{k}{2}ln(1-2s{\sigma }^2),$$

(43)

and

$${\psi }^{\prime }\left(s\right)=1+\frac{k{\sigma }^2}{1-2s{\sigma }^2}.$$

(44)

It follows that

$$\mathbb{E}\left\{ln\left(1+\sum _{i=1}^k{Y}_i^2\right)\right\}\ge e·\underset{\alpha \ge 0}{sup}\left\{\frac{\alpha e^{-\alpha }}{{(1+2\alpha {\sigma }^2)}^{k/2}}\left(1+\frac{k{\sigma }^2}{1+2\alpha {\sigma }^2}\right)ln\left(1+\frac{k{\sigma }^2}{1+2\alpha {\sigma }^2}\right)\right\}.$$

(45)

The Jensen upper bound, $ln(1+k{\sigma }^2)$, and the lower bound (45) are displayed in Figure 1 for ${\sigma }^2=1$ and $k=1,2,\dots ,100$. As can be seen, the bounds are quite close. Interestingly, the choice $\alpha =1/\left(k{\sigma }^2\right)$ yields results that are very close to those of the optimal $\alpha$.

Another instance of this example is the circularly symmetric complex Gaussian channel whose signal-to-noise ratio (SNR), Z, is a random variable (e.g., due to fading), which is known to both the transmitter and the receiver. The capacity is given by $C=\mathbb{E}\{ln(1+gZ\left)\right\}$, where g is a certain deterministic gain factor and the expectation is with respect to the randomness of Z. For simplicity, let us assume that Z is distributed exponentially, i.e.,

$$p\left(z\right)=\left\{\begin{array}{cc}\theta e^{-\theta z}\hfill & z\ge 0\hfill \\ 0\hfill & z<0\hfill \end{array}\right.$$

(46)

where the parameter $\theta >0$ is given. In this case,

$$\varphi (-\alpha )=\frac{\theta e^{-\alpha }}{\theta +g\alpha }$$

(47)

and

$$\psi (-\alpha )=ln\theta -ln(\theta +g\alpha )-\alpha ,$$

(48)

and so,

$$\mathbb{E}\{ln(1+gZ)\}\ge e\theta ·\underset{\alpha \ge 0}{sup}\frac{\alpha e^{-\alpha }}{\theta +g\alpha }·\left(1+\frac{g}{g+\theta \alpha }\right)ln\left(1+\frac{g}{g+\theta \alpha }\right).$$

(49)

In Figure 2, we plot this lower bound as a function of $\theta$ for $g=5$ and compare it to the Jensen upper bound, $ln(1+g/\theta )$ (red curve) and to the lower bound of [22] (Sect. 4.1, Example 1). As can be seen, the lower bound proposed here is considerably tighter, especially for small $\theta$.

Example 6. 

Yet another example of this family of Jensen-like inequalities applies to obtaining a lower bound to $\mathbb{E}\left\{X^t\right\}$, where t is an arbitrary real. For a given t, let $s\ge 0$ be either larger than $1-t$ or smaller than $-t$, and consider the case where $f\left(x\right)=x^{t+s}$, $g\left(x\right)=-slnx$ and $h\left(x\right)=e^x$. Then,

$$\begin{array}{}\mathrm{(50)}& \hfill \mathbb{E}\left\{X^t\right\}& =& \mathbb{E}\left\{e^{-slnX}{X}^{t+s}\right\}\hfill \mathrm{(51)}& & \ge & \mathbb{E}\left\{exp\left[s\left(-lna-\frac{1}{a}(X-a)\right)\right]·\left[b^{t+s}+(t+s)b^{t+s-1}(X-b)\right]\right\}\hfill \mathrm{(52)}& & =& e^{s[1-lna]}\varphi \left(-\frac{s}{a}\right)\left[b^{t+s}+(t+s)b^{t+s-1}\left({\psi }^{\prime }\left(-\frac{s}{a}\right)-b\right)\right].\hfill \end{array}$$

Choosing $b={\psi }^{\prime }(-s/a)$, and changing the optimization variable a into $\alpha =1/a$, we obtain

$$\mathbb{E}\left\{X^t\right\}\ge \underset{\alpha \ge 0}{sup}{\left(\alpha e\right)}^s\varphi (-\alpha s){\left[{\psi }^{\prime }(-\alpha s)\right]}^{t+s}.$$

(53)

More specifically, if $X={\sum }_{i=1}^n{Y}_i$, where $\left\{Y_i\right\}$ are Bernoulli i.i.d., with parameter p, then $\varphi \left(s\right)={(p{e}^s+q)}^n$, where $q=1-p$. We then obtain

$$\mathbb{E}\left\{X^t\right\}\ge \underset{\alpha \ge 0}{sup}{\left(\alpha e\right)}^s{(p{e}^{-\alpha s}+q)}^n·{\left(\frac{np{e}^{-\alpha s}}{p{e}^{-\alpha s}+q}\right)}^{t+s}.$$

(54)

Selecting $\alpha =1/\left(np\right)$, we obtain

$$\mathbb{E}\left\{X^t\right\}\ge {\left(np\right)}^t·\frac{e^s{(p{e}^{-s/\left(np\right)}+q)}^n{e}^{-s(t+s)/\left(np\right)}}{{(p{e}^{-s/\left(np\right)}+q)}^{t+s}}.$$

(55)

The first factor is ${\left(\mathbb{E}X\right)}^t$. The second factor tends to unity as n grows, because $p{e}^{-s/np}+q\approx p(1-s/\left(np\right))+q=1-s/n$, and so, ${(p{e}^{-s/np}+q)}^n\approx {(1-s/n)}^n\approx e^{-s}$. For $t\ge 1$ and $t\le 0$, the function $f\left(x\right)=x^t$ is convex, and so, ${\left(\mathbb{E}X\right)}^t$ is the ordinary Jensen lower bound. In this case, the bound is valuable if the multiplicative factor,

$$\frac{e^s{(p{e}^{-s/\left(np\right)}+q)}^n{e}^{-s(t+s)/\left(np\right)}}{{(p{e}^{-s/\left(np\right)}+q)}^{t+s}},$$

is larger than unity. If $0<t<1$, the function $f\left(x\right)=x^t$ is concave, and then ${\left(\mathbb{E}X\right)}^t$ is an upper bound. Of course, the parameter s can be optimized, too. Some numerical results for $t=0.5$ are depicted in Figure 3. As can be seen, the upper and the lower bounds are fairly close.

Another application of this example is related to estimation theory. Let $\theta \in \mathbb{R}$ and let $Y_1,\dots ,Y_n$ be i.i.d., with mean $\theta$ and variance ${\sigma }^2$. Consider the t-th moment of the estimation error, ${\mathbb{E}}_{\theta }|\frac{1}{n}{\sum }_{i=1}^n{Y}_i{-\theta |}^t$. Defining $X={\left(\frac{1}{n}{\sum }_{i=1}^n{Y}_i-\theta \right)}^2$, we have

$$\varphi \left(s\right)=\frac{1}{\sqrt{1-2s{\sigma }^2/n}};\psi \left(s\right)=-\frac{1}{2}ln\left(1-\frac{2s{\sigma }^2}{n}\right).$$

(56)

and so,

$$\varphi (-\alpha s)=\frac{1}{\sqrt{1+2\alpha s{\sigma }^2/n}};{\psi }^{\prime }(-\alpha s)=\frac{{\sigma }^2/n}{1+2\alpha s{\sigma }^2/n}.$$

(57)

$$\begin{array}{}\mathrm{(58)}& \hfill {\mathbb{E}}_{\theta }|\frac{1}{n}\sum _{i=1}^n{Y}_i-\theta {|}^t& =& {\mathbb{E}}_{\theta }{X}^{t/2}\hfill & & \ge & \frac{{\left(\alpha e\right)}^s}{\sqrt{1+2\alpha s{\sigma }^2/n}}{\left(\frac{{\sigma }^2/n}{1+2\alpha s{\sigma }^2/n}\right)}^{t/2+s}\hfill \mathrm{(59)}& & =& {\left(\frac{{\sigma }^2}{n}\right)}^{t/2+s}·\frac{{\left(\alpha e\right)}^s}{{(1+2\alpha s{\sigma }^2/n)}^{(t+1)/2+s}}.\hfill \end{array}$$

with either $s\ge 1-t/2$ or $s\le -t/2$. For $\alpha =\zeta n/{\sigma }^2$ ($\zeta >0$ being a constant), we have:

$${\mathbb{E}}_{\theta }|\frac{1}{n}\sum _{i=1}^n{Y}_i-\theta {|}^t\ge \frac{{\sigma }^t}{n^{t/2}}·\underset{\zeta >0,s>1-t/2}{sup}\frac{{\left(\zeta e\right)}^s}{{(1+2\zeta s)}^{(t+1)/2+s}}$$

(60)

where for $t\in [0,2]$, the first factor, ${\sigma }^t/n^{t/2}$, is the Jensen upper bound. The second factor,

$${\mu }_t=\underset{\zeta >0,s>1-t/2}{sup}\frac{{\left(\zeta e\right)}^s}{{(1+2\zeta s)}^{(t+1)/2+s}},$$

(61)

is the gap between the Jensen upper bound and the proposed lower bound. In Figure 4, we display this factor. The result ${\mu }_2=1$ is expected, because for $t=2$ and $s=0$, the calculation is trivially exact. Note that the maximization over $\zeta$, for a given s, can be carried out in closed form by equating to zero the partial derivative of $ln[{\left(\zeta e\right)}^s/{(1+2\zeta s)}^{(t+1)/2+s}]$ with respect to $\zeta$. The optimal $\zeta$ turns out to be equal to $1/(t+1)$ (independently of s), and so,

$${\mu }_t=\underset{s>1-t/2}{sup}{\left(\frac{t+1}{t+2s+1}\right)}^{(t+1)/2}·{\left(\frac{e}{t+2s+1}\right)}^s.$$

(62)

Finally, it should be pointed out that this family of Jensen-like bounds opens the door also to lower-bound calculations on the form $\mathbb{E}\left\{f\right(X)/g(X\left)\right\}$, where f is non-negative convex and g is non-negative and concave. Using the fact the identity $1/s={\int }_0^{\infty }{e}^{-st}dt$, we have:

$$\begin{array}{}\mathrm{(63)}& \hfill \mathbb{E}\left\{\frac{f\left(X\right)}{g\left(X\right)}\right\}& =& \mathbb{E}\left\{f\left(X\right)·{\int }_0^{\infty }{e}^{-tg\left(X\right)}dt\right\}\hfill \mathrm{(64)}& & =& {\int }_0^{\infty }\mathbb{E}\left\{e^{-tg\left(X\right)}f\left(X\right)\right\}dt\hfill \end{array}$$

and we can apply the same ideas as before to the integrand, having the freedom to optimize the bound parameters with possible dependence on t.

5. A Product of Two Non-Negative Convex Functions

The last family of Jensen-like bounds that we present in this work is associated with the product of two non-negative convex functions. Let both f and g be non-negative convex functions of $x\ge 0$. Then,

$$\begin{array}{}\mathrm{(65)}& \hfill \mathbb{E}\left\{f\right(X\left)g\right(X\left)\right\}& \ge & \mathbb{E}\{[f\left(a\right)+f^{\prime }\left(a\right)(X-a)]·g\left(X\right)\}\hfill \mathrm{(66)}& & =& [f\left(a\right)-a{f}^{\prime }\left(a\right)]\mathbb{E}\left\{g\left(X\right)\right\}+f^{\prime }\left(a\right)\mathbb{E}\left\{Xg\left(X\right)\right)\}\hfill & & \ge & [f\left(a\right)-a{f}^{\prime }\left(a\right)]\mathbb{E}\left\{[g\left(b\right)+g^{\prime }\left(b\right)(X-b)]\right\}+\hfill \mathrm{(67)}& & & f^{\prime }\left(a\right)\mathbb{E}\left\{X[g\left(c\right)+g^{\prime }\left(c\right)(X-c)]\right\}f\left(a\right)\ge a{f}^{\prime }\left(a\right)\ge 0\hfill & & =& [f\left(a\right)-a{f}^{\prime }\left(a\right)]·[g\left(b\right)-b{g}^{\prime }\left(b\right)+g^{\prime }\left(b\right)\mathbb{E}\left\{X\right\}]+\hfill \mathrm{(68)}& & & f^{\prime }\left(a\right)[(g\left(c\right)-c{g}^{\prime }\left(c\right))\mathbb{E}\left\{X\right\}+g^{\prime }\left(c\right)\mathbb{E}\left\{X^2\right\}\}].\hfill \end{array}$$

The optimal b and c are $b^{\ast }=\mathbb{E}\left\{X\right\}$ and $c^{\ast }=\mathbb{E}\left\{X^2\right\}/\mathbb{E}\left\{X\right\}$, respectively. Thus,

$$\mathbb{E}\left\{f\left(X\right)g\left(X\right)\right\}\ge [f\left(a\right)-a{f}^{\prime }\left(a\right)]·g\left(\mathbb{E}\left\{X\right\}\right)+f^{\prime }\left(a\right)\mathbb{E}\left\{X\right\}·g\left(\frac{\mathbb{E}\left\{X^2\right\}}{\mathbb{E}\left\{X\right\}}\right).$$

(69)

Let

$$a^{\ast }=\frac{\mathbb{E}\left\{X\right\}·g(\mathbb{E}\left\{X^2\right\}/\mathbb{E}\left\{X\right\})}{g\left(\mathbb{E}\right\{X\left\}\right)}$$

(70)

and assume that $f\left(a^{\ast }\right)\ge a^{\ast }{f}^{\prime }\left(a^{\ast }\right)\ge 0$. Then, $a^{\ast }$ is the optimal value of a, which yields

$$\mathbb{E}\left\{f\left(X\right)g\left(X\right)\right\}\ge f\left(\frac{\mathbb{E}\left\{X\right\}·g(\mathbb{E}\left\{X^2\right\}/\mathbb{E}\left\{X\right\})}{g\left(\mathbb{E}\right\{X\left\}\right)}\right)·g\left(\mathbb{E}\left\{X\right\}\right).$$

(71)

More generally, when X and Y are two random variables with a joint distribution, the above derivation easily extends to

$$\mathbb{E}\left\{f\left(X\right)g\left(Y\right)\right\}\ge f\left(\frac{\mathbb{E}\left\{X\right\}·g\left(\mathbb{E}\right\{XY\}/\mathbb{E}\{X\left\}\right)}{g\left(\mathbb{E}\right\{Y\left\}\right)}\right)·g\left(\mathbb{E}\left\{Y\right\}\right).$$

(72)

If f and g are both concave, rather than convex, then the inequalities are reversed.

Example 7. 

Consider again the example of the capacity of the AWGN with a random SNR, $c\left(Z\right)=ln(1+gZ)$, and suppose that we wish to bound the variance of $c\left(Z\right)$ in order to assess the fluctuations (e.g., for the purpose of bounding the outage probability). Then, obviously,

$$Var\left\{c\left(Z\right)\right\}=\mathbb{E}\left\{c^2\left(Z\right)\right\}-{\left[\mathbb{E}\left\{c\left(Z\right)\right\}\right]}^2=\mathbb{E}\left\{{ln}^2(1+gZ)\right\}-{\left[\mathbb{E}\{ln(1+gZ)\}\right]}^2.$$

(73)

To upper bound $Var\left\{c\right(Z\left)\right\}$, we may derive an upper bound to $\mathbb{E}\left\{{ln}^2(1+gZ)\right\}$ and a lower bound to $\mathbb{E}\{ln(1+gZ\left)\right\}$. For the latter, a lower bound was already proposed earlier in Example 5. For the former, we may use the present inequality with the choice $f\left(z\right)=g\left(z\right)=ln(1+gz)$, which can easily be shown to satisfy the requirements. We then obtain the following upper bound, which depends merely on the first two moments of Z:

$$\mathbb{E}\left\{{ln}^2(1+gZ)\right\}\le ln(1+g\mathbb{E}\left\{Z\right\})·ln\left(1+\frac{g\mathbb{E}\left\{Z\right\}ln(1+g\mathbb{E}\left\{Z^2\right\}/\mathbb{E}\left\{Z\right\})}{ln(1+g\mathbb{E}\{Z\left\}\right)}\right).$$

(74)

Interestingly, the function ${ln}^2(1+gx)$ is neither convex nor concave, yet our approach offers an upper bound, which is fairly easy to calculate provided that one can compute the first two moments of Z.

6. Conclusions

In this work, we have revisited the Jensen inequality on the basis of taking advantage of the freedom to optimize the choice of the supporting line that is tangential to the given convex function. This optimal choice might be different than the ordinary one when the convex function does not stand alone, but it is rather only part of a more complicated expression. This more complicated expression can sometimes be created in an artificial manner, such as in Examples 2, 5 and 6. The resulting bounds depend on either the first two moments of the independent variable, X, or on its MGF and its derivative. Both types of moments often lend themselves to relatively easy calculations. The proposed methodology can be used both for improving on the ordinary Jensen inequality (such as in Examples 2 and 4), and for generating lower bounds to expectations of non-convex or even concave (rather than convex) functions (such as in Examples 1, 2, 5 and 7). Several families of Jensen-like inequalities have been derived along with a demonstration of numerical examples with application to information theory. The tightness of the inequalities obtained was also demonstrated in those examples.