Robust Universal Inference

Abstract

Learning and making inference from a finite set of samples are among the fundamental problems in science. In most popular applications, the paradigmatic approach is to seek a model that best explains the data. This approach has many desirable properties when the number of samples is large. However, in many practical setups, data acquisition is costly and only a limited number of samples is available. In this work, we study an alternative approach for this challenging setup. Our framework suggests that the role of the train-set is not to provide a single estimated model, which may be inaccurate due to the limited number of samples. Instead, we define a class of “reasonable” models. Then, the worst-case performance in the class is controlled by a minimax estimator with respect to it. Further, we introduce a robust estimation scheme that provides minimax guarantees, also for the case where the true model is not a member of the model class. Our results draw important connections to universal prediction, the redundancy-capacity theorem, and channel capacity theory. We demonstrate our suggested scheme in different setups, showing a significant improvement in worst-case performance over currently known alternatives.

1. Introduction

One of the major challenges in statistics and machine learning is making predictions and inference from a limited number of samples. This problem is mostly evident in modern statistics (big data), where the dimension of the problem is very large compared to the number of samples in hand, or in cases where data acquisition is relatively costly, and only a small number of samples is available (such as in complicated clinical trails). The standard approach in many applications is to seek a model that best explains the data. For example, empirical risk minimization (ERM) [1] is a commonly used criterion in predictive modeling. Minimizing the empirical risk has many desirable properties. Under different loss functions, we may attain consistency, unbiasedness, and other favorable attributes. In parametric estimation, perhaps the most popular approach is maximum likelihood. Here, again, we seek parameters that maximize the likelihood of the given set of observations.

However, what happens if our specific instance of data does not represent the true model well enough (as happens in high-dimensional problems)? Is it still desirable to choose the single model that best explains it?

In this work, we study an alternative approach for this challenging setup. Here, instead of choosing a model that best describes the data, we define a class of models that describe it with high confidence. Then, we seek a scheme that minimizes the worst-case loss in the class. This way, we control the performance over a class of reasonable models and provide explicit worst-case guarantees, even when the given data fail to accurately represent the true model. This scheme is, in fact, a data-driven approach of minimax estimation, as later discussed. Further, we show it provides worst-case guarantees for the expected regret of future samples. This property makes our framework applicable both for inference and prediction tasks.

One of the major challenges of our suggested scheme is to characterize the model class. In this work, we consider a class of models that corresponds to a confidence region of the unknown parameters. This way we provide a PAC-like generalization bound, as the true model is a member of this class with high confidence. Then, we introduce a more robust approach which drops the model class assumptions and provides stronger performance guarantees for the derived estimator. We demonstrate our suggested approach in classical inference problems and more challenging large alphabet probability estimation. We further study a real-world example, motivated from the medical domain. Our suggested approach introduces favorable worst-case performance, for every given instance of data, at a typically low cost on the average. This demonstrates an “insurance-like” trade-off; we pay a small cost on the average to avoid a great loss if “something bad happens” (that is, the observed samples do not represent the true model well enough).

The rest of this manuscript is organized as follows. In Section 2, we review related work to our problem. We introduce our suggested framework and some of its basic properties in Section 3. Then, we extend the framework to a more robust estimation scheme in Section 4. We demonstrate our suggested scheme in several setups. In Section 5, we study the unknown normal mean problem, while in Section 6 we focus on multinomial probability estimation. We consider a more challenging large alphabet probability estimation problem in Section 7. Finally, we study a real-work breast cancer problem in Section 8. We conclude with a discussion in Section 9.

2. Previous Work

Minimax estimation has been extensively studied over the years. Here, we briefly review the more relevant results for our work. Let $x^n\sim p_{\theta }^n$ be a collection of n i.i.d. samples, drawn from a distribution $p_{\theta }$, where $\theta$ is a fixed and unknown parameter. Let $\mathrm{\Theta }$ be a given class of parameters. Assume that $\theta \in \mathrm{\Theta }$. Let $\widehat{\theta }\triangleq \widehat{\theta }\left(x^n\right)$ be an estimator of $\theta$ from $x^n$. Let $R(\theta ,\widehat{\theta })$ be a risk function which measures the expected error between the true parameter $\theta$ and its corresponding estimate $\widehat{\theta }$. For example, $R(\theta ,\widehat{\theta })={\mathbb{E}}_{x^n\sim p_{\theta }^n}{(\theta -\widehat{\theta }\left(x^n\right))}^2$ is the mean squared error. The minimax risk [2] is defined as

$$r_{mm}=\underset{\widehat{\theta }}{inf}\underset{\theta \in \mathrm{\Theta }}{sup}R(\theta ,\widehat{\theta }).$$

(1)

A minimax estimator ${\widehat{\theta }}_{mm}$ satisfies ${sup}_{\theta \in \mathrm{\Theta }}R(\theta ,{\widehat{\theta }}_{mm})=r_{mm}$, if such exists. In words, ${\widehat{\theta }}_{mm}$ minimizes the worst-case risk for a given class of parameters $\mathrm{\Theta }$. Finding the minimax estimator is, in general, not an easy task. However, the optimal solution is characterized by several important properties.

Let ${\widehat{\theta }}_{\pi }={\int }_{\theta \in \mathrm{\Theta }}\theta \pi \left(\theta \right)d\theta$ be a Bayes estimator with respect to some prior $\pi \left(\theta \right)$ over $\mathrm{\Theta }$. In words, ${\widehat{\theta }}_{\pi }$ is a weighted average of $\theta \in \mathrm{\Theta }$, according to a given weight function $\pi \triangleq \pi \left(\theta \right)$. Let $r_{\pi }={\int }_{\theta \in \mathrm{\Theta }}R(\theta ,{\widehat{\theta }}_{\pi })d\pi \left(\theta \right)$ be the average risk with respect to $\pi$. One of the basic results in the minimax theory suggests that if $r_{\pi }=r_{mm}$, then ${\widehat{\theta }}_{\pi }$ is a minimax estimator and $\pi$ is a least favorable prior (satisfying $r_{\pi }\ge r_{{\pi }^{\prime }}$ for any ${\pi }^{\prime }$) [2]. Importantly, if a Bayes estimator has a constant risk, it is minimax. Note that this is not a necessary condition.

For example, consider the problem of estimating the mean of a d-dimensional Gaussian vector. Here, it can be shown that the maximum likelihood estimator (MLE) is also the minimax estimator with respect to the squared error. Interestingly, in this example, the MLE is known to be inadmissible for $d2$; assuming that the mean is finite, the famous James–Stein estimator [3] dominates the MLE, as it achieves a lower mean squared error (where the phenomenon is more evident as the mean is closer to zero) [2,3]. Additional examples for the minimax estimators are provided in [2].

The minimax formulation was studied in a variety of setups. In [4], the authors considered minimax estimation of parameters over $L^p$ loss and provided key analytical results. These results were further studied and generalized (for example, see in [5]). Bickel studied minimax estimation of the normal mean when the parameter space is restricted [6]. Later, Marchand and Perron considered the case where the norm of the normal mean is bounded [7]. The minimax approach is also applicable to supervised learning problems. In [8], the authors considered minimax classification with fixed first- and second-order moments. Eban et al. developed a classification approach by minimizing the worst-case hinge loss subject to fixed low-order marginals [9]. Razaviyayn et al. fitted a model that minimizes the maximal correlation under fixed pairwise marginals to design a robust classification scheme [10]. Farnia et al. described a minimax approach for supervised learning by generalizing the maximum entropy principle [11].

The minimax approach has many applications, as it defines a conservative estimate for a given class of models. A variety of examples spans different fields including optimization [12], signal processing [13,14], communications [15], and others [16].

It is important to emphasize that the minimax problem (1), and its corresponding solution, heavily depend on the assumption that the unknown parameter $\theta$ is a member of the given class of parameters $\mathrm{\Theta }$. However, what happens if this assumption is false, and $\theta \notin \mathrm{\Theta }$ (as discussed, for example, in [17,18,19])? Furthermore, how do we choose $\mathrm{\Theta }$ in practice? If we choose $\mathrm{\Theta }$ to be too large, we might control a class of models that are unreasonable. On the other hand, if $\mathrm{\Theta }$ is too small, we may violate the assumption that $\theta \in \mathrm{\Theta }$. Finally, notice that the minimax problem is typically concerned with the expected worst-case performance (the risk). However, in many real-world applications we are given a single instance of data, which may be quite costly to acquire. Therefore, we require worst-case performance guarantees for this specific instance of data. In this work, we address these concerns and suggest a robust, data-driven, universal estimation scheme for a given set of observations.

3. The Suggested Inference Scheme

For the purpose of our presentation, we use the following additional notations. Let ${\mathrm{\Theta }}_r$ be a restricted class of parameters and denote $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ as the corresponding restricted class of parametric distributions. Assume that the true model $p_{\theta }$ is a member of $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ (or alternatively, $\theta \in {\mathrm{\Theta }}_r$). For example, $p_{\theta }=\mathcal{N}(\theta ,1)$ is a normal distribution with an unknown mean $\theta$ and a unit variance, while $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ is a set of all normal distributions with $\theta \in {\mathrm{\Theta }}_r=[{\theta }_a,{\theta }_b]$ (henceforth, restricted to $[{\theta }_a,{\theta }_b]$) and a unit variance. Let $\mathcal{P}=\left\{p\right|p\left(x\right)\ge 0,\int p\left(x\right)dx=1\}$ be the class of all probability measures. Let $q\triangleq q(·|x^n)$ be a probability measure which estimates $p_{\theta }$ given the samples $x^n$. Notice that as opposed to the presentation in (1), the estimator q is with respect to the entire probability distribution $p_{\theta }$, and not just unknown parameter $\theta$. We measure the estimate’s accuracy using the Kullback–Leibler ($\mathrm{KL}$) divergence between the true underlying distribution $p_{\theta }$ and q, formally defined as $D_{\mathrm{KL}}(p_{\theta }\left|\right|q)=\int p_{\theta }\left(x\right)log\frac{p_{\theta }\left(x\right)}{q\left(x\right)}dx$. The KL divergence is a widely used measure for the discrepancy between two probability distributions, with many desirable properties [20]. In addition, the KL divergence serves as an upper bound for a collection of popular discrepancy measures (for example, the Pinsker inequality [21] and the universality results in [22,23]). In this sense, by minimizing the KL divergence, we implicity bound from above a large set of common performance merits.

Ultimately, our goal is to find an estimate q that minimizes $D_{\mathrm{KL}}(p_{\theta }\left|\right|q)$ for the unknown $\theta$. Thus, we consider a minimax formulation

$$\begin{array}{c}\hfill \underset{q}{min}\underset{p_{\theta }\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)}{sup}{D}_{\mathrm{KL}}(p_{\theta }\left|\right|q)\end{array}$$

(2)

where $q\in \mathcal{P}$ is the minimizer of the worst-case divergence over the class $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$, if such exists. In words, q minimizes the worst possible divergence, over the restricted model class $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$. To avoid an overload of notation, we assume that $q\in \mathcal{P}$ throughout the text, unless otherwise stated. We observe several differences between (1) and (2). First, the formulation in (1) considers an estimate $\widehat{\theta }$, and by that implicitly restricts the solution to be a parametric distribution $p_{\widehat{\theta }}$ of the same family as $p_{\theta }$. On the other hand, (2) considers the entire distribution and does not impose any restrictions on the solution. Second, the standard minimax formulation (1) focuses on the risk. Our approach considers the estimation accuracy for every given instance of data ($q\triangleq q(·|x^n)$, as defined above). Third, notice that $D_{\mathrm{KL}}(p_{\theta }\left|\right|q)$ can also be viewed as the expected log-loss regret, where the expectation is with respect to a future sample, $D_{\mathrm{KL}}(p_{\theta }\left|\right|q)={\mathbb{E}}_{x\sim p_{\theta }}\left(l(x,q)-l(x,p_{\theta })\right)$ and $l(x,q)=-log\left(q\right(x\left)\right)$ is the logarithmic loss. This means that while (1) focuses on the expected loss with respect to the given samples, (2) considers the expected performance of future samples. We further discussed these points in Section 5, Section 6 and Section 8.

In practice, one of the major challenges in applying any minimax formulation is the choice (or design) of the parametric class. Specifically, using the notations of (2), choosing a larger class ${\mathrm{\Theta }}_r$ is more likely to include the true model $\theta$, but may also include unreasonable worst-case models (for example, ${\mathrm{\Theta }}_r=\mathbb{R}$ in the unknown normal mean example above). On the other hand, choosing a more restrictive ${\mathrm{\Theta }}_r$ may violate our assumption that $\theta \in {\mathrm{\Theta }}_r$. Therefore, a trade-off between the two seems inevitable. In the following, we focus on the design and characterization of a set ${\mathrm{\Theta }}_r$ that depends on the given samples, ${\mathrm{\Theta }}_r\left(x^n\right)$. In other words, we use the train-set $x^n$ to construct a minimal-size restricted model class ${\mathrm{\Theta }}_r\left(x^n\right)$ that contains the true parameter $\theta$ with high confidence. Then, we solve the minimax problem (2) with respect to it and attain an estimator that minimizes the maximal divergence in the class (or equivalently, the expected log-loss regret for future samples).

Designing and Controlling the Restricted Model Class

Our first objective is to construct a minimal-size ${\mathrm{\Theta }}_r\left(x^n\right)$ such that $\theta \in {\mathrm{\Theta }}_r\left(x^n\right)$ with high confidence. For this purpose, we turn to classical statistics and construct a confidence region for the desired parameter $\theta$. A confidence region of level $100(1-\alpha )\%$ is defined as a region $\mathcal{T}$ such that $P(\theta \in \mathcal{T})=1-\alpha$. Notice that $\mathcal{T}$ is random and depends on the samples $x^n$, while $\theta$ is an unknown (non-random) parameter. Further, notice that a confidence region $\mathcal{T}$ is data-dependent and does not require knowledge of the true parameter $\theta$. Obviously, there are many ways to define $\mathcal{T}$ to satisfy the above. We are interested in a confidence region that has a minimal expected volume. For example, consider n i.i.d. samples from $\mathcal{N}(\theta ,1)$, as discussed above. Let $\overline{x}$ be the sample mean. Then, the minimal-size $100(1-\alpha )\%$ confidence interval is $\left[\overline{x}\pm z_{\frac{\alpha }{2}}\frac{1}{\sqrt{n}}\right]$, where $z_{\alpha }$ is the upper $100\alpha$ percentile of a standard normal distribution [24].

Given the restrictive model class, we would like to solve the minimax problem defined in (2). A general form of this problem was extensively studied over the years, mainly in the context of universal compression and universal prediction [15]. There, $D_{\mathrm{KL}}(p_{\theta }\left|\right|q)$ is the expected number of extra bits (over the optimal code-book), required to code samples from $p_{\theta }$ using a code designed for q. The celebrated redundancy-capacity theorem demonstrates a basic connection between the desired formulation (2) and channel capacity theory. Let $T\sim \pi$ be a source variable, X be a target variable and $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ be the set of transition probabilities from X to T. In other words, T is a message, transmitted through a noisy channel, characterized by $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$. The received (noisy) message is denoted by X. Let $I(T;X)$ be the mutual information between T and X, and $C\left({\mathrm{\Theta }}_r\right)\triangleq {sup}_{\pi }I(T;X)$ be the corresponding channel capacity. The redundancy-capacity theorem [25,26,27] suggests that for $C\left({\mathrm{\Theta }}_r\right)\infty$, the minimax formulation presented in (2) is equivalent to

$$\begin{array}{c}\hfill \underset{\pi \left(\theta \right)}{sup}{\int }_{\theta \in {\mathrm{\Theta }}_r}\pi \left(\theta \right)D_{\mathrm{KL}}(p_{\theta }\left|\right|q_{\pi })d\theta =\underset{\pi }{sup}I(T;X)\triangleq C\left({\mathrm{\Theta }}_r\right)\end{array}$$

(3)

where $\pi \left(\theta \right)$ is a weight function for every $\theta \in {\mathrm{\Theta }}_r$ and $q_{\pi }={\int }_{\theta \in {\mathrm{\Theta }}_r}\pi \left(\theta \right)p_{\theta }d\theta$ is a mixture distribution. In words, solving (2) is equivalent to solving a channel capacity problem. Furthermore, the source distribution which maximizes the mutual information between the source and the target (and henceforth achieves the channel capacity) is a mixture over $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$. This solution is quite similar to the solution of (1); in both cases, we obtain a Bayes estimator over the given class, while the least favorable prior (or equivalently, the capacity achieving prior), if such exists, attains the maximal average risk (the channel capacity). See examples in Section 5 and Section 6 for further detail. It is important to emphasize that while $\theta$ is a fixed and unknown parameter, $\pi \left(\theta \right)$ is a weight function for every $\theta \in {\mathrm{\Theta }}_r$, and $q_{\pi }$ is a weighted average over $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$.

The redundancy-capacity theorem shows that the solution to the minimax problem (2) is achieved by solving the channel capacity problem (3). We apply the capacity-redundancy theorem to our suggested class ${\mathrm{\Theta }}_r\left(x^n\right)$ to attain the desired solution. Theorem 1 below summarizes our parametric inference approach.

Theorem 1.

Let $x^n\sim p_{\theta }^n$ be a collection of n i.i.d. samples, drawn from an unknown distribution $p_{\theta }$. Let ${\mathrm{\Theta }}_r\left(x^n\right)$ be a $100(1-\alpha )\%$ confidence region for the parameter θ. Assume that $C\left({\mathrm{\Theta }}_r\left(x^n\right)\right)\infty$. Then, with probability $1-\alpha$ (over the samples), $C\left({\mathrm{\Theta }}_r\left(x^n\right)\right)$ is the minimal worst-case divergence and $q_{\pi }$ is the corresponding minimax estimator, denoted as the mixture model.

Theorem 1 establishes a PAC-like generalization bound for parametric inference. It defines the worst-case expected performance of future samples (with respect to a logarithmic loss, as discussed above), at a confidence level of $1-\alpha$ over the drawn samples. Specifically, with probability $1-\alpha$ we have that ${\mathbb{E}}_{x\sim p_{\theta }}\left(l(x,q_{\pi })-l(x,p_{\theta })\right)\le C\left({\mathrm{\Theta }}_r\left(x^n\right)\right)$ for the entire parametric class. It is important to emphasize that the resulting minimax estimator $q_{\pi }$ is data-dependent, as it is a mixture over the data-driven restricted model class.

Solving the channel capacity problem is, in general, not an easy task. However, there exist several cases where the solution to (3) holds a closed-form expression, or an efficient computational routine. We demonstrate basic examples in Section 5 and Section 6.

4. A Generalized Inference Scheme beyond the Restricted Class

In the previous section, we derive a minimax solution to (2) under the assumption that $p_{\theta }\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)$ (equivalently, $\theta \in {\mathrm{\Theta }}_r$), with high confidence. Unfortunately, it does not provide any guarantee for the event where $p_{\theta }\notin \mathcal{P}\left({\mathrm{\Theta }}_r\right)$. We now consider a general setup where $p_{\theta }$ is not necessarily in $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ as well as introduce a more robust approach which addresses this case.

Let $\mathcal{P}\left(\mathrm{\Theta }\right)$ be a (non-restricted) model class that is known to contain the true parametric model $p_{\theta }$. Here, we define $\mathrm{\Theta }$ as the set of all possible parameter values, such that $\theta \in \mathrm{\Theta }$. For example, $\mathcal{P}\left(\mathrm{\Theta }\right)$ is a class of all normal distributions with an unknown mean and a unit variance ($\mathrm{\Theta }=\mathbb{R}$), in the normal mean example above. As before, we would like to find a distribution q that minimizes $D_{\mathrm{KL}}(p_{\theta }\left|\right|q)$. Simple calculus shows that

$$\begin{array}{c}\hfill D_{\mathrm{KL}}(p_{\theta }\left|\right|q)=D_{\mathrm{KL}}(p_{\theta }\left|\right|p_{{\theta }^{\prime }})+\int p_{\theta }\left(x\right)log\frac{p_{{\theta }^{\prime }}\left(x\right)}{q\left(x\right)}dx\end{array}$$

(4)

for any choice of $p_{{\theta }^{\prime }}$. Specifically, (4) holds for any model in the restricted model class, $p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)$. Notice that in this case, the first term of (4) is an error induced by the restrictive model class, independent of the choice of q. The second term is the residual, which depends on q. Notice that the first term only depends on $p_{\theta }$ and the reference distribution $p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)$. This means that by choosing a model class $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ that is too “far” (or from the true distribution), we face a large overhead term that is independent of the estimator q. On the other hand, the second term depends on q, and may be universally bounded. In other words, we are interested in a universal bound of the form

$$\begin{array}{cc}\hfill & \underset{q}{min}\underset{p_{\theta }\in \mathcal{P}\left(\mathrm{\Theta }\right)}{sup}\underset{p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)}{sup}\left(\int p_{\theta }\left(x\right)log\frac{p_{{\theta }^{\prime }}\left(x\right)}{q\left(x\right)}dx\right).\hfill \end{array}$$

(5)

Interestingly, notice that (5) may be equivalently written as

$$\begin{array}{cc}\hfill & \underset{q}{min}\underset{p_{\theta }\in \mathcal{P}\left(\mathrm{\Theta }\right)}{sup}\underset{p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)}{sup}\left(\int p_{\theta }\left(x\right)log\frac{p_{\theta }\left(x\right)}{q\left(x\right)}dx-\int p_{\theta }\left(x\right)log\frac{p_{\theta }\left(x\right)}{p_{{\theta }^{\prime }}\left(x\right)}dx\right)=\hfill \\ \hfill & \underset{q}{min}\underset{p_{\theta }\in \mathcal{P}\left(\mathrm{\Theta }\right)}{sup}\left(D_{\mathrm{KL}}(p_{\theta }\left|\right|q)-\underset{p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)}{inf}{D}_{\mathrm{KL}}(p_{\theta }\left|\right|p_{{\theta }^{\prime }})\right).\hfill \end{array}$$

(6)

This means that (5) is just a constrained variant of (2). Therefore, similarly to (2), we would like to represent (5) as a (constrained) channel capacity problem.

Definition 1.

Let $p_{\theta }\in \mathcal{P}\left(\mathrm{\Theta }\right)$ be an unknown probability distribution. Let $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ be a restrictive model class (that does not necessarily contain $p_{\theta }$). Assume that ${\mathrm{\Theta }}_r$ is bounded. Define

$$\begin{array}{cc}\hfill F(\mathrm{\Theta },{\mathrm{\Theta }}_r)\triangleq & \underset{\pi \left(\theta \right)}{sup}{\int }_{\theta \in \mathrm{\Theta }}\pi \left(\theta \right)\left(D_{\mathrm{KL}}(p_{\theta }\left|\right|q_{\pi })-\underset{{\theta }^{\prime }\in {\mathrm{\Theta }}_r}{min}{D}_{\mathrm{KL}}(p_{\theta }\left|\right|p_{{\theta }^{\prime }})\right)d\theta =\hfill \\ \hfill & \underset{\pi \left(\theta \right)}{sup}\left(I(T;X)-{\mathbb{E}}_{\pi \left(\theta \right)}\underset{{\theta }^{\prime }\in {\mathrm{\Theta }}_r}{min}{D}_{\mathrm{KL}}(p_{\theta }\left|\right|p_{{\theta }^{\prime }})\right).\hfill \end{array}$$

(7)

As in (2), $I(T;X)$ is the mutual information between a source variable $T\sim \pi$, and a target variable X, that is characterized by the transition probabilities $\mathcal{P}\left(\mathrm{\Theta }\right)$. The constraint is simply the expected divergence (with respect to $\pi$) between $p_{\theta }$ and its closest projection in ${\mathrm{\Theta }}_r$. The term $q_{\pi }$ is a mixture distribution over $\mathcal{P}\left(\mathrm{\Theta }\right)$, according to the prior $\pi$. Notice that here, the mixture is over the non-restricted model class, as opposed to (2), where the mixture is over $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$. We denote this distribution, $q_{\pi }$, as the projected mixture distribution.

Theorem 2.

Let $p_{\theta }\in \mathcal{P}\left(\mathrm{\Theta }\right)$ be an unknown probability distribution. Let $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ be a restrictive model class (that does not necessarily contain $p_{\theta }$). Assume that ${\mathrm{\Theta }}_r$ is bounded. Then, for $F(\mathrm{\Theta },{\mathrm{\Theta }}_r)\infty$ the following holds:

$$\begin{array}{c}\hfill \underset{q}{min}\underset{p_{\theta }\in \mathcal{P}\left(\mathrm{\Theta }\right)}{sup}\underset{p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)}{sup}\left(\int p_{\theta }\left(x\right)log\frac{p_{{\theta }^{\prime }}\left(x\right)}{q\left(x\right)}dx\right)=F(\mathrm{\Theta },{\mathrm{\Theta }}_r).\end{array}$$

(8)

A proof of Theorem 2 is provided in Appendix A. Theorem 2 establishes a redundancy-capacity result, similarly to (2). It shows that (5) may be obtained by solving a constrained channel capacity problem, and the distribution which achieves it is, again, a mixture distribution. This result is further discussed in [28] in a different (asymptotic) setup.

In addition, notice that for a bounded ${\mathrm{\Theta }}_r$ the formulation in (5) may be equivalently written as

$$\underset{q}{min}\underset{p_{\theta }\in \mathcal{P}\left(\mathrm{\Theta }\right)}{sup}\int p_{\theta }\left(x\right)log\frac{p_{\theta }^{\ast }\left(x\right)}{q\left(x\right)}dx,$$

where $p_{\theta }^{\ast }={argmin}_{p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)}{D}_{\mathrm{KL}}(p_{\theta }\left|\right|p_{{\theta }^{\prime }})$. In other words, $F(\mathrm{\Theta },{\mathrm{\Theta }}_r)$ is also the optimal universal minimizer of the second term of (4), for a specific (greedy) choice of $p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)$ that minimizes the first term. This result may be viewed as a “triangle inequality” for the KL divergence: given a reference set $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$, the KL divergence $D_{\mathrm{KL}}(p_{\theta }\left|\right|q)$ is bounded from above by the closest projection in $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ to $p_{\theta }$, plus an overhead-term $F(\mathrm{\Theta },{\mathrm{\Theta }}_r)$. It is important to emphasize that $F(\mathrm{\Theta },{\mathrm{\Theta }}_r)$ is not new to the universal coding literature. In fact, it was introduced in [17] as relative redundancy, in the context of robust codes for universal compression. However, it was mostly studied in an asymptotic regime, where n i.i.d. variables $X^n$ are simultaneously compressed. However, it was mostly studied in an asymptotic regime, where n i.i.d. variables Xⁿ are simultaneously compressed [28].

Similarly to the channel capacity problem, the term $F(\mathrm{\Theta },{\mathrm{\Theta }}_r)$ holds a closed-form analytical expression only in several special cases. Therefore, we introduce a simple iterative algorithm, which provides an optimal solution to it (as indicated in [28]). Our suggested routine is similar in spirit to the Blahut–Arimoto algorithm [21], which is typically applied to intractable channel capacity problems.

Theorem 3.

Let $\mathcal{P}\left(\mathrm{\Theta }\right)$ and $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ be two model classes. Let $F(\mathrm{\Theta },{\mathrm{\Theta }}_r)$ follow the definition above. Assume that ${\mathrm{\Theta }}_r$ is bounded. Then, for $F(\mathrm{\Theta },{\mathrm{\Theta }}_r)\infty$ the following holds:

$$\begin{array}{c}\hfill F(\mathrm{\Theta },{\mathrm{\Theta }}_r)=\underset{\varphi \left(\theta \right),\psi (\theta ,x)}{sup}{\int }_{\theta \in \mathrm{\Theta }}{\int }_x\varphi \left(\theta \right)p_{\theta }\left(x\right)log\frac{\psi (\theta ,x)}{\varphi \left(\theta \right)}\frac{p_{\theta }^{\ast }\left(x\right)}{p_{\theta }\left(x\right)}dxd\theta \end{array}$$

(9)

where $\varphi \left(\theta \right)$ and $\psi (\theta ,x)$ are probability distributions (over the variable θ, for any given x), and $p_{\theta }^{\ast }\left(x\right)={argmin}_{{\theta }^{\prime }\in {\mathrm{\Theta }}_r}{D}_{\mathrm{KL}}(p_{\theta }\left|\right|p_{{\theta }^{\prime }})$. Further, the solution to (9) may be attained by the following iterative projection algorithm:

• For a fixed $\varphi \left(\theta \right)$, we set $\psi (\theta ,x)=\frac{\varphi \left(\theta \right)p_{\theta }\left(x\right)}{{\int }_{\theta \in \mathrm{\Theta }}\varphi \left(\theta \right)p_{\theta }\left(x\right)d\theta }$
• For a fixed $\psi (\theta ,x)$, we set $\varphi \left(\theta \right)=\frac{{\prod }_x{\tilde{\psi }(\theta ,x)}^{p_{\theta }\left(x\right)}}{{\int }_{\theta \in \mathrm{\Theta }}{\prod }_x{\tilde{\psi }(\theta ,x)}^{p_{\theta }\left(x\right)}d\theta }$ where $\tilde{\psi }(\theta ,x)=\psi (\theta ,x)\frac{p_{\theta }^{\ast }\left(x\right)}{p_{\theta }\left(x\right)}$.

Finally, the distribution q that achieves $F(\mathrm{\Theta },{\mathrm{\Theta }}_r)$ is given by $q_{\mathrm{\Theta }}={\int }_{\theta \in \mathrm{\Theta }}{\varphi }^{\ast }\left(\theta \right)p_{\theta }d\theta$, where ${\varphi }^{\ast }\left(\theta \right)$ is $\varphi \left(\theta \right)$ at the final iteration of the algorithm.

A proof for this theorem is provided in Appendix B.

In many practical cases, the choice of a model class $\mathcal{P}\left(\mathrm{\Theta }\right)$ is not a trivial task. For example, consider a real-world setup where a domain expert suggests that the underlying model follows a Normal distribution with an unknown mean. However, we would like to design a scheme that does not heavily rely on this assumption. Therefore, we may consider the general case where $\mathcal{P}\left(\mathrm{\Theta }\right)$ is the simplex of all possible probability distributions. This important special case described in the following section.

The Normalized Maximum Likelihood

Consider a model class $\mathcal{P}\left(\mathrm{\Theta }\right)=\mathcal{P}=\left\{p\right|p\left(x\right)\ge 0,\int p\left(x\right)dx=1\}$. Here, the solution to (7) holds a closed form expression.

Theorem 4.

Let $p_{\theta }\in \mathcal{P}$ be an unknown probability distribution where $\mathcal{P}=\left\{p\right|p\left(x\right)\ge 0,\int p\left(x\right)dx=1\}$. Let $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ be a restrictive model class. Assume that ${\mathrm{\Theta }}_r$ is bounded and $Z\triangleq \int {max}_{p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)}{p}_{{\theta }^{\prime }}\left(x\right)dx$. Let $\Gamma \left({\mathrm{\Theta }}_r\right)\triangleq log\left(Z\right)$. For $\Gamma \left({\mathrm{\Theta }}_r\right)\infty$,

$$\underset{q}{min}\underset{p_{\theta }\in \mathcal{P}}{sup}\underset{p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)}{sup}\int p_{\theta }\left(x\right)log\frac{p_{{\theta }^{\prime }}\left(x\right)}{q\left(x\right)}dx\triangleq \Gamma \left({\mathrm{\Theta }}_r\right)$$

and the model q that achieves the minimum is the normalized maximum likelihood (NML) [29], $q_{nml}\left(x\right)={max}_{p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)}{p}_{{\theta }^{\prime }}\left(x\right)/Z$

This theorem is an immediate application of Shtarkov’s NML result [29]. It suggests that given a model class $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ which does not necessarily contain the true model p, the NML estimator $q_{nml}$ minimizes the worst-case regret over all possible distributions and guarantees an overhead of at most $\Gamma \left({\mathrm{\Theta }}_r\right)$ bits, compared to the best model in the class $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$, for any possible p. Further, it is shown that this result is tight, in the sense that there exist probability distributions $p\in \mathcal{P}\left(\mathrm{\Theta }\right)$ and $p_{{\theta }^{\prime }}\in \mathcal{P}\left({\mathrm{\Theta }}_r\right)$ that achieve the $\Gamma \left({\mathrm{\Theta }}_r\right)$ term. Notice that for every $\mathcal{P}\left(\mathrm{\Theta }\right)$ and $\mathcal{P}\left({\mathrm{\Theta }}_r\right)$ that satisfy the conditions above, we have that $C\left({\mathrm{\Theta }}_r\right)\le F(\mathrm{\Theta },{\mathrm{\Theta }}_r)\le \Gamma \left({\mathrm{\Theta }}_r\right).$ This means that under more restrictive assumptions we attain tighter worst-case performance guarantees, as expected. We now demonstrate our suggested methods in synthetic and real-world problems.

5. The Normal Distribution

Let us first study the classical unknown mean problem in the Gaussian case. Consider n i.i.d. samples, drawn from a d-dimensional multivariate normal distribution with an unknown mean $\mu$ and a known covariance matrix $\Sigma$. The $100(1-\alpha )\%$ confidence region for $\mu$ is ${\mathcal{M}}_r=\left\{\mu \right|{(\overline{x}-\mu )}^T{\Sigma }^{-1}(\overline{x}-\mu )\le \frac{1}{n}{\chi }_d^2(1-\alpha )\}$, where $\overline{x}$ is the sample mean, and ${\chi }_d^2$ is a Chi-squared distribution with d degrees of freedom. First, we would like to solve the minimax problem (2) with respect to ${\mathcal{M}}_r$. We apply the redundancy-capacity theorem (3) and define a corresponding channel, $X=M+Z$, where M is a random vector, taking values over the domain ${\mathcal{M}}_r$, while $Z\sim \mathcal{N}(0,\Sigma )$, independent of M (see [30] for detail). This formulation is also known as an amplitude-constrained capacity problem. We show (Appendix C) that it is equivalent to the generic case where $Z\sim \mathcal{N}(0,I_d)$ and $M\in {\mathcal{M}}_r^{\prime }$ where ${\mathcal{M}}_r^{\prime }=\left\{\mu \right|{\mu }^T\mu \le \frac{1}{n}{\chi }_d^2(1-\alpha )\}$. Notice that the domain of M is now restricted to a d-dimensional ball (defined by ${\mathcal{M}}_r^{\prime }$) and our goal is to find the capacity achieving distribution of M. It has been shown [31] that the solution to this problem is achieved when M is supported on a finite number of concentric spheres. Recently, the authors of [32] studied the necessary conditions under which the solution is a single sphere, centered at the origin. Specifically, they derived the largest radius $r_d$ for which the capacity achieving distribution is uniform on the sphere of the d-dimensional ball. This means that if the radius defined by ${\mathcal{M}}_r^{\prime }$ is smaller than $r_d$, then the solution to our minimax problem is immediate. Applying Dytso et al. analysis to our problem, we attain the following result.

Theorem 5.

Let $x^n$ be a collection of n i.i.d. samples from a d-dimensional multivariate normal distribution with an unknown mean μ and a known covariance matrix Σ. Let ${\mathcal{M}}_r$ be a $100(1-\alpha )\%$ confidence region for μ. Let $r_d$ be the largest radius for which the capacity achieving distribution is uniform on the sphere of a d-ball, as defined in Table 1 of [32]. Then, for any $n\ge {\chi }_d^2(1-\alpha )/r_d^2$, the solution to the minimax problem (2) over the confidence region ${\mathcal{M}}_r$ is attained by a uniform mixture of Gaussians with means on the confidence region, $q_{\pi }\propto {\int }_{\mu \in \mathcal{O}\left({\mathcal{M}}_r\right)}\mathcal{N}(\mu ,\Sigma )d\mu$ where $\mathcal{O}\left({\mathcal{M}}_r\right)=\left\{\mu \right|{(\overline{x}-\mu )}^T{\Sigma }^{-1}(\overline{x}-\mu )=\frac{1}{n}{\chi }_d^2(1-\alpha )\}$.

For example, let $\alpha =0.05$ and $d=2$. We have that $r_d=2.454$ (as appears in Table 1 of [32]), and the solution to (2) over ${\mathcal{M}}_r$ is given by $q_{\pi }\propto {\int }_{\mu \in \mathcal{O}\left(\mathcal{M}\right)}\mathcal{N}(\mu ,\Sigma )d\mu$, for every $n\ge 1$. The left chart of Figure 1 illustrates the shape of $q_{\pi }$ in this case. This Gaussian mixture shape may seem counterintuitive at a first glance, as $x^n$ are known to be drawn from a normal distribution. However, the reason is quite clear. Our inference criterion strives to control a set of Gaussian models. Therefore, the optimal solution is not necessarily the most likely model in the set, but a mixture of models.

Let us now turn to the projected mixture distribution and the NML. The right chart of Figure 1 demonstrates the shape of these estimators for $d=1$ and ${\mathcal{M}}_r=[-1.5,1.5]$.

First, we notice that the projected mixture is again a Gaussian mixture, with means outside of the confidence interval. On the other hand, the NML solution is not a Gaussian mixture; simple calculus shows that

$$\begin{array}{c}\hfill q_{nml}\left(x\right)\propto \left\{\begin{array}{cc}\frac{1}{\sqrt{2\pi }}exp(-\frac{1}{2}{(x+a)}^2)\hfill & x-a\hfill \\ \frac{1}{\sqrt{2\pi }}\hfill & -a\le x\le a\hfill \\ \frac{1}{\sqrt{2\pi }}exp(-\frac{1}{2}{(x-a)}^2)\hfill & xa\hfill \end{array}\right.\end{array}$$

(10)

for a symmetric confidence interval $[-a,a]$.

Let us illustrate the performance of our suggested methods. We draw n i.i.d. samples from a standard normal distribution $p_{\mu }\sim \mathcal{N}(0,1)$ where the mean $\mu =0$ is unknown and the variance is known. We apply our suggested methods (with $\alpha =0.05$) and evaluate the $\mathrm{KL}$ divergence from the true distribution, $D_{\mathrm{KL}}(p_{\mu }\left|\right|q(·|x^n))$. We compare our results with the performance of the MLE, $D_{\mathrm{KL}}\left(p\right||q_{\mathrm{mle}}\left(x^n\right))$, where $q_{\mathrm{mle}}\left(x^n\right)=\mathcal{N}(\overline{x},1)$. Notice that the MLE is also known to be the minimax solution to (1) in this setting. We repeat this experiment k = 10,000 times, for different sample sizes n. For each n we evaluate the mean ${\mathbb{E}}_{x^n\sim p_{\mu }^n}{D}_{\mathrm{KL}}(p_{\mu }\left|\right|q(·|x^n))$, the variance ${\mathrm{var}}_{x^n\sim p_{\mu }^n}{D}_{\mathrm{KL}}(p_{\mu }\left|\right|q(·|x^n))$ and the worst-case ${max}_{x^n\in {\mathcal{X}}_k}{D}_{\mathrm{KL}}(p_{\mu }\left|\right|q(·|x^n))$, where ${\mathcal{X}}_k$ is the set of k random draws of $x^n$ from $p_{\mu }^n$. Figure 2 demonstrates our results. Notice that we lose some accuracy, on the average, with all of our methods, compared to the MLE. On the other hand, the variance of the MLE is significantly greater, which suggests that it is less reliable for a given instance of data. Finally, we notice a significant gain in the worst-case performance. This behavior is not surprising: our approach strives to control the worst-case performance for each given draw. In this sense, we may view our approach as an “insurance policy”—we pay a small cost on the average, but attain a more stable estimator and gain significantly if “something bad happens” (that is, we observe $x^n$ that do not represent the true model well enough). Notice that this phenomenon is more evident when the inference problem is more challenging (smaller n’s). As we compare our suggested models to each other, we notice that the mixture distribution is the most conservative (that is, smallest cost and smallest gain), while the projected mixture is the least conservative. The reason is quite clear: in about $(1-\alpha )$ of the draws, the true parameter lies within the confidence region, and the mixture distribution is closer to it. This implies better performance on the average and worse performance in the extremes. Interestingly, the NML behaves as a compromise between the two. This is mostly as the NML does not assume that $\mu \in {\mathcal{M}}_r$ (better than the mixture in the worst-case). However, it also unnecessarily controls non-Gaussian models (worse than the projected mixture).

Let us now illustrate our suggested approach in a high-dimensional setting, $p=\mathcal{N}(\underline{1},I_d)$. Figure 3 compares the mixture estimator (which demonstrates a reasonable compromise between mean and worst-case performance) with the MLE and the James–Stein (JS) estimator. As we can see, the JS estimator slightly outperforms MLE on the average (as discussed in [3]), while the mixture distribution is very close to them. However, as we focus on the variance and the worst-case performance, the mixture distribution demonstrates a significant improvement, as expected. It is important to mention that in a zero mean case, the JS estimator achieves a significantly lower mean error (as discussed in [2]) and a remarkable increase in variance and worst-case performance. These results are omitted for brevity.

6. The Multinomial Distribution

We now turn to an additional important example of finite alphabet distributions. Let $x^n$ be n i.i.d. draws from a multinomial distribution over an alphabet size m. Notice that here, the parametric family spans the entire simplex. Therefore, we omit the parametric subscript $\theta$ to avoid an overload of notation, and regard p as the unknown vector of parameters. As discussed above, we would first like to construct a minimal-volume confidence region for p, denoted as ${\mathcal{P}}_r$. Unfortunately, there exists no closed-form solution in the multinomial case. Therefore, we turn to an approximate confidence region suggested in [33]. As many other approximation techniques [34,35], Sison and Glaz derive a rectangular region ${\mathcal{P}}_{sg}=\left\{p\right|p_l\left(i\right)\le p\left(i\right)\le p_u\left(i\right)\forall i=1,\cdots ,m\}$ which demonstrates a smaller expected volume compared to alternatives. Our first step is to define a subset of Sison and Glaz region, ${\mathcal{P}}_r\subset {\mathcal{P}}_{sg}$, which corresponds to valid probability distributions, ${\mathcal{P}}_r=\left\{p\right|p\in {\mathcal{P}}_{sg},\sum p\left(i\right)=1\}$. Notice that ${\mathcal{P}}_r$ is a convex set, and denote its set of vertexes as $\mathcal{V}\left({\mathcal{P}}_r\right)$. We show (Appendix D) that the solution to (2) over ${\mathcal{P}}_r$ is attained by solving (3) over $\mathcal{V}\left({\mathcal{P}}_r\right)$. This means that instead of considering the entire class ${\mathcal{P}}_r$, we only need to focus on the discrete set $\mathcal{V}\left({\mathcal{P}}_r\right)$.

Unfortunately, there exists no closed-form solution to (3) in this setting. However, as the cardinality of $\pi$ is finite (as we optimize over $\mathcal{V}\left({\mathcal{P}}_r\right)$), we may apply the Blahut–Arimoto algorithm [21] and attain a numerical solution, at a relatively small computational cost. Finally, we derive the projected mixture and the NML. As mentioned above, the parametric family spans the entire simplex. This means that the two methods are identical, and obtained by applying the NML over ${\mathcal{P}}_r$.

We now demonstrate our suggested approach. Let $x^n$ be i.i.d. draws from a Zipf’s law distribution over an alphabet size $m=5$ and a parameter $s=1.01$, $p\left(i\right)\propto i^{-s}$. The Zipf’s law distribution is a commonly used benchmark distribution, mostly in modeling of natural (real-world) quantities. It is widely used in physical and social sciences, linguistics, economics, and many other fields [36,37,38]. As in Section 5, we compare our suggested methods to the MLE. In addition, we consider the popular Laplace estimator, which adds a single count to all events, followed by a MLE. In our experiments we focus on an enhanced variant of Laplace [39], which adds $1/2$ to all events, $q_{\mathrm{lap}}\left(n_i\right)\propto n_i+1/2$, where $n_i$ is the number of appearances of the i^th symbol in $x^n$. This variant holds important universality properties and is widely known as the Krichevsky–Trofimov estimator [39,40].

We repeat each experiment k = 10,000 times and report the estimated mean, variance, and worst-case performance, as in the Gaussian case. Figure 4 demonstrates the results we achieve. We omit the MLE as it typically results in an unbounded divergence (in cases where at least a single symbol fails to appear).

As in the unknown normal mean problem, we notice that in more challenging setups (small n), our worst-case gain is quite remarkable. This gain narrows down as n increases, and all the estimators converge to the same solution. In addition, we observe a significant gain in expectation when n is small. It is important to emphasize that when the underlying distribution is easier to infer (all $p\left(x\right)$ are bounded away from zero, as with the uniform distribution), the advantage of using the minimax approach is less evident (similarly to the large n regime in the Zipf’s law example).

7. Large Alphabet Probability Estimation

In the large alphabet regime, we study a multinomial distribution where $mn$. This problem has been extensively studied over the years, with many applications ranging from language processing to biological studies [41]. Here, traditional methods like MLE are typically ineffective, as they assign a zero probability to unseen events. Several alternatives have been suggested over the years. In his seminal work, Laplace [42] suggested to add a single count to all events, followed by a maximum likelihood estimator. The work of Laplace was later generalized to a class of add-constant estimators [39], with the important special case of the Krichevsky–Trofimov estimator (as discussed in Section 6). A significant milestone in the history of large alphabet probability estimation was established in the work of Good and Turing [43]. Their approach suggests that unseen events shall be assigned a probability proportional to the number of events that appear once. To this day, Good–Turing estimators are the most commonly used methods in practical problems (see, for example, Section 1.4 in [41]). Despite the great interest in large alphabet estimation, provably-optimal schemes remain elusive [41]. Moreover, the accuracy of existing methods do not allow us to construct practical confidence regions. In fact, Paninski [44] showed that in the large alphabet setup, the minimal expected worst-case divergence is unbounded, and grows like $log(m/n)$. Therefore, it is quite difficult to define a small enough restrictive model class that contains p with high confidence. In this case, we introduce an alternative approach for the design of ${\mathcal{P}}_r$, followed by an NML estimator.

The Leave-One-Out Hypothesis Class

Define the convergence rate of $q(·|x^n)$ as ${\Delta }_p\left(n\right)=$$\mathbb{E}\left(D_{\mathrm{KL}}\left(p\right||q(·|x^n))-D_{\mathrm{KL}}\left(p\right||q(·|x^{n+1}))\right).$ We say that an estimator $q(·;x^n)$ is proper if it satisfies, for every p,

• $\mathbb{E}{D}_{\mathrm{KL}}\left(p\right||q(·|x^n))\infty$ for all $n\ge 0$
• ${\Delta }_p\left(n\right)\ge 0$ for all $n\ge n_0$
• ${\Delta }_p\left(n\right)$ is monotonically non-increasing for all $n\ge n_0$

The first condition states that the expected loss is finite for any n. The second condition indicates that asymptotically, adding more samples only improves the expected accuracy. The third condition says that the rate of the improvement is non-increasing in the number of samples. For example, the improvement from 100 to 101 samples is greater than the improvement from 1000 to 1001 samples, on the average. We now define the leave-one-out model class. Let $x_{[-i]}^{n-1}=\{x_1,...,x_{i-1},x_{i+1},...,x_n\}$ be the leave-one-out set of $x^n$, excluding the $i^{th}$ sample. Let $q(·|x_{[-i]}^{n-1})$ be the corresponding proper estimate. The leave-one-out (loo) model class is defined as ${\mathcal{P}}_{loo}={\left\{q(·|x_{[-i]}^{n-1})\right\}}_{i=1}^n$. Theorem 6 below establishes that on the average, the accuracy of the best model in ${\mathcal{P}}_{loo}$ is bounded from above by accuracy of $q(·|x^n)$, plus an additional vanishing overhead term.

Theorem 6.

Let q be a proper estimator. Then,

$$\begin{array}{c}\hfill \mathbb{E}\left(\underset{i}{min}{D}_{\mathrm{KL}}\left(p\left|\right|q\left(·|x_{[-i]}^{n-1}\right)\right)\right)\le \mathbb{E}\left(D_{\mathrm{KL}}\left(p\left|\right|q\left(·|x^n\right)\right)\right)+o\left(\frac{1}{n}\right).\end{array}$$

(11)

A proof for this Theorem is provided in Appendix E. Notice that the inequality is due to the convexity of the different operators. This means that typically, we expect the inequality to be strict. In other words, given a proper estimator q, Theorem 6 shows that on the average, there exists at least a single model in ${\mathcal{P}}_{loo}$ that is better than $q(·|x^n)$, up to a vanishing overhead term of $o\left(\frac{1}{n}\right)$. In Appendix F we show that any add-constant (Laplace) estimator satisfies (11). Further, our experiments indicate that the same property holds for the Good–Turing estimator. This motivates the use of these estimators in the design of ${\mathcal{P}}_r={\mathcal{P}}_{loo}$, as suggested by (4).

Let us now demonstrate our suggested scheme. In each experiment, we draw n samples from a multinomial distribution over an alphabet size $m=1000$. We apply the Krichevsky–Trofimov estimator, $q\left(n_i\right)\propto n_i+1/2$, and a Good–Turing estimator, following the implementation of Gale [45]. We compare these estimators to our suggested scheme; we construct a loo model class using Good–Turing, followed by an NML estimator. In addition, we compare the NML with a simple uniform average over the loo model class. A comprehensive description of our suggested scheme is provided in Appendix G. To emphasize the difference between the suggested schemes, we compare each estimator with a natural oracle$p_{nat}\left(x^n\right)$; an estimator who knows the true model p, but is restricted to assign the same probability to symbols that appear the same number of times in $x^n$. The performance of this oracle serves as a lower bound [41]. Figure 5 and Figure 6 demonstrate the results we achieve for a Zipf’s law distribution $p\left(i\right)\propto i^{-s}$ with a parameter value of $s=1.01$ (left) and $s=1.5$ (center). In addition, we consider a geometric distribution $p\left(i\right)={(1-s)}^{i-1}s$ with $s=0.05$ (right). We report the expected difference (regret) between $D_{\mathrm{KL}}\left(p\right||q(·|x^n))$ and $D_{\mathrm{KL}}\left(p\right||p_{nat}(·|x^n))$ in Figure 5, while the worst-case regret is presented in Figure 6. We omit the uncompetitive performance of the Krichevsky–Trofimov estimator.

As we observe Figure 5, we notice that our suggested NML method outperforms Good–Turing when the alphabet size is relatively small. As n increases, the improvement becomes less evident as the restrictive model class converges to $q(·|x^n)$. Further, we notice that a uniform average over the loo model class is also favorable, but demonstrates a slighter improvement. Finally, we compare the worst-case performance in Figure 6. Here, again, we notice a significant improvement as in the previous experiments. For example, for $n=40$ and a Zipf’s law distribution ($s=1.5$), the Good–Turing results in a regret of $0.72$ bits while the uniform mixture is $0.58$ bits and the NML is only $0.43$ bits.

8. Real-World Example

Let us now introduce a real-world example. The Wisconsin breast cancer study (https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic), accesed on 14 June 2021) considers 569 diagnosed tumors, of which 357 are benign (B) and 212 are malignant (M) [46]. Each tumor is characterized by 32 features, including its size, texture, surface, and more. We would like to study the radius of benign tumors and assess its probability function. This probability is of high interest as it allows us, for example, to control type-I error in a future hypothesis testing (the probability of deciding a tumor is malignant, given that it is benign).

The medical domain knowledge suggests that the size of the tumor follows a normal distribution, with different parameters for the B and M tumors. Therefore, the standard approach is to estimate the parameters from the given data. For simplicity, we assume the true variance is known (estimated from the entire population) and focus on the unknown mean.

As in the previous sections, we study the performance of different estimation schemes. We draw n samples from the B class, and apply the MLE and the suggested NML scheme. Notice that we focus on the NML as it is the most robust approach for the modeling assumption (and henceforth most suitable for such a clinical trail). We repeat this experiment 10,000 times for every value of n and report the mean, variance, and worst-case KL divergence between the “true empirical distribution” (based on all the B samples that we have) and each estimator that we examine. Figure 7 demonstrates the results we achieve. As we can see, our suggested approach attains a significantly better worst-case results.

It is important to emphasize that the MLE is the solution to the classical minimax estimation scheme (1), under the assumption that the data is generated from normal distribution (see Section 2). Our approach with the NML relaxes this strong restriction and attains a significant improvement in the worst-case performance.

9. Discussion and Conclusions

In this work, we study a minimax inference framework. Our suggested scheme considers a class of models, defined by the parametric confidence region of the given samples. Then, we control the worst-case performance within this class. Our formulation relaxes some strong modeling assumptions of the classical minimax framework and considers a robust inference scheme for the complete unknown distribution. The attained solution draws fundamental connections to basic concepts in information theory. We demonstrate the performance of our suggested framework in classical inference problems, including normal and multinomial distributions. In addition, we demonstrate our suggested scheme on more challenging large alphabet probability estimation problems. Finally, we study a real-world breast cancer example. In all of these settings we introduce a significant improvement in the worst-case, at a typically low cost on the average. This demonstrates an “insurance-like” trade-off; we pay a small cost on the average to avoid a great loss if “something bad happens” (that is, the observed samples do not represent the true model well enough).

It is important to emphasize that our suggested scheme is not limited to confidence region model classes. In fact, in many cases, exact confidence regions are difficult to attain, or result in model classes that are too large to control (for example, large alphabet problems with many unseen symbols). In these cases, we consider alternative forms of “reasonable” classes of models. One possible solution is the leave-one-out (LOO) class, discussed in Section 7. Additional alternatives are bootstrap confidence regions, Markov Chain Monte Carlo (MCMC) sampling and others.

Finally, our suggested scheme may be generalized to a supervised learning framework. For example, consider a linear regression problem. The standard approach is to estimate the regression coefficients that best explain the data (typically by least squares analysis). However, notice we may also construct confidence intervals for the sought coefficients. This way, we can define a restricted model class (similarly to Section 3), and seek minimax estimates with respect to it. This idea may be generalized to more complex learning schemes such as deep neural networks. Specifically, we may construct a restricted model class as the vicinity of some class of parameters that the network converges to, and control the corresponding worst-case performance. We consider this direction for our future work.