Update of Prior Probabilities by Minimal Divergence

Abstract

The present paper investigates the update of an empirical probability distribution with the results of a new set of observations. The update reproduces the new observations and interpolates using prior information. The optimal update is obtained by minimizing either the Hellinger distance or the quadratic Bregman divergence. The results obtained by the two methods differ. Updates with information about conditional probabilities are considered as well.

1. Introduction

The present work is inspired by the current practices in Information Geometry [1,2,3] where minimization of divergences is an important tool. In Statistical Physics a divergence is called a relative entropy. Its importance was noted rather late in the twentieth century, after the work of Jaynes on the maximal entropy principle [4]. Estimation in the presence of hidden variables by minimizing a divergence function is briefly discussed in Chapter 8 of [2].

Assume now that some observation or experiment yields new statistical data. The approach is then to look for a probability distribution that reproduces the newly observed probabilities and that interpolates the data with missing information coming from a prior.

No further model assumptions are imposed. Hence, the statistical model under consideration consists of all probability distributions that are consistent with the newly obtained empirical data. Internal consistency of the empirical data ensures that the model is not empty. The update is the model point that minimizes the chosen divergence function from the prior to the manifold of the model.

In the context of Maximum Likelihood Estimation (MLE) one usually adopts a parameterized model. The dimension of the model can be kept low and properties of the model can be used to ease the calculations. One assumes that the new data can lead to a more accurate estimation of the limited number of model parameters. It can then happen that the model is misspecified [5] and that the update is only a good approximation of the empirical data.

Here, the model is dictated by the newly acquired empirical data and the update is forced to reproduce the measured data. Finding the probability distribution is then an underdetermined problem. Minimization of the divergence from the prior probability distribution solves the underdetermination.

In Bayesian statistics, the update $q\left(B\right)$ of the probability $p\left(B\right)$ of an event B equals

$$\begin{array}{ccc}\hfill q\left(B\right)& =& p^{\mathrm{emp}}\left(A\right)p\left(B\right|A)+p^{\mathrm{emp}}\left(A^c\right)p\left(B\right|A^c).\hfill \end{array}$$

(1)

The quantities $p^{\mathrm{emp}}\left(A\right)$ and $p^{\mathrm{emp}}\left(A^c\right)$ are the empirical probabilities obtained after repeated measurement of event A and its complement $A^c$. Expression (1) has been called Jeffrey conditioning [6]. It implies the sufficiency conditions $q\left(B\right|A)=p(B\left|A\right)$ and $q\left(B\right|A^c)=p\left(B\right|A^c)$. It is an updating rule used in Radical Probabilism [7]. This expression is also obtained when minimizing the Hellinger distance between the prior and the model manifold. A proof of the latter follows later on in Section 4.

The present approach is a special case of minimizing a divergence function in the presence of linear constraints. See the introduction of [8] for an overview of early applications of this technique. Two classes of generalized distance functions satisfy a natural set of axioms: the f-divergences of Csiszár and the generalized Bregman divergences. The squared Hellinger distance belongs to the former class. The other divergence function considered here is the square Bregman divergence. Both Hellinger and square Bregman have special properties that make it easy to work with them.

A broad class of generalized Bregman divergences satisfies the Pythagorean equality [8,9]. Pythagorean inequalities hold for an even larger class [10]. The Pythagorean relations derived in the present work make use of the specific properties of the Hellinger distance and of the quadratic Bregman divergence. It is unclear how to prove them for more general divergences.

One incentive for starting the present work is a paper of Banerjee, Guo, and Wang [11,12]. They consider the problem of predicting a random variable $Z_1$ given observations of a random variable $Z_2$. It is well-known that the conditional expectation, as defined by Kolmogorov, is the optimal predictor. They show that this statement remains true when the metric distance is replaced by a Bregman divergence. It is shown in Theorem 2 below that a proof in a more general context yields a deviating result.

The next Section fixes notations. Section 3 collects some results about the squared Hellinger distance and the quadratic Bregman divergence. Section 4 discusses the optimal choice and contains the Theorems 1 and 2. The proof of the theorems can be adapted to cover the situation that a subsequent measurement also yields information on conditional probabilities. This is shown in Section 4.3. Section 5 treats a simple example. A final section summarizes the results of the paper.

2. Empirical Data

Consider a probability space $\mathsf{\Omega },\mu$. A measurable subset A of $\mathsf{\Omega }$ is called an event. Its probability is denoted $p\left(A\right)$ and is given by

$$\begin{array}{ccc}\hfill p\left(A\right)& =& {\int }_{\mathsf{\Omega }}{\mathbb{I}}_A\left(x\right)\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

where ${\mathbb{I}}_A\left(x\right)$ equals 1 when $x\in A$ and 0 otherwise. The conditional expectation of a random variable f given an event A with non-vanishing probability $p\left(A\right)$ is given by

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mu }f|A& =& \frac{1}{p\left(A\right)}{\mathbb{E}}_{\mu }f{\mathbb{I}}_A.\hfill \end{array}$$

The probability space $\mathsf{\Omega },\mu$ reflects the prior knowledge of the system at hand. When new data become available an update procedure is used to select the posterior probability space. The latter is denoted $\mathsf{\Omega },\nu$ in what follows. The corresponding probability of an event A is denoted $q\left(A\right)$.

The outcome of repeated experiments is the empirical probability distribution of the events, denoted $p^{\mathrm{emp}}\left(A\right)$. The question at hand is then to establish a criterion for finding the update $\nu$ of the probability distribution $\mu$ that is as close as possible to $\mu$ while reproducing the empirical results.

The event A defines a partition $A,A^c$ of the probability space $\mathsf{\Omega },\mu$. As before $A^c$ denotes the complement of A in $\mathsf{\Omega }$. In what follows a slightly more general situation is considered in which the event A is replaced by a partition ${\left(O_i\right)}_{i=1}^n$ of the measure space $\mathsf{\Omega },\mu$ into subsets with non-vanishing probability. The notations $p_i$ and ${\mu }_i$ are used, with

$$\begin{array}{c}\hfill p_i=p\left(O_i\right)\mathrm{and}\mathrm{d}{\mu }_i\left(x\right)=\frac{1}{p_i}{\mathbb{I}}_{O_i}\left(x\right)\mathrm{d}\mu \left(x\right).\end{array}$$

(2)

Introduce the random variable g defined by $g\left(x\right)=i$ when $x\in O_i$. Repeated measurement of the random variable g yields the empirical probabilities

$$\begin{array}{ccc}\hfill p_i^{\mathrm{emp}}& =& \mathrm{Emp} \mathrm{Prob}\{x:g(x)=i\}.\hfill \end{array}$$

They may deviate from the prior probabilities $p_i$. In some cases one also measures the conditional probabilities

$$\begin{array}{ccc}\hfill p^{\mathrm{emp}}\left(B\right|O_i)& =& \mathrm{Emp} \mathrm{Prob} \mathrm{of}B\mathrm{given} \mathrm{that}g\left(x\right)=i\hfill \end{array}$$

of some other event B.

3. A Geometric Approach

In this section two divergences are reviewed, the squared Hellinger distance and the quadratic Bregman divergence.

3.1. Squared Hellinger Distance

For simplicity the present section is restricted to the case that the sample space $\mathsf{\Omega }$ is the real line.

Given two probability measures $\mu$ and $\sigma$, both absolutely continuous w.r.t. the Lebesgue measure, the squared Hellinger distance is the divergence $D_2\left(\sigma \right|\left|\mu \right)$ defined by

$$\begin{array}{ccc}\hfill D_2\left(\sigma \right|\left|\mu \right)& =& \frac{1}{2}{\int }_{\mathbb{R}}{\left(\sqrt{\frac{\mathrm{d}\sigma }{\mathrm{d}x}}-\sqrt{\frac{\mathrm{d}\mu }{\mathrm{d}x}}\right)}^2\mathrm{d}x.\hfill \end{array}$$

It satisfies

$$\begin{array}{ccc}\hfill D_2\left(\sigma \right|\left|\mu \right)& =& 1-{\int }_{\mathbb{R}}\sqrt{\frac{\mathrm{d}\sigma }{\mathrm{d}x}\frac{\mathrm{d}\mu }{\mathrm{d}x}}\mathrm{d}x.\hfill \end{array}$$

Let ${\left(O_i\right)}_i$ be a partition of $\mathsf{\Omega },\mu$ and let $g\left(x\right)=i$ when x belongs to $O_i$, as before. Let $p_i$ and ${\mu }_i$ be defined by (2). Consider the following functions of i, with i in $\{1,\dots ,n\}$,

$$\begin{array}{ccc}\hfill {\tau }^{\left(1\right)}\left(i\right)& =& \mu ,\mathrm{independent} \mathrm{of}i,\hfill \\ \hfill {\tau }^{\left(2\right)}\left(i\right)& =& {\mu }_i,\hfill \\ \hfill {\tau }^{\left(3\right)}\left(i\right)& =& {\sigma }_i,\hfill \end{array}$$

where each of the ${\sigma }_i$ is a probability distribution with support in $O_i$. The empirical expectation of a function $f\left(i\right)$ is given by ${\mathbb{E}}^{\mathrm{emp}}f={\sum }_i{p}_i^{\mathrm{emp}}f\left(i\right)$.

Proposition 1.

If $p_i^{\mathrm{emp}}>0$ for all i and ${\sum }_i{p}_i^{\mathrm{emp}}=1$ then one has

$$\begin{array}{c}\hfill {\mathbb{E}}^{\mathrm{emp}}{D}_2({\tau }^{\left(1\right)}\left|\right|{\tau }^{\left(3\right)})\ge {\mathbb{E}}^{\mathrm{emp}}{D}_2({\tau }^{\left(1\right)}\left|\right|{\tau }^{\left(2\right)})\end{array}$$

with equality if and only if ${\sigma }_i={\mu }_i$ for all i.

First prove the following two lemmas.

Lemma 1.

Assume that the probability measure ${\nu }_i$ is absolutely continuous w.r.t. the measure ${\mu }_i$, with Radon-Nikodym derivative given by $\mathrm{d}{\nu }_i\left(x\right)=f_i\left(x\right)\mathrm{d}{\mu }_i$. Then one has

$$\begin{array}{ccc}\hfill D_2\left(\mu \right||{\sigma }_i)-D_2\left(\mu \right||{\nu }_i)& =& \sqrt{p_i}\left[D_2({\mu }_i\left|\right|{\sigma }_i)-D_2({\mu }_i\left|\right|{\nu }_i)\right]\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill D_2({\mu }_i\left|\right|{\nu }_i)& =& 1-{\int }_{O_i}\sqrt{f_i\left(x\right)}\mathrm{d}{\mu }_i\left(x\right).\hfill \end{array}$$

Proof. 

One calculates

$$\begin{array}{ccc}\hfill D_2\left(\mu \right||{\sigma }_i)-D_2\left(\mu \right||{\nu }_i)& =& {\int }_{\mathbb{R}}\sqrt{\frac{\mathrm{d}\mu }{\mathrm{d}x}}\left[\sqrt{\frac{\mathrm{d}{\nu }_i}{\mathrm{d}x}}-\sqrt{\frac{\mathrm{d}{\sigma }_i}{\mathrm{d}x}}\right]\mathrm{d}x\hfill \\ & =& \sqrt{p_i}{\int }_{O_i}\sqrt{\frac{\mathrm{d}{\mu }_i}{\mathrm{d}x}}\left[\sqrt{\frac{\mathrm{d}{\nu }_i}{\mathrm{d}x}}-\sqrt{\frac{\mathrm{d}{\sigma }_i}{\mathrm{d}x}}\right]\mathrm{d}x\hfill \\ & =& \sqrt{p_i}\left[{\int }_{O_i}\sqrt{f_i\left(x\right)}\mathrm{d}{\mu }_i\left(x\right)-{\int }_{O_i}{\left[\frac{\mathrm{d}{\mu }_i}{\mathrm{d}x}\frac{\mathrm{d}{\sigma }_i}{\mathrm{d}x}\right]}^{1/2}\mathrm{d}x\right]\hfill \\ & =& \sqrt{p_i}\left[{\int }_{O_i}\sqrt{f_i\left(x\right)}\mathrm{d}{\mu }_i\left(x\right)-1+D_2({\mu }_i\left|\right|{\sigma }_i)\right].\hfill \end{array}$$

Now take ${\sigma }_i={\nu }_i$ to obtain the desired results. □

Lemma 2.

(Pythagorean relation) For any i is

$$\begin{array}{ccc}\hfill D_2\left(\mu \right||{\sigma }_i)& =& D_2\left(\mu \right||{\mu }_i)+\sqrt{p_i}{D}_2({\mu }_i\left|\right|{\sigma }_i).\hfill \end{array}$$

Proof. 

The proof follows by taking ${\nu }_i={\mu }_i$ in the previous lemma. □

Proof. 

(Proposition 1)

From the previous lemma it follows that $D_2({\tau }^{\left(1\right)}\left|\right|{\tau }^{\left(3\right)})\ge D_2({\tau }^{\left(1\right)}\left|\right|{\tau }^{\left(2\right)})$. Note that ${\sigma }_i={\mu }_i$ implies that ${\tau }^{\left(3\right)}={\tau }^{\left(2\right)}$ and hence $D_2({\tau }^{\left(1\right)}\left|\right|{\tau }^{\left(3\right)})=D_2({\tau }^{\left(1\right)}\left|\right|{\tau }^{\left(2\right)})$. Conversely, if

$$\begin{array}{ccc}\hfill {\mathbb{E}}^{\mathrm{emp}}{D}_2({\tau }^{\left(1\right)}\left|\right|{\tau }^{\left(3\right)})& =& {\mathbb{E}}^{\mathrm{emp}}{D}_2({\tau }^{\left(1\right)}\left|\right|{\tau }^{\left(2\right)})\hfill \end{array}$$

then it follows from the previous lemma that ${\mathbb{E}}^{\mathrm{emp}}{D}_2({\tau }^{\left(2\right)}\left|\right|{\tau }^{\left(3\right)})=0$. If in addition $p_i^{\mathrm{emp}}>0$ for all i then it follows that for all i

$$\begin{array}{ccc}\hfill 0& =& D_2\left({\tau }^{\left(2\right)}\left(i\right)\left|\right|{\tau }^{\left(3\right)}\left(i\right)\right).\hfill \end{array}$$

Because the squared Hellinger distance is a divergence, this implies that ${\tau }^{\left(2\right)}\left(i\right)={\tau }^{\left(3\right)}\left(i\right)$, which is equivalent with ${\mu }_i={\sigma }_i$. □

3.2. Bregman Divergence

In the present section the squared Hellinger distance, which is an f-divergence, is replaced by a divergence of the Bregman type. In addition let $\mathsf{\Omega }$ be a finite set equipped with the counting measure $\rho$. It assigns to each subset A of $\mathsf{\Omega }$ the number of elements in A. This number is denoted $\left|A\right|$. The expectation value ${\mathbb{E}}_{\mu }f$ of a random variable f w.r.t. the probability measure $\mu$ is given by

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\mu }f& =& \sum _{k\in \mathsf{\Omega }}\mu \left(k\right)f\left(k\right).\hfill \end{array}$$

Given a partition of $\mathsf{\Omega }$ into sets $O_i$ one can define conditional probability measures with probability mass function ${\rho }_i$ given by

$$\begin{array}{ccc}\hfill {\rho }_i\left(k\right)& =& \frac{1}{|O_i|}\mathrm{if} k\in O_i,\hfill \\ & =& 0\mathrm{otherwise}.\hfill \end{array}$$

(3)

Similarly, conditional probability measures with probability mass function ${\mu }_i$ are given by

$$\begin{array}{ccc}\hfill {\mu }_i\left(k\right)& =& \frac{\mu \left(k\right)}{\mu \left(O_i\right)}\mathrm{if}k\in O_i,\hfill \\ & =& 0\mathrm{otherwise}.\hfill \end{array}$$

(4)

Fix a strictly convex function $\varphi :\mathbb{R}\mapsto \mathbb{R}$. The Bregman divergence of the probability measures $\sigma$ and $\mu$ is defined by

$$\begin{array}{ccc}\hfill D_{\varphi }\left(\sigma \right|\left|\mu \right)& =& F\left(\sigma \right)-F\left(\mu \right)-\langle \nabla F,\sigma -\mu \rangle \hfill \end{array}$$

with

$$\begin{array}{c}\hfill F\left(\sigma \right)=\sum _k\varphi \left(\sigma \left(k\right)\right)\mathrm{and}{\nabla }_{k}F\left(\sigma \right)={\varphi }^{\prime }\left(\sigma \left(k\right)\right).\end{array}$$

In the case that $\varphi \left(x\right)=x^2/2$, which is used below, it becomes

$$\begin{array}{ccc}\hfill D_{\varphi }\left(\sigma \right|\left|\mu \right)& =& \frac{1}{2}\sum _k{\left[\sigma \left(k\right)-\mu \left(k\right)\right]}^2.\hfill \end{array}$$

(5)

For convenience, this case is referred to as the quadratic Bregman divergence.

The following result, obtained with the quadratic Bregman divergence, is more elegant than the result of Lemma 2.

Proposition 2.

Consider the quadratic Bregman divergence $D_{\varphi }$ as given by (5). Let ${\nu }_i=p_i{\mu }_i+(1-p_i){\rho }_i$. Let ${\sigma }_i$ be any probability measure with support in $O_i$. Then the following Pythagorean relation holds.

$$\begin{array}{ccc}\hfill D_{\varphi }\left(\mu \right||{\sigma }_i)& =& D_{\varphi }\left(\mu \right||{\nu }_i)+D_{\varphi }({\nu }_i\left|\right|{\sigma }_i).\hfill \end{array}$$

Proof. 

One calculates

$$\begin{array}{ccc}\hfill D_{\varphi }(\mu ||{\sigma }_i)-D_{\varphi }\left(\mu \right||{\nu }_i)& =& D_{\varphi }({\nu }_i\left|\right|{\sigma }_i)+\sum _x\left[\mu \left(x\right)-{\nu }_i\left(x\right)\right]\left[{\varphi }^{\prime }\left({\nu }_i\left(x\right)\right)-{\varphi }^{\prime }\left({\sigma }_i\left(x\right)\right)\right]\hfill \\ & =& D_{\varphi }({\nu }_i\left|\right|{\sigma }_i)+\sum _{x\in O_i}\left[p_i{\mu }_i\left(x\right)-{\nu }_i\left(x\right)\right]\left[{\varphi }^{\prime }\left({\nu }_i\left(x\right)\right)-{\varphi }^{\prime }\left({\sigma }_i\left(x\right)\right)\right]\hfill \\ & =& D_{\varphi }({\nu }_i\left|\right|{\sigma }_i)-(1-p_i)\frac{1}{|O_i|}\sum _{x\in O_i}\left[{\varphi }^{\prime }\left({\nu }_i\left(x\right)\right)-{\varphi }^{\prime }\left({\sigma }_i\left(x\right)\right)\right].\hfill \end{array}$$

Use now that ${\varphi }^{\prime }\left(u\right)=u$ and the normalization of the probability measures ${\nu }_i$ and ${\sigma }_i$ to find the desired result. □

4. The Optimal Choice

4.1. Updated Probabilities

The following result proves that the standard Kolmogorovian definition of the conditional probability minimizes the Hellinger distance between the prior probability measure $\mu$ and the updated probability measure $\nu$. The optimal choice of the updated probability measure $\nu$ is given by corresponding probabilities $q\left(B\right)$. They satisfy

$$\begin{array}{ccc}\hfill q\left(B\right)& =& \sum _{i=1}^n{p}_i^{\mathrm{emp}}p\left(B\right|O_i)\mathrm{for} \mathrm{any} \mathrm{event}B.\hfill \end{array}$$

Theorem 1.

Let be given a partition ${\left(O_i\right)}_{i=1}^n$ of the probability space $\mathsf{\Omega },\mu$ with $\mathsf{\Omega }=\mathbb{R}$. Let ${\mu }_i$ be given by (2). Let $p_i=p\left(O_i\right)>0$ denote the probability of the event $O_i$ and let be given strictly positive empirical probabilities $p_i^{\mathrm{emp}}$, $i=1,\dots ,n$. The squared Hellinger distance $D_2\left(\sigma \right|\left|\mu \right)$ as a function of σ is minimal if and only if ${\sigma }_i={\mu }_i$ for all i. Here, σ is any probability measure on Ω satisfying

$$\begin{array}{c}\hfill \sigma =\sum _{i=1}^n{p}_i^{\mathrm{emp}}{\sigma }_i,\end{array}$$

and each of the ${\sigma }_i$ is a probability measure with support in $O_i$ and absolutely continuous w.r.t. ${\mu }_i$.

Note that the probability measure $\nu$ given by

$$\begin{array}{ccc}\hfill \nu \left(x\right)& =& \sum _{i=1}^n{p}_i^{\mathrm{emp}}{\mu }_i\left(x\right)\hfill \end{array}$$

uses the Kolmogorovian conditional probability as the predictor because the probabilities determined by the ${\mu }_i$ are obtained from the prior probability distribution $\mu$ by $p_i\left(x\right)=p\left(x\right|O_i)$. By the above theorem this predictor is the optimal one w.r.t. the squared Hellinger distance.

Proof. 

With the notations of the previous section is

$$\begin{array}{ccc}\hfill D_2\left(\sigma \right|\left|\mu \right)& =& {\mathbb{E}}^{\mathrm{emp}}{D}_2({\tau }^{\left(1\right)}\left|\right|{\tau }^{\left(3\right)}).\hfill \end{array}$$

Proposition 1 shows that it is minimal if and only if ${\sigma }_i={\mu }_i$ for all i. □

Next, consider the use of the quadratic Bregman divergence in the context of a finite probability space.

Theorem 2.

Let be given a partition ${\left(O_i\right)}_{i=1}^n$ of the finite probability space $\mathsf{\Omega },\mu$. Let ${\rho }_i$ be the counting measure on $O_i$ defined by (3). Let ${\mu }_i$ be given by (2). Let $p_i=p\left(O_i\right)>0$ denote the probability of the event $O_i$ and let be given strictly positive empirical probabilities $p_i^{\mathrm{emp}}$, $i=1,\dots ,n$ summing up to 1. Assume that

$$\begin{array}{c}\hfill p_i^{\mathrm{emp}}\ge p_i\left[1-|O_i|{\mu }_i\left(x\right)\right]for all x\in O_i and for i=1,\dots ,n.\end{array}$$

(6)

Then the following hold.

(a)

A probability distribution ν is defined by $\nu ={\sum }_i{p}_i^{\mathrm{emp}}{\nu }_i$ with

$$\begin{array}{ccc}\hfill {\nu }_i& =& \left(1-\frac{p_i}{p_i^{\mathrm{emp}}}\right){\rho }_i+\frac{p_i}{p_i^{\mathrm{emp}}}{\mu }_i.\hfill \end{array}$$

(7)

(b)

Let σ be any probability measure on Ω satisfying $\sigma ={\sum }_{i=1}^n{p}_i^{\mathrm{emp}}{\sigma }_i$, where each of the ${\sigma }_i$ is a probability distribution with support in $O_i$. Then the quadratic Bregman divergence satisfies the Pythagorean relation

$$\begin{array}{ccc}\hfill D_{\varphi }\left(\sigma \right|\left|\mu \right)& =& D_{\varphi }\left(\nu \right|\left|\mu \right)+\sum _{i=1}^n{\left(p_i^{\mathrm{emp}}\right)}^2{D}_{\varphi }({\sigma }_i\left|\right|{\nu }_i).\hfill \end{array}$$

(8)

(c)

The quadratic Bregman divergence $D_{\varphi }\left(\sigma \right|\left|\mu \right)$ is minimal if and only if $\sigma =\nu$.

Proof. 

• (a)

The assumption (6) guarantees that the ${\nu }_i\left(x\right)$ are probabilities.

• (b)

One calculates

$$\begin{array}{ccc}\hfill D_{\varphi }\left(\sigma \right|\left|\mu \right)-D_{\varphi }\left(\nu \right|\left|\mu \right)& =& \frac{1}{2}\sum _x\left[\sigma \left(x\right)-\nu \left(x\right)\right]\left[\sigma \left(x\right)+\nu \left(x\right)-2\mu \left(x\right)\right]\hfill \\ & =& \sum _{i=1}^n{p}_i^{\mathrm{emp}}\frac{1}{2}\sum _{x\in O_i}\left[{\sigma }_i\left(x\right)-{\nu }_i\left(x\right)\right]\hfill \\ & & \times \left[p_i^{\mathrm{emp}}{\sigma }_i\left(x\right)+p_i^{\mathrm{emp}}{\nu }_i\left(x\right)-2p_i{\mu }_i\left(x\right)\right]\hfill \\ & =& \sum _{i=1}^n{\left(p_i^{\mathrm{emp}}\right)}^2\frac{1}{2}\sum _{x\in O_i}{\left[{\sigma }_i\left(x\right)-{\nu }_i\left(x\right)\right]}^2\hfill \\ & & +\sum _{i=1}^n{p}_i^{\mathrm{emp}}\sum _{x\in O_i}\left[{\sigma }_i\left(x\right)-{\nu }_i\left(x\right)\right](p_i^{\mathrm{emp}}-p_i){\rho }_i\left(x\right)\hfill \\ & =& \sum _{i=1}^n{\left(p_i^{\mathrm{emp}}\right)}^2{D}_{\varphi }({\sigma }_i\left|\right|{\nu }_i).\hfill \end{array}$$

In the above calculation the third line is obtained by eliminating $p_i{\mu }_i$ using the definition of ${\nu }_i$. This gives

$$\begin{array}{ccc}& & p_i^{\mathrm{emp}}{\sigma }_i\left(x\right)+p_i^{\mathrm{emp}}{\nu }_i\left(x\right)-2p_i{\mu }_i\left(x\right)\hfill \\ & =& p_i^{\mathrm{emp}}{\sigma }_i\left(x\right)+p_i^{\mathrm{emp}}{\nu }_i\left(x\right)-2p_i^{\mathrm{emp}}\left[{\nu }_i\left(x\right)-\left(1-\frac{p_i}{p_i^{\mathrm{emp}}}\right){\rho }_i\left(x\right)\right]\hfill \\ & =& p_i^{\mathrm{emp}}\left[{\sigma }_i\left(x\right)-{\nu }_i\left(x\right)\right]+2(p_i^{\mathrm{emp}}-p_i){\rho }_i\left(x\right).\hfill \end{array}$$

The term

$$\begin{array}{c}\hfill \sum _{i=1}^n{p}_i^{\mathrm{emp}}\sum _{x\in O_i}\left[{\sigma }_i\left(x\right)-{\nu }_i\left(x\right)\right](p_i^{\mathrm{emp}}-p_i){\rho }_i\left(x\right)\end{array}$$

vanishes because ${\rho }_i\left(x\right)$ is constant on the set $O_i$ and the probability measures ${\nu }_i$ and ${\sigma }_i$ have support in $O_i$.

(c)

From (b) it follows that $D_{\varphi }\left(\sigma \right|\left|\mu \right)\ge D_{\varphi }\left(\nu \right|\left|\mu \right)$, with equality when $\sigma =\nu$.

Conversely, when $D_{\varphi }\left(\sigma \right|\left|\mu \right)=D_{\varphi }\left(\nu \right|\left|\mu \right)$ then (8) implies that

$$\begin{array}{ccc}\hfill \sum _{i=1}^n{\left(p_i^{\mathrm{emp}}\right)}^2{D}_{\varphi }({\sigma }_i\left|\right|{\nu }_i)& =& 0.\hfill \end{array}$$

The empirical probabilities are strictly positive by assumption. Hence, it follows that $D_{\varphi }\left(\mu \right||{\sigma }_i)=D_{\varphi }\left(\mu \right||{\nu }_i)$ for all i and hence, that ${\sigma }_i={\nu }_i$ for all i. The latter implies $\sigma =\nu$. □

The optimal update $\nu$ can be written as

$$\begin{array}{c}\hfill \nu =\sum _i\left[(p_i^{\mathrm{emp}}-p_i){\rho }_i+p_i{\mu }_i\right]=\mu +\sum _i(p_i^{\mathrm{emp}}-p_i){\rho }_i.\end{array}$$

This result is in general quite different from the update proposed by Theorem 1, which is

$$\begin{array}{ccc}\hfill \nu & =& \sum _i{p}_i^{\mathrm{emp}}{\mu }_i.\hfill \end{array}$$

The updates proposed by the two theorems coincide only in the special cases that either $p_i^{\mathrm{emp}}=p_i$ for all i or that ${\mu }_i={\rho }_i$ for all i. In the latter case the prior distribution $\mu ={\sum }_i{p}_i{\rho }_i$ is replaced by the update $\nu ={\sum }_i{p}_i^{\mathrm{emp}}{\rho }_i$.

The entropy of the update when event $O_i$ is observed, according to Theorem 1, equals $S\left({\nu }_i\right)=S\left({\mu }_i\right)$. According to Theorem 2 it equals

$$\begin{array}{ccc}\hfill S\left({\nu }_i\right)& =& S\left(\left[1-\frac{p_i}{p_i^{\mathrm{emp}}}\right]{\rho }_i+\frac{p_i}{p_i^{\mathrm{emp}}}{\mu }_i\right).\hfill \end{array}$$

If $p_i\le p_i^{\mathrm{emp}}$ then it follows that

$$\begin{array}{ccc}\hfill S\left({\nu }_i\right)& \ge & \left[1-\frac{p_i}{p_i^{\mathrm{emp}}}\right]S\left({\rho }_i\right)+\frac{p_i}{p_i^{\mathrm{emp}}}S\left({\mu }_i\right)\hfill \\ & \ge & S\left({\mu }_i\right).\hfill \end{array}$$

The former inequality follows because the entropy is a concave function. The latter follows because entropy is maximal for the uniform distribution ${\rho }_i$. On the other hand, if $p_i>p_i^{\mathrm{emp}}$ then one has

$$\begin{array}{ccc}\hfill S\left({\mu }_i\right)& =& S\left(\left[1-\frac{p_i^{\mathrm{emp}}}{p_i}\right]{\rho }_i+\frac{p_i^{\mathrm{emp}}}{p_i}{\nu }_i\right)\hfill \\ & \ge & \left[1-\frac{p_i^{\mathrm{emp}}}{p_i}\right]S\left({\rho }_i\right)+\frac{p_i^{\mathrm{emp}}}{p_i}S\left({\nu }_i\right)\hfill \\ & \ge & S\left({\nu }_i\right).\hfill \end{array}$$

In the latter case the decrease of the entropy is stronger than in the case of the update based on the squared Hellinger distance. In conclusion, the update relying on the quadratic Bregman divergence looses details of the prior distribution by making a convex combination with a uniform distribution weighed with the probabilities of the observation. It does this moreso for the events with observed probability larger than predicted; this is when $p_i^{\mathrm{emp}}>p_i$.

Note that Theorem 2 cannot always be applied because it contains restrictions on the empirical probabilities. In particular, if the prior probability $\mu \left(x\right)$ of some point x in $\mathsf{\Omega }$ vanishes then the condition (6) requires that the empirical probability $p_i^{\mathrm{emp}}$ of the partition $O_i$ to which the point x belongs is larger than or equal to the prior probability $p_i$.

4.2. Update of Conditional Probabilities

The two previous theorems assume that no empirical information is available about conditional probabilities. If such information is present then an optimal choice should make use of it. In one case the solution of the problem is straightforward. If the probabilities $p_i^{\mathrm{emp}}$ are available together with all conditional probabilities $p^{\mathrm{emp}}\left(B\right|O_i)$ and there exists an update $\nu$ which reproduces these results then it is unique. Two cases remain: (1) The information about the conditional probabilities is incomplete; (2) the information is internally inconsistent – no update exists which reproduces the data.

Let us tackle the problem by considering the case that the only information that is available besides the probabilities $p_i^{\mathrm{emp}}$ is the vector of conditional probabilities $p^{\mathrm{emp}}\left(B\right|O_i)$ of a fixed event B, given the outcome of the measurement of the random variable g as introduced in Section 2.

The following result is independent of the choice of divergence function.

Proposition 3.

Fix an event B in Ω. Assume that the conditional probabilities $p\left(B\right|O_i)$, $i=1,\dots ,n$, are strictly positive and strictly less than 1. Assume in addition that $p_i^{\mathrm{emp}}{p}^{\mathrm{emp}}\left(B\right|O_i)\le 1$ for all i. Then there exists an update ν with corresponding probabilities $q(·)$ such that $q\left(O_i\right)=p_i^{\mathrm{emp}}$ and $q\left(B\right|O_i)=p^{\mathrm{emp}}\left(B\right|O_i)$, $i=1,\dots ,n$.

Proof. 

An obvious choice is to take $\nu$ of the form $\nu ={\sum }_i{p}_i^{\mathrm{emp}}{\nu }_i$ with ${\nu }_i$ of the form

$$\begin{array}{c}\hfill \mathrm{d}{\nu }_i\left(x\right)=\left[a_i{\mathbb{I}}_{B\cap O_i}\left(x\right)+b_i{\mathbb{I}}_{B^c\cap O_i}\left(x\right)\right]\mathrm{d}\mu \left(x\right),\end{array}$$

with $a_i\ge 0$ and $b_i\ge 0$. Normalization of the ${\nu }_i$ gives the conditions

$$\begin{array}{ccc}\hfill 1& =& a_{i}p(B\cap O_i)+b_{i}p(B^c\cap O_i).\hfill \end{array}$$

(9)

Reproduction of the conditional probabilities gives the conditions

$$\begin{array}{c}\hfill p^{\mathrm{emp}}\left(B\right|O_i)=\frac{q(B\cap O_i)}{q\left(O_i\right)}=a_i\frac{p(B\cap O_i)}{p_i^{\mathrm{emp}}}.\end{array}$$

The latter gives

$$\begin{array}{c}\hfill a_i=\frac{p_i^{\mathrm{emp}}}{p_i}\frac{p^{\mathrm{emp}}\left(B\right|O_i)}{p\left(B\right|O_i)}.\end{array}$$

The normalization condition (9) becomes

$$\begin{array}{c}\hfill 1=p_i^{\mathrm{emp}}{p}^{\mathrm{emp}}\left(B\right|O_i)+b_{i}p(B^c\cap O_i).\end{array}$$

It has a positive solution for $b_i$ because $p_i^{\mathrm{emp}}{p}^{\mathrm{emp}}\left(B\right|O_i)\le 1$ and $p(B^c\cap O_i)>0$. □

4.3. The Hellinger Case

The optimal updates can be derived easily from Theorem 1. Double the partition by introduction of the following sets

$$\begin{array}{c}\hfill O_i^{+}=B\cap O_{i}and{O}_i^{-}=B^c\cap O_i.\end{array}$$

They have prior probabilities $p_i^{\pm }=p\left(O_i^{\pm }\right)$. Corresponding prior measures ${\mu }_i^{\pm }$ are defined by

$$\begin{array}{c}\hfill \mathrm{d}{\mu }_i^{\pm }\left(x\right)=\frac{1}{p_i^{\pm }}{\mathbb{I}}_{O_i^{\pm }}\left(x\right)\mathrm{d}\mu \left(x\right).\end{array}$$

The empirical probability of the set $O_i^{+}$ is taken equal to $p_i^{\mathrm{emp}}{p}^{\mathrm{emp}}\left(B\right|O_i)$, that of $O_i^{-}$ equals $p_i^{\mathrm{emp}}[1-p^{\mathrm{emp}}\left(B\right|O_i)]$. The optimal update $\nu$ follows from Theorem 1 and is given by

$$\begin{array}{ccc}\hfill \mathrm{d}\nu \left(x\right)& =& \sum _i{p}_i^{\mathrm{emp}}{p}^{\mathrm{emp}}\left(B\right|O_i)\mathrm{d}{\mu }_i^{+}\left(x\right)+\sum _i{p}_i^{\mathrm{emp}}[1-p^{\mathrm{emp}}\left(B\right|O_i)]\mathrm{d}{\mu }_i^{-}\left(x\right).\hfill \end{array}$$

By construction it is

$$\begin{array}{c}\hfill q\left(O_i^{+}\right)=p_i^{\mathrm{emp}}{p}^{\mathrm{emp}}\left(B\right|O_i)andq\left(O_i^{-}\right)=p_i^{\mathrm{emp}}[1-p^{\mathrm{emp}}\left(B\right|O_i)].\end{array}$$

One now verifies that $q\left(O_i\right)=p_i^{\mathrm{emp}}$ and $q\left(B\right|O_i)=p^{\mathrm{emp}}\left(B\right|O_i)$, which is the intended result.

4.4. The Bregman Case

Next consider the optimization with the quadratic Bregman divergence. Probability distributions ${\rho }_i^{\pm }$ are defined by

$$\begin{array}{ccc}\hfill {\rho }_i^{\pm }\left(x\right)& =& \frac{1}{|O_i^{\pm }|}{\mathbb{I}}_{O_i^{\pm }}\left(x\right).\hfill \end{array}$$

Introduce the notations

$$\begin{array}{ccc}\hfill r_i^{+}& =& \frac{p_i^{+}}{p_i^{\mathrm{emp}}{p}^{\mathrm{emp}}\left(B\right|O_i)},\hfill \\ \hfill r_i^{-}& =& \frac{p_i^{-}}{p_i^{\mathrm{emp}}[1-p^{\mathrm{emp}}\left(B\right|O_i)]},\hfill \\ & & \\ \hfill {\nu }_i^{\pm }\left(x\right)& =& (1-r_i^{\pm }){\rho }_i^{\pm }+r_i^{\pm }{\mu }_i^{\pm }\left(x\right).\hfill \end{array}$$

Then the condition for Theorem 2 to hold is that ${\nu }_i^{\pm }\left(x\right)\ge 0$ for all $x,i$. The optimal probability distribution $\nu$ is given by

$$\begin{array}{ccc}\hfill \nu \left(x\right)& =& \sum _i{p}_i^{\mathrm{emp}}{p}^{\mathrm{emp}}\left(B\right|O_i){\nu }_i^{+}\left(x\right)+\sum _i{p}_i^{\mathrm{emp}}[1-p^{\mathrm{emp}}\left(B\right|O_i)]{\nu }_i^{-}\left(x\right)\hfill \\ & =& \sum _i\left[p_i^{\mathrm{emp}}{p}^{\mathrm{emp}}\left(B\right|O_i)-p_i^{+}\right]{\rho }_i^{+}+\sum _i{p}_i^{+}{\mu }_i^{+}\hfill \\ & & +\sum _i\left[p_i^{\mathrm{emp}}[1-p^{\mathrm{emp}}\left(B\right|O_i)]-p_i^{-}\right]{\rho }_i^{-}+\sum _i{p}_i^{-}{\mu }_i^{-}\hfill \\ & =& \sum _i{p}_i^{\mathrm{emp}}{p}^{\mathrm{emp}}\left(B\right|O_i)\left[{\rho }_i^{+}-{\rho }_i^{-}\right]\hfill \\ & & -\sum _i{p}_i^{+}{\rho }_i^{+}+\sum _i[p_i^{\mathrm{emp}}-p_i^{-}]{\rho }_i^{-}\hfill \\ & & +\mu .\hfill \end{array}$$

5. Example

Assume that the prior probability distribution is binomial with parameters $n,\lambda$, where n is known with certainty. The probability mass function is given by

$$\begin{array}{ccc}\hfill \mu \left(k\right)=\mathrm{Prob}(X=k)& =& \left(\genfrac{}{}{0pt}{}{n}{k}\right){\lambda }^k{(1-\lambda )}^{n-k}k=0,1,2,\dots ,n.\hfill \end{array}$$

The probability distribution and the value of the parameter $\lambda$ are for instance the result of theoretical modeling of the experiment. Or they are obtained from a different kind of experiment.

The experiment under consideration yields accurate values for the probability $p^{\mathrm{emp}}$ of the two events $X=1$ and $X=2$. The problem at hand is to predict by extrapolation the probability of the event $X=k$ for other values of k. A fit of the data with a binomial distribution is likely to fail because two accurate data points are given to determine a single parameter $\lambda$. The binomial model can be misspecified.

The geometric approach followed in the present paper yields an update from the binomial distribution to another distribution, one which is reproducing the data. The update is conducted in an unbiased manner. Quite often one is tempted to replace the model, in the case of the binomial model, by a model with one extra free parameter.

Let us see what are the results of minimizing divergence functions. The probability space $\mathsf{\Omega }$ is the set of integers $0,1,2,\dots ,n$ equipped with the uniform measure. Choose events

$$\begin{array}{c}\hfill O_1=\left\{1\right\},O_2=\left\{2\right\},O_3=\mathsf{\Omega }\setminus (O_1\cup O_2).\end{array}$$

This gives for $p_i:=\mathrm{Prob}(X\in O_i)$

$$\begin{array}{c}\hfill p_1=\mu \left(1\right)=n\lambda {(1-\lambda )}^{n-1},p_2=\mu \left(2\right)=\frac{1}{2}n(n-1){\lambda }^2{(1-\lambda )}^{n-2},p_3=1-p_2-p_3.\end{array}$$

The optimal update according to Theorem 1, minimizing the Hellinger distance, is given by the probabilities

$$\begin{array}{ccc}\hfill \nu \left(B\right)& =& \sum _i{p}_i^{\mathrm{emp}}\mu \left(B\right|O_i).\hfill \end{array}$$

In particular, the probability mass function $\nu \left(k\right):=\nu \left(\right\{k\left\}\right)$ becomes

$$\begin{array}{ccc}\hfill \nu \left(1\right)& =& p_1^{\mathrm{emp}},\hfill \\ \hfill \nu \left(2\right)& =& p_2^{\mathrm{emp}},\hfill \\ \hfill \nu \left(k\right)& =& \frac{p_3^{\mathrm{emp}}}{p_3}\mu \left(k\right)otherwise.\hfill \end{array}$$

The optimal update according to Theorem 2, minimizing the quadratic Bregman divergence, is given by (7). The auxiliary measures ${\mu }_i$, ${\rho }_i$, and ${\nu }_i$ have probability mass functions given by

$$\begin{array}{c}\hfill {\mu }_i\left(k\right)={\rho }_i\left(k\right)={\nu }_i={\delta }_{k,i}\mathrm{for}i=1,2,\end{array}$$

and

$$\begin{array}{ccc}\hfill {\mu }_3\left(k\right)& =& (1-{\delta }_{k,1})(1-{\delta }_{k,2})\frac{\mu \left(k\right)}{p_3},\hfill \\ \hfill {\rho }_3\left(k\right)& =& (1-{\delta }_{k,1})(1-{\delta }_{k,2})\frac{1}{n-2}\hfill \\ \hfill {\nu }_3\left(k\right)& =& (1-{\delta }_{k,1})(1-{\delta }_{k,2})\left[(1-\frac{p_3}{p_3^{\mathrm{emp}}})\frac{1}{n-2}+\frac{\mu \left(k\right)}{p_3^{\mathrm{emp}}}\right].\hfill \end{array}$$

The probability mass function $\nu \left(k\right):=\nu \left(\right\{k\left\}\right)$ becomes

$$\begin{array}{ccc}\hfill \nu \left(k\right)& =& p_1^{\mathrm{emp}}{\nu }_1\left(k\right)+p_2^{\mathrm{emp}}{\nu }_2\left(k\right)+p_3^{\mathrm{emp}}{\nu }_3\left(k\right)\hfill \\ & =& p_1^{\mathrm{emp}}\mathrm{if}k=1,\hfill \\ & =& p_2^{\mathrm{emp}}\mathrm{if}k=2,\hfill \\ & =& \frac{p_3^{\mathrm{emp}}-p_3}{n-2}+\mu \left(k\right)\mathrm{otherwise}.\hfill \end{array}$$

The condition (6) is the requirement that all $\nu \left(k\right)$ are non-negative. Because the probabilities $\mu \left(k\right)$ can become very small this essentially means that $p_3^{\mathrm{emp}}$ should be larger than $p_3$. The amount of probability missing in the empirical probabilities $p_1^{\mathrm{emp}}$ and $p_2^{\mathrm{emp}}$ is equally distributed over the remaining $n-1$ points of $\mathsf{\Omega }$. On the other hand, when minimizing the Hellinger distance the excess or shortage of probability is compensated by multiplying all remaining probabilities by a constant factor.

A numerical comparison with $n=20$ and $\lambda =1/8$ is found in Figure 1. The empirical values are $p_1^{\mathrm{emp}}=0.15$ and $p_2^{\mathrm{emp}}=0.25$. The difference with the prior values $p_1\simeq 0.19774$ and $p_2\simeq 0.26836$ is made large enough to amplify the effects of the update.

6. Summary

It is well known that the use of unmodified prior conditional probabilities is the optimal way for updating a probability distribution after new data become available. The update procedure minimizes the Hellinger distance between prior and posterior probability distributions. For the sake of completeness a proof is given in Theorem 1.

Alternatively, one can minimize the quadratic Bregman divergence instead of the Hellinger distance. The result is given in Theorem 2. The conservation of probability is handled in a different way in the two cases, either by multiplying prior probabilities with a suitable factor or by adding an appropriate term.

The example of Section 5 shows that the two update procedures have different effects and that neither of them may be satisfactory. This raises the question whether the present approach should be improved by choosing divergences other than Hellinger or Bregman.

In the present research, the work of Banerjee, Guo, and Wang [11] was considered as well. They prove that minimization of the Hellinger distance can be replaced by minimization of a Bregman divergence, without modifying the outcome. It is shown in Theorem 2 that, in a different context, the use of the Bregman divergence yields results quite distinct from those obtained by minimizing the Hellinger distance.