Bayesian Update with Importance Sampling: Required Sample Size

Abstract

Importance sampling is used to approximate Bayes’ rule in many computational approaches to Bayesian inverse problems, data assimilation and machine learning. This paper reviews and further investigates the required sample size for importance sampling in terms of the ${\chi }^2$-divergence between target and proposal. We illustrate through examples the roles that dimension, noise-level and other model parameters play in approximating the Bayesian update with importance sampling. Our examples also facilitate a new direct comparison of standard and optimal proposals for particle filtering.

1. Introduction

Importance sampling is a mechanism to approximate expectations with respect to a target distribution using independent weighted samples from a proposal distribution. The variance of the weights—quantified by the ${\chi }^2$-divergence between target and proposal— gives both necessary and sufficient conditions on the sample size to achieve a desired worst-case error over large classes of test functions. This paper contributes to the understanding of importance sampling to approximate the Bayesian update, where the target is a posterior distribution obtained by conditioning the proposal to observed data. We consider illustrative examples where the ${\chi }^2$-divergence between target and proposal admits a closed formula and it is hence possible to characterize explicitly the required sample size. These examples showcase the fundamental challenges that importance sampling encounters in high dimension and small noise regimes where target and proposal are far apart. They also facilitate a direct comparison of standard and optimal proposals for particle filtering.

We denote the target distribution by $\mu$ and the proposal by $\pi$ and assume that both are probability distributions in Euclidean space ${\mathbb{R}}^d$. We further suppose that the target is absolutely continuous with respect to the proposal and denote by g the un-normalized density between target and proposal so that, for any suitable test function $\phi ,$

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^d}\phi \left(u\right)\mu \left(du\right)=\frac{{\int }_{{\mathbb{R}}^d}\phi \left(u\right)g\left(u\right)\pi \left(du\right)}{{\int }_{{\mathbb{R}}^d}g\left(u\right)\pi \left(du\right)}.\end{array}$$

(1)

We write this succinctly as $\mu \left(\phi \right)=\pi \left(\phi g\right)/\pi \left(g\right).$ For simplicity of exposition, we will assume that g is positive $\pi$-almost surely. Importance sampling approximates $\mu \left(\phi \right)$ using independent samples ${\left\{u^{\left(n\right)}\right\}}_{n=1}^N$ from the proposal $\pi ,$ computing the numerator and denominator in (1) by Monte Carlo integration,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mu \left(\phi \right)& \approx \frac{\frac{1}{N}{\sum }_{n=1}^N\phi \left(u^{\left(n\right)}\right)g\left(u^{\left(n\right)}\right)}{\frac{1}{N}{\sum }_{n=1}^{N}g\left(u^{\left(n\right)}\right)}\hfill \\ \hfill & =\sum _{n=1}^N{w}^{\left(n\right)}\phi \left(u^{\left(n\right)}\right),w^{\left(n\right)}:=\frac{g\left(u^{\left(n\right)}\right)}{{\sum }_{\ell =1}^{N}g\left(u^{\left(\ell \right)}\right)}.\hfill \end{array}\end{array}$$

(2)

The weights $w^{\left(n\right)}$—called autonormalized or self-normalized since they add up to one—can be computed as long as the un-normalized density g can be evaluated point-wise; knowledge of the normalizing constant $\pi \left(g\right)$ is not needed. We write (2) briefly as $\mu \left(\phi \right)\approx {\mu }^N\left(\phi \right)$, where ${\mu }^N$ is the random autonormalized particle approximation measure

$${\mu }^N:=\sum _{n=1}^N{w}^{\left(n\right)}{\delta }_{u^{\left(n\right)}},u^{\left(n\right)}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\pi .$$

(3)

This paper is concerned with the study of importance sampling in Bayesian formulations to inverse problems, data assimilation and machine learning tasks [1,2,3,4,5], where the relationship $\mu \left(du\right)\propto g\left(u\right)\pi \left(du\right)$ arises from application of Bayes’ rule $\mathbb{P}\left(u\right|y)\propto \mathbb{P}(y\left|u\right)\mathbb{P}\left(u\right)$; we interpret $u\in {\mathbb{R}}^d$ as a parameter of interest, $\pi \equiv \mathbb{P}\left(u\right)$ as a prior distribution on u, $g\left(u\right)\equiv g(u;y)\equiv \mathbb{P}\left(y\right|u)$ as a likelihood function which tacitly depends on observed data $y\in {\mathbb{R}}^k$, and $\mu \equiv \mathbb{P}\left(u\right|y)$ as the posterior distribution of u given $y.$ With this interpretation and terminology, the goal of importance sampling is to approximate posterior expectations using prior samples. Since the prior has fatter tails than the posterior, the Bayesian setting poses further structure into the analysis of importance sampling. In addition, there are several specific features of the application of importance sampling in Bayesian inverse problems, data assimilation and machine learning that shape our presentation and results.

First, Bayesian formulations have the potential to provide uncertainty quantification by computing several posterior quantiles. This motivates considering a worst-case error analysis [6] of importance sampling over large classes of test functions $\phi$ or, equivalently, bounding a certain distance between the random particle approximation measure ${\mu }^N$ and the target $\mu ,$ see [1]. As we will review in Section 2, a key quantity in controlling the error of importance sampling with bounded test functions is the ${\chi }^2$-divergence between target and proposal, given by

$$d_{{\chi }^2}(\mu \parallel \pi )=\frac{\pi \left(g^2\right)}{\pi {\left(g\right)}^2}-1.$$

Second, importance sampling in inverse problems, data assimilation and machine learning applications is often used as a building block of more sophisticated computational methods, and in such a case there may be little or no freedom in the choice of proposal. For this reason, throughout this paper we view both target and proposal as given and we focus on investigating the required sample size for accurate importance sampling with bounded test functions, following a similar perspective as [1,7,8]. The complementary question of how to choose the proposal to achieve a small variance for a given test function is not considered here. This latter question is of central interest in the simulation of rare events [9] and has been widely studied since the introduction of importance sampling in [10,11], leading to a plethora of adaptive importance sampling schemes [12].

Third, high dimensional and small noise settings are standard in inverse problems, data assimilation and machine learning, and it is essential to understand the scalability of sampling algorithms in these challenging regimes. The curse of dimension of importance sampling has been extensively investigated [1,13,14,15,16,17]. The early works [13,14] demonstrated a weight collapse phenomenon, by which unless the number of samples is scaled exponentially with the dimension of the parameter, the maximum weight converges to one. The paper [1] also considered small noise limits and further emphasized the need to define precisely the dimension of learning problems. Indeed, while many inverse problems, data assimilation models and machine learning tasks are defined in terms of millions of parameters, their intrinsic dimension can be substantially smaller since $\left(i\right)$ all parameters may not be equally important; $\left(ii\right)$ a priori information about some parameters may be available; and $\left(iii\right)$ the data may be lower dimensional than the parameter space. If the intrinsic dimension is still large, which occurs often in applications in geophysics and machine learning, it is essential to leverage the correlation structure of the parameters or the observations by performing localization [18,19,20]. Local particle filters are reviewed in [21] and their potential to beat the curse of dimension is investigated from a theoretical viewpoint in [16]. Localization is popular in ensemble Kalman filters [20] and has been employed in Markov chain Monte Carlo [22,23]. Our focus in this paper is not on localization but rather on providing a unified and accessible understanding of the roles that dimension, noise-level and other model parameters play in approximating the Bayesian update. We will do so through examples where it is possible to compute explicitly the ${\chi }^2$-divergence between target and proposal, and hence the required sample size.

Finally, in the Bayesian context the normalizing constant $\pi \left(g\right)$ represents the marginal likelihood and is often computationally intractable. This motivates our focus on the autonormalized importance sampling estimator in (2), which estimates both$\pi \left(g\phi \right)$ and $\pi \left(g\right)$ using Monte Carlo integration, as opposed to un-normalized variants of importance sampling [8].

Main Goals, Specific Contributions and Outline

The main goal of this paper is to provide a rich and unified understanding of the use of importance sampling to approximate the Bayesian update, while keeping the presentation accessible to a large audience. In Section 2 we investigate the required sample size for importance sampling in terms of the ${\chi }^2$-divergence between target and proposal. Section 3 builds on the results in Section 2 to illustrate through numerous examples the fundamental challenges that importance sampling encounters when approximating the Bayesian update in small noise and high dimensional settings. In Section 4 we show how our concrete examples facilitate a new direct comparison of standard and optimal proposals for particle filtering. These examples also allow us to identify model problems where the advantage of the optimal proposal over the standard one can be dramatic.

Next, we provide further details on the specific contributions of each section and link them to the literature. We refer to [1] for a more exhaustive literature review.

• Section 2 provides a unified perspective on the sufficiency and necessity of having a sample size of the order of the ${\chi }^2$-divergence between target and proposal to guarantee accurate importance sampling with bounded test functions. Our analysis and presentation are informed by the specific features that shape the use of importance sampling to approximate Bayes’ rule. The key role of the second moment of the ${\chi }^2$-divergence has long been acknowledged [24,25], and it is intimately related to an effective sample size used by practitioners to monitor the performance of importance sampling [26,27]. A topic of recent interest is the development of adaptive importance sampling schemes where the proposal is chosen by minimizing—over some admissible family of distributions—the ${\chi }^2$-divergence with respect to the target [28,29]. The main original contributions of Section 2 are Proposition 2 and Theorem 1, which demonstrate the necessity of suitably increasing the sample size with the ${\chi }^2$-divergence along singular limit regimes. The idea of Proposition 2 is inspired by [7], but adapted here from relative entropy to ${\chi }^2$-divergence. Our results complement sufficient conditions on the sample size derived in [1] and necessary conditions for un-normalized (as opposed to autonormalized) importance sampling in [8].
• In Section 3, Proposition 4 gives a closed formula for the ${\chi }^2$-divergence between posterior and prior in a linear-Gaussian Bayesian inverse problem setting. This formula allows us to investigate the scaling of the ${\chi }^2$-divergence (and thereby the rate at which the sample size needs to grow) in several singular limit regimes, including small observation noise, large prior covariance and large dimension. Numerical examples motivate and complement the theoretical results. Large dimension and small noise singular limits were studied in [1] in a diagonal setting. The results here are generalized to a nondiagonal setting, and the presentation is simplified by using the closed formula in Proposition 4. Moreover, we include singular limits arising from large prior covariance. In an infinite dimensional setting, Corollary 1 establishes an equivalence between absolute continuity, finite ${\chi }^2$-divergence and finite intrinsic dimension. A similar result was proved in more generality in [1] using the advanced theory of Gaussian measures in Hilbert space [30]; our presentation and proof here are elementary, while still giving the same degree of understanding.
• In Section 4 we follow [1,13,14,15,31] and investigate the use of importance sampling to approximate Bayes’ rule within one filtering step in a linear-Gaussian setting. We build on the examples and results in Section 3 to identify model regimes where the performance of standard and optimal proposals can be dramatically different. We refer to [2,32] for an introduction to standard and optimal proposals for particle filtering and to [33] for a more advanced presentation. The main original contribution of this section is Theorem 2, which gives a direct comparison of the ${\chi }^2$-divergence between target and standard/optimal proposals. This result improves on [1], where only a comparison between the intrinsic dimension was established.

2. Importance Sampling and ${\chi }^2$-Divergence

The aim of this section is to demonstrate the central role of the ${\chi }^2$-divergence between target and proposal in determining the accuracy of importance sampling. In Section 2.1 we show how the ${\chi }^2$-divergence arises in both sufficient and necessary conditions on the sample size for accurate importance sampling with bounded test functions. Section 2.2 describes a well-known connection between the effective sample size and the ${\chi }^2$-divergence. Our investigation of importance sampling to approximate the Bayesian update—developed in Section 3 and Section 4—will make use of a closed formula for the ${\chi }^2$-divergence between Gaussians, which we include in Section 2.3 for later reference.

2.1. Sufficient and Necessary Sample Size

Here we provide general sufficient and necessary conditions on the sample size in terms of

$$\rho :=d_{{\chi }^2}(\mu \parallel \pi )+1.$$

We first review upper-bounds on the worst-case bias and mean-squared error of importance sampling with bounded test functions, which imply that accurate importance sampling is guaranteed if $N\gg \rho$. The proof of the bound for the mean-squared error can be found in [1] and the bound for the bias in [2].

Proposition 1

(Sufficient Sample Size). It holds that

$$\begin{array}{cc}\hfill \underset{{\left|\phi \right|}_{\infty }\le 1}{sup}\left|\mathbb{E}\left[{\mu }^N\left(\phi \right)-\mu \left(\phi \right)\right]\right|& \le \frac{4}{N}\rho ,\hfill \\ \hfill \underset{{\left|\phi \right|}_{\infty }\le 1}{sup}\mathbb{E}\left[{\left({\mu }^N\left(\phi \right)-\mu \left(\phi \right)\right)}^2\right]& \le \frac{4}{N}\rho .\hfill \end{array}$$

The next result shows the existence of bounded test functions for which the error may be large with a high probability if $N\ll \rho .$ The idea is taken from [7], but we adapt it here to obtain a result in terms of the ${\chi }^2$-divergence rather than relative entropy. We denote by $\mathsf{g}:=g/\pi \left(g\right)$ the normalized density between $\mu$ and $\pi ,$ and note that $\rho =\pi \left({\mathsf{g}}^2\right)=\mu \left(\mathsf{g}\right).$

Proposition 2

(Necessary Sample Size). Let $U\sim \mu .$ For any $N\ge 1$ and $\alpha \in (0,1)$ there exists a test function φ with ${\left|\phi \right|}_{\infty }\le 1$ such that

$$\mathbb{P}\left(|{\mu }^N\left(\phi \right)-\mu \left(\phi \right)|=\mathbb{P}(\mathsf{g}\left(U\right)>\alpha \rho \right))\ge 1-\frac{N}{\alpha \rho }.$$

(4)

Proof. 

Observe that for the test function $\phi \left(u\right):=\mathbb{1}\left\{\mathsf{g}\right(u)\le \alpha \rho \}$, we have $\mu \left(\phi \right)=\mathbb{P}\left(\mathsf{g}\left(U\right)\le \alpha \rho \right).$ On the other hand, ${\mu }^N\left(\phi \right)=1$ if and only if $\mathsf{g}\left(u^{\left(n\right)}\right)\le \alpha \rho$ for all $1\le n\le N$. This implies that

$$\begin{array}{c}\hfill \mathbb{P}\left(|{\mu }^N\left(\phi \right)-\mu \left(\phi \right)|=\mathbb{P}(\mathsf{g}\left(U\right)>\alpha \rho )\right)\ge 1-N\mathbb{P}(\mathsf{g}\left(u^{\left(1\right)}\right)>\alpha \rho )\ge 1-\frac{N}{\alpha \rho }.\end{array}$$

(5)

□

The power of Proposition 2 is due to the fact that in some singular limit regimes the distribution of $\mathsf{g}\left(U\right)$ concentrates around its expected value $\rho .$ In such a case, for any fixed $\alpha \in (0,1)$ the probability of the event $\mathsf{g}\left(U\right)>\alpha \rho$ will not vanish as the singular limit is approached. This idea will become clear in the proof of Theorem 1 below.

In Section 3 and Section 4 we will investigate the required sample size for importance sampling approximation of the Bayesian update in various singular limits, where target and proposal become further apart as a result of reducing the observation noise, increasing the prior uncertainty or increasing the dimension of the problem. To formalize the discussion in a general abstract setting, let ${\left\{({\mu }_{\theta },{\pi }_{\theta })\right\}}_{\theta >0}$ be a family of targets and proposals such that ${\rho }_{\theta }:=d_{{\chi }^2}({\mu }_{\theta }\parallel {\pi }_{\theta })\to \infty$ as $\theta \to \infty .$ The parameter $\theta$ may represent for instance the size of the precision of the observation noise, the size of the prior covariance or a suitable notion of dimension. Our next result shows a clear dichotomy in the performance of importance sampling along the singular limit depending on whether the sample size grows sublinearly or superlinearly with ${\rho }_{\theta }.$

Theorem 1.

Suppose that ${\rho }_{\theta }\to \infty$ and that $\mathcal{V}:={sup}_{\theta }\frac{\mathbb{V}\left[{\mathsf{g}}_{\theta }\left(U_{\theta }\right)\right]}{{\rho }_{\theta }^2}<1.$ Let $\delta >0.$

i 

If $N_{\theta }={\rho }_{\theta }^{1+\delta },$ then

$$\underset{\theta \to \infty }{lim}\underset{{\left|\phi \right|}_{\infty }\le 1}{sup}\mathbb{E}\left[{\left({\mu }_{\theta }^{N_{\theta }}\left(\phi \right)-{\mu }_{\theta }\left(\phi \right)\right)}^2\right]=0.$$

(6)

ii 

If $N_{\theta }={\rho }_{\theta }^{1-\delta }$, then there exists a fixed $c\in (0,1)$ such that

$$\underset{\theta \to \infty }{lim}\underset{{\left|\phi \right|}_{\infty }\le 1}{sup}\mathbb{P}\left(|{\mu }_{\theta }^{N_{\theta }}\left(\phi \right)-{\mu }_{\theta }\left(\phi \right)|>c\right)=1.$$

(7)

Proof. 

The proof of $\left(i\right)$ follows directly from Proposition 1. For $\left(ii\right)$ we fix $\alpha \in (0,1-\mathcal{V})$ and $c\in \left(0,1-\frac{\mathcal{V}}{{(1-\alpha )}^2}\right)$. Let ${\phi }_{\theta }\left(u\right):=\mathbb{1}({\mathsf{g}}_{\theta }\left(u\right)\le \alpha {\rho }_{\theta })$ as in the proof of Proposition 2. Then,

$$\begin{array}{c}\hfill \mathbb{P}\left({\mathsf{g}}_{\theta }\left(U_{\theta }\right)>\alpha {\rho }_{\theta }\right)\ge 1-\mathbb{P}\left(|{\rho }_{\theta }-{\mathsf{g}}_{\theta }\left(U_{\theta }\right)|\ge (1-\alpha ){\rho }_{\theta }\right)\ge 1-\frac{\mathbb{V}\left[{\mathsf{g}}_{\theta }\left(U_{\theta }\right)\right]}{{(1-\alpha )}^2{\rho }_{\theta }^2}\ge 1-\frac{\mathcal{V}}{{(1-\alpha )}^2}>c.\end{array}$$

The bound in (5) implies that

$$\mathbb{P}\left(|{\mu }_{\theta }^{N_{\theta }}\left({\phi }_{\theta }\right)-{\mu }_{\theta }\left({\phi }_{\theta }\right)|>c\right)\ge \mathbb{P}\left(|{\mu }_{\theta }^N\left({\phi }_{\theta }\right)-{\mu }_{\theta }\left({\phi }_{\theta }\right)|=\mathbb{P}({\mathsf{g}}_{\theta }\left(U_{\theta }\right)>\alpha {\rho }_{\theta }\right))\ge 1-\frac{N_{\theta }}{\alpha {\rho }_{\theta }}.$$

This completes the proof, since if $N_{\theta }={\rho }_{\theta }^{1-\delta }$ the right-hand side goes to 1 as $\theta \to \infty$. □

Remark 1.

As noted in [1], the bound in Proposition 1 is sharp in the asymptotic limit $N\to \infty .$ This implies that, for any fixed θ, the bound $4{\rho }_{\theta }/N$ becomes sharp as $N\to \infty .$ We point out that this statement does not provide direct understanding of the joint limit $\theta ,N_{\theta }\to \infty$ analyzed in Theorem 1.

The assumption that $\mathcal{V}<1$ can be verified for some singular limits of interest, in particular for small noise and large prior covariance limits studied in Section 3 and Section 4; details will be given in Example 1. While the assumption $\mathcal{V}<1$ may fail to hold in high dimensional singular limit regimes, the works [13,14] and our numerical example in Section 4.4 provide compelling evidence of the need to suitably scale N with $\rho$ along those singular limits in order to avoid a weigh-collapse phenomenon. Further theoretical evidence was given for un-normalized importance sampling in [8].

2.2. ${\chi }^2$-Divergence and Effective Sample Size

The previous subsection provides theoretical nonasymptotic and asymptotic evidence that a sample size larger than $\rho$ is necessary and sufficient for accurate importance sampling. Here we recall a well known connection between the ${\chi }^2$-divergence and the effective sample size

$$\mathrm{ESS}:=\frac{1}{{\sum }_{n=1}^N{\left(w^{\left(n\right)}\right)}^2},$$

(8)

widely used by practitioners to monitor the performance of importance sampling. Note that always $1\le \mathrm{ESS}\le N$; it is intuitive that $\mathrm{ESS}=1$ if the maximum weight is one and $\mathrm{ESS}=N$ if the maximum weight is $1/N.$ To see the connection between ESS and $\rho$, note that

$$\begin{array}{cc}\hfill \frac{\mathrm{ESS}}{N}& =\frac{1}{N{\sum }_{n=1}^N{\left(w^{\left(n\right)}\right)}^2}=\frac{{\left({\sum }_{n=1}^{N}g\left(u^{\left(n\right)}\right)\right)}^2}{N{\sum }_{n=1}^{N}g{\left(u^{\left(n\right)}\right)}^2}=\frac{{\left(\frac{1}{N}{\sum }_{n=1}^{N}g\left(u^{\left(n\right)}\right)\right)}^2}{\frac{1}{N}{\sum }_{n=1}^{N}g{\left(u^{\left(n\right)}\right)}^2}\approx \frac{\pi {\left(g\right)}^2}{\pi \left(g^2\right)}.\hfill \end{array}$$

Therefore, $\mathrm{ESS}\approx N/\rho :$ if the sample-based estimate of $\rho$ is significantly larger than N, ESS will be small which gives a warning sign that a larger sample size N may be needed.

2.3. ${\chi }^2$-Divergence between Gaussians

We conclude this section by recalling an analytical expression for the ${\chi }^2$-divergence between Gaussians. In order to make our presentation self-contained, we include a proof in Appendix A.

Proposition 3.

Let $\mu =\mathcal{N}(m,C)$ and $\pi =\mathcal{N}(0,\Sigma )$. If $2\Sigma \succ C$, then

$$\rho =\frac{|\Sigma |}{\sqrt{|2\Sigma -C|\left|C\right|}}exp\left(m^{\prime }{(2\Sigma -C)}^{-1}m\right).$$

Otherwise, $\rho =\infty$.

It is important to note that nondegenerate Gaussians $\mu =\mathcal{N}(m,C)$ and $\pi =\mathcal{N}(0,\Sigma )$ in ${\mathbb{R}}^d$ are always equivalent. However, $\rho =\infty$ unless $2\Sigma \succ C.$ In Section 3 and Section 4 we will interpret $\mu$ as a posterior and $\pi$ as a prior, in which case automatically $\Sigma \succ C$ and $\rho <\infty .$

3. Importance Sampling for Inverse Problems

In this section we study the use of importance sampling in a linear Bayesian inverse problem setting where the target and the proposal represent, respectively, the posterior and the prior distribution. In Section 3.1 we describe our setting and we also derive an explicit formula for the ${\chi }^2$-divergence between the posterior and the prior. This explicit formula allows us to determine the scaling of the ${\chi }^2$-divergence in small noise regimes (Section 3.2), in the limit of large prior covariance (Section 3.3) and in a high dimensional limit (Section 3.4). Our overarching goal is to show how the sample size for importance sampling needs to grow along these limiting regimes in order to maintain the same level of accuracy.

3.1. Inverse Problem Setting and ${\chi }^2$-Divergence between Posterior and Prior

Let $A\in {\mathbb{R}}^{k\times d}$ be a given design matrix and consider the linear inverse problem of recovering $u\in {\mathbb{R}}^d$ from data $y\in {\mathbb{R}}^k$ related by

$$y=Au+\eta ,\eta \sim \mathcal{N}(0,\Gamma ),$$

(9)

where $\eta$ represents measurement noise. We assume henceforth that we are in the underdetermined case $k\le d$, and that A is full rank. We follow a Bayesian perspective and set a Gaussian prior on u, $u\sim \pi =\mathcal{N}(0,\Sigma ).$ We assume throughout that $\Sigma$ and $\Gamma$ are given symmetric positive definite matrices. The solution to the Bayesian formulation of the inverse problem is the posterior distribution $\mu$ of u given $y.$ We are interested in studying the performance of importance sampling with proposal $\pi$ (the prior) and target $\mu$ (the posterior). We recall that under this linear-Gaussian model the posterior distribution is Gaussian [2], and we denote it by $\mu =\mathcal{N}(m,C)$. In order to characterize the posterior mean m and covariance C, we introduce standard data assimilation notation

$$\begin{array}{cc}\hfill S& :=A\Sigma A^{\prime }+\Gamma ,\hfill \\ \hfill K& :=\Sigma A^{\prime }{S}^{-1},\hfill \end{array}$$

where K is the Kalman gain. Then we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill m& =Ky,\hfill \\ \hfill C& =(I-KA)\Sigma .\hfill \end{array}\end{array}$$

(10)

Proposition 3 allows us to obtain a closed formula for the quantity $\rho =d_{{\chi }^2}(\mu \parallel \pi )+1$, noting that (10) implies that

$$\begin{array}{cc}\hfill 2\Sigma -C& =(I+KA)\Sigma \hfill \\ \hfill & =\Sigma +\Sigma A^{\prime }{S}^{-1}A\Sigma \succ 0.\hfill \end{array}$$

The proof of the following result is then immediate and therefore omitted.

Proposition 4.

Consider the inverse problem (9) with prior $u\sim \pi =\mathcal{N}(0,\Sigma )$ and posterior $\mu =\mathcal{N}(m,C)$ with m and C defined in (10). Then $\rho =d_{{\chi }^2}(\mu \parallel \pi )+1$ admits the explicit characterization

$$\rho =\left(\right|I+KA\left|\right|I-{KA\left|\right)}^{-\frac{1}{2}}exp\left(y^{\prime }{K}^{\prime }{[(I+KA)\Sigma ]}^{-1}Ky\right).$$

In the following two subsections we employ this result to derive by direct calculation the rate at which the posterior and prior become further apart —in ${\chi }^2$-divergence— in small noise and large prior regimes. To carry out the analysis we use parameters ${\gamma }^2,{\sigma }^2>0$ to scale the noise covariance, ${\gamma }^2\Gamma ,$ and the prior covariance, ${\sigma }^2\Sigma .$

3.2. Importance Sampling in Small Noise Regime

To illustrate the behavior of importance sampling in small noise regimes, we first introduce a motivating numerical study. A similar numerical setup was used in [13] to demonstrate the curse of dimension of importance sampling. We consider the inverse problem setting in Equation (9) with $d=k=5$ and noise covariance ${\gamma }^2\Gamma .$ We conduct 18 numerical experiments with a fixed data y. For each experiment, we perform importance sampling 400 times and report in Figure 1 a histogram with the largest autonormalized weight in each of the 400 realizations. The 18 experiments differ in the sample size N and the size of the observation noise ${\gamma }^2.$ In both Figure 1a,b we consider three choices of N (rows) and three choices of ${\gamma }^2$ (columns). These choices are made so that in Figure 1a it holds that $N={\gamma }^{-4}$ along the bottom-left to top-right diagonal, while in Figure 1b $N={\gamma }^{-6}$ along the same diagonal.

We can see from Figure 1a that $N={\gamma }^{-4}$ is not a fast enough growth of N to avoid weight collapse: the histograms skew to the right along the bottom-left to top-right diagonal, suggesting that weight collapse (i.e., one weight dominating the rest, and therefore the variance of the weights being large) is bound to occur in the joint limit $N\to \infty ,$$\gamma \to 0$ with $N={\gamma }^{-4}$. In contrast, the histograms in Figure 1b skew to the left along the same diagonal, suggesting that the probability of weight collapse is significantly reduced if $N={\gamma }^{-6}$. We observe a similar behavior with other choices of dimension d by conducting experiments with sample sizes $N={\gamma }^{-d+1}$ and $N={\gamma }^{-d-1}$, and we include the histograms with $d=k=4$ in Appendix C. Our next result shows that these empirical findings are in agreement with the scaling of the ${\chi }^2$-divergence between target and proposal in the small noise limit.

Proposition 5.

Consider the inverse problem setting

$$y=Au+\eta ,\eta =\mathcal{N}(0,{\gamma }^2\Gamma ),u\sim \pi =\mathcal{N}(0,\Sigma ).$$

Let ${\mu }_{\gamma }$ denote the posterior and let ${\rho }_{\gamma }=d_{{\chi }^2}({\mu }_{\gamma }\parallel \pi )+1.$ Then, for almost every $y,$

$${\rho }_{\gamma }\sim \mathcal{O}\left({\gamma }^{-k}\right)$$

in the small noise limit $\gamma \to 0.$

Proof. 

Let $K_{\gamma }=\Sigma A^{\prime }{(A\Sigma A^{\prime }+{\gamma }^2\Gamma )}^{-1}$ denote the Kalman gain. We observe that $K_{\gamma }\to \Sigma A^{\prime }{(A\Sigma A^{\prime })}^{-1}$ under our standing assumption that A is full rank. Let $U^{\prime }\Xi V$ be the singular value decompostion of ${\Gamma }^{-\frac{1}{2}}A{\Sigma }^{\frac{1}{2}}$ and ${\left\{{\xi }_i\right\}}_{i=1}^k$ be the singular values. Then we have

$$\begin{array}{cc}\hfill K_{\gamma }A& \sim {\Sigma }^{\frac{1}{2}}{A}^{\prime }{\Gamma }^{-\frac{1}{2}}{({\Gamma }^{-\frac{1}{2}}A\Sigma A^{\prime }{\Gamma }^{-\frac{1}{2}}+{\gamma }^{2}I)}^{-1}{\Gamma }^{-\frac{1}{2}}A{\Sigma }^{\frac{1}{2}}\hfill \\ \hfill & =V^{\prime }{\Xi }^{\prime }U{(U^{\prime }\Xi V{V}^{\prime }{\Xi }^{\prime }U+{\gamma }^{2}I)}^{-1}{U}^{\prime }\Xi V\hfill \\ \hfill & \sim {\Xi }^{\prime }{(\Xi {\Xi }^{\prime }+{\gamma }^{2}I)}^{-1}\Xi ,\hfill \end{array}$$

where here “∼” denotes matrix similarity. It follows that $I+K_{\gamma }A$ converges to a finite limit, and so does the exponent $y^{\prime }{K}_{\gamma }^{\prime }{\Sigma }^{-1}{(I+K_{\gamma }A)}^{-1}{K}_{\gamma }y$ in Proposition 4. On the other hand,

$$\begin{array}{c}\hfill \left(\right|I+K_{\gamma }A\left|\right|I-K_{\gamma }A{\left|\right)}^{-\frac{1}{2}}={\left(\prod _{i=1}^k\frac{{\gamma }^2}{{\xi }_i^2+{\gamma }^2}\right)}^{-\frac{1}{2}}\sim \mathcal{O}\left({\gamma }^{-k}\right)\end{array}$$

as $\gamma \to 0$. The conclusion follows. □

Remark 2.

The scaling of ρ with ${\gamma }^2$ obtained in Proposition 5 agrees with the lower bound reported in Table 1 in [1], which was derived in a diagonalized setting.

3.3. Importance Sampling and Prior Scaling

Here we illustrate the behavior of importance sampling in the limit of large prior covariance. We start again with a motivating numerical example, similar to the one reported in Figure 1. The behavior is analogous to the small noise regime, which is expected since the ratio of prior and noise covariances determines the closeness between target and proposal. Figure 2 shows that when $d=k=5$ weight collapse is observed frequently when the sample size N grows as ${\sigma }^4,$ but not so often with sample size $N={\sigma }^6$. Similar histograms with $d=k=4$ are included in Appendix C. These empirical results are in agreement with the theoretical growth rate of the ${\chi }^2$-divergence between target and proposal in the limit of large prior covariance, as we prove next.

Proposition 6.

Consider the inverse problem setting

$$y=Au+\eta ,\eta \sim \mathcal{N}(0,\Gamma ),u\sim {\pi }_{\sigma }=\mathcal{N}(0,{\sigma }^2\Sigma ).$$

Let ${\mu }_{\sigma }$ denote the posterior and ${\rho }_{\sigma }=d_{{\chi }^2}({\mu }_{\sigma }\parallel {\pi }_{\sigma })+1.$ Then, for almost every $y,$

$${\rho }_{\sigma }\sim \mathcal{O}\left({\sigma }^d\right)$$

in the large prior limit $\sigma \to \infty .$

Proof. 

Let ${\Sigma }_{\sigma }={\sigma }^2\Sigma$, let $K_{\sigma }={\Sigma }_{\sigma }{A}^{\prime }{(A{\Sigma }_{\sigma }{A}^{\prime }+\Gamma )}^{-1}$ be the Kalman gain. Observing that $K_{\sigma }=K_{\gamma =\frac{1}{\sigma }}$, we apply Proposition 5 and deduce that when $\sigma \to \infty$:

• $K_{\sigma }\to \Sigma A^{\prime }{(A\Sigma A^{\prime }+{\gamma }^2\Gamma )}^{-1}$;
• $I+K_{\sigma }A$ has a well-defined and invertible limit;
• $|I-K_{\sigma }{A|}^{-\frac{1}{2}}\sim \mathcal{O}\left({\sigma }^k\right)$.

On the other hand, we notice that the quadratic term

$$K_{\sigma }^{\prime }{\Sigma }_{\sigma }^{-1}{(I+K_{\sigma }A)}^{-1}{K}_{\sigma }={\sigma }^{-2}{K}_{\sigma }^{\prime }\Sigma {(I+K_{\sigma }A)}^{-1}{K}_{\sigma }$$

vanishes in limit. The conclusion follows by Proposition 4. □

3.4. Importance Sampling in High Dimension

In this subsection we study importance sampling in high dimensional limits. To that end, we let ${\left\{a_i\right\}}_{i=1}^{\infty },$${\left\{{\gamma }_i^2\right\}}_{i=1}^{\infty }$ and ${\left\{{\sigma }_i^2\right\}}_{i=1}^{\infty }$ be infinite sequences and we define, for any $d\ge 1,$

$$\begin{array}{cc}\hfill A_{1:d}& :=\mathrm{diag}\left\{a_1,\dots ,a_d\right\}\in {\mathbb{R}}^{d\times d},\hfill \\ \hfill {\Gamma }_{1:d}& :=\mathrm{diag}\left\{{\gamma }_1^2,\dots ,{\gamma }_d^2\right\}\in {\mathbb{R}}^{d\times d},\hfill \\ \hfill {\Sigma }_{1:d}& :=\mathrm{diag}\left\{{\sigma }_1^2,\dots ,{\sigma }_d^2\right\}\in {\mathbb{R}}^{d\times d}.\hfill \end{array}$$

We then consider the inverse problem of reconstructing $u\in {\mathbb{R}}^d$ from data $y\in {\mathbb{R}}^d$ under the setting

$$y=A_{1:d}u+\eta ,\eta \sim \mathcal{N}(0,{\Gamma }_{1:d}),u\sim {\pi }_{1:d}=\mathcal{N}(0,{\Sigma }_{1:d}).$$

(11)

We denote the corresponding posterior distribution by ${\mu }_{1:d},$ which is Gaussian with a diagonal covariance. Given observation y, we may find the posterior distribution ${\mu }_i$ of $u_i$ by solving the one dimensional linear-Gaussian inverse problem

$$y_i=a_i{u}_i+{\eta }_i,{\eta }_i\sim \mathcal{N}(0,{\gamma }_i^2),1\le i\le d,$$

(12)

with prior ${\pi }_i=\mathcal{N}(0,{\sigma }_i^2).$ In this way we have defined, for each $d\in \mathbb{N}\cup \{\infty \},$ an inverse problem with prior and posterior

$${\pi }_{1:d}=\prod _{i=1}^d{\pi }_i,{\mu }_{1:d}=\prod _{i=1}^d{\mu }_i.$$

(13)

In Section 3.4.1 we include an explicit calculation in the one dimensional inverse setting (12), which will be used in Section 4.4 to establish the rate of growth of ${\rho }_d=d_{{\chi }^2}({\mu }_{1:d}\parallel {\pi }_{1:d})$ and thereby how the sample size needs to be scaled along the high dimensional limit $d\to \infty$ to maintain the same accuracy. Finally, in Section 3.4.3 we establish from first principles and our simple one dimensional calculation the equivalence between $\left(i\right)$ certain notion of dimension being finite; $\left(ii\right)$${\rho }_{\infty }<\infty ;$ and $\left(iii\right)$ absolute continuity of ${\mu }_{1:\infty }$ with respect to ${\pi }_{1:\infty }.$

3.4.1. One Dimensional Setting

Let $a\in \mathbb{R}$ be given and consider the one dimensional inverse problem of reconstructing $u\in \mathbb{R}$ from data $y\in \mathbb{R}$, under the setting

$$y=au+\eta ,\eta \sim \mathcal{N}(0,{\gamma }^2),u\sim \pi =\mathcal{N}(0,{\sigma }^2).$$

(14)

By defining

$$g\left(u\right):=exp\left(-\frac{a^2}{2{\gamma }^2}{u}^2+\frac{ay}{{\gamma }^2}u\right),$$

we can write the posterior density $\mu \left(du\right)$ as $\mu \left(du\right)\propto g\left(u\right)\pi \left(du\right).$ The next result gives a simplified closed formula for $\rho =d_{{\chi }^2}(\mu \parallel \pi )+1.$ In addition, it gives a closed formula for the Hellinger integral

$$\mathcal{H}(\mu ,\pi ):=\frac{\pi \left(g^{\frac{1}{2}}\right)}{\pi {\left(g\right)}^{\frac{1}{2}}},$$

which will facilitate the study of the case $d=\infty$ in Section 3.4.3.

Lemma 1.

Consider the inverse problem in (14). Let $\lambda :=a^2{\sigma }^2/{\gamma }^2$ and $z^2:=\frac{y^2}{a^2{\sigma }^2+{\gamma }^2}.$ Then, for any $\ell >0,$

$$\begin{array}{cc}\hfill \frac{\pi \left(g^{\ell }\right)}{\pi {\left(g\right)}^{\ell }}& =\frac{{(\lambda +1)}^{\frac{\ell }{2}}}{\sqrt{\ell \lambda +1}}exp\left(\frac{({\ell }^2-\ell )\lambda }{2(\ell \lambda +1)}{z}^2\right).\hfill \end{array}$$

(15)

In particular,

$$\begin{array}{cc}\hfill \rho & =\frac{\lambda +1}{\sqrt{2\lambda +1}}exp\left(\frac{\lambda }{2\lambda +1}{z}^2\right),\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill \mathcal{H}(\mu ,\pi )& =\sqrt{\frac{2\sqrt{\lambda +1}}{\lambda +2}}exp\left(-\frac{\lambda z^2}{4(\lambda +2)}\right).\hfill \end{array}$$

(17)

Proof. 

A direct calculation shows that

$$\pi \left(g\right)=\frac{1}{\sqrt{\lambda +1}}exp\left(\frac{1}{2}\frac{\lambda y^2}{a^2{\sigma }^2+{\gamma }^2}\right).$$

The same calculation, but replacing ${\gamma }^2$ by ${\gamma }^2/\ell$ and $\lambda$ by $\ell \lambda$, gives similar expressions for $\pi \left(g^{\ell }\right)$, which leads to (15). The other two equations follow by setting ℓ to be 2 and $\frac{1}{2}$. □

Lemma 1 will be used in the two following subsections to study high dimensional limits. Here we show how this lemma also allows us to verify directly that the assumption $\mathcal{V}<1$ in Theorem 1 holds in small noise and large prior limits.

Example 1.

Consider a sequence of inverse problems of the form (14) with $\lambda =a^2{\sigma }^2/{\gamma }^2$ approaching infinity. Let ${\left\{({\mu }_{\lambda },{\pi }_{\lambda })\right\}}_{\lambda >0}$ be the corresponding family of posteriors and priors and let ${\mathsf{g}}_{\lambda }$ be the normalized density. Lemma 1 implies that

$$\begin{array}{cc}\hfill \frac{{\pi }_{\lambda }\left({\mathsf{g}}_{\lambda }^3\right)}{{\pi }_{\lambda }{\left({\mathsf{g}}_{\lambda }^2\right)}^2}=& \frac{2\lambda +1}{\sqrt{(3\lambda +1)(\lambda +1)}}exp\left(\frac{\lambda }{(2\lambda +1)(3\lambda +1)}{z}^2\right)\to \frac{2}{\sqrt{3}}<2,\hfill \end{array}$$

as $\lambda \to \infty$. This implies that, for λ sufficiently large,

$$\frac{\mathbb{V}\left[{\mathsf{g}}_{\lambda }\left(U_{\lambda }\right)\right]}{{\rho }_{\lambda }^2}=\frac{{\pi }_{\lambda }\left({\mathsf{g}}_{\lambda }^3\right)}{{\pi }_{\lambda }{\left({\mathsf{g}}_{\lambda }^2\right)}^2}-1<1.$$

3.4.2. Large Dimensional Limit

Now we investigate the behavior of importance sampling in the limit of large dimension, in the inverse problem setting (11). We start with an example similar to the ones in Figure 1 and Figure 2. Figure 3 shows that for $\lambda =1.3$ fixed, weight collapse happens frequently when the sample size N grows polynomially as $d^2$ but not so often if N grows at rate $\mathcal{O}\left({\prod }_{i=1}^d\left(\frac{\lambda +1}{\sqrt{2\lambda +1}}{e}^{\frac{\lambda z_i^2}{2\lambda +1}}\right)\right)$. These empirical results are in agreement with the growth rate of ${\rho }_d$ in the large d limit.

Proposition 7.

For any $d\in \mathbb{N}\cup \{\infty \},$

$$\begin{array}{cc}\hfill {\rho }_d& =\prod _{i=1}^d\left(\frac{{\lambda }_i+1}{\sqrt{2{\lambda }_i+1}}{e}^{\frac{{\lambda }_i{z}_i^2}{2{\lambda }_i+1}}\right),\hfill \\ \hfill {\mathbb{E}}_{z_{1:d}}\left[{\rho }_d\right]& =\prod _{i=1}^d({\lambda }_i+1).\hfill \end{array}$$

Proof. 

The formula for ${\rho }_d$ is a direct consequence of Equation (16) and the product structure. Similarly, we have

$$\begin{array}{cc}\hfill {\mathbb{E}}_{z_i}\left[\frac{{\lambda }_i+1}{\sqrt{2{\lambda }_i+1}}{e}^{\frac{{\lambda }_i{z}_i^2}{2{\lambda }_i+1}}\right]& =\frac{{\lambda }_i+1}{\sqrt{2{\lambda }_i+1}}{\int }_{\mathbb{R}}\frac{1}{\sqrt{2\pi }}{e}^{-\frac{z_i^2}{2}+\frac{{\lambda }_i{z}_i^2}{2{\lambda }_i+1}}d{z}_i\hfill \\ \hfill & =\frac{{\lambda }_i+1}{\sqrt{2{\lambda }_i+1}}{\int }_{\mathbb{R}}\frac{1}{\sqrt{2\pi }}{e}^{-\frac{z_i^2}{2(2{\lambda }_i+1)}}d{z}_i\hfill \\ \hfill & ={\lambda }_i+1.\hfill \end{array}$$

□

Proposition 7 implies that, for $d\in \mathbb{N}\cup \{\infty \},$

$$\begin{array}{cc}\hfill \underset{{\left|\phi \right|}_{\infty }\le 1}{sup}\mathbb{E}\left[{\left({\mu }_{1:d}^N\left(\phi \right)-{\mu }_{1:d}\left(\phi \right)\right)}^2\right]& \le \frac{4}{N}\prod _{i=1}^d\left(\frac{{\lambda }_i+1}{\sqrt{2{\lambda }_i+1}}{e}^{\frac{{\lambda }_i{z}_i^2}{2{\lambda }_i+1}}\right),\hfill \\ \hfill \mathbb{E}\left[\underset{{\left|\phi \right|}_{\infty }\le 1}{sup}\mathbb{E}\left[{\left({\mu }_{1:d}^N\left(\phi \right)-{\mu }_{1:d}\left(\phi \right)\right)}^2\right]\right]& \le \frac{4}{N}\prod _{i=1}^d({\lambda }_i+1).\hfill \end{array}$$

Note that the outer expected value in the latter equation averages over the data, while the inner one averages oversampling from the prior ${\pi }_{1:d}$. This suggests that

$$log\mathbb{E}\left[\underset{{\left|\phi \right|}_{\infty }\le 1}{sup}\mathbb{E}\left[{\left({\mu }_{1:d}^N\left(\phi \right)-{\mu }_{1:d}\left(\phi \right)\right)}^2\right]\right]\lesssim \sum _{i=1}^d{\lambda }_i-logN.$$

The quantity $\tau :={\sum }_{i=1}^d{\lambda }_i$ had been used as an intrinsic dimension of the inverse problem (11). This simple heuristic together with Theorem 1 suggest that increasing Nexponentially with $\tau$ is both necessary and sufficient to maintain accurate importance sampling along the high dimensional limit $d\to \infty .$ In particular, if all coordinates of the problem play the same role, this implies that N needs to grow exponentially with d, a manifestation of the curse of dimension of importance sampling [1,13,14].

3.4.3. Infinite Dimensional Singularity

Finally, we investigate the case $d=\infty .$ Our goal in this subsection is to establish a connection between the effective dimension, the quantity ${\rho }_{\infty }$ and absolute continuity. The main result, Corollary 1, had been proved in more generality in [1]. However, our proof and presentation here requires minimal technical background and is based on the explicit calculations obtained in the previous subsections and in the following lemma.

Lemma 2.

It holds that ${\mu }_{1:\infty }$ is absolutely continuous with respect to ${\pi }_{1:\infty }$ if and only if

$$\mathcal{H}({\mu }_{1:\infty },{\pi }_{1:\infty })=\prod _{i=1}^{\infty }\frac{{\pi }_i\left(g_i^{\frac{1}{2}}\right)}{{\pi }_i{\left(g_i\right)}^{\frac{1}{2}}}>0,$$

(18)

where $g_i$ is an un-normalized density between ${\mu }_i$ and ${\pi }_i.$ Moreover, we have the following explicit characterizations of the Hellinger integral $\mathcal{H}({\mu }_{1:\infty },{\pi }_{1:\infty })$ and its average with respect to data realizations,

$$\begin{array}{cc}\hfill \mathcal{H}({\mu }_{1:\infty },{\pi }_{1:\infty })& =\prod _{i=1}^{\infty }\left(\sqrt{\frac{2\sqrt{{\lambda }_i+1}}{{\lambda }_i+2}}{e}^{-\frac{{\lambda }_i{z}_i^2}{4({\lambda }_i+2)}}\right),\hfill \\ \hfill {\mathbb{E}}_{z_{1:\infty }}\left[\mathcal{H}({\mu }_{1:\infty },{\pi }_{1:\infty })\right]& =\prod _{i=1}^{\infty }\frac{2{({\lambda }_i+1)}^{\frac{1}{4}}}{\sqrt{3{\lambda }_i+4}}.\hfill \end{array}$$

Proof. 

The formula for the Hellinger integral is a direct consequence of Equation (17) and the product structure. On the other hand,

$$\begin{array}{cc}\hfill {\mathbb{E}}_{z_i}\left[\sqrt{\frac{2\sqrt{{\lambda }_i+1}}{{\lambda }_i+2}}{e}^{-\frac{{\lambda }_i{z}_i^2}{4({\lambda }_i+2)}}\right]& =\frac{\sqrt{2}{({\lambda }_i+1)}^{\frac{1}{4}}}{\sqrt{{\lambda }_i+2}}{\int }_{\mathbb{R}}\frac{1}{\sqrt{2\pi }}{e}^{-\frac{{\lambda }_i{z}_i^2}{4({\lambda }_i+2)}-\frac{z_i^2}{2}}d{z}_i\hfill \\ \hfill & =\frac{2{({\lambda }_i+1)}^{\frac{1}{4}}}{\sqrt{3{\lambda }_i+4}}.\hfill \end{array}$$

The proof of the equivalence between finite Hellinger integral and absolute continuity is given in Appendix B. □

Corollary 1.

The following statements are equivalent:

i 

$\tau ={\sum }_{i=1}^{\infty }{\lambda }_i<\infty$;

ii 

${\rho }_{\infty }<\infty$ for almost every y;

iii 

${\mu }_{1:\infty }\ll {\pi }_{1:\infty }$ for almost every y.

Proof. 

Observe that ${\lambda }_i\to 0$ is a direct consequence of all three statements, so we will assume ${\lambda }_i\to 0$ from now on.

$\left(i\right)\iff \left(ii\right):$ By Proposition 7,

$$\begin{array}{c}\hfill log\left({\mathbb{E}}_{z_{1:\infty }}\left[{\rho }_{\infty }\right]\right)=\sum _{i=1}^{\infty }log(1+{\lambda }_i)=\mathcal{O}\left(\sum _{i=1}^{\infty }{\lambda }_i\right),\end{array}$$

since $log(1+{\lambda }_i)\approx {\lambda }_i$ for large i.

$\left(i\right)\iff \left(iii\right):$ Similarly, we have

$$\begin{array}{cc}\hfill log\left({\mathbb{E}}_{z_{1:\infty }}\left[\mathcal{H}({\mu }_{1:\infty },{\pi }_{1:\infty })\right]\right)& =-\frac{1}{4}\sum _{i=1}^{\infty }log\frac{{(3{\lambda }_i+4)}^2}{16({\lambda }_i+1)}\hfill \\ \hfill & =-\frac{1}{4}\sum _{i=1}^{\infty }log\left(1+\frac{9{\lambda }_i^2+8{\lambda }_i}{16{\lambda }_i+16}\right)\hfill \\ \hfill & =-\frac{1}{4}\mathcal{O}\left(\sum _{i=1}^{\infty }{\lambda }_i\right).\hfill \end{array}$$

The conclusion follows from Lemma 2. □

4. Importance Sampling for Data Assimilation

In this section, we study the use of importance sampling in a particle filtering setting. Following [13,14,15] we focus on one filtering step. Our goal is to provide a new and concrete comparison of two proposals, referred to as standard and optimal in the literature [1]. In Section 4.1 we introduce the setting and both proposals and show that the ${\chi }^2$-divergence between target and standard proposal is larger than the ${\chi }^2$-divergence between target and optimal proposal. Section 4.3 and Section 4.4 identify small noise and large dimensional limiting regimes where the sample size for the standard proposal needs to grow unboundedly to maintain the same level of accuracy, but the required sample size for the optimal proposal remains bounded.

4.1. One-Step Filtering Setting

Let M and H be given matrices. We consider the one-step filtering problem of recovering $v_0,v_1$ from y, under the following setting

$$\begin{array}{cc}\hfill v_1& =M{v}_0+\xi ,v_0\sim \mathcal{N}(0,P),\xi \sim \mathcal{N}(0,Q),\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill y& =H{v}_1+\zeta ,\zeta \sim \mathcal{N}(0,R).\hfill \end{array}$$

(20)

Similar to the setting in SubSection 3.1, we assume that $P,Q,R$ are symmetric positive definite and that M and H are full rank. From a Bayesian point of view, we would like to sample from the target distribution ${\mathbb{P}}_{v_0,v_1|y}$. To achieve this, we can either use ${\pi }_{\mathrm{std}}={\mathbb{P}}_{v_1|v_0}{\mathbb{P}}_{v_0}$ or ${\pi }_{\mathrm{opt}}={\mathbb{P}}_{v_1|v_0,y}{\mathbb{P}}_{v_0}$ as the proposal distribution.

The standard proposal ${\pi }_{\mathrm{std}}$ is the prior distribution of $(v_0,v_1)$ determined by the prior $v_0\sim \mathcal{N}(0,P)$ and the signal dynamics encoded in Equation (19). Then assimilating the observation y leads to an inverse problem [1,2] with design matrix, noise covariance and prior covariance given by

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill A_{\mathrm{std}}& :=H,\hfill \\ \hfill {\Gamma }_{\mathrm{std}}& :=R,\hfill \\ \hfill {\Sigma }_{\mathrm{std}}& :=MP{M}^{\prime }+Q.\hfill \end{array}\end{array}$$

(21)

We denote ${\pi }_{\mathrm{std}}=\mathcal{N}(0,{\Sigma }_{\mathrm{std}})$ the prior distribution and by ${\mu }_{\mathrm{std}}$ the corresponding posterior distribution.

The optimal proposal ${\pi }_{\mathrm{opt}}$ samples from $v_0$ and the conditional kernel $v_1|v_0,y.$ Then assimilating y leads to the inverse problem [1,2]

$$y=HM{v}_0+H\xi +\zeta ,$$

where the design matrix, noise covariance and prior covariance are given by

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill A_{\mathrm{opt}}& :=HM,\hfill \\ \hfill {\Gamma }_{\mathrm{opt}}& :=HQ{H}^{\prime }+R,\hfill \\ \hfill {\Sigma }_{\mathrm{opt}}& :=P.\hfill \end{array}\end{array}$$

(22)

We denote ${\pi }_{\mathrm{opt}}=\mathcal{N}(0,{\Sigma }_{\mathrm{opt}})$ the prior distribution and ${\mu }_{\mathrm{std}}$ the corresponding posterior distribution.

4.2. ${\chi }^2$ -Divergence Comparison between Standard and Optimal Proposal

Here we show that

$${\rho }_{\mathrm{std}}:=d_{{\chi }^2}({\mu }_{\mathrm{std}}\parallel {\pi }_{\mathrm{std}})+1>d_{{\chi }^2}({\mu }_{\mathrm{opt}}\parallel {\pi }_{\mathrm{opt}})+1=:{\rho }_{\mathrm{opt}}.$$

The proof is a direct calculation using the explicit formula in Proposition 4. We introduce, as in Section 3, standard Kalman notation

$$\begin{array}{c}\hfill K_{\mathrm{std}}:={\Sigma }_{\mathrm{std}}{A}_{\mathrm{std}}^{\prime }{S}_{\mathrm{std}}^{-1},S_{\mathrm{std}}:=A_{\mathrm{std}}{\Sigma }_{\mathrm{std}}{A}_{\mathrm{std}}^{\prime }+{\Gamma }_{\mathrm{std}},\\ \hfill K_{\mathrm{opt}}:={\Sigma }_{\mathrm{opt}}{A}_{\mathrm{opt}}^{\prime }{S}_{\mathrm{opt}}^{-1},S_{\mathrm{opt}}:=A_{\mathrm{opt}}{\Sigma }_{\mathrm{opt}}{A}_{\mathrm{opt}}^{\prime }+{\Gamma }_{\mathrm{opt}}.\end{array}$$

It follows from the definitions in (21) and (22) that

$$\begin{array}{cc}\hfill S_{\mathrm{std}}& =H(MP{M}^{\prime }+Q)H+R\hfill \\ \hfill & =HMP{M}^{\prime }H+HQ{H}^{\prime }+R\hfill \\ \hfill & =S_{\mathrm{opt}}.\hfill \end{array}$$

Since $S_{\mathrm{std}}=S_{\mathrm{opt}}$ we drop the subscripts in what follows and denote both simply by $S.$

Theorem 2.

Consider the one-step filtering setting in Equations (19) and (20). If M and H are full rank and $P,Q,R$ are symmetric positive definite, then, for almost every $y,$

$${\rho }_{\mathrm{std}}>{\rho }_{\mathrm{opt}}.$$

Proof. 

By Proposition 4 we have

$$\begin{array}{cc}\hfill {\rho }_{\mathrm{std}}& =\left(\right|I-K_{\mathrm{std}}{A}_{\mathrm{std}}\left|\right|I+K_{\mathrm{std}}{A}_{\mathrm{std}}{\left|\right)}^{-\frac{1}{2}}exp\left(y^{\prime }{K}_{\mathrm{std}}^{\prime }{\left[(I+K_{\mathrm{std}}{A}_{\mathrm{std}}){\Sigma }_{\mathrm{std}}\right]}^{-1}{K}_{\mathrm{std}}y\right),\hfill \\ \hfill {\rho }_{\mathrm{opt}}& =\left(\right|I-K_{\mathrm{opt}}{A}_{\mathrm{opt}}\left|\right|I+K_{\mathrm{opt}}{A}_{\mathrm{opt}}{\left|\right)}^{-\frac{1}{2}}exp\left(y^{\prime }{K}_{\mathrm{opt}}^{\prime }{\left[(I+K_{\mathrm{opt}}{A}_{\mathrm{opt}}){\Sigma }_{\mathrm{std}}\right]}^{-1}{K}_{\mathrm{opt}}y\right).\hfill \end{array}$$

Therefore, it suffices to prove the following two inequalities:

$$\begin{array}{cc}\hfill |I-K_{\mathrm{std}}{A}_{\mathrm{std}}\left|\right|I+K_{\mathrm{std}}{A}_{\mathrm{std}}|& <|I-K_{\mathrm{opt}}{A}_{\mathrm{opt}}\left|\right|I+K_{\mathrm{opt}}{A}_{\mathrm{opt}}|,\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill K_{\mathrm{std}}^{\prime }{\left[(I+K_{\mathrm{std}}{A}_{\mathrm{std}}){\Sigma }_{\mathrm{std}}\right]}^{-1}{K}_{\mathrm{std}}& \prec K_{\mathrm{opt}}^{\prime }{\left[(I+K_{\mathrm{opt}}{A}_{\mathrm{opt}}){\Sigma }_{\mathrm{std}}\right]}^{-1}{K}_{\mathrm{opt}}.\hfill \end{array}$$

(24)

We start with inequality (24). Note that

$$\begin{array}{cc}\hfill (I+K_{\mathrm{std}}{A}_{\mathrm{std}}){\Sigma }_{\mathrm{std}}& ={\Sigma }_{\mathrm{std}}+{\Sigma }_{\mathrm{std}}{A}_{\mathrm{std}}^{\prime }{S}^{-1}{A}_{\mathrm{std}}{\Sigma }_{\mathrm{std}},\hfill \\ \hfill (I+K_{\mathrm{opt}}{A}_{\mathrm{opt}}){\Sigma }_{\mathrm{opt}}& ={\Sigma }_{\mathrm{opt}}+{\Sigma }_{\mathrm{opt}}{A}_{\mathrm{opt}}^{\prime }{S}^{-1}{A}_{\mathrm{opt}}{\Sigma }_{\mathrm{opt}}.\hfill \end{array}$$

Using the definitions in (21) and (22) it follows that

$$\begin{array}{cc}\hfill K_{\mathrm{std}}^{\prime }{\Sigma }_{\mathrm{std}}^{-1}{(I+K_{\mathrm{std}}{A}_{\mathrm{std}})}^{-1}{K}_{\mathrm{std}}& =H{\left\{{(MP{M}^{\prime }+Q)}^{-1}+H^{\prime }SH\right\}}^{-1}{H}^{\prime }\hfill \\ \hfill & \prec H{\left\{{\left(MP{M}^{\prime }\right)}^{-1}+H^{\prime }SH\right\}}^{-1}{H}^{\prime }\hfill \\ \hfill & =K_{\mathrm{opt}}^{\prime }{\Sigma }_{\mathrm{opt}}^{-1}{(I+K_{\mathrm{opt}}{A}_{\mathrm{opt}})}^{-1}{K}_{\mathrm{opt}}.\hfill \end{array}$$

For inequality (23), we notice that

$$\begin{array}{cc}\hfill & K_{\mathrm{std}}{A}_{\mathrm{std}}=(MP{M}^{\prime }+Q)H^{\prime }{S}^{-1}H=M\tilde{P}{M}^{\prime }{H}^{\prime }{S}^{-1}H\sim {\left(H^{\prime }{S}^{-1}H\right)}^{\frac{1}{2}}M\tilde{P}{M}^{\prime }{\left(H^{\prime }{S}^{-1}H\right)}^{\frac{1}{2}},\hfill \\ \hfill & K_{\mathrm{opt}}{A}_{\mathrm{opt}}=P{M}^{\prime }{H}^{\prime }{S}^{-1}HM\sim {\left(H^{\prime }{S}^{-1}H\right)}^{\frac{1}{2}}MP{M}^{\prime }{\left(H^{\prime }{S}^{-1}H\right)}^{\frac{1}{2}},\hfill \end{array}$$

where $\tilde{P}:=P+M^{†}Q{M}^{{}^{\prime }†}.$ Therefore

$$K_{\mathrm{opt}}{A}_{\mathrm{opt}}\prec K_{\mathrm{std}}{A}_{\mathrm{std}}$$

which, together with $K_{\mathrm{std}}{A}_{\mathrm{std}}\prec I,$ implies that

$$\begin{array}{cc}\hfill |I-K_{\mathrm{std}}{A}_{\mathrm{std}}\left|\right|I+K_{\mathrm{std}}{A}_{\mathrm{std}}|-|I-K_{\mathrm{opt}}{A}_{\mathrm{opt}}\left|\right|I+K_{\mathrm{opt}}{A}_{\mathrm{opt}}|=& |I-{\left(K_{\mathrm{std}}{A}_{\mathrm{std}}\right)}^2|-|I-{\left(K_{\mathrm{opt}}{A}_{\mathrm{opt}}\right)}^2|>0,\hfill \end{array}$$

as desired. □

Remark 3.

It is well known that if the signal dynamics are deterministic, i.e., if $Q=0$ in (19), then the standard and optimal proposal agree, and therefore ${\rho }_{\mathrm{opt}}={\rho }_{\mathrm{std}}$. Theorem 2 shows that ${\rho }_{\mathrm{std}}>{\rho }_{\mathrm{opt}}$ provided that Q is positive definite. Further works that have investigated theoretical and practical benefits of the optimal proposal over the standard proposal include [1,2,31,34]. In particular, [1] shows that use of the optimal proposal reduces the intrinsic dimension. Theorem 2 compares directly the ${\chi }^2$-divergence, which is the key quantity that determines the performance of importance sampling.

4.3. Standard and Optimal Proposal in Small Noise Regime

It is possible that along a certain limiting regime, $\rho$ diverges for the standard proposal but not for the optimal proposal. This has been observed in previous work [1,31], and here we provide some concrete examples using the scaling results from Section 3. Precisely, consider the following one-step filtering setting

$$\begin{array}{cc}\hfill v_1& =M{v}_0+\xi ,v_0\sim \mathcal{N}(0,P),\xi \sim \mathcal{N}(0,Q),\hfill \\ \hfill y& =H{v}_1+\zeta ,\zeta \sim \mathcal{N}(0,r^{2}R),\hfill \end{array}$$

where $r\to 0$. Let ${\mu }_{\mathrm{opt}}^{\left(r\right)},{\mu }_{\mathrm{std}}^{\left(r\right)}$ be the optimal/standard targets and ${\pi }_{\mathrm{opt}}^{\left(r\right)},{\pi }_{\mathrm{std}}^{\left(r\right)}$ be the optimal/standard proposals. We assume that $M\in {\mathbb{R}}^{d\times d}$ and $H\in {\mathbb{R}}^{k\times d}$ are full rank.

Proposition 8.

If $r\to 0$, then we have

$$\begin{array}{cc}\hfill & {\rho }_{\mathrm{opt}}^{\left(r\right)}<\infty ,\hfill \\ \hfill & {\rho }_{\mathrm{std}}^{\left(r\right)}\sim \mathcal{O}\left(r^{-k}\right).\hfill \end{array}$$

Proof. 

Consider the two inverse problems that correspond to ${\mu }_{\mathrm{opt}}^{\left(r\right)},{\pi }_{\mathrm{opt}}^{\left(r\right)}$ and ${\mu }_{\mathrm{std}}^{\left(r\right)},{\pi }_{\mathrm{std}}^{\left(r\right)}$. Note that the two problems have identical prior and design matrix. Let ${\Gamma }_{\mathrm{opt}}^{\left(r\right)}$ and ${\Gamma }_{\mathrm{std}}^{\left(r\right)}$ denote the noise in those two inverse problems. When r goes to 0, we observe that

$$\begin{array}{cc}\hfill & {\Gamma }_{\mathrm{opt}}^{\left(r\right)}=r^{2}R+HQ{H}^{\prime }\to HQ{H}^{\prime },\hfill \\ \hfill & {\Gamma }_{\mathrm{std}}^{\left(r\right)}=r^{2}R\to 0.\hfill \end{array}$$

Therefore, the limit of ${\rho }_{\mathrm{opt}}^{\left(r\right)}$ converges to a finite value, but Lemma 5 implies that ${\rho }_{\mathrm{std}}^{\left(r\right)}$ diverges at rate $\mathcal{O}\left(r^{-k}\right)$. □

4.4. Standard and Optimal Proposal in High Dimension

The previous subsection shows that the standard and optimal proposals can have dramatically different behavior in the small noise regime $r\to 0.$ Here we show that both proposals can also lead to dramatically different behavior in high dimensional limits. Precisely, as a consequence of Corollary 1 we can easily identify the exact regimes where both proposals converge or diverge in limit. The notation is analogous to that in Section 4.4, and so we omit the details.

Proposition 9.

Consider the sequence of particle filters defined as above. We have the following convergence criteria:

1. 

${\mu }_{\mathrm{opt}}^{(1:\infty )}\ll {\pi }_{\mathrm{opt}}^{(1:\infty )}$ and ${\rho }_{\mathrm{opt}}<\infty$ if and only if ${\sum }_{i=1}^{\infty }\frac{h_i^2{m}_i^2{p}_i^2}{h_i^2{q}_i^2+r_i^2}<\infty$,

2. 

${\mu }_{\mathrm{std}}^{(1:\infty )}\ll {\pi }_{\mathrm{std}}^{(1:\infty )}$ and ${\rho }_{\mathrm{std}}<\infty$ if and only if ${\sum }_{i=1}^{\infty }\frac{h_i^2{m}_i^2{p}_i^2}{r_i^2}<\infty$ and ${\sum }_{i=1}^{\infty }\frac{h_i^2{q}_i^2}{r_i^2}<\infty$.

Proof. 

By direct computation, we have

$$\begin{array}{cc}\hfill & {\lambda }_{\mathrm{std}}^{\left(i\right)}=\frac{h_i^2{m}_i^2{p}_i^2+h_i^2{q}_i^2}{r_i^2}=\frac{h_i^2{m}_i^2{p}_i^2}{r_i^2}+\frac{h_i^2{q}_i^2}{r_i^2},\hfill \\ \hfill & {\lambda }_{\mathrm{opt}}^{\left(i\right)}=\frac{h_i^2{m}_i^2{p}_i^2}{h_i^2{q}_i^2+r_i^2}.\hfill \end{array}$$

Theorem 1 gives the desired result. □

Example 2.

As a simple example where absolute continuity holds for the optimal proposal but not for the standard one, let $h_i=m_i=p_i=r_i=1.$ Then ${\rho }_{\mathrm{std}}=\infty ,$ but ${\rho }_{\mathrm{opt}}<\infty$ provided that ${\sum }_{i=1}^{\infty }\frac{1}{q_i^2+1}<\infty .$