Bayesian Surrogate Analysis and Uncertainty Propagation ^†

Abstract

The quantification of uncertainties of computer simulations due to input parameter uncertainties is paramount to assess a model’s credibility. For computationally expensive simulations, this is often feasible only via surrogate models that are learned from a small set of simulation samples. The surrogate models are commonly chosen and deemed trustworthy based on heuristic measures, and substituted for the simulation in order to approximately propagate the simulation input uncertainties to the simulation output. In the process, the contribution of the uncertainties of the surrogate itself to the simulation output uncertainties is usually neglected. In this work, we specifically address the case of doubtful surrogate trustworthiness, i.e., non-negligible surrogate uncertainties. We find that Bayesian probability theory yields a natural measure of surrogate trustworthiness, and that surrogate uncertainties can easily be included in simulation output uncertainties. For a Gaussian likelihood for the simulation data, with unknown surrogate variance and given a generalized linear surrogate model, the resulting formulas reduce to simple matrix multiplications. The framework contains Polynomial Chaos Expansions as a special case, and is easily extended to Gaussian Process Regression. Additionally, we show a simple way to implicitly include spatio-temporal correlations. Lastly, we demonstrate a numerical example where surrogate uncertainties are in part negligible and in part non-negligible.

1. Introduction

Uncertainty quantification of simulations has gained increasing attention, e.g., in the field of Computational Engineering, in order to address doubtful parameter choices and assess the models’ credibility. Surrogate models have become a popular tool to propagate simulation input uncertainties to the simulation output, particularly for modern day applications with high computational cost and many uncertain model parameters. For that, a parametrized surrogate model (synonyms: meta-model, emulator) is learned from a finite set of simulation samples. i.e., the surrogate is a function of the uncertain simulation input parameters that is ’fitted’ to the simulation output data. The quality of this fit is then judged by heuristic diagnostics, and the surrogate deemed trustworthy, respectively. A key aspect of this procedure is that the surrogate can be evaluated much faster than the simulation, and still retains a reasonable approximation to the simulation.

The simulation is then substituted with the surrogate model in order to compute the marginal probability density function of the simulation output. The simulation uncertainties are thus inferred from the surrogate model instead of the original simulation model at a significantly reduced computational effort. While this practice allows for obtaining estimates on uncertainties of expensive simulations in the first place, the contribution of the uncertainty of the surrogate itself to the total simulation uncertainty is commonly neglected. In other words, the estimation of the surrogate parameters based on the finite set of simulation samples entails an additional uncertainty in the sought-for uncertainty of the simulation output. The purpose of this paper is to investigate this surrogate uncertainty as the natural measure for the surrogate’s trustworthiness, and how the surrogate uncertainty affects the simulation output uncertainty.

In many cases, the surrogate uncertainty is indeed small if the heuristic diagnostics naively imply so. If the heuristic diagnostics imply that the surrogate is not trustworthy, one may resort to two options: (i) Acquire more simulation data until the surrogate is trustworthy. This is limited by the computational budget and the surrogate’s convergence properties. (ii) Shrink the parameter space, e.g., omit a number of uncertain simulation parameters by assuming definite parameter values. However, in some cases, (i) is not feasible and (ii) is not desired. In this contribution, we demonstrate how to include surrogate uncertainties if the user deals with a surrogate model with doubtful trustworthiness.

Popular surrogate models are Polynomial Chaos Expansions [1,2,3] and Gaussian Process Regression [4,5], the latter of which has had its renaissance recently from within the machine learning community. In this work, we assume a Gaussian likelihood for the simulation data with unknown variance and given a generalized linear surrogate model (i.e., linear in the surrogate parameters) that includes Polynomial Chaos Expansions as a special case and is easily extended to Gaussian Process Regression. Other Bayesian perspectives on Uncertainty Quantification of computer simulations with these popular surrogate models are given in [6,7,8,9,10,11,12,13]. A comprehensive collection of reviews on Uncertainty Quantification, from the point of view of computational engineering and applied mathematics, can be found in [14]. In [7,15], a statistician’s perspective is discussed. Here, we will use Bayesian Probability Theory [16].

2. Bayesian Uncertainty Quantification

We start with the general structure of uncertainty propagation problems based on surrogate models in Section 2.1. In Section 2.2, we analyze a generalized linear surrogate model with a Gaussian likelihood for the simulation data with unknown surrogate variance. In Section 2.3, we proceed to use the surrogate model to propagate the input uncertainties to the output, and show how the surrogate’s uncertainties too can be included.

2.1. General Structure of the Problem

The goal in this paper is to quantify the uncertainties of the simulation results for the observable $z^{\left(x\right)}$ at different measurement points $x=1,...,N_x$ in the simulation domain. e.g., z could be the mechanical stress resulting from a structural analysis with a finite element simulation, where x could denote the location of the measurement probe in or on the analyzed structure. $z^{\left(x\right)}$ depends on unknown or uncertain model parameters $\mathit{a}={\left\{a_i\right\}}_{i=1}^{N_a}$, which are generally inferred from experimental data ${\mathit{d}}_{\exp }$. Based on these data, Bayes’ theorem allows for determining the posterior probability density function (pdf) for $\mathit{a}$,

$$\begin{array}{c}\hfill p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right),\end{array}$$

(1)

where all background information on the experiment is subsumed in $\mathcal{I}$. The implications of the background information $\mathcal{I}$ will be discussed later. This pdf will be assumed to be (almost) arbitrary but given in the following considerations. It usually is the result of a statistical data analysis of the foregoing experiment. This experiment could be the measurement of some material property needed for the simulation, e.g., viscosity for a computational fluid dynamics simulation. The uncertainty of the model parameters $\mathit{a}$ entails an uncertainty in the simulated observable $z^{\left(x\right)}$, and the latter is determined by the marginalization rule,

$$\begin{array}{cc}\hfill p\left(z^{\left(x\right)}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)& =\int p\left(z^{\left(x\right)}\mid \mathit{a},\cancel{{\mathit{d}}_{\exp }},\mathcal{I}\right)p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)d{V}_{\mathit{a}}.\hfill \end{array}$$

(2)

In the first pdf, we have struck out ${\mathit{d}}_{\exp }$ because the knowledge of $\mathit{a}$ suffices to perform the simulation to obtain $z^{\left(x\right)}$. If $\mathit{a}$ consists of only one or two parameters ($N_a=1,2$), then the numerical evaluation of the integral over the model parameters $\mathit{a}$ will typically require a few dozens to hundreds of simulations. The uncertainty propagation is then done and no surrogate is needed. However, this is the trivial case, and usually $\mathit{a}$ will consist of way more parameters. Let us assume that $\mathit{a}$ consists of, e.g., four parameters. That would imply the need for performing simulations at least ${10}^5$ times, which is way too CPU expensive for most real problems. This can be avoided if the simulations are replaced by a surrogate model that approximates the observable z by a suitable parametrized surrogate function $z_{\mathrm{sur}}=g\left(\mathit{a}\right|\mathit{c})$, where $\mathit{c}$ are yet unknown parameters. The simulation may yield the observable $z^{\left(x\right)}$ at different sites x in the domain; however, x could also denote the time-instance in non-static problems. Clearly, the parameters of the surrogate model will also depend on those positions. Thus, we actually have

$$\begin{array}{cc}\hfill z^{\left(x\right)}\approx z_{\mathrm{sur}}^{\left(x\right)}& =g\left(\mathit{a}\right|{\mathit{c}}^{\left(x\right)}).\hfill \end{array}$$

(3)

The unknown parameters will be inferred from a suitable training data. To this end, simulations are performed for a finite set of model parameters ${\mathit{A}}_s={\left\{{\mathit{a}}_s^{\left(i\right)}\right\}}_{i=1}^{N_s}$ and the corresponding observables ${\mathit{Z}}_s={\left\{z_s^{\left(x\right),\left(i\right)}\right\}}_{i,x=1}^{N_s,N_x}$ are computed and combined in ${\mathit{D}}_{\mathrm{sim}}=\{{\mathit{A}}_s,{\mathit{Z}}_s\}$. The surrogate parameters ${\mathit{c}}^{\left(x\right)}$ are then inferred from ${\mathit{D}}_{\mathrm{sim}}$, and the surrogate is thus constructed. We now proceed to substitute the simulation for the surrogate, $z^{\left(x\right)}\to z_{\mathrm{sur}}^{\left(x\right)}$, in order to solve Equation (2) at a significantly reduced computational cost. This implies that the background information has changed. We will denote this as $\tilde{\mathcal{I}}$, suggesting that we take the observable z entering the integral in Equation (2) from the surrogate model Equation (3) rather than from the expensive simulation. More precisely, instead of Equation (2), we now need to consider

$$\begin{array}{cc}\hfill p\left(z^{\left(x\right)}\mid {\mathit{d}}_{\exp },{\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}}\right)& =\int d{V}_{\mathit{a}}p\left(z^{\left(x\right)}\mid \mathit{a},{\mathit{D}}_{\mathrm{sim}},\cancel{{\mathit{d}}_{\exp }},\tilde{\mathcal{I}}\right)p\left(\mathit{a}\mid \cancel{{\mathit{D}}_{\mathrm{sim}}},{\mathit{d}}_{\exp },\tilde{\mathcal{I}}\right).\hfill \end{array}$$

(4)

As far as the (second) pdf for the model parameters is concerned, we can omit the information on the training set ${\mathit{D}}_{\mathrm{sim}}$, as it does not tell us anything about the model parameters. This pdf is actually the same as that in Equation (1), i.e., $p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\tilde{\mathcal{I}}\right)=p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)$, as it makes no difference for the model parameters how we solve the equations underlying the simulation. In the first pdf, we can omit the information on the experiment ${\mathit{d}}_{\exp }$, as we only need the simulation data ${\mathit{D}}_{\mathrm{sim}}$ to fix the surrogate model, which in turn defines the observable z. The first pdf can be further specified by the marginalization rule upon introducing the surrogate parameters $\mathit{C}={\left\{{\mathit{c}}^{\left(x\right)}\right\}}_{x=1}^{N_x}$, where x is an index denoting the measurement points as introduced for $z^{\left(x\right)}$,

$$\begin{array}{cc}\hfill p\left(z^{\left(x\right)}\mid \mathit{a},{\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}}\right)& =\int d{V}_{\mathit{C}}p\left(z^{\left(x\right)}\mid \mathit{C},\mathit{a},\cancel{{\mathit{D}}_{\mathrm{sim}}},\tilde{\mathcal{I}}\right)p\left(\mathit{C}\mid \cancel{\mathit{a}},{\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}}\right).\hfill \end{array}$$

The first pdf is uniquely fixed by the knowledge of $\mathit{a}$ and $\mathit{C}$, hence ${\mathit{D}}_{\mathrm{sim}}$ is superfluous. Similarly, in the second pdf, where $\mathit{C}$ is inferred from the training data, additional model parameters without the corresponding observables’ values z, are useless. In summary, substituting the latter equation into Equation (4), we have

$$\begin{array}{cc}\hfill p\left(z^{\left(x\right)}\mid {\mathit{d}}_{\exp },{\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}}\right)& =\int \int d{V}_{\mathit{a}}d{V}_{\mathit{C}}p\left(z^{\left(x\right)}\mid \mathit{C},\mathit{a},\tilde{\mathcal{I}}\right)p\left(\mathit{C}\mid {\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}}\right)p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right).\hfill \end{array}$$

(5)

The first pdf is rather simple. According to the background information $\tilde{\mathcal{I}}$, we will determine the observable via the surrogate model. Since the necessary parameters ${\mathit{c}}^{\left(x\right)}\in \mathit{C}$ and $\mathit{a}$ are part of the conditional complex, the surrogate model allows only one value

$$\begin{array}{cc}\hfill z_{\mathrm{sur}}^{\left(x\right)}& =g(\mathit{a}\mid {\mathit{c}}^{\left(x\right)}).\hfill \end{array}$$

for the observable. This means that $p\left({\mathit{z}}^{\left(x\right)}\mid \mathit{C},\mathit{a},\tilde{\mathcal{I}}\right)$ is equivalent to the probability density function for ${\mathit{z}}^{\left(x\right)}$ given ${\mathit{z}}^{\left(x\right)}=g(\mathit{a}\mid {\mathit{c}}^{\left(x\right)})$. Hence, the pdf is a Dirac-delta distribution

$$\begin{array}{cc}\hfill p\left(z^{\left(x\right)}\mid \mathit{C},\mathit{a},\tilde{\mathcal{I}}\right)& =\delta \left[z^{\left(x\right)}-g(\mathit{a}\mid {\mathit{c}}^{\left(x\right)})\right].\hfill \end{array}$$

(6)

Finally, we have

$$\begin{array}{cc}\hfill p\left(z^{\left(x\right)}\mid {\mathit{d}}_{\exp },{\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}}\right)& =\int \int d{V}_{\mathit{a}}d{V}_{\mathit{C}}\delta \left[z^{\left(x\right)}-g(\mathit{a}\mid {\mathit{c}}^{\left(x\right)})\right]p\left(\mathit{C}\mid {\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}}\right)p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right).\hfill \end{array}$$

(7)

Before we can evaluate this integral, we first need to determine the two terms, which have their own independent significance. The last term is the result of a data analysis of a specific foregoing experiment, and will therefore not be treated here. We will suppress the background information in the following, as ambiguities should no longer occur.

2.2. Bayesian Analysis and Selection of the Surrogate Model

We recall that Equation (7) allows for determining the pdf for the observable based on the pdf for the model parameters, and the pdf for the parameters of the surrogate model $p\left(\mathit{C}\mid {\mathit{D}}_{\mathrm{sim}}\right)$ that we will determine now. To this end, we have to specify the form of the surrogate model. We use the expansion

$$\begin{array}{c}\hfill z_{\mathrm{sur}}=\sum _{\nu =1}^{N_p}{c}_{\nu }{\Phi }_{\nu }\left(\mathit{a}\right),\end{array}$$

(8)

in terms of basis functions ${\Phi }_{\nu }\left(\mathit{a}\right)$ and expansion coefficients $c_{\nu }$. No further specification is needed at this point. Without loss of generality, we will use multi-variate Legendre polynomials for the numerical examples. This expansion is similar to the frequently used generalized Polynomial Chaos Expansfion [1], where the polynomials ${\Phi }_{\nu }\left(\mathit{a}\right)$ are orthogonal with respect to the $L^2$ inner product with the prior of $\mathit{a}$, $p\left(\mathit{a}\right)$, as an integration measure. However, here we actually consider a posterior $p\left(\mathit{a}\mid {\mathit{d}}_{exp}\right)$ that generally has no standard form, for which no standard orthogonal polynomial chaos basis is known, and for which conditional independence of the model parameters $\mathit{a}$ does not hold. Polynomial Chaos Expansions have been extended to arbitrary probability measures [17] and dependent parameters [18], but, in the present context; however, these polynomials are not of primary interest and would only complicate the numerical evaluation. Note that the approach presented here does not demand any orthogonality properties for the basis, and thus avoid the practical problems encountered with the construction of such orthogonal bases. Parameters here may have complex dependence structures, and the only requirement for the probability distribution $p\left(\mathit{a}\mid {\mathit{d}}_{\exp }\right)$ is that the integrals with respect to it exist. As outlined in Section 2.1, $N_s$ simulations are performed for a set of model parameters ${\mathit{A}}_s={\left\{{\mathit{a}}_s^{\left(i\right)}\right\}}_{i=1}^{N_s}$ and the corresponding observables ${\mathit{Z}}_s$ are computed. The theory is so far agnostic to the experimental design of these simulations, and it is therefore not of concern here. Now, we want to determine the pdf for the surrogate parameters $\mathit{C}$, which are combined in a matrix with elements $C_{\nu ,x}$, where $\nu$ enumerates the surrogate basis functions and x enumerates the measurement positions in the domain for which the observables are computed. We abbreviated the simulation data by the quantity ${\mathit{D}}_{\mathrm{sim}}=\{{\mathit{A}}_s,{\mathit{Z}}_s\}$, where the matrix ${\mathit{Z}}_s$ has the elements ${\left({\mathit{Z}}_s\right)}_{i,x}$, which represent the observable $z^{\left(x\right)}$ at position x corresponding to the model parameter vector ${\mathit{a}}_s^{\left(i\right)}$. The sought-for pdf follows from Bayes’ theorem

$$\begin{array}{cc}\hfill p\left(\mathit{C}\mid {\mathit{D}}_{\mathrm{sim}}\right)& =p\left(\mathit{C}\mid {\mathit{Z}}_s,{\mathit{A}}_s\right)\propto p\left({\mathit{Z}}_s\mid \mathit{C},{\mathit{A}}_s\right)p\left(\mathit{C}\mid {\mathit{A}}_s\right)\propto p\left({\mathit{Z}}_s\mid \mathit{C},{\mathit{A}}_s\right).\hfill \end{array}$$

The proportionality constant is not required in the ensuing considerations, and we have assumed an ignorant, uniform prior for the coefficients $\mathit{c}$, i.e., $p\left(\mathit{C}\mid \mathcal{I}\right)=const$. We note that this is also the transformation invariant Riemann prior (see Appendix B). However, any prior that is conjugate to the likelihood will retain analytical tractability. For the likelihood, we need the total misfit, which is given by

$$\begin{array}{cc}\hfill {\chi }^2& =\sum _{i=1}^{N_s}\sum _{x=1}^{N_x}{\left({\left({\mathit{Z}}_s\right)}_{i,x}-\sum _{\nu =1}^{N_p}{\left(M_s\right)}_{i,\nu }{\left(\mathit{C}\right)}_{\nu ,x}\right)}^2=\sum _{i,x}{\left({\mathit{Z}}_s-M_s\mathit{C}\right)}_{i,x}^2\hfill \\ \hfill & =\mathrm{tr}\left\{{\left({\mathit{Z}}_s-M_s\mathit{C}\right)}^T\left({\mathit{Z}}_s-M_s\mathit{C}\right)\right\},\hfill \end{array}$$

(9)

with ${\left(M_s\right)}_{i,\nu }={\Phi }_{\nu }\left({\mathit{a}}_s^{\left(i\right)}\right)$ and $N_{sx}=N_s·N_x$. We assume a Gaussian type of likelihood, i.e.,

$$\begin{array}{cc}\hfill p\left({\mathit{Z}}_s\mid \mathit{C},{\mathit{A}}_s,\Delta \right)& =\frac{{\Delta }^{-N_{sx}}}{Z^{\prime }}exp\left\{-\frac{{\chi }^2}{2{\Delta }^2}\right\}.\hfill \end{array}$$

(10)

with normalization $Z^{\prime }$. We have mentioned the $\Delta$-dependence of the normalization explicitly, while the rest of of the normalization is irrelevant in the present context. Usually, the misfit entering the likelihood comes from the noise of the data. In the present case, however, there is no noise (merely a tiny numerical error), but the surrogate model is presumably not an exact description of the simulation data and $\Delta$ covers the corresponding uncertainty. However, the uncertainty level $\Delta$ is not known and has to be marginalized over. Along with the appropriate Jeffreys’ prior, $p\left(\mathit{C},\Delta \right)=p\left(\mathit{C}\right)p\left(\Delta \right)$, $p\left(\Delta \right)=\frac{1}{\Delta },p\left(\mathit{C}\right)=const.$ (see Appendix B), the integration over $\Delta$ yields

$$\begin{array}{cc}\hfill p\left(\mathit{C}\mid {\mathit{Z}}_s,{\mathit{A}}_s\right)& =\frac{1}{Z}{\left({\chi }^2\right)}^{-\frac{N_{sx}}{2}}.\hfill \end{array}$$

(11)

with terms independent of $\mathit{C}$ subsumed in the normalization Z. For computing the mean, variance, and evidence, we first complete the square in Equation (9) to get a quadratic form in $\mathit{C}$, which can then be integrated analytically (see Appendix A). The result is

$$\begin{array}{cccc}\hfill {〈\mathit{C}〉}_{\mathit{a}}& =H_s^{-1}{M}_s^T{\mathit{Z}}_s,\hfill & \hfill H_s& =M_s^T{M}_s,\hfill \end{array}$$

(12a)

$$\begin{array}{cccc}\hfill {〈\Delta C_{\nu x}\Delta C_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a}}& =\frac{{\chi }_{\min }^2}{(N_s-N_p)N_x-2}{\left(H_s^{-1}\right)}_{\nu ,{\nu }^{\prime }}{\delta }_{x{x}^{\prime }},\hfill & \hfill {\chi }_{\min }^2& =\mathrm{tr}\left\{{\mathit{Z}}_s^T\left(11-M_s{H}_s^{-1}{M}_s^T\right){\mathit{Z}}_s\right\}.\hfill \end{array}$$

(12b)

We argue that the prefactor of $H_s^{-1}$ is the Bayesian estimate for ${\Delta }^2$, the variance of the Gaussian in Equation (10). This reasoning is similar to [19]. Note that ${\mathit{Z}}_s$ is a matrix of size $N_s\times N_x$, containing the data vectors of length $N_s$ for each measurement site x. As shown in Appendix A, Equation (A6), the evidence for a particular set of surrogate models is computed as

$$\begin{array}{c}\hfill p\left({\left\{z_{\mathrm{sur}}^{\left(x\right)}\right\}}_{x=1}^{N_x}\mid {\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}}\right)=Z={\Omega }_{N_{px}}{\left|H_s\right|}^{-\frac{1}{2}}{\left({\chi }_{\min }^2\right)}^{-\frac{N_{sx}-N_{px}}{2}}\frac{\Gamma \left(\frac{N_{px}}{2}\right)\Gamma \left(\frac{N_{sx}-N_{px}}{2}\right)}{\Gamma \left(\frac{N_{sx}}{2}\right)},\end{array}$$

(13)

where ${\Omega }_{N_{px}}$ is the solid angle in $N_{px}$ dimensions. The evidence is the probability for a surrogate model given the data. Note that this quantity does not depend on $p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)$. This is reasonable because the analysis of the experimental data should be independent of the analysis of the simulation data. However, $p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)$ will typically be used for the experimental design of the simulation data acquisition. By comparing the evidence for different models, the user can choose a particular surrogate model or, if the results do not overwhelmingly suggest one single model, average the results for the surrogate analysis and the following uncertainty propagation over several plausible models. Note that the evidence is the pillar of a Bayesian procedure to select a surrogate model, and is distinct from the procedure of incorporating the trustworthiness or uncertainty of the surrogate in the subsequent uncertainty propagation.

2.3. Bayesian Uncertainty Propagation with Surrogate Models

Now that we have selected the surrogate model and determined the ingredients of Equation (7), we can determine the pdf for the observables in the light of the experimental data and the simulation results of the training set. The form in Equation (5) allows an easy evaluation of the mean value by using Equation (12) (see also Equation (A3)),

$$\begin{array}{cc}\hfill 〈z^{\left(x\right)}〉& =\int \int d{V}_{\mathit{a}}d{V}_{\mathit{C}}f(\mathit{a}\mid {\mathit{c}}^{\left(x\right)})p\left(\mathit{C}\mid {\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}}\right)p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)\hfill \\ \hfill & =\sum _{\nu }\int d{V}_{\mathit{a}}{\Phi }_{\nu }\left(\mathit{a}\right){〈C_{\nu x}〉}_{\mathit{a}}p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right).\hfill \end{array}$$

(14)

Similarly, we obtain (see Equations (12) and (A8))

$$\begin{array}{cc}\hfill 〈z^{\left(x\right)}{z}^{\left(x^{\prime }\right)}〉& =\int \int d{V}_{\mathit{a}}d{V}_{\mathit{C}}f(\mathit{a}\mid {\mathit{c}}^{\left(x\right)})f(\mathit{a}\mid {\mathit{c}}^{\left(x^{\prime }\right)})p\left(\mathit{C}\mid {\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}}\right)p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)\hfill \\ \hfill & =\sum _{\nu {\nu }^{\prime }}\int d{V}_{\mathit{a}}{\Phi }_{\nu }\left(\mathit{a}\right){\Phi }_{{\nu }^{\prime }}\left(\mathit{a}\right){〈C_{\nu x}{C}_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a}}p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)\hfill \\ \hfill & =\sum _{\nu {\nu }^{\prime }}\int d{V}_{\mathit{a}}{\Phi }_{\nu }\left(\mathit{a}\right){\Phi }_{{\nu }^{\prime }}\left(\mathit{a}\right)\left({〈C_{\nu x}〉}_{\mathit{a}}{〈C_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a}}+{〈\Delta C_{\nu x}\Delta C_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a}}\right)p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right).\hfill \end{array}$$

(15)

The covariance then follows from

$$\begin{array}{cc}\hfill 〈\Delta z^{\left(x\right)}\Delta z^{\left(x^{\prime }\right)}〉& =〈z^{\left(x\right)}{z}^{\left(x^{\prime }\right)}〉-〈z^{\left(x\right)}〉〈z^{\left(x^{\prime }\right)}〉.\hfill \end{array}$$

(16)

If we neglected the uncertainty of the surrogate, i.e.,

$$\begin{array}{cc}\hfill p\left(\mathit{C}\mid D_{sim}\right)& =\delta (\mathit{C}-\widehat{\mathit{C}}),\hfill \\ \hfill \widehat{\mathit{C}}& ={〈\mathit{C}〉}_{\mathit{a}},\hfill \end{array}$$

then we retain the widely known special case of ’perfectly trustworthy’ surrogates

$$\begin{array}{cc}\hfill 〈z^{\left(x\right)}{z}^{\left(x^{\prime }\right)}〉& =\sum _{\nu {\nu }^{\prime }}\int d{V}_{\mathit{a}}{\Phi }_{\nu }\left(\mathit{a}\right){\Phi }_{{\nu }^{\prime }}\left(\mathit{a}\right){〈C_{\nu x}〉}_{\mathit{a}}{〈C_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a}}p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right).\hfill \end{array}$$

Thus, the first part in the integral of Equation (15) is the uncertainty of the observable due to experimental uncertainties and given the surrogate model, while the second term adds the uncertainty of the surrogate itself. The term ${〈\Delta C_{\nu x}\Delta C_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a}}$ is commonly neglected, but easily computed. This result also suggests a natural measure for the trustworthiness of the surrogate model, which is directly linked to the specific experiment:

$$\begin{array}{c}\hfill \frac{{\sum }_{\nu {\nu }^{\prime }}\int d{V}_{\mathit{a}}{\Phi }_{\nu }\left(\mathit{a}\right){\Phi }_{{\nu }^{\prime }}\left(\mathit{a}\right){〈\Delta C_{\nu x}\Delta C_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a}}p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)}{{\sum }_{\nu {\nu }^{\prime }}\int d{V}_{\mathit{a}}{\Phi }_{\nu }\left(\mathit{a}\right){\Phi }_{{\nu }^{\prime }}\left(\mathit{a}\right){〈C_{\nu x}〉}_{\mathit{a}}{〈C_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a}}p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)}<ϵ.\end{array}$$

(17)

If the surrogate uncertainties are, on average, smaller than the experimental uncertainties by a few orders of magnitude, e.g., $ϵ={10}^{-3}$, then they may be neglected. However, $ϵ$ is the user’s choice. Note that this result does not spare the user to solve the foregoing surrogate model selection problem by e.g., computing evidence. This work only demonstrates how surrogate uncertainties can be incorporated and a practical rule when they could be neglected, given that the surrogate model has already been selected before.

3. Numerical Example

Here, we demonstrate an application where surrogate uncertainties were in part negligible and in part non-negligible. We apply our method to a computational fluid dynamics simulation of aortic hemodynamics, i.e., blood flow in an aorta resembled by the simplified geometry of an upside down umbrella stick. The simulation depends on a non-Newtonian viscosity model with four parameters $\mathit{a}=\{a_1,a_2,a_3,a_4\}$. The model was accompanied by viscosity measurements of human blood samples, thus determining $p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)$. This posterior turned out to have a complex landscape that cannot be reasonably approximated by standard distributions. Particularly, strong correlations and multi-modality were observed, i.e., $p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)\ne {\prod }_{i}p\left(a_i\mid {\mathit{d}}_{\exp },\mathcal{I}\right)$. This means that vanilla Polynomial Chaos Expansions could not be applied without an undesirable transformation to conditionally independent variables. The posterior is described in detail in [20]. Based on $p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)$, $N_s=100$ parameter samples ${\mathit{a}}_s$ were chosen and the simulation evaluated accordingly. The output, ${\mathit{Z}}_s$, was the absolute values of the wall shear stress that the blood flow exerts on the aortic wall, for $N_x=10$ measurement probes at different locations, each for $N_t=101$ time-instances equidistantly spaced over one cardiac cycle (ca. 1 s). Further details on the simulation are not relevant here but are documented [20]. A simulation time on the order of 150 CPU hours per sample suggested to use a surrogate for the inference. For the surrogate’s basis functions, ${\Phi }_{\nu }\left(\mathit{a}\right)$, we found multi-variate Legendre polynomials of up to order two sufficient. The numerical integrals were computed with Riemannian quadrature and convergence checked with successive grid refinement; however, stochastic integration would work just as well. A sketch example on how to implement this procedure computationally efficient via vectorisation in parameter space can be found at https://github.com/Sranf/BayesianSurrogate_sketch.git (1 January 2021).

In Figure 1, we compare the simulation uncertainty (including surrogate uncertainty) as computed with our Bayesian approach (Equation (15)) to the naive estimate for the simulation uncertainty (without surrogate uncertainty, i.e., neglecting ${〈\Delta C_{\nu x}\Delta C_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a}}$ in Equation (15). The surrogate uncertainties in the first half (left hand side) are relatively small, comprising only a few percent of the total uncertainty, and could possibly be neglected. In the second half (right-hand side), however, the surrogate uncertainties make up to ∼50% of the total uncertainty. This demonstrates that simulation uncertainties inferred via surrogate models can be severely underestimated if the surrogate uncertainties are neglected, and subsequently lead to overconfidence in the simulation model. In practice, one would acquire more data in order to reduce the surrogate uncertainties, e.g., more data at later time-instances in Figure 1 are particularly promising. This was limited here not only by the computational budget, but also the impracticality in that dynamic simulations require the full evaluation of all previous time instances where the surrogate is already reasonably accurate. Thus, the procedure of instead explicitly including the surrogate uncertainties here also has proven to be practical. A similar situation is to be expected for most transient simulations, as uncertainties will usually increase as time progresses.

4. Discussion

In this work, we have assumed a Gaussian likelihood for the simulation data, with unknown variance, for a surrogate that is linear in its parameters. Surrogates that are nonlinear in its parameters (e.g., neural networks) may promise higher capacity, however at the expense of losing analytical tractability of the surrogate uncertainty entirely. Other likelihood functions might be useful if further information is available, such as bounds on the observable (Gamma- or Beta-likelihood).

The result is a simple formula to incorporate surrogate uncertainties in the simulation uncertainties. This formula will be particularly useful if ’convergence’ in the sense of finding the coefficients of e.g., a Polynomial Chaos Expansion is doubtful or not achievable due to a limit to the computational budget. The formula immediately suggests an intrinsic measure for the trustworthiness of the surrogate, distinct from commonly used ad hoc diagnostics. This measure is not to be confused with the evidence and should not be used for model selection because it would not preclude over-fitting, etc. It is merely a measure for the trustworthiness of the already selected surrogate.

Let us now explore the connections of this work to Polynomial Chaos Expansions (PCE) and Gaussian Process Regression (GPR). PCE is a special case of our generalized linear surrogate model, in that the basis functions of the surrogate are chosen such that

$$\begin{array}{c}\hfill \int {\Phi }_{\nu }\left(\mathit{a}\right){\Phi }_{{\nu }^{\prime }}\left(\mathit{a}\right)p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)d{V}_{\mathit{a}}:={\delta }_{\nu ,{\nu }^{\prime }}.\end{array}$$

(18)

The double sum in Equation (15) then contracts to a single sum, and the diagonal of the term for the surrogate uncertainty, ${〈\Delta C_{\nu x}\Delta C_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a}}$, survives. This is expected, in that PCE is defined such that the basis functions are uncorrelated, but still the expansion coefficients must be uncertain to a finite degree, and this must carry over to the simulation uncertainty. A severe limitation of PCE is that it is rather difficult to find basis function sets $\left\{{\Phi }_{\nu }\right\}$ that fulfill Equation (18), depending on $p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)$. For most practical purposes, one demands (i) conditional independence of the simulation parameters, i.e., $p\left(\mathit{a}\mid {\mathit{d}}_{\exp },\mathcal{I}\right)={\prod }_{i}p\left(a_i\mid {\mathit{d}}_{\exp },\mathcal{I}\right)$, as well as (ii) simple standard distributions for $p\left(a_i\mid {\mathit{d}}_{\exp },\mathcal{I}\right)$, in order to find a solution (usually a tensor-product) to Equation (18). Known albeit tedious work-arounds are for (i) variable transformations and numerical orthonormalisation [18] and for (ii) PCE-constructions for arbitrary pdfs [17]. Note that also [17] demands (i) conditional independence of the simulation parameters. Note that our approach is not afflicted by above considerations, unless Equation (18) is specifically demanded. In the numerical example above, neither (i) nor (ii) were applicable. Finding a variable transformation in order to fulfill (i) or numerical construction of orthonormal basis functions can be difficult, and particularly inconvenient if sophisticated priors $p\left(\mathit{a}\mid \mathcal{I}\right)$ are being used, e.g., Jeffreys’ generalized prior. An interesting alternative would be to model the input dependencies with vine copulas [21] in order to overcome the limitations of PCE addressed here. Unfortunately, no obvious vine copula was found for the example presented here.

Gaussian Process Regression would correspond to a change in the prior for $z^{\left(x\right)}$ in Equation (6) as follows:

$$\begin{array}{cc}\hfill p\left(z^{\left(x\right)}\mid \mathit{C},\mathit{a},\theta ,\tilde{\mathcal{I}}\right)& =\mathcal{N}(g(\mathit{a}\mid {\mathit{c}}^{\left(x\right)})\left|K\left(\mathit{\theta }\right)\right).\hfill \end{array}$$

(19)

where $\mathcal{N}$ denotes a normal distribution, and K is the prior’s covariance matrix and defined by the parametrized covariance function k, ${\left[K\right]}_{ij}=k({\mathit{a}}^{\left(i\right)},{\mathit{a}}^{\left(j\right)}\mid \mathit{\theta })$. This in turn would change ${({\mathit{Z}}_s-M_s\mathit{C})}^T({\mathit{Z}}_s-M_s\mathit{C})\to {({\mathit{Z}}_s-M_s\mathit{C})}^T{K}^{-1}({\mathit{Z}}_s-M_s\mathit{C})$ in Equation (9). By again completing the square and following the same procedure, the corresponding results for mean Equation (12a), variance Equation (12b), and evidence Equation (13) are then retained by a simple substitution of

$$\begin{array}{cccc}\hfill H_s& \to {\tilde{H}}_s,\hfill & \hfill {\tilde{H}}_s& =M_s^T{K}_s^{-1}{M}_s,\hfill \\ \hfill {\chi }_{min}^2& \to {\tilde{\chi }}_{min}^2,\hfill & \hfill {\tilde{\chi }}_{min}^2& =\mathrm{tr}\left\{{\mathit{Z}}_s^T(K_s^{-1}-K_s^{-1}{M}_s{\tilde{H}}_s{M}_s^T{K}_s^{-1}){\mathit{Z}}_s\right\},\hfill \end{array}$$

where ${\left[K_s\right]}_{ij}=k({\mathit{a}}_s^{\left(i\right)},{\mathit{a}}_s^{\left(j\right)}\mid \mathit{\theta })$ is the likelihood’s covariance matrix evaluated at ${\mathit{A}}_s$ for the data set ${\mathit{Z}}_s$ at given $\mathit{\theta }$. Equations (14) and (15) would preserve their form with the substitution

$$\begin{array}{cc}\hfill {\Phi }_{\nu }\left(\mathit{a}\right){〈\mathit{C}〉}_{\mathit{a}}& \to {\Phi }_{\nu }\left(\mathit{a}\right){〈\mathit{C}〉}_{\mathit{a},\mathit{\theta }}+K_{*}^T{K}_s^{-1}{M}_s{〈\mathit{C}〉}_{\mathit{a},\mathit{\theta }},\hfill \\ \hfill {〈\Delta C_{\nu x}\Delta C_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a}}& \to {〈\Delta C_{\nu x}\Delta C_{{\nu }^{\prime }{x}^{\prime }}〉}_{\mathit{a},\mathit{\theta }}\left(K-K_{*}^T{K}_s^{-1}{K}_{*}\right),\hfill \end{array}$$

where the subscript $\theta$ acknowledges that the right-hand side now depends on $\mathit{\theta }$, and ${\left[K_{*}\right]}_{ij}=k({\mathit{a}}^{\left(i\right)},{\mathit{a}}_s^{\left(j\right)}\mid \mathit{\theta })$ is the covariance between the training set ${\mathit{A}}_s$ and the ’test set’, i.e., the integration variable $\mathit{a}$. Note that the additionally introduced hyperparameters $\mathit{\theta }$ would require the choice of a prior for $\mathit{\theta }$ and marginalization wrt $\mathit{\theta }$ in Equation (7), and subsequently also Equations (12)–(15).

We now discuss the implications of $\tilde{\mathcal{I}}$ in contrast to the original background information $\mathcal{I}$. $\mathcal{I}$ contains, most importantly, that the observable z is uniquely determined by the simulation for a given set of input parameters $\mathit{a}$. A prerequisite here was that the simulation is converged. For example, for finite element simulations, this would be a given mesh-converged spatial discretization. The proposition $\tilde{\mathcal{I}}$ additionally assumes Equations (3) and (6), so that it can be used to replace Equation (2) by Equation (4). Formally, this means, to get from Equations (2)–(4), we replace $p(z\mid \mathit{a},\mathcal{I})\to p(z\mid \mathit{a},\mathit{c},\tilde{\mathcal{I}})$, where $p(z\mid \mathit{a},\mathcal{I})=\delta (z-z(\mathit{a}\left)\right)$, $p(z\mid \mathit{a},\mathit{c},\tilde{\mathcal{I}})=\delta (z-z_{sur}\left(\mathit{a}\right))=\delta (z-g(\mathit{a}\mid \mathit{c}))$. For example, $\tilde{\mathcal{I}}$ contains in comparison to $\mathcal{I}$ the additional assumption that we can use the value for z as predicted/approximated by the surrogate model. It also means that we introduce additional, artificial, and usually unknown regression parameters $\mathit{c}$ that need to be marginalized over. The additional uncertainty introduced by this approximation (i.e., the surrogate assumption) is encoded in $p\left(\mathit{c}\right|{\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}})$, and is correctly incorporated into the simulation observable uncertainties in Equation (15). What is important here is that $p(z\mid {\mathit{d}}_{\exp },{\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}})\ne p(z\mid {\mathit{d}}_{\exp },\cancel{{\mathit{D}}_{\mathrm{sim}}},\mathcal{I})$ in general (the latter is computationally infeasible), but $p(z\mid {\mathit{d}}_{\exp },{\mathit{D}}_{\mathrm{sim}},\tilde{\mathcal{I}})\approx p(z\mid {\mathit{d}}_{\exp },{\mathit{D}}_{\mathrm{sim}},\mathcal{I})$ if Equation (3) holds and $p(\mathit{C}\mid {\mathit{D}}_{\mathrm{sim}})\approx \delta (\mathit{C}-\widehat{\mathit{C}})$, i.e., if the surrogate is indeed a good approximation and the posterior for the surrogate parameters is sharply peaked at $\widehat{\mathit{C}}$. Very often, this posterior is not sharply peaked; then, we can just gather more data until it is, or, if that is not possible, we can at least avoid overconfidence induced by neglecting these uncertainties. A numerical example of where this is the case has been demonstrated above.

We have modeled spatial correlations by introducing a location index x, and assumed that the expansion coefficients at different sites, ${\mathit{c}}^{\left(x\right)}$ and ${\mathit{c}}^{\left(x^{\prime }\right)}$, are conditionally independent. This assumption is reasonable, in that the expansion coefficients are arbitrary mathematical constructs and no physically motivated model for their correlation is known. The spatial correlation, however, is retained in $z^{\left(x\right)}$, as was originally intended. More general models for spatial correlations can easily be implemented by substitution of ${\delta }_{x{x}^{\prime }}$ with a spatial covariance matrix in Equation (12b). Note that this would require an additional marginalization wrt the (typically nonlinear) hyperparameters of the spatial covariance matrix. By introducing a compound index $\tilde{x}=(x,t)$ and substituting $x\to \tilde{x}$, we find a simple generalization to spatio-temporal correlations. This is equivalent to re-ordering spatial and temporal indices into a single sequence. While this procedure is convenient and requires only minor changes in the numerical implementation, it implicitly assumes conditional independence of spatial and temporal correlations. Analogous to above, general temporal correlations can be modeled by a substitution of ${\delta }_{\tilde{x}{\tilde{x}}^{\prime }}$ in Equation (12b) with a temporal covariance matrix, again requiring an additional marginalization wrt the latter’s hyperparameters.

5. Conclusions

We presented a Bayesian analysis of surrogate models and its associated uncertainty propagation problem in the context of uncertainty quantification of computer simulations. The assumptions were a generalized linear surrogate model (linear in its parameters, not the variable) and a Gaussian likelihood with unknown variance. Additionally, spatial and temporal correlations have been discussed. The result suggests a measure of trustworthiness of the surrogate by quantifying the ratio of the surrogate uncertainty to the total uncertainty, in contrast to commonly used heuristic diagnostics. The main result, however, is a rather simple rule to include surrogate uncertainties in the sought-for uncertainties of the simulation output. This is useful particularly for problems where the surrogate’s trustworthiness is doubtful and cannot be improved. The connections to Polynomial Chaos Expansions and Gaussian Process Regression have been discussed. A numerical example demonstrated that simulation uncertainties can be significantly underestimated if surrogate uncertainties are neglected.