Expectation-Maximization Algorithm for the Calibration of Complex Simulator Using a Gaussian Process Emulator

Abstract

The approximated non-linear least squares (ALS) tunes or calibrates the computer model by minimizing the squared error between the computer output and real observations by using an emulator such as a Gaussian process (GP) model. A potential defect of the ALS method is that the emulator is constructed once and it is no longer re-built. An iterative method is proposed in this study to address this difficulty. In the proposed method, the tuning parameters of the simulation model are calculated by the conditional expectation (E-step), whereas the GP parameters are updated by the maximum likelihood estimation (M-step). These EM-steps are alternately repeated until convergence by using both computer and experimental data. For comparative purposes, another iterative method (the max-min algorithm) and a likelihood-based method are considered. Five toy models are tested for a comparative analysis of these methods. According to the toy model study, both the variance and bias of the estimates obtained from the proposed EM algorithm are smaller than those from the existing calibration methods. Finally, the application to a nuclear fusion simulator is demonstrated.

1. Introduction

Modern researchers have attempted to develop and use simulation code instead of excessively expensive or infeasible physical experiments in many fields. Advanced computer technology has made it possible to realize very complicated simulations. In most cases, the simulation code contains several unknown parameters (or universal constants). A classic method for determining the universal constants in computer models is non-linear least squares estimation (NLSE). It makes the sum of the squared error between the real observations and the computer responses as minimum. The NLSE, however, will become too computationally expensive or infeasible in terms of time when the computer model is time-consuming to run. In such cases, a statistical emulator can be used to determine the universal constants in the computer model so that the simulator or emulator can represent the real experiments effectively well. This process is known as “code tuning” or “calibration” [1,2,3,4].

Code tuning is defined as the procedure of enhancing the consistency between code responses and experimental data via determining the parameters inside the simulation code [5]. A differentiation can be made between tuning parameters and calibration parameters [3]. However, we use these two terms (calibration and code tuning) interchangeably.

Our work focuses on calibration in a frequentist manner [6,7,8], rather than a Bayesian approach [2,9,10]. A Bayesian calibration with a bias correction was introduced by Kennedy and O’Hagan [2]. Contributions to this topic include the related works of sequential tuning [11], multivariate computer outputs [12], identifiability [13,14], multi-fidelity [9], good designs for calibration [10], GP models for big data [7], and various metamodeling applications [8,15,16].

Calibration has often been performed within a framework, in the literature, in which the code suffers from a systematic bias or discrepancy for any parameter values. This shows a view that the mathematical equations inside the simulator may not perfectly describe the real world [2,12]. This framework may be more realistic, but it is beyond the range of this study. Thus, presentation in this study is focused on a statistical method that does not include the simulation discrepancy [1]. It would be, however, possible to extend our framework when the shape of the bias were available.

Cox et al. [1] studied an approximated NLSE in the form of approximated nonlinear least squares (ALS) for calibration. They utilized the Gaussian process (GP) as an emulator (or a surrogate or metamodel) of the complex simulator. The ALS first builds the GP model from the computer data. Then, it regards the built GP model as if it were true simulator. This surrogation makes the NLSE computationally feasible. We adopt the GP model as an emulator of the complex simulator, in this study. The GP model has been used successfully in constructing a regression model for complex data [17,18,19,20,21] and in analyzing computer experiments [22,23,24,25].

A drawback of the ALS is that the emulator (a GP model) is constructed once from the computer data only and it is not re-modeled after that. To address this defect, Seo et al. [4] considered an iterative modification of the ALS, which they referred it as the “max-min” algorithm. The max-min algorithm improves the calibration accuracy and acquires stable results. In this study, we propose another iterative method based on the expectation-maximization (EM) algorithm. We anticipate that this new method is more effective than existing approaches, including the ALS, max-min algorithm, and likelihood-based calibration. The proposed method is compared with existing calibration methods using five toy-models, in which the true values are known a priori.

This report is arranged as follows: a GP model for simulation experiments is described in Section 2. Existing methods including the ALS, max-min algorithm, and likelihood-based calibration are briefly introduced in Section 3. EM-based calibration is presented in Section 4. In Section 5, a toy model study is described. The application to a nuclear fusion simulator is presented in Section 6. Discussions are outlined in Section 7, followed by a summary in Section 8. Further details including technical specifications are provided in the Appendix. Certain contents of this study are inevitably similar to those of Reference [4,6] because the problem setting of these studies was very similar.

2. GP Model for Computer Experiments

Sacks et al. [22] proposed the adoption of the GP model for the analysis of deterministic computer experiments. The GP model can be expressed as follows, for the response $Y\left(\mathbf{t}\right)$ at site $\mathbf{t}$:

$$Y\left(\mathbf{t}\right)=\sum _{i=1}^p{\beta }_i{f}_i\left(\mathbf{t}\right)+Z\left(\mathbf{t}\right),$$

(1)

where ${\beta }_i$s are unknown regression coefficients, $\left\{f_i(·)\right\}$ is a set of known functions, and $Z(·)$ is the GP (random variables) with mean zero and covariances ${\sigma }_Z^{2}R\left(\mathbf{t}\right)$. In the above equation, the first term indicates a regression function and the second part ($Z\left(\mathbf{t}\right)$) stands for the departure from the assumed regression function. The stochastic process term allows the predictions (5) to interpolate the deterministic responses. For ${\mathbf{t}}_i=\{t_{i1},...t_{id}\}$ and ${\mathbf{t}}_j=\{t_{j1},...t_{jd}\}$, the covariance between $Z\left({\mathbf{t}}_i\right)$ and $Z\left({\mathbf{t}}_j\right)$ is denoted by

$$\mathrm{Cov}(Z\left({\mathbf{t}}_i\right),Z\left({\mathbf{t}}_j\right))=V({\mathbf{t}}_i,{\mathbf{t}}_j)={\sigma }_Z^{2}R({\mathbf{t}}_i,{\mathbf{t}}_j),$$

(2)

where ${\sigma }_Z^2$ is the variance of $Z(·)$ and $R({\mathbf{t}}_i,{\mathbf{t}}_j)$ is a correlation function. We select the Gaussian correlation family [26]:

$$R({\mathbf{t}}_i,{\mathbf{t}}_j)=exp\left(-\sum _{k=1}^d{\theta }_k{(t_{ik}-t_{jk})}^2\right),$$

(3)

where ${\theta }_k^{\prime }$s are non-negative parameters. We define $v^{\prime }\left({\mathbf{t}}_0\right)$ and $f^{\prime }\left({\mathbf{t}}_0\right)$ as

$$v^{\prime }\left({\mathbf{t}}_0\right)=[V({\mathbf{t}}_0,{\mathbf{t}}_1),\dots ,V({\mathbf{t}}_0,{\mathbf{t}}_n)],f^{\prime }\left({\mathbf{t}}_0\right)=[f_1\left({\mathbf{t}}_0\right),\dots ,f_p\left({\mathbf{t}}_0\right)].$$

(4)

In this case, $v^{\prime }\left({\mathbf{t}}_0\right)$ is a correlation vector between the observed (or training) sites and a prediction site ${\mathbf{t}}_0$, and $f^{\prime }\left({\mathbf{t}}_0\right)$ is a design (or functional) vector of ${\mathbf{t}}_0$.

When the correlation function $R(·,·)$ is assigned and the observations y at sites $\mathbf{t}$ are given, the best linear unbiased predictor (BLUP) of $Y\left({\mathbf{t}}_0\right)$ is [26]

$$\widehat{Y}\left({\mathbf{t}}_0\right)=[v^{\prime }\left({\mathbf{t}}_0\right),f^{\prime }\left({\mathbf{t}}_0\right)]{\left[\begin{array}{cc}V& F\\ F^{\prime }& 0\end{array}\right]}^{-1}\left[\begin{array}{c}y\\ 0\end{array}\right]=f^{\prime }\left({\mathbf{t}}_0\right)\widehat{\beta }+v^{\prime }\left({\mathbf{t}}_0\right)V^{-1}(y-F\widehat{\beta }),$$

(5)

where $V=V(\mathbf{t},\mathbf{t})$ is an $n\times n$ covariance metrix, $F={\left[f_j\left({\mathbf{t}}_i\right)\right]}_{1\le i\le n,1\le j\le p}$ is an $n\times p$ design matrix, and $\widehat{\beta }={\left(F^{\prime }{V}^{-1}F\right)}^{-1}{F}^{\prime }{V}^{-1}y$ is the GLSE (generalized least squares estimator) of $\beta$. Furthermore, the hyper-parameters in V are generally unknown, and hence, we usually estimate the hyper-parameters using the MLE (maximum likelihood estimation) from the observations y. These estimates are subsequently plugged into (5), causing (5) to become the so-called empirical BLUP (EBLUP) of $Y\left({\mathbf{t}}_0\right)$ [26]. We used the “DiceKriging” [27] package of the R program. A brief description of the MLE calculation is provided in Appendix A.

We can build the prediction models differently according to the combination of ${\theta }^{\prime }s$ and ${\beta }^{\prime }s$ in (1) and (3). We construct the following two models, in this study:

$$\begin{array}{cc}\hfill & Y\left(x\right)={\beta }_0+{\beta }_1{t}_1+\dots +{\beta }_d{t}_d+Z\left(\mathbf{t}\right)+\epsilon ,\\ & \mathrm{M}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l} 1:\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h} \mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{o}\mathrm{n}\theta ,\mathrm{M}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l} \mathrm{2}:\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h} d \mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{t}{\theta }^{\prime }s.\end{array}$$

(6)

In the above, “common $\theta$” indicates that d number of $\theta$${}^{\prime }s$ are constrained to be a common ${\theta }_c$ so that ${\theta }_1={\theta }_2=\cdots ={\theta }_d:={\theta }_c$. The final error term ($ϵ$) is for the randomness or measurement error in the real experiments. The error term is not applied to the computer responses because only a deterministic computer model is considered in this study. Other models such as the one from variable selection algorithms [28,29,30] are of course usable. We recommend References [26,31] for further details on the GP model.

3. Existing Calibration Methods

3.1. Data Structure and Notations

The experimental data and computer data are subscrived by “E” and “C”, respectively, and the combined (both) data are denoted by “B”. Let $\tau$ be a vector of calibration parameters with the dimension q and $\underline{T}$ represent the input variables of the simulator corresponding to $\tau$. We denote X for the experimental input variables with the dimension p. In addition, let $n_C,n_E$ be the sample sizes and $n_B=n_C+n_E$. More details are presented in Appendix B.

3.2. ALS Method

In this subsection, we briefly describe the ALS method considered in Reference [1]. If the simulation code is computationally costly to run, it is very difficult to optimize numerically a certain quantity from the code directly in terms of time. In this case, the ALS employs the GP model as a surrogate or an emulator of the computer code. The ALS first estimates the GP parameters using the MLE for the simulation data. Thereafter, it regards the built GP model as if it were true simulator. The ALS determines $\widehat{\tau }$ by minimizing the following approximated residual sum of squares (ARSS):

$$ARSS\left(\tau \right)=\sum _{i=1}^{n_E}{\left[y_{E,i}-\widehat{Y}(\tau ,{\mathbf{x}}_{E,i})\right]}^2,$$

(7)

where $\widehat{Y}(\tau ,{\mathbf{x}}_E)$ is the EBLUP of $Y\left({\mathbf{x}}_0\right)$, as in (5).

The advantage of the ALS method is that it does not need additional runs of the simulator to calculate $ARSS\left(\tau \right)$ after a GP surrogate has been constructed from a computer dataset. Because there is no explicit minimizer in $ARSS\left(\tau \right)$, we employ the quasi-Newton iteration in the “optim” package of the R program.

A potential disadvantage of the ALS method is that the emulator (a GP model) is constructed once from the simulation data and it is no longer re-built. To address this defect, an iterative method of the ALS, namely the max-min algorithm, was considered by Seo et al. Reference [4].

3.3. Max-Min Algorithm

The tuning constants and GP parameters are alternately estimated by ALS method and the MLE, in the max-min algorithm. This method utilizes both experimental and computer data repeatedly until convergence. The steps are outlined as follows:

• Step 1: Find the MLE of the GP parameters ${\widehat{\theta }}^{\prime }s$, ${\widehat{\beta }}^{\prime }s$, and ${\widehat{\sigma }}_Z^2$ in Equations (2) and (6) using the simulation data ($T_S=({\tau }_S,{\mathbf{x}}_S)$ and $y_S$) only.
• Step 2: Determine $\widehat{\tau }$, which minimizes the $ARSS\left(\tau \right)$ in (7) with the estimates ${\widehat{\theta }}^{\prime }s$, ${\widehat{\beta }}^{\prime }s$, and ${\widehat{\sigma }}_Z^2$ from Step 1.
• Step 3: [maximization] Find the MLE of the GP parameters ${\widehat{\theta }}^{\prime }s$, ${\widehat{\beta }}^{\prime }s$, and ${\widehat{\sigma }}_Z^2$ using the combined data ($T_B$ and $y_B$), where $T_B=\left(\begin{array}{c}{T}_E^{*}\\ T_S\end{array}\right),y_B=\left(\begin{array}{c}{y}_E\\ y_S\end{array}\right),T_E^{*}={\{t_{E,1}^{*},...,t_{E,n_E}^{*}\}}^{\prime }$, and ${t_{E,i}^{*}}^{\prime }=(\widehat{\tau },x_{E,i})$.
• Step 4: [minimization] Determine $\widehat{\tau }$, which minimizes the $ARSS\left(\tau \right)$ in (7) with the estimates ${\widehat{\theta }}^{\prime }s$, ${\widehat{\beta }}^{\prime }s$, and ${\widehat{\sigma }}_Z^2$ from Step 3.
• Step 5: Repeat Step 3 and Step 4 until convergence, such as ${\sum }_{i=1}^q|{\widehat{\tau }}_i^{old}-{\widehat{\tau }}_i^{new}|/|{\widehat{\tau }}_i^{old}|<ϵ$.

In Steps 1 and 3, a GP model is constructed for the prediction. Steps 2 and 4 are the same in minimizing $ARSS\left(\tau \right)$, but Step 2 utilizes computer data only in the prediction $\widehat{Y}\left(x_E\right)$, whereas Step 4 uses both data. In Step 3, $\widehat{\tau }$ is the estimate obtained from the previous step. Steps 1 and 3 are the same in terms of seaching the MLE, but Step 1 utilizes computer data only, whereas Step 3 utilizes both data. Quasi-Newton numerical algorithms are employed for the optimizations in Steps 2–4.

Seo et al. [4] demonstrated the max-min method works better than the ALS. One defect of this method is that it needs more computing time than the ALS. Further details on this algorithm, including the stopping rule, can be found in [4].

3.4. Likelihood-Based Calibration

Cox et al. [1] considered the full likelihood function using the combined data for all parameters, including the calibration parameters $\tau$; variance parameter ${\sigma }_{ϵE}^2$ and GP parameters $\theta$, $\underline{\beta }$, and ${\sigma }^2$. Thus, all parameters are estimated simultaneously by the MLE method. This method is called as the full MLE, and it is applied to a GP model. The minus two times profile log likelihood function of all parameters is, without constants,

$$-2logL(\tau ,{\psi }_B;{\underline{y}}_B,X_B)=n_{B}log\widehat{{\sigma }_B^2}+log\left|V_B\right|,$$

(8)

where

$$\widehat{{\sigma }_B^2}={({\underline{y}}_B-F_B{\underline{\widehat{\beta }}}_B)}^t{V}_B^{-1}({\underline{y}}_B-F_B{\underline{\widehat{\beta }}}_B)/n_B,$$

(9)

$${\underline{\widehat{\beta }}}_B={\left({F_B}^t{V}_B^{-1}{F}_B\right)}^{-1}{F_B}^t{V}_B^{-1}{\underline{y}}_B,$$

(10)

in which ${\psi }_B=({\underline{\theta }}_B,{\underline{\beta }}_B,{\sigma }_B^2,{\gamma }_E)$ with ${\underline{\widehat{\beta }}}_B$ and $\widehat{{\sigma }_B^2}$ inserted, and ${\gamma }_E={\sigma }_{ϵE}^2/{\sigma }^2$.

Cox et al. [1] considered other likelihood-based approaches. One method is the so-called separated MLE (SMLE). This maximizes the conditional likelihood of the experimental data given the simulation data. In this case, the GP parameters $\underline{\theta }$, $\underline{\beta }$, and ${\sigma }^2$ are determined by the marginal MLE from the computer data. These estimates are then inserted into in obtaining the conditional MLE. This result is subsequently maximized with respect to $\tau$ and ${\gamma }_E$, to acquire estimates of the above parameters. Advantages of likelihood-based methods for calibration are that they simultaneously or conditionally utilize the combined data, and so enrich the calibration methods. The SMLE was demonstrated to be superior to the full MLE in Reference [1]. Thus, in this study, we compare the SMLE with the proposed method. Further details of the SMLE are presented in Appendix C.

4. Proposed Method: EM-Based Calibration

An EM algorithm is an iterative method for determining the MLE or maximum a posteriori (MAP) estimates of parameters in statistical models [32], where the model usually depends on the unobserved latent variables. One example of the unobserved latent variables is the calibration parameters in computer experiments. The EM iteration alternates between an expectation (E) step and a maximization (M) step. The E-step calculates the expected value of the log likelihood at the current parameter estimates provided from the M-step. The M-step computes the parameters by maximizing the expected log likelihood determined in the E-step. These parameter estimates are subsequently employed to determine the distribution of the latent variables (or calibration parameters) in the following E-step [32].

The EM algorithm was described and given its name in 1977 in a paper by Dempster et al. [33]. Since then, the EM algorithm has been applied to many research areas, including computational statistics, machine learning, computer vision, hidden Markov models, item response theory, and computed tomography. See References [34,35,36] for further information on the EM algorithm and its applications.

To date, the EM algorithm has not been applied to the calibration problem of complex computer code. Because the tuning parameters in real experiments can be treated as unobserved latent variables, an EM algorithm may be appropriate to obtain the distribution of the tuning parameters. The steps of the proposed method for a given GP model are presented as follows:

• Initialization: Provide initial values ($\widehat{\tau }$) for $\tau$ from prior information on $\tau$.
• M-step: Determine the MLE ($\widehat{\psi }$) of the GP parameters from the combined data in which $\widehat{\tau }$ are inserted into the experimental data.
• E-step: Set $\widehat{\tau }$ as the conditional expectation of $\tau$ given the estimates ($\widehat{\psi }$ from the M-step) of the GP parameters obtained under the combined data.
• Iterate: E- and M-steps until convergence.

In the k-th iteration of the E-step, the conditional expectation of $\tau$ is actually the expectation of the posterior of $\tau$:

$$\begin{array}{cc}\hfill E({\mathit{\tau }}^{*\left(\mathit{k}\right)}|\mathit{y};\mathit{x},{\mathsf{\Psi }}^{\left(k\right)})& =\int \tau f(\tau |\mathit{y};\mathit{x},{\mathsf{\Psi }}^{\left(k\right)})d\tau \hfill \\ \hfill & =\int \tau \frac{f(\tau ,\mathit{y};\mathit{x},{\mathsf{\Psi }}^{\left(k\right)})}{f(\mathit{y};\mathit{x},{\mathsf{\Psi }}^{\left(k\right)})}d\tau \hfill \\ \hfill & =\int \tau \frac{f(\mathit{y}|\tau ;\mathit{x},{\mathsf{\Psi }}^{\left(k\right)})p(\tau ;\mathit{x},{\mathsf{\Psi }}^{\left(k\right)})}{\int f(\tau ;\mathit{x},{\mathsf{\Psi }}^{\left(k\right)})p(\tau ;\mathit{x},{\mathsf{\Psi }}^{\left(k\right)})d\tau }d\tau \hfill \\ \hfill & =\int \tau \frac{G\times p(\tau ;\mathit{x},{\mathsf{\Psi }}^{\left(k\right)})}{\int G\times p(\tau ;\mathit{x},{\mathsf{\Psi }}^{\left(k\right)})d\tau }d\tau \hfill \\ \hfill & ⟹{\tau }^{*(\mathit{k}+\mathit{1})},\hfill \end{array}$$

(11)

where $G=f\left({\underline{y}}_B\right|\tau ;{\underline{x}}_B,{\widehat{\psi }}^{\left(k\right)})=\mathrm{p}\mathrm{d}\mathrm{f}\left(\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{d}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{f}\mathrm{u}\mathrm{n}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\right)\mathrm{o}\mathrm{f}MN(F_B{\widehat{\beta }}^{\left(k\right)},{\widehat{\sigma }}^{2\left(k\right)}{V_B}^{\left(k\right)})$, ${\psi }^{\left(k\right)}=({\underline{\theta }}^{\left(k\right)},{\underline{\beta }}^{\left(k\right)},{\sigma }^{2\left(k\right)}{\gamma }_E^{2\left(k\right)})$, and $p(\tau ;{\underline{x}}_B,{\widehat{\psi }}^{\left(k\right)})$ is a prior (pdf) of $\tau$. Note that $F_B$ and $V_B$ are functions of $\tau$. A numerical integration method by Reference [37] in the R package “cubature” [38] was used for the calculation of (11). We set a uniform distribution as the prior of $\tau$. Other priors can easily be incorporated.

A slightly modified likelihood function from (8) is employed to calculate the MLE in the M-step: $-2logL({\psi }_B;{\underline{y}}_B,X_B,\widehat{\tau })$, where $\widehat{\tau }$ is obtained from the E-step. The M-step is basically the same as Step 3 in the max-min algorithm. The major difference between the two algorithms is that the max-min minimizes the ARSS, whereas the EM calculates the conditional expectation. One defect of the EM method is that it requires more computing time than the ALS or Kennedy–O’Hagan (KOH) method [2].

In each iteration of the E- and M-steps, parameters in the emulator and the combined data are updated. We expect this updating procedure affects positively to the estimation of the calibration parameter. The EM and max-min use the combined data for constructing an emulator, whereas the ALS uses the computer data only. The addition of relevant data generally enhances the prediction capacity of the emulator.

It is worth noting that the EM algorithm converges effectively, according to our experience when executing it in the test function study. The median iteration number is approximately 10 (the first and third quartiles are 6 and 28 iterations, respectively) based on 10 trials for test function 1, as described in detail in the following section.

Table 1. A classification of the calibration methods based on $\tau$ estimation, emulator building, and the number of outer iterations. Acronyms: NLSE = nonlinear least squares estimation, ALS = approximated NLSE, MLE (A) = maximum likelihood estimation calculated from data A, FMLE = full MLE, SMLE = separated MLE, Cond Expect (D) = conditional expectation based on data D, EM = expectation and maximization algorithm, Max-min = max-min algorithm described in Section 3.3, KOH = Kenney-O’Hagan method.

Method	$\mathit{\tau }$ Estimation	Emulator Building	Outer Iterations
NLSE	NLSE		1
ALS	ALS (E)	MLE (C)	1
FMLE	MLE (C, E, ${\psi }_B$ ) (all together)		1
SMLE	MLE (C|E)	MLE (C)	1
EM	Cond Expect (C, E)	MLE (C, E)	many
Max-min	ALS (E)	MLE (C, E)	many
KOH	Cond Expect (E)	MLE(C)	1

presents a classification of the calibration methods based on the $\tau$ estimation, emulator building, and outer iteration. It can be observed that the EM algorithm can be viewed as an extension of the KOH method.

5. Test Function Study

In this section, we describe the application of the calibration methods to test functions (or toy models) in which the true tuning parameters were known a priori. A set of five toy models in different situations were arranged for a comparison of the methods. These test functions are simple toy models, i.e., easy to compute. However, we treated these functions as if they were time-consuming simulators.

The computer data and experimental data with sample sizes $n_C$ and $n_E$, respectively, were generated by

$$y_C=Y({\tau }_C,{\mathbf{x}}_C),y_E=Y({\tau }^{*},{\mathbf{x}}_E)+e.$$

The five test functions along with $n_E=n_C=30$ and with the true constants of $\tau$ are described as follows:

$$\begin{array}{cc}\hfill \mathbf{Test}\mathbf{function}\mathbf{1}:Y(\tau ,x)& ={\tau }_{1}exp({\tau }_2+x_1)+{\tau }_1{x}_2^2-{\tau }_2{x}_3^2\hfill \\ \hfill \mathrm{Computer}\mathrm{data}:& T_1\sim U(0,5),T_2\sim U(0,4),\hfill \\ \hfill & x_1\sim U(-3,3),x_2\sim U(-3,3),x_3\sim U(0,6)\hfill \\ \hfill \mathrm{Experimental}\mathrm{data}:& {\tau }_1=2,{\tau }_2=2,{\sigma }_E^2=1.\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathbf{Test}\mathbf{function}\mathbf{2}:Y(\tau ,x)& ={\tau }_{1}exp({\tau }_2+x_1+{\tau }_3)+{\tau }_1{\tau }_3{x}_2^2-{\tau }_2{x}_3^2-{\tau }_{3}log\left(x_4\right)\hfill \\ \hfill \mathrm{Computer}\mathrm{data}:& T_1\sim U(0,5),T_2\sim U(0,4),T_3\sim U(1,5),\hfill \\ \hfill & x_1\sim U(-3,4),x_2\sim U(-3,3),x_3\sim U(0,6),x_4\sim U(1,5)\hfill \\ \hfill \mathrm{Experimental}\mathrm{data}:& {\tau }_1=2,{\tau }_2=1,{\tau }_3=3,{\sigma }_E^2=1.\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathbf{Test}\mathbf{function}\mathbf{3}:Y(\tau ,x)& ={\tau }_{1}exp\left(\right|x_1+x_2\left|\right)+{\tau }_2(x_4+1.2x_5+1)/2.5+{\tau }_3({\tau }_2+2x_3+x_4)\hfill \\ \hfill & +{\tau }_4(x_1+x_3-{\tau }_4{x}_5-x_6)+2cos\left(6(x_2+x_3+x_3)\right)\hfill \\ \hfill \mathrm{Computer}\mathrm{data}:& T_1\sim U(0,5),T_2\sim U(0,3),T_3\sim U(0,7),T_4\sim U(0,6),\hfill \\ \hfill & x_1\sim U(-2,6),x_2\sim U(-6,6),x_3\sim U(-5,6),x_4\sim U(-6,6)\hfill \\ \hfill & x_5\sim U(-6,4),x_6\sim U(-6,3)\hfill \\ \hfill \mathrm{Experimental}\mathrm{data}:& {\tau }_1=2,{\tau }_2=1,{\tau }_3=4,{\tau }_4=3,{\sigma }_E^2=6.\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathbf{Test}\mathbf{function}\mathbf{4}:Y(\tau ,x)& ={\tau }_1{x}_1^2+{\tau }_2{x}_2+{\tau }_{3}cos\left(x_3\pi \right)+{\tau }_{4}sin\left(x_4\pi \right)\hfill \\ \hfill \mathrm{Computer}\mathrm{data}:& T_1\sim U(0,5),T_2\sim U(0,5),T_3\sim U(0,7),T_4\sim U(0,5)\hfill \\ \hfill & x_1\sim U(0,3),x_2\sim U(0,3),x_3\sim U(0,2),x_4\sim U(0,2)\hfill \\ \hfill \mathrm{Experimental}\mathrm{data}:& {\tau }_1=1,{\tau }_2=2,{\tau }_3=3,{\tau }_4=2,{\sigma }_E^2=4.\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathbf{Test}\mathbf{function}\mathbf{5}:Y(\tau ,x)& =exp({\tau }_2{x}_1+x_2)/(3+{\tau }_1)+{\tau }_2\sqrt{(x_4+x_5)}+{\tau }_1{(x_3+x_6)}^2\hfill \\ \hfill \mathrm{Computer}\mathrm{data}:& T_1\sim U(0,5),T_2\sim U(1,5)\hfill \\ \hfill & x_1\sim U(-0.5,0.5),x_2\sim U(-0.5,0.5),x_3\sim U(-0.5,0.5),\hfill \\ \hfill & x_4\sim U(0,2)x_5\sim U(0,2),x_6\sim U(0,1)\hfill \\ \hfill \mathrm{Experimental}\mathrm{data}:& {\tau }_1=2,{\tau }_2=2,{\sigma }_E^2=0.25.\hfill \end{array}$$

Optimal Latin hypercube designs [39,40] were used for sampling in the independent variables for real experiments (${\mathbf{x}}_E$) and for computer experiments $({\tau }_S,{\mathbf{x}}_S)$. A total of 30 different designs for computer data were employed for each test function to take into account uncertainty in the design, whereas the real experimental design was fixed.

As a result, the average of estimates and the standard deviations from 30 trials are presented. The averaged Euclidean distance from the estimates to the true values was computed to evaluate the performance of the methods. (In addition to the Euclidean distance, one can consider an weighted distance such as the Mahalanobis distance [41]. It may be more meaningful than the Euclidean distance in the sense that the Mahalanobis distance takes into account the covariances among estimates of calibration parameters. The weighted distance is not considered in this study, but may be useful in the future study.) The root mean squared error (RMSE) of the estimates is also provided. The formula for the RMSE is as follows:

$$RMSE\left(\widehat{\tau }\right)=\sqrt{Bia{s}^2\left(\widehat{\tau }\right)+\sum _{i=1}^q\left(std({\widehat{\tau }}_i\right){)}^2},$$

(12)

where $Bias\left(\widehat{\tau }\right)$ is the averaged Euclidean distance to the true constants and $std\left({\widehat{\tau }}_i\right)$ is the standard deviation of each estimate obtained from 30 replications.

Table 2. Result for the test function 1 with true constants ${\tau }_1^{*}=2$, ${\tau }_2^{*}=2$. The values presented are the averaged estimates of ${\tau }^{*}$, the averaged Euclidean distance from the estimates to the true constants, the root mean squared error (RMSE) of the estimates. The standard deviations are in parentheses. The acronyms are same as Table 1. The number 1 or 2 following each method stands for the Gaussian process Model 1 or Model 2.

Method	Average of ${\widehat{\mathit{\tau }}}_1$	Average of ${\widehat{\mathit{\tau }}}_2$	Average Distance	RMSE
ALS 1	1.609 (0.207)	1.773 (0.246)	0.527 (0.168)	0.617
ALS 2	1.448 (0.249)	1.602 (0.351)	0.749 (0.288)	0.864
SMLE 1	1.790 (0.245)	1.758 (0.275)	0.462 (0.147)	0.591
SMLE 2	2.228 (0.409)	1.465 (0.267)	0.703 (0.280)	0.856
Max-min 1	2.296 (0.135)	1.966 (0.224)	0.377 (0.109)	0.459
Max-min 2	2.382 (0.399)	2.151 (0.575)	0.718 (0.342)	1.003
EM 1	2.024 (0.212)	1.828 (0.204)	0.276 (0.197)	0.103
EM 2	2.098 (0.234)	2.894 (0.166)	0.264 (0.177)	0.390

,

Table 3. Same as Table 2 but for test function 2 with true values ${\tau }_1^{*}=2$, ${\tau }_2^{*}=1$, ${\tau }_3^{*}=3$.

Method	Average of ${\widehat{\mathit{\tau }}}_1$	Average of ${\widehat{\mathit{\tau }}}_2$	Average of ${\widehat{\mathit{\tau }}}_3$	Average Distance	RMSE
ALS 1	1.801 (0.324)	1.349 (0.225)	3.301 (0.319)	0.598 (0.232)	0.784
ALS 2	1.715 (0.244)	1.107 (0.322)	3.034 (0.344)	0.570 (0.209)	0.779
SMLE 1	1.737 (0.565)	1.080 (0.527)	2.957 (0.519)	0.873 (0.398)	1.276
SMLE 2	1.912 (0.347)	0.901 (0.381)	2.853 (0.416)	0.585 (0.353)	0.884
Max-min 1	1.706 (0.293)	0.842 (0.127)	3.003 (0.435)	0.535 (0.327)	0.760
Max-min 2	1.768 (0.231)	1.106 (0.314)	3.054 (0.283)	0.511 (0.176)	0.702
EM 1	2.108 (0.238)	1.207 (0.116)	2.975 (1.164)	0.371 (0.109)	0.484
EM 2	2.163 (0.239)	1.261 (0.160)	2.980 (0.112)	0.414(0.128)	0.516

,

Table 4. Same as Table 2 but for test function 3 with true values ${\tau }_1^{*}=2$, ${\tau }_2^{*}=1$, ${\tau }_3^{*}=4$, ${\tau }_4^{*}=3$.

Method	Average of ${\widehat{\mathit{\tau }}}_1$	Average of ${\widehat{\mathit{\tau }}}_2$	Average of ${\widehat{\mathit{\tau }}}_3$	Average of ${\widehat{\mathit{\tau }}}_4$	Average Distance	RMSE
ALS 1	2.314 (0.408)	1.185 (0.335)	3.911 (0.578)	3.367 (0.571)	0.963 (0.514)	1.366
ALS 2	2.224 (0.587)	1.175 (0.341)	3.713 (0.865)	3.323 (0.793)	1.324 (0.547)	1.895
SMLE 1	1.923 (0.465)	1.316 (0.472)	3.713 (0.515)	3.309 (0.648)	1.080 (0.443)	1.513
SMLE 2	1.861 (0.446)	1.365 (0.643)	3.423 (0.749)	3.541 (0.655)	1.609 (0.347)	2.047
Max-min 1	2.060 (0.321)	1.166 (0.300)	3.626 (0.553)	3.315 (0.528)	0.902(0.463)	1.261
Max-min 2	2.039 (0.544)	1.469 (0.339)	3.753 (0.785)	2.845 (0.726)	1.208 (0.612)	1.736
EM 1	2.583 (0.198)	1.522 (0.119)	3.668 (0.250)	3.029 (0.254)	0.938 (0.139)	1.030
EM 2	2.480 (0.239)	1.403 (0.252)	3.681 (0.325)	3.024 (0.223)	0.859 (0.158)	1.007

,

Table 5. Same as Table 2 but for test function 4 with true values ${\tau }_1^{*}=1$, ${\tau }_2^{*}=2$, ${\tau }_3^{*}=3$, ${\tau }_4^{*}=2$.

Method	Average of ${\widehat{\mathit{\tau }}}_1$	Average of ${\widehat{\mathit{\tau }}}_2$	Average of ${\widehat{\mathit{\tau }}}_3$	Average of ${\widehat{\mathit{\tau }}}_4$	Average Distance	RMSE
ALS 1	0.935 (0.237)	1.753 (0.396)	3.304 (0.908)	1.951 (0.610)	1.025 (0.628)	1.568
ALS 2	0.874 (0.250)	1.852 (0.414)	2.765 (0.846)	1.729 (0.672)	1.149 (0.456)	1.650
SMLE 1	0.845 (0.291)	1.975 (0.758)	2.875 (0.532)	1.852 (0.607)	1.069 (0.437)	1.566
SMLE 2	1.008 (0.515)	1.895 (0.895)	2.740 (0.475)	1.816 (0.953)	1.398 (0.53)	2.038
Max-min 1	0.562 (0.261)	2.155 (0.472)	3.087 (0.549)	2.049 (0.560)	0.955 (0.291)	1.348
Max-min 2	0.600 (0.230)	2.319 (0.549)	3.114 (0.749)	2.104 (0.407)	1.095 (0.362)	1.510
EM 1	1.160 (0.198)	2.017 (0.296)	3.383 (0.190)	2.418 (0.169)	0.718 (0.134)	0.841
EM 2	0.921 (0.268)	1.863 (0.397)	3.379 (0.248)	2.498 (0.160)	0.797 (0.149)	0.976

and

Table 6. Same as Table 2 but for test function 5 with true values ${\tau }_1^{*}=2$, ${\tau }_2^{*}=3$.

Method	Average of ${\widehat{\mathit{\tau }}}_1$	Average of ${\widehat{\mathit{\tau }}}_2$	Average Distance	RMSE
ALS 1	1.796 (0.274)	2.076 (0.130)	0.361 (0.078)	0.471
ALS 2	1.745 (0.253)	2.060 (0.105)	0.365 (0.093)	0.456
SMLE 1	2.025 (0.307)	2.038 (0.116)	0.287 (0.157)	0.436
SMLE 2	2.157 (0.483)	2.041 (0.154)	0.436 (0.295)	0.667
Max-min 1	2.074 (0.374)	2.001 (0.100)	0.340 (0.192)	0.515
Max-min 2	1.889 (0.292)	2.045 (0.125)	0.298 (0.152)	0.436
EM 1	2.054 (0.140)	2.085 (0.098)	0.189 (0.054)	0.255
EM 2	2.030 (0.143)	2.102 (0.085)	0.172(0.094)	0.239

present the results for each test function. The averaged Euclidean distance from the estimates to the true constants and the RMSEs of the estimates are displayed in each table. The standard deviations are the numbers in parentheses. In terms of the RMSE, the proposed method offered superior results over the ALS, SMLE, and max-min methods. In particular, the EM shows less bias and lower RMSE than the other methods. We have this result again in Figure 1, Figure 2, Figure 3, Figure 4 and Figure 5, which present the boxplots of the distance to the true value for each of the five test functions. In many cases, the medians of the EM estimates were nearer to the true constants than those from the SMLE, ALS, and max-min methods. The box lengths of the EM estimates were shorter than those obtained from the other methods. It is notable that the max-min was more effective than the ALS and SMLE in test functions 2–4. One plausible reason that the EM was superior to the max-min is that the numerical integration for the conditional expectation in the EM may be more stable than the numerical optimization of the ARSS in the max-min.

The computing times of the max-min and EM algorithms were much longer than those of the ALS and SMLE (see

Table 7. Averaged computing time in seconds for tuning methods with $n_E=n_C=30$ from 30 trials for each of test functions 2, 4, and 5.

Function	No. of $\mathit{\tau }$	No. of X	ALS	SMLE	Max-Min	EM
Test 2	3	4	149	150	661	471
Test 4	4	6	185	295	1207	1300
Test 5	2	6	23	117	435	320

). The times were obtained using a personal computer with an Intel i5 CPU (3.6 GHz) and 16 Giga bytes of memory. Thus, we faced the limitation in extending our test functions to a higher-dimensional calibration parameters because of the heavy computing time in the max-min and EM algorithms. Numerical integration method for more than 10 dimensions may not be practical in the EM algorithm. In such high-dimensional cases, a Monte Carlo integration technique would be useful, but it requires more computing time.

6. Application to Nuclear Fusion Simulator

We present, in this section, the application of the calibration methods to computer code known as “Baldur” [42], which is a time-consuming simulator for the energy confinement time in a nuclear fusion device. It is called a “tokamak” in the Russian language. The mathematical model is simply expressed as follows, for a known complex function f calculated from the Baldur code:

$$y=f({\tau }_1,{\tau }_2,{\tau }_3,{\tau }_4,x_1,x_2,x_3,x_4),$$

(13)

where $x_1$ is the input heating power, $x_2$ is the toroidal plasma current, $x_3$ is the line average electron density, and $x_4$ is the toroidal magnetic field. Calibration parameters $\tau =({\tau }_1,{\tau }_2,{\tau }_3,{\tau }_4)$ determine the energy transfer, where each parameter is respectively, related to drift waves, rippling, resistive ballooning, and the critical value of ${\eta }_i$ [42].

The experimental data consisted of $\mathbf{x}=\{x_1,x_2,x_3,x_4\}$ with sample size 42 from the Poloidal Divertor Experiment (PDX) tokamak at Princeton. The computer data consisted of $(\tau ,\mathbf{x})$ with sample size 64 from the Baldur code. The details on data can be found in References [1,42].

Table 8. Estimates of the tuning parameters by various calibration methods from the nuclear fusion data. The last column shows computing time in seconds for tuning methods. The acronyms are same as Figure 1.

Method	${\widehat{\mathit{\tau }}}_1$	${\widehat{\mathit{\tau }}}_2$	${\widehat{\mathit{\tau }}}_3$	${\widehat{\mathit{\tau }}}_4$	Computing Time
ALS 1	1.012	2.035	1.110	1.308	337
ALS 2	1.195	1.593	0.956	1.035	1017
SMLE 1	1.120	2.055	0.118	1.303	585
SMLE 2	1.290	1.342	1.204	1.157	1230
Max-min 1	0.670	1.125	0.468	1.826	1674
Max-min 2	0.732	2.592	1.311	1.034	8042
EM 1	−0.459	3.227	1.750	1.407	1481
EM 2	1.064	2.428	1.048	1.030	7717

presents the results of the $\tau$ estimation when using the ALS, SMLE, max-min, and EM algorithms on the basis of GP Model 1 and Model 2 from the tokamak data. The results were obtained using R program on a personal computer with an Intel i5 CPU (3.6 GHz) and 16 Giga bytes of memory.

Figure 6 depicts the residuals ($y_E-\widehat{Y}\left(T_E\right)$) plotted according to the predicted values $\widehat{Y}\left(T_E\right)$ that were obtained by various methods using GP Model 1 and Model 2. The residual plots from all methods exhibited linear trends. The trend for the EM algorithm was the lowest among the methods.

7. Discussion

Certain basic limitations exist when calibrating computer models to real data. We have experienced that the performance of the calibration methods was influenced significantly by the designs of both the physical and computer experiments [43,44]. Thus, a sequential designing approach must be very useful in practice [10,11,16]. Relevant experimental designs under a sequential tuning will improve the calibration very well.

Iterative versions of calibration methods, including the KOH method, may be available. For example, an iterative version of the SMLE method can be summarized as follows:

• Step 1: Acquire $\widehat{\tau }$ by maximizing the conditional likelihood function of $\tau$ on the E-data, given $(y_C,{\widehat{\psi }}_C)$.
• Step 2: [maximization] Acquire ${\widehat{\psi }}_B$ by maximizing the likelihood from the both data, in which $\widehat{\tau }$ are inserted into the E-data.
• Step 3: [maximization] Acquire $\widehat{\tau }$ by maximizing the conditional likelihood function of $\tau$ on the E-data, given $(y_C,{\widehat{\psi }}_B)$.
• Step 4: Iterate Steps 2 and 3 until convergence.

The estimates $\widehat{\tau }$ may vary according to the selected emulator. Thus an importance in calibration is the selection of the GP surrogate with some regression variables and GP parameters. We only used simple models (6) in this study. If the optimal GP model is built by the model selection algorithm, the result could be different.

8. Summary

The ALS method using a GP emulator is a basic calibration technique for complex computer models. However, it exhibits the potential drawback that the emulator is constructed once and it is no longer re-built. To overcome this defect, an iterative (EM) algorithm has been proposed in this study. The calibration parameters of the simulation code are calculated by the conditional expectation (E-step), whereas the GP parameters are updated by maximum likelihood estimation (M-step). These EM steps are alternately repeated until convergence by using both computer and experimental data.

We examined five test functions for a performance comparison. We confirmed that our proposed method (the EM algorithm) provided better results than the SMLE, ALS, and max-min methods in terms of the RMSE. The disadvantage of the proposed method is that it is more time-consuming than the ALS, because the EM algorithm needs to optimize complex functions and compute the conditional expectation using numerical integration based on the combined data. Nonetheless, the EM method can provide improved calibration as well as a superior emulator of the computer code compared to non-iterative methods, including the ALS and SMLE.