A Multilevel Monte Carlo Approach for a Stochastic Optimal Control Problem Based on the Gradient Projection Method

Abstract

A multilevel Monte Carlo (MLMC) method is applied to simulate a stochastic optimal problem based on the gradient projection method. In the numerical simulation of the stochastic optimal control problem, the approximation of expected value is involved, and the MLMC method is used to address it. The computational cost of the MLMC method and the convergence analysis of the MLMC gradient projection algorithm are presented. Two numerical examples are carried out to verify the effectiveness of our method.

1. Introduction

The stochastic optimal control problem has been widely used in engineering, finance and economics. Many scholars have studied the stochastic optimal control problem for different controlled systems. Because it is difficult to find analytical solutions for these problems, the application of numerical approximation is a good choice. As with numerical methods for determining optimal control problems (see, e.g., [1,2,3,4,5,6]), numerical methods for stochastic optimal control problems have also been extensively studied. For optimal control problems governed by PDE with a random coefficient, the authors of [7,8,9] studied numerical approximations using different methods for different problems. For SDE optimal control problems, the stochastic maximum principle in [10,11], Bellman dynamic programming principle in [12] and Martingale method in [13] have been used to study numerical approximation in recent years.

The gradient projection method is a common numerical optimization method. In [14,15,16,17], the gradient projection method is used. For the numerical simulation of stochastic optimal control problems, whether gradient projection or other optimization methods are used, the approximation of expected value is always involved. For a class stochastic optimal control problem, the authors of [16] combined the gradient projection method with conditional expectation (which was used to solve forward and backward stochastic differential equations) to solve the stochastic optimal control problem. This method is difficult because it involves conditional expectation and numerical methods of solving forward and backward stochastic differential equations. In [15], the expectation was calculated using the Monte Carlo (MC) method, which is easy to understand and implement. However, the convergence speed of the MC method is slow. If we want it to produce a relatively small allowable error, we need to use a large sample. The MLMC method is a commonly used method of improving the slow convergence rate. For the MLMC method and its application, we can refer to the literature [7,18,19,20,21,22,23,24]. It is worth mentioning that in reference [18], an MLMC method is proposed for the robust optimization of PDEs with random coefficients. The MLMC method can effectively overcome adverse effects when the iterative solution is near the exact solution. However, the proof of the convergence for the gradient algorithm is not given.

In this work, we apply the gradient projection method with the MLMC method to solve a stochastic optimal control problem. An expected value is needed to compute in the simulation of the stochastic optimal control problem. To reduce the influence of statistical and discrete errors brought about by the calculation of the gradient, we use the MLMC method to estimate the gradient in each iteration. In the process of iteration, the mean square error (MSE) is dynamically updated. We prove the convergence of the gradient projection method combined with MLMC, and also extend the theory of MLMC which is suitable for our stochastic optimal control problem.

The rest of this paper is organized as follows. We describe the stochastic optimal control problem in Section 2. In Section 3, we review the gradient projection method and MLMC theory. In Section 4, we expand the existing MLMC theory and apply it to the gradient projection method for the stochastic optimal problem. The convergence analysis is also presented in this section. Some numerical experiments are carried out to verify the validity of our method in Section 5. Main contributions and future work are presented in Section 6.

2. Stochastic Optimal Control Problem

Let $(\mathsf{\Omega },\mathcal{F},{\left\{{\mathcal{F}}_t\right\}}_{t\ge 0},\mathbb{P})$ be a complete probability space and $L_{\mathcal{F}}^2([0,T];\mathbb{R})$ be a real-valued square-integrable ${\mathcal{F}}_t$-adapted process space such that ${\left|\right|y\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}<\infty ,$ where ${\left\{{\mathcal{F}}_t\right\}}_{t\ge 0}$ is a natural filtration generated by a one-dimensional standard Brownian motion ${\left\{W_t\right\}}_{t\ge 0}$, and 

$${\left|\right|y\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}={\left({\int }_{\mathsf{\Omega }}{\int }_0^T{\left|y\right|}^{2}dtd\mathbb{P}\left(\omega \right)\right)}^{1/2}.$$

(1)

The objective function of the optimal control problem that we consider is

$$\underset{u\in U_{ad}}{min}J(y,u)={\int }_0^{T}E\left[h\left(y\right)\right]dt+{\int }_0^{T}j\left(u\right)dt,$$

(2)

where $h(·),j(·)$ are first-order continuous derivative functions. $u\in U_{ad}$ is a deterministic control. $U_{ad}$ is a closed convex control set in the control space $L^2(0,T)$. $E(·)$ stands for expectation, which is defined by $E\left(h\right)={\int }_{\mathsf{\Omega }}h\left(\omega \right)d\mathbb{P}\left(\omega \right)$. The stochastic process $y\left(u\right)\in L_{\mathcal{F}}^2([0,T];\mathbb{R})$ is generated by the following stochastic differential equation:

$$dy=f(t,y,u)dt+g(t,y)d{W}_t,y\left(0\right)=y_0.$$

(3)

Assume that f is a continuous differentiable function with respect to $t,y,u$; and g is a continuous differentiable function with respect to $t,y$. Under these continuous differentiable assumptions, we can find that for the problem (3) there exists a unique solution $y(·)\in L_{\mathcal{F}}^2([0,T];\mathbb{R})$ with $(y_0,u(·))\in \mathbb{R}\times U_{ad}$ (this can be seen in [25]). Here, $y(·)$ is a function of $u(·)$. For the optimal control problem (2)–(3) there exists a unique solution. Let $u^{*}$ be the optimal solution.

In the following, C denotes different positive constants at different occurrences, and is independent of discrete parameters, sample parameters and iteration times.

3. Review of Gradient Projection Method and MLMC Method

The gradient projection method is a common numerical method for optimization problems. The gradient projection method for stochastic optimal control problems usually contains expectation, and MLMC is an important method for calculating expectations. This section introduces the gradient projection and MLMC methods.

3.1. Gradient Projection Method

Let $J\left(u\right)=J\left(y\right(u),u)$, where $y\left(u\right)$ is the solution of (3). We assume that $J\left(u\right)$ is a convex function, $U=L^2([0,T],\mathbb{R})$ is a Hilbert space and K is a closed convex subset of U. Let $b(·,·)$ be a symmetric and positive definite bilinear form and define $\mathbf{b}:U\to U$ by $(\mathbf{b}u,v)=b(u,v)$. Combining the first order optimization condition with the projection operator ${\mathbf{P}}_K:U\to K$, we can obtain (see [15,16])

$$u^{*}={\mathbf{P}}_K(u^{*}-\rho {\mathbf{b}}^{-1}{J}^{\prime }\left(u^{*}\right)),$$

(4)

where $J^{\prime }\left(u\right)$ is the Gâteaux derivative of $J\left(u\right)$, $\rho$ is a positive constant.

We introduce a uniform partition for time intervals: $0=t_0^N<t_1^N<\cdots <t_N^N=T,t_{n+1}^N-t_n^N=T/N=h.$ Let $I_n^N=(t_{n-1}^N,t_n^N]$. The piecewise constant space $U_N$ is denoted by

$$U_N=\left\{u\in U:u=\sum _{n=0}^N{\alpha }_n{\chi }_{I_n^N},a.e.,{\alpha }_n\in \mathbb{R}\right\},$$

(5)

where ${\chi }_{I_n^N}$ is the characteristic function of $I_n^N$. $u^{*}$ can be approximated by:

$$u^{*,N}={\mathbf{P}}_{K_N}(u^{*,N}-\rho {\mathbf{b}}^{-1}{J}^{\prime }\left(u^{*,N}\right)),$$

(6)

where $K_N=K\cap U_N$. Based on the analysis of (4)–(6), we can get an iterative scheme for the numerical approximation of (2)–(3) as below:

$$\left\{\begin{array}{c}b(u_{i+\frac{1}{2}}^N,v)=b(u_i^N,v)-{\rho }_i(J_N^{\prime }\left(u_i^N\right),v),v\in K_N,\hfill \\ u_{i+1}^N={\mathbf{P}}_{K_N}^b\left(u_{i+\frac{1}{2}}^N\right),\hfill \end{array}\right.$$

(7)

where ${\rho }_i$ is the iterative step size, and $J_N^{\prime }$ is the numerical approximation of $J^{\prime }(·)$. The error between $J^{\prime }(·)$ and $J_N^{\prime }(·)$ is represented by the following formula:

$${ϵ}_N=\underset{i}{sup}\left|\right|J^{\prime }\left(u_i^N\right)-J_N^{\prime }\left(u_i^N\right)\left|\right|.$$

For the iterative scheme (7), the following convergence results hold (see [16]).

Lemma 1.

Assume that $J^{\prime }(·)$ is Lipschitz continuous and uniformly monotone in the neighborhood of $u^{*}$ and $u^{*,N}$, i.e.,  there exist positive constants c and C such that

$$\begin{array}{cc}\hfill & \left|\right|J^{\prime }\left(u^{*}\right)-J^{\prime }\left(v\right)\left|\right|\le C\left|\right|u^{*}-v\left|\right|\forall v\in K,\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill & (J^{\prime }\left(u^{*}\right)-J^{\prime }\left(v\right),u^{*}-v)\ge c\left|\right|u^{*}-v{\left|\right|}^2\forall v\in K,\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill & \left|\right|J^{\prime }\left(u^{*,N}\right)-J^{\prime }\left(v\right)\left|\right|\le C\left|\right|u^{*,N}-v\left|\right|\forall v\in K_N,\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & (J^{\prime }\left(u^{*,N}\right)-J^{\prime }\left(v\right),u^{*,N}-v)\ge c\left|\right|u^{*,N}-v{\left|\right|}^2\forall v\in K_N.\hfill \end{array}$$

(11)

Suppose that

$${ϵ}_N=\underset{i}{sup}\left|\right|J^{\prime }\left(u_i^N\right)-J_N^{\prime }\left(u_i^N\right)\left|\right|\to 0,N\to \infty ,$$

(12)

and ${\rho }_i$ can be chosen such that $0<1-2c{\rho }_i+(1+2C){\rho }_i^2\le {\delta }^2$ for some constant $0<\delta <1$. Then the iteration scheme (7) is convergent, i.e.,

$$\left|\right|u^{*}-u_i^N\left|\right|\to 0,(i\to \infty ,N\to \infty ).$$

(13)

In the iterative scheme (7), the gradient of the objective function (2) needs to be calculated. Using the stochastic maximum principle, the gradient is solved more conveniently by introducing an adjoint equation:

$$-dp=[h^{\prime }\left(y\right)+p{f}_y^{\prime }(t,y,u)-p{g}_y^{\prime }{(t,y)}^2]dt+p{g}_y^{\prime }(t,y)d{W}_t,p\left(T\right)=0.$$

(14)

The detail deduction can be seen in [26,27]. Thus the derivative of $J\left(u\right)$ can be denoted by

$$J^{\prime }\left(u\right)v={\int }_0^T(E\left[p{f}_u^{\prime }(t,y,u)\right]+j^{\prime }\left(u\right))vdt,\forall v\in U.$$

(15)

According to the Riesz representation theorem, we can get

$$J^{\prime }\left(u\right)=E\left[p{f}_u^{\prime }(t,y,u)\right]+j^{\prime }\left(u\right).$$

(16)

3.2. MLMC Method

In [19,20,21], the quantities of interest in the MLMC method are scalar value. The authors of [18,21]) make an extension of the MLMC theory to fit function value. In the gradient projection algorithm, the quantities of interest are unknown functions. In this section, we will first briefly review the theoretical knowledge of MLMC. Then, we discuss in detail how to estimate the expectation for the gradient projection method.

3.2.1. Scalar-Valued Quantities of Output

The quantity we are interested in is $A:\mathsf{\Omega }\to \mathbb{R}$. Generally, the exact sample of A is not known. We can only get an approximate sample $A_h\left(\omega \right)$. It is assumed that A has an $\alpha$ order weak convergence property, i.e.,

$$|E[A_h-A]|\le C{h}^{\alpha },$$

(17)

and the computational cost satisfies

$$\mathcal{C}\left(A_h\left(\omega \right)\right)\le C{h}^{-\gamma },$$

(18)

where $\gamma$ is a positive constant. Usually, $\alpha ,\gamma$ depend on the algorithm itself.

For the MLMC method (see, e.g., [19,21]), we consider the multiple approximations $A_{h_0},A_{h_1}$, ⋯, $A_{h_L}$ of A, where $h_l={\mathbb{M}}^{-l}T$ ($l=0,1,\cdots ,L$) represents time step size at the lth level. Here, $\mathbb{M}$ is a positive integer (in the later numerical experiments, $\mathbb{M}=2$). The expectation of $E\left[A_{h_L}\right]$ can be defined by

$$E\left[A_{h_L}\right]=E\left[A_{h_0}\right]+\sum _{l=1}^{L}E[A_{h_l}-A_{h_{l-1}}]=\sum _{l=0}^{L}E\left[Y_l\right],$$

(19)

where $Y_l=A_{h_l}-A_{h_{l-1}},A_{h_{-1}}=0$. For each $E\left[Y_l\right]$, we apply the standard MC method to estimate. If the sample number is $M_l$, we can obtain

$$\widehat{Y_l}=M_l^{-1}\sum _{i=1}^{M_l}{Y}_l\left({\omega }_i\right)=\frac{1}{M_l}\sum _{i=1}^{M_l}(A_{h_l}\left(w_i\right)-A_{h_{l-1}}\left(w_i\right)).$$

(20)

In order to maintain a high correlation between samples under fine and coarse meshes, we must produce samples $A_{h_l}\left({\omega }_i\right),A_{h_{l-1}}\left({\omega }_i\right)$ in the same Brownian motion path. Combining (19) and (20), we can get the multilevel estimation $Y={\sum }_{l=0}^L\widehat{Y_l}.$ Because the expectation operator is linear, and each expectation is estimated independently, we have

$$E\left[{\widehat{Y}}_l\right]=E[A_{h_l}-A_{h_{l-1}}],Var\left[Y\right]=\sum _{l=0}^L\frac{1}{M_l}Var\left[Y_l\right].$$

(21)

Moreover, Y is known as an approximation of $E\left[A\right]$, and its mean square error (MSE) can be described as:

$$\begin{array}{cc}\hfill E\left[{(Y-E\left[A\right])}^2\right]& =E\left[{(Y-E\left[Y\right]+E\left[Y\right]-E\left[A\right])}^2\right]\hfill \\ \hfill & =\sum _{l=0}^L\frac{1}{M_l}Var\left[Y_l\right]+{\left(E[A_{h_L}-A]\right)}^2,\hfill \end{array}$$

(22)

where the first term is the statistical error and the second term is the algorithm error. To make the MSE less than ${ϵ}^2$, we may let both terms be less than ${ϵ}^2/2$. Denote the cost of taking a sample of $Y_l$ by ${\mathcal{C}}_l$, and the sample size by $M_l$. The total cost can be represented as

$$\mathcal{C}\left(Y\right)=\sum _{l=0}^L{M}_l{\mathcal{C}}_l.$$

(23)

How should we choose $M_l$ such that the multilevel estimated variance is less than ${ϵ}^2$? In [21], we find that the optimal sample number can be selected as

$$M_l=⌈\frac{2}{{ϵ}^2}\sqrt{Var\left[Y_l\right]{\mathcal{C}}_l^{-1}}\sum _{i=0}^L\sqrt{Var\left[Y_i\right]{\mathcal{C}}_i}⌉.$$

(24)

Here the symbol $\lceil ·\rceil$ indicates rounding up.

The complexity theorem for the MLMC is given in [19,20,21] when the quantities of interest are scalar. Usually, l can not be taken from 0, because when the grid is too coarse, the correlation of equation (SDE, SPDE) is lost. Especially when the quantity is a function, an interpolation operator is needed. If the grid is too coarse, interpolation can cause large errors.

3.2.2. Function Valued Quantities of Output

When the interest quantity is a function in Equation (16), a natural problem is how to apply classical MLMC theory.

When the samples are a vector or matrix, the discrete time step size of each level is different. $A_{h_l}\left(\omega \right)$ and $A_{h_{l-1}}\left(\omega \right)$ are not compatible. Thus they cannot be subtracted directly. The most natural idea is processed by interpolation and compression. Reference [28] introduced an abstract operator $I_{l_1}^{l_2}:{\mathbb{R}}^{{\mathbb{M}}^{l_1}+1}\to {\mathbb{R}}^{{\mathbb{M}}^{l_2}+1}$ for one-dimensional cases. If $l_1<l_2$, the operator is a bounded linear prolongation operator. If $l_1>l_2$, the operator is a bounded linear compression operator. If $l_1=l_2$, it is an identity operator. We also require $I_{l_1}^{l_2}=I_{l_3}^{l_2}{I}_{l_1}^{l_3}$. In real applications, $I_{l_1}^{l_2}$ is often a linear interpolation operator. However, in order to be consistent with the previous control space (5), we define $I_l:{\mathbb{R}}^{{\mathbb{M}}^l+1}\to U_{{\mathbb{M}}^l}$. Redefine

$$E\left[A_{h_L}\right]=E\left[I_0{A}_{h_0}\right]+\sum _{l=1}^{L}E[I_l{A}_{h_l}-I_{l-1}{A}_{h_{l-1}}]=\sum _{l=0}^{L}E\left[{\overline{Y}}_l\right],$$

(25)

$${\overline{Y}}_l=\frac{1}{M_l}\sum _{i=1}^{M_l}\left[I_l^{}{A}_{h_l}\left({\omega }_i\right)-I_{l-1}^{}{A}_{h_{l-1}}\left(w_i\right)\right],$$

(26)

and

$$Y=\sum _{l=0}^L{\overline{Y}}_l,$$

(27)

where $I_{-1}{A}_{h_{-1}}=\mathbf{0}$, $Y\in U_{{\mathbb{M}}^L}$. The MLMC theory of extending to vectors or functions can be found in [18,21]. The MSE of each A is less than ${ϵ}^2$, i.e.,

$$E\left[{(Y-E\left[A\right])}^2\right]\le {ϵ}^2.$$

(28)

The optimal sample number is determined by the maximum variance. Thus, the optimal sample number can be revised as follows:

$$M_l=⌈\frac{2}{{ϵ}^2}\sqrt{\left|\right|Var\left[{\overline{Y}}_l\right]{\left|\right|}_{\infty }{\mathcal{C}}_l^{-1}}\sum _{i=0}^L\sqrt{\left|\right|Var\left[{\overline{Y}}_i\right]{\left|\right|}_{\infty }{\mathcal{C}}_i}⌉.$$

(29)

Next we consider the termination condition of the MLMC method, when the bias term is less than ${ϵ}^2/2$ (see, e.g., [20]). $a\lesssim Cb$ if and only if $a\le Cb$ and $b\le a$. Choosing $\mathbb{M}=4$, we may assume that

$$\left|\right|E[I_l{A}_{h_l}-A]{\left|\right|}_{\infty }\simeq 4^{-\alpha l},\left|\right|E[I_l{A}_{h_l}-I_{l-1}{A}_{h_{l-1}}]{\left|\right|}_{\infty }\simeq 4^{-\alpha l},$$

(30)

where $I_l$ is a bounded linear prolongation operator, such as linear interpolation. Using the inverse triangle inequality, we can obtain

$$\begin{array}{cc}\hfill \left|\right|E[I_l{A}_{h_l}-I_{l-1}{A}_{h_{l-1}}]{\left|\right|}_{\infty }& =\left|\right|E[I_l{A}_{h_l}-A+A-I_{l-1}{A}_{h_{l-1}}]{\left|\right|}_{\infty }\hfill \\ \hfill & \ge \left|\right|E[A-I_{l-1}{A}_{h_{l-1}}]{\left|\right|}_{\infty }-\left|\right|E[I_l{A}_{h_l}-A]{\left|\right|}_{\infty }.\hfill \end{array}$$

(31)

From (30)–(31), we can derive

$$\begin{array}{cc}\hfill \left|\right|E[A-I_{l-1}{A}_{h_{l-1}}]{\left|\right|}_{\infty }& \simeq 4^{-\alpha (l-1)}\hfill \\ \hfill & =C{4}^{-\alpha l}{4}^{\alpha }\hfill \\ \hfill & \simeq 4^{\alpha }\left|\right|E[I_l{A}_{h_l}-A]{\left|\right|}_{\infty }.\hfill \end{array}$$

(32)

Furthermore, we get

$$\left|\right|E[I_l{A}_{h_l}-A]{\left|\right|}_{\infty }\le C{(4^{\alpha }-1)}^{-1}\left|\right|E[I_l{A}_{h_l}-I_{l-1}{A}_{h_{l-1}}]{\left|\right|}_{\infty }.$$

(33)

Combining Equation (32) with (33), we can derive the following error estimate,

$$\left|\right|E[I_l{A}_{h_l}-I_{l-1}{A}_{h_{l-1}}]{\left|\right|}_{\infty }\le C\frac{1}{\sqrt{2}}(4^{\alpha }-1)ϵ.$$

(34)

To ensure that the bias term can be less than ${ϵ}^2/2$, we use the following formula:

$$max\left\{\frac{1}{4}\left|\right|{\overline{Y}}_{l-1}{\left|\right|}_{\infty },\left|\right|{\overline{Y}}_l{\left|\right|}_{\infty }\right\}\le C\frac{1}{\sqrt{2}}(4^{\alpha }-1)ϵ.$$

(35)

Based on the above analysis, the complexity theorem of MLMC is given below.

Theorem 1.

Suppose that there are positive constants $\alpha ,\beta ,\gamma$,  $\alpha \ge \frac{1}{2}min\{\beta ,\gamma \}$, and 

$$\left|\right|E[I_l{A}_{h_l}-A]{\left|\right|}_{\infty }\le C{4}^{-\alpha l},\left|\right|Var\left[{\overline{Y}}_l\right]{\left|\right|}_{\infty }\le C{4}^{-\beta l},{\mathcal{C}}_l\le C{4}^{\gamma l}.$$

(36)

Then, there exists a positive integer L and a sequence ${\left\{M_l\right\}}_{l=0}^L$ such that, for any $ϵ<e^{-1}$

$$\left|\right|E\left[{(Y-E\left[A\right])}^2\right]{\left|\right|}_{\infty }\le {ϵ}^2,$$

(37)

and the cost

$${\mathcal{C}}_{mlmc}\le \left\{\begin{array}{c}C{ϵ}^{-2},if\beta >\gamma ,\hfill \\ C{ϵ}^{-2}{\left(\log ϵ\right)}^2,if\beta =\gamma ,\hfill \\ C{ϵ}^{-2-\frac{(\gamma -\beta )}{\alpha }}if\beta <\gamma .\hfill \end{array}\right.$$

(38)

The proof of Theorem 1 is similar to that of Theorem 3.1 in [20]. The norm can be replaced by $\left|\right|·{\left|\right|}_{L^p\left(D\right)}$, $1\le p\le \infty$.

Next, we introduce a combination of the MLMC method with the gradient projection method.

4. MLMC Method Based on Gradient Projection

For the expectation form $E\left[p{f}_u^{\prime }(t,y,u)\right]$ in formula (16), a more general expectation estimation form may occur in the numerical approximation of stochastic optimal control problems. We consider the expectation formula

$$E\left[A\right]=E\left[f\right(y\left)g\right(p\left)\right],$$

(39)

where y, p are the solutions of the state equation and the adjoint equation, respectively. We assume that f, g have continuous derivatives. The numerical approximation is denoted by $A_{h_L}=f\left(y_{h_L}\right)g\left(p_{h_L}\right)$. Before the theoretical analysis, we first make the following assumptions:

Hypothesis 1 (H1).

Assume that the error estimate of state y is as

$$\left|\right|I_L{y}_{h_L}-y{\left|\right|}_{L^4(\mathsf{\Omega },L^4[0,T])}^2\le C{h}_L^{{\beta }_y}.$$

(40)

Hypothesis 2 (H2).

Assume that the error estimate of the adjoint state p is as

$$\left|\right|I_L{p}_{h_L}-p{\left|\right|}_{L^4(\mathsf{\Omega },L^4[0,T])}^2\le C{h}_L^{{\beta }_p}.$$

(41)

Hypothesis 3 (H3).

Assume that the cost of calculating approximate sample A is as

$${\mathcal{C}}_L\le C\left(h_L^{-{\gamma }_1}+h_L^{-{\gamma }_2}\right),$$

(42)

where the first and second terms are costs in sampling y, p, respectively.

4.1. Classic Monte Carlo Method

Let $A\in L^2(\mathsf{\Omega },L^2[0,T])$. $E\left[A\right]$ is estimated by the average value of the sample, i.e.,

$$E_M\left[A\right]=\frac{1}{M}\sum _{i=1}^{M}A\left({\omega }_i\right),$$

(43)

where $A\left(w_i\right)\in L^2[0,T]$. For a fixed M, we have $E_M\left[A\right]\in L^2[0,T]$. Because the exact sample is taken here, there is only statistical error (see, e.g., [7,22]). The statistical error can be given by the following lemma.

Lemma 2.

Assume that $A\in L^2(\mathsf{\Omega },L^2[0,T])$. Then, for any $M\in \mathbb{N}$, we have

$$\left|\right|E\left[A\right]-E_M{\left[A\right]\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\le M^{-\frac{1}{2}}{\left|\right|A\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}.$$

(44)

The approximation of $E\left[A\right]$ can be defined by

$$E_M\left[A_{h_L}\right]=\frac{1}{M}\sum _{i=1}^M{A}_{h_L}\left({\omega }_i\right),$$

(45)

where $A_{h_L}\left({\omega }_i\right)$, $i=1,\cdots ,M$ is independent and identically distributed. From the formula (45), we can obviously find that there are two sources of error: one is statistical error, and the other is discrete error. The detailed error estimate is described as follows.

Theorem 2.

Let assumptions H1–H2 hold and f, g be Lipschitz continuous. Then we have

$$\begin{array}{cc}\hfill & \left|\right|E\left[A\right]-E_M\left[I_L{A}_{h_L}\right]{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill & \le C\left(M^{-\frac{1}{2}}\left|\right|I_L{A}_{h_L}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}+h_L^{{\beta }_y}+h_L^{{\beta }_p}\right),\hfill \end{array}$$

(46)

where $I_l{A}_{h_l}=f\left(I_l{y}_{h_l}\right)g\left(I_l{p}_{h_l}\right)$.

Proof. 

Firstly, applying Lemma 2 and triangle inequality, we obtain

$$\begin{array}{cc}\hfill \left|\right|E\left[A\right]-E_M\left[I_L{A}_{h_L}\right]{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}& \le \left|\right|E\left[f\left(y\right)g\left(p\right)\right]-E\left[I_L{A}_{h_L}\right]{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill & +\left|\right|E\left[I_L{A}_{h_L}\right]-E_M\left[I_L{A}_{h_L}\right]{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill & \le C{M}^{-\frac{1}{2}}\left|\right|I_L{A}_{h_L}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill & +\left|\right|E\left[f\left(y\right)g\left(p\right)\right]-E\left[f\left(I_L{y}_{h_L}\right)g\left(p\right)\right]{\left|\right|}_{L^2[0,T]}\hfill \\ \hfill & +\left|\right|E\left[f\left(I_L{y}_{h_L}\right)g\left(I_L{p}_{h_L}\right)\right]-E\left[f\left(I_L{y}_{h_L}\right)g\left(p\right)\right]{\left|\right|}_{L^2[0,T]}.\hfill \end{array}$$

(47)

Secondly, using Cauchy–Schwartz inequality and Lipschitz continuity, we have

$$\begin{array}{cc}\hfill & \left|\right|E\left[f\left(y\right)g\left(p\right)\right]-E\left[f\left(I_L{y}_{h_L}\right)g\left(p\right)\right]{\left|\right|}_{L^2[0,T]}\hfill \\ \hfill & +\left|\right|E\left[f\left(I_L{y}_{h_L}\right)g\left(I_L{p}_{h_L}\right)\right]-E\left[f\left(I_L{y}_{h_L}\right)g\left(p\right)\right]{\left|\right|}_{L^2[0,T]}\hfill \\ \hfill & \le C\left(h_L^{{\beta }_y}+h_L^{{\beta }_p}\right).\hfill \end{array}$$

(48)

Combining (47) with (48), we can derive the desired result.    □

From Theorem 2, we can find that the numbers of statistical samples are affected by the step size. Based on this, the complexity theorem of MC is given below:

Theorem 3.

Let assumptions H1–H3 hold, $A\in L^2(\mathsf{\Omega },L^2[0,T])$ and f, g be Lipschitz continuous. Then, the MC sample number M can be derived as

$$M=O\left({(h_L^{{\beta }_p}+h_L^{{\beta }_y})}^{-2}\right),$$

(49)

the error bound yields

$$\left|\right|E\left[A\right]-E_M\left[I_L{A}_{h_L}\right]{\left]\right||}_{L^2(\mathsf{\Omega },L^2[0,T])}\le C\left(h_L^{{\beta }_p}+h_L^{{\beta }_y}\right),$$

(50)

and the total cost satisfies

$${\mathcal{C}}_{mc}\le C(h_L^{-{\gamma }_1}+h_L^{-{\gamma }_2}){(h_L^{{\beta }_p}+h_L^{{\beta }_y})}^{-2}.$$

(51)

Proof. 

Selecting $M=O\left({(h_L^{{\beta }_p}+h_L^{{\beta }_y})}^{-2}\right)$, from Theorem 2 we have inequality (50). The formula (51) can be obtained with assumption $H_3$.    □

4.2. Multilevel Monte Carlo Method

According to Equations (26) and (27), a multilevel estimator is established for A in the following theorem.

Theorem 4.

Let assumptions H1–H2 hold, $A\in L^2(\mathsf{\Omega },L^2[0,T])$ and f, g be Lipschitz continuous. Then, using (26) and (27), the error of MLMC expectation for A is established as follows:

$$\left|\right|E\left[A\right]-{Y\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\le C\left[h_L^{{\beta }_y}+h_L^{{\beta }_p}+\sum _{l=0}^L{M}_l^{-\frac{1}{2}}(h_l^{{\beta }_y}+h_l^{{\beta }_p})\right].$$

(52)

Proof. 

Using the triangle inequality, we get

$$\begin{array}{cc}\hfill & \left|\right|E\left[A\right]-{Y\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill & \le \left|\right|E\left[A\right]-E\left[I_L{A}_{h_L}\right]\left|\right|+\left|\right|E\left[I_L{A}_{h_L}\right]-Y{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}:=I_1+I_2,\hfill \end{array}$$

(53)

where

$$I_1=\left|\right|E\left[A\right]-E\left[I_L{A}_{h_L}\right]\left|\right|,I_2=\left|\right|E\left[I_L{A}_{h_L}\right]-Y{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}.$$

(54)

Now, the aim is to estimate the terms $I_1$ and $I_2$. For $I_1$, employing Theorem 2, we can obtain

$$\left|\right|E\left[A\right]-E\left[I_L{A}_{h_L}\right]\left|\right|\le C\left(h_L^{{\beta }_y}+h_L^{{\beta }_p}\right).$$

(55)

For the term $I_2$, using the triangle inequality, Lemma 2, Cauchy–Schwarz inequality and Lipschitz continuity, we get

$$\begin{array}{cc}\hfill & \left|\right|E\left[A_{h_L}\right]-Y{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill & \le \sum _{l=0}^L\left|\right|E[I_l{A}_{h_l}-I_{l-1}{A}_{h_{l-1}}]-E_{M_l}[I_l{A}_{h_l}-I_{l-1}{A}_{h_{l-1}}]{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill & \le \sum _{l=0}^L{M}_l^{-\frac{1}{2}}\left|\right|f\left(I_l{y}_{h_l}\right)g\left(I_l{p}_{h_l}\right)-f\left(I_{l-1}{y}_{h_{l-1}}\right)g\left(I_{l-1}{p}_{h_{l-1}}\right){\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill & \le \sum _{l=0}^L{M}_l^{-\frac{1}{2}}\left(\left|\right|f\left(I_l{y}_{h_l}\right)g\left(I_l{p}_{h_l}\right)-f\left(y\right)g\left(p\right){\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\right.\hfill \\ \hfill & \left.+\left|\right|f\left(I_{l-1}{y}_{h_{l-1}}\right)g\left(I_{l-1}{p}_{h_{l-1}}\right)-f\left(y\right)g\left(p\right){\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\right)\hfill \\ \hfill & \le C\sum _{l=0}^L{M}_l^{-\frac{1}{2}}(h_l^{{\beta }_y}+h_l^{{\beta }_p}),\hfill \end{array}$$

(56)

where $h_{l-1}=\mathbb{M}{h}_l$ is used in the above derivation. Hence, substituting estimates of $I_1$,$I_2$ into equation (53), we derive

$$\left|\right|E\left[A\right]-{Y\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\le C\left[h_L^{{\beta }_y}+h_L^{{\beta }_p}+\sum _{l=0}^L{M}_l^{-\frac{1}{2}}(h_l^{{\beta }_y}+h_l^{{\beta }_p})\right],$$

(57)

which is the desired result.    □

The formula (52) shows that ${\left\{M_l\right\}}_{l=0}^L$ is selected by balancing discrete and statistical errors. We choose sample number ${\left\{M_l\right\}}_{l=0}^L$ such that

$$\sum _{l=0}^L{M}_l^{-\frac{1}{2}}(h_l^{{\beta }_y}+h_l^{{\beta }_p})\le C_0(h_L^{{\beta }_y}+h_L^{{\beta }_p}),$$

(58)

and the total cost ${\mathcal{C}}_{mlmc}=\sum _{l=0}^L{\mathcal{C}}_l{M}_l$ is as little as possible. According to [7], this is a convex optimization minimization problem. Therefore, there exists an optimal sample number at each level. Thus, we introduce a Lagrange function as

$$L(\mu ,M)=\sum _{l=0}^L{\mathcal{C}}_l{M}_l+\mu \left(\sum _{l=0}^L{M}_l^{-\frac{1}{2}}(h_l^{{\beta }_y}+h_l^{{\beta }_p})-C_0(h_L^{{\beta }_y}+h_L^{{\beta }_p})\right).$$

(59)

Letting the derivative of $L(\mu ,N)$ with respect to $N_l$ be zero, we can derive

$$M_l\approx ⌈{(h_l^{-{\gamma }_1}+h_l^{-{\gamma }_2})}^{-\frac{2}{3}}{(h_l^{{\beta }_y}+h_l^{{\beta }_p})}^{\frac{2}{3}}{\left(\sum _{l=0}^L{(h_l^{-{\gamma }_1}+h_l^{-{\gamma }_2})}^{\frac{1}{3}}{(h_l^{{\beta }_y}+h_l^{{\beta }_p})}^{\frac{2}{3}}\right)}^2{(h_L^{{\beta }_y}+h_L^{{\beta }_p})}^{-2}⌉.$$

(60)

Based on the above analysis, the complexity theorem of the MLMC method based on gradient projection is given as follows.

Theorem 5.

Let assumptions H1–H3 hold, $A\in L^2(\mathsf{\Omega },L^2[0,T])$ and f, g be Lipschitz continuous. Then, the MLMC estimator (27) can be obtained by the following choice of ${\left\{M_l\right\}}_{l=0}^L$,

$$M_l=\left\{\begin{array}{c}O\left({(h_l^{-{\gamma }_1}+h_l^{-{\gamma }_2})}^{-\frac{2}{3}}{(h_l^{{\beta }_y}+h_l^{{\beta }_p})}^{\frac{2}{3}}{(h_L^{{\beta }_y}+h_L^{{\beta }_p})}^{-2}\right),if\tau >0,\hfill \\ O\left({(h_l^{-{\gamma }_1}+h_l^{-{\gamma }_2})}^{-\frac{2}{3}}{(h_l^{{\beta }_y}+h_l^{{\beta }_p})}^{\frac{2}{3}}{(L+1)}^2{(h_L^{{\beta }_y}+h_L^{{\beta }_p})}^{-2}\right),if\tau =0,\hfill \\ O\left({(h_l^{-{\gamma }_1}+h_l^{-{\gamma }_2})}^{-\frac{2}{3}}{(h_l^{{\beta }_y}+h_l^{{\beta }_p})}^{\frac{2}{3}}{h}_L^{\frac{2\tau }{3}}{(h_L^{{\beta }_y}+h_L^{{\beta }_p})}^{-2}\right),if\tau <0,\hfill \end{array}\right.$$

(61)

where

$$\tau =min\{2{\beta }_y-{\gamma }_1,2{\beta }_p-{\gamma }_1,{\beta }_p+{\beta }_y-{\gamma }_1,2{\beta }_p-{\gamma }_2,2{\beta }_y-{\gamma }_2,{\beta }_y+{\beta }_p-{\gamma }_2\}.$$

(62)

Then the error bound is yielded as

$$\left|\right|E\left[A\right]-{Y\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\le C\left(h_L^{{\beta }_y}+h_L^{{\beta }_p}\right).$$

(63)

And the total computational cost ${\mathcal{C}}_{mlmc}$ is asymptotically bounded by $L\to \infty$

$${\mathcal{C}}_{mlmc}\le \left\{\begin{array}{c}C{(h_L^{{\beta }_y}+h_L^{{\beta }_p})}^{-2},if\tau >0,\hfill \\ C{(L+1)}^3{(h_L^{{\beta }_y}+h_L^{{\beta }_p})}^{-2},if\tau =0,\hfill \\ C{h}_L^{\tau }{(h_L^{{\beta }_y}+h_L^{{\beta }_p})}^{-2},if\tau <0.\hfill \end{array}\right.$$

(64)

Proof. 

Firstly, we prove (63). Using Theorem 4 and $h_{l-1}=M{h}_l$, choosing

$$M_l\approx {(h_l^{-{\gamma }_1}+h_l^{-{\gamma }_2})}^{-\frac{2}{3}}{(h_l^{{\beta }_y}+h_l^{{\beta }_p})}^{\frac{2}{3}}{(h_L^{{\beta }_y}+h_L^{{\beta }_p})}^{-2},l=0,\cdots ,L,$$

(65)

we obtain

$$\begin{array}{cc}\hfill \left|\right|E\left[A\right]-{Y\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}& \le C\left[h_L^{{\beta }_y}+h_L^{{\beta }_p}+\sum _{l=0}^L{M}_l^{-\frac{1}{2}}(h_l^{{\beta }_y}+h_l^{{\beta }_p})\right]\hfill \\ \hfill & \le C\left[h_L^{{\beta }_y}+h_L^{{\beta }_p}+(h_L^{{\beta }_y}+h_L^{{\beta }_p})\sum _{l=0}^L{(h_l^{-{\gamma }_1}+h_l^{-{\gamma }_2})}^{\frac{1}{3}}{(h_l^{{\beta }_y}+h_l^{{\beta }_p})}^{\frac{2}{3}}\right]\hfill \\ \hfill & \le C\left(h_L^{{\beta }_y}+h_L^{{\beta }_p}\right),\hfill \end{array}$$

(66)

thereby (63) is proved.

For formula (64), we discuss the proof of the first case that $\tau >0$. It is similar for the latter two cases.

According to hypothesis $H_3$, we get

$${\mathcal{C}}_{mlmc}=\sum _{l=0}^L{N}_l{\mathcal{C}}_l\le C{(h_L^{{\beta }_y}+h_L^{{\beta }_p})}^{-2}\sum _{l=0}^L{(h_l^{-{\gamma }_1}+h_l^{-{\gamma }_2})}^{\frac{1}{3}}{(h_l^{{\beta }_y}+h_l^{{\beta }_p})}^{\frac{2}{3}}.$$

(67)

Let $\tau >0$, as $L\to \infty$. Then

$$\sum _{l=0}^L{(h_l^{-{\gamma }_1}+h_l^{-{\gamma }_2})}^{\frac{1}{3}}{(h_l^{{\beta }_y}+h_l^{{\beta }_p})}^{\frac{2}{3}}=O\left(1\right),$$

(68)

Thus we have

$${\mathcal{C}}_{mlmc}\le C{(h_L^{{\beta }_y}+h_L^{{\beta }_p})}^{-2}.$$

(69)

Here we complete the proof.    □

4.3. Gradient Projection Based on Optimization

Following the analysis of the former parts, the numerical iterative algorithm for the optimal control problem (2)–(3) is given in [15].

From Lemma 1, Corollary 3.2 of [16], we know that their algorithm is convergent. The error of the scheme in [15] has two main sources. One is the error caused by estimating the expected $E\left[p{f}_u^{\prime }(t,y,u)\right]$; the other is caused by the Euler scheme, which cannot be ignored. The error of the estimation for expectation has little effect on the step size ${\rho }_i$ at the initial iteration step. When ${\rho }_i$ is small, the estimated error from expectations may completely distort the gradient direction. This causes the algorithm to be unable to converge, i.e., the iterative error does not subsequently decrease.

To make the gradient valid, it is necessary that the MSE satisfies ${ϵ}^{\left(i\right)}<\eta \left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2(0,T)}$, where $\eta \in (0,1)$ ($\eta$ is determined by the optimal control problem). To reduce the influence of statistical error, discrete error and unnecessary computation cost, we use MLMC to estimate expectation.

4.4. MLMC Gradient Projection Algorithm

For the algorithm presented in [15], the MSE cannot decrease for a fixed time step size, no matter how much the number of samples is increased, because it is a biased estimate. Therefore, in an efficient algorithm, the time step size will not be fixed. Here we apply the MLMC method to estimate the gradient, which can ensure that the MSE of each iteration is within an allowable range. The detail step is presented in Algorithm 1. Norm $\left|\right|·{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}$ is induced by the inner product ${(u,v)}_{L^2\left(\mathsf{\Omega }\right),L^2[0,T]}={\int }_0^T{\int }_{\mathsf{\Omega }}uvd\mathbb{P}\left(\omega \right)dt$. $J_{(L_i,M_i)}^{\prime }\left(u^{(i-1)}\right)=Y+j^{\prime }\left(u^{(i-1)}\right)$ is the MLMC approximation of (16). The definition of Y can be seen in (27). Here, $A=p{f}_u^{\prime }(t,y,u)$ is involved in (28), Theorems 4 and 5.

Algorithm 1: MLMC gradient projection based optimization

1: input $\tau$, ${ϵ}^{\left(1\right)}$, $i_{max}$, $u^{\left(0\right)}$, $\eta$ 2: for i = 1,$\cdots ,i_{max}$ do 3:     estimate $J_{(L_i,M_i)}^{\prime }\left(u^{(i-1)}\right)$ 4:      estimate $u^{\left(i\right)}$ 5:   if  $\left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2[0,T]}\le \tau$ then 6:          return $u^{\left(i\right)}$ 7:   end if 8:   if ${ϵ}^{\left(i\right)}\ge \eta \left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2[0,T]}$ or  ${ϵ}^{\left(i\right)}\le {\eta }^2\left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2[0,T]}$ then 9:         ${ϵ}^{(i+1)}=\eta \left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2[0,T]}$               or ${ϵ}^{(i+1)}=max\{\tau ,\eta ||u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2[0,T]}\}$ 10:   else 11:        ${ϵ}^{(i+1)}={ϵ}^{\left(i\right)}$ 12:   end if 13: end for

Line 1: $\tau$ is the iterative permissible error; ${ϵ}^{\left(1\right)}$ is the given initial MSE; $i_{max}$ is the max iterative steps; $u^{\left(0\right)}$ is the initial control function; $\eta \in (0,1)$ is a given parameter.

Line 3: We use the MLMC method to estimate gradient according to MSE ${ϵ}^{\left(i\right)}$.

Line 4: We use the gradient projection formula (7) to update the control function. To determine the new optimal iterative step size ${\rho }_i$, we need to calculate the objective function. After a small number of iterations, the change of the objective function value is small, which makes the MSE very small and the computation cost of MLMC very large, especially through the Armijio search method. So for the iterative step size here we simply use ${\rho }_i=\frac{1}{\sqrt{i+d}}$, where d is a positive constant.

Lines 8–12 determine the MSE of the next iteration, ensure that the MSE of the MLMC estimation gradient will not affect the iterative error of each iteration, and also ensure that the MSE cannot be less than the iterative error of each iteration. This avoids the waste of unnecessary samples; especially when the iteration is close to termination, a very small error change will lead to a large sample number difference.

Noting that the sample number $M_i$ and max level $L_i$ are related to ${ϵ}^{\left(i\right)}$, and

$$\begin{array}{c}\hfill J_{(L_i,M_i)}^{\prime }\left(u^{(i-1)}\right)\in U_{{\mathbb{M}}^{L_i}},\end{array}$$

(70)

from Algorithm 1, we can see

$$\begin{array}{c}\hfill {ϵ}^{\left(i\right)}\simeq \left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2[0,T]}\end{array}$$

(71)

hold.

4.5. Convergence Analysis of the Algorithm

First of all, we consider the accuracy of applying MLMC to estimate a gradient. For the accuracy of the gradient estimate, Theorem 6.1 of [18] is discussed in detail. The gradient estimated by MLMC is the exact gradient of a discrete objective function. When the objective function is convex, the MLMC estimation remains convex.

Algorithm 1 eliminates the impact of $\left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2[0,T]}$ from discrete and statistical errors as far as possible. According to formula (16), line 3 and line 9 in Algorithm 1, we have

$$\left|\right|J^{\prime }\left(u^{(i-1)}\right)-J_{(L_i,M_i)}^{\prime }\left(u^{(i-1)}\right){\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\le C\left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])},$$

(72)

where $J_{(L_i,M_i)}^{\prime }\left(u^{(i-1)}\right)$ is the MLMC estimate of the $J^{\prime }\left(u^{\left(i\right)}\right)$.

From Lemma 1, Corollary 3.2 of [16] and Theorem 1 of [15], we know that Algorithm 1 is convergent and that the final iterative error is not affected by discrete or statistical errors. We have the following convergence theorem as $\tau \to 0$:

Theorem 6.

Suppose all the conditions of Lemma 1 hold,

$$\left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\to 0,i\to \infty ,$$

(73)

and ${\rho }_i$ satisfies $0<1-2c{\rho }_i+\overline{C}{\rho }_i^2\le \frac{{\delta }^2}{4}$, where $\delta \in (0,1)$. Then, Algorithm 1 based on multilevel estimation is convergent, i.e.,

$$\left|\right|u^{*}-u^{\left(i\right)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}\to 0,(i\to \infty ).$$

(74)

Proof. 

Triangle inequality implies

$$\left|\right|u^{*}-u^{\left(i\right)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2\le 2\left|\right|u^{*}-u^{*,N(i-1)}{\left|\right|}_{L^2[0,T]}^2+2\left|\right|u^{*,N(i-1)}-u^{\left(i\right)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2,$$

(75)

where the definition of $u^{*,N(i-1)}$ can be seen (6). Let ${\omega }_1=u^{*}-{\rho }_i{J}^{\prime }\left(u^{*}\right)$, ${\omega }_2=u^{*,N(i-1)}-{\rho }_i{J}^{\prime }\left(u^{*,N(i-1)}\right)$, ${\omega }_3=u^{(i-1)}-{\rho }_i{J}_{(L_i,M_i)}^{\prime }$. Then, for the second term of the right hand side (RHS) of (75), we have

$$\begin{array}{cc}\hfill & \left|\right|u^{*,N(i-1)}-u^{\left(i\right)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2=\left|\right|{\mathbf{P}}_{K_{N(i-1)}}\left({\omega }_2\right)-{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_3\right){\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2\hfill \\ \hfill & \le 2\left|\right|{\mathbf{P}}_{K_{N(i-1)}}\left({\omega }_2\right)-{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_2\right){\left|\right|}_{L^2[0,T]}^2+2\left|\right|{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_2\right)-{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_3\right){\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2.\hfill \end{array}$$

(76)

For the first term of the RHS of (76), we can get

$$\begin{array}{cc}\hfill & \left|\right|{\mathbf{P}}_{K_{N(i-1)}}\left({\omega }_2\right)-{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_2\right){\left|\right|}_{L^2[0,T]}^2\hfill \\ \hfill & \le 2\left|\right|{\mathbf{P}}_{K_{N(i-1)}}\left({\omega }_2\right)-u^{*}{\left|\right|}_{L^2[0,T]}^2+2\left|\right|u^{*}-{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_2\right){\left|\right|}_{L^2[0,T]}^2\hfill \\ \hfill & \le C\left(\left|\right|{\mathbf{P}}_{K_{N(i-1)}}\left({\omega }_2\right)-{\mathbf{P}}_{K_{N(i-1)}}\left({\omega }_1\right){\left|\right|}_{L^2[0,T]}^2+\left|\right|{\mathbf{P}}_{K_{N(i-1)}}\left({\omega }_1\right)-{\mathbf{P}}_K\left({\omega }_1\right){\left|\right|}_{L^2[0,T]}^2\right.\hfill \\ \hfill & \left.+\left|\right|{\mathbf{P}}_K\left({\omega }_1\right)-{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_1\right){\left|\right|}_{L^2[0,T]}+\left|\right|{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_1\right)-{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_2\right){\left|\right|}_{L^2[0,T]}\right)\hfill \\ \hfill & \le C\left(\left|\right|u^{*}-u^{*,N(i-1)}{\left|\right|}_{L^2[0,T]}^2+\left|\right|{\mathbf{P}}_K\left({\omega }_1\right)-{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_1\right){\left|\right|}_{L^2[0,T]}^2\right.\hfill \\ \hfill & \left.+\left|\right|{\mathbf{P}}_{K_{N(i-1)}}\left({\omega }_1\right)-{\mathbf{P}}_K\left({\omega }_1\right){\left|\right|}_{L^2[0,T]}^2\right).\hfill \end{array}$$

(77)

The second term of the RHS of (77) can be written as

$$\begin{array}{cc}\hfill & \left|\right|{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_2\right)-{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_3\right){\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2\hfill \\ \hfill & \le \left|\right|u^{*,N(i-1)}-u^{(i-1)}-{\rho }_i(J^{\prime }\left(u^{*,N(i-1)}\right)-J_{(L_i,M_i)}^{\prime }\left(u^{(i-1)}\right)){\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2\hfill \\ \hfill & \le \left|\right|u^{*,N(i-1)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2\hfill \\ \hfill & -2{\rho }_i{(u^{*,N(i-1)}-u^{(i-1)},J^{\prime }\left(u^{*,N(i-1)}\right)-J_{(L_i,M_i)}^{\prime }\left(u^{(i-1)}\right))}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill & +{\rho }_i^2\left|\right|J^{\prime }\left(u^{*,N(i-1)}\right)-J_{(L_i,M_i)}^{\prime }\left(u^{(i-1)}\right){\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2.\hfill \end{array}$$

(78)

Using (11), (72) and Cauchy–Schwartz inequality, we can obtain for the second term of the RHS of (78)

$$\begin{array}{cc}\hfill & -2{\rho }_i{(u^{*,N(i-1)}-u^{(i-1)},J^{\prime }\left(u^{*,N(i-1)}\right)-J_{(L_i,M_i)}^{\prime }\left(u^{(i-1)}\right))}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill =& -2{\rho }_i{(u^{*,N(i-1)}-u^{(i-1)},J^{\prime }\left(u^{*,N(i-1)}\right)-J^{\prime }\left(u^{(i-1)}\right))}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill & -2{\rho }_i{(u^{*,N(i-1)}-u^{(i-1)},J^{\prime }\left(u^{(i-1)}\right)-J_{(L_i,M_i)}^{\prime }\left(u^{(i-1)}\right))}_{L^2(\mathsf{\Omega },L^2[0,T])}\hfill \\ \hfill \le & -2c{\rho }_i\left|\right|u^{*,N(i-1)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2+{\rho }_{i}C{\left|\right|{ϵ}^{\left(i\right)}\left|\right|}_{L^2\left(\mathsf{\Omega }\right)}\left|\right|u^{*,N(i-1)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}.\hfill \end{array}$$

(79)

Substituting (77)–(79) into (76), we have

$$\begin{array}{cc}\hfill & \left|\right|u^{*,N(i-1)}-u^{\left(i\right)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2\hfill \\ \hfill & \le 2(1-2c{\rho }_i+\overline{C}{\rho }_i^2)\left|\right|u^{*,N(i-1)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2+C{\left|\right|{ϵ}^{\left(i\right)}\left|\right|}_{L^2\left(\mathsf{\Omega }\right)}^2.\hfill \end{array}$$

(80)

Triangle inequality and (80) imply

$$\begin{array}{cc}\hfill & \left|\right|u^{*,N(i-1)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2\hfill \\ \hfill & \le 2\left|\right|u^{*,N(i-1)}-u^{\left(i\right)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2+2\left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2\hfill \\ \hfill & \le 4(1-2c{\rho }_i+\overline{C}{\rho }_i^2)\left|\right|u^{*,N(i-1)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2+2\left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2\hfill \\ & +C{\left|\right|{ϵ}^{\left(i\right)}\left|\right|}_{L^2\left(\mathsf{\Omega }\right)}^2.\hfill \end{array}$$

(81)

Choosing appropriate ${\rho }_i$, we can get

$$\begin{array}{c}\hfill \left|\right|u^{*,N(i-1)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2\le C\left(\left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}+{\left|\right|{ϵ}^{\left(i\right)}\left|\right|}_{L^2\left(\mathsf{\Omega }\right)}^2\right).\end{array}$$

(82)

Combining (75) with (76) and (82), we have

$$\begin{array}{cc}\hfill & \left|\right|u^{*}-u^{\left(i\right)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}^2\hfill \\ \hfill & \le C\left(\left|\right|u^{*}-u^{*,N(i-1)}{\left|\right|}_{L^2[0,T]}^2+\left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2(\mathsf{\Omega },L^2[0,T])}+{\left|\right|{ϵ}^{\left(i\right)}\left|\right|}_{L^2\left(\mathsf{\Omega }\right)}^2\right.\hfill \\ \hfill & \left.+\left|\right|{\mathbf{P}}_K\left({\omega }_1\right)-{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_1\right){\left|\right|}_{L^2[0,T]}^2+\left|\right|{\mathbf{P}}_{K_{N(i-1)}}\left({\omega }_1\right)-{\mathbf{P}}_K\left({\omega }_1\right){\left|\right|}_{L^2[0,T]}^2\right).\hfill \end{array}$$

(83)

It is known from Theorem 3.1 of [16] that

$$\left|\right|u^{*}-u^{*,N(i-1)}{\left|\right|}_{L^2[0,T]}^2\to 0,\mathrm{as}i\to \infty .$$

When $i\to \infty$, according to the assumption and (71), we have $\left|\right|{ϵ}^{\left(i\right)}{\left|\right|}_{L^2\left(\mathsf{\Omega }\right)}\to 0$. The fact that $K_{N\left(i\right)}$ is dense in K ($N\left(i\right)\to \infty$) implies that $\left|\right|{\mathbf{P}}_K\left({\omega }_1\right)-{\mathbf{P}}_{K_{N\left(i\right)}}\left({\omega }_1\right){\left|\right|}_{L^2[0,T]}\to 0$ and $\left|\right|{\mathbf{P}}_{K_{N(i-1)}}\left({\omega }_1\right)-{\mathbf{P}}_K\left({\omega }_1\right){\left|\right|}_{L^2[0,T]}\to 0$. This completes the proof. □

5. Numerical Experiments

In this section, two numerical examples are presented. The effectiveness of Algorithm 1 is analyzed numerically.

5.1. Example 1

This example comes from the literature [16]. It is as follows:

$$\begin{array}{c}\hfill \underset{u\in L^2(0,T)}{min}J\left(u\right)=\frac{1}{2}{\int }_0^{T}E\left[{(y-y_d)}^2\right]dt+\frac{1}{2}{\int }_0^T{u}^{2}dt,\\ \hfill s.td{y}_t=u\left(t\right)y\left(t\right)dt+\sigma y\left(t\right)d{W}_t,y\left(0\right)=y_0,\end{array}$$

(84)

The optimal control problem is equivalent to

$$\left\{\begin{array}{cc}dy=uydt+\sigma yd{W}_t,\hfill & \hfill y\left(0\right)=y_0,\\ -dp=(y-y_d+up-p{\sigma }^2)dt+p\sigma d{W}_t,\hfill & \hfill p\left(T\right)=0,\\ u=-E\left[py\right].\hfill \end{array}\right.$$

(85)

The exact solution of the optimal control problem is

$$\begin{array}{c}\hfill u^{*}=\frac{T-t}{\frac{1}{y_0}-Tt+\frac{1}{2}{t}^2},y_d=\frac{e^{{\sigma }^{2}t}-{(T-t)}^2}{\frac{1}{y_0}-Tt+\frac{1}{2}{t}^2}+1.\end{array}$$

(86)

The parameters are chosen as $u^{\left(0\right)}=1$, $T=1$, $y_0=1$, $\sigma =0.05,0.1,0.2$, $\tau ={10}^{-4}$, ${ϵ}^{\left(1\right)}=5e-2$, $\eta =0.5$. Firstly, the parameters ${\alpha }_y,{\alpha }_p,{\beta }_y,{\beta }_p,{\gamma }_1,{\gamma }_2$ need to be simulated when MLMC is used to estimate the gradient. Because the Euler algorithm is used for numerically simulating adjoint equation and state equation, the relevant parameters have been determined theoretically (the details are referred to in [20,29,30]). Therefore, ${\alpha }_y={\alpha }_p=1$, ${\beta }_y={\beta }_p=\frac{1}{2}$, ${\gamma }_1={\gamma }_2=1$. The experimental results are shown in Figure 1, Figure 2 and Figure 3 with $h=1/64$. The final error values $\left|\right|u^{\left(i\right)}-u^{*}{\left|\right|}_{L^2[0,T]}$ are $6.4\times {10}^{-4}$, $1.22\times {10}^{-3}$, $9.9\times {10}^{-3}$, respectively.

Partial information about the computation process when $\sigma =0.2$ is listed in

Table 1. Behavior of MLMC gradient projection-based optimization ($\sigma =0.2$).

i	${\mathit{ϵ}}^{\left(\mathit{i}\right)}$	$\left|\right|{\mathit{u}}^{\left(\mathit{i}\right)}-{\mathit{u}}^{(\mathit{i}-1)}\left|\right|$	${\mathit{M}}_3$	${\mathit{M}}_4$	${\mathit{M}}_5$	${\mathit{M}}_6$	${\mathit{M}}_7$	${\mathit{t}}^{\left(\mathit{i}\right)}/\mathit{s}$
3	$4.5\times {10}^{-2}$	$5.6\times {10}^{-2}$	178	3	1			0.08
7	$9.1\times {10}^{-3}$	$1.3\times {10}^{-2}$	4273	55	10			0.09
11	$2.6\times {10}^{-3}$	$3.8\times {10}^{-3}$	53,203	749	126			0.20
15	$8.4\times {10}^{-4}$	$1.3\times {10}^{-3}$	508,312	6715	1181	254		1.70
19	$3.1\times {10}^{-4}$	$4.9\times {10}^{-4}$	3,816,292	51,158	8526	1880	463	11.63
23	$1.3\times {10}^{-4}$	$2.0\times {10}^{-4}$	23,879,767	321,838	54,732	11,887	2892	67.07

. In the table, $M_k$ is the number of samples, k means time step $h=T/2^k$. The total time of iteration is $2.4E+4$s for $\sigma =0.2$.

From Figure 1, Figure 2 and Figure 3 and Table 1, it can be seen that ${ϵ}^{\left(i\right)}\simeq \left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2[0,T]}\to 0$, Algorithm 1 is convergent, and the MLMC method is efficient.

From Figure 3, we can see that when the $\tau$ gradually decreases, the error decreases slowly ($\sigma =0.1,\sigma =0.2$). This is mainly caused by the direction we choose is the negative gradient direction, which is the disadvantage of the gradient descent method itself.

5.2. Example 2

This example comes from [15]; it is as follows:

$$\begin{array}{c}\hfill \underset{u\ge 0}{min}J=0.5[c_1{\int }_0^{T}E\left[{(y-y_d)}^2\right]+c_2{\int }_0^T{u}^{2}dt,\\ \hfill s.td{y}_t=[u\left(t\right)-r\left(t\right)]dt+\sigma d{W}_t,y\left(0\right)=y_0,\end{array}$$

(87)

This optimal control problem is equivalent to:

$$\left\{\begin{array}{cc}dy=(u-r)dt+\sigma d{W}_t,\hfill & \hfill y\left(0\right)=y_0,\\ -dp=(y-y_d)dt,\hfill & \hfill p\left(T\right)=0,\\ u=max(0,-\frac{c_1}{c_2}E\left[p\right]),\hfill \end{array}\right.$$

(88)

where

$$\begin{array}{c}\hfill c_1=c_2=1,y_0=0,y_d=0.5Tt-0.25t^2+1,r=0.5(T-T),u^{*}=T-t.\end{array}$$

(89)

The parameters are selected as $u^{\left(0\right)}=1$, $T=1$, $\sigma =0.1,3,5$, $\tau ={10}^{-4}$, ${ϵ}^{\left(1\right)}=1\times {10}^{-1}$, $\eta =0.5$. The computational results are shown in Figure 4, Figure 5 with $h=1/64$. The final error values $\left|\right|u^{\left(i\right)}-u^{*}{\left|\right|}_{L^2[0,T]}$ are $3.4\times {10}^{-4}$, $1.39\times {10}^{-3}$, $8.34\times {10}^{-4}$ respectively.

When $\sigma =5$, partial information about the computation process is listed in

Table 2. Behavior of MLMC gradient projection-based optimization ($\sigma =5$).

i	${\mathit{ϵ}}^{\left(\mathit{i}\right)}$	$\left|\right|{\mathit{u}}^{\left(\mathit{i}\right)}-{\mathit{u}}^{(\mathit{i}-1)}\left|\right|$	${\mathit{M}}_3$	${\mathit{M}}_4$	${\mathit{M}}_5$	${\mathit{M}}_6$	${\mathit{t}}^{\left(\mathit{i}\right)}/\mathit{s}$
3	$7.6\times {10}^{-2}$	$7.7\times {10}^{-2}$	2785	19	3		0.05
7	$7.2\times {10}^{-3}$	$9.9\times {10}^{-3}$	329,983	2050	267		0.56
11	$1.6\times {10}^{-3}$	$2.3\times {10}^{-3}$	6,642,693	42,434	5277		9.23
15	$4.4\times {10}^{-4}$	$6.6\times {10}^{-4}$	85,982,855	549,800	69,370		119.30
19	$1.4\times {10}^{-4}$	$2.2\times {10}^{-4}$	821,441,011	5,264,257	656,891	82,911	1160.74

. In this table, $M_k$ has the same meaning as in Table 1. The total time of iteration is $1.841\times {10}^4$ s when $\sigma =5$.

From Figure 4, Figure 5 and Figure 6 and Table 2, it can also be seen that ${ϵ}^{\left(i\right)}\simeq \left|\right|u^{\left(i\right)}-u^{(i-1)}{\left|\right|}_{L^2[0,T]}\to 0$, Algorithm 1 is convergent, and the MLMC method is efficient.

In the numerical approximation of the stochastic optimal control problem, statistical error (or MSE) and discrete error have a great influence on the convergence of the gradient projection method. If we use a Monte Carlo method as in [15], the optimization iteration method (gradient projection method) may not converge for a fixed large time step. In order to ensure the convergence of the iteration (gradient projection method), it is necessary to select a small time step size h and a large sample size M. This means that each optimization iteration will take a lot of time.

Our Algorithm 1 is actually an MLMC method with variable step size. The number of samples in each iteration step is different, and the sample size is also different with different time step sizes. It is optimal in a sense, just as the MLMC is superior to the MC method. Compared with the method in [16], our method is much simpler.

6. Conclusions

In this paper, an MLMC method based on gradient projection is used to approximate a stochastic optimal control problem. One of the contributions is that MLMC method is used to compute expectation, reducing the influence of statistical and discrete errors on the convergence of the gradient projection algorithm. The other contribution is that the convergence proof of Algorithm 1 is given; to the best of our knowledge, this is not found elsewhere. Two numerical examples are carried out to verify the effectiveness of the proposed algorithm.

Our method can be applied to simulate other stochastic optimal control problems with expected value in the optimality conditions (optimal control of SDE or SPDE).

The MLMC is used to reduce the MSE. Other methods include the most important sampling method (see, e.g., [31,32]), quasi-Monte Carlo method and multilevel quasi-Monte Carlo method (see, e.g., [33,34]). In future work, we can use the importance sampling MLMC method and the multi-level quasi-Monte Carlo method to approximate the stochastic optimal control problem with gradient projection or other optimization methods.