Forward-Backward Sweep Method for the System of HJB-FP Equations in Memory-Limited Partially Observable Stochastic Control

Abstract

Memory-limited partially observable stochastic control (ML-POSC) is the stochastic optimal control problem under incomplete information and memory limitation. To obtain the optimal control function of ML-POSC, a system of the forward Fokker–Planck (FP) equation and the backward Hamilton–Jacobi–Bellman (HJB) equation needs to be solved. In this work, we first show that the system of HJB-FP equations can be interpreted via Pontryagin’s minimum principle on the probability density function space. Based on this interpretation, we then propose the forward-backward sweep method (FBSM) for ML-POSC. FBSM is one of the most basic algorithms for Pontryagin’s minimum principle, which alternately computes the forward FP equation and the backward HJB equation in ML-POSC. Although the convergence of FBSM is generally not guaranteed in deterministic control and mean-field stochastic control, it is guaranteed in ML-POSC because the coupling of the HJB-FP equations is limited to the optimal control function in ML-POSC.

1. Introduction

In many practical applications of the stochastic optimal control theory, several constraints need to be considered. In the cases of small devices [1,2] and biological systems [3,4,5,6,7,8], for example, incomplete information and memory limitation become predominant because their sensors are extremely noisy and their memory resources are severely limited. To take into account one of these constraints, incomplete information, partially observable stochastic control (POSC) has been extensively studied in the stochastic optimal control theory [9,10,11,12,13]. However, because POSC cannot take into account the other constraint, memory limitation, it is not practical enough for designing memory-limited controllers for small devices and biological systems. To resolve this problem, memory-limited POSC (ML-POSC) has recently been proposed [14]. Because ML-POSC formulates noisy observation and limited memory explicitly, ML-POSC can take into account both incomplete information and memory limitation in the stochastic optimal control problem.

However, ML-POSC cannot be solved in a similar way as completely observable stochastic control (COSC), which is the most basic stochastic optimal control problem [15,16,17,18]. In COSC, the optimal control function depends only on the Hamilton–Jacobi–Bellman (HJB) equation, which is a time-backward partial differential equation given a terminal condition (Figure 1a) [15,16,17,18]. Therefore, the optimal control function of COSC can be obtained by solving the HJB equation backward in time from the terminal condition, which is called the value iteration method [19,20,21]. In contrast, the optimal control function of ML-POSC depends not only on the HJB equation but also on the Fokker–Planck (FP) equation, which is a time-forward partial differential equation given an initial condition (Figure 1b) [14]. Because the HJB equation and the FP equation interact with each other through the optimal control function in ML-POSC, the optimal control function of ML-POSC cannot be obtained by the value iteration method.

To propose an algorithm to solve ML-POSC, we first show that the system of HJB-FP equations can be interpreted via Pontryagin’s minimum principle on the probability density function space. Pontryagin’s minimum principle is one of the most representative approaches to the deterministic optimal control problem, which converts it into the two-point boundary value problem of the forward state equation and the backward adjoint equation [22,23,24,25]. We formally show that the system of HJB-FP equations is an extension of the system of adjoint and state equations from the deterministic optimal control problem to the stochastic optimal control problem.

The system of HJB-FP equations also appears in the mean-field stochastic control (MFSC) [26,27,28]. Although the relationship between the system of HJB-FP equations and Pontryagin’s minimum principle has been briefly mentioned in MFSC [29,30,31], its details have not yet been investigated. In this work, we investigate it in more detail by deriving the system of HJB-FP equations in a similar way to Pontryagin’s minimum principle. We note that our derivations are formal, not analytical, and more mathematically rigorous proofs remain future challenges. However, our results are consistent with many conventional results and also provide a useful perspective in proposing an algorithm.

We then propose the forward-backward sweep method (FBSM) for ML-POSC. FBSM is an algorithm to compute the forward FP equation and the backward HJB equation alternately, which can be interpreted as an extension of the value iteration method. FBSM has been proposed in Pontryagin’s minimum principle of the deterministic optimal control problem, which computes the forward state equation and the backward adjoint equation alternately [32,33,34]. Because FBSM is easy to implement, it has been used in many applications [35,36]. However, the convergence of FBSM is not guaranteed in deterministic control except for special cases [37,38] because the coupling of adjoint and state equations is not limited to the optimal control function (Figure 1c). In contrast, we show that the convergence of FBSM is generally guaranteed in ML-POSC because the coupling of the HJB-FP equations is limited only to the optimal control function (Figure 1b).

FBSM is called the fixed-point iteration method in MFSC [39,40,41,42]. Although the fixed-point iteration method is the most basic algorithm to solve MFSC, its convergence is not guaranteed for the same reason as deterministic control (Figure 1d). Therefore, ML-POSC is a special and nice class of optimal control problems where FBSM or the fixed-point iteration method is guaranteed to converge.

This paper is organized as follows: In Section 2, we formulate ML-POSC. In Section 3, we derive the system of HJB-FP equations of ML-POSC from the viewpoint of Pontryagin’s minimum principle. In Section 4, we propose FBSM for ML-POSC and prove its convergence. In Section 5, we apply FBSM to the linear-quadratic-Gaussian (LQG) problem. In Section 6, we verify the convergence of FBSM by numerical experiments. In Section 7, we discuss our work. In Appendix A, we briefly review Pontryagin’s minimum principle of deterministic control. In Appendix B, we derive the system of HJB-FP equations of MFSC from the viewpoint of Pontryagin’s minimum principle. In Appendix C, we show the detailed derivations of our results.

2. Memory-Limited Partially Observable Stochastic Control

In this section, we briefly review the formulation of ML-POSC [14], which is the stochastic optimal control problem under incomplete information and memory limitation.

2.1. Problem Formulation

This subsection outlines the formulation of ML-POSC [14]. The state of the system $x_t\in {\mathbb{R}}^{d_x}$ at time $t\in [0,T]$ evolves by the following stochastic differential equation (SDE):

$$\begin{array}{cc}\hfill d{x}_t& =b(t,x_t,u_t)dt+\sigma (t,x_t,u_t)d{\omega }_t,\hfill \end{array}$$

(1)

where $x_0$ obeys $p_0\left(x_0\right)$, $u_t\in {\mathbb{R}}^{d_u}$ is the control, and ${\omega }_t\in {\mathbb{R}}^{d_{\omega }}$ is the standard Wiener process. In COSC [15,16,17,18], because the controller can completely observe the state $x_t$, it determines the control $u_t$ based on the state $x_t$ as $u_t=u(t,x_t)$. By contrast, in POSC [9,10,11,12,13] and ML-POSC [14], the controller cannot directly observe the state $x_t$ and instead obtains the observation $y_t\in {\mathbb{R}}^{d_y}$, which evolves by the following SDE:

$$\begin{array}{cc}\hfill d{y}_t& =h(t,x_t)dt+\gamma \left(t\right)d{\nu }_t,\hfill \end{array}$$

(2)

where $y_0$ obeys $p_0\left(y_0\right)$, and ${\nu }_t\in {\mathbb{R}}^{d_{\nu }}$ is the standard Wiener process. In POSC [9,10,11,12,13], because the controller can completely memorize the observation history $y_{0:t}:=\left\{y_{\tau }\right|\tau \in [0,t]\}$, it determines the control $u_t$ based on the observation history $y_{0:t}$ as $u_t=u(t,y_{0:t})$. In ML-POSC [14], by contrast, because the controller cannot completely memorize the observation history $y_{0:t}$, it compresses the observation history $y_{0:t}$ into the finite-dimensional memory $z_t\in {\mathbb{R}}^{d_z}$, which evolves by the following SDE:

$$\begin{array}{c}\hfill d{z}_t=c(t,z_t,v_t)dt+\kappa (t,z_t,v_t)d{y}_t+\eta (t,z_t,v_t)d{\xi }_t,\end{array}$$

(3)

where $z_0$ obeys $p_0\left(z_0\right)$, $v_t\in {\mathbb{R}}^{d_v}$ is the control, and ${\xi }_t\in {\mathbb{R}}^{d_{\xi }}$ is the standard Wiener process. The memory dimension $d_z$ is determined by the available memory size of the controller. In addition, the memory noise ${\xi }_t$ represents the intrinsic stochasticity of the memory to be used. Therefore, unlike the conventional POSC, ML-POSC can explicitly take into account the memory size and noise of the controller. Furthermore, because the memory dynamics (3) depends on the memory control $v_t$, it can be optimized through the memory control $v_t$, which is expected to realize the optimal compression of the observation history $y_{0:t}$ into the limited memory $z_t$. In ML-POSC [14], the controller determines the state control $u_t$ and the memory control $v_t$ based on the memory $z_t$ as follows:

$$\begin{array}{c}\hfill u_t=u(t,z_t),v_t=v(t,z_t).\end{array}$$

(4)

The objective function of ML-POSC is given by the following expected cumulative cost function:

$$\begin{array}{c}\hfill J[u,v]:={\mathbb{E}}_{p(x_{0:T},y_{0:T},z_{0:T};u,v)}\left[{\int }_0^{T}f(t,x_t,u_t,v_t)dt+g\left(x_T\right)\right],\end{array}$$

(5)

where f is the cost function, g is the terminal cost function, $p(x_{0:T},y_{0:T},z_{0:T};u,v)$ is the probability of $x_{0:T}$, $y_{0:T}$, and $z_{0:T}$ given u and v as parameters, and ${\mathbb{E}}_p[·]$ is the expectation with respect to the probability p. Because the cost function f depends on the memory control $v_t$, ML-POSC can explicitly take into account the memory control cost, which is also impossible with the conventional POSC.

ML-POSC is the problem of finding the optimal state control function $u^{*}$ and the optimal memory control function $v^{*}$ that minimize the expected cumulative cost function $J[u,v]$ as follows:

$$\begin{array}{c}\hfill u^{*},v^{*}:=arg\underset{u,v}{min}J[u,v].\end{array}$$

(6)

ML-POSC first formulates the finite-dimensional and stochastic memory dynamics explicitly, then optimizes the memory control by considering the memory control cost. As a result, unlike the conventional POSC, ML-POSC is a practical framework for memory-limited controllers where the memory size, noise, and cost are imposed and non-negligible.

The previous work [14] has shown the validity and effectiveness of ML-POSC. In the LQG problem of conventional POSC, the observation history $y_{0:T}$ can be compressed into the Kalman filter without a loss of performance [10,18,43]. Because the Kalman filter is finite-dimensional, it can be interpreted as the finite-dimensional memory $z_t$ and discussed in terms of ML-POSC. The previous work [14] has proven that the optimal memory dynamics of ML-POSC become the Kalman filter in this problem, which indicates that ML-POSC is a consistent framework with the conventional POSC. Furthermore, the previous work [14] has demonstrated the effectiveness of ML-POSC in the LQG problem with memory limitation and in the non-LQG problem by numerical experiments.

2.2. Problem Reformulation

Although the formulation of ML-POSC in the previous subsection is intuitive, it is inconvenient for further mathematical investigations. To address this problem, we reformulate ML-POSC in this subsection. The formulation in this subsection is simpler and more general than that in the previous subsection.

First, we define an extended state $s_t$ as follows:

$$\begin{array}{c}\hfill s_t:=\left(\begin{array}{c}{x}_t\\ z_t\end{array}\right)\in {\mathbb{R}}^{d_s},\end{array}$$

(7)

where $d_s=d_x+d_z$. The extended state $s_t$ evolves by the following SDE:

$$\begin{array}{c}\hfill d{s}_t=\tilde{b}(t,s_t,{\tilde{u}}_t)dt+\tilde{\sigma }(t,s_t,{\tilde{u}}_t)d{\tilde{\omega }}_t,\end{array}$$

(8)

where $s_0$ obeys $p_0\left(s_0\right)$, ${\tilde{u}}_t\in {\mathbb{R}}^{d_{\tilde{u}}}$ is the control, and ${\tilde{\omega }}_t\in {\mathbb{R}}^{d_{\tilde{\omega }}}$ is the standard Wiener process. ML-POSC determines the control ${\tilde{u}}_t\in {\mathbb{R}}^{d_{\tilde{u}}}$ based on the memory $z_t$ as follows:

$$\begin{array}{c}\hfill {\tilde{u}}_t=\tilde{u}(t,z_t).\end{array}$$

(9)

The extended state SDE (8) includes the previous SDEs (1)–(3) as a special case because they can be represented as follows:

$$\begin{array}{cc}\hfill & d{s}_t=\left(\begin{array}{c}b(t,x_t,u_t)\\ c(t,z_t,v_t)+\kappa (t,z_t,v_t)h(t,x_t)\end{array}\right)dt\hfill \\ \hfill & +\left(\begin{array}{ccc}\sigma (t,x_t,u_t)& O& O\\ O& \kappa (t,z_t,v_t)\gamma \left(t\right)& \eta (t,z_t,v_t)\end{array}\right)\left(\begin{array}{c}d{\omega }_t\\ d{\nu }_t\\ d{\xi }_t\end{array}\right),\hfill \end{array}$$

(10)

where $p_0\left(s_0\right)=p_0\left(x_0\right)p_0\left(z_0\right)$.

The objective function of ML-POSC is given by the following expected cumulative cost function:

$$\begin{array}{c}\hfill J\left[\tilde{u}\right]:={\mathbb{E}}_{p(s_{0:T};\tilde{u})}\left[{\int }_0^T\tilde{f}(t,s_t,{\tilde{u}}_t)dt+\tilde{g}\left(s_T\right)\right].\end{array}$$

(11)

where $\tilde{f}$ is the cost function and $\tilde{g}$ is the terminal cost function. It is obvious that this objective function (11) is more general than that in the previous subsection (5).

ML-POSC is the problem of finding the optimal control function ${\tilde{u}}^{*}$ that minimizes the expected cumulative cost function $J\left[\tilde{u}\right]$ as follows:

$$\begin{array}{c}\hfill {\tilde{u}}^{*}:=arg\underset{\tilde{u}}{min}J\left[\tilde{u}\right].\end{array}$$

(12)

In the following sections, we mainly consider the formulation of this subsection because it is simpler and more general than that in the previous subsection. Moreover, we omit $\widetilde{·}$ for simplicity of notation.

3. Pontryagin’s Minimum Principle

If the control $u_t$ is determined based on the extended state $s_t$ as $u_t=u(t,s_t)$, ML-POSC is the same problem with COSC of the extended state, and its optimality conditions can be obtained in the conventional way [15,16,17,18]. In reality, however, because ML-POSC determines the control $u_t$ based only on the memory $z_t$ as $u_t=u(t,z_t)$, its optimality conditions cannot be obtained in a similar way as COSC. In the previous work [14], the optimality conditions of ML-POSC were obtained by employing a mathematical technique of MFSC [30,31].

In this section, we obtain the optimality conditions of ML-POSC by employing Pontryagin’s minimum principle [22,23,24,25] on the probability density function space (Figure 2 (bottom right)). The conventional approach in ML-POSC [14] and MFSC [30,31] can be interpreted as a conversion from Bellman’s dynamic programming principle (Figure 2 (top right)) to Pontryagin’s minimum principle (Figure 2 (bottom right)) on the probability density function space.

In Appendix A, we briefly review Pontryagin’s minimum principle in deterministic control (Figure 2 (left)). In this section, we obtain the optimality conditions of ML-POSC in a similar way as Appendix A (Figure 2 (right)). Furthermore, in Appendix B, we obtain the optimality conditions of MFSC in a similar way as Appendix A (Figure 2 (right)). MFSC is more general than ML-POSC except for the partial observability. In particular, the expected Hamiltonian is non-linear with respect to the probability density function in MFSC, while it is linear in ML-POSC.

Although our derivations are formal, not analytical, and more mathematically rigorous proofs remain future challenges, our results are consistent with the conventional results of COSC [15,16,17,18], ML-POSC [14], and MFSC [26,27,28,30,31], and also provide a useful perspective in proposing an algorithm.

3.1. Preliminary

In this subsection, we show a useful result in obtaining Pontryagin’s minimum principle. Given arbitrary control functions u and $u^{\prime }$, $J\left[u\right]-J\left[u^{\prime }\right]$ can be calculated as follows:

$$\begin{array}{c}\hfill J\left[u\right]-J\left[u^{\prime }\right]={\int }_0^T\left({\mathbb{E}}_{p(t,s)}\left[\mathcal{H}(t,s,u,w^{\prime })\right]-{\mathbb{E}}_{p(t,s)}\left[\mathcal{H}(t,s,u^{\prime },w^{\prime })\right]\right)dt,\end{array}$$

(13)

where $\mathcal{H}$ is the Hamiltonian, which is defined as follows:

$$\begin{array}{c}\hfill \mathcal{H}\left(t,s,u,w\right):=f(t,s,u)+{\mathcal{L}}_{u}w(t,s).\end{array}$$

(14)

${\mathcal{L}}_u$ is the backward diffusion operator, which is defined as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{u}w(t,s)& :=\sum _{i=1}^{d_s}{b}_i(t,s,u)\frac{\partial w(t,s)}{\partial s_i}+\frac{1}{2}\sum _{i,j=1}^{d_s}{D}_{ij}(t,s,u)\frac{{\partial }^{2}w(t,s)}{\partial s_i\partial s_j},\hfill \end{array}$$

(15)

where $D(t,s,u):=\sigma (t,s,u){\sigma }^{\top }(t,s,u)$. $w^{\prime }(t,s)$ is the solution of the following Hamilton–Jacobi–Bellman (HJB) equation driven by $u^{\prime }$:

$$\begin{array}{c}\hfill -\frac{\partial w^{\prime }(t,s)}{\partial t}=\mathcal{H}\left(t,s,u^{\prime },w^{\prime }\right),\end{array}$$

(16)

where $w^{\prime }(T,s)=g\left(s\right)$. $p(t,s)$ is the solution of the following Fokker–Planck (FP) equation driven by u:

$$\begin{array}{c}\hfill \frac{\partial p(t,s)}{\partial t}={\mathcal{L}}_u^{†}p(t,s),\end{array}$$

(17)

where $p(0,s)=p_0\left(s\right)$. ${\mathcal{L}}_u^{†}$ is the forward diffusion operator, which is defined as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_u^{†}p(t,s)& :=-\sum _{i=1}^{d_s}\frac{\partial \left(b_i(t,s,u)p(t,s)\right)}{\partial s_i}+\frac{1}{2}\sum _{i,j=1}^{d_s}\frac{{\partial }^2\left(D_{ij}(t,s,u)p(t,s)\right)}{\partial s_i\partial s_j}.\hfill \end{array}$$

(18)

${\mathcal{L}}_u^{†}$ is the conjugate of ${\mathcal{L}}_u$ as follows:

$$\begin{array}{c}\hfill \int w(t,s){\mathcal{L}}_u^{†}p(t,s)ds=\int p(t,s){\mathcal{L}}_{u}w(t,s)ds.\end{array}$$

(19)

We derive Equation (13) in Appendix C.1.

3.2. Necessary Condition

In this subsection, we show the necessary condition of the optimal control function of ML-POSC. It corresponds to Pontryagin’s minimum principle on the probability density function space (Figure 2 (bottom right)). If $u^{*}$ is the optimal control function of ML-POSC (12), then the following equation is satisfied:

$$\begin{array}{c}\hfill u^{*}(t,z)=arg\underset{u}{min}{\mathbb{E}}_{p_t^{*}\left(x\right|z)}\left[\mathcal{H}\left(t,s,u,w^{*}\right)\right],a.s.{}^{\forall }t\in [0,T],{}^{\forall }z\in {\mathbb{R}}^{d_z},\end{array}$$

(20)

where $w^{*}(t,s)$ is the solution of the following HJB equation driven by $u^{*}$:

$$\begin{array}{c}\hfill -\frac{\partial w^{*}(t,s)}{\partial t}=\mathcal{H}\left(t,s,u^{*},w^{*}\right),\end{array}$$

(21)

where $w^{*}(T,s)=g\left(s\right)$. $p_t^{*}\left(x\right|z):=p^{*}(t,s)/\int p^{*}(t,s)dx$ is the conditional probability density function of state x given memory z, and $p^{*}(t,s)$ is the solution of the following FP equation driven by $u^{*}$:

$$\begin{array}{c}\hfill \frac{\partial p^{*}(t,s)}{\partial t}={\mathcal{L}}_{u^{*}}^{†}{p}^{*}(t,s),\end{array}$$

(22)

where $p^{*}(0,s)=p_0\left(s\right)$. We derive this result in Appendix C.2.

In deterministic control, Pontryagin’s minimum principle can be expressed by the derivatives of the Hamiltonian (Figure 2 (bottom left)). Similarly, the system of HJB-FP Equations (21) and (22) can be expressed by the variations of the expected Hamiltonian

$$\begin{array}{c}\hfill \overline{\mathcal{H}}(t,p,u,w):={\mathbb{E}}_{p\left(s\right)}\left[\mathcal{H}\left(t,s,u,w\right)\right]\end{array}$$

(23)

as follows:

$$\begin{array}{cc}\hfill \frac{\partial p^{*}(t,s)}{\partial t}& =\frac{\delta \overline{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta w}\left(s\right),\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill -\frac{\partial w^{*}(t,s)}{\partial t}& =\frac{\delta \overline{\mathcal{H}}(t,p^{*},u^{*},w^{*})}{\delta p}\left(s\right),\hfill \end{array}$$

(25)

where $p^{*}(0,s)=p_0\left(s\right)$ and $w^{*}(T,s)=g\left(s\right)$ (Figure 2 (bottom right)). Therefore, the system of HJB-FP equations can be interpreted via Pontryagin’s minimum principle on the probability density function space.

3.3. Sufficient Condition

Pontryagin’s minimum principle (20) is only a necessary condition and generally not a sufficient condition. Pontryagin’s minimum principle (20) becomes a necessary and sufficient condition if the expected Hamiltonian $\overline{\mathcal{H}}(t,p,u,w)$ is convex with respect to p and u. We obtain this result in Appendix C.3.

3.4. Relationship with Bellman’s Dynamic Programming Principle

From Bellman’s dynamic programming principle on the probability density function space (Figure 2 (top right)) [14], the optimal control function of ML-POSC is given by the following equation:

$$\begin{array}{c}\hfill u^{*}(t,z,p)=arg\underset{u}{min}{\mathbb{E}}_{p\left(x\right|z)}\left[\mathcal{H}\left(t,s,u,\frac{\delta V^{*}(t,p)}{\delta p}\left(s\right)\right)\right],\end{array}$$

(26)

where $V^{*}(t,p)$ is the value function on the probability density function space, which is the solution of the following Bellman equation:

$$\begin{array}{c}\hfill -\frac{\partial V^{*}(t,p)}{\partial t}={\mathbb{E}}_{p\left(s\right)}\left[\mathcal{H}\left(t,s,u^{*},\frac{\delta V^{*}(t,p)}{\delta p}\left(s\right)\right)\right],\end{array}$$

(27)

where $V^{*}(T,p)={\mathbb{E}}_{p\left(s\right)}\left[g\left(s\right)\right]$. More specifically, the optimal control function of ML-POSC is given by $u^{*}(t,z)=u^{*}(t,z,p^{*})$, where $p^{*}$ is the solution of the FP Equation (22).

Because the Bellman Equation (27) is a functional differential equation, it cannot be solved even numerically. To resolve this problem, the previous work [14] converted the Bellman Equation (27) into the HJB Equation (21) by defining

$$\begin{array}{c}\hfill w^{*}(t,s):=\frac{\delta V^{*}(t,p^{*})}{\delta p}\left(s\right),\end{array}$$

(28)

where $p^{*}$ is the solution of FP Equation (22). This approach can be interpreted as the conversion from Bellman’s dynamic programming principle (Figure 2 (top right)) to Pontryagin’s minimum principle (Figure 2 (bottom right)) on the probability density function space.

3.5. Relationship with Completely Observable Stochastic Control

In the COSC of the extended state, the control $u_t$ is determined based on the extended state $s_t$ as $u_t=u(t,s_t)$. Therefore, in the COSC of the extended state, Pontryagin’s minimum principle on the probability density function space is given by the following equation:

$$\begin{array}{c}\hfill u^{*}(t,s)=arg\underset{u}{min}\mathcal{H}\left(t,s,u,w^{*}\right),a.s.{}^{\forall }t\in [0,T],{}^{\forall }s\in {\mathbb{R}}^{d_s},\end{array}$$

(29)

where $w^{*}(t,s)$ is the solution of the HJB Equation (21). Because this proof is almost identical to that of Section 3.2, it is omitted in this paper.

While the optimal control function of ML-POSC (20) depends on the FP equation and the HJB equation, the optimal control function of COSC (29) depends only on the HJB equation. From this nice property of COSC, Equation (29) is not only a necessary condition but also a sufficient condition without assuming the convexity of the expected Hamiltonian. We derive this result in Appendix C.4.

This result is consistent with the conventional result of COSC [15,16,17,18]. Unlike ML-POSC and MFSC, COSC can be solved by Bellman’s dynamic programming principle on the state space. In COSC, Pontryagin’s minimum principle on the probability density function space is equivalent to Bellman’s dynamic programming principle on the state space. Because Bellman’s dynamic programming principle on the state space is a necessary and sufficient condition, Pontryagin’s minimum principle on the probability density function space may also become a necessary and sufficient condition.

4. Forward-Backward Sweep Method

In this section, we propose FBSM for ML-POSC and then prove its convergence by employing the interpretation of the system of HJB-FP equations by Pontryagin’s minimum principle introduced in the previous section.

4.1. Forward-Backward Sweep Method

In this subsection, we propose FBSM for ML-POSC, which is summarized in Algorithm 1. FBSM is an algorithm to compute the forward FP equation and the backward HJB equation alternately. More specifically, in the initial step of FBSM, we initialize the control function $u_{0:T-dt}^0$ and obtain $p_{0:T}^0$ by computing the FP equation forward in time from the initial condition. In the backward step, we obtain $w_{0:T}^1$ by computing the HJB equation backward in time from the terminal condition and simultaneously update the control function from $u_{0:T-dt}^0$ to $u_{0:T-dt}^1$ by minimizing the conditional expected Hamiltonian. In the forward step, we obtain $p_{0:T}^2$ by computing the FP equation forward in time from the initial condition and simultaneously update the control function from $u_{0:T-dt}^1$ to $u_{0:T-dt}^2$ by minimizing the conditional expected Hamiltonian. By iterating the backward and forward steps, the objective function of ML-POSC $J\left[u_{0:T-dt}^k\right]$ monotonically decreases and finally converges to the local minimum at which the control function of ML-POSC $u_{0:T-dt}^k$ satisfies Pontryagin’s minimum principle.

Pontryagin’s minimum principle is only a necessary condition of the optimal control function, not a sufficient condition. Therefore, the control function obtained by FBSM is not necessarily the global optimum except in the case where the expected Hamiltonian is convex. Nevertheless, the control function obtained by FBSM is expected to be superior to most control functions because it is locally optimal.

FBSM has been used in deterministic control [32,34,35,38] and MFSC [39,40,41,42]. However, the convergence of FBSM for these problems is not guaranteed because the backward dynamics depend on the forward dynamics even without the optimal control function (Figure 1c,d). In contrast, the convergence of FBSM is guaranteed in ML-POSC because the backward HJB equation does not depend on the forward FP equation without the optimal control function (Figure 1b). More specifically, in FBSM for ML-POSC, the objective function $J\left[u_{0:T-dt}^k\right]$ monotonically decreases and finally converges to Pontryagin’s minimum principle. In the following subsections, we prove this nice property of FBSM for ML-POSC.

Algorithm 1: Forward-Backward Sweep Method (FBSM)

//— Initial step —// $k\leftarrow 0$ $p_0^k\left(s\right)\leftarrow p_0\left(s\right)$ for $t=0$ to $T-dt$ do    Initialize $u_t^k\left(z\right)$    $p_{t+dt}^k\left(s\right)\leftarrow p_t^k\left(s\right)+{\mathcal{L}}_{u_t^k}^{†}{p}_t^k\left(s\right)dt$ end for while $J\left[u_{0:T-dt}^k\right]$ do not converge do    if k is even then      //— Backward step —//      $w_T^{k+1}\left(s\right)\leftarrow g\left(s\right)$      for $t=T-dt$ to 0 do         $u_t^{k+1}\left(z\right)\leftarrow arg{min}_u{\mathbb{E}}_{p_t^k\left(x\right|z)}\left[\mathcal{H}(t,s,u,w_{t+dt}^{k+1})\right]$         $w_t^{k+1}\left(s\right)\leftarrow w_{t+dt}^{k+1}\left(s\right)+\mathcal{H}(t,s,u_t^{k+1},w_{t+dt}^{k+1})dt$      end for    else      //— Forward step —//      $p_0^{k+1}\left(s\right)\leftarrow p_0\left(s\right)$      for $t=0$ to $T-dt$ do         $u_t^{k+1}\left(z\right)\leftarrow arg{min}_u{\mathbb{E}}_{p_t^{k+1}\left(x\right|z)}\left[\mathcal{H}(t,s,u,w_{t+dt}^k)\right]$         $p_{t+dt}^{k+1}\left(s\right)\leftarrow p_t^{k+1}\left(s\right)+{\mathcal{L}}_{u_t^{k+1}}^{†}{p}_t^{k+1}\left(s\right)dt$      end for    end if    $k\leftarrow k+1$ end while return $u_{0:T-dt}^k$

4.2. Preliminary

In this subsection, we show an important result in proving the convergence of FBSM for ML-POSC. We suppose that $u_{0:t-dt,t+dt:T-dt}:=\{u_0,...,u_{t-dt},u_{t+dt},...,u_{T-dt}\}$ is given and only $u_t$ is optimized as follows:

$$\begin{array}{c}\hfill u_t^{*}:=arg\underset{u_t}{min}J\left[u_{0:T-dt}\right].\end{array}$$

(30)

In ML-POSC, $u_t^{*}$ can be calculated as follows:

$$\begin{array}{c}\hfill u_t^{*}\left(z\right)=arg\underset{u_t}{min}{\mathbb{E}}_{p_t\left(x\right|z)}\left[\mathcal{H}\left(t,s,u_t,w_{t+dt}\right)\right],a.s.{}^{\forall }z\in {\mathbb{R}}^{d_z},\end{array}$$

(31)

where $w_{t+dt}\left(s\right)$ is the solution of the following time-discretized HJB equation driven by $u_{t+dt:T-dt}$:

$$\begin{array}{c}\hfill w_{\tau }\left(s\right)=w_{\tau +dt}\left(s\right)+\mathcal{H}\left(\tau ,s,u_{\tau },w_{\tau +dt}\right)dt,\tau \in \{t+dt,...,T-dt\},\end{array}$$

(32)

where $w_T\left(s\right)=g\left(s\right)$. $p_t\left(x\right|z):=p_t\left(s\right)/\int p_t\left(s\right)dx$ is the conditional probability density function of state x given memory z, and $p_t\left(s\right)$ is the solution of the following time-discretized FP equation driven by $u_{0:t-dt}$:

$$\begin{array}{c}\hfill p_{\tau +dt}\left(s\right)=p_{\tau }\left(s\right)+{\mathcal{L}}_{u_{\tau }}^{†}{p}_{\tau }\left(s\right)dt,\tau \in \{0,...,t-dt\},\end{array}$$

(33)

where $p_0\left(s\right)$. Equation (31) is obtained by the similar way to Pontyragin’s minimum principle in Appendix C.5 and also by the time discretization method in Appendix C.6.

Importantly, $w_{t+dt}$ does not depend on $u_t$ in ML-POSC (Figure 3a) while ${\lambda }_{t+dt}$ and $w_{t+dt}$ depend on $u_t$ in deterministic control (Figure 3b) and MFSC (Figure 3c), respectively. Therefore, $u_t^{*}$ can be obtained without modifying $w_{t+dt}$ in ML-POSC, which is essentially different from deterministic control and MFSC. From this nice property, the convergence of FBSM is guaranteed in ML-POSC.

4.3. Monotonicity

In FBSM for ML-POSC, the objective function is monotonically non-increasing with respect to the update of the control function at each time step. More specifically,

$$\begin{array}{c}\hfill J[u_{0:t-dt}^k,u_{t:T-dt}^{k+1}]\le J[u_{0:t}^k,u_{t+dt:T-dt}^{k+1}]\end{array}$$

(34)

is satisfied in the backward step, and

$$\begin{array}{c}\hfill J[u_{0:t-dt}^{k+1},u_{t:T-dt}^k]\ge J[u_{0:t}^{k+1},u_{t+dt:T-dt}^k]\end{array}$$

(35)

is satisfied in the forward step. We prove this result in Appendix C.7. Furthermore, in FBSM for ML-POSC, the objective function is monotonically non-increasing with respect to the update of the control function at each iteration step as follows:

$$\begin{array}{c}\hfill J\left[u_{0:T-dt}^{k+1}\right]\le J\left[u_{0:T-dt}^k\right].\end{array}$$

(36)

Equation (36) is obviously satisfied from Equations (34) and (35).

4.4. Convergence to Pontryagin’s Minimum Principle

We assume that $J\left[u_{0:T-dt}\right]$ has a lower bound. From Equation (36), FBSM for ML-POSC is guaranteed to converge to the local minimum. Furthermore, we assume that if the candidate of $u_t^{k+1}$ includes $u_t^k$, then set $u_t^{k+1}$ at $u_t^k$. Under these assumptions, FBSM for ML-POSC converges to Pontryagin’s minimum principle (20). More specifically, if $J\left[u_{0:T-dt}^{k+1}\right]=J\left[u_{0:T-dt}^k\right]$ holds, $u_{0:T-dt}^{k+1}$ satisfies Pontryagin’s minimum principle (20). We prove this result in Appendix C.8.

Therefore, unlike deterministic control and MFSC, in FBSM for ML-POSC, the objective function $J\left[u_{0:T-dt}^k\right]$ monotonically decreases and finally converges to the local minimum at which the control function $u_{0:T-dt}^k$ satisfies Pontryagin’s minimum principle (20).

5. Linear-Quadratic-Gaussian Problem

In this section, we apply FBSM to the LQG problem of ML-POSC [14]. In the LQG problem of ML-POSC, the system of HJB-FP equations is reduced from partial differential equations to ordinary differential equations.

5.1. Problem Formulation

In the LQG problem of ML-POSC, the extended state SDE (8) is given as follows [14]:

$$\begin{array}{c}\hfill d{s}_t=\left(A\left(t\right)s_t+B\left(t\right)u_t\right)dt+\sigma \left(t\right)d{\omega }_t,\end{array}$$

(37)

where $s_0$ obeys the Gaussian distribution $p_0\left(s_0\right):=\mathcal{N}\left(s_0\left|{\mu }_0,{\mathsf{\Lambda }}_0\right.\right)$ where ${\mu }_0$ is the mean vector and ${\mathsf{\Lambda }}_0$ is the precision matrix. The objective function (11) is given as follows:

$$\begin{array}{c}\hfill J\left[u\right]:={\mathbb{E}}_{p(s_{0:T};u)}\left[{\int }_0^T\left(s_t^{\top }Q\left(t\right)s_t+u_t^{\top }R\left(t\right)u_t\right)dt+s_T^{\top }P{s}_T\right],\end{array}$$

(38)

where $Q\left(t\right)⪰O$, $R\left(t\right)\succ O$, and $P⪰O$. The LQG problem of ML-POSC is the problem of finding the optimal control function $u^{*}$ that minimizes the objective function $J\left[u\right]$ as follows:

$$\begin{array}{c}\hfill u^{*}:=arg\underset{u}{min}J\left[u\right].\end{array}$$

(39)

5.2. Pontryagin’s Minimum Principle

In the LQG problem of ML-POSC, Pontryagin’s minimum principle (20) can be calculated as follows [14]:

$$\begin{array}{c}\hfill u^{*}(t,z)=-R^{-1}{B}^{\top }\left(\mathsf{\Pi }K\left(\mathsf{\Lambda }\right)(s-\mu )+\mathsf{\Psi }\mu \right),a.s.{}^{\forall }t\in [0,T],{}^{\forall }z\in {\mathbb{R}}^{d_z},\end{array}$$

(40)

where $K\left(\mathsf{\Lambda }\right)$ is defined as follows:

$$\begin{array}{c}\hfill K\left(\mathsf{\Lambda }\right):=\left(\begin{array}{cc}O& {\mathsf{\Lambda }}_{xx}^{-1}{\mathsf{\Lambda }}_{xz}\\ O& I\end{array}\right),\end{array}$$

(41)

where $\mu \left(t\right)$ and $\mathsf{\Lambda }\left(t\right)$ are the mean vector and the precision matrix of the extended state, respectively, which correspond to the solution of the FP Equation (22). We note that ${\mathbb{E}}_{p_t\left(z\right|x)}\left[s\right]=K\left(\mathsf{\Lambda }\right)(s-\mu )+\mu$ is satisfied. $\mu \left(t\right)$ and $\mathsf{\Lambda }\left(t\right)$ are the solutions of the following ordinary differential equations (ODEs):

$$\begin{array}{cc}\hfill \dot{\mu }& =\left(A-B{R}^{-1}{B}^{\top }\mathsf{\Psi }\right)\mu ,\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill \dot{\mathsf{\Lambda }}& =-{\left(A-B{R}^{-1}{B}^{\top }\mathsf{\Pi }K\left(\mathsf{\Lambda }\right)\right)}^{\top }\mathsf{\Lambda }-\mathsf{\Lambda }\left(A-B{R}^{-1}{B}^{\top }\mathsf{\Pi }K\left(\mathsf{\Lambda }\right)\right)-\mathsf{\Lambda }\sigma {\sigma }^{\top }\mathsf{\Lambda },\hfill \end{array}$$

(43)

where $\mu \left(0\right)={\mu }_0$ and $\mathsf{\Lambda }\left(0\right)={\mathsf{\Lambda }}_0$. $\mathsf{\Psi }\left(t\right)$ and $\mathsf{\Pi }\left(t\right)$ are the control gain matrices of the deterministic and stochastic extended state, respectively, which correspond to the solution of the HJB Equation (21). $\mathsf{\Psi }\left(t\right)$ and $\mathsf{\Pi }\left(t\right)$ are the solutions of the following ODEs:

$$\begin{array}{cc}\hfill -\dot{\mathsf{\Psi }}& =Q+A^{\top }\mathsf{\Psi }+\mathsf{\Psi }A-\mathsf{\Psi }B{R}^{-1}{B}^{\top }\mathsf{\Psi },\hfill \end{array}$$

(44)

$$\begin{array}{cc}\hfill -\dot{\mathsf{\Pi }}& =Q+A^{\top }\mathsf{\Pi }+\mathsf{\Pi }A-\mathsf{\Pi }B{R}^{-1}{B}^{\top }\mathsf{\Pi }+{(I-K\left(\mathsf{\Lambda }\right))}^{\top }\mathsf{\Pi }B{R}^{-1}{B}^{\top }\mathsf{\Pi }(I-K\left(\mathsf{\Lambda }\right)),\hfill \end{array}$$

(45)

where $\mathsf{\Psi }\left(T\right)=\mathsf{\Pi }\left(T\right)=P$. The ODE of $\mathsf{\Psi }$ (44) is the Riccati equation [16,17,18], which also appears in the LQG problem of COSC. In contrast, the ODE of $\mathsf{\Pi }$ (45) is the partially observable Riccati equation [14], which appears only in the LQG problem of ML-POSC. The above result is obtained in [14].

The ODE of $\mathsf{\Psi }$ (44) can be solved backward in time from the terminal condition. Using $\mathsf{\Psi }$, the ODE of $\mu$ (42) can be solved forward in time from the initial condition. In contrast, the ODEs of $\mathsf{\Pi }$ (45) and $\mathsf{\Lambda }$ (43) cannot be solved in a similar way as the ODEs of $\mathsf{\Psi }$ (44) and $\mu$ (42) because they interact with each other, which is a similar problem to the system of HJB-FP equations.

5.3. Forward-Backward Sweep Method

In the LQG problem of ML-POSC, FBSM is reduced from Algorithm 1 to Algorithm 2. $\mathcal{F}(\mathsf{\Lambda },\mathsf{\Pi })$ and $\mathcal{G}(\mathsf{\Lambda },\mathsf{\Pi })$ are defined by the right-hand sides of the ODEs of $\mathsf{\Lambda }$ (43) and $\mathsf{\Pi }$ (45), respectively, as follows:

$$\begin{array}{cc}\hfill \mathcal{F}(\mathsf{\Lambda },\mathsf{\Pi })& :=-{\left(A-B{R}^{-1}{B}^{\top }\mathsf{\Pi }K\left(\mathsf{\Lambda }\right)\right)}^{\top }\mathsf{\Lambda }-\mathsf{\Lambda }\left(A-B{R}^{-1}{B}^{\top }\mathsf{\Pi }K\left(\mathsf{\Lambda }\right)\right)-\mathsf{\Lambda }\sigma {\sigma }^{\top }\mathsf{\Lambda },\hfill \\ \hfill \mathcal{G}(\mathsf{\Lambda },\mathsf{\Pi })& :=Q+A^{\top }\mathsf{\Pi }+\mathsf{\Pi }A-\mathsf{\Pi }B{R}^{-1}{B}^{\top }\mathsf{\Pi }+{(I-K\left(\mathsf{\Lambda }\right))}^{\top }\mathsf{\Pi }B{R}^{-1}{B}^{\top }\mathsf{\Pi }(I-K\left(\mathsf{\Lambda }\right)).\hfill \end{array}$$

This result is obtained in Appendix C.9. Importantly, in the LQG problem of ML-POSC, FBSM computes the ODEs of $\mathsf{\Lambda }$ (43) and $\mathsf{\Pi }$ (45) instead of the FP Equation (22) and the HJB Equation (21).

Algorithm 2: Forward-Backward Sweep Method (FBSM) in the LQG problem

//— Initial step —// $k\leftarrow 0$ ${\mathsf{\Lambda }}_0^k\leftarrow {\mathsf{\Lambda }}_0$ for $t=0$ to $T-dt$ do    Initialize ${\mathsf{\Pi }}_{t+dt}^k$    ${\mathsf{\Lambda }}_{t+dt}^k\leftarrow {\mathsf{\Lambda }}_t^k+\mathcal{F}({\mathsf{\Lambda }}_t^k,{\mathsf{\Pi }}_{t+dt}^k)dt$ end for while $J\left[u_{0:T-dt}^k\right]$ do not converge do    if k is even then      //— Backward step —//      ${\mathsf{\Pi }}_T^{k+1}\leftarrow P$      for $t=T-dt$ to 0 do         ${\mathsf{\Pi }}_t^{k+1}\leftarrow {\mathsf{\Pi }}_{t+dt}^{k+1}+\mathcal{G}({\mathsf{\Lambda }}_t^k,{\mathsf{\Pi }}_{t+dt}^{k+1})dt$      end for    else      //— Forward step —//      ${\mathsf{\Lambda }}_0^{k+1}\leftarrow {\mathsf{\Lambda }}_0$      for $t=0$ to $T-dt$ do         ${\mathsf{\Lambda }}_{t+dt}^{k+1}\leftarrow {\mathsf{\Lambda }}_t^{k+1}+\mathcal{F}({\mathsf{\Lambda }}_t^{k+1},{\mathsf{\Pi }}_{t+dt}^k)dt$      end for    end if    $k\leftarrow k+1$ end while return $u_{0:T-dt}^k$

6. Numerical Experiments

In this section, we verify the convergence of FBSM in ML-POSC by performing numerical experiments on the LQG and non-LQG problems. The setting of the numerical experiments is the same as the previous work [14].

6.1. LQG Problem

In this subsection, we verify the convergence of FBSM for ML-POSC by conducting a numerical experiment on the LQG problem. We consider state $x_t\in \mathbb{R}$, observation $y_t\in \mathbb{R}$, and memory $z_t\in \mathbb{R}$, which evolve by the following SDEs:

$$\begin{array}{cc}\hfill d{x}_t& =\left(x_t+u_t\right)dt+d{\omega }_t,\hfill \end{array}$$

(46)

$$\begin{array}{cc}\hfill d{y}_t& =x_{t}dt+d{\nu }_t,\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill d{z}_t& =v_{t}dt+d{y}_t,\hfill \end{array}$$

(48)

where $x_0$ and $z_0$ obey the standard Gaussian distributions, $y_0$ is an arbitrary real number, ${\omega }_t\in \mathbb{R}$ and ${\nu }_t\in \mathbb{R}$ are independent standard Wiener processes, and $u_t=u(t,z_t)\in \mathbb{R}$ and $v_t=v(t,z_t)\in \mathbb{R}$ are the controls. The objective function to be minimized is given as follows:

$$\begin{array}{c}\hfill J[u,v]:={\mathbb{E}}_{p(x_{0:10},y_{0:10},z_{0:10};u,v)}\left[{\int }_0^{10}\left(x_t^2+u_t^2+v_t^2\right)dt\right].\end{array}$$

(49)

Therefore, the objective of this problem is to minimize the state variance with small state and memory controls.

This problem corresponds to the LQG problem, which is defined by (37) and (38). By defining $s_t:=(x_t,z_t)\in {\mathbb{R}}^2$, ${\tilde{u}}_t:=(u_t,v_t)\in {\mathbb{R}}^2$, and ${\tilde{\omega }}_t:=({\omega }_t,{\nu }_t)\in {\mathbb{R}}^2$, the SDEs (46)–(48) can be rewritten as follows:

$$\begin{array}{cc}\hfill & d{s}_t=\left(\left(\begin{array}{cc}1& 0\\ 1& 0\end{array}\right)s_t+{\tilde{u}}_t\right)dt+d{\tilde{\omega }}_t,\hfill \end{array}$$

(50)

which corresponds to (37). Furthermore, the objective function (49) can be rewritten as follows:

$$\begin{array}{c}\hfill J\left[\tilde{u}\right]:={\mathbb{E}}_{p(s_{0:10};\tilde{u})}\left[{\int }_0^{10}\left(s_t^{\top }\left(\begin{array}{cc}1& 0\\ 0& 0\end{array}\right)s_t+{\tilde{u}}_t^{\top }{\tilde{u}}_t\right)dt\right],\end{array}$$

(51)

which corresponds to (38).

We apply the FBSM of the LQG problem (Algorithm 2) to this problem. ${\mathsf{\Pi }}^0\left(t\right)$ is initialized by ${\mathsf{\Pi }}^0\left(t\right)=O$. To solve the ODEs of ${\mathsf{\Pi }}^k\left(t\right)$ and ${\mathsf{\Lambda }}^k\left(t\right)$, we use the fourth-order Runge–Kutta method. Figure 4 shows the control gain matrix ${\mathsf{\Pi }}^k\left(t\right)\in {\mathbb{R}}^{2\times 2}$ and the precision matrix ${\mathsf{\Lambda }}^k\left(t\right)\in {\mathbb{R}}^{2\times 2}$ obtained by FBSM. The color of each curve represents the iteration k. The darkest curve corresponds to the first iteration $k=0$, and the brightest curve corresponds to the last iteration $k=50$. Importantly, ${\mathsf{\Pi }}^k\left(t\right)$ and ${\mathsf{\Lambda }}^k\left(t\right)$ converge with respect to the iteration k.

Figure 5a shows the objective function $J\left[u^k\right]$ with respect to iteration k. The objective function $J\left[u^k\right]$ monotonically decreases with respect to iteration k, which is consistent with Section 4.3. This monotonicity of FBSM is the nice property of ML-POSC that is not guaranteed in deterministic control and MFSC. The objective function $J\left[u^k\right]$ finally converges, and $u^k$ satisfies Pontryagin’s minimum principle from Section 4.4.

Figure 5b–d compare the performance of the control function $u^k$ at the first iteration $k=0$ and the last iteration $k=50$ by performing a stochastic simulation. At the first iteration $k=0$, the distributions of state and memory are unstable, and the cumulative cost diverges. In contrast, at the last iteration $k=50$, the distributions of state and memory are stabilized and the cumulative cost is smaller. This result indicates that FBSM improves the performance in ML-POSC.

Although Figure 5b–d look similar to Figure 2d–f in the previous work [14], they are comparing different things. While Figure 5b–d demonstrate the performance improvement by the FBSM iteration, the previous work [14] compares the performance of the partially observable Riccati Equation (45) with that of the conventional Riccati Equation (44).

6.2. Non-LQG Problem

In this subsection, we verify the convergence of FBSM in ML-POSC by conducting a numerical experiment on the non-LQG problem. We consider state $x_t\in \mathbb{R}$, observation $y_t\in \mathbb{R}$, and memory $z_t\in \mathbb{R}$, which evolve by the following SDEs:

$$\begin{array}{cc}\hfill d{x}_t& =u_{t}dt+d{\omega }_t,\hfill \end{array}$$

(52)

$$\begin{array}{cc}\hfill d{y}_t& =x_{t}dt+d{\nu }_t,\hfill \end{array}$$

(53)

$$\begin{array}{cc}\hfill d{z}_t& =d{y}_t,\hfill \end{array}$$

(54)

where $x_0$ and $z_0$ obey the Gaussian distributions $p_0\left(x_0\right)=\mathcal{N}\left(x_0\right|0,0.01)$ and $p_0\left(z_0\right)=\mathcal{N}\left(z_0\right|0,0.01)$, respectively. $y_0$ is an arbitrary real number, ${\omega }_t\in \mathbb{R}$ and ${\nu }_t\in \mathbb{R}$ are independent standard Wiener processes, and $u_t=u(t,z_t)\in \mathbb{R}$ is the control. For the sake of simplicity, memory control is not considered. The objective function to be minimized is given as follows:

$$\begin{array}{c}\hfill J\left[u\right]:={\mathbb{E}}_{p(x_{0:1},y_{0:1},z_{0:1};u)}\left[{\int }_0^1\left(Q(t,x_t)+u_t^2\right)dt+10x_1^2\right],\end{array}$$

(55)

where

$$\begin{array}{c}\hfill Q(t,x):=\left\{\begin{array}{cc}1000\hfill & (0.3\le t\le 0.6,0.1\le |x|\le 2.0),\hfill \\ 0\hfill & \left(others\right).\hfill \end{array}\right.\end{array}$$

(56)

The cost function is high in $0.3\le t\le 0.6$ and $0.1\le \left|x\right|\le 2.0$, which represents the obstacles. In addition, the terminal cost function is the lowest at $x=0$, which represents the desirable goal. Therefore, the system should avoid the obstacles and reach the goal with a small control. Because the cost function is non-quadratic, it is a non-LQG problem.

We apply the FBSM (Algorithm 1) to this problem. $u^0(t,z)$ is initialized by $u^0(t,z)=0$. To solve the HJB equation and the FP equation, we use the finite-difference method. Figure 6 shows $w^k(t,s)$ and $p^k(t,s)$ obtained by FBSM at the first iteration $k=0$ and at the last iteration $k=50$. From Appendix C.6, $w^k(t,s)$ is given as follows:

$$\begin{array}{c}\hfill w^k(t,s)={\mathbb{E}}_{p\left(s_{t+dt:1}\right|s_t=s;u^k)}\left[{\int }_t^1\left(Q(\tau ,x_{\tau })+{\left(u_{\tau }^k\right)}^2\right)d\tau +10x_1^2\right].\end{array}$$

(57)

Because $u^0(t,z)=0$, $w^0(t,s)$ reflects the cost function corresponding to the obstacles and the goal (Figure 6a–e). In contrast, because $u^{50}(t,z)\ne 0$, $w^{50}(t,s)$ becomes more complex (Figure 6f–j). In particular, while $w^0(t,s)$ does not depend on memory z, $w^{50}(t,s)$ depends on memory z, which indicates that the control function $u^{50}(t,z)$ is adjusted by the memory z. We note that $w^0(1,s)$ (Figure 6e) and $w^{50}(1,s)$ (Figure 6j) are the same because they are given by the terminal cost function as $w^0(1,s)=w^{50}(1,s)=10x^2$. Furthermore, while $p^0(t,s)$ is a unimodal distribution (Figure 6k–o), $p^{50}(t,s)$ is a bimodal distribution (Figure 6p–t), which can avoid the obstacles.

Figure 7a shows the objective function $J\left[u^k\right]$ with respect to iteration k. The objective function $J\left[u^k\right]$ monotonically decreases with respect to iteration k, which is consistent with Section 4.3. This monotonicity of FBSM is the nice property of ML-POSC that is not guaranteed in deterministic control and MFSC. The objective function $J\left[u^k\right]$ finally converges, and its $u^k$ satisfies Pontryagin’s minimum principle from Section 4.4.

Figure 7b,c compare the performance of the control function $u^k$ at the first iteration $k=0$ and the last iteration $k=50$ by conducting the stochastic simulation. At the first iteration $k=0$, the obstacles cannot be avoided, which results in a higher objective function. In contrast, at the last iteration $k=50$, the obstacles can be avoided, which results in a lower objective function. This result indicates that FBSM improves the performance in ML-POSC.

Although Figure 7b,c look similar to Figure 3a,b in the previous work [14], they are comparing different things. While Figure 7b,c demonstrate the performance improvement by the FBSM iteration, the previous work [14] compares the performance of ML-POSC with the local LQG approximation of the conventional POSC.

7. Discussion

In this work, we first showed that the system of HJB-FP equations corresponds to Pontryagin’s minimum principle on the probability density function space. Although the relationship between the system of HJB-FP equations and Pontryagin’s minimum principle has been briefly mentioned in MFSC [29,30,31], its details have not yet been investigated. We addressed this problem by deriving the system of HJB-FP equations in a similar way to Pontryagin’s minimum principle. We then proposed FBSM to ML-POSC. Although the convergence of FBSM is generally not guaranteed in deterministic control [32,34,35,38] and MFSC [39,40,41,42], we proved the convergence in ML-POSC by noting the fact that the update of the current control function does not affect the future HJB equation in ML-POSC. Therefore, ML-POSC is a special and nice class where FBSM is guaranteed to converge.

Our derivation of Pontryagin’s minimum principle on the probability density function space is formal, not analytical. Therefore, more mathematically rigorous proofs should be pursued in future work. Nevertheless, because our results are consistent with the conventional results of COSC [15,16,17,18], ML-POSC [14], and MFSC [26,27,28,30,31], they would be reliable except for special cases. Furthermore, our results provide a unified perspective on FBSM in deterministic control [32,34,35,38] and the fixed-point iteration method in MFSC [39,40,41,42], which have been studied independently. It clarifies the different properties of ML-POSC from deterministic control and MFSC, which ensures the convergence of FBSM.

The regularized FBSM has recently been proposed in deterministic control, which is guaranteed to converge even in the general deterministic control [44,45]. Our work gives an intuitive reason why the regularized FBSM is guaranteed to converge. In the regularized FBSM, the Hamiltonian is regularized, which makes the update of the control function smaller. When the regularization is sufficiently strong, the effect of the current control function on the future backward dynamics would be negligible. Therefore, the regularized FBSM of deterministic control would be guaranteed to converge for a similar reason to the FBSM of ML-POSC. However, the convergence of the regularized FBSM is much slower because the stronger regularization makes the update of the control function smaller. The FBSM of ML-POSC does not suffer from such a problem because the future backward dynamics already do not depend on the current control function without regularization.

Our work gives a hint about a modification of the fixed-point iteration method to ensure convergence in MFSC. Although the fixed-point iteration method is the most basic algorithm in MFSC, its convergence is not guaranteed [39,40,41,42]. Our work showed that the fixed-point iteration method is equivalent to the FBSM on the probability density function space. Therefore, the idea of regularized FBSM may also be applied to the fixed-point iteration method. More specifically, the fixed-point iteration method may be guaranteed to converge by regularizing the expected Hamiltonian.

In FBSM, we solve the HJB equation and the FP equation using the finite-difference method. However, because the finite-difference method is prone to the curse of dimensionality, it is difficult to solve high-dimensional ML-POSC. To resolve this problem, two directions can be considered. One direction is the policy iteration method [21,46,47]. Although the policy iteration method is almost the same as FBSM, only the update of the control function is different. While FBSM updates the system of HJB-FP equations and the control function simultaneously, the policy iteration method updates them separately. In the policy iteration method, the system of HJB-FP equations becomes linear, which can be solved by the sampling method [48,49,50]. Because the sampling method is more tractable than the finite-difference method, the policy iteration method may allow high-dimensional ML-POSC to be solved. Furthermore, the policy iteration method has recently been studied in MFSC [51,52,53]. However, its convergence is not guaranteed except for special cases in MFSC. In a similar way to FBSM, the convergence of the policy iteration method may be guaranteed in ML-POSC.

The other direction is machine learning. Neural network-based algorithms have recently been proposed in MFSC, which can solve high-dimensional problems efficiently [54,55]. By extending these algorithms, high-dimensional ML-POSC may be solved efficiently. Furthermore, unlike MFSC, the coupling of the HJB-FP equations is limited only to the optimal control function in ML-POSC. By exploiting this nice property, more efficient algorithms may be devised for ML-POSC.