Dynamic Expectation Maximization Algorithm for Estimation of Linear Systems with Colored Noise

Abstract

The free energy principle from neuroscience has recently gained traction as one of the most prominent brain theories that can emulate the brain’s perception and action in a bio-inspired manner. This renders the theory with the potential to hold the key for general artificial intelligence. Leveraging this potential, this paper aims to bridge the gap between neuroscience and robotics by reformulating an FEP-based inference scheme—Dynamic Expectation Maximization—into an algorithm that can perform simultaneous state, input, parameter, and noise hyperparameter estimation of any stable linear state space system subjected to colored noises. The resulting estimator was proved to be of the form of an augmented coupled linear estimator. Using this mathematical formulation, we proved that the estimation steps have theoretical guarantees of convergence. The algorithm was rigorously tested in simulation on a wide variety of linear systems with colored noises. The paper concludes by demonstrating the superior performance of DEM for parameter estimation under colored noise in simulation, when compared to the state-of-the-art estimators like Sub Space method, Prediction Error Minimization (PEM), and Expectation Maximization (EM) algorithm. These results contribute to the applicability of DEM as a robust learning algorithm for safe robotic applications.

1. Introduction

The free energy principle (FEP) has emerged from neuroscience as a unifying theory of the brain [1] and has begun to guide the search for a brain-inspired learning algorithm for robots. Many attempts have been made in this direction, including the state and input observer [2,3], the adaptive controller for robot manipulators [4,5,6], the body perception and action scheme for humanoid robots [7], the robot navigation of ground robots [8] etc. However, the design of a parameter estimation algorithm for linear systems with colored noise remains unexplored. Since the design of an accurate parameter estimator for dynamic systems sits at the core of control systems and robotics, the reformulation of FEP into a brain-inspired estimation algorithm has an influential impact on the industry and applied robotics.

A wide range of estimators have been proposed in the literature for linear time-invariant (LTI) systems [9], including the blind system identification [10,11,12]. However, most of them assume the noises to be temporally uncorrelated (white noise), which is often violated in practice. This results in biased estimation for the least-square (LS)-based methods [13], and an inaccurate convergence for the iterative methods [14]. Although many attempts have been made to solve this problem, mainly through bias compensation methods [15,16], none of them perform state, input, parameter, and noise estimation for systems with colored noises [17]. The only method that does it is the Dynamic Expectation Maximization (DEM) [18] algorithm, which uses FEP to invert a highly nonlinear and hierarchical brain model from sensory data. However, the disconnect between neuroscience and control system literature hinders the wide use of this method for practical robotics applications. Although FEP-based tools have already been applied to practical robotics applications [2,3,4,5,6,7,8,19,20], there is a gap in the literature on the applications of DEM owing to the mathematical formidability of the theory and its lack of formalism in the control systems domain. Therefore, it is important to reformulate DEM for the control systems audience. While DEM from computational neuroscience focuses on emulating the brain’s perception through the hierarchical abstraction of a number of interacting non-linear dynamic systems, our work focuses on reformulating it into a blind system-identification algorithm for an LTI system with colored noise, which is a well-known challenge in robotics. In this attempt, we keep all the brain-related approximations intact, thereby aiming for a biologically plausible parameter estimation algorithm.

According to an FEP proposed by Karl Friston, the brain’s inference mechanism is a gradient ascent over its free energy, where free energy is the information-theoretic measure that bounds the brain’s sensory surprisal [21]. FEP emerges as a unified brain theory [22] by providing a mathematical description of brain functions [23]; unifying action and perception [24]; connecting physiological constructs like memory, attention, value, reinforcement, and salience [23]; and remaining consistent with Freudian ideas [25]. Similarities of FEP with reinforcement learning [26], neural networks [21,27], PID controller [28], Kalman Filter [2] and active learning [24] open up possibilities for biologically plausible parameter estimation algorithms. Although FEP emerged as a brain theory, the recent works have pushed the boundaries towards systems that survive over time, such as social and cultural dynamics. Notable works include the variational approach to culture [29], collective intelligence [30], cumulative cultural evolution [31], etc.

Numerous methods have been proposed based on the FEP framework. Predictive coding [32] models perception through a hierarchy of dynamical systems [27] with the brain’s priors at the top, minimizing the prediction error at each level of the hierarchy. Bayesian message passing algorithms [33,34] use similar ideas for belief propagation. Active inference [24,35] uses FEP to model the brain’s action and perception under one framework. On the perception side, there are two main type of methods to deal with dynamic systems: variational filtering and generalized filtering. Variational filtering [18,36] uses mean-field approximation (conditional independence between densities), whereas generalized filtering [37] does not. DEM [18] is a type of variational filtering that uses a Laplace approximation [38] (a fixed-form assumption for the conditional density of variables), whereas [36] uses ensemble dynamics to model the free form of the conditional densities. We focus on DEM for this work.

DEM is an FEP-based variational inference algorithm that models the brain’s inference process as a maximization of its free energy for state, input, parameter, and noise estimation from data. Although the method shows high similarity to the variational inference [39], the key difference is in the use of generalized coordinates, which enables DEM to track the evolution of the trajectory of states instead of just the point estimates. This renders DEM with the capability of gracefully handling colored noises, a feature that conventional point estimators such as the Kalman Filter (KF) lacks. The modeling of noise color as analytic using generalized coordinates results in an improved state estimation under colored noise for LTI systems [2,3] and for nonlinear filtering [40], which directly improves the parameter estimation accuracy, making DEM a topic of interest [20]. This work directly impacts various sub-domains of robotics community: input estimation to the industrial automation community for fault detection systems, state estimation to the control systems community, parameter estimation to the system identification community, and hyperparameter estimation to the signal processing community.

With this paper, we aim to present DEM to the robotics audience as a robot learning algorithm for the blind system identification of LTI systems with colored noise. We elaborate on various components of DEM that are most relevant to the robotics community: (1) the derivation of the free energy objectives from Bayesian principles, (2) the modeling of colored noise using generalized coordinates, and (3) the simplification of update rules for LTI systems with colored noise. We reformulate DEM for LTI systems into a form that is widely used in the robotics domain and use this mathematical formulations to prove that the estimation steps of DEM have theoretical guarantees of convergence [41]. In our prior work, we have discussed the stability conditions of our DEM-based linear state and input observer design [2]. This convergence guarantees and stability criteria are essential for robot safety while in operation and is of high relevance to the robotics community. Through extensive simulations on a range of random systems, we show that DEM is a competitive estimator when compared to other benchmarks in the control systems domain. The core contributions of the paper are:

• Reformulating DEM into an estimation algorithm for LTI systems with colored noise (Section 12).
• Proving that the estimator has theoretical guarantees of convergence for the estimation steps (Section 14).
• Proving through rigorous simulation that DEM outperforms the state-of-the-art system identification methods for parameter estimation under colored noise (Section 16).

2. Problem Statement

Consider the linear plant dynamics given in Equation (1), where $\mathbf{A}$, $\mathbf{B}$ and $\mathbf{C}$ are constant system matrices, $\mathbf{x}\in {\mathbb{R}}^n$ is the hidden state, $\mathbf{v}\in {\mathbb{R}}^r$ is the input and $\mathbf{y}\in {\mathbb{R}}^m$ is the output.

$$\begin{array}{c}\hfill \dot{\mathbf{x}}=\mathbf{A}\mathbf{x}+\mathbf{B}\mathbf{v}+\mathbf{w},\end{array}\begin{array}{c}\hfill \mathbf{y}=\mathbf{C}\mathbf{x}+\mathbf{z}.\end{array}$$

(1)

Here, $\mathbf{w}\in {\mathbb{R}}^n$ and $\mathbf{z}\in {\mathbb{R}}^m$ represent the process and measurement noise, respectively. The notations of the plant are denoted in boldface, whereas its estimates are denoted in nonboldface letters. The noises in this paper are generated through the convolution of white noise with a Gaussian kernel. The use of colored noise is motivated by the fact that in robotics, the unmodelled dynamics and the non-linearity errors can enter the plant dynamics through the noise terms, thereby violating the white noise assumption in practice [20].

The goal of this paper is to reformulate DEM for an LTI system such that given the output of the system $\mathbf{y}$, the estimator computes the associated states x, inputs v, parameter $\theta$ containing the matrices $A,B$ and C, hyperparameters $\lambda$ that model the noise precision $(\mathsf{\Pi }=e^{\lambda })$, and the uncertainties of all its estimates $({\mathsf{\Sigma }}^x,{\mathsf{\Sigma }}^v,{\mathsf{\Sigma }}^{\theta },{\mathsf{\Sigma }}^{\lambda })$, with the help of the prior (learned) knowledge encoded in the robot brain, such that the estimate best predicts the data. The schematic of the proposed robot brain’s inference process is given in Figure 1.

3. Preliminaries

To reformulate DEM into an estimation algorithm for LTI systems, this section introduces the key concepts and terminologies that are familiar to the FEP audience.

3.1. Generative Model

The generative model (plant model) is the robot brain’s estimate of the generative process in the environment that generated data. The robot brain infers this model via model evidencing from the measurement data. The key idea behind DEM to deal with colored noise is to model the trajectory of the time-varying components (states, for example) using generalized coordinates. The use of generalized coordinates is new to the control systems literature and is different from the familiar definition in classical mechanics. In mechanics, it is the minimum number of independent coordinates that define the configuration of a system, whereas in DEM, it is the vector defining the motion of a point using its higher derivatives. In DEM, the emphasis is on tracking the trajectories of states, inputs and outputs instead of their point estimates. The states are expressed in generalized coordinates using its higher-order derivatives, i.e., $\tilde{x}={[x{x}^{\prime }{x}^{″}\dots ]}^T$. The generalized state vector $\tilde{x}$ with an order of generalized motion of p will have $p+1$ terms, with the copy of the state vector as the first term, followed by its p derivatives. The variables in generalized coordinates are denoted by a tilde, and its components (higher derivatives) are denoted by primes. The evolution of states is written as:

$$\begin{array}{cc}\hfill & x^{\prime }=Ax+Bv+w\hfill \\ \hfill & x^{″}=A{x}^{\prime }+B{v}^{\prime }+w^{\prime }\hfill \\ & \dots \hfill \end{array}\begin{array}{cc}\hfill & y=Cx+z\hfill \\ \hfill & \dot{y}=C{x}^{\prime }+z^{\prime }\hfill \\ & \dots \hfill \end{array}$$

(2)

The colored noises are modeled such that the covariance of noise derivatives $\tilde{z}={[z,z^{\prime },z^{″},\dots ]}^T$ and $\tilde{w}={[w,w^{\prime },w^{″},\dots ]}^T$ are well defined (to be explained in Section 3.3). The generative model representing the system is compactly written as:

$$\begin{array}{cc}\hfill & \dot{\tilde{x}}=D^x\tilde{x}=\tilde{A}\tilde{x}+\tilde{B}\tilde{v}+\tilde{w}\hfill \end{array}\begin{array}{cc}\hfill & \tilde{y}=\tilde{C}\tilde{x}+\tilde{z}\hfill \end{array}$$

(3)

where $D^x={\left[\begin{array}{ccccc}0& 1& & & \\ & 0& 1& & \\ & & .& .& \\ & & & 0& 1\\ & & & & 0\end{array}\right]}_{(p+1)\times (p+1)}\otimes I_{n\times n}$, performs the block derivative operation, equivalent to shifting up all components in generalized coordinates by one block. A similar definition holds for $D^v$ (appears later) with size $r(d+1)\times r(d+1)$, where p and d are the order of generalized motion of states and inputs, respectively. Here, $\tilde{A}=I_{p+1}\otimes A$, $\tilde{B}=I_{p+1}\otimes B$, and $\tilde{C}=I_{p+1}\otimes C$, where ⊗ is the Kronecker tensor product.

3.2. Parameters and Hyperparameters

To simplify the parameter estimation steps of DEM for LTI systems (Section 13.2) and to facilitate the convergence proof (Section 14), we introduce an alternative generative model which is the direct reformulation of Equation (3) given by:

$$\begin{array}{c}\hfill \dot{\tilde{x}}=M\theta +\tilde{w},\tilde{y}=N\theta +\tilde{z},\theta =\left[\begin{array}{c}vec(A^T)\\ vec(B^T)\\ vec(C^T)\end{array}\right]\end{array}$$

(4)

where

$$M=\left[\begin{array}{ccc}{I}_n\otimes x^T& I_n\otimes v^T& I_n\otimes O_{1\times m}\\ I_n\otimes x^{\prime T}& I_n\otimes v^{\prime T}& I_n\otimes O_{1\times m}\\ \dots & \dots & \dots \end{array}\right],N=\left[\begin{array}{ccc}{I}_n\otimes O_{1\times n}& I_n\otimes O_{1\times r}& I_m\otimes x^T\\ I_n\otimes O_{1\times n}& I_n\otimes O_{1\times r}& I_m\otimes x^{\prime T}\\ \dots & \dots & \dots \end{array}\right].$$

(5)

Throughout the paper, $\theta$ represents the set of parameters, whereas $\lambda =\left[\begin{array}{c}{\lambda }^z\\ {\lambda }^w\end{array}\right]$ represents the hyperparameters that define the precision matrices (inverse covariance matrices) of the observation noise and the process noise. For noise modelling, we parametrize the noise precisions using an exponential relation with the hyperparameters, given by [18]:

$${\mathsf{\Pi }}^w({\lambda }^w)=e^{{\lambda }^w}{\mathsf{\Omega }}^w,{\mathsf{\Pi }}^z({\lambda }^z)=e^{{\lambda }^z}{\mathsf{\Omega }}^z,$$

(6)

where ${\mathsf{\Omega }}^w$ and ${\mathsf{\Omega }}^z$ represent constant matrices encoding how different noises are correlated. Here ${\mathsf{\Pi }}^w$ and ${\mathsf{\Pi }}^z$ are the inverse covariances $\{{({\mathsf{\Sigma }}^w)}^{-1},{({\mathsf{\Sigma }}^z)}^{-1}\}$ or precisions of the noises. This parameterization ensures the selection of positive definite noise precision matrices through hyperparameter updates. We assume zero cross-correlation between w and z. We also assume that $\theta$ and $\lambda$ are time-invariant.

3.3. Colored Noise

The next step towards handling the colored noise is to embed the noise correlation between different components in the generalized noises $\tilde{w}$ and $\tilde{z}$ given in Equation (3). DEM uses generalized coordinates, which models a correlation between noise derivatives through the temporal precision matrix S (inverse of covariance matrix). The generalized noise correlation is assumed to be due to a Gaussian filter, where S can be calculated as [18]:

$$S({\sigma }^2)={\left[\begin{array}{cccc}1& 0& -\frac{1}{2{\sigma }^2}& ..\\ 0& \frac{1}{2{\sigma }^2}& 0& ..\\ -\frac{1}{2{\sigma }^2}& 0& \frac{3}{4{\sigma }^4}& ..\\ ..& ..& ..& ..\end{array}\right]}_{(p+1)\times (p+1)}^{-1}$$

(7)

where $\sigma$ (from the Gaussian kernel) denotes the noise smoothness level. While ${\sigma }^2\to 0$ denotes white noise, nonzero ${\sigma }^2$ denotes colored noise. The covariance between noise derivatives increases exponentially with the order of noise derivatives. Simulations show that derivatives above 6 can be neglected [18]. The generalized noise precision matrices are given by ${\tilde{\mathsf{\Pi }}}^w=S({\sigma }^2)\otimes {\mathsf{\Pi }}^w$ and ${\tilde{\mathsf{\Pi }}}^z=S({\sigma }^2)\otimes {\mathsf{\Pi }}^z$, where ${\mathsf{\Pi }}^w$ and ${\mathsf{\Pi }}^z$ are the inverse noise covariances.

3.4. Generalized Motion of the Outputs and Noises

The generalized motion of output $\tilde{\mathbf{y}}$ is practically expensive and inaccessible to robotics systems. Moreover, most sensors such as encoders operate with discrete measurements. Therefore, as a pre-processing step for estimation (refer to Figure 1), $\tilde{\mathbf{y}}$ should be computed from discrete measurements. The goal is to first express the discrete measurements as a function of output derivatives and then invert the function to compute the derivatives from the discrete measurements. Given the p temporal derivatives $\tilde{\mathbf{y}}$ at time t, the p output sequence surrounding $\mathbf{y}$ can be approximated using Taylor series as follows [18]:

$$\hat{\mathbf{y}}=\left[\begin{array}{c}..\\ \mathbf{y}(t-dt)\\ \mathbf{y}(t)\\ \mathbf{y}(t+dt)\\ ..\end{array}\right]=(Y\otimes I_m)\tilde{\mathbf{y}},Y_{ij}=\frac{{\left[\left(i-ceil\left(\frac{p+1}{2}\right)\right)dt\right]}^{j-1}}{(j-1)!},$$

(8)

where $i,j=1,2,\dots ,p+1$, $ceil(.)$ is the smallest integer function and $\hat{y}$ has the size $m(p+1)\times 1$. Therefore, generalized motion of output $\tilde{\mathbf{y}}$ at time t is:

$$\tilde{\mathbf{y}}=(Y^{-1}\otimes I_m)\hat{\mathbf{y}}.$$

(9)

Using $\tilde{\mathbf{y}}$ embeds more temporal information about the plant into the data in the form of conditional trajectories, with the disadvantage of a time latency of $\frac{p}{2}dt$ in estimation. For robotic systems with high sampling rates, this latency in estimation is negligible [3,20]. The next section employs this generalized output along with the generative model for observer designs.

3.5. Notations and Conventions

Throughout the paper, the superscript notation will be used to represent the quantity being referred to, and the subscript notation will be used to represent the derivative operation. For example, ${\tilde{\mathsf{\Pi }}}_{\lambda }^w$ represents the derivative of the generalized precision of the process noise w with respect to the hyperparameters $\lambda$. The tilde operator $(\tilde{x})$ is used to represent a quantity in its generalized coordinates, whereas the bar operator $(\overline{F})$ is used to represent the time integral of a quantity. All the probability distributions will be represented by the $p(.)$ notation, whereas its expectation will be represented by the $\langle p(.)\rangle$ notation.

4. Free Energy Objectives

With the preliminaries in place, we can build up towards the complete DEM algorithm. Eventually, in Section 8 and Section 12, we will see that it is an optimization algorithm that finds the best estimates for states, inputs, parameters, and hyperparameters for given measurement data. This result is achieved by optimizing two objective functions, which are the core objectives under the entire Free Energy Principle: the free energy and the free action [1]. Here we derive and simplify this objective.

The derivation starts from the fundamentals of Variational Inference (VI) [39]. In VI, the posterior distribution $p(\vartheta /y)$ of parameter $\vartheta$, given the measurement y, is expressed as:

$$p(\vartheta /y)=\frac{p(y/\vartheta )p(\vartheta )}{p(y)}=\frac{p(\vartheta ,y)}{p(y)}=\frac{p(\vartheta ,y)}{\int p(\vartheta ,y)d\vartheta }.$$

(10)

However, the marginalization over $\vartheta$ to calculate $p(y)$ is often intractable because the search space of $\vartheta$ is large. A widely used technique is to introduce a variational distribution $q(\vartheta )$ known as the recognition density, which acts as an approximate representation of the posterior distribution with $q(\vartheta )\approx p(\vartheta /y)$. A common method used among variational Bayes algorithms is to minimize the Kullback–Leibler (KL) divergence between both the distributions, defined as:

$$KL(q(\vartheta )||p(\vartheta /y))={\langle lnq(\vartheta )\rangle }_{q(\vartheta )}-{\langle lnp(\vartheta /y)\rangle }_{q(\vartheta )},$$

(11)

where ${\langle .\rangle }_{q(\vartheta )}$ represents the expectation over $q(\vartheta )$. Substituting Equation (10) in Equation (11) and using $\int q(\vartheta )d\vartheta =1$ yields:

$$KL(q(\vartheta )||p(\vartheta /y))={\langle lnq(\vartheta )\rangle }_{q(\vartheta )}-{\langle lnp(\vartheta ,y)\rangle }_{q(\vartheta )}+lnp(y).$$

(12)

The rearrangement of terms yield:

$$\begin{array}{cc}\hfill lnp(y)& ={\langle lnp(\vartheta ,y)\rangle }_{q(\vartheta )}-{\langle lnq(\vartheta )\rangle }_{q(\vartheta )}+KL(q(\vartheta )||p(\vartheta /y)),\hfill \\ \hfill & =\underset{\mathrm{free}\mathrm{energy}}{\underbrace{{\langle U(\vartheta ,y)\rangle }_{q(\vartheta )}+H{(\vartheta )}_{q(\vartheta )}}}+KL(q(\vartheta )||p(\vartheta /y)),\hfill \end{array}$$

(13)

where $lnp(y)$ is the log-evidence, $U(\vartheta ,y)=lnp(\vartheta ,y)$ is the internal energy and $H{(\vartheta )}_{q(\vartheta )}=-{\langle lnq(\vartheta )\rangle }_{q(\vartheta )}$ is the entropy of the density. The free energy term in Equation (13) is defined as the sum of an energy term and its entropy. It acts as the lower bound on the log-evidence because the KL divergence term $KL(q(\vartheta )||p(\vartheta /y))$ is always positive. The maximization of free energy minimizes the divergence term in Equation (13) because the log-evidence is independent of $q(\vartheta )$, thereby rendering the variational density $q(\vartheta )$ as a close approximation of $p(\vartheta /y)$. Therefore, the difficult evaluation of an intractable integral term in Equation (10) is converted into a much simpler optimization problem of maximizing the free energy. This reduces the problem of inference to a direct optimization of its free energy objectives and is the fundamental idea behind variation inference. The free energy term in Equation (13) is equivalent to the Evidence Lower Bound (ELBO) [39]. It can be simplified as:

$$\begin{array}{cc}\hfill F& ={\langle U(\vartheta ,y)\rangle }_{q(\vartheta )}+H{(\vartheta )}_{q(\vartheta )}=\int q(\vartheta )U(\vartheta ,y)d\vartheta -\int q(\vartheta )lnq(\vartheta )d\vartheta \hfill \\ \hfill & =\int q(\vartheta )lnp(\vartheta ,y)d\vartheta -\int q(\vartheta )lnq(\vartheta )d\vartheta .\hfill \end{array}$$

(14)

However, when the parameter set to be estimated includes both the time-variant and the time-invariant parameters, the free action is used as the objective function to be maximized. The free action is defined as the time integral of the free energy and is given by:

$$\overline{F}=\overline{V}(\vartheta )+\overline{H}(\vartheta )=\int {\langle U(\vartheta ,y)\rangle }_{q(\vartheta )}dt+\int H{(\vartheta )}_{q(\vartheta )}dt,$$

(15)

where $V(\vartheta )={\langle U(\vartheta ,y)\rangle }_{q(\vartheta )}$ is called the variational free energy (VFE).

The next sections will deal with two main assumptions used in DEM to simplify the free energy objectives, namely the Laplace approximation and the mean-field assumption. The aim is to derive the simplified free energy objectives for an LTI system under these assumptions.

5. Laplace Approximation

The first common approach to simplify the free energy objective is to assume the variational density $q(\vartheta )$ to be Gaussian in nature with variational parameters $\vartheta$ and ${\mathsf{\Sigma }}^{\theta }$ as its mode and covariance, respectively, [38]. Here, the inverse of ${\mathsf{\Sigma }}^{\vartheta }$ (denoted by ${\mathsf{\Pi }}^{\vartheta }$ and known as the conditional precision), represents the confidence in estimation. The recognition density takes the following form:

$$q(\vartheta )=\mathcal{N}(\vartheta :{\mu }^{\vartheta },{\mathsf{\Sigma }}^{\vartheta })=\frac{1}{\sqrt{{(2\pi )}^n|{\mathsf{\Sigma }}^{\vartheta }|}}{e}^{-\frac{1}{2}{(\vartheta -{\mu }^{\vartheta })}^T{\mathsf{\Pi }}^{\vartheta }(\vartheta -{\mu }^{\vartheta })}.$$

(16)

There are two main advantages with this approximation:

• It simplifies the internal energy expression $U(\vartheta ,y)$,
• It facilitates an easy computation of the conditional precision ${\mathsf{\Pi }}^{\vartheta }$ (derived in Section 7.2) as the negative curvature of the internal energy at it’s mode ${\mu }^{\vartheta }$.

Therefore, the main aim of this section is to simplify the expression for internal energy $U(\vartheta ,y)$ using the Laplace approximation, for an LTI system with its states, inputs, and outputs expressed in generalized coordinates.

The internal energy in Equation (13) can be expressed as the sum of log-likelihood and prior terms as:

$$U(\vartheta ,y)=lnp(\vartheta ,y)=ln\underset{\begin{array}{c}\mathrm{generative}\\ \mathrm{model}\end{array}}{\underbrace{p(y/\vartheta )}}+ln\underset{\mathrm{prior}}{\underbrace{p(\vartheta )}}.$$

(17)

The parameter set $\vartheta$ includes two types of parameters:

• the states and inputs, which are time-varying and therefore expressed in generalized coordinates,
• the parameters and hyperparameters, which are time-invariant and not expressed in generalized coordinates.

Equation (17) can be simplified by assuming the conditional independence of $\tilde{x}$ and $\tilde{v}$ with $\theta$ and $\lambda$. This factorization separates the deterministic quantities from the stochastic ones, thereby providing a separation of temporal scales. This is one of the core ideas behind DEM, which will be detailed in Section 6. With the redefinition of $\vartheta =\{\tilde{x},\tilde{v},\theta ,\lambda \}$, Equation (17) simplifies to:

$$\begin{array}{cc}\hfill U(\vartheta ,y)=& lnp(y/\tilde{x},\tilde{v},\theta ,\lambda )+lnp(\tilde{x},\tilde{v},\theta ,\lambda )\hfill \\ \hfill =& lnp(y/\tilde{x},\tilde{v},\theta ,\lambda )+lnp(\tilde{x}/\tilde{v},\theta ,\lambda )+lnp(\tilde{v})+lnp(\theta )+lnp(\lambda ).\hfill \end{array}$$

(18)

DEM combines the new sensory information y coming from the environment with the robot brain’s priors (refer to Figure 1) in a Bayesian fashion, through the internal energy $U(\vartheta ,y)$ expression given in Equation (18). The next sections will deal with simplifying $U(\vartheta ,y)$ and its action $\overline{U}$ by first modeling the probability distributions for the generative model and the priors.

5.1. Generative Model

The probability distribution $p(y/\tilde{x},\tilde{v},\theta ,\lambda )$, given in Equation (18), represents the generative model that predicts the output from the current parameter estimates. This probability can be assumed to be Gaussian-distributed, centered around the model’s output prediction $\tilde{C}\tilde{x}$ (from Equation (3)), with the same uncertainty as that of the measurement noise ${\tilde{\mathsf{\Sigma }}}^z$. The distribution $p(y/\vartheta )$ becomes:

$$p(\tilde{y}/\vartheta )=\frac{1}{\sqrt{{(2\pi )}^{m(p+1)}|{\tilde{\mathsf{\Sigma }}}^z|}}{e}^{-\frac{1}{2}{(\tilde{y}-\tilde{C}\tilde{x})}^T{\tilde{\mathsf{\Pi }}}^z(\tilde{y}-\tilde{C}\tilde{x})}.$$

(19)

Since the robot cannot directly measure $\tilde{x}$ in Equation (3), we track the motion of the generalized states through the approximation $\dot{\tilde{x}}={\tilde{x}}^{\prime }=D^x\tilde{x}$. The prediction for motion is $\tilde{A}\tilde{x}+\tilde{B}\tilde{v}$ with an uncertainty of ${\tilde{\mathsf{\Sigma }}}^w$. The Gaussian distribution becomes:

$$p({\tilde{x}}^{\prime }/\tilde{x},\tilde{v},\vartheta ,\lambda )=\frac{1}{\sqrt{{(2\pi )}^{n(p+1)}|{\tilde{\mathsf{\Sigma }}}^w|}}{e}^{-\frac{1}{2}{(D^x\tilde{x}-\tilde{A}\tilde{x}-\tilde{B}\tilde{v})}^T{\tilde{\mathsf{\Pi }}}^w(D^x\tilde{x}-\tilde{A}\tilde{x}-\tilde{B}\tilde{v})}.$$

(20)

5.2. Prior Distributions

The remaining distributions $p(\tilde{v}),p(\theta )$ and $p(\lambda )$ are the priors of the robot brain that can be transferred from prior (learned) experiences to the inference process. Similar to the previous section, a Gaussian prior is placed over the inputs $p(\tilde{v})=\mathcal{N}(\tilde{v}:{\eta }^{\tilde{v}},L^{\tilde{v}})$ as well, with mean ${\eta }^{\tilde{v}}$ and prior covariance $L^{\tilde{v}}={(P^{\tilde{v}})}^{-1}$, as:

$$p(\tilde{v})=\frac{1}{\sqrt{{(2\pi )}^{r(d+1)}|L^{\tilde{v}}|}}{e}^{-\frac{1}{2}{(\tilde{v}-{\eta }^{\tilde{v}})}^T{P}^{\tilde{v}}(\tilde{v}-{\eta }^{\tilde{v}})}.$$

(21)

The prior distribution of parameters $\theta \in {\mathbb{R}}^l$ is assumed to be a Gaussian-centred around the prior parameter value ${\eta }^{\theta }$, with the prior covariance $L^{\theta }={(P^{\theta })}^{-1}$:

$$p(\theta )=\mathcal{N}(\theta :{\eta }^{\theta },L^{\theta })=\frac{1}{\sqrt{{(2\pi )}^l|L^{\theta }|}}{e}^{-\frac{1}{2}{(\theta -{\eta }^{\theta })}^T{P}^{\theta }(\theta -{\eta }^{\theta })}.$$

(22)

Similarly, a Gaussian prior is placed over the hyperparameters $\lambda \in {\mathbb{R}}^2$:

$$p(\lambda )=\mathcal{N}(\lambda :{\eta }^{\lambda },L^{\lambda })=\frac{1}{\sqrt{{(2\pi )}^2|L^{\lambda }|}}{e}^{-\frac{1}{2}{(\lambda -{\eta }^{\lambda })}^T{P}^{\lambda }(\lambda -{\eta }^{\lambda })}.$$

(23)

A higher value of $P^{\tilde{v}},P^{\theta }$ and $P^{\lambda }$ represents the robot’s higher confidence in its prior estimates ${\eta }^{\tilde{v}}$, ${\eta }^{\theta }$, and ${\eta }^{\lambda }$, respectively.

5.3. Simplification of the Internal Energy Action $\overline{U}$

This section aims at using the distributions from the previous sections to simplify $\overline{U}$. The logarithm of a Gaussian prior after dropping constants takes the general form:

$$\begin{array}{cc}\hfill lnp(\theta )=& ln\mathcal{N}(\theta :{\eta }^{\theta },L^{\theta })=-\frac{1}{2}{(\theta -{\eta }^{\theta })}^T{(L^{\theta })}^{-1}(\theta -{\eta }^{\theta })-\frac{1}{2}ln|L^{\theta }|\hfill \end{array}$$

(24)

Therefore, substituting Equations (19)–(23) in Equation (18) and simplifying it using the prediction error terms ${ϵ}^{\tilde{x}}=D^x\tilde{x}-\tilde{A}\tilde{x}-\tilde{B}\tilde{v},{ϵ}^{\tilde{y}}=\tilde{y}-\tilde{C}\tilde{x},{ϵ}^{\tilde{v}}=\tilde{v}-{\eta }^{\tilde{v}}{ϵ}^{\theta }=\theta -{\eta }^{\theta },$ and ${ϵ}^{\lambda }=\lambda -{\eta }^{\lambda }$, after dropping constants, yields:

$$\begin{array}{cc}\hfill U(\vartheta ,y)=& -\frac{1}{2}{ϵ}^{\theta T}{P}^{\theta }{ϵ}^{\theta }+\frac{1}{2}ln|P^{\theta }|-\frac{1}{2}{ϵ}^{\lambda T}{P}^{\lambda }{ϵ}^{\lambda }+\frac{1}{2}ln|P^{\lambda }|-\frac{1}{2}{\widetilde{ϵ}}^T\tilde{\mathsf{\Pi }}\widetilde{ϵ}+\frac{1}{2}ln|\tilde{\mathsf{\Pi }}|,\hfill \end{array}$$

(25)

where $\widetilde{ϵ}=\left[\begin{array}{c}{ϵ}^{\tilde{y}}\\ {ϵ}^{\tilde{v}}\\ {ϵ}^{\tilde{x}}\end{array}\right]=\left[\begin{array}{c}\tilde{y}-\tilde{C}\tilde{x}\\ \tilde{v}-{\eta }^{\tilde{v}}\\ D^x\tilde{x}-\tilde{A}\tilde{x}-\tilde{B}\tilde{v}\end{array}\right]$, and $\tilde{\mathsf{\Pi }}=diag({\tilde{\mathsf{\Pi }}}^z,P^{\tilde{v}},{\tilde{\mathsf{\Pi }}}^w)$. Here $diag(.,.)$ is the block diagonal operator. Grouping the internal energy terms of the temporal and nontemporal parameters yields:

$$U(\vartheta ,y)=U(t)+U(\theta )+U(\lambda ),$$

(26)

where

$$U(t)=-\frac{1}{2}{\widetilde{ϵ}}^T\tilde{\mathsf{\Pi }}\widetilde{ϵ}+\frac{1}{2}ln|\tilde{\mathsf{\Pi }}|.$$

(27)

Summing up the internal energy of all the temporal parameters over time yields the action of internal energy as follows:

$$\begin{array}{cc}\hfill \overline{U}& =U(\theta )+U(\lambda )+\sum _{t}U(t)\hfill \\ \hfill & =-\frac{1}{2}{ϵ}^{\theta T}{P}^{\theta }{ϵ}^{\theta }+\frac{1}{2}ln|P^{\theta }|-\frac{1}{2}{ϵ}^{\lambda T}{P}^{\lambda }{ϵ}^{\lambda }+\frac{1}{2}ln|P^{\lambda }|-\sum _t\frac{1}{2}{\widetilde{ϵ}}^T\tilde{\mathsf{\Pi }}\widetilde{ϵ}+\frac{1}{2}\sum _{t}ln|\tilde{\mathsf{\Pi }}|.\hfill \end{array}$$

(28)

It can be observed from Equation (28) that the robot’s priors $({\eta }^{\tilde{v}},P^{\tilde{v}},{\eta }^{\theta },P^{\theta },{\eta }^{\lambda },P^{\lambda })$ enter the free energy objective through the internal energy term. Intuitively, this can be seen as the direct influence of the robot’s prior beliefs on the inference process through the mismatches in the robot’s predictions. These weighed prediction errors drive the robot’s desire to maintain an equilibrium between its internal model and the generative process in the environment. A large mismatch between the robot’s predictions and the data results in a large prediction error, which gets precision-weighted and enters the free energy objective through its internal energy.

6. Mean-Field Approximation

The second widely used assumption for the simplification of free energy objectives is the factorization of parameters into independent subdensities for the recognition density [18], given by:

$$q(\vartheta )=\prod _{i}q({\theta }^i)=q(\tilde{x})q(\tilde{v})q(\theta )q(\lambda ).$$

(29)

This approximation assumes the conditional independence between subdensities (states and parameters, for example). The subdensities are assumed to interact with each other only through the mean-field quantities. The strong biological plausibility of this approximation in terms of biological brain’s inference process is delineated in [27]. The main advantage of this approximation is the simplification of $\overline{V}$ in Equation (15). However, the mathematical proof for this simplification is missing from the DEM literature [18]. Therefore, this section aims to fill this gap by deriving these proofs by delineating all the intermediate assumptions.

6.1. Simplification of the Entropy Action $\overline{H}$

The entropy of the density in Equation (14), given by $H{(\vartheta )}_{q(\vartheta )}=-{\langle lnq(\vartheta )\rangle }_{q(\vartheta )}$, can be simplified for all parameters as:

$$H{(\vartheta )}_{q(\vartheta )}=-\int \int \int \int q(\tilde{x},\tilde{v},\theta ,\lambda )lnq(\tilde{x},\tilde{v},\theta ,\lambda )d\tilde{x}d\tilde{v}d\theta d\lambda$$

(30)

Substituting the mean-field assumption given in Equation (29) in Equation (30) and using the property of normalized recognition densities $\int q({\theta }^i)d{\theta }^i=1$ yields:

$$H{(\vartheta )}_{q(\vartheta )}=H(\theta )+H(\lambda )+H(t),$$

(31)

where $H(\theta )=-{\langle lnq(\theta )\rangle }_{q(\theta )}$, $H(\lambda )=-{\langle lnq(\lambda )\rangle }_{q(\lambda )}$ and $H(t)=-{\langle lnq(\tilde{x})\rangle }_{q(\tilde{x})}-$${\langle lnq(\tilde{v})\rangle }_{q(\tilde{v})}$. We place the Laplace approximation over the marginals of the recognition densities of all parameters as:

$$\begin{array}{cc}\hfill q(\theta )=& \mathcal{N}(\theta :{\mu }^{\theta },{\mathsf{\Sigma }}^{\theta })=\frac{1}{\sqrt{{(2\pi )}^l|{\mathsf{\Sigma }}^{\theta }|}}{e}^{-\frac{1}{2}{(\theta -{\mu }^{\theta })}^T{\mathsf{\Pi }}^{\theta }(\theta -{\mu }^{\theta })}\hfill \\ \hfill q(\lambda )=& \mathcal{N}(\lambda :{\mu }^{\lambda },{\mathsf{\Sigma }}^{\lambda })=\frac{1}{\sqrt{{(2\pi )}^2|{\mathsf{\Sigma }}^{\lambda }|}}{e}^{-\frac{1}{2}{(\lambda -{\mu }^{\lambda })}^T{\mathsf{\Pi }}^{\lambda }(\lambda -{\mu }^{\lambda })}\hfill \\ \hfill q(\tilde{x})=& \mathcal{N}(\tilde{x}:{\mu }^{\tilde{x}},{\mathsf{\Sigma }}^{\tilde{x}})=\frac{1}{\sqrt{{(2\pi )}^{n(p+1)}|{\mathsf{\Sigma }}^{\tilde{x}}|}}{e}^{-\frac{1}{2}{(\tilde{x}-{\mu }^{\tilde{x}})}^T{\mathsf{\Pi }}^{\tilde{x}}(\tilde{x}-{\mu }^{\tilde{x}})}\hfill \\ \hfill q(\tilde{v})=& \mathcal{N}(\tilde{v}:{\mu }^{\tilde{v}},{\mathsf{\Sigma }}^{\tilde{v}})=\frac{1}{\sqrt{{(2\pi )}^{r(d+1)}|{\mathsf{\Sigma }}^{\tilde{v}}|}}{e}^{-\frac{1}{2}{(\tilde{v}-{\mu }^{\tilde{v}})}^T{\mathsf{\Pi }}^{\tilde{v}}(\tilde{v}-{\mu }^{\tilde{v}})}.\hfill \end{array}$$

(32)

The recognition densities given in Equation (32) might look similar to the priors distributions given in Equations (21)–(23), mainly due to the common Gaussian distribution. The prior distributions are centered around the prior mean and prior covariances, whereas the recognition density is centered around the mean ${\mu }^i$ and conditional covariance ${\mathsf{\Sigma }}^i$ of the i-th parameter set.

The action of entropy can be calculated by substituting Equation (32) in Equation (31) and summing up the entropy terms of time dependent parameters with respect to time. Upon dropping the constant terms, it simplifies to:

$$\begin{array}{c}\hfill \overline{H}=H(\theta )+H(\lambda )+\sum _{t}H(t)=\frac{1}{2}ln|{\mathsf{\Sigma }}^{\theta }|+\frac{1}{2}ln|{\mathsf{\Sigma }}^{\lambda }|+\frac{1}{2}\sum _{t}ln|{\mathsf{\Sigma }}^{\tilde{x}}|+\frac{1}{2}\sum _{t}ln|{\mathsf{\Sigma }}^{\tilde{v}}|,\end{array}$$

(33)

Equation (33) shows how the uncertainty in the robot’s inference directly enters the objective $\overline{F}$, through $\overline{H}$.

6.2. Mean-Field Terms

Given the simplified expressions for $\overline{U}$ and $\overline{H}$, the next step towards finding $\overline{F}$ in Equation (15) is to evaluate $\overline{V}$ given by:

$$\begin{array}{c}\hfill \overline{V}=\int {\langle U(y,\vartheta )\rangle }_{q(\vartheta )}dt\end{array}$$

(34)

$U(y,\vartheta )$ can be simplified using the second-degree Taylor series expansion near the mean ${\mu }^{\vartheta }=\{{\mu }^{\tilde{x}},{\mu }^{\tilde{v}},{\mu }^{\theta },{\mu }^{\lambda }\}$ as:

$$\begin{array}{cc}\hfill U(y,\tilde{x},\tilde{v},\theta ,\lambda )=& U(y,{\mu }^{\vartheta })+\sum _{i=1}^{4}U{(y,{\mu }^{\vartheta })}_{{\theta }^i}({\theta }^i-{\mu }^i)+\sum _{i=1}^4\sum _{j=1}^4{({\theta }^i-{\mu }^i)}^{T}U{(y,{\mu }^{\vartheta })}_{{\theta }^i{\theta }^j}({\theta }^j-{\mu }^j),\hfill \end{array}$$

(35)

where we use the shorthand $U(y,{\mu }^{\vartheta })=U(y,\vartheta ){|}_{\vartheta ={\mu }^{\vartheta }}$ and $U{(y,{\mu }^{\vartheta })}_{\vartheta }=U{(y,\vartheta )}_{\vartheta }{|}_{\vartheta ={\mu }^{\vartheta }}$. This approximation of the internal energy has nontrivial implications in terms of the biological brain’s decision-making process [21]. The second order approximation is justified because the Laplace and mean-field approximations entail an internal energy that is quadratic in $\tilde{x}$, $\tilde{v}$, and $\theta$, as given in Equation (25), thereby reducing all its higher-order derivatives to zero. Moreover, the higher derivatives of $U(y,{\mu }^{\vartheta })$ with respect to $\lambda$, reduce to zero because of the assumptions made in Section 9. By differentiating Equation (25) at the mean ${\mu }^i$, $U{(y,{\mu }^{\vartheta })}_{{\theta }^i}$ can be found to be all zeroes, which upon substitution in Equation (35) simplifies to:

$$U(y,\vartheta )=U(y,{\mu }^{\vartheta })+\sum _{i=1}^4\sum _{j=1}^4{({\theta }^i-{\mu }^i)}^{T}U{(y,{\mu }^{\vartheta })}_{{\theta }^i{\theta }^j}({\theta }^j-{\mu }^j).$$

(36)

Substituting it in Equation (34), upon simplification yields:

$$\overline{V}=\overline{U}(y,{\mu }^{\vartheta })+\int Wdt$$

(37)

where

$$\begin{array}{cc}\hfill W& =\frac{1}{2}\sum _{i,j=1}^4{\langle {({\theta }^i-{\mu }^i)}^{T}U{(y,{\mu }^{\vartheta })}_{{\theta }^i{\theta }^j}({\theta }^j-{\mu }^j)\rangle }_{q({\vartheta }^i)q({\vartheta }^j)}\hfill \\ \hfill & =\frac{1}{2}\sum _{i,j=1}^4{\langle tr\left({({\theta }^i-{\mu }^i)}^{T}U{(y,{\mu }^{\vartheta })}_{{\theta }^i{\theta }^j}({\theta }^j-{\mu }^j)\right)\rangle }_{q({\vartheta }^i)q({\vartheta }^j)}\hfill \\ \hfill & =\frac{1}{2}\sum _{i,j=1}^4{\langle tr\left(({\theta }^j-{\mu }^j){({\theta }^i-{\mu }^i)}^{T}U{(y,{\mu }^{\vartheta })}_{{\theta }^i{\theta }^j}\right)\rangle }_{q({\vartheta }^i)q({\vartheta }^j)}\hfill \\ \hfill & =\frac{1}{2}\sum _{i,j=1}^{4}tr\left[{\langle ({\theta }^j-{\mu }^j){({\theta }^i-{\mu }^i)}^T\rangle }_{q({\vartheta }^i)q({\vartheta }^j)}U{(y,{\mu }^{\vartheta })}_{{\theta }^i{\theta }^j}\right]=\frac{1}{2}\sum _{i,j=1}^{4}tr\left[{\mathsf{\Sigma }}^{ij}U{(y,{\mu }^{\vartheta })}_{{\theta }^i{\theta }^j}\right]\hfill \end{array}$$

Since the parameters ${\vartheta }^i$ are assumed to be independent of each other, the covariance between them drops to zero, resulting in:

$$\begin{array}{cc}\hfill \overline{V}=& \overline{U}(y,{\mu }^{\vartheta })+\frac{1}{2}\int \sum _{i=1}^{4}tr[{\mathsf{\Sigma }}^{i}U{(y,{\mu }^{\vartheta })}_{{\theta }^i{\theta }^i}]dt\hfill \\ \hfill =& \overline{U}(y,{\mu }^{\vartheta })+\int \left[W^{\tilde{x}}+W^{\tilde{v}}+W^{\theta }+W^{\lambda }\right]dt,\hfill \end{array}$$

(38)

where

$$W^{{\vartheta }^i}=\frac{1}{2}tr\left[{\mathsf{\Sigma }}^{i}U{(y,{\mu }^{\vartheta })}_{{\theta }^i{\theta }^i}\right]$$

(39)

is defined as the mean-field term. Equation (38) shows that $\overline{V}$ is the sum of internal energy action and the mean field terms. The simplification of $\overline{V}$ is one of the main advantages of using mean-field approximation. However, this approximation can be relaxed to build Generalized Filtering [37,42], which is mainly relevant to nonlinear identification. It involves the modeling of parameters and hyperparameters in generalized coordinates (together with states) for online system identification. However, in this work, we take a simpler approach.

7. Simplified Free Energy Objectives

This section aims at simplifying the free energy objectives using the results from Section 5 and Section 6.

7.1. Simplification of Free Action

$\overline{F}$ is simplified by substituting Equations (28), (33) and (38) into Equation (15), yielding:

$$\begin{array}{cc}\hfill \overline{F}=& -\frac{1}{2}({ϵ}^{\theta T}{P}^{\theta }{ϵ}^{\theta }){|}_{\vartheta ={\mu }^{\vartheta }}+\frac{1}{2}ln|P^{\theta }|-\frac{1}{2}({ϵ}^{\lambda T}{P}^{\lambda }{ϵ}^{\lambda }){|}_{\vartheta ={\mu }^{\vartheta }}+\frac{1}{2}ln|P^{\lambda }|-\sum _t\frac{1}{2}({\widetilde{ϵ}}^T\tilde{\mathsf{\Pi }}\widetilde{ϵ}){|}_{\vartheta ={\mu }^{\vartheta }}\hfill \\ & +\frac{1}{2}\sum _t(ln|\tilde{\mathsf{\Pi }}{|)|}_{\vartheta ={\mu }^{\vartheta }}+\frac{1}{2}\sum _t(ln|{\mathsf{\Sigma }}^{\tilde{x}}|+ln|{\mathsf{\Sigma }}^{\tilde{v}}|)+\frac{1}{2}ln|{\mathsf{\Sigma }}^{\theta }|+\frac{1}{2}ln|{\mathsf{\Sigma }}^{\lambda }|\hfill \\ & +\frac{1}{2}\sum _{t}tr[{\mathsf{\Sigma }}^{\tilde{x}}U{(y,{\mu }^{\vartheta })}_{\tilde{x}\tilde{x}}+{\mathsf{\Sigma }}^{\tilde{v}}U{(y,{\mu }^{\vartheta })}_{\tilde{v}\tilde{v}}+{\mathsf{\Sigma }}^{\theta }U{(y,{\mu }^{\vartheta })}_{\theta \theta }+{\mathsf{\Sigma }}^{\lambda }U{(y,{\mu }^{\vartheta })}_{\lambda \lambda }].\hfill \end{array}$$

(40)

We highlight three important terms in Equation (40): the combined prediction error of (generalized) outputs, inputs, and states $E=\frac{1}{2}({\widetilde{ϵ}}^T\tilde{\mathsf{\Pi }}\widetilde{ϵ}){|}_{\vartheta ={\mu }^{\vartheta }}$, the log determinant of noise precision $G=(ln|\tilde{\mathsf{\Pi }}|){|}_{\vartheta ={\mu }^{\vartheta }}$, and the mean field term $W^{{\vartheta }^i}=\frac{1}{2}tr\left[{\mathsf{\Sigma }}^{i}U{(y,{\mu }^{\vartheta })}_{{\theta }^i{\theta }^i}\right]$. These terms will be used rigorously in the rest of the document.

7.2. Simplification of the Parameter Precisions

One of the main advantages of Laplace approximation is the simple evaluation of the covariance associated with the parameter estimation. This is achieved by setting the gradient of the free action with respect to individual parameter covariance as zero. The free action gradients with respect to covariance of paramaters ${\vartheta }^i$ is given by:

$$\begin{array}{cc}\hfill \frac{\partial \overline{F}}{\partial {\mathsf{\Sigma }}^i}=& \frac{\partial }{\partial {\mathsf{\Sigma }}^i}\left(\frac{1}{2}ln|{\mathsf{\Sigma }}^i|+\frac{1}{2}\sum _{t}tr({\mathsf{\Sigma }}^{i}U{(y,{\mu }^{\vartheta })}_{{\vartheta }^i{\vartheta }^i}))\right)=\frac{1}{2}\left({({\mathsf{\Sigma }}^i)}^{-1}+\overline{U}{(y,{\mu }^{\vartheta })}_{{\vartheta }^i{\vartheta }^i}\right).\hfill \end{array}$$

(41)

The optimal parameter covariance is the covariance that maximizes the free action with zero gradients. Assuming that the parameter covariances are time-invariant, and equating the gradients to zero yields ${\mathsf{\Pi }}^i=-\overline{U}{(y,{\mu }^{\vartheta })}_{{\vartheta }^i{\vartheta }^i}$, resulting in:

$$\begin{array}{c}\hfill {\mathsf{\Pi }}^{\tilde{x}}=-U{(y,{\mu }^{\vartheta })}_{\tilde{x}\tilde{x}},{\mathsf{\Pi }}^{\tilde{v}}=-U{(y,{\mu }^{\vartheta })}_{\tilde{v}\tilde{v}},{\mathsf{\Pi }}^{\theta }=-\overline{U}{(y,{\mu }^{\vartheta })}_{\theta \theta },{\mathsf{\Pi }}^{\lambda }=-\overline{U}{(y,{\mu }^{\vartheta })}_{\lambda \lambda }\end{array}$$

(42)

From Equation (42), it is clear that the precision of parameters can be estimated just by evaluating the negative curvature of the internal energy at the conditional mode. These precision values denote the confidence of the estimator in the parameter estimate. Ideally, the parameter precision increases as the estimation process proceeds. Intuitively, the robot is more confident about its estimates (higher precision) when its predictions on the sensory data show the least variance.

7.3. Free Action at Optimal Precision

Substituting Equation (42) into Equation (39) at optimal precisions results in constant mean field terms. Therefore, the mean field terms in the free action given in Equation (40), reduces to a constant under the optimal precision given by Equation (42). Therefore, the free action at optimal precision for an LTI system reduces to:

$$\begin{array}{cc}\hfill \overline{F}=& -\frac{1}{2}\sum _t\left[\underset{\mathrm{prediction}\mathrm{error}\mathrm{of}\mathrm{outputs}}{\underbrace{{(\tilde{\mathbf{y}}-\tilde{C}\tilde{x})}^T{\tilde{\mathsf{\Pi }}}^z(\tilde{\mathbf{y}}-\tilde{C}\tilde{x})\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.}}+\underset{\mathrm{prediction}\mathrm{error}\mathrm{of}\mathrm{inputs}}{\underbrace{{(\tilde{v}-{\eta }^{\tilde{v}})}^T{P}^{\tilde{v}}(\tilde{v}-{\eta }^{\tilde{v}})\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.}}\right]\hfill \\ & -\frac{1}{2}\underset{\mathrm{prediction}\mathrm{error}\mathrm{of}\mathrm{parameters}}{\underbrace{{(\theta -{\eta }^{\theta })}^T{P}^{\theta }(\theta -{\eta }^{\theta })\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.}}-\frac{1}{2}\underset{\mathrm{prediction}\mathrm{error}\mathrm{of}\mathrm{hyperparameters}}{\underbrace{{(\lambda -{\eta }^{\lambda })}^T{P}^{\lambda }(\lambda -{\eta }^{\lambda })\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.}}\hfill \\ & -\frac{1}{2}\sum _t\underset{\mathrm{prediction}\mathrm{error}\mathrm{of}(\mathrm{generalized})\mathrm{states}}{\underbrace{{(D^x\tilde{x}-\tilde{A}\tilde{x}-\tilde{B}\tilde{v})}^T{\tilde{\mathsf{\Pi }}}^w(D^x\tilde{x}-\tilde{A}\tilde{x}-\tilde{B}\tilde{v})\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.}}+\underset{\mathrm{state}\mathrm{and}\mathrm{input}\mathrm{entropy}}{\underbrace{\frac{1}{2}{n}_{t}ln|{\mathsf{\Sigma }}^X|}}\hfill \\ & +\frac{1}{2}{n}^t\underset{\mathrm{noise}\mathrm{entropy}}{\underbrace{\left[ln|{\tilde{\mathsf{\Pi }}}^z|+ln|P^{\tilde{v}}|+ln|{\tilde{\mathsf{\Pi }}}^w|\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.}}+\underset{\mathrm{parameter}\mathrm{entropy}}{\underbrace{\frac{1}{2}ln|{\mathsf{\Sigma }}^{\theta }{P}^{\theta }|}}+\underset{\mathrm{hyperparameter}\mathrm{entropy}}{\underbrace{\frac{1}{2}ln|{\mathsf{\Sigma }}^{\lambda }{P}^{\lambda }|}}\hfill \end{array}$$

(43)

Note that the free action is not a function of the latent variables ($\vartheta =\{\tilde{x},\tilde{v},\theta ,\lambda \}$) but the sufficient statistics (${\mu }^{\vartheta },{\mathsf{\Sigma }}^{\vartheta }$) of the approximate posterior. For example, the weighted output prediction error term of $\overline{F}$ is ${(\tilde{\mathbf{y}}-\tilde{C}\tilde{x})}^T{\tilde{\mathsf{\Pi }}}^z(\tilde{\mathbf{y}}-\tilde{C}\tilde{x})\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.={(\tilde{\mathbf{y}}-{\mu }^{\tilde{C}}{\mu }^{\tilde{x}})}^T{\tilde{\mathsf{\Pi }}}^z\left|{}_{{\lambda }^z={\mu }^{{\lambda }^z}}\right.(\tilde{\mathbf{y}}-{\mu }^{\tilde{C}}{\mu }^{\tilde{x}})$, where $\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.$ denotes the evaluation at $\vartheta ={\mu }^{\vartheta },(\tilde{x}={\mu }^{\tilde{x}},\tilde{v}={\mu }^{\tilde{v}},\theta ={\mu }^{\theta },\lambda ={\mu }^{\lambda })$. We regroup the time-dependent components into one variable $X=\left[\begin{array}{c}{\mu }^{\tilde{x}}\\ {\mu }^{\tilde{v}}\end{array}\right]$, and use capital letters for the mean estimates of time-independent components ($\mathsf{\Theta }={\mu }^{\theta },\mathsf{\Lambda }={\mu }^{\lambda }$). Note that $X,\mathsf{\Theta }$, and $\mathsf{\Lambda }$ are part of the generative model and are the mean estimate of the components of plant dynamics.

7.4. Equivalence with the EM Algorithm

One of the most popular approaches to solve the Expectiation Maximization (EM) algorithm for state space models is to use the Maximum Likelihood Estimation (MLE), where the objective function is the log-likelihood of data, given by [43]:

$$\begin{array}{cc}\hfill & lnL=-\frac{1}{2}\sum _t{(x_t-A{x}_{t-1}-B{u}_{t-1})}^T{\mathsf{\Pi }}^w(x_t-A{x}_{t-1}-B{u}_{t-1})-\frac{n_t}{2}ln|{\mathsf{\Sigma }}^w|\hfill \\ \hfill & -\frac{n_t}{2}ln|{\mathsf{\Sigma }}^z|-\frac{1}{2}ln|{\mathsf{\Sigma }}^x|-\frac{1}{2}{(x_0-\mu )}^T{\mathsf{\Pi }}^x(x_0-\mu )-\frac{1}{2}\sum _t{(y_t-C{x}_t)}^T{\mathsf{\Pi }}^z(y_t-C{x}_t).\hfill \end{array}$$

(44)

Comparing the objective functions of EM and DEM given by Equation (44) and Equation (43), respectively, DEM is equivalent to EM when:

• the mean field terms are neglected,
• the generalized motion is not considered, and
• the robot’s priors on $\vartheta$ are not considered.

Therefore, DEM can be considered as a generalized version of the EM algorithm with additional capabilities.

The free action at optimal precision (Equation (43)) is the sum of prediction errors (PE) and entropy of generalized states, parameters, noise, and hyperparameters and is independent of mean-field terms. Although the mean-field terms turns into a constant at optimal precision, their gradients do not. This property is leveraged in Section 8 to account for the uncertainties in the parameter estimation during state estimation and vice versa, through the gradient of mean-field terms.

8. Update Rules for Estimation

The Free Energy objectives (Section 4), together with the two simplifying approximations (Section 5 and Section 6), will be combined with an optimization procedure to form the ultimate DEM algorithm. The optimization procedure itself consists of the following two sets of update rules:

• a gradient ascent over its free action $\overline{F}$ for the time invariant parameters $\theta$ and $\lambda$,
• a gradient ascent over its free energy F for the time varying parameters X,

where F and $\overline{F}$ are related through $F=\frac{\partial \overline{F}}{\partial t}$. The core idea is that the time varying parameters $\tilde{x}$ and $\tilde{v}$ can be estimated online from the robot’s instantaneous free energy, whereas the time-invariant parameters $\theta$ and $\lambda$ can be estimated from its free action after observing a sequence of data. Accordingly, the update rules for both the gradient ascends are given by

$$\frac{\partial {\mu }^i}{\partial a}=k^i\frac{\partial \overline{F}}{\partial {\mu }^i},$$

(45)

for the $a^{th}$ parameter update of the time invariant parameters and

$$\frac{\partial {\mu }^i}{\partial t}=D{\mu }^i+k^i\frac{\partial F}{\partial {\mu }^i},$$

(46)

for the time=varying parameters, where $k^i$ is the learning rate. The presence of $D{\mu }^i$ term in Equation (46) differentiates the update rule from the general gradient ascent equation used in machine learning. This is to accommodate the boundary condition that when F is maximized, $\frac{\partial F}{\partial {\mu }^i}=0$ and ${\dot{\mu }}^i=D{\mu }^i$. In other words, when the free energy is maximized, the motion of the generalized states becomes their generalized motion [44]. However, the update equations for the time-invariant parameters, $\theta$ and $\lambda$, do not require the $D{\mu }^i$ term. Therefore, the update equations at $t^{th}$ time, $a^{th}$ parameter update step, and $b^{th}$ hyperparameter update step, after regrouping the (generalized) states and inputs as $X=\left[\begin{array}{c}{\mu }^{\tilde{x}}\\ {\mu }^{\tilde{v}}\end{array}\right]$, is given by:

$$\frac{\partial X}{\partial t}=DX+k^X\frac{\partial F}{\partial X},\frac{\partial \mathsf{\Theta }}{\partial a}=k^{\mathsf{\Theta }}\frac{\partial \overline{F}}{\partial \mathsf{\Theta }},\frac{\partial \mathsf{\Lambda }}{\partial b}=k^{\mathsf{\Lambda }}\frac{\partial \overline{F}}{\partial \mathsf{\Lambda }},$$

(47)

where $D=diag(D^x,D^v)$. Note that the gradient update rules are written not on the latent variables ($\tilde{x},\tilde{v},\theta$ and $\lambda$), but on their mean estimates ($X,\mathsf{\Theta }$ and $\mathsf{\Lambda }$). Since the update rules should be implemented in the discrete domain for robotics applications, Equation (47) is discretized under local linearization with the corresponding Jacobians as $J^X=D+k^X\frac{{\partial }^{2}F}{\partial X^2},J^{\mathsf{\Theta }}=k^{\mathsf{\Theta }}\frac{{\partial }^2\overline{F}}{\partial {\mathsf{\Theta }}^2}$, and $J^{\mathsf{\Lambda }}=k^{\mathsf{\Lambda }}\frac{{\partial }^2\overline{F}}{\partial {\mathsf{\Lambda }}^2}$. The (generalized) state and input update at time t, parameter update at step a and hyperparameter update at step b are given by:

$$\begin{array}{cc}\hfill X_{t+\Delta t}& =X_t+\left(e^{J^X\Delta t}-I\right){(J^X)}^{-1}\frac{\partial X}{\partial t},\hfill \\ \hfill {\mathsf{\Theta }}_{a+1}& ={\mathsf{\Theta }}_a+\left(e^{J^{\mathsf{\Theta }}}-I\right){(J^{\mathsf{\Theta }})}^{-1}\frac{\partial \mathsf{\Theta }}{\partial a},\hfill \\ \hfill {\mathsf{\Lambda }}_{b+1}& ={\mathsf{\Lambda }}_b+\left(e^{J^{\mathsf{\Lambda }}}-I\right){(J^{\mathsf{\Lambda }})}^{-1}\frac{\partial \mathsf{\Lambda }}{\partial b}.\hfill \end{array}$$

(48)

Equation (48) shows that the update rules are dependent only on the gradients and curvatures of the free energy objectives.

8.1. The DEM Algorithm

The DEM algorithm is an iterative model inversion algorithm that uses Equations (42) and (48) to perform estimation on causal dynamic systems. It can be expressed using three steps:

• D step: (generalized) state and input estimation,
• E step: parameter estimation,
• M step: noise hyperparameter estimation,

a nomenclature that is similar to the EM algorithm. Figure 2 shows an intuitive block diagram that demonstrates the inference process of DEM as a coupled dynamics between D, E, and M steps. The data from the environment and the robot brain’s prior distributions are used to infer the generative process. The pseudocode given in Algorithm 1 demonstrates how DEM performs estimation using only the gradient and curvatures of the free energy objectives. The next sections will focus on deriving the algebraic expressions for these quantities.

8.2. Updated Equations for Estimation

The free energy gradients in Equation (47) can be evaluated by differentiating Equation (40) with ${\mu }^i$. Substituting the resulting expression in Equation (47), upon simplification yields:

$$\begin{array}{cc}\hfill \dot{X}& =DX+k^X\left[-E_X+W_X^X+W_X^{\mathsf{\Theta }}+W_X^{\mathsf{\Lambda }}\right]\hfill \\ \hfill \frac{\partial \mathsf{\Theta }}{\partial a}& =k^{\mathsf{\Theta }}\left[-P^{\theta }{ϵ}^{\theta }\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.+\sum _t\left(-E_{\mathsf{\Theta }}+W_{\mathsf{\Theta }}^X+W_{\mathsf{\Theta }}^{\mathsf{\Theta }}+W_{\mathsf{\Theta }}^{\mathsf{\Lambda }}\right)\right]\hfill \\ \hfill \frac{\partial \mathsf{\Lambda }}{\partial b}& =k^{\mathsf{\Lambda }}\left[-P^{\lambda }{ϵ}^{\lambda }\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.+\sum _t(-E_{\mathsf{\Lambda }}+W_{\mathsf{\Lambda }}^X+W_{\mathsf{\Lambda }}^{\mathsf{\Theta }}+W_{\mathsf{\Lambda }}^{\mathsf{\Lambda }}+G_{\mathsf{\Lambda }})\right],\hfill \end{array}$$

(49)

where $E_{{\mu }^i}=\frac{\partial E}{\partial {\mu }^i}=\frac{1}{2}\frac{\partial ({\widetilde{ϵ}}^T\tilde{\mathsf{\Pi }}\widetilde{ϵ})\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.}{\partial {\mu }^i}$ is the gradient of the prediction error with respect to ${\mu }^i$, $W_{{\mu }^j}^{{\mu }^i}=\frac{\partial W^{{\mu }^i}}{\partial {\mu }^j}=\frac{1}{2}\frac{\partial }{\partial {\mu }^j}tr\left[{\mathsf{\Sigma }}^{i}U{(y,{\mu }^{\vartheta })}_{{\mu }^i{\mu }^i}\right]$ is the gradient of the mean field term of ${\mu }^i$ with respect to ${\mu }^j$, and $G_{\mathsf{\Lambda }}=\frac{1}{2}\frac{\partial }{\partial \mathsf{\Lambda }}ln|\tilde{\mathsf{\Pi }}|\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.$ is the gradient of the log-determinant of the precision matrix with respect to $\mathsf{\Lambda }$. From Equation (49), the Jacobians that are required for the updates given in Equation (48) can be evaluated as:

$$\begin{array}{cc}\hfill J^X& =D+k^X\left[-E_{XX}+W_{XX}^X+W_{XX}^{\mathsf{\Theta }}+W_{XX}^{\mathsf{\Lambda }}\right]\hfill \\ \hfill J^{\mathsf{\Theta }}& =k^{\mathsf{\Theta }}\left[-P^{\theta }+\sum _t\left(-E_{\mathsf{\Theta }\mathsf{\Theta }}+W_{\mathsf{\Theta }\mathsf{\Theta }}^X+W_{\mathsf{\Theta }\mathsf{\Theta }}^{\mathsf{\Theta }}+W_{\mathsf{\Theta }\mathsf{\Theta }}^{\mathsf{\Lambda }}\right)\right]\hfill \\ \hfill J^{\mathsf{\Lambda }}& =k^{\mathsf{\Lambda }}\left[-P^{\lambda }+\sum _t\left(-E_{\mathsf{\Lambda }\mathsf{\Lambda }}+W_{\mathsf{\Lambda }\mathsf{\Lambda }}^X+W_{\mathsf{\Lambda }\mathsf{\Lambda }}^{\mathsf{\Theta }}+W_{\mathsf{\Lambda }\mathsf{\Lambda }}^{\mathsf{\Lambda }}+G_{\mathsf{\Lambda }\mathsf{\Lambda }}\right)\right]\hfill \end{array}$$

(50)

The Jacobians are employed in different loops corresponding to the D, E, and M steps in the Algorithm 1.

Algorithm 1: Dynamics Expectation Maximization

8.3. Update Equation for Precision of Estimates

The uncertainty in estimation is represented by the inverse of precision matrices ${\mathsf{\Pi }}^{{\vartheta }^i}$. Differentiating Equations (25) and (28) twice and substituting it into Equation (42) upon simplification yield:

$$\begin{array}{cc}\hfill {\mathsf{\Pi }}^{\tilde{x},\tilde{v}}& =-U{(y,{\mu }^{\vartheta })}_{\left[\begin{array}{c}\tilde{x}\\ \tilde{v}\end{array}\right]\left[\begin{array}{c}\tilde{x}\\ \tilde{v}\end{array}\right]}=E_{XX}\hfill \\ \hfill {\mathsf{\Pi }}^{\theta }& =-\overline{U}{(y,{\mu }^{\vartheta })}_{\theta \theta }=P^{\theta }+\sum _t{E}_{\mathsf{\Theta }\mathsf{\Theta }}\hfill \\ \hfill {\mathsf{\Pi }}^{\lambda }& =-\overline{U}{(y,{\mu }^{\vartheta })}_{\lambda \lambda }=P^{\lambda }+\sum _t\left(E_{\mathsf{\Lambda }\mathsf{\Lambda }}-G_{\mathsf{\Lambda }\mathsf{\Lambda }})\right)\hfill \end{array}$$

(51)

The only unknowns in the DEM update equations given by Equations (49)–(51) are the gradients and curvatures of E, W, and G. Section 9, Section 10 and Section 11 will deal with evaluating the simplified algebraic expressions for these gradients and curvatures. Using these simplifications, Section 12 will proceed towards expanding Algorithm 1.

9. Gradients of (Log Determinant of) Precision

This section aims at evaluating the gradients of the log determinant of noise precision ($G_{\mathsf{\Lambda }},G_{\mathsf{\Lambda }\mathsf{\Lambda }}$) that are required for the hyperparameter update rules of the DEM algorithm. The precision matrix for hyperparameter estimation is modeled as:

$$\begin{array}{cc}\hfill \tilde{\mathsf{\Pi }}& =diag({\tilde{\mathsf{\Pi }}}^z,P^{\tilde{v}},{\tilde{\mathsf{\Pi }}}^w)=diag(e^{{\lambda }^z}(S\otimes {\mathsf{\Omega }}^z),P^{\tilde{v}},e^{{\lambda }^w}(S\otimes {\mathsf{\Omega }}^w)),\hfill \end{array}$$

(52)

where S is the noise smoothness matrix given by Equation (7) and ${\mathsf{\Omega }}^z,{\mathsf{\Omega }}^w$ are the constant matrices given in Equation (6). Here the precision matrix is parametrized using $\lambda =\left[\begin{array}{c}{\lambda }^z\\ {\lambda }^w\end{array}\right]$, which is the only unknown in Equation (52). Therefore, the log determinant of precision and its gradients can be written as:

$$\begin{array}{cc}\hfill G& =ln|\tilde{\mathsf{\Pi }}|\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.,G_{\mathsf{\Lambda }}=\frac{1}{2}\left[\begin{array}{c}tr({\tilde{\mathsf{\Pi }}}_{{\lambda }^z}{\tilde{\mathsf{\Pi }}}^{-1})\\ tr({\tilde{\mathsf{\Pi }}}_{{\lambda }^w}{\tilde{\mathsf{\Pi }}}^{-1})\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.,\hfill \\ \hfill G_{{\mathsf{\Lambda }}^i{\mathsf{\Lambda }}^j}& =\frac{1}{2}tr({\tilde{\mathsf{\Pi }}}_{{\lambda }^i{\lambda }^j}{\tilde{\mathsf{\Pi }}}^{-1}-{\tilde{\mathsf{\Pi }}}_{{\lambda }^i}{\tilde{\mathsf{\Pi }}}^{-1}{\tilde{\mathsf{\Pi }}}_{{\lambda }^j}{\tilde{\mathsf{\Pi }}}^{-1})\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.,\hfill \end{array}$$

(53)

where ${\lambda }^i$ is the $i^{th}$ element in $\lambda$. The gradients of precision in Equation (53) can be evaluated by differentiating Equation (52) as:

$$\begin{array}{cc}\hfill & {\tilde{\mathsf{\Pi }}}_{{\lambda }^z}=diag(e^{{\lambda }^z}(S\otimes {\mathsf{\Omega }}^z),O,O)=diag({\tilde{\mathsf{\Pi }}}^z,O,O),\hfill \\ \hfill & {\tilde{\mathsf{\Pi }}}_{{\lambda }^w}=diag(O,O,e^{{\lambda }^w}(S\otimes {\mathsf{\Omega }}^z))=diag(O,O,{\tilde{\mathsf{\Pi }}}^z),\hfill \\ \hfill & {\tilde{\mathsf{\Pi }}}_{{\lambda }^z{\lambda }^w}=O,{\tilde{\mathsf{\Pi }}}_{{\lambda }^w{\lambda }^z}=O.\hfill \end{array}$$

(54)

Substituting Equations (52) and (54) in Equation (53) yields:

$$G_{\mathsf{\Lambda }}=\frac{1}{2}\left[\begin{array}{c}tr(I_{n^{{\tilde{\mathsf{\Pi }}}^z}})\\ tr(I_{n^{{\tilde{\mathsf{\Pi }}}^w}})\end{array}\right]=\frac{1}{2}\left[\begin{array}{c}{n}^{{\tilde{\mathsf{\Pi }}}^z}\\ n^{{\tilde{\mathsf{\Pi }}}^w}\end{array}\right],$$

(55)

where $I_{n^{{\tilde{\mathsf{\Pi }}}^z}}$ is the identity matrix of size $n^{{\tilde{\mathsf{\Pi }}}^z}$, which is the size of the ${\tilde{\mathsf{\Pi }}}^z$ matrix. ${\mathsf{\Pi }}^w$ and ${\mathsf{\Pi }}^z$ are modeled to have an exponential relation with $\lambda$, so that any updates on $\lambda$ would result in positive semi-definite precision matrices. However, this relation entails an infinitely differentiable precision matrix with respect to $\lambda$, increasing the computational complexity of the algorithm. Therefore, an approximation is made by forcefully setting ${\tilde{\mathsf{\Pi }}}_{{\lambda }^z{\lambda }^z}={\tilde{\mathsf{\Pi }}}_{{\lambda }^w{\lambda }^w}=O$, while maintaining the exponential relation between $\tilde{\mathsf{\Pi }}$ and $\lambda$, thereby ensuring that the optimization process proceeds along the correct gradients ${\tilde{\mathsf{\Pi }}}_{\lambda }$. Together with Equation (54), this approximation results in ${\tilde{\mathsf{\Pi }}}_{\lambda \lambda }=O$. This assumption has two direct consequences:

• It simplifies all the update rules given in Equations (49) and (50),
• It simplifies the precision update rule for hyperparameters given in Equation (51).

A direct consequence of this approximation is in the simplification of $G_{\mathsf{\Lambda }\mathsf{\Lambda }}$ in Equation (53), expressed as:

$$G_{\mathsf{\Lambda }\mathsf{\Lambda }}=-\frac{1}{2}\left[\begin{array}{cc}tr({\tilde{\mathsf{\Pi }}}_{{\lambda }^z}{\tilde{\mathsf{\Pi }}}^{-1}{\tilde{\mathsf{\Pi }}}_{{\lambda }^z}{\tilde{\mathsf{\Pi }}}^{-1})& tr({\tilde{\mathsf{\Pi }}}_{{\lambda }^z}{\tilde{\mathsf{\Pi }}}^{-1}{\tilde{\mathsf{\Pi }}}_{{\lambda }^w}{\tilde{\mathsf{\Pi }}}^{-1})\\ tr({\tilde{\mathsf{\Pi }}}_{{\lambda }^w}{\tilde{\mathsf{\Pi }}}^{-1}{\tilde{\mathsf{\Pi }}}_{{\lambda }^z}{\tilde{\mathsf{\Pi }}}^{-1})& tr({\tilde{\mathsf{\Pi }}}_{{\lambda }^w}{\tilde{\mathsf{\Pi }}}^{-1}{\tilde{\mathsf{\Pi }}}_{{\lambda }^w}{\tilde{\mathsf{\Pi }}}^{-1})\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.,$$

(56)

which upon substitution of Equations (52) and (54) yields:

$$G_{\mathsf{\Lambda }\mathsf{\Lambda }}=-\frac{1}{2}\left[\begin{array}{cc}tr(I_{n^{{\tilde{\mathsf{\Pi }}}^z}})& O\\ O& tr(I_{n^{{\tilde{\mathsf{\Pi }}}^w}})\end{array}\right]=-\frac{1}{2}\left[\begin{array}{cc}{n}^{{\tilde{\mathsf{\Pi }}}^z}& O\\ O& n^{{\tilde{\mathsf{\Pi }}}^w}\end{array}\right],$$

(57)

where $n^{{\tilde{\mathsf{\Pi }}}^z}=m(p+1)$ and $n^{{\tilde{\mathsf{\Pi }}}^w}=n(p+1)$ are the sizes of ${\tilde{\mathsf{\Pi }}}^z$ and ${\tilde{\mathsf{\Pi }}}^w$, respectively. From Equations (55) and (57), $G_{\mathsf{\Lambda }}$ and $G_{\mathsf{\Lambda }\mathsf{\Lambda }}$ are constants, and can be pre-computed in Algorithm 1.

10. Gradients of Prediction Error

As opposed to the result above, the gradients of the prediction errors are not constant, as is shown in this section.

10.1. Gradients of Prediction Error along (Generalized) States

The error in prediction of (generalized) outputs, inputs and states is represented together by $\widetilde{ϵ}$, which makes up the precision weighted prediction error defined by $E=\frac{1}{2}{\widetilde{ϵ}}^T\tilde{\mathsf{\Pi }}\widetilde{ϵ}\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.$, where $\tilde{\mathsf{\Pi }}=diag({\tilde{\mathsf{\Pi }}}^z,P^{\tilde{v}},{\tilde{\mathsf{\Pi }}}^w)$. The error and its gradient are:

$$\widetilde{ϵ}=\left[\begin{array}{c}\tilde{\mathbf{y}}-\tilde{C}\tilde{x}\\ \tilde{v}-{\eta }^{\tilde{v}}\\ D^x\tilde{x}-\tilde{A}\tilde{x}-\tilde{B}\tilde{v}\end{array}\right],{\widetilde{ϵ}}_X\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=\left[\begin{array}{cc}-\tilde{C}& O\\ O& I\\ D^x-\tilde{A}& -\tilde{B}\end{array}\right].$$

(58)

The gradient of prediction error with respect to X can be simplified as:

$$E_X={\widetilde{ϵ}}_X^T\tilde{\mathsf{\Pi }}\widetilde{ϵ}\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=A_{1}X+B_1\left[\begin{array}{c}\tilde{\mathbf{y}}\\ -{\eta }^{\tilde{v}}\end{array}\right],$$

(59)

where $A_1=\left[\begin{array}{cc}{\tilde{C}}^T{\tilde{\mathsf{\Pi }}}^z\tilde{C}+{(D^x-\tilde{A})}^T{\tilde{\mathsf{\Pi }}}^w(D^x-\tilde{A})& -{(D^x-\tilde{A})}^T{\tilde{\mathsf{\Pi }}}^w\tilde{B}\\ -{\tilde{B}}^T{\tilde{\mathsf{\Pi }}}^w(D^x-\tilde{A})& P^{\tilde{v}}+{\tilde{B}}^T{\tilde{\mathsf{\Pi }}}^w\tilde{B}\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.$ and $B_1=\left[\begin{array}{cc}-{\tilde{C}}^T{\tilde{\mathsf{\Pi }}}^z& O\\ O& P^{\tilde{v}}\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.$. Since $E_X$ is linear in X, differentiating Equation (59) with respect to X yields a simple expression for curvature $E_{XX}$ as:

$$\begin{array}{c}\hfill E_{XX}={\widetilde{ϵ}}_X^T\tilde{\mathsf{\Pi }}{\widetilde{ϵ}}_X\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=\left[\begin{array}{cc}{\tilde{C}}^T{\tilde{\mathsf{\Pi }}}^z\tilde{C}+{(D^x-\tilde{A})}^T{\tilde{\mathsf{\Pi }}}^w(D^x-\tilde{A})& -{(D^x-\tilde{A})}^T{\tilde{\mathsf{\Pi }}}^w\tilde{B}\\ -{\tilde{B}}^T{\tilde{\mathsf{\Pi }}}^w(D^x-\tilde{A})& P^{\tilde{v}}+{\tilde{B}}^T{\tilde{\mathsf{\Pi }}}^w\tilde{B}\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right..\end{array}$$

(60)

10.2. Gradients of Prediction Error along Parameters

To evaluate the gradients of prediction error along the parameters $\mathsf{\Theta }$, the reformulated definition of $\widetilde{ϵ}$ is used:

$$\widetilde{ϵ}=\left[\begin{array}{c}\tilde{\mathbf{y}}-N\theta \\ \tilde{v}-{\eta }^{\tilde{v}}\\ D^x\tilde{x}-M\theta \end{array}\right],{\widetilde{ϵ}}_{\mathsf{\Theta }}=({\widetilde{ϵ}}_{\theta }){|}_{\theta ={\mu }^{\theta }}=\left[\begin{array}{c}-N\\ O\\ -M\end{array}\right],$$

(61)

where M and N are given in Equation (5). This is to ensure that the variable $\mathsf{\Theta }$ can be separated out of the expression for $E_{\mathsf{\Theta }}$ such that it is linear in $\mathsf{\Theta }$ as follows:

$$\begin{array}{cc}\hfill E_{\mathsf{\Theta }}& =({\widetilde{ϵ}}_{\theta }^T\tilde{\mathsf{\Pi }}\widetilde{ϵ}){|}_{\vartheta ={\mu }^{\vartheta }}=-\left[\begin{array}{ccc}{N}^T& O& M^T\end{array}\right]\tilde{\mathsf{\Pi }}\left[\begin{array}{c}\tilde{\mathbf{y}}-N\theta \\ \tilde{v}-\tilde{\eta }\\ D^x\tilde{x}-M\theta \end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\hfill \\ \hfill & =(N^T{\tilde{\mathsf{\Pi }}}^{z}N+M^T{\tilde{\mathsf{\Pi }}}^{w}M)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\mathsf{\Theta }-\left[\begin{array}{cc}{N}^T{\tilde{\mathsf{\Pi }}}^z& M^T{\tilde{\mathsf{\Pi }}}^w{D}^x\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\left[\begin{array}{c}\tilde{\mathbf{y}}\\ {\mu }^{\tilde{x}}\end{array}\right]\hfill \\ \hfill & =A_2\mathsf{\Theta }-B_2\left[\begin{array}{c}\tilde{\mathbf{y}}\\ {\mu }^{\tilde{x}}\end{array}\right].\hfill \end{array}$$

(62)

Since $E_{\mathsf{\Theta }}$ is linear in $\mathsf{\Theta }$, differentiating Equation (62) with respect to $\mathsf{\Theta }$ yields a simple expression for $E_{\mathsf{\Theta }\mathsf{\Theta }}$ as:

$$E_{\mathsf{\Theta }\mathsf{\Theta }}=({\widetilde{ϵ}}_{\theta }^T\tilde{\mathsf{\Pi }}{\widetilde{ϵ}}_{\theta })\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=\left[N^T{\tilde{\mathsf{\Pi }}}^{z}N+M^T{\tilde{\mathsf{\Pi }}}^{w}M\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=A_2.$$

(63)

10.3. Gradients of Prediction Error along Hyperparameters

The gradients of prediction error along the hyperparameters $\mathsf{\Lambda }$ is simpler, and is given by $E_{\mathsf{\Lambda }}=\frac{1}{2}{\widetilde{ϵ}}^T{\tilde{\mathsf{\Pi }}}_{\lambda }\widetilde{ϵ}\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.,E_{\mathsf{\Lambda }\mathsf{\Lambda }}=\frac{1}{2}{\widetilde{ϵ}}^T{\tilde{\mathsf{\Pi }}}_{\lambda \lambda }\widetilde{ϵ}\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.$, which upon using ${\tilde{\mathsf{\Pi }}}_{\lambda \lambda }=0$, gives:

$$\begin{array}{cc}\hfill E_{\mathsf{\Lambda }}& =\frac{1}{2}tr({\tilde{\mathsf{\Pi }}}_{\lambda }\widetilde{ϵ}{\widetilde{ϵ}}^T)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=\frac{1}{2}\left[\begin{array}{c}tr({\tilde{\mathsf{\Pi }}}^z(\tilde{\mathbf{y}}-\tilde{C}\tilde{x}){(\tilde{\mathbf{y}}-\tilde{C}\tilde{x})}^T)\\ tr({\tilde{\mathsf{\Pi }}}^w(D^x\tilde{x}-\tilde{A}\tilde{x}-\tilde{B}\tilde{u}){(D^x\tilde{x}-\tilde{A}\tilde{x}-\tilde{B}\tilde{u})}^T)\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.,\hfill \\ \hfill E_{\mathsf{\Lambda }\mathsf{\Lambda }}& =\frac{1}{2}{\widetilde{ϵ}}^T{\tilde{\mathsf{\Pi }}}_{\lambda \lambda }\widetilde{ϵ}\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=0.\hfill \end{array}$$

(64)

In summary, Equations (59), (62) and (64) represent the gradients of the prediction error term, whereas Equations (60), (63) and (64) represent its curvatures. The next section will deal with evaluating the analytic expressions for all the gradients and curvatures of the mean field term.

11. Gradients of Mean Field Terms

This section aims to derive the analytic expressions for the mean field terms and their gradients: $W_{{\mu }^j}^{{\mu }^i}$ and $W_{{\mu }^j{\mu }^j}^{{\mu }^i},\forall \mu \in \{X,\mathsf{\Theta },\mathsf{\Lambda }\}$.

11.1. Gradients of Mean Field Terms along Hyperparameters

In this section, we prove that all the gradients and curvatures of $W^{\mathsf{\Lambda }}$ (namely $W_{{\mu }^i}^{\mathsf{\Lambda }}$ and $W_{{\mu }^i{\mu }^i}^{\mathsf{\Lambda }}$) are zeroes. The mean field term for hyperparameters $\mathsf{\Lambda }$ can be expressed as:

$$W^{\mathsf{\Lambda }}=\frac{1}{2}tr\left[{\mathsf{\Sigma }}^{\lambda }U{(y,{\mu }^{\vartheta })}_{\lambda \lambda }\right].$$

(65)

To compute the gradients of $W^{\mathsf{\Lambda }}$, we need the curvature of internal energy with respect to $\lambda$. This can be evaluated by first differentiating Equation (25) with respect to $\lambda$ and evaluating it at $\vartheta ={\mu }^{\vartheta }$, which yields:

$$\begin{array}{cc}\hfill U{(y,{\mu }^{\vartheta })}_{\lambda }=U{(y,\vartheta )}_{\lambda }\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.& =-P^{\lambda }{ϵ}^{\lambda }\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.-\frac{1}{2}({\widetilde{ϵ}}^T{\tilde{\mathsf{\Pi }}}_{\lambda }\widetilde{ϵ})\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.+G_{\mathsf{\Lambda }},\hfill \end{array}$$

(66)

where $G_{\mathsf{\Lambda }}$ is given by Equation (55). Upon further differentiation, we get:

$$\begin{array}{cc}\hfill U{(y,{\mu }^{\vartheta })}_{\lambda \lambda }& =-P^{\lambda }-\frac{1}{2}({\widetilde{ϵ}}^T{\tilde{\mathsf{\Pi }}}_{\lambda \lambda }\widetilde{ϵ})\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.+G_{\mathsf{\Lambda }\mathsf{\Lambda }}.\hfill \end{array}$$

(67)

The assumption of ${\tilde{\mathsf{\Pi }}}_{\lambda \lambda }=0$ applied to Equation (67) yields:

$$\begin{array}{c}\hfill U{(y,{\mu }^{\vartheta })}_{\lambda \lambda }=-P^{\lambda }+G_{\mathsf{\Lambda }\mathsf{\Lambda }}\end{array}$$

(68)

which contains only constants. Therefore, the assumption of ${\tilde{\mathsf{\Pi }}}_{\lambda \lambda }=0$ reduces all the gradients and curvatures of mean field terms of $\mathsf{\Lambda }$ to zeros:

$$W_{{\mu }^i}^{\mathsf{\Lambda }}=\frac{1}{2}tr\left[{\mathsf{\Sigma }}^{\lambda }U{(y,{\mu }^{\vartheta })}_{\lambda \lambda {\mu }^i}\right]=0,W_{{\mu }^i{\mu }^i}^{\mathsf{\Lambda }}=0.$$

(69)

Since the internal energy given in Equation (25) is quadratic in ${\vartheta }^i$, and since ${\tilde{\mathsf{\Pi }}}_{\lambda \lambda }=0$, all the gradients and curvatures of the mean field term of ${\vartheta }^i$ with respect to itself are zeros:

$$W_{{\mu }^i}^{{\mu }^i}=\frac{1}{2}tr\left[{\mathsf{\Sigma }}^{{\vartheta }^i}U{(y,{\mu }^{\vartheta })}_{{\vartheta }^i{\vartheta }^i{\mu }^i}\right]=0,W_{{\mu }^i{\mu }^i}^{{\mu }^i}=0.$$

(70)

11.2. Gradients of Mean Field Terms along Generalized States

The mean-field term of the combined generalized states X can be expressed as:

$$W^X=\frac{1}{2}tr\left[{\mathsf{\Sigma }}^{X}U{(y,{\mu }^{\vartheta })}_{XX}\right].$$

(71)

The curvature of internal energy with respect to X can be calculated by differentiating Equation (25) with respect to X twice, resulting in $U{(y,{\mu }^{\vartheta })}_{XX}=-\frac{1}{2}{\widetilde{ϵ}}_X^T\tilde{\mathsf{\Pi }}{\widetilde{ϵ}}_X$. Substituting it in Equation (71) upon differentiation with $\mathsf{\Theta }$ yields:

$$\begin{array}{cc}\hfill W_{{\mathsf{\Theta }}^i}^X& =-\frac{1}{2}tr({\mathsf{\Sigma }}^X{\widetilde{ϵ}}_{X{\theta }^i}^T\tilde{\mathsf{\Pi }}{\widetilde{ϵ}}_X)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=-\frac{1}{2}tr\left({\mathsf{\Sigma }}^X\left[\begin{array}{ccc}-{\tilde{C}}_{{\theta }^i}^T& O& -{\tilde{A}}_{{\theta }^i}^T\\ O& O& -{\tilde{B}}_{{\theta }^i}^T\end{array}\right]\tilde{\mathsf{\Pi }}\left[\begin{array}{cc}-\tilde{C}& O\\ O& I\\ D^x-\tilde{A}& -\tilde{B}\end{array}\right]\right)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\hfill \\ & =-\frac{1}{2}tr\left[{\mathsf{\Sigma }}^X\left[\begin{array}{cc}{\tilde{C}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^z\tilde{C}-{\tilde{A}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^w(D^x-\tilde{A})& {\tilde{A}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^w\tilde{B}\\ -{\tilde{B}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^w(D^x-\tilde{A})& {\tilde{B}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^w\tilde{B}\end{array}\right]\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.,\hfill \end{array}$$

(72)

where the gradient of the mean field with respect to $\mathsf{\Theta }$ is given by $W_{\mathsf{\Theta }}^X={\left[\begin{array}{cccc}{W}_{{\mathsf{\Theta }}^1}^X& W_{{\mathsf{\Theta }}^2}^X& \dots & W_{{\mathsf{\Theta }}^i}^X\end{array}\right]}^T$. Similarly, the elements of the curvature matrix of the mean field term with respect to $\mathsf{\Theta }$ is given by:

$$\begin{array}{c}\hfill W_{{\mathsf{\Theta }}^i{\mathsf{\Theta }}^j}^X=-\frac{1}{2}tr({\mathsf{\Sigma }}^X{\widetilde{ϵ}}_{X{\theta }^i}^T\tilde{\mathsf{\Pi }}{\widetilde{ϵ}}_{X{\theta }^j})\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=-\frac{1}{2}tr\left[{\mathsf{\Sigma }}^X\left[\begin{array}{cc}{\tilde{C}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^z{\tilde{C}}_{{\theta }^j}+{\tilde{A}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^w{\tilde{A}}_{{\theta }^j}& {\tilde{A}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^w{\tilde{B}}_{{\theta }^j}\\ {\tilde{B}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^w{\tilde{A}}_{{\theta }^j}& {\tilde{B}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^w{\tilde{B}}_{{\theta }^j}^T\end{array}\right]\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right..\end{array}$$

(73)

The gradient and curvature of the mean field term of X with respect to $\mathsf{\Lambda }$ can be evaluated as:

$$\begin{array}{cc}\hfill W_{\mathsf{\Lambda }}^X& =-\frac{1}{2}tr({\mathsf{\Sigma }}^X{\widetilde{ϵ}}_X^T{\tilde{\mathsf{\Pi }}}_{\lambda }{\widetilde{ϵ}}_X)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=-\frac{1}{2}tr({\tilde{\mathsf{\Pi }}}_{\lambda }{\widetilde{ϵ}}_X{\mathsf{\Sigma }}^X{\widetilde{ϵ}}_X^T)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\hfill \\ & =-\frac{1}{2}\left[\begin{array}{c}tr({\tilde{\mathsf{\Pi }}}^z\tilde{C}{\mathsf{\Sigma }}^x{\tilde{C}}^T)\\ tr({\tilde{\mathsf{\Pi }}}^w\left[\begin{array}{cc}(D^x-\tilde{A}),& -\tilde{B}\end{array}\right]{\mathsf{\Sigma }}^X\left[\begin{array}{c}{(D^x-\tilde{A})}^T\\ -{\tilde{B}}^T\end{array}\right])\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\hfill \\ \hfill W_{\mathsf{\Lambda }\mathsf{\Lambda }}^X& =-\frac{1}{2}tr({\tilde{\mathsf{\Pi }}}_{\lambda \lambda }{\widetilde{ϵ}}_X{\mathsf{\Sigma }}^X{\widetilde{ϵ}}_X^T)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=0,\hfill \end{array}$$

(74)

where ${\mathsf{\Sigma }}^x$ is a component of ${\mathsf{\Sigma }}^X=[\begin{array}{cc}{\mathsf{\Sigma }}^{\tilde{x}}& {\mathsf{\Sigma }}^{\tilde{x}\tilde{v}}\\ {\mathsf{\Sigma }}^{\tilde{x}\tilde{v}}& {\mathsf{\Sigma }}^{\tilde{v}}\end{array}]$. Here the curvature $W_{\mathsf{\Lambda }\mathsf{\Lambda }}^X$ vanishes due to the assumption that ${\tilde{\mathsf{\Pi }}}_{\lambda \lambda }=0$.

11.3. Gradients of Mean Field Terms along Parameters

The mean-field term of the parameters $\mathsf{\Theta }$ can be expressed as

$$W^{\mathsf{\Theta }}=\frac{1}{2}tr\left[{\mathsf{\Sigma }}^{\theta }U{(y,{\mu }^{\vartheta })}_{\theta \theta }\right]=\frac{1}{2}tr\left[{\mathsf{\Sigma }}^{\theta }(-P^{\theta }-{\widetilde{ϵ}}_{\theta }^T\tilde{\mathsf{\Pi }}{\widetilde{ϵ}}_{\theta })\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right..$$

(75)

Differentiating Equation (75) with X and substituting Equation (61) in it yields the gradient as:

$$W_X^{\mathsf{\Theta }}=-\frac{1}{2}\left[\begin{array}{c}tr\left({\mathsf{\Sigma }}^{\theta }\left(N_{X^1}^T{\tilde{\mathsf{\Pi }}}^{z}N+M_{X^1}^T{\tilde{\mathsf{\Pi }}}^{w}M\right)\right)\\ tr\left({\mathsf{\Sigma }}^{\theta }\left(N_{X^2}^T{\tilde{\mathsf{\Pi }}}^{z}N+M_{X^2}^T{\tilde{\mathsf{\Pi }}}^{w}M\right)\right)\\ \dots \end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.,$$

(76)

and the elements of the curvature matrix as:

$$W_{X^i{X}^j}^{\mathsf{\Theta }}=-\frac{1}{2}tr\left({\mathsf{\Sigma }}^{\theta }\left(N_{X^i}^T{\tilde{\mathsf{\Pi }}}^z{N}_{X^j}+M_{X^i}^T{\tilde{\mathsf{\Pi }}}^w{M}_{X^j}^T\right)\right)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right..$$

(77)

Differentiating Equation (75) with respect to $\lambda$ yields the gradient and curvature as:

$$\begin{array}{cc}\hfill W_{\mathsf{\Lambda }}^{\mathsf{\Theta }}& =-\frac{1}{2}tr({\mathsf{\Sigma }}^{\theta }{\widetilde{ϵ}}_{\theta }^T{\tilde{\mathsf{\Pi }}}_{\lambda }{\widetilde{ϵ}}_{\theta })\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=-\frac{1}{2}tr({\tilde{\mathsf{\Pi }}}_{\lambda }{\widetilde{ϵ}}_{\theta }{\mathsf{\Sigma }}^{\theta }{\widetilde{ϵ}}_{\theta }^T)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.,\hfill \\ \hfill & =-\frac{1}{2}\left[\begin{array}{c}tr({\tilde{\mathsf{\Pi }}}_{{\lambda }^z}{\widetilde{ϵ}}_{\theta }{\mathsf{\Sigma }}^{\theta }{\widetilde{ϵ}}_{\theta }^T)\\ tr({\tilde{\mathsf{\Pi }}}_{{\lambda }^w}{\widetilde{ϵ}}_{\theta }{\mathsf{\Sigma }}^{\theta }{\widetilde{ϵ}}_{\theta }^T)\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=-\frac{1}{2}\left[\begin{array}{c}tr({\tilde{\mathsf{\Pi }}}^{z}N{\mathsf{\Sigma }}^{\theta }{N}^T)\\ tr({\tilde{\mathsf{\Pi }}}^{w}M{\mathsf{\Sigma }}^{\theta }{M}^T)\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\hfill \\ \hfill W_{\mathsf{\Lambda }\mathsf{\Lambda }}^{\mathsf{\Theta }}& =-\frac{1}{2}tr({\tilde{\mathsf{\Pi }}}_{\lambda \lambda }{\widetilde{ϵ}}_{\theta }{\mathsf{\Sigma }}^{\theta }{\widetilde{ϵ}}_{\theta }^T)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=0.\hfill \end{array}$$

(78)

Here, $W_{\mathsf{\Lambda }\mathsf{\Lambda }}^{\mathsf{\Theta }}$ vanishes due to the assumption that ${\tilde{\mathsf{\Pi }}}_{\lambda \lambda }=0$.

12. The Complete DEM Algorithm

By combining the gradients found from Section 9, Section 10, and Section 11 with the Algorithm 1, we can finalize the full DEM algorithm so that it can iteratively compute the estimates and the associated precisions from data.

12.1. DEM Estimates

The main equations that are required to perform the update rules of DEM given in Equation (48) can be summarized as:

$$\begin{array}{cc}\hfill \dot{X}& =DX+k^X(-E_X+W_X^{\mathsf{\Theta }}),J^X=D+k^X(-E_{XX}+W_{XX}^{\mathsf{\Theta }})\hfill \\ \hfill \frac{\partial \mathsf{\Theta }}{\partial a}& =k^{\mathsf{\Theta }}\left[-P^{\theta }{ϵ}^{\theta }\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.+\sum _t\left(-E_{\mathsf{\Theta }}+W_{\mathsf{\Theta }}^X\right)\right],J^{\mathsf{\Theta }}=k^{\mathsf{\Theta }}\left[-P^{\theta }+\sum _t\left(-E_{\mathsf{\Theta }\mathsf{\Theta }}+W_{\mathsf{\Theta }\mathsf{\Theta }}^X\right)\right]\hfill \\ \hfill \frac{\partial \mathsf{\Lambda }}{\partial b}& =k^{\mathsf{\Lambda }}\left[-P^{\lambda }{ϵ}^{\lambda }\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.+\sum _t(-E_{\mathsf{\Lambda }}+W_{\mathsf{\Lambda }}^X+W_{\mathsf{\Lambda }}^{\mathsf{\Theta }}+G_{\mathsf{\Lambda }})\right],J^{\mathsf{\Lambda }}=k^{\mathsf{\Lambda }}(-P^{\lambda }+n^t{G}_{\mathsf{\Lambda }\mathsf{\Lambda }})\hfill \end{array}$$

(79)

where $E_X,E_{XX},E_{\mathsf{\Theta }},E_{\mathsf{\Theta }\mathsf{\Theta }},E_{\mathsf{\Lambda }},W_X^{\mathsf{\Theta }},W_{XX}^{\mathsf{\Theta }},W_{\mathsf{\Theta }}^X,W_{\mathsf{\Theta }\mathsf{\Theta }}^X,W_{\mathsf{\Lambda }}^X,$$W_{\mathsf{\Lambda }}^{\mathsf{\Theta }},G_{\mathsf{\Lambda }},and{G}_{\mathsf{\Lambda }\mathsf{\Lambda }}$ are given by Equations (55), (57), (59), (60), (62)–(64), (72)–(74), (76)–(78), respectively. The hyperparameter update rule can be further simplified to reduce the computational complexity as:

$$\frac{\partial \mathsf{\Lambda }}{\partial b}=-k^{\mathsf{\Lambda }}{P}^{\lambda }{ϵ}^{\lambda }\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.+\frac{k^{\mathsf{\Lambda }}{n}^t}{2}\left[\begin{array}{c}{n}^{{\tilde{\mathsf{\Pi }}}^z}\\ n^{{\tilde{\mathsf{\Pi }}}^w}\end{array}\right]-\frac{k^{\mathsf{\Lambda }}}{2}\left[\begin{array}{c}tr({\tilde{\mathsf{\Pi }}}^z{A}_3)\\ tr({\tilde{\mathsf{\Pi }}}^w{B}_3)\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.$$

(80)

where

$$\begin{array}{cc}\hfill A_3& =\sum _t\left((\tilde{\mathbf{y}}-\tilde{C}\tilde{x}){(\tilde{\mathbf{y}}-\tilde{C}\tilde{x})}^T+N{\mathsf{\Sigma }}^{\theta }{N}^T+\tilde{C}{\mathsf{\Sigma }}^{\tilde{x}\tilde{x}}{\tilde{C}}^T\right)\hfill \\ \hfill B_3& =\sum _t\left((D^x\tilde{x}-\tilde{A}\tilde{x}-\tilde{B}\tilde{u}){(D^x\tilde{x}-\tilde{A}\tilde{x}-\tilde{B}\tilde{u})}^T+M{\mathsf{\Sigma }}^{\theta }{M}^T+\left[\begin{array}{cc}(D^x-\tilde{A}),& -\tilde{B}\end{array}\right]{\mathsf{\Sigma }}^X\left[\begin{array}{c}{(D^x-\tilde{A})}^T\\ -{\tilde{B}}^T\end{array}\right]\right).\hfill \end{array}$$

Substituting Equation (57) to the expression for $J^{\mathsf{\Lambda }}$ in Equation (79) yields:

$$J^{\mathsf{\Lambda }}=-k^{\mathsf{\Lambda }}(P^{\lambda }+\frac{n^t}{2}\left[\begin{array}{cc}{n}^{{\tilde{\mathsf{\Pi }}}^z}& O\\ O& n^{{\tilde{\mathsf{\Pi }}}^w}\end{array}\right]),$$

(81)

which is independent of $\mathsf{\Lambda }$. This reduces the algorithm’s computational complexity, as $J^{\mathsf{\Lambda }}$ can now be pre-computed.

12.2. Precision of Estimates

This section simplifies the precision for DEM’s estimates for an LTI system. The confidence in the estimate of (generalized) states and inputs can be simplified using Equations (51) and (60) as:

$$\begin{array}{cc}\hfill & {\mathsf{\Pi }}^{\tilde{x},\tilde{v}}=E_{XX}=\left[\begin{array}{cc}{\tilde{C}}^T{\tilde{\mathsf{\Pi }}}^z\tilde{C}+{(D^x-\tilde{A})}^T{\tilde{\mathsf{\Pi }}}^w(D^x-\tilde{A})& -{(D^x-\tilde{A})}^T{\tilde{\mathsf{\Pi }}}^w\tilde{B}\\ -{\tilde{B}}^T{\tilde{\mathsf{\Pi }}}^w(D^x-\tilde{A})& P^{\tilde{v}}+{\tilde{B}}^T{\tilde{\mathsf{\Pi }}}^w\tilde{B}\end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right..\hfill \end{array}$$

(82)

From Equation (82), the precisions for state and input estimation are ${\mathsf{\Pi }}^{\tilde{x}\tilde{x}}={\tilde{C}}^T{\tilde{\mathsf{\Pi }}}^z\tilde{C}+{(D^x-\tilde{A})}^T{\tilde{\mathsf{\Pi }}}^w(D^x-\tilde{A})$ and ${\mathsf{\Pi }}^{\tilde{v}\tilde{v}}=P^{\tilde{v}}+{\tilde{B}}^T{\tilde{\mathsf{\Pi }}}^w\tilde{B}$, respectively. The cross-correlation between the (generalized) states and inputs are given by $-{\tilde{B}}^T{\tilde{\mathsf{\Pi }}}^w(D^x-\tilde{A})$. Since ${\mathsf{\Pi }}^{\tilde{x},\tilde{v}}$ is independent of X, it can be updated outside the D step.

Combining the results of Equations (51) and (63) yields the precision of parameter estimates ${\mathsf{\Pi }}^{\theta }$, which is independent of $\mathsf{\Theta }$, as:

$${\mathsf{\Pi }}^{\theta }=P^{\theta }+\sum _t{E}_{\mathsf{\Theta }\mathsf{\Theta }}=P^{\theta }+\sum _t\left[N^T{\tilde{\mathsf{\Pi }}}^{z}N+M^T{\tilde{\mathsf{\Pi }}}^{w}M\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right..$$

(83)

From Equations (51), (57) and (64), the precision of hyperparameter estimation is:

$${\mathsf{\Pi }}^{\lambda }=P^{\lambda }+\sum _t\left(E_{\mathsf{\Lambda }\mathsf{\Lambda }}-G_{\mathsf{\Lambda }\mathsf{\Lambda }})\right)=P^{\lambda }+\frac{n^t}{2}diag(n^{{\tilde{\mathsf{\Pi }}}^z},n^{{\tilde{\mathsf{\Pi }}}^w}),$$

(84)

which is a constant and hence is never updated in the algorithm. In conclusion, the estimation using Equation (79), along with the precision of these estimates given by Equations (82)–(84) completely define the DEM algorithm for an LTI system with colored noises. The complete DEM algorithm is given in Algorithm 2.

Algorithm 2: Dynamics Expectation Maximization

13. Translation into Simplified Mathematical Form

Although the pseudocode derived in the previous sections is sufficient to replicate the DEM algorithm for an LTI system with colored noise, it is not sufficient to analyze DEM using the standard control systems tools for stability checks, convergence, etc. Therefore, in this section we translate the algorithm into a simplified mathematical form that control engineers can easily analyze. The following subsections aim at converting the DEM updates into a coupled linear system.

13.1. State and Input Estimation as a Linear Observer

This section deals with reformulating the D step of DEM for an LTI system as a (generalized) state and input observer. Substituting Equation (59) in Equation (79) with a learning rate of $k^X=1$ yields [2]:

$$\dot{X}=\left[\begin{array}{c}\dot{\tilde{x}}\\ \dot{\tilde{v}}\end{array}\right]=(D-A_1)X-B_1\left[\begin{array}{c}\tilde{\mathbf{y}}\\ -{\eta }^{\tilde{v}}\end{array}\right]+W_X^{\mathsf{\Theta }}.$$

(85)

We now aim to mathematically prove that the (generalized) state and input observer of DEM can be reduced into an augmented LTI system, for which an exact discretization can be performed. We proceed by simplifying the mean field terms in Equation (39) as:

$$\begin{array}{cc}\hfill W_X^{\mathsf{\Theta }}=& -\frac{1}{2}\left[\begin{array}{c}tr\left({\mathsf{\Sigma }}^{\theta }(N_{X^1}^T{\tilde{\mathsf{\Pi }}}^{z}N+M_{X^1}^T{\tilde{\mathsf{\Pi }}}^{w}M)\right)\\ tr\left({\mathsf{\Sigma }}^{\theta }(N_{X^2}^T{\tilde{\mathsf{\Pi }}}^{z}N+M_{X^2}^T{\tilde{\mathsf{\Pi }}}^{w}M)\right)\\ \dots \end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\hfill \\ \hfill =& -\frac{1}{2}\left[\begin{array}{c}vec{({\mathsf{\Sigma }}^{\theta T})}^T(I\otimes (N_{X^1}^T{\tilde{\mathsf{\Pi }}}^z)vec(N))+\\ vec{({\mathsf{\Sigma }}^{\theta T})}^T(I\otimes (M_{X^1}^T{\tilde{\mathsf{\Pi }}}^w)vec(M))\\ \dots \end{array}\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\hfill \\ \hfill =& (Z_X^{N}vec(N)+Z_X^{M}vec(M))\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.,\hfill \end{array}$$

(86)

where,

$$Z_X^N=-\frac{1}{2}(I\otimes vec{({\mathsf{\Sigma }}^{\theta T})}^T)\left[\begin{array}{c}I\otimes (N_{X^1}^T{\tilde{\mathsf{\Pi }}}^z)\\ I\otimes (N_{X^2}^T{\tilde{\mathsf{\Pi }}}^z)\\ \dots \end{array}\right],Z_X^M=-\frac{1}{2}(I\otimes vec{({\mathsf{\Sigma }}^{\theta T})}^T)\left[\begin{array}{c}I\otimes (M_{X^1}^T{\tilde{\mathsf{\Pi }}}^w)\\ I\otimes (M_{X^2}^T{\tilde{\mathsf{\Pi }}}^w)\\ \dots \end{array}\right].$$

(87)

Since M and N can be obtained from linear transformation of X, $vec(M)$ and $vec(N)$ can be written as:

$$vec(M)=Z^{M}X\mathrm{and}vec(N)=Z^{N}X,$$

(88)

where $Z^M$ and $Z^N$ are matrices with elements 0 and 1. This leads to the mean field term being expressed as a linear transformation of X:

$$W_X^{\mathsf{\Theta }}=(Z_X^N{Z}^N+Z_X^M{Z}^M)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.X.$$

(89)

Substituting Equation (89) into Equation (85) simplifies the observer as:

$$\begin{array}{cc}\hfill \dot{X}& =A_{4}X+B_4\left[\begin{array}{c}\tilde{\mathbf{y}}\\ -{\eta }^{\tilde{v}}\end{array}\right],\hfill \\ \hfill A_4& =D-A_1+(Z_X^N{Z}^N+Z_X^M{Z}^M)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\mathrm{and}{B}_4=-B_1.\hfill \end{array}$$

(90)

The (generalized) state and input observer given by Equation (90) is of the form of an augmented LTI system. Therefore, an exact discretization can be used to solve it without using the second order gradient $J^X$ as given in Equation (48). This reduces the algorithm’s computational complexity because $E_{XX}$ and $W_{XX}^{\mathsf{\Theta }}$ for $J^X$ calculation are no longer necessary. Figure 3 shows the simplified control diagram of the observer. The stability condition of this observer (under known $\theta$ and $\lambda$) and its similarity with the Kalman Filter is discussed in our prior work [2]. To evaluate $W_X^{\mathsf{\Theta }}$, one could either use Equation (89) or Equation (76). Equation (90) is derived mainly for simplification and exact discretization.

13.2. Parameter Estimation—System Identification

This section aims to mathematically prove that the E step can be reduced to an augmented LTI system, for which an exact discretization can be performed. We proceed by first simplifying the parameter update equation given in Equation (79):

$$\frac{\partial \mathsf{\Theta }}{\partial a}=-P^{\theta }(\mathsf{\Theta }-{\eta }^{\theta })\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.+\sum _t(-E_{\mathsf{\Theta }}+W_{\mathsf{\Theta }}^X).$$

(91)

Grouping all $W_{{\mathsf{\Theta }}^i}^X$ using Equation (72) yields:

$$\begin{array}{cc}\hfill W_{\mathsf{\Theta }}^X& =-\frac{1}{2}\left(I\otimes vec{({\mathsf{\Sigma }}^{XT})}^T\right)\left[\begin{array}{c}I\otimes {\widetilde{ϵ}}_{X{\theta }^1}^T\tilde{\mathsf{\Pi }}\\ I\otimes {\widetilde{ϵ}}_{X{\theta }^2}^T\tilde{\mathsf{\Pi }}\\ \dots \end{array}\right]vec({\widetilde{ϵ}}_X)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.=Z_{\theta }^{ϵ}vec({\widetilde{ϵ}}_X)\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\hfill \end{array}$$

(92)

where

$$\begin{array}{cc}\hfill {\widetilde{ϵ}}_{X{\theta }^i}^T\tilde{\mathsf{\Pi }}& =-\left[\begin{array}{ccc}{\tilde{C}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^z& O& {\tilde{A}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^w\\ O& O& {\tilde{B}}_{{\theta }^i}^T{\tilde{\mathsf{\Pi }}}^w\end{array}\right],vec({\widetilde{ϵ}}_X)=-vec\left[\begin{array}{cc}\tilde{C}& O\\ O& O\\ \tilde{A}& \tilde{B}\end{array}\right]+vec\left[\begin{array}{cc}O& O\\ O& I\\ D^x& O\end{array}\right]=-Z^{\theta }\theta +Z^I.\hfill \end{array}$$

(93)

Here $Z^{\theta }$ and $Z^I$ are constant matrices with elements 0 and 1. Substituting $vec({\widetilde{ϵ}}_X)$ from Equation (93) in Equation (92) yields:

$$W_{\mathsf{\Theta }}^X=-(Z_{\theta }^{ϵ}{Z}^{\theta }){{|}_{\vartheta ={\mu }^{\vartheta }}\mathsf{\Theta }+(Z_{\theta }^{ϵ}{Z}^I)|}_{\vartheta ={\mu }^{\vartheta }}.$$

(94)

Substituting Equations (62) and (94) in Equation (91), simplifies the parameter update equation to

$$\begin{array}{cc}\hfill \frac{\partial \mathsf{\Theta }}{\partial a}& =A_5\mathsf{\Theta }+B_5,\hfill \\ \hfill A_5& =-[P^{\theta }+n^t(Z_{\theta }^{ϵ}{Z}^{\theta }){|}_{\vartheta ={\mu }^{\vartheta }}+\sum _t{A}_2],B_5=P^{\theta }{\eta }^{\theta }+n^t(Z_{\theta }^{ϵ}{Z}^I){|}_{\vartheta ={\mu }^{\vartheta }}+\sum _t{B}_2\left[\begin{array}{c}\tilde{\mathbf{y}}\\ {\mu }^{\tilde{x}}\end{array}\right],\hfill \end{array}$$

(95)

where $A_2$ and $B_2$ are given in Equation (62). Equation (95) is a linear differential equation in $\mathsf{\Theta }$ for which an exact discretization can be computed. For each $\mathsf{\Theta }$ update in Algorithm 2, $A_2$ and $B_2$ are also updated, consequently updating $A_5$ and $B_5$. Therefore, Equation (95) is equivalent to a linear time-varying system. Figure 4 shows the simplified parameter estimation step of the robot brain. To evaluate $W_{\mathsf{\Theta }}^X$, one could use either Equation (72) or Equation (94). Equation (94) was derived mainly for the exact discretization and for the convergence proof in Section 14.

13.3. Hyperparameter Update

The update equation in Equation (80) can be simplified as:

$$\begin{array}{cc}\hfill \frac{\partial \mathsf{\Lambda }}{\partial b}=& k^{\mathsf{\Lambda }}\left[-P^{\lambda }{ϵ}^{\lambda }+\frac{n^t}{2}\left[\begin{array}{c}{n}^{{\tilde{\mathsf{\Pi }}}^z}\\ n^{{\tilde{\mathsf{\Pi }}}^w}\end{array}\right]-\frac{1}{2}\left[\begin{array}{c}{e}^{{\lambda }^z}tr((S\otimes {\mathsf{\Omega }}^z)A_3)\\ e^{{\lambda }^w}tr((S\otimes {\mathsf{\Omega }}^w)B_3)\end{array}\right]\right]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right.\hfill \\ \hfill =& -\frac{k^{\mathsf{\Lambda }}}{2}\left[\begin{array}{cc}tr((S\otimes {\mathsf{\Omega }}^z)A_3)& O\\ O& tr((S\otimes {\mathsf{\Omega }}^w)B_3)\end{array}\right]e^{\mathsf{\Lambda }}-(k^{\mathsf{\Lambda }}{P}^{\lambda })\mathsf{\Lambda }+(k^{\mathsf{\Lambda }}{P}^{\lambda }{\eta }^{\lambda }+\frac{k^{\mathsf{\Lambda }}{n}^t}{2}\left[\begin{array}{c}{n}^{{\tilde{\mathsf{\Pi }}}^z}\\ n^{{\tilde{\mathsf{\Pi }}}^w}\end{array}\right])\hfill \\ \hfill =& a_1{e}^{\mathsf{\Lambda }}+a_2\mathsf{\Lambda }+a_3,\hfill \end{array}$$

(96)

where $a_1,a_2$, and $a_3$ are constants that are independent of $\mathsf{\Lambda }$. Since Equation (96) is nonlinear in $\mathsf{\Lambda }$, an approximate discretization like the conventional Gauss–Newton update scheme given in Equation (48) should be used for the M step. In summary, the D and E steps follow an exact discretization, whereas the M step follows an approximate discretization.

14. Convergence Proof for Parameter and Hyperparameter Estimation

In robotics, it is important that learning algorithms provide a stable solution, especially when robot safety during operation is a concern. Therefore, a proof of convergence for DEM is important for its widespread use in robotics as a learning algorithm. However, the DEM literature lacks any such mathematical proof of convergence for the estimator. Therefore, this section aims at providing one for the parameter and hyperparameter estimation step on LTI systems.

Since the update equation given by Equation (95) is a linear differential equation, proving that $A_5\prec O$ is sufficient to prove that $\mathsf{\Theta }$ converges to a stable solution. Substituting the expression for $A_2$ from Equation (62) to the $A_5$ in Equation (95), yields:

$$A_5=-[P^{\theta }+n^t{Z}_{\theta }^{ϵ}{Z}^{\theta }+\sum _t(N^T{\tilde{\mathsf{\Pi }}}^{z}N+M^T{\tilde{\mathsf{\Pi }}}^{w}M)]\left|{}_{\vartheta ={\mu }^{\vartheta }}\right..$$

(97)

Since the prior precision matrix can be chosen to be positive definite, $P^{\theta }\succ O$. It is straightforward to note from the expression for $A_2$ in Equation (62) that ${\sum }_t{A}_2\succ O$, because ${\tilde{\mathsf{\Pi }}}^z\succ O$, ${\tilde{\mathsf{\Pi }}}^w\succ O\Rightarrow N^T{\tilde{\mathsf{\Pi }}}^{z}N\succ O$, and $M^T{\tilde{\mathsf{\Pi }}}^{w}M\succ O$. Therefore, the proof of convergence is complete if we prove that $Z_{\theta }^{ϵ}{Z}^{\theta }\succ O$. Simplifying the expressions for $Z_{\theta }^{ϵ}$ and $Z^{\theta }$ from Equations (92) and (93), after some nontrivial linear algebra [41], yields:

$$Z_{\theta }^{ϵ}{Z}^{\theta }=\frac{1}{2}{\frac{\partial \tilde{\theta }}{\partial \theta }}^T{Z}_1{Z}_2\frac{\partial \tilde{\theta }}{\partial \theta }$$

(98)

where

$$\begin{array}{cc}\hfill Z_1=& diag({\tilde{\mathsf{\Pi }}}^w\otimes I,{\tilde{\mathsf{\Pi }}}^w\otimes I,{\tilde{\mathsf{\Pi }}}^z\otimes I),\frac{\partial \tilde{\theta }}{\partial \theta }=diag(vec{\tilde{A}}_{vec{A}^T}^T,vec{\tilde{B}}_{vec{B}^T}^T,vec{\tilde{C}}_{vec{C}^T}^T),\hfill \\ \hfill Z_2=& \left[\begin{array}{ccc}I\otimes {\mathsf{\Sigma }}^{\tilde{x}\tilde{x}T}& I\otimes {\mathsf{\Sigma }}^{\tilde{v}\tilde{x}T}& O\\ I\otimes {\mathsf{\Sigma }}^{\tilde{x}\tilde{v}T}& I\otimes {\mathsf{\Sigma }}^{\tilde{v}\tilde{v}T}& O\\ O& O& I\otimes {\mathsf{\Sigma }}^{\tilde{x}\tilde{x}T}\end{array}\right].\hfill \end{array}$$

It is straightforward from Equation (98) that $Z_{\theta }^{ϵ}{Z}^{\theta }\succ O$ because $Z_1\succ O$, and $Z_2\succ O$. Combining all the results from this section, $P^{\theta }\succ O$, $Z_{\theta }^{ϵ}{Z}^{\theta }\succ O$ and ${\sum }_t{A}_2\succ O,\Rightarrow A_5\prec O$. This completes the proof that the parameter estimation step of DEM converges for an LTI system. Similarly, from Equation (81), $J^{\mathsf{\Lambda }}\prec O$ proves the convergence of hyperparameter estimation step. For a detailed account of the linear algebra behind the proof of convergence, readers may refer to [41].

15. A Demonstrative Example

This section aims to provide the proof of concept for DEM through simulation for the estimation of an LTI system with colored noise. Since the algorithm can find an infinite number of solutions for a black box estimation of $\tilde{x}$, $\tilde{v}$, $\theta$ and $\lambda$ from $\mathbf{y}$, a black box estimation is not ideal as a demonstrative example. Therefore, we restrict this section to the joint estimation of x, A, B, ${\mathsf{\Pi }}^w$, and ${\mathsf{\Pi }}^z$ from known $\mathbf{y}$ and C.

15.1. Generative Model

A stable LTI system of the form Equation (3) was selected, with randomly generated parameters ${\theta }^i\in [-1,1]$ having

$$A=\left[\begin{array}{cc}0.0484& 0.7535\\ -0.7617& -0.2187\end{array}\right],B=\left[\begin{array}{c}0.3604\\ 0.0776\end{array}\right],C=\left[\begin{array}{cc}0.2265& -0.4786\\ 0.4066& -0.2641\\ 0.3871& 0.3817\\ -0.1630& -0.9290\end{array}\right].$$

A Gaussian bump input signal of $v=e^{-0.25{(t-12)}^2}$ was centered around $t=12s$ and sampled at $dt=0.1s$ till $T=32s$ was used. The colored noise was generated with a smoothness value of $\sigma =0.5$ for the Gaussian kernel. The noise precisions were ${\mathsf{\Pi }}^w=e^8{I}_2$ and ${\mathsf{\Pi }}^z=e^8{I}_4$, making ${\lambda }^z={\lambda }^w=8$. The embedding order of the generalized motion of states and inputs were $p=6$ and $d=2$, respectively.

15.2. Priors for Estimation

As discussed in Section 5.2, three prior distributions are necessary for the algorithm. Since the inputs are known, the input prior ${\eta }^v$ is initialized with the known input v, and a tight prior precision of $P^v=e^{32}{I}_1$ is used to restrict any changes in v. Similarly, since the parameter C is known, the corresponding prior parameters in ${\eta }^{\theta }$ are initialized with C, with tight priors of $P^{{\theta }^i}=e^{32}$. The prior parameters ${\eta }^{{\theta }^i}$ for the unknown A and B matrices are randomly sampled from the range of [−2,2], and a low prior precision of $P^{{\theta }^i}=e^6$ is used to encourage exploratory behavior. In summary, ${\eta }^{\theta }={\left[\begin{array}{ccc}vec{(rand{(2,2)}^T)}^T& vec{(rand{(2,1)}^T)}^T& vec{(C^T)}^T\end{array}\right]}^T$ and $P^{\theta }=diag(e^6{I}_4,e^6{I}_2,e^{32}{I}_8)$. Since the hyperparameters are unknown, their priors were set to zero $\lambda ={[00]}^T$, with a prior precision of $P^{\lambda }=e^3{I}_2$ to encourage exploration.

15.3. Results of Estimation

The data $\mathbf{y}$ generated from the system in Section 15.1 was used to run the DEM algorithm given in Algorithm 2. Figure 5a demonstrates the successful state estimation of the algorithm. The results of parameter estimation (A and B) are shown in Figure 5b. The updates began from randomly selected priors ${\eta }^{\theta }$, marked by red circles, to finally converge.

Table 1. DEM’s estimate of A and B converges to real value.

	${\mathit{\theta }}_1$	${\mathit{\theta }}_2$	${\mathit{\theta }}_3$	${\mathit{\theta }}_4$	${\mathit{\theta }}_5$	${\mathit{\theta }}_6$
Real	0.048	0.753	−0.761	−0.218	0.360	0.077
Estimate	0.034	0.714	−0.769	−0.219	0.333	0.098

shows that the DEM’s estimate of A and B are close to the real values.

This confirms that the parameter estimation can converge close to the real parameters, even when ${\eta }^{\theta }$ is randomly selected from the range ${\eta }^{\theta }\in [-2,2]$ that is double the size of the real parameter range ${\theta }^i\in [-1,1]$. Figure 5c shows the successful hyperparameter convergence close to ${\lambda }^z={\lambda }^w=8$.

DEM’s confidence on its estimates increase with the E step iterations, as can be seen from Figure 6a, which demonstrates an increase in parameter precision ${\mathsf{\Pi }}^{\theta }$. A similar trend can be observed for ${\mathsf{\Pi }}^X$. However, ${\mathsf{\Pi }}^{\lambda }$ remains a constant during the entire algorithm, as proved in Section 12.2. The key idea behind DEM’s inference is the maximization of free energy objectives. Read together, Figure 5 and Figure 6 demonstrates that DEM successfully estimates $\tilde{x}$, $\theta$ and $\lambda$, with increasing confidence on its estimates as the estimation proceeds by maximizing $\overline{F}$ from Equation (40). In summary, DEM can be used for the joint estimation of states, parameters and hyperparameters of an LTI system, subjected to colored noise.

16. Benchmarking

This section deals with benchmarking DEM against the state-of-the-art parameter estimation methods such as Expectation Maximization (EM), Subspace method (SS), and Prediction Error Minimization (PEM), for black-box estimation (fully unknown x, $\theta$ and $\lambda$).

16.1. Evaluation Metric for Parameter Estimation

For the black box identification, with completely unknown x, $\theta$, and $\lambda$, there are infinite solutions with accurate input–output mapping. However, for LTI systems, there exists a unique transformation for identical systems. We use the companion canonical form to check the validity of parameter estimation by transforming both the real and the estimated parameters into their companion canonical form and then using the (square of) Euclidean distance between them as the sum of squared error (SSE) in parameter estimation. This evaluation metric will be used for parameter estimation in the next section.

16.2. Simulation Setup

A total of 500 (5 × 100) different randomly generated stable systems were used with five different noise smoothness values for parameter estimation. All systems were selected with same number of parameters $n^{\theta }=14$ ($n=2,m=4$ and $r=1$), with each ${\theta }^i\in [-1,1]$, while ensuring that A matrix is stable. All the noises were generated with the precision of $e^6({\mathsf{\Pi }}^w=e^6{I}_{2\times 2},{\mathsf{\Pi }}^z=e^6{I}_{4\times 4})$, with the embedding orders of states and inputs as $p=6$ and $d=2$. A Gaussian bump of $v=e^{-0.25*{(t-12)}^2}$ was used as the input signal with $dt=0.5s$ and $T=32s$. The prior parameter was randomly initialized such that all ${\eta }^{{\theta }^i}\in [-2,2]$ with a tight prior precision of $P^{\theta }=e^4{I}_{14\times 14}$. Both the hyperparameter priors ${\eta }^{{\lambda }^i}$ were set to zero, with a prior precision of $P^{\lambda }=e^{-4}{I}_{2\times 2}$.

The System Identification toolbox from MATLAB was used for SS ($n4sid()$) and PEM methods. The solution of SS was used to initialize PEM. An implementation of EM algorithm for state space models was written in MATLAB based on [45]. $n4sid()$ is inherently designed to handle colored noise, whereas the implemented EM algorithm is not. The code for the DEM algorithm will be openly available at: https://github.com/ajitham123/DEM_LTI.

16.3. Results

The results shown in Figure 7 demonstrate the superior performance of DEM in comparison with EM, PEM, and SS, with minimum SSE during parameter estimation across different noise smoothness. Additionally, EM and PEM exploded occasionally ($<5\%$ times), resulting in outliers in SSE, which were removed for better visualization. DEM demonstrated a consistent performance without generating any such outliers or exploding solutions, which could be explained by DEM’s convergence guarantees for parameter estimation under colored noise [41], as proved in Section 14. In summary, DEM is a competitive parameter estimator for LTI systems with colored noise.

17. Discussion

The quest for a brain-inspired learning algorithm for robots has culminated in the free energy principle that postulates biological brain’s perception as an optimization over its free energy objectives. FEP is of prime importance to robotics because of the use of generalized coordinates that enables it to gracefully handle colored noises. Colored noises appear in real robotics systems through the unmodeled dynamics and the non-linearity errors in the model, thereby providing an advantage for DEM during estimation when compared to other estimators. An example could be the unmodeled wind disturbances acting on an unmanned aerial vehicle while in flight, or the non linearity errors in the dynamic model of a robotic manipulator arm involved in a pick and place operation. The scope of this work spans across the blind system identification of such linear dynamic systems with colored noise.

The fundamental difference between this work and the prior work is in the reformulation of DEM for an LTI system. While DEM from computational neuroscience focuses on emulating the biological brain’s perception through the hierarchical abstraction of a number of non-linear dynamic systems that interact with each other, our work focuses on reducing this method into an algorithm for the system identification of an LTI system with colored noise, which is a well-known problem in robotics. This reformulation enables the standard analysis for convergence, stability and unbiased estimation, which is an essential analysis in practical robotics. It also enables DEM to be compared with other existing estimation algorithms in a control systems domain. The widespread use of DEM in robotics necessitates these mathematical analyses, especially when concerning the stable and safe operation of robots in industry and during human–robot interaction.

An algorithm with proved convergence for estimation is preferred for safe robotic applications. Therefore, one of the main contribution of this work was the reduction of the estimation algorithm into a coupled augmented system to prove the convergence of parameter and hyperparameter estimation steps. This work also demonstrated the successful applicability of DEM for the estimation of a randomly selected LTI system. Furthermore, we showed through rigorous simulations on a wide range of randomly generated LTI systems that DEM is a competitive algorithm for system identification under colored noise, thereby widening the scope of DEM to a large number of LTI systems in robotics.

One of the main drawbacks of the algorithm is its higher computational complexity when compared to the estimation algorithms that do not keep track of the trajectory of states. Therefore, future work can focus on the online estimation using DEM with reduced computational load. Future work can also focus on extending this algorithm for linear time varying systems to deal with robots with changing system parameters while in operation—a delivery drone dropping deliveries in mid-flight, for example. From a practical robotics point of view, DEM’s parameter estimation module can be directly applied to a wide range of robots such as quadrotors, robotic arms, wheeled robots, etc. for black-box system identification, the input estimation module can be employed for fault-detection systems, and the hyperparameter estimation module can be used for online noise estimation for robust control. DEM can also be extended with a control loop for active inference to perform simultaneous perception and action on robots. This would result in the development of cognitive robots that can learn the generative model in the environment by interacting with it and actively seeking new information (active learning) for uncertainty resolution. This would influence multiple domains in robotics such as human–robot interaction for task learning, swarm robotics for collective learning and distributed control, informative path planning of aerial robots for environment monitoring, etc. The development of such brain-inspired autonomous agents sits at the core of cognitive robotics research. In summary, DEM has a huge potential to be the bioinspired learning algorithm for future robots.

18. Conclusions

The free energy principle from neuroscience has a great potential to be one of the most prominent frameworks for learning and control for the autonomous systems in future. Therefore, this paper converted the FEP-based inference scheme called DEM into a joint state, input, parameter, and hyperparameter estimation algorithm for LTI systems with colored noise. We derived the mathematical framework of DEM for LTI systems to prove that the resulting estimator is a combination of linear estimators that are coupled. We provided the proof of convergence for the estimation steps. Through rigorous simulations on randomly generated linear systems with colored noise at varying smoothness levels, we demonstrated that the DEM algorithm outperforms EM, PEM, and SS methods for parameter estimation with minimal estimation error. In light of the potential for DEM to solve the parameter estimation problem, the future research will aim at applying DEM to a quadcopter flying in wind.