A Hierarchical Bayesian Model for Inferring and Decision Making in Multi-Dimensional Volatile Binary Environments

Abstract

The ability to track the changes of the surrounding environment is critical for humans and animals to adapt their behaviors. In high-dimensional environments, the interactions between each dimension need to be estimated for better perception and decision making, for example in volatile or social cognition tasks. We develop a hierarchical Bayesian model for inferring and decision making in multi-dimensional volatile environments. The hierarchical Bayesian model is composed of a hierarchical perceptual model and a response model. Using the variational Bayes method, we derived closed-form update rules. These update rules also constitute a complete predictive coding scheme. To validate the effectiveness of the model in multi-dimensional volatile environments, we defined a probabilistic gambling task modified from a two-armed bandit. Simulation results demonstrated that an agent endowed with the proposed hierarchical Bayesian model is able to infer and to update its internal belief on the tendency and volatility of the sensory inputs. Based on the internal belief of the sensory inputs, the agent yielded near-optimal behavior following its response model. Our results pointed this model a viable framework to explain the temporal dynamics of human decision behavior in complex and high dimensional environments.

1. Introduction

Natural environments are volatile, with ever changing sensory distributions and reward contingencies [1]. In a volatile environment, a biological agent must maintain stable internal states and be able to efficiently capture effective sensory information at the same time [2,3]. These seemingly contradictory requirements are unified into Bayesian inference [4], an optimal probability inference process.

Neuroscience research has shown that Bayesian inference underlies brain functions, such as perception, memory and decision-making, and resulting adaptive animal behaviors [3,5,6,7,8,9,10,11]. Adaptive behaviors are rooted in perceptual inferences and adaptive behavioral responses [12,13,14,15,16]. To understand the mechanisms of adaptive behaviors, one basic approach is to employ a generative model to infer the probabilistic distribution of sensory information and reproduce the temporal dynamics of human perception and decision-making in dynamic environments [17,18]. In this view, a Bayesian agent with a generative model is able to transform sensory inputs into behavioral responses [19]. With appropriate choice of parameters, a Bayesian agent could account for human decision behaviors.

“Observing the observer” is a meta Bayesian framework to simulate the perception processes of humans [20,21]. Perceptual and response models are two key components of this framework. According to this theoretic framework, inversion of the perceptual and response models can map from sensory inputs to response actions based on variational free energy principle [15,22,23] or Bayes’ rule.

To deal with volatile environments, volatility models, such as Hierarchical Gaussian Filtering [23], are developed to deliver an estimation of the changes of the environment. Accumulating evidences from the research on human learning and perception have shown that volatile Bayesian models (e.g., Hierarchical Gaussian Filtering) well explain human behaviors, especially, in changing environments. For example, saccadic response speed can be modulated by prediction precision of the belief on sensory inputs [24]. The volatility of the sensory environment and changes in sensory inputs are overestimated by adults with autism spectrum disorders [14]. This overestimation of volatility leads to the reduced precision of prior belief on sensory inputs. In human social learning, hierarchical prediction errors are encoded by midbrain and septum activity [25]. These evidences manifest that hierarchical Bayesian inference provides an optimal scheme to diminish surprise and reduce uncertainty in a volatility world.

To gain theoretical understandings of decision making under uncertainty with finite resource, the multi-armed bandit problem has been formulated as an abstraction [26,27,28,29]. The goal of the multi-armed bandit problem is to maximize the overall rewards through a series of choices. In neuroscience, multi-armed bandit problem is widely used to investigate economic decision making, contingent learning and human social behavior [30,31,32,33,34,35]. Animals and humans often have to make perceptual inference and settle on a series of decisions in a complex volatility environment. In general, the state space of decision making is high-dimensional. For example, in social interactions, the behaviors of multi-agents play important roles in the decisions of each individual. In a particular situation, agents employ internal models to observe other agents’ behaviors and to simulate their belief about actions [34,35]. The interactions between agents result in complex and correlated behaviors such as competition, cooperation, prediction and judgment. To describe multi-agents’ behaviors in social tasks, models that are able to capture dynamic information and noisy correlation in multi-dimensional state space need to be developed [36].

Bayesian networks are widely used for the inference of features from observed data [37,38,39,40]. In recent years, hierarchical Bayesian networks are developed to model the compositional nature of complex features for recognition tasks [41,42]. To solve perceptual inference and decision making problems in high-dimensional volatile binary environments, in this paper, we develop a hierarchical Bayesian model to infer time-varying hidden states of multi-armed bandits and maximize rewards given uncertain high-dimensional sensory inputs.

In summary, our model is promising to solve complex inference and decision making problems in realistic environments, which are intrinsically dynamic and high dimensional. In addition, our model could be applied to reveal computational mechanisms underlying human cognition and behaviors [43,44].

The rest of this paper is structured as follows. Section 2 introduces the hierarchical Bayesian perceptual model in high-dimensional volatile binary environment. Section 3 derives a set of closed form update equations for perceptual inference. Section 4 develops a response model for reward maximization in volatile multi-armed bandits as a typical example. Experimental results are given in Section 5. Finally, the paper is concluded with discussions.

2. Hierarchical Bayesian Perceptual Model

2.1. Beyond Independency

As a classic task in neuroscience and reinforcement learning, a multi-armed bandit challenges the agent with uncertain reward distribution, revealing rewards probabilistically. Since the agent has to estimate both the mean reward (for exploitation) and the precision of mean reward (for exploration), the multi-armed bandit captures the exploration–exploitation tradeoff dilemma in reward maximization under uncertainty [26,45].

Put simply, a one-armed bandit can be considered as a random binary number generator described by a Bernoulli distribution

$$\begin{array}{c}\hfill \mathrm{Bern}(x_0;{\mu }_0)={\mu }_0^{x_0}{(1-{\mu }_0)}^{1-x_0},\end{array}$$

(1)

where $x_0\in \{0,1\}$, the state of the one-armed bandit, represents “reward” ($x_0=1$) or “no reward” ($x_0=0$). ${\mu }_0\in [0,1]$ is the probability of being in the reward state. For a multi-armed bandit, $x_0^{\left(i\right)}$ and ${\mu }_0^{\left(i\right)}$ denote the observation and expectation of reward in the i-th arm. The binary vector ${\mathit{x}}_0={[x_0^{\left(1\right)},x_0^{\left(2\right)},\cdots ,x_0^{\left(d_0\right)}]}^T$ constitutes a binary pattern corresponding to the state of rewarding or non-rewarding of the arms at time t, with $d_0$ being the total number of the arms. Throughout the paper we use the notation $\left(i\right)$ in the superscript to indicate the i-th element of a vector.

Assuming independence between the reward distributions of the arms, the joint probability of being in state ${\mathit{x}}_0$ is given by the product of reward probabilities of the arms, equivalent to

$$p\left({\mathit{x}}_0\right)=exp(\sum _{i=1}^{d_0}[x_0^{\left(i\right)}ln{\mu }_0^{\left(i\right)}+(1-x_0^{\left(i\right)})ln(1-{\mu }_0^{\left(i\right)})]).$$

(2)

However, this independent model is not able to capture possible interaction structure of the arms.

In volatile environments, the reward distributions are non-stationary and often evolve dependently on each other, showing time-variant interaction strength. To quantitatively describe the interactions among the arms of a multi-armed bandit, we introduce the concept that there are low-order interactions among the natural parameters of the underlying multivariate Bernoulli distribution. Denoted by ${\mathit{x}}_1$, the natural parameter vector is mapped to a point ${\mathit{\mu }}_0$ in the probability space through a multivariate element-wise sigmoid function $\mathit{s}$

$${\mathit{\mu }}_0\left(t\right)=\mathit{s}({\mathit{x}}_{\mathbf{1}}\left(t\right),{\mathit{\zeta }}_1),$$

(3)

with the i-th element of ${\mathit{\mu }}_0\left(t\right)$ being

$$\begin{array}{cc}\hfill {\mu }_0^{\left(i\right)}\left(t\right)=& s(x_1^{\left(i\right)}\left(t\right),{\zeta }_1^{\left(i\right)})\hfill \\ \hfill =& \frac{1}{1+exp(-{\zeta }_1^{\left(i\right)}{x}_1^{\left(i\right)}\left(t\right))},\hfill \\ \hfill & i\in \{1,2,\cdots ,d_1\}.\hfill \end{array}$$

(4)

$d_1=d_0$ is the dimension of ${\mathit{x}}_1$. $\mathit{s}({\mathit{x}}_{\mathbf{1}},{\mathit{\zeta }}_1)$ is a vector-valued function defined by

$${[s(x_1^{\left(1\right)},{\zeta }_1^{\left(1\right)}),s(x_1^{\left(2\right)},{\zeta }_1^{\left(2\right)}),\cdots ,s(x_1^{\left(d_1\right)},{\zeta }_1^{\left(d_1\right)})]}^T.$$

The parameter vector

$${\mathit{\zeta }}_1={[{\zeta }_1^{\left(1\right)},{\zeta }_1^{\left(2\right)},\cdots ,{\zeta }_1^{\left(d_1\right)}]}^T$$

is the inverse temperature, with positive elements $({\zeta }_1^{\left(i\right)}>0)$.

2.2. Perceiving Tendency and Volatility

In volatile environments, variables of interest, such as reward, are subject to changes. The changes of a variable are again subject to changes, and so forth. The nested nature of volatility is a hallmark of collective phenomena as observed in animal swarms, the financial market and social behavior. To quantitatively describe volatility and pairwise correlations of high dimensional variables, we have developed a hierarchical volatility model, called General Hierarchical Brownian Filter (GHBF), based on the idea of nested Brownian motions [46]. Following this framework, we develop here a hierarchical perceptual model to estimate both the tendency and volatility in the states of a multi-armed bandit (Figure 1). More specifically, the natural parameters ${\mathit{x}}_{\mathbf{1}}$ of the underlying multivariate Bernoulli distribution is modeled by a general Brownian motion with pervasion matrix ${\mathbf{\Sigma }}_1\in {\mathbb{R}}^{d_1\times d_1}$

$${\mathit{x}}_1=\mathcal{B}(t;{\mathbf{\Sigma }}_1).$$

(5)

This Brownian motion captures the tendency of the learned parameter vector ${\mathit{x}}_1$. The volatility (i.e., uncertainties and pairwise correlations) in ${\mathit{x}}_1$ is given by ${\mathbf{\Sigma }}_1\in {\mathbb{R}}^{d_1\times d_1}$, which is a symmetric positive definite matrix by definition.

Considering the fact that the pervasion intensity ${\mathbf{\Sigma }}_1$ is a symmetric positive definite matrix, it could be uniquely represented by a lower triangular matrix ${\mathit{L}}_1\in {\mathbb{R}}^{d_1\times d_1}$ according to Cholesky decomposition

$${\mathbf{\Sigma }}_{\mathbf{1}}={\mathit{L}}_{\mathbf{1}}{\mathit{L}}_{\mathbf{1}}^T.$$

To further evaluate the volatility ${\mathbf{\Sigma }}_1$ (i.e., uncertainties and pairwise correlations) in ${\mathit{x}}_1$, we assume that its decomposition ${\mathit{L}}_{\mathbf{1}}$ is modeled by a general Brownian motion in its parameterized space. To be exact, the elements of ${\mathit{L}}_{\mathbf{1}}$ is parametrized by a $d_2=d_1(d_1+1)/2$ dimensional vector ${\mathit{y}}_{\mathbf{2}}$, which results from concatenating the lower triangle elements of ${\mathit{L}}_{\mathbf{1}}$ in a column-wise fashion. The element in i-th row and j-th column of ${\mathit{L}}_{\mathbf{1}}$ is given by

$$\begin{array}{cc}\hfill & {\mathit{L}}_{\mathbf{1}}^{(i,j)}=l_1^{(i,j)}=\hfill \\ \hfill & \left\{\begin{array}{cc}2sinh(y_2^{(\frac{(2d_1-j+2)(j-1)}{2}+i-j+1)}),\hfill & 1\le j<i\le d_1\hfill \\ exp(y_2^{(\frac{(2d_1-i+2)(i-1)}{2}+1)}),\hfill & j=i\hfill \end{array}\right.\hfill \end{array}$$

(6)

where $sinh(·)$ denotes a hyperbolic sine function. The vector ${\mathit{y}}_2$ gives the logarithm of volatility in the second level

$${\mathit{y}}_2={\mathit{W}}_{\mathbf{2}}{\mathit{x}}_{\mathbf{2}}+{\mathit{b}}_{\mathbf{2}}.$$

(7)

The coefficient matrix ${\mathit{W}}_2$ is a $d_2$-by-$d_2$ diagonal matrix and represents the coupling strength from level two to level one. Here, ${\mathit{W}}_2$ can simply take the form of a diagonal matrix spanned from a column vector ${\mathit{w}}_2$ with all positive elements

$${\mathit{W}}_{\mathbf{2}}^{(i,i)}={\mathit{w}}_2^{\left(i\right)}.$$

${\mathit{b}}_{\mathbf{2}}$ and ${\mathit{x}}_{\mathbf{2}}\in {\mathbb{R}}^{d_2}$ represents trend and time-varying fluctuation in log-volatility of the natural parameter respectively. We may further assume that ${\mathit{x}}_{\mathbf{2}}$ evolves as a general Brownian motion with pervasion matrix ${\mathbf{\Sigma }}_2\in {\mathbb{R}}^{d_2\times d_2}$

$${\mathit{x}}_2=\mathcal{B}(t;{\Sigma }_2).$$

(8)

We can rewrite the coupling (Equations (6) and (7)) as

$${\mathit{L}}_1={\mathit{F}}_2({\mathit{x}}_2;{\mathit{w}}_2,{\mathit{b}}_2).$$

(9)

In the second level, the pervasion matrix ${\mathbf{\Sigma }}_2$ is chosen as a diagonal matrix. Let ${\mathit{L}}_2\in {\mathbb{R}}^{d_2\times d_2}$ be the unique Cholesky decomposition of ${\mathbf{\Sigma }}_2$. We simply assume that ${\mathit{L}}_2$ is a constant diagonal matrix spanned by vector ${\mathit{\lambda }}_{top}\in {\mathbb{R}}^{d_2}$ with all positive components.

Figure 1 shows an overview of the hierarchical perceptual model. With this model, a Bayesian agent receives a series of sensory inputs or observations ${\mathit{u}}_s=\left\{\mathit{u}\left(t_1\right),\mathit{u}\left(t_2\right),\cdots ,\mathit{u}\left(t_K\right)\right\}$. K is the total number of trials. At time $t_k$, the sensory input $\mathit{u}\left(t_k\right)$ to the agent is determined by the state ${\mathit{x}}_0\left(t_k\right)$ of the bandit deterministically, i.e., with a delta distribution $\delta (·)$

$$P(\mathit{u}\left(t_k\right)\mid {\mathit{x}}_0\left(t_k\right))=\delta (\mathit{u}\left(t_k\right)={\mathit{x}}_0\left(t_k\right)).$$

(10)

In summary, the hierarchical perceptual model constitutes a generative model for sensory observations $\mathit{u}\left(t\right)$ based on hidden representations of the tendency (${\mathit{x}}_1$) and the volatility (${\mathit{x}}_2$) of the observations.

3. Perceptual Inference Approximated by Variational Approximation

The aforementioned hierarchical perceptual model is constructed based on general continuous Brownian motions. It remains to develop update rules to estimate the posterior distributions for the hidden representations ${\mathit{x}}_1$ and ${\mathit{x}}_2$. In order to derive a family of analytical and efficient updates, we discretize the continuous Brownian motions by applying Eulerian method. Sampling interval (SI) $ϵ\left(t_k\right)=t_k-t_{k-1}$ is defined by the time that elapses between the arrival of consecutive sensory inputs $\mathit{u}\left(t_{k-1}\right)$ and $\mathit{u}\left(t_k\right)$.

We use the variational Bayesian method [15,20,22,23] to reach an approximation to the posterior distributions of ${\mathit{x}}_1\left(t\right)$ and ${\mathit{x}}_2\left(t\right)$ given the sensory input $\mathit{u}\left(t\right)$ (i.e., observation). To this end, we maximize the negative free energy, which is the lower bound of log-model evidence, to yield a variational approximation posterior (cf. Appendices Appendix A and Appendix B)

$$q\left({\mathit{x}}_h\left(t_k\right)\right)=\frac{1}{{\mathcal{Z}}_h}exp\left(V_h\left({\mathit{x}}_h\left(t_k\right)\right)\right),h=1,2,$$

where ${\mathcal{Z}}_h$ is a normalization constant. $V_h({\mathit{x}}_h\left(t_k\right)$ is the variational energy given by

$$V_h({\mathit{x}}_h\left(t_k\right)=E_{q({\mathit{x}}_s\setminus \left\{{\mathit{x}}_h\right\}\left(t_k\right))}\left[lnp({\mathit{x}}_s\left(t_k\right),\mathit{u}\left(t_k\right)\mid {\mathit{\psi }}_s,ϵ\left(t_k\right))\right].$$

(11)

Here we introduced the notation ${\mathit{x}}_s=\left\{{\mathit{x}}_0,{\mathit{x}}_1,{\mathit{x}}_2\right\}$ to denote the set of all hidden states, ${\mathit{\psi }}_s=\left\{{\mathit{w}}_2,{\mathit{b}}_2,{\mathit{\lambda }}_{top},{\mathit{\zeta }}_1\right\}$ for the hyperparameters of the model, ${\mathit{x}}_s\setminus \left\{{\mathit{x}}_h\right\}$ for excluding ${\mathit{x}}_h$ from the set ${\mathit{x}}_s$, $E_{q\left(\mathit{x}\right)}\left(v\right)$ for the expectation of v under the distribution $q\left(\mathit{x}\right)$.

In order to complete the derivations, Gaussian quadratic form approximation is used as in [46]. In general, the variational energy $V_h\left({\mathit{x}}_h\left(t_k\right)\right)$ will deviate from a Gaussian quadratic form. We have to use a Gaussian quadratic form

$${\overline{V}}_h\left({\mathit{x}}_h\left(t_k\right)\right)=-\frac{1}{2}{({\mathit{x}}_h\left(t_k\right)-{\mathit{\mu }}_h\left(t_k\right))}^T{\mathit{P}}_h\left(t_k\right)({\mathit{x}}_h\left(t_k\right)-{\mathit{\mu }}_h\left(t_k\right))$$

as an efficient approximation of $V_h\left({\mathit{x}}_h\left(t_k\right)\right)$. ${\mathit{P}}_h\left(t_k\right)$ is given by the inverse of the Hessian matrix at the last state ${\mathit{\mu }}_h\left(t_{k-1}\right)$, ${\mathit{P}}_h\left(t_k\right)={\left({\mathit{C}}_h\left(t_k\right)\right)}^{-1}=-{\nabla }^{2}V\left({\mathit{\mu }}_h\left(t_{k-1}\right)\right)$, and then a local maximum point ${\mathit{\mu }}_h\left(t_k\right)$ is found as the mode of the posterior Gaussian distribution. This approximation is made by neglecting higher order terms of the logarithm of $q\left({\mathit{x}}_h\left(t_k\right)\right)$, and assuming Gaussian quadratic forms

$$\begin{array}{cc}\hfill & {\mathit{x}}_h\left(t_k\right)\mid \mathit{u}\left(t_k\right),{\mathit{\psi }}_s\sim \mathcal{N}({\mathit{\mu }}_h\left(t_k\right),{\mathit{C}}_h\left(t_k\right)).\hfill \\ \hfill & h=1,2\hfill \end{array}$$

(12)

Under this approximation, the inference of the posterior distributions of ${\mathit{x}}_h$ is reduced to the estimation of the mean ${\mathit{\mu }}_h\left(t_k\right)$ and the covariance matrix ${\mathit{C}}_h\left(t_k\right)$, or equivalently the precision matrix ${\mathit{P}}_h\left(t_k\right)\equiv {\left({\mathit{C}}_h\left(t_k\right)\right)}^{-1}$. Following [46], the update rules for the posterior distributions of ${\mathit{x}}_1$ and ${\mathit{x}}_2$ are derived.

At the bottom (zeroth) level of the hierarchical perceptual model, we can directly determine multivariate Bernoulli distribution $q\left({\mathit{x}}_0\left(t_k\right)\right)$ with expectation:

$${\mathit{\mu }}_0\left(t_k\right)=\mathit{u}\left(t_k\right).$$

(13)

At the first level, following Equation (11), $V_1\left({\mathit{x}}_1\right)$ can be calculated as

$$\begin{array}{cc}\hfill & V_1\left({\mathit{x}}_1\left(t_k\right)\right)=E_{q({\mathit{x}}_s\setminus \left\{{\mathit{x}}_1\right\}\left(t_k\right))}[lnp({\mathit{x}}_s\left(t_k\right),\mathit{u}\left(t_k\right)\mid {\mathit{\psi }}_s,ϵ\left(t_k\right))]\hfill \\ \hfill =& lnp(\mathit{u}\left(t_k\right)\mid {\mathit{x}}_0\left(t_k\right))+E_{q\left({\mathit{x}}_0\left(t_k\right)\right)}[lnp({\mathit{x}}_0\left(t_k\right)\mid {\mathit{x}}_1\left(t_k\right))]\hfill \\ & +E_{q({\mathit{x}}_1\left(t_k\right),{\mathit{x}}_2\left(t_k\right))}[lnp({\mathit{x}}_1\left(t_k\right)\mid {\mathit{x}}_2\left(t_k\right),{\mathit{W}}_2,{\mathit{b}}_2,ϵ\left(t_k\right))]\hfill \\ \hfill \approx & {\mathit{\mu }}_0^T\left(t_k\right)ln\mathit{s}({\mathit{x}}_1\left(t_k\right);{\zeta }_1)+{\left(\mathbf{1}-{\mathit{\mu }}_0\left(t_k\right)\right)}^{T}ln\left(\mathbf{1}-\mathit{s}({\mathit{x}}_1\left(t_k\right);{\mathit{\zeta }}_1)\right)\hfill \\ & -\frac{1}{2}{({\mathit{x}}_1\left(t_k\right)-{\mathit{\mu }}_1\left(t_{k-1}\right))}^T{\left(ϵ\left(t_k\right){\widehat{\mathbf{\Sigma }}}_1\left(t_k\right)+{\mathit{C}}_1\left(t_{k-1}\right)\right)}^{-1}({\mathit{x}}_1\left(t_k\right)-{\mathit{\mu }}_1\left(t_{k-1}\right)).\hfill \end{array}$$

where $\mathbf{1}$ is a $d_0$ dimensional column vector in which all elements are 1. Here we use the approximation

$${\left(ϵ\left(t_k\right){\mathbf{\Sigma }}_1\left(t_k\right)+{\mathit{C}}_1\left(t_{k-1}\right)\right)}^{-1}\approx {(ϵ\left(t_k\right){\widehat{\mathbf{\Sigma }}}_1\left(t_k\right)+{\mathit{C}}_1\left(t_{k-1}\right))}^{-1},$$

(14)

with ${\widehat{\mathbf{\Sigma }}}_1\left(t_k\right)$ computed from the second level

$$\begin{array}{cc}\hfill & {\widehat{\mathbf{\Sigma }}}_1\left(t_k\right)={\widehat{\mathit{L}}}_1\left(t_k\right){\widehat{\mathit{L}}}_1^T\left(t_k\right)\hfill \\ \hfill & {\widehat{\mathit{L}}}_1\left(t_k\right)={\mathit{F}}_2({\mathit{\mu }}_2\left(t_{k-1}\right);{\mathit{w}}_2,{\mathit{b}}_2).\hfill \end{array}$$

(15)

The variational energy $V_1\left({\mathit{x}}_1\left(t_k\right)\right)$ is not a standard Gaussian quadratic form, so we have to employ a Gaussian quadratic form ${\overline{V}}_1\left({\mathit{x}}_1\left(t_k\right)\right)$ to approximate it. To obtain this approximation form, we give the gradient and Hessian matrix of $V_1\left({\mathit{x}}_1\left(t_k\right)\right)$ as follows:

$$\begin{array}{cc}\hfill & \nabla V_1\left({\mathit{x}}_1\left(t_k\right)\right)={\mathit{\mu }}_0\left(t_k\right)-\mathit{s}({\mathit{x}}_1\left(t_k\right);{\mathit{\zeta }}_1)\hfill \\ \hfill & -\frac{1}{2}{\left(ϵ\left(t_k\right){\widehat{\mathbf{\Sigma }}}_1\left(t_k\right)+{\mathit{C}}_1\left(t_{k-1}\right)\right)}^{-1}({\mathit{x}}_1\left(t_k\right)-{\mathit{\mu }}_1\left(t_{k-1}\right))\hfill \end{array}$$

(16)

and

$$\begin{array}{cc}\hfill & {\nabla }^2{V}_1\left({\mathit{x}}_1\left(t_k\right)\right)=-\mathrm{diag}\left(\mathit{s}({\mathit{x}}_1\left(t_k\right);{\mathit{\zeta }}_1)\odot (1-\mathit{s}({\mathit{x}}_1\left(t_k\right);{\mathit{\zeta }}_1))\right)\hfill \\ \hfill & -\frac{1}{2}{\left(ϵ\left(t_k\right){\widehat{\mathbf{\Sigma }}}_1\left(t_k\right)+{\mathit{C}}_1\left(t_{k-1}\right)\right)}^{-1}\hfill \end{array}$$

(17)

where the operator ⊙ is the Hadamard product. The operation $\mathrm{diag}\left(\mathit{v}\right)$ is to transform a vector $\mathit{v}$ into a diagonal square matrix with the elements of $\mathit{v}$ on the principal diagonal.

Under the Gaussian quadratic form approximation which is based on a single-step Newton method, the tendency of ${\mathit{x}}_0\left(t_k\right)$ is captured by

$${\mathit{\mu }}_1\left(t_k\right)={\mathit{\mu }}_1\left(t_{k-1}\right)+\mathrm{diag}\left({\mathit{\zeta }}_1\right){\mathit{C}}_1\left(t_k\right){\mathit{PE}}_0\left(t_k\right)$$

(18)

where ${\mathit{PE}}_0\left(t_k\right)$ is the prediction error

$${\mathit{PE}}_0\left(t_k\right)={\mathit{\mu }}_0\left(t_k\right)-{\widehat{\mathit{\mu }}}_0\left(t_k\right),$$

(19)

where ${\widehat{\mathit{\mu }}}_0\left(t_k\right)\equiv {[{\widehat{\mu }}_0^{\left(1\right)}\left(t_k\right),{\widehat{\mu }}_0^{\left(2\right)}\left(t_k\right),\cdots ,{\widehat{\mu }}_0^{\left(d_0\right)}\left(t_k\right)]}^T$ is the prediction according to Equation (3)

$${\widehat{\mathit{\mu }}}_0\left(t_k\right)=\mathit{s}({\mathit{\mu }}_1\left(t_{k-1}\right),{\mathit{\zeta }}_1).$$

(20)

In Equation (19), the prediction error is scaled by the covariance matrix ${\mathit{C}}_1\left(t_k\right)$ of the approximate Gaussian distribution, which is converted from the precision matrix

$$\begin{array}{cc}\hfill {\mathit{C}}_1\left(t_k\right)\equiv & {\left({\mathit{P}}_1\left(t_k\right)\right)}^{-1}\hfill \\ \hfill {\mathit{P}}_1\left(t_k\right)=& {\widehat{\mathbf{\Pi }}}_1\left(t_k\right)+\mathrm{diag}{\left({\mathit{\zeta }}_1\right)}^2{\widehat{\mathit{C}}}_0\left(t_k\right).\hfill \end{array}$$

(21)

Here ${\widehat{\mathit{C}}}_0\left(t_k\right)=\mathrm{diag}\left({\widehat{\mathit{\sigma }}}_0\left(t_k\right)\right)$ is the diagonal square matrix containing the observed variance

$${\widehat{\mathit{\sigma }}}_0\left(t_k\right)=\left[\begin{array}{cc}& {\widehat{\mu }}_0^{\left(1\right)}\left(t_k\right)(1-{\widehat{\mu }}_0^{\left(1\right)}\left(t_k\right))\\ & {\widehat{\mu }}_0^{\left(2\right)}\left(t_k\right)(1-{\widehat{\mu }}_0^{\left(2\right)}\left(t_k\right))\\ & ⋮\\ & {\widehat{\mu }}_0^{\left(d_0\right)}\left(t_k\right)(1-{\widehat{\mu }}_0^{\left(d_0\right)}\left(t_k\right))\end{array}\right].$$

(22)

Prediction precision ${\widehat{\mathbf{\Pi }}}_1\left(t_k\right)$ is given by

$${\widehat{\mathbf{\Pi }}}_1\left(t_k\right)={(ϵ\left(t_k\right){\widehat{\mathbf{\Sigma }}}_1\left(t_k\right)+{\mathit{C}}_1\left(t_{k-1}\right))}^{-1}.$$

(23)

At the second level, the volatility, consisting of the uncertainties and pairwise correlations in natural parameters, is inferred by similar variational approximation method [46]. The mean is updated by

$$\begin{array}{cc}\hfill {\mathit{\mu }}_2\left(t_k\right)& ={\mathit{\mu }}_2\left(t_{k-1}\right)+ϵ\left(t_k\right){\mathit{C}}_2\left(t_k\right){\mathit{W}}_2^T\hfill \\ \hfill & ·{\widehat{\mathit{L}}}_{g1}\left(t_k\right)\left({\mathbf{\Omega }}_1\left(t_k\right)\otimes {\mathit{I}}_{d_1}\right)\mathbf{vec}\left({\mathbf{\Delta }}_1^T\left(t_k\right)\right).\hfill \end{array}$$

(24)

Here the function $\mathbf{vec}\left({\mathit{M}}_{m\times n}\right)$ is the vectorization of a matrix $\mathit{M}$, a linear operation, to obtain a column vector of length $m\times n$ by concatenating the columns of the matrix $\mathit{M}$ consecutively from column 1 to column n. The operator ⊗ is Kronecker product. ${\mathbf{\Delta }}_1\left(t_k\right)$ is given by

$$\begin{array}{cc}\hfill {\mathbf{\Delta }}_1\left(t_k\right)=& \left[{\mathit{C}}_1\left(t_k\right)+{\mathit{PE}}_1\left(t_k\right){\mathit{PE}}_1^T\left(t_k\right)\right]{\widehat{\mathbf{\Pi }}}_1\left(t_k\right)-{\mathit{I}}_{d_1}.\hfill \end{array}$$

(25)

The constant matrix ${\mathit{I}}_d$ is a d-by-d unit square matrix. ${\mathit{PE}}_1\left(t_k\right)$ is the prediction error on the hidden state ${\mathit{x}}_1$

$${\mathit{PE}}_1\left(t_k\right)={\mathit{\mu }}_1\left(t_k\right)-{\mathit{\mu }}_1\left(t_{k-1}\right).$$

(26)

${\widehat{\mathit{L}}}_{g1}\left(t_k\right)$ is given by

$$\begin{array}{cc}\hfill & {\widehat{\mathit{L}}}_{g1}\left(t_k\right)\hfill \\ \hfill =& \left[\begin{array}{cc}& exp\left({({\mathit{W}}_2^{\left(1\right)})}^T{\mathit{\mu }}_2\left(t_{k-1}\right)+b_2^{\left(1\right)}\right){\mathit{e}}_2^T\left(1\right)\\ & 2cosh\left({({\mathit{W}}_2^{\left(2\right)})}^T{\mathit{\mu }}_2\left(t_{k-1}\right)+b_2^{\left(2\right)}\right){\mathit{e}}_2^T\left(2\right)\\ & exp\left({({\mathit{W}}_2^{\left(3\right)})}^T{\mathit{\mu }}_2\left(t_{k-1}\right)+b_2^{\left(3\right)}\right){\mathit{e}}_2^T\left(3\right)\\ & 2cosh\left({({\mathit{W}}_2^{\left(4\right)})}^T{\mathit{\mu }}_2\left(t_{k-1}\right)+b_2^{\left(4\right)}\right){\mathit{e}}_2^T\left(4\right)\\ & ⋮\\ & exp\left({({\mathit{W}}_2^{\left(d_2\right)})}^T{\mathit{\mu }}_2\left(t_{k-1}\right)+b_2^{\left(d_2\right)}\right){\mathit{e}}_2^T\left(d_2\right)\end{array}\right]\hfill \end{array},$$

(27)

where the constant vector ${\mathit{e}}_2\left(d_2\right)$ is a $d_1^2$-dimension column vector. The j-th component in ${\mathit{e}}_2^T\left(d_2\right)$ is 1 if $j=i$ or 0 if $j\ne i$. The column vector ${\mathit{W}}_2^{\left(i\right)}$ is the i-th row in the coefficient matrix ${\mathit{W}}_2$. ${\mathbf{\Omega }}_1\left(t_k\right)$ is

$${\mathbf{\Omega }}_1\left(t_k\right)={\widehat{\mathit{L}}}_1^T\left(t_k\right){\widehat{\mathbf{\Pi }}}_1\left(t_k\right).$$

(28)

The precision matrix is updated by

$$\begin{array}{cc}\hfill & {\mathit{P}}_2\left(t_k\right)={\widehat{\mathbf{\Pi }}}_2\left(t_k\right)+{\mathit{W}}_2^T{\widehat{\mathit{L}}}_{g1}\left(t_k\right)\hfill \\ \hfill & ϵ{\left(t_k\right)}^2{\mathit{K}}_{d_1{d}_1}\left[{\mathbf{\Omega }}_1^T\left(t_k\right)\otimes \left[{\mathbf{\Omega }}_1\left(t_k\right){\mathbf{\Delta }}_1\left(t_k\right)\right]\right.\hfill \\ \hfill & +\left[{\mathbf{\Delta }}_1^T\left(t_k\right){\mathbf{\Omega }}_1^T\left(t_k\right)\right]\otimes {\mathbf{\Omega }}_1\left(t_k\right)\hfill \\ \hfill & \left.+{\mathbf{\Omega }}_1^T\left(t_k\right)\otimes {\mathbf{\Omega }}_1\left(t_k\right)\right]\hfill \\ \hfill +& ϵ{\left(t_k\right)}^2\left[\left[{\mathit{L}}_1^T\left(t_k\right){\mathbf{\Delta }}_1^T\left(t_k\right){\mathbf{\Omega }}_1^T\left(t_k\right)\right]\otimes {\widehat{\mathbf{\Pi }}}_1\left(t_k\right)\right.\hfill \\ \hfill & +\left[{\mathit{L}}_1^T\left(t_k\right){\mathbf{\Omega }}_1^T\left(t_k\right)\right]\otimes \left[{\widehat{\mathbf{\Pi }}}_1\left(t_k\right){\mathbf{\Delta }}_1\left(t_k\right)\right]\hfill \\ \hfill & \left.+\left[{\mathit{L}}_1^T\left(t_k\right){\mathbf{\Omega }}_1^T\left(t_k\right)\right]\otimes {\widehat{\mathbf{\Pi }}}_1\left(t_k\right)\right]\hfill \\ \hfill -& ϵ\left(t_k\right)\left[{\mathit{I}}_{d_1}\otimes \left[{\widehat{\mathbf{\Pi }}}_1\left(t_k\right){\mathbf{\Delta }}_1\left(t_k\right)\right]\right]\hfill \\ \hfill & {\widehat{\mathit{L}}}_{g1}^T\left(t_k\right){\mathit{W}}_2\hfill \\ \hfill -& {\mathit{W}}_2^T\mathbf{diag}\left(\mathbf{lvec}\left({\mathit{\delta }}_1\left(t_k\right)\right)\right){\mathit{W}}_2\hfill \end{array}$$

(29)

where the function $\mathbf{lvec}\left(\mathit{L}\right)$ is to transform a lower triangular matrix $\mathit{L}$ into a column vector obtained by column stacking except all constant zero elements in the upper triangle part of the matrix. The prediction precision matrix ${\widehat{\mathbf{\Pi }}}_2$ at the second level is given by

$${\widehat{\mathbf{\Pi }}}_2\left(t_k\right)={(ϵ\left(t_k\right){\mathbf{\Sigma }}_2+{\mathit{C}}_2\left(t_{k-1}\right))}^{-1}.$$

(30)

The notation ${\mathit{K}}_{mn}$ denotes a $mn$-by-$mn$ commutation matrix. ${\mathit{\delta }}_1\left(t_k\right)$ is defined as

$${\mathit{\delta }}_1\left(t_k\right)=ϵ\left(t_k\right)\left[{\mathbf{\Delta }}_1^T\left(t_k\right){\mathbf{\Omega }}_1^T\left(t_k\right)\right]\odot {\widehat{\mathit{L}}}_1\left(t_k\right).$$

(31)

4. Decision Making in Volatile Multi-Armed Bandits

To illustrate decision making on the basis of perceptual inference in volatile environments, we introduce, as a toy example, a two-armed bandit problem, which is a complex variant of a one-armed bandit gambling task in [30,47]. In this task, a cautious gambler is asked to bet on the outcomes of a two-armed bandit, and to maximize its overall score (Figure 2). We use upper-case letters A and B to denote the two arms of the bandit, and the notations $x_0^{\left(1\right)}$ and $x_0^{\left(2\right)}$ for the states of arm A and B respectively. On each trail, the states of the two arms, i.e., the binary vector ${\mathit{x}}_0={[x_0^{\left(1\right)},x_0^{\left(2\right)}]}^T$, will be revealed to the gambler at the same time after the gambler makes a choice. There are two options available for the gambler to choose from. The first option “Same” represents the congruent states of the two arms, i.e., ${[0,0]}^T$ or ${[1,1]}^T$. The second option “Different” represents incongruent states of the two arms, i.e., ${[1,0]}^T$ or ${[0,1]}^T$. Once the gambler makes a decision (to choose “Same” or “Different”), the two arms would randomly generate their states by employing two univariate Bernoulli distributions (Equation (1)). To model a volatile environment, the state distributions of the arms are time-varying (Figure 3).

The gambler’s response a is encoded as:

$$a=\left\{\begin{array}{cc}0,\hfill & \mathrm{for}\hspace{0.17em}\mathrm{choice}\hspace{0.17em}‘\mathrm{Different}’\hfill \\ 1,\hfill & \mathrm{for}\hspace{0.17em}\mathrm{choice}\hspace{0.17em}‘\mathrm{Same}’\hfill \end{array}\right..$$

(32)

The gambler is rewarded if its choice matches the outcome of the bandit. To include volatility also in rewards, the magnitude of reward is varied from trial to trial. The reward is sampled from a reward set ${\mathbb{S}}_r=\left\{1,2,3,\cdots ,N_r\right\}$, with equal probability of each reward being chosen $P\left(k\right)=1/N_r,\forall k\in {\mathbb{S}}_r$.

The gambler starts the experiment with zero score. On each trial, once the chosen option turns out to be correct, the corresponding reward associated to the choice will be added to its overall score.

To maximize reward, a response model has to be defined. To this end, we first denote the rewards obtained for the correct choice of “Different” and “Same” as $r_0$ and $r_1$, respectively, can construct a reward table for each trial (

Table 1. Reward table.

		a
		0	1
${\mathit{x}}_0$	(0,0)	0	$r_1$
	(1,1)	0	$r_1$
	(1,0)	$r_0$	0
	(0,1)	$r_0$	0

).

Then we write a reward (utility) function $r({\mathit{x}}_0,a)$ on a trial basis according to the reward table

$$\begin{array}{cc}\hfill r({\mathit{x}}_0,a)& =(1-{(x_0^{\left(1\right)}-x_0^{\left(2\right)})}^2){[a-{(x_0^{\left(1\right)}-x_0^{\left(2\right)})}^2]}^2{r}_1\hfill \\ \hfill & +{(x_0^{\left(1\right)}-x_0^{\left(2\right)})}^2{[a-{(x_0^{\left(1\right)}-x_0^{\left(2\right)})}^2]}^2{r}_0.\hfill \end{array}$$

(33)

Given the predicted state ${\widehat{\mathit{\mu }}}_0$ (Equation (21)), the expected reward of decision a under the corresponding predicted distribution $q({\mathit{x}}_0,{\widehat{\mathit{\mu }}}_0)$ is given by the value function

$$\begin{array}{cc}\hfill & Q(a,{\widehat{\mathit{\mu }}}_0)=\sum _{{\mathit{x}}_0}r({\mathit{x}}_0,a)\mathrm{Bern}({\mathit{x}}_0;{\widehat{\mathit{\mu }}}_0)\hfill \\ \hfill =& \sum _{{\mathit{x}}_0}r({\mathit{x}}_0,a)\mathrm{Bern}(x_0^{\left(1\right)};{\widehat{\mu }}_0^{\left(1\right)})\mathrm{Bern}(x_0^{\left(2\right)};{\widehat{\mu }}_0^{\left(2\right)})\hfill \\ \hfill =& a^2{r}_1[(1-{\widehat{\mu }}_0^{\left(1\right)})(1-{\widehat{\mu }}_0^{\left(2\right)})+{\widehat{\mu }}_0^{\left(1\right)}{\widehat{\mu }}_0^{\left(2\right)}]\hfill \\ \hfill & +{(a-1)}^2{r}_0[(1-{\widehat{\mu }}_0^{\left(1\right)}){\widehat{\mu }}_0^{\left(2\right)}+{\widehat{\mu }}_0^{\left(1\right)}(1-{\widehat{\mu }}_0^{\left(2\right)})]\hfill \\ \hfill =& \left\{\begin{array}{cc}{r}_1[(1-{\widehat{\mu }}_0^{\left(1\right)})(1-{\widehat{\mu }}_0^{\left(2\right)})+{\widehat{\mu }}_0^{\left(1\right)}{\widehat{\mu }}_0^{\left(2\right)}],\hfill & a=1\hfill \\ r_0[(1-{\widehat{\mu }}_0^{\left(1\right)}){\widehat{\mu }}_0^{\left(2\right)}+{\widehat{\mu }}_0^{\left(1\right)}(1-{\widehat{\mu }}_0^{\left(2\right)})],\hfill & a=0\hfill \end{array}\right.\hfill \end{array}$$

(34)

The agent makes decisions according to a Boltzmann distribution constructed from the value function. The probability of choosing action a is defined by

$$\begin{array}{c}\hfill P_a=\frac{exp\left(Q(a,{\widehat{\mathit{\mu }}}_0)\right)}{{\sum }_{b}exp\left(Q(b,{\widehat{\mathit{\mu }}}_0)\right)}.\end{array}$$

(35)

For the binary decision-making task considered here, the probability of choosing action $a=1$ is reduced to a sigmoid function

$$\begin{array}{cc}\hfill P_1=& \frac{1}{1+exp(-(Q(1,{\widehat{\mathit{\mu }}}_0)-Q(0,{\widehat{\mathit{\mu }}}_0)))}\hfill \\ \hfill =& s(Q(1,{\widehat{\mathit{\mu }}}_0)-Q(0,{\widehat{\mathit{\mu }}}_0),1),\hfill \end{array}$$

(36)

where $s(·,·)$ the sigmoid function defined in Equation (4).

In fact, a biological agent maximizes long-term rewards, instead of immediate rewards, using decision noise as a mechanician to tradeoff exploration and exploitation. We introduce a probability weighting function [47,48] with a noise parameter ${\zeta }_a>0$ to include decision noise. The probability of choosing action $a=1$ is

$$\begin{array}{c}\hfill P(a=1\mid {\widehat{\mathit{\mu }}}_0,{\zeta }_a)=\frac{P_1^{{\zeta }_a}}{P_1^{{\zeta }_a}+{(1-P_1)}^{{\zeta }_a}}.\end{array}$$

(37)

Up to now, we have defined a response model (Equations (33)–(37)) based on Bayesian decision theory to maximize expected rewards. The response model is a function of the decision evidence $(Q(1,{\widehat{\mathit{\mu }}}_0)-Q(0,{\widehat{\mathit{\mu }}}_0))$, i.e., the difference between expected rewards for the two options (“Different”, “Same”). If the decision evidence is positive, the probability of choosing “Same” exceeds $0.5$, and the optimal action is to choose “Same” or $a=1$. If the decision evidence is a negative number, the probability of choosing “Different” exceeds $0.5$ and the optimal action is the option “Different” or $a=0$.

5. Simulation Results

The combination of the perceptual model (Equations (5)–(10)) and the response model (Equations (33)–(37)) constitute a Bayesian model (denoted by ${\mathcal{M}}_1$) for decision making in volatile multi-armed bandits. To assess the model’s ability to adapt to volatility, we simulated a gambler with the proposed Bayesian decision model to solve the two-armed bandit task (Figure 2). In the simulation, trials are organized into seventeen blocks, each of which contains 15 trials (Figure 3). The state expectations of the bandit change across blocks, resulting in volatility in sensory inputs. The reward set is specified as ${\mathbb{S}}_r=\{1,2,3,4\}$.

For an ideal observer, it has the access to the actual state $\mathit{u}\left(t\right)={[u^{\left(1\right)}\left(t\right),u^{\left(2\right)}\left(t\right)]}^T$ generated by the bandit at each time t (Figure 3). Given this ideal information, the ideal observer could make the ideal actions $a_{ideal}\left(t\right)$

$$a_{ideal}\left(t\right)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\hspace{0.17em}{u}^{\left(1\right)}\left(t\right)\ne u^{\left(2\right)}\left(t\right)\hfill \\ 1,\hfill & \mathrm{if}\hspace{0.17em}{u}^{\left(1\right)}\left(t\right)=u^{\left(2\right)}\left(t\right)\hfill \end{array}\right..$$

(38)

Based on this series of ideal actions, the cumulative reward obtained by the ideal observer could be computed.

To measure the performance of decision making behavior in the above gambling task, we define a probabilistically optimal reference for comparison. For this purpose, we consider an informed agent, who is given the expectation of the states of the volatile bandit ${[P(x_0^{\left(1\right)}\left(t\right)=1),P(x_0^{\left(1\right)}\left(t\right)=1)]}^T$. The informed agent needs not learn the states of the bandit, and it uses the same action selection mechanism (Equations (34)–(37)) of the response model ${\pi }_r$ to obtain the probabilistically optimal expectation of response action $a\left(t\right)$, denoted by $P^{*}(a\left(t\right)=1)$. For a decision-making agent, only if the agent fully understands the volatile environments, the expectation of its action $a\left(t\right)$ can completely coincide with the probabilistically optimal expectation of response action $P^{*}(a\left(t\right)=1)$. Overestimating or underestimating the environmental states will lead to the expectation of the agent’s action $a\left(t\right)$ to deviate from the optimal behavior. Therefore, $P^{*}(a\left(t\right)=1)$ constitutes the optimal decision making behavior of a learning agent could reach. The deviation of a learning agent with sensory inputs ${\mathit{u}}_s$, actions $a_s$ and rewards ${\mathit{r}}_s$ from the informed agent in decision making behavior is measured by regret $R\left(P(a\left(t_k\right)=1)\right|{\mathit{u}}_s,a_s,{\mathit{r}}_s)$, defined by

$$\begin{array}{c}\hfill R\left(P(a\left(t_k\right)=1)\right|{\mathit{u}}_s,a_s,{\mathit{r}}_s)=\sum _{k=1}^K\left|P(a\left(t_k\right)=1)-P^{*}(a\left(t_k\right)=1)\right|,\end{array}$$

(39)

where $P(a\left(t_k\right)=1)$ is generated by the learning agent.

5.1. Dynamics of Bayesian Decision Making

We employed a Bayesian agent ${\mathcal{M}}_1$, which is endowed with the proposed hierarchical perceptual model and binary response model (Figure 4), to perform the above gambling task (Figure 2). All free parameters of our Bayesian agent ${\mathcal{M}}_1$ is defined in Appendix C, and forms a random variable vector denoted by ${\mathit{\xi }}_1$. Their initial sufficient statistics of all parameters are listed in

Table 2. Parameters of our hierarchical Bayesian model. Parameters labeled by ‘Free’ are optimized by the inversion of the model. Fixed parameters are constant and not optimized. The notation $\mathbf{1}$ is a constant column vector with all components being 1. The notation $\mathbf{0}$ is a zero vector. The matrix ${\mathit{O}}_d$ is a d by d constant matrix in which all elements are 0. The notation $\mathrm{logit}(·)$ denotes a logistic function $\mathrm{logit}\left(x\right)=ln(\frac{x}{1-x}).$ Given all initial priors, we search for the optimal priors on all optimized parameters ${\mathit{\mu }}_{\xi }$ according to the free energy principle (Equations (A19) and (A21)).

Name	Description	Initial Value	Fixed or Free
Parameters of our Bayesian perceptual model
$d_0=d_{\mathit{u}}$	Dimension of sensory input $\mathit{u}$	2	constant
$d_1$	Dimension of ${\mathit{x}}_1$	2	constant
$d_2$	Dimension of ${\mathit{x}}_2$	3	constant
$ϵ\left(t_k\right)$	Sampling interval $ϵ\left(t_k\right)$	1	constant
${\alpha }_{{\mathit{\lambda }}_{top}}$	Upper bound on ${\mathit{\lambda }}_{top}$	$\sqrt{0.1}·\mathbf{1}$	constant
${\mathit{\lambda }}_{top}$	Volatility of ${\mathit{x}}_2$		Free
${\mathit{\mu }}_{{\mathit{\lambda }}_{top}^G}$	Mean of ${\mathit{\lambda }}_{top}^G$	$\mathrm{logit}\left(0.1\right)·\mathbf{1}$
${\mathit{C}}_{{\mathit{\lambda }}_{top}^G}$	Covariance of ${\mathit{\lambda }}_{top}^G$	$1\times {10}^{-2}{\mathit{I}}_{d_2}$
${\alpha }_{{\mathit{w}}_2}$	Upper bound on ${\mathit{w}}_2$	$1·\mathbf{1}$	constant
${\mathit{w}}_2$	Coupling strength		Free
${\mathit{\mu }}_{{\mathit{w}}_2^G}$	Mean of ${\mathit{w}}_2^G$	$\mathrm{logit}\left(0.25\right)·\mathbf{1}$
${\mathit{C}}_{{\mathit{w}}_2^G}$	Covariance of ${\mathit{w}}_2^G$	$1\times {10}^{-2}·{\mathit{I}}_{d_2}$
${\mathit{b}}_2$	Coupling bias	$\mathbf{0}$	Fixed
${\mathit{\mu }}_{{\mathit{b}}_2}$	Mean of ${\mathit{b}}_2$	$\mathbf{0}$
${\mathit{C}}_{{\mathit{b}}_2}$	Covariance of ${\mathit{b}}_2$	${\mathit{O}}_3$
${\mathit{\mu }}_2\left(t_0\right)$	Prior mean of ${\mathit{x}}_2$		Free
${\mathit{\mu }}_{{\mathit{\mu }}_2\left(t_0\right)}$	Mean of ${\mathit{\mu }}_2\left(t_0\right)$	$ln\left(0.16\right)·\mathbf{1}$
${\mathit{C}}_{{\mathit{\mu }}_2\left(t_0\right)}$	Covariance of ${\mathit{\mu }}_2\left(t_0\right)$	$1\times {10}^{-2}·{\mathit{I}}_3$
${\mathit{C}}_2\left(t_0\right)$	Prior covariance of ${\mathit{x}}_2$		Free
${\mathit{\mu }}_{{\mathit{c}}_2^G}$	Mean of ${\mathit{c}}_2^G$	$ln\left(1\right)$
${\mathit{C}}_{{\mathit{c}}_2^G}$	Covariance of ${\mathit{c}}_2^G$	${\mathit{I}}_{d_2}$
${\mathit{\mu }}_1\left(t_0\right)$	Prior mean of ${\mathit{x}}_1$		Free
${\mathit{\mu }}_{{\mathit{\mu }}_1\left(t_0\right)}$	Mean of ${\mathit{\mu }}_1\left(t_0\right)$	$\mathbf{0}$
${\mathit{C}}_{{\mathit{\mu }}_1\left(t_0\right)}$	Covariance of ${\mathit{\sigma }}_1^G$	${\mathit{O}}_{d_2}$
${\mathit{C}}_1\left(t_0\right)$	Prior covariance of ${\mathit{x}}_1$		Free
${\mathit{\mu }}_{{\mathit{\sigma }}_1^G}$	Mean of ${\mathit{\sigma }}_1^G$	$ln\left(0.16\right)·\mathbf{1}$
${\mathit{C}}_{{\mathit{\sigma }}_1^G}$	Covariance of ${\mathit{\sigma }}_1^G$	$0.1{\mathit{I}}_{d_1}$
${\mathit{\zeta }}_1$	Coefficient		Fixed
${\mathit{\mu }}_{{\mathit{\zeta }}_1^G}$	Mean of ${\mathit{\zeta }}_1^G$	$\mathbf{0}$
${\mathit{C}}_{{\mathit{\zeta }}_1^G}$	Covariance of ${\mathit{\zeta }}_1^G$	${\mathit{O}}_2$
Parameters of our response model
$d_a$	Dimension of a	1	Fixed
${\mathit{\zeta }}_a$	Coefficient		Fixed
${\mu }_{{\zeta }_a^G}$	Mean of ${\zeta }_a^G$	$ln\left(2\right)$
$C_{{\zeta }_a^G}$	Covariance of ${\zeta }_a^G$	0

. In details, the optimization of the free parameters was carried out in three steps as follows before the model was used for the gambling task.

(1)

Generating synthetic data. According to the expected states of the arms (Figure 3), we randomly generated a sequence of multivariate binary inputs

$${\mathit{u}}_s=\left\{\mathit{u}\left(t_1\right),\mathit{u}\left(t_2\right),\mathit{u}\left(t_3\right),\cdots ,\mathit{u}\left(t_K\right)\right\},(K=255).$$

Then the series of ideal actions $a_s=\left\{a_{ideal}\left(t_1\right),a_{ideal}\left(t_2\right),\cdots ,a_{ideal}\left(t_K\right)\right\}$ is computed by Equation (38). The random reward sequence ${\mathit{r}}_s=\left\{\mathit{r}\left(t_1\right),\mathit{r}\left(t_2\right),\mathit{r}\left(t_3\right),\cdots ,\mathit{r}\left(t_K\right)\right\}$ is generated from uniform distribution $\mathcal{U}(1,4)$ based on the reward set ${\mathbb{S}}_r=\left\{1,2,3,4\right\}$.

(2)

Initializing sufficient statistics of all random parameters. To allow our model to work well for sensory inputs, we choose particular initial sufficient statistics of the random parameter vector ${\mathit{\xi }}_1$, and determined the prior distribution of ${\mathit{\xi }}_1$. The configuration for the parameters of the Bayesian agent (Figure 4) is shown in Table 2.

(3)

Maximizing negative free energy. To obtain the optimal prior parameters $\left({\mathit{\mu }}_{{\mathit{\xi }}_1}^{*},{\mathit{C}}_{{\mathit{\xi }}_1}^{*}\right)$ of the parameter ${\mathit{\xi }}_1$, we maximize negative free energy (Equations (A19)–(A21)) by using the quasi-Newton Broyden–Fletcher–Goldfarb–Shanno method based on a line search framework [49].

In order to reveal the dynamic interaction between the natural parameters of the two-armed bandit, we show one example gambling process performed by the Bayesian agent characterized by the optimal parameters. The Bayesian agent tracked online the tendency ${\mathit{\mu }}_1$ of the natural parameters ${\mathit{x}}_1$ associated to the bandit (Figure 5), so that it is able to make decisions based on the estimated decision evidence. The evolution of ${\mathit{\mu }}_1$ follows well of the trend of expected states of the bandit (Figure 3), generating good prediction of the states (Figure 6b,c).

After the observation of the tentative rewards (Figure 6a) during the cue phase of a trial, the Bayesian agent makes a choice in the decision phase according to the perceptual model and the response model. In the first block of the simulation (trial 1 to 15, Figure 3), the two arms have the same expected states of maximal uncertainty, i.e., $P(x_0^{\left(1\right)}=1)=P(x_0^{\left(2\right)}=1)=0.5$, and the binary state patterns of the two-armed bandit are equal probable. Both of the belief states ${\mu }_0^{\left(1\right)},{\mu }_0^{\left(2\right)}$ of two arms fluctuates around 0.5 (Figure 6). During this block, the prediction correlation ${\widehat{\rho }}_1$ fluctuates and decreases slightly towards zero, reflecting the fact that the states of the two arms are uncorrelated. From the second block to the tenth block (trial 16 to 150, Figure 3), the expected states of the two arms are incongruent. Therefore, the changes in the prediction tendency ${\widehat{\mu }}_1^{\left(1\right)}$ (Figure 5a, as well as in the predicted mean ${\widehat{\mu }}_0^{\left(1\right)}$ of arm A in Figure 6b) are on average in opposite directions as the changes in the prediction tendency ${\widehat{\mu }}_1^{\left(2\right)}$ (Figure 5b, as well as in the predicted mean ${\widehat{\mu }}_0^{\left(2\right)}$ of arm B in Figure 6c). Meanwhile, the prediction correlation ${\widehat{\rho }}_1$ continues to decrease during this stage (Figure 6d), manifesting the incongruency of the two arms. From the eleventh block to the seventeenth block (trial 151 to 255, Figure 3), the changes in ${\widehat{\mu }}_1^{\left(1\right)}$ and ${\widehat{\mu }}_1^{\left(2\right)}$ share the same trend (Figure 5), so do the changes in ${\widehat{\mu }}_0^{\left(1\right)}$ and ${\widehat{\mu }}_0^{\left(2\right)}$ (Figure 6b,c), due to the fact that the two arms have the same expected states. Consequently, the prediction correlation ${\widehat{\rho }}_1$ continues to increase during this stage (Figure 6d).

The log-volatility in the natural parameters (${\mu }_2^{\left(1\right)}$ and ${\mu }_2^{\left(3\right)}$, i.e., internal representation of the expected states) of the two arms has notable changes from the third block to the fourteenth block (trial 31 to 210, Figure 7a,c). The changes are more evident from the sixth block to the fourteenth block, during which volatility is more vigorous. From the second to the tenth blocks (trial 31 to 150), the expected states of the two arms are not equal. Instead, they become incongruent (Figure 3). During this period, the log-volatility state ${\mu }_2^{\left(2\right)}$, corresponding to the prediction correlation ${\widehat{\rho }}_1$, decreased and kept a descending trend (Figure 7b). This is consistent with the fact that the two arms are incongruent at the time. As a contrast, from the eleventh block to the seventeenth block, the expected states of the two arms are equal (trial 151 to 255, Figure 3), therefore, the Bayesian learner discovered an increasing log-volatility state ${\mu }_2^{\left(2\right)}$ during this stage (Figure 7b).

In Figure 8, our Bayesian agent ${\mathcal{M}}_1$ is compared with the informed agent. In our Bayesian decision model, the evidence for decision-making is quantified by the probability $P\left(a\right(t)=1)$ (red solid line in Figure 8a). It is close to the probabilistically optimal expectation of response action $P^{*}(a\left(t\right)=1)$ given by the informed agent (blue dashed line in Figure 8a). The action selection behavior of our Bayesian agent was similar to the optimal probability decision pattern. In this experiment, the regret $R\left(P(a\left(t_k\right)=1)\right|{\mathit{u}}_s,a_s,{\mathit{r}}_s)$ is worked out by substituting $P\left(a\right(t)=1)$ generated by our Bayesian agent into Equation (39), and is 27.3588.

Since the Bayesian agent made decisions based on the estimated decision evidence, it may be distracted by high rewards associated to the wrong action. In the third block (trial 31 to 45), the rewards of the option “Same” were higher than the option “Different”, the Bayesian agent were biased towards choosing “Same”, reflexing the fact that less likely but highly rewarded actions are worth to be tried (Figure 6e). This phenomenon was also evident in the beginning of the eleventh block (trial 151 to 155), where high rewards were more often assigned to the option “Different”. The Bayesian agent from time to time reduced the probability of choosing the option “Same”, leading to select “Different” more often (Figure 6e). The cumulative reward obtained by the Bayesian agent maintains a linear increasing trend irrespective of the volatility (red line in Figure 6f and Figure 8b), keeping close to the reward gained by the informed agent (blue line in Figure 8b).

5.2. Bayesian Model Selection

In order to evaluate the proposed hierarchical Bayesian model for inferring and decision making, we adopt the Bayesian model selection methodology [50]. It is a general principle to favor a model that achieves balanced tradeoff between complexity and flexibility. The proposed hierarchical Bayesian model has the sophisticated complexity to capture volatility in a multiscale fashion. We compare it with a well-known baseline model in psychology and Reinforcement Learning (RL), namely the Rescorla–Wagner (RW) model [26,51]. As a special case of the Temporal-Difference Learning method, the Rescorla–Wagner model updates value estimations based on prediction errors [26].

To perform fair comparisons, we construct a variant of the RW model using the same response model as the proposed hierarchical Bayesian model (cf. Appendix G). The agent with the RW model and the above response model is denoted by ${\mathcal{M}}_2$. Under the same variational Bayesian learning scheme, we search the optimal parameters for ${\mathcal{M}}_2$ on each sequence of sensory inputs (Appendix D).

We conducted a Bayesian model selection experiment to compare the proposed Bayesian agent ${\mathcal{M}}_1$ based on a variant of GHBF with the agent ${\mathcal{M}}_2$ based on the RW model. The detailed simulation was performed as the following steps.

(1)

Generating synthetic dataset $\mathcal{D}$. According to Figure 3, we randomly generated 100 sequences of multivariate binary inputs ${\mathit{u}}_s=\left\{\mathit{u}\left(t_1\right),\mathit{u}\left(t_2\right),\mathit{u}\left(t_3\right),\cdots ,\mathit{u}\left(t_K\right)\right\}$ ($K=255$). Then the series of ideal actions $a_s=\left\{a_{ideal}\left(t_1\right),a_{ideal}\left(t_2\right),\cdots ,a_{ideal}\left(t_K\right)\right\}$ are computed according to Equation (38). Random reward sequences ${\mathit{r}}_s=\left\{\mathit{r}\left(t_1\right),\mathit{r}\left(t_2\right),\mathit{r}\left(t_3\right),\cdots ,\mathit{r}\left(t_K\right)\right\}$ are generated from uniform distribution $\mathcal{U}(1,4)$ based on the reward set ${\mathbb{S}}_r$. Here we used the notation $\mathcal{D}$ to denote the set of sensory and action sequences

$$\mathcal{D}=\left\{{\mathit{u}}_s,{\mathit{r}}_s,a_s|{\mathit{u}}_s\hspace{0.17em}\mathrm{and}\hspace{0.17em}{\mathit{r}}_s\hspace{0.17em}\mathrm{are}\hspace{0.17em}\mathrm{repeatedly}\hspace{0.17em}\mathrm{generated}\right\}.$$

(2)

Initializing sufficient statistics of all random parameters in our Bayesian agent ${\mathcal{M}}_1$. We choose particular initial sufficient statistics of a parameter vector ${\mathit{\xi }}_1$ to allow the Bayesian agent ${\mathcal{M}}_1$ to work well on all sequences of sensory inputs. Then we determined the prior distribution of ${\mathit{\xi }}_1$. All configurations for parameters of the agent based on GHBF (Figure 4) are shown in Table 2.

(3)

Initializing sufficient statistics of all random parameters in the RW-agent ${\mathcal{M}}_2$. We determined a particular initial value of a parameter vector ${\mathit{\xi }}_2$ (

Table A2. Parameters of the Rescorla–Wagner model. Parameters labeled by ‘Free’ are optimized by the inversion of the model. Fixed parameters are constant and not optimized. The notation $\mathbf{0}$ is a zero vector. Given all initial priors, we search the optimal priors on the free parameters ${\mathit{\mu }}_{\xi }$ according to the free energy principle (Equations (A19) and (A21)).

Name	Description	Initial Value	Fixed or Free
Parameters of Rescorla–Wagner model
$d_{\mathit{u}}$	Dimension of $\mathit{u}$	2	constant
$d_0$	Dimension of ${\mathit{\mu }}_0$	2	constant
${\mathit{\mu }}_0\left(t_0\right)$	Prior initial state		Fixed
${\mathit{\mu }}_{{\mathit{\mu }}_0^G\left(t_0\right)}$	Mean of ${\mathit{\mu }}_0^G\left(t_0\right)$	$\mathbf{0}$
${\mathit{C}}_{{\mathit{\mu }}_0^G\left(t_0\right)}$	Covariance of ${\mathit{\mu }}_0^G\left(t_0\right)$	${\mathit{I}}_{d_0}$
$\alpha$	Learning rate $\alpha$		Free
${\mu }_{{\alpha }^G}$	Mean of ${\alpha }^G$	0
$C_{{\alpha }^G}$	Covariance of ${\alpha }^G$	$0.01$

) for the agent ${\mathcal{M}}_2$. All configurations for parameters of the agent based on Rescorla–Wagner model were shown in Table A2. The response model of the RW model uses the same parameter configuration as in the Bayesian agent in step 2.

(4)

Maximizing negative free energy. On each sequence of sensory inputs, we performed an optimization method to obtain the optimal prior parameters $\left({\mathit{\mu }}_{{\mathit{\xi }}_1}^{*},{\mathit{C}}_{{\mathit{\xi }}_1}^{*}\right)$ of the parameter ${\mathit{\xi }}_1$ for the agent ${\mathcal{M}}_1$ and the optimal prior parameters $\left({\mathit{\mu }}_{{\mathit{\xi }}_2}^{*},{\mathit{C}}_{{\mathit{\xi }}_2}^{*}\right)$ of the parameter ${\mathit{\xi }}_2$ for the agent ${\mathcal{M}}_2$ according to Equation (A21) respectively. In this paper, we implemented the quasi-Newton Broyden–Fletcher–Goldfarb–Shanno method based on a line search framework [49] to obtain negative free energy maximization (Equations (A19)–(A21)).

(5)

Evaluating negative free energy. On each sequence of sensory inputs, we can evaluate the maximum negative free energies ${\mathcal{F}}_{{\mathit{\xi }}_1}^{*}$ for the agent ${\mathcal{M}}_1$ and ${\mathcal{F}}_{{\mathit{\xi }}_2}^{*}$ for the agent ${\mathcal{M}}_2$ according to Equation (A22). Then Bayesian Factors are evaluated by Equations (A33) and (A34).

In each gambling task, two agents ${\mathcal{M}}_1,{\mathcal{M}}_2$ generate their time-courses of the predicted states ${\mathit{\mu }}_0\left(t\right)$ on the expectation of the states of the two-armed bandit. The predicted states are recorded into a $d_0\times K$ matrix $\mathit{T}$

$$\mathit{T}=[{\mu }_0\left(t_1\right),{\mu }_0\left(t_2\right),\cdots ,{\mu }_0\left(t_K\right)]\in {\mathbb{R}}^{d_0\times K}.$$

Given any pair of sequences (randomly generated in step 1) $({\mathit{u}}_s,a_s,{\mathit{r}}_s)\in \mathcal{D}$, our Bayesian agent ${\mathcal{M}}_1$ and the RW agent dynamically infer the states, and form inference trajectories ${\mathit{T}}_1({\mathit{u}}_s,a_s,{\mathit{r}}_s)$ and ${\mathit{T}}_2({\mathit{u}}_s,a_s,{\mathit{r}}_s)$ respectively. The mean dynamic inference trajectory ${\overline{T}}_i$ and the standard deviation ${\sigma }_{{\mathit{T}}_i}$ are computed as

$$\begin{array}{cc}\hfill {\overline{\mathit{T}}}_i=& \frac{1}{\left|\mathcal{D}\right|}\sum _{({\mathit{u}}_s,a_s,{\mathit{u}}_s)\in \mathcal{D}}{\mathit{T}}_i({\mathit{u}}_s,a_s,{\mathit{r}}_s)\hfill \\ \hfill {\sigma }_{{\mathit{T}}_i}=& \sqrt{\frac{1}{\left|\mathcal{D}\right|-1}\sum _{({\mathit{u}}_s,a_s,{\mathit{r}}_s)\in \mathcal{D}}{\left[T_i({\mathit{u}}_s,a_s,{\mathit{r}}_s)-{\overline{T}}_i\right]}^2},\hfill \end{array}$$

(40)

where the notation $\left|\mathcal{D}\right|$ is the number of elements in the dataset $\mathcal{D}$ (e.g., $\left|\mathcal{D}\right|=100$).

Figure 9 shows that both the Bayesian agent (red lines) and the RW agent (blue dashed lines) are able to track the ground truth or real probabilities well, showing quick jumps at the points where the ground truth undertakes remodeling. However, the RW agent produces more variable predictions, as shown by larger standard deviation (blue shaded areas vs. red shaded areas). The RW agent often overshoots its estimation (blue dashed lines vs. black lines). These results demonstrate that the RW agent overfits observations.

In each gambling task, given the sensory inputs ${\mathit{u}}_s$, actions $a_s$ and rewards ${\mathit{r}}_s$, two agents ${\mathcal{M}}_1,{\mathcal{M}}_2$ generate their series of the expectations of the response action $P\left(a\right(t)=1)$, denoted by $\left\{P_1(a\left(t_k\right)=1)|k=1,2,\cdots ,K\right\}$ and $\left\{P_2(a\left(t_k\right)=1)|k=1,2,\cdots ,K\right\}$ respectively. Their regrets can be evaluated by substituting $P_i(a\left(t_k\right)=1)$ into Equation (40). The mean regret ${\overline{R}}_i$ and the standard deviation ${\sigma }_{{\mathit{T}}_i}$ on the synthetic dataset $\mathcal{D}$ are computed as

$$\begin{array}{cc}\hfill {\overline{R}}_i=& \frac{1}{\left|\mathcal{D}\right|}\sum _{({\mathit{u}}_s,a_s,{\mathit{r}}_s)\in \mathcal{D}}R\left(P_i(a\left(t_k\right)=1)\right|{\mathit{u}}_s,a_s,{\mathit{r}}_s)\hfill \\ \hfill {\sigma }_{{\mathit{R}}_i}=& \sqrt{\frac{1}{\left|\mathcal{D}\right|-1}\sum _{({\mathit{u}}_s,a_s,{\mathit{r}}_s)\in \mathcal{D}}{\left[R\left(P_i(a\left(t_k\right)=1)\right|{\mathit{u}}_s,a_s,{\mathit{r}}_s)-{\overline{R}}_i\right]}^2}.\hfill \end{array}$$

(41)

The mean ${\overline{R}}_1$ and standard deviation ${\sigma }_{{\mathit{R}}_1}$ of our Bayesian agent (based on GHBF) ${\mathcal{M}}_1$ are smaller than the mean ${\overline{R}}_2$ and standard deviation ${\sigma }_{{\mathit{R}}_2}$ of the RW-agent (based on the RW model) ${\mathcal{M}}_2$ (Figure 10).

To evaluate the two models more formally, given the three sequences $({\mathit{u}}_s,{\mathit{r}}_s,a_s)$, we computed Bayesian Factor $BF$ without Bayesian Information Criterion (BIC)

$$\begin{array}{cc}\hfill BF:=& \frac{p({\mathit{u}}_s,{\mathit{r}}_s,a_s|{\mathcal{M}}_1)}{p({\mathit{u}}_s,{\mathit{r}}_s,a_s|{\mathcal{M}}_2)}\hfill \\ \hfill \approx & exp({\mathcal{F}}_{{\mathit{\xi }}_1}^{*}-{\mathcal{F}}_{{\mathit{\xi }}_2}^{*}),\hfill \end{array}$$

(42)

and Bayesian Factor $B{F}_{BIC}$ with BIC (cf. Appendices Appendix E and Appendix F)

$$\begin{array}{c}\hfill B{F}_{BIC}:=exp\left({\mathcal{F}}_{{\mathit{\xi }}_1}^{*}-{\mathcal{F}}_{{\mathit{\xi }}_2}^{*}-\frac{d_{{\mathit{\xi }}_1}-d_{{\mathit{\xi }}_2}}{2}ln\left(K\right)\right),\end{array}$$

(43)

where $d_{{\mathit{\xi }}_i}$ is the number of free parameters estimated by the model. The notations ${\mathcal{F}}_{{\mathit{\xi }}_1}^{*},{\mathcal{F}}_{{\mathit{\xi }}_2}^{*}$ are respectively the maximal negative free energies of the two agents ${\mathcal{M}}_1,{\mathcal{M}}_2$ on the given pair of the sequences $({\mathit{u}}_s,{\mathit{r}}_s,a_s)$. Under both measures, Bayesian Factors on the observation dataset $\mathcal{D}$ are concentrated on the range larger than 100 (i.e., $BF>100$, $B{F}_{BIC}>100$) (Figure 11a,b), meaning decisive evidence for the Bayesian agent outperforming the RW agent according to

Table A1. Bayes Factors and interpretations.

Bayesian Factor $\mathit{BF}$	Interpretations
$0<BF<\frac{1}{100}$	Decisive evidence for ${\mathcal{M}}_2$
$\frac{1}{100}<BF<\frac{1}{10}$	Strong evidence for ${\mathcal{M}}_2$
$\frac{1}{10}<BF<\frac{1}{3}$	Moderate evidence for ${\mathcal{M}}_2$
$\frac{1}{3}<BF<1$	Weak evidence for ${\mathcal{M}}_2$
$1<BF<3$	Weak evidence for ${\mathcal{M}}_1$
$3<BF<10$	Moderate evidence for ${\mathcal{M}}_1$
$10<BF<100$	Strong evidence for ${\mathcal{M}}_1$
$BF>100$	Decisive evidence for ${\mathcal{M}}_1$

.

6. Discussion

6.1. Contributions of This Work

In this article, we have introduced a hierarchical Bayesian model that describes how to infer volatility (i.e., environmental uncertainty and correlations) in a multi-dimensional space. In this model, the bottom level is to learn the state expectation of a multi-armed bandit, which is described by a multivariate Bernoulli distribution. The natural parameter ${\mathit{x}}_1$ of the Bernoulli distribution is learned by the first level. Under the Brownian and Gaussian assumption on ${\mathit{x}}_1$, volatility can be strictly determined by the Cholesky Decomposition of pervasion intensity of Brownian motion ${\mathit{x}}_1$ [46]. Therefore, we can define the volatility in ${\mathit{x}}_1$ as the Cholesky Decomposition of pervasion intensity of ${\mathit{x}}_1$. Next, the volatility in ${\mathit{x}}_1$ can be represented by ${\mathit{x}}_2$, with evolves as a Brownian motion. The low-order interactions between the dimensions of the Bernoulli distribution and the environmental uncertainties are captured in the second level, corresponding to ${\mathit{x}}_2$.

The hierarchical Bayesian model assumes that the tendency of a binary pattern evolves as a general Brownian motion at the first level. The tendency can be updated by Equation (19), where prediction error ${\mathit{PE}}_0\left(t_k\right)$ is the information gap between the agent’s belief and sensory input. This quantity is a target that the agent should learn to diminish. The parameter vector ${\mathit{\zeta }}_1$ functions as weighting vector to weight prediction error ${\mathit{PE}}_0\left(t_k\right)$. The covariance ${\mathit{C}}_1\left(t_k\right)$ plays the role of complex adaptive learning rate in Equation (19).

In principle, the proposed model could be easily generalized to a Bayesian framework for decision making in high-dimensional multinary environments, by defining appropriate forms of perceptual models and response models. In this study, the input space was assumed to be binary. For multinary environments, the representations of the tendency of the inputs could be defined accordingly to form a hierarchical perceptual model. Here we derived a response model from Bayesian decision theory with the goal of maximizing expected rewards or minimizing expected risk or loss [47,52]. For other problems of interest, it is sufficient to construct a compatible response model addressing the particular optimization criteria of the question. For example, recognition and navigation tasks could be formulated in the proposed Bayesian framework to cope with the interactions between multimodal information [53,54].

In summary, the main contributions of this work are twofold. First, the model captures the correlations between the dimensions of the sensory space, and is able to make decisions contingent on the structure of the sensory inputs. Simulations show that our model is applicable to complex inference and decision making tasks that could not be tackled by methods with independence assumptions of the high dimensional input features. Second, the model represents the tendency and volatility of the sensory inputs in a hierarchical manner based on the idea of nested Brownian motions. The resulting hierarchical computational framework naturally allows the interactions between layers, and is able to track the dynamics of the environment.

6.2. Related Works

The proposed hierarchical Bayesian model is most related to the Rescorla–Wagner model [35,51,55,56]. Equation (18) has the form of a generalized form of the Rescorla–Wagner equation in reinforcement learning [26]

$${\mathit{\mu }}_1\left(t_k\right)={\mathit{\mu }}_1\left(t_{k-1}\right)+\Gamma \Delta {\mathit{\mu }}_1\left(t_k\right),$$

where $\Delta {\mathit{\mu }}_1\left(t_k\right)$ is an error signal (or learned target) at time $t_k$. In the cognitive neuroscience field, some variants of the RW model have been introduced for the behavioral paradigm of multi-armed bandits. However, due to the limitation of the RW model, it is difficult to capture the volatility of the signal. More importantly, since the learning rate in RW model is constant, it is difficult to interpret the subject’s dynamic process of capturing effective information during the experiment. As an example, given the reward $R\left(t_{k-1}\right)$ at time $t_k$, the standard RW model estimates the value state variable V by

$$\begin{array}{c}\hfill V\left(t_k\right)=V\left(t_{k-1}\right)+\alpha PE\left(t_k\right)\\ \hfill PE\left(t_k\right)=R\left(t_{k-1}\right)-V\left(t_{k-1}\right),\end{array}$$

(44)

which is simplified to

$$\begin{array}{c}\hfill V\left(t_k\right)=(1-\alpha )V\left(t_{k-1}\right)+\alpha R\left(t_{k-1}\right).\end{array}$$

(45)

It is clear to see that the learning rate $\alpha$ plays a role of a moving average, weighting initial value $V\left(t_0\right)$ and a reward sequence $R\left(t_1\right),R\left(t_2\right),\cdots ,R\left(t_k\right)$. This is an inflexible filtering method to cope with volatility. For small learning rate $\alpha$, the RW model prefers to predict based on the input history, a good scenario for slow changing signals. The RW model with large learning rate prefers to rely on most recent rewards, a good scenario for fast changing signals. However, the RW model did not unify the two learning processes (i.e., the learning rate is not able be adapted depending on the environment and agent state). In this sense, our hierarchical Bayesian model provides a theoretically justified mechanism to adapt learning rate dynamically according to the volatility of the environment and the states of the agent.

For a single-step update, the time complexity of our hierarchical Bayesian model is $O\left(d_0^4\right)$ (Equations (13)–(31)), while the time complexity of the RW model is $O\left(d_0\right)$. In our model, capturing volatility to form adaptive learning rate leads to a higher computational cost. Experiments show that this computational cost is necessary for the model to flexibly adapt to volatile environments. On the synthetic dataset $\mathcal{D}$, the trajectories of the state estimation formed by our hierarchical Bayesian model (light red shadow area in Figure 9) are distributed narrower than those of the RW model (light blue shadow area in Figure 9), indicating the stability and robustness of the proposed model.

6.3. Strengths and Limitations

Our hierarchical Bayesian model is general enough to be easily applied in high-dimensional environments. The number of parameters of the model scales quadratically with respect to the dimension of the input space. Given the number of dimensions $d_0$, corresponding to the dimension of ${\mathit{x}}_0\left(t\right)$ at the bottom level (i.e., sensory input $\mathit{u}\left(t\right)$), the dimension of the parameter ${\mathit{\xi }}_1$ of our perceptual model is $d_0+2d_1+5d_2=\frac{d_0(d_0+5)}{2}$ (cf. Appendix C). In the Bayesian learning process, the optimization algorithm (i.e., quasi-Newton Broyden–Fletcher–Goldfarb–Shanno method) needs to numerically evaluate the gradient of the negative free energy with respect to each component of the model parameter ${\mathit{\xi }}_1$. For a large number of dimensions $d_0$, parallel computing framework based on CPU and GPU need to be developed in order to improve the evaluation efficiency of numerical gradients.

6.4. Future Work

In this paper, we construct a hierarchical Bayesian model for inferring and decision-making in multivariate volatile binary environment, and test and validate it on a synthetic dataset. We plan to use this model to explain human decision-making behaviors and brain activities. To this end, we need to collect behavior and neuroimaging data while human subjects are performing the same task of multi-armed bandit as defined in this paper. For theoretical interest, the mechanism of the adaptive learning rate and correlation among natural parameters are worthy for further clarification in our hierarchical model, and we look forward to analyzing these critical mechanisms in future investigation.

7. Conclusions

We have introduced a hierarchical Bayesian model for decision making in high-dimensional volatile environments, and derived a family of interpretable closed form update rules. Based on this framework, we define a Bayesian agent endowed with the proposed hierarchical Bayesian model, as a mentalizing model of a biological agent, to perform an abstract multi-armed bandit task. Simulations show that our model is applicable to complex tasks that could not be tackled by models with independency assumptions. Crucially, the proposed model contains a hierarchical perceptual model that is able to capture different covariances (e.g., prediction covariance, posterior covariance, likelihood covariance). As an important indicator of mental process, prediction correlation is dynamically estimated in the second level of the hierarchical perceptual model. Prediction correlation describes quantitatively (weak) pairwise interactions among different perception quantities (e.g., natural parameters of multi-armed bandits). In conclusion, the proposed hierarchical Bayesian model provides a powerful tool to solve complex perception and decision making problems in high-dimensional volatile environments [57], as well as to quantify complex phenomena such such as perceptual decision making, spatial navigation, social interactions and exploratory behaviors [58,59,60,61,62,63].