Accelerating Convergence of Langevin Dynamics via Adaptive Irreversible Perturbations

Abstract

Irreversible perturbations in Langevin dynamics have been widely recognized for their role in accelerating convergence in simulations of multi-modal distributions $\pi \left(\theta \right)$. A commonly used and easily computed standard irreversible perturbation is $J\nabla log\pi \left(\theta \right)$, where J is a skew-symmetric matrix. However, Langevin dynamics employing a fixed-scale standard irreversible perturbation encounter a trade-off between local exploitation and global exploration, associated with small and large scales of standard irreversible perturbation, respectively. To address this trade-off, we introduce the adaptive irreversible perturbations Langevin dynamics, where the scale of the standard irreversible perturbation changes adaptively. Through numerical examples, we demonstrate that adaptive irreversible perturbations in Langevin dynamics can enhance performance compared to fixed-scale irreversible perturbations.

1. Introduction

Markov chain Monte Carlo (MCMC) algorithms represent the gold standard for Bayesian inference. One widely employed category of MCMC algorithms is based on Langevin dynamics (LD), which employs the gradient of the negative logarithm of the target distribution (energy function) to define a stochastic differential equation (SDE). Recent research has focused on improving the efficiency of LD-based MCMC algorithms [1,2,3,4]. While LD-based MCMC algorithms have made notable strides in simulating high-dimensional systems to address big data challenges, they exhibit a fast mixing rate, primarily when the energy landscape is uncomplicated. However, when dealing with simulations of multi-modal distributions, the LD approach is susceptible to becoming ensnared in local energy minima, leading to biased statistical inferences. To improve convergence in complex energy landscapes, various strategies have been proposed. One widely recognized and effective approach involves introducing specific perturbations to LD, a technique that has demonstrated promising utility in the literature [5,6,7,8,9].

It is well established that specific perturbations to LD can accelerate the convergence of the dynamics toward the target distribution. Hwang et al. [5] demonstrate that the inclusion of a weighted divergence-free drift in LD leads to accelerated convergence to target distribution. Rey-Bellet and Spiliopoulos [6] demonstrate that introducing appropriate reversible and irreversible perturbations to diffusion processes can have several beneficial effects. These include reducing the spectral gap of the generator, increasing the rate function for large deviations, and decreasing the asymptotic variance of estimators. One notable example of a reversible perturbation is the Riemannian manifold Langevin dynamics (RMLD), as introduced by Girolami and Calderhead [10]. This approach involves defining a Riemannian metric to modify the computation of distances and gradients. It is also well-documented [6,7,11] that certain irreversible perturbations to the LD approach can enhance its convergence while still maintaining its invariant distribution. Given the individual successes of both irreversible and reversible perturbations, such as the RMLD, in enhancing Langevin samplers, Zhang [9] has introduced a novel approach involving geometry-informed irreversible perturbations to accelerate the convergence of LD-based algorithms for Bayesian computation. Zhang’s work combines irreversible and reversible perturbations by introducing a unique form of irreversible perturbation for RMLD that is informed by the underlying geometry. It is demonstrated that this innovative irreversible perturbation can lead to improved estimation performance compared to traditional irreversible perturbations that do not consider geometric characteristics.

While perturbations that account for the geometric characteristics of the invariant distribution, such as RMLD and geometry-informed irreversible perturbations, notably accelerate the convergence of LD towards its invariant distribution, considering these geometric features often necessitates the computation of metrics like the expected Fisher information and Hessian matrices in each iteration. This substantially elevates the computational overhead per iteration. So, in many application scenarios, for computational convenience, we often choose standard irreversible perturbations, $\Gamma \left(\theta \right)=Jlog\pi \left(\theta \right)$, with J denoting a constant skew-symmetric matrix and $\pi \left(\theta \right)$ denoting the target distribution. Indeed, with a large magnitude of J, the flow $Jlog\pi \left(\theta \right)$ will result in rapid exploration of the energy function. However, this introduces additional constraints, as arbitrarily increasing the magnitude of J not only leads to a significant discretization error but also weakens the ability of LD to exploit the local geometry. Therefore, we expect J’s magnitude to be adaptable in the utilization of an LD sampler with irreversible perturbations. In most cases, the magnitude of J remains relatively small. Then, increasing the magnitude of J adaptively enables LD to rapidly explore the energy function. Once LD reaches new regions, the magnitude of J adapts to be small, ensuring stable sampling. This ensures that the LD sampler with irreversible perturbations possesses excellent exploration capabilities while maintaining stable local exploitation capabilities. So far, the concept of adaptively changing the magnitude of J has not been implemented. This is because a simple alteration in the magnitude of J would result in LD not converging to the target distribution.

In this paper, we propose an adaptively irreversible replica-exchange Langevin dynamics (AIRELD) algorithm by combining the parallel tempering strategy replica exchange Langevin dynamics (RELD) [1] and perturbation strategy. Within AIRELD, the magnitude of irreversible perturbations adjust dynamically, correlating with the temperature changes in the parallel tempering process. At higher temperatures, the irreversible perturbation scale amplifies, resulting in rapid exploration of the energy function. Consequently, this enhancement raises the probability of escaping local traps, fostering a comprehensive exploration of the entire domain. Conversely, with lower temperatures, the behavior of LD takes on the characteristics of gradient flow, emphasizing the utilization of local geometry. This, in turn, results in stable local sampling. The adaptive irreversible perturbations accelerate the convergence of LD and, if needed, can be employed in conjunction with the stochastic gradient Langevin dynamics (SGLD) algorithm to leverage the computational advantages of stochastic gradient methods.

The organization of the paper is as follows: In Section 2, we offer an overview of the parallel tempering algorithm, RELD, and irreversible perturbations within LD, which have the potential to enhance sampling efficiency from target distribution. Simultaneously, In Section 3, we introduce our innovative algorithm AIRELD and extend it to the scenario of stochastic gradients. Section 4 is devoted to presenting simulation studies that highlight the AIRELD approach’s performance compared to the choice of standard irreversible perturbations and RELD. Lastly, Section 5 provides a summary of our findings and outlines potential directions for future research.

2. Improving the Performance of Langevin Dynamics

We begin by revisiting relevant background information on Langevin samplers, RELD, and irreversible perturbations in LD.

A frequently employed LD sampling algorithm is the LD, which is characterized by the SDE

$$\mathrm{d}{\theta }_t=-\nabla U\left({\theta }_t\right)\mathrm{d}t+\sqrt{2\tau }\mathrm{d}{\mathit{W}}_t,$$

(1)

where $\theta \in {\mathbb{R}}^d$, ${\left\{{\mathit{W}}_t\right\}}_{t\ge 0}$ represents a standard d-dimensional Brownian motion, $U(·)$ is the energy function, and $\tau >0$ is the temperature parameter. Under reasonable regularity assumptions, the process ${\left\{{\theta }_t\right\}}_{t\ge 0}$ possesses a unique invariant distribution, which is absolutely continuous with respect to the Lebesgue measure with density

$$\pi \left(\theta \right)=\frac{e^{-U\left(\theta \right)/\tau }}{{\int }_{{\mathbb{R}}^d}{e}^{-U\left(\theta \right)/\tau }\mathrm{d}\theta }.$$

(2)

In the context of Bayesian statistics, when dealing with independent data, $y_1,\dots ,y_N$, the posterior distribution is $\pi \left(\theta \right)\propto p\left(\theta \right){\prod }_{i=1}^{N}f\left(y_i\right|\theta )$, where $p\left(\theta \right)$ represents the prior density, and $f\left(y_i\right|\theta )$ represents the likelihood for the ith observation. In this particular framework, $U\left(\theta \right)=-logp\left(\theta \right)-{\sum }_{i=1}^{N}logf\left(y_i\right|\theta )$.

2.1. Replica Exchange Langevin Dynamics

The temperature parameter $\tau$ in LD (1) plays a crucial role in the sampling process of the high-dimensional energy function $U\left(\theta \right)$. In practice, the LD approach is often executed with a fixed temperature. A high temperature $\tau$ endows the LD with an enhanced exploration capability. It significantly accelerates the convergence to the flattened invariant distribution across the entire domain. However, although the distribution of ${\left\{{\theta }_t\right\}}_{t\ge 0}$ converges more rapidly to the global invariant distribution $\pi \left(\theta \right)$, it is less concentrated around the global minima of $U\left(\theta \right)$. On the other hand, when the temperature parameter $\tau$ is small, the convergence of ${\left\{{\theta }_t\right\}}_{t\ge 0}$ becomes slow, and particles may remain trapped in a local minimum under a limited computational budget without effectively exploring the entire domain. As a result, over a finite period, the obtained samples are distant from the invariant distribution $\pi \left(\theta \right)$. Therefore, the effectiveness of utilizing a constant temperature is quite limited. Hence, in the tempering strategy, temperature is considered as an auxiliary variable, lower temperatures concentrate the distribution’s mass around the global optimum, resulting in higher energy barriers and local traps. On the other hand, higher temperatures flatten the distribution, reduce energy barriers, and facilitate transitions between different modes of distribution. Two extensively researched methods in the field of tempering are worth noting. The first is parallel tempering, which has been investigated in the context of MCMC for the purpose of sampling from multi-modal distributions, such as RELD. For further insights, refer to works by Earl and Deem [12]; Sindhikara et al. [13]; Deng et al. [14], as well as their associated references. Additionally, another well-explored approach is simulated tempering, a technique akin to parallel tempering, both aimed at accelerating the convergence of LD. This method, introduced by Marinari and Parisi [15] and further developed by Geyer and Thompson [16], has a primary distinction from replica exchange: while the replica exchange tracks multiple diffusion dynamics driven by different temperatures and accelerates convergence through swapping, the simulated tempering focuses on a single diffusion process but treats temperature as a stochastic process.

A recent and potent parallel tempering algorithm, known as RELD, a specialized variant of the parallel tempering algorithm, has been introduced to accelerate the convergence of the SDE, as illustrated in Figure 1.

RELD bridges the gap between global exploration and local exploitation by employing a high-temperature particle for exploration and a low-temperature particle for exploitation, allowing them to swap simultaneously. RELD consists of the following LD with temperatures ${\tau }_1<{\tau }_2$:

$$\begin{array}{cc}\hfill & \mathrm{d}{\theta }_t^{\left(1\right)}=-\nabla U\left({\theta }_t^{\left(1\right)}\right)\mathrm{d}t+\sqrt{2{\tau }_1}\mathrm{d}{\mathit{W}}_t^{\left(1\right)},\hfill \\ \hfill & \mathrm{d}{\theta }_t^{\left(2\right)}=-\nabla U\left({\theta }_t^{\left(2\right)}\right)\mathrm{d}t+\sqrt{2{\tau }_2}\mathrm{d}{\mathit{W}}_t^{\left(2\right)}.\hfill \end{array}$$

(3)

The invariant distribution of $({\theta }_t^{\left(1\right)},{\theta }_t^{\left(2\right)})$ is

$$\pi \left({\theta }^{\left(1\right)},{\theta }^{\left(2\right)}\right)\propto e^{-\frac{U\left({\theta }^{\left(1\right)}\right)}{{\tau }_1}-\frac{U\left({\theta }^{\left(2\right)}\right)}{{\tau }_2}}.$$

(4)

The primary concept behind the replica exchange is to enhance the low-temperature particle’s capacity for comprehensive global exploration through the exchange of its position with the high-temperature particle. Concurrently, the low-temperature particle possesses good local exploitation capability, making its trajectory the preferred choice for the sampling outcome.

By allowing the exchange of positions between the two particles, the positions may transition from $\left({\theta }_t^{\left(1\right)},{\theta }_t^{\left(2\right)}\right)$ to $\left({\theta }_{t+\mathrm{d}t}^{\left(2\right)},{\theta }_{t+\mathrm{d}t}^{\left(1\right)}\right)$ at a swapping rate denoted as $rS\left({\theta }_t^{\left(1\right)},{\theta }_t^{\left(2\right)}\right)\mathrm{d}t$. Here, the constant $r\ge 0$ represents the swapping intensity, and $S(·,·)$ is defined as:

$$S\left({\theta }_t^{\left(1\right)},{\theta }_t^{\left(2\right)}\right)=e^{\left(\frac{1}{{\tau }_1}-\frac{1}{{\tau }_2}\right)\left(U\left({\theta }_t^{\left(1\right)}\right)-U\left({\theta }_t^{\left(2\right)}\right)\right)}$$

(5)

In such scenarios, RELD operates as a Markov jump process, which is reversible, leading to the same invariant distribution (4).

2.2. Irreversible Perturbations

When faced with a multi-modal target distribution, characterized by numerous local minima of $U\left(\theta \right)=-log\pi \left(\theta \right)$, the convergence to invariant distribution for the LD may be characterized by a slow pace. In such circumstances, it can be beneficial to modify the dynamics through adjustments to the drift, facilitating the escape of the dynamics from the local minima of $U\left(\theta \right)$.

Consider the following LD:

$$\mathrm{d}{\theta }_t=[\nabla log\pi \left({\theta }_t\right)+\Gamma \left({\theta }_t\right)]\mathrm{d}t+\sqrt{2}\mathrm{d}{\mathit{W}}_t.$$

(6)

when $\Gamma \left(\theta \right)\equiv 0$, the process exhibits reversibility and possesses $\pi \left(\theta \right)$ as its invariant distribution. However, if $\Gamma \left(\theta \right)\ne 0$, then the resultant process will generally lack time-reversibility, unless $\Gamma \left(\theta \right)$ can be expressed as a multiple of $\nabla log\pi \left(\theta \right)$ [17].

However, even with an irreversible perturbation, it is still possible to preserve the invariant distribution of the system. By considering the Fokker–Planck equation, it can be shown that if $\Gamma \left(\theta \right)$ is selected such that $\nabla ·(\Gamma (\theta \left)\pi \right(\theta \left)\right)=0$, then $\pi$ will remain the invariant distribution. It is clear that numerous (in fact, infinitely many) approaches can be employed to alter the reversible dynamics while maintaining the invariant distribution. A natural inquiry arises: can the inclusion of an irreversible term enhance the rate of convergence to invariant distribution, and if so, is there an optimal perturbation choice that maximizes this rate of convergence to invariant distribution? To date, this problem has not been systematically resolved, and no general conclusions have been reached. Only a small subset of problems has been addressed. For example, Lelièvre et al. [18] provide a comprehensive analysis of the best choice for an irreversible perturbation in the context of a linear system, one that leads to the fastest convergence. In practical applications, a frequently employed selection in the literature is the standard irreversible perturbation $\Gamma \left(\theta \right)=J\nabla log\pi \left(\theta \right)$, with J representing a constant skew-symmetric matrix, i.e., $J=-J^T$. The advantage of this choice is clear, as it necessitates only one additional matrix–vector multiplication for implementation.

The advantages of incorporating irreversible perturbations are widely acknowledged. Hwang et al. [5] present a significant finding, revealing that, under certain conditions, the spectral gap, representing the difference between the top two eigenvalues of the Markov semigroup generator, increases when $\Gamma \left(\theta \right)\ne 0$. In the works of Rey-Bellet and Spiliopoulos [6,7], they introduce the concept of a large deviation rate function as a performance measure for sampling from the invariant distribution. By establishing a connection between large deviation rate function and the asymptotic variance of the long-term average estimator, they illustrate that introducing an appropriate perturbation $\Gamma \left(\theta \right)$ not only increases the large deviation rate function but also diminishes the asymptotic variance of the estimator. The use of irreversible proposals in the Metropolis-adjusted Langevin algorithm (MALA) was studied by Ottobre et al. [19].

3. Adaptively Irreversible Replica-Exchange Langevin Dynamics

While perturbations that account for the geometric characteristics of the invariant distribution, such as RMLD and geometry-informed irreversible perturbations, notably accelerate the convergence of LD toward its invariant distribution, the consideration of these geometric features often necessitates the computation of metrics like expected Fisher information and Hessian matrices during each iteration. This substantially elevates the computational overhead per iteration. As a result, in most practical scenarios, we frequently adopt the standard irreversible perturbation $\Gamma \left(\theta \right)=J\nabla log\pi \left(\theta \right)$, with J denoting a constant skew-symmetric matrix. Using $\Gamma \left(\theta \right)=J\nabla log\pi \left(\theta \right)$, since $\pi \left(\theta \right)\propto \exp (-U(\theta \left)\right)$, Equation (6) transforms into the following expression:

$$\begin{array}{cc}\hfill \mathrm{d}\theta \left(t\right)& =[\nabla log\pi \left({\theta }_t\right)+\Gamma \left({\theta }_t\right)]\mathrm{d}t+\sqrt{2}\mathrm{d}{\mathit{W}}_t,\hfill \\ \hfill & =-(I+J)\nabla U\left(\theta \right)\mathrm{d}t+\sqrt{2}\mathrm{d}{\mathit{W}}_t.\hfill \end{array}$$

(7)

From the above discussion in Section 2.2, we know that the introduction of the standard irreversible perturbation $\Gamma \left(\theta \right)=J\nabla log\pi \left(\theta \right)$ leads to a certain improvement in the convergence speed of LD. In fact, we can provide a rudimentary explanation from the LD (7). With the presence of J, and without altering the invariant distribution, both the magnitude and direction of $\nabla U\left(\theta \right)$ undergo some changes. When the magnitude of J is very small, LD (7) behaves quite similarly to standard LD. However, when the magnitude of J is significantly larger, it enables LD to transition from one peak to another in a multi-modal invariant distribution, all without altering the invariant distribution. It is worth noting that throughout the entire process, J remains constant.

During the implementation of LD (7), we notice that the magnitude of J introduces a trade-off between two opposing effects, commonly referred to as global exploration and local exploitation. In more detail, a larger magnitude of J results in a larger gradient, thereby increasing the probability of escaping from local minima and facilitating exploration across the entire domain. Conversely, a smaller magnitude of J causes the LD to behave more like a gradient flow, emphasizing the exploitation of local geometry, leading to a more concentrated invariant distribution. Hence, a simple idea came to mind: when the magnitude of J is very small, it can slightly improve the convergence speed while ensuring the stability of LD for sampling. Conversely, when the magnitude of J is relatively large, it can aid LD in exploring the various peaks of a multi-modal target distribution. Consequently, our proposed naive approach entails alternating between small and large magnitudes of J during LD execution, with these transitions occurring at manually defined intervals. We introduced these straightforward variations with the expectation of improving the operational efficiency of LD. Unfortunately, this artificial swapping led to the failure of LD to converge to the invariant distribution. Given that the manually determined swaps resulted in non-convergence of LD, we attempted to find an adaptive swapping strategy. An attempt involves utilizing the temperature swapping criterion defined in the RELD. This criterion can be used to adaptively adjust the magnitude of the skew-symmetric matrix J. Incorporating this adjustment, we introduce the adaptively irreversible replica-exchange Langevin dynamics (AIRELD) algorithm, outlined in Algorithm 1.

Algorithm 1: Adaptively irreversible replica-exchange Langevin dynamics

AIRELD is characterized by the following LD with temperatures ${\tau }_1<{\tau }_2$:

$$\begin{array}{cc}\hfill & \mathrm{d}{\theta }_t^{\left(1\right)}=-\left(I+J\right)\nabla U\left({\theta }_t^{\left(1\right)}\right)\mathrm{d}t+\sqrt{2{\tau }_1}\mathrm{d}{\mathit{W}}_t^{\left(1\right)},\hfill \\ \hfill & \mathrm{d}{\theta }_t^{\left(2\right)}=-\left(I+J\right)\nabla U\left({\theta }_t^{\left(2\right)}\right)\mathrm{d}t+\sqrt{2{\tau }_2}\mathrm{d}{\mathit{W}}_t^{\left(2\right)}.\hfill \end{array}$$

(8)

In contrast to the standard LD approach, the LD approach in Equation (8) introduces perturbations $\Gamma \left(\theta \right)=J\nabla log\pi \left(\theta \right)$ in the drift. The ensuing dynamics are described by

$$\mathrm{d}{\theta }_t=\left(\nabla log\pi \left({\theta }_t\right)+\Gamma \left({\theta }_t\right)\right)\mathrm{d}t+\sqrt{2}\mathrm{d}{\mathit{W}}_t,$$

(9)

where $\Gamma (·)$ is a smooth vector field. The stationary Fokker–Planck Equation (9) then takes the form

$$\nabla ·(\Gamma (\theta \left)\pi \right(\theta \left)\right)=0.$$

Consequently, any divergence-free vector field perturbations respecting the target distribution can be employed to construct irreversible ergodic dynamics, with $\pi \left(\theta \right)$ as the invariant distribution. Given $\nabla ·(J\nabla log\pi (\theta \left)\pi \right(\theta \left)\right)=0$, the invariant distribution of $({\theta }_t^{\left(1\right)},{\theta }_t^{\left(2\right)})$ in Equation (8) is

$$\pi \left({\theta }^{\left(1\right)},{\theta }^{\left(2\right)}\right)\propto e^{-\frac{U\left({\theta }^{\left(1\right)}\right)}{{\tau }_1}-\frac{U\left({\theta }^{\left(2\right)}\right)}{{\tau }_2}}.$$

(10)

AIRELD facilitates the exchange of positions between the two particles using the swapping rule in RELD, which possesses the invariance property. Therefore, the invariant distribution of AIRELD remains as expressed in (10). Our experiments also affirm that the AIRELD approach retains the same invariant distribution.

Stochastic Gradient Langevin Dynamics

The SGLD approach stands out as the most effective approach for addressing computational challenges in Bayesian inference when confronted with large datasets. In order to reduce computational requirements per iteration, Welling and Teh [20] introduced the innovative SGLD method, which combines the strengths of LD and stochastic gradients. In each iteration of LD (1), a subset of n samples is randomly selected with replacement from the entire dataset, and the gradient with respect to these n samples is computed. Consequently, $\nabla U\left({\theta }_t\right)$ in (1) is replaced by an unbiased estimator $\widehat{U}\left({\theta }_t\right)=\frac{N}{n}{\sum }_{i\in {\mathcal{S}}_n}\nabla U_i\left({\theta }_t\right)$, where $U_i\left({\theta }_t\right)=-logp\left(y_i\right|{\theta }_t)-\frac{1}{N}logp\left({\theta }_t\right)$ and $S_n$ is a random sample from $\{1,\cdots ,N\}$ without replacement. The detailed SGLD approach is outlined in Algorithm 2. When the size of the selected subset, n, is much smaller than the total number of samples, N, it is evident that the SGLD approach offers a significantly reduced computational cost per iteration compared to MALA. On the other hand, Welling provided informal proof that the SGLD approach converges to the true posterior distribution as the step size approaches zero. Therefore, the SGLD approach offers certain advantages when computing resources are limited.

Algorithm 2: Stochastic Gradient Langevin Dynamics

The SGLD approach, when employed with a variable and diminishing step size, has been demonstrated to be consistent. In other words, the invariant distribution of the discretized system is equivalent to that of the continuous system [21]. However, the use of a decreasing step size mitigates the computational cost savings associated with computing stochastic gradients. To address this, a version of the SGLD approach with a fixed step size was introduced by Vollmer et al. [22]. This work also includes theoretical characterizations of asymptotic and finite-time bias and variance. In our numerical experiments, we utilize the stochastic gradient version of the Langevin algorithm with a fixed step size to illustrate that the SGLD approach can be combined with adaptive irreversible perturbations.

However, directly substituting the gradient $\nabla U\left({\theta }_t\right)$ with its unbiased estimator, the stochastic gradient $\widehat{U}\left({\theta }_t\right)$, in AIRELD is not feasible. It is apparent that the unbiased estimators $\widehat{U}\left({\theta }_t^{\left(1\right)}\right)$ and $\widehat{U}\left({\theta }_t^{\left(2\right)}\right)$ in $\widehat{S}=r{e}^{\left(\frac{1}{{\tau }_1}-\frac{1}{{\tau }_2}\right)\left(\widehat{U}\left({\theta }_t^{\left(1\right)}\right)-\widehat{U}\left({\theta }_t^{\left(2\right)}\right)\right)}$ do not offer an unbiased estimator of $S=r{e}^{\left(\frac{1}{{\tau }_1}-\frac{1}{{\tau }_2}\right)\left(U\left({\theta }_t^{\left(1\right)}\right)-U\left({\theta }_t^{\left(2\right)}\right)\right)}$ after a non-linear transformation, resulting in a large bias. Deng et al. [14] introduce replica exchange stochastic gradient Langevin dynamics (RESLGD) to automatically mitigate this bias by assuming normality on the stochastic energy, i.e., $\widehat{U}\left(\theta \right)\sim \mathcal{N}\left(U\left(\theta \right),{\sigma }^2\right)$. Consequently, it follows that

$$\widehat{U}\left({\theta }^{\left(1\right)}\right)-\widehat{U}\left({\theta }^{\left(2\right)}\right)=U\left({\theta }^{\left(1\right)}\right)-U\left({\theta }^{\left(2\right)}\right)+\sqrt{2}\sigma {\mathit{W}}_1,$$

(11)

where ${\mathit{W}}_1$ follows the standard normal distribution. Within this assumption, an unbiased estimator of $S=r{e}^{\left(\frac{1}{{\tau }_1}-\frac{1}{{\tau }_2}\right)\left(U\left({\theta }_t^{\left(1\right)}\right)-U\left({\theta }_t^{\left(2\right)}\right)\right)}$ is obtained, given by

$$\widehat{S}=e^{\left(\frac{1}{{\tau }_1}-\frac{1}{{\tau }_2}\right)\left(\widehat{U}\left({\theta }^{\left(1\right)}\right)-\widehat{U}\left({\theta }^{\left(2\right)}\right)-\left(\frac{1}{{\tau }_1}-\frac{1}{{\tau }_2}\right){\sigma }^2\right)},$$

(12)

and the correction term $\left(\frac{1}{{\tau }_1}-\frac{1}{{\tau }_2}\right){\sigma }^2$ aims to eliminate the bias from the swaps. In practice, the exact variance ${\sigma }^2$ is rarely known and is subject to estimation. Hence, stochastic approximation (SA) is employed. In each SA step, one obtains an unbiased sample variance ${\tilde{\sigma }}^2$ for the true ${\sigma }^2$ and updates the adaptive estimate ${\widehat{\sigma }}_{t+1}^2$ through

$${\widehat{\sigma }}_{t+1}^2=\left(1-{\gamma }_t\right){\widehat{\sigma }}_t^2+{\gamma }_t{\tilde{\sigma }}_{t+1}^2,$$

(13)

where ${\gamma }_t$ is the smoothing factor at the t-th SA step. With this adjustment, Algorithm 3 outlines the adaptively irreversible replica-exchange stochastic gradient Langevin dynamics (AIRESGLD) algorithm.

Algorithm 3: Adaptively irreversible replica-exchange stochastic gradient Langevin dynamics.

4. Experiments

4.1. Parameters of a Normal Distribution

This example is consistent with the one utilized in work by Girolami and Calderhead [10]. Consider a dataset of $\mathbb{R}$-valued data denoted as $\mathbf{X}={\left\{X_i\right\}}_{i=1}^N\sim$$\mathcal{N}\left(\mu ,{\sigma }^2\right)$, where our goal is to infer the parameters $\mu$ and $\sigma$. In this example, the state space is represented as $\theta ={[\mu ,\sigma ]}^T$. The prior distribution over $\mu$ and $\sigma$ is intentionally chosen to be uniform (flat) and, therefore, improper. The log-posterior distribution can be expressed as follows:

$$logp(\mu ,\sigma \mid \mathbf{X})=-\frac{N}{2}log2\pi -Nlog\sigma -\sum _{i=1}^N\frac{{\left(X_i-\mu \right)}^2}{2{\sigma }^2}.$$

(14)

The log-posterior gradient can be expressed as follows:

$$\nabla logp(\mu ,\sigma \mid \mathbf{X})=\left[\begin{array}{c}{M}_1\left(\mu \right)/{\sigma }^2\\ -N/\sigma +M_2\left(\mu \right)/{\sigma }^3\end{array}\right]$$

where $M_1\left(\mu \right)={\sum }_{i=1}^N\left(X_i-\mu \right)$, and $M_2\left(\mu \right)={\sum }_{i=1}^N\left(X_i-\right.$${\mu )}^2$.

In this experiment, we conduct a comparison involving LD, RELD, AIRELD, and standard irreversible perturbation LD through the performance of the estimators of ${\varphi }_1(\mu ,\sigma )=\mu +\sigma$ and ${\varphi }_2(\mu ,\sigma )={\mu }^2+{\sigma }^2$. Our assessment is based on various performance metrics, including mean squared error (MSE), bias, and variance.

In our experimental configuration, we use a dataset comprising $N=30$ data points. This dataset is generated by sampling from a normal distribution with true parameters ${\mu }_{\mathrm{true}}=0$ and ${\sigma }_{\mathrm{true}}=10$. We adopt a constant step size of $h={10}^{-3}$. For RELD and AIRELD, we use two temperatures: a low temperature ${\tau }_1=0.1$ and a high temperature ${\tau }_2=1.5$, with a swapping intensity of 1. For standard irreversible LD (7), we set $J=0.1\times \left[\begin{array}{cc}0& 1\\ -1& 0\end{array}\right]$. Our simulations involve generating $M=1000$ independent trajectories for each system, with each trajectory consisting of $T={10}^6$ steps. We initialize the system with $\mu =5$ and $\sigma =20$. To ensure the reliability of our results, we subject each trajectory to a burn-in period of 5000 iterations.

We visualize the MSE, squared bias, and variance of the resulting estimators for ${\varphi }_1(\mu ,\sigma )=\mu +\sigma$ and ${\varphi }_2(\mu ,\sigma )={\mu }^2+{\sigma }^2$ in Figure 2 and Figure 3, respectively, where Irr represents the LD with standard irreversible perturbation. From the results in the graphs, the AIRELD approach has superior performance compared to RELD, which can be attributed to the introduction of perturbations. Additionally, we can see that the AIRELD approach’s performance surpasses that of the standard irreversible LD approach, thanks to the utilization of adaptive perturbations. In Figure 4 and Figure 5, we also provide results under the scenario of stochastic gradients, showing a high consistency with outcomes obtained using true gradients.

4.2. Mixture of 25 Two-Dimensional Gaussian Distribution

In this experiment, we observed the early-stage performances of various algorithms. In the experiments, we set the the target distribution $\pi \left(\theta \right)$ as a mixture of 25 two-dimensional Gaussian probability density functions, defined as follows:

$$\pi \left({\theta }_1,{\theta }_2\right)=\sum _{i=1}^{25}{w}_i·1/\left(2\pi \kappa \right)·exp\left\{-1/\left(2\kappa \right)·\left({\left({\theta }_1-{\xi }_i^{\left(1\right)}\right)}^2+{\left({\theta }_2-{\xi }_i^{\left(2\right)}\right)}^2\right)\right\}.$$

Here, ${\xi }_i=\left({\xi }_i^{\left(1\right)},{\xi }_i^{\left(2\right)}\right)$ represents the center of the i-th Gaussian probability density function, and $w_i$ is the corresponding weight. It is worth noting that we set all the covariance matrices as $\kappa ·I_2$, where $I_2$ denotes the two-dimensional identity matrix, and $\kappa$ is a positive constant. In our experiments, we choose ${\xi }_i$ in a successive pattern, starting with $(0,0),(0,1),\dots ,(0,4),\dots ,(4,0),(4,1),\dots ,(4,4)$, with their corresponding weights set as $1/325,1/325,\dots ,25/325$. With these settings, it can be easily demonstrated that $U\left({\theta }_1,{\theta }_2\right)=-log\pi \left({\theta }_1,{\theta }_2\right)$ is non-convex and exhibits multiple local minima.

In this experiment, with stochastic gradients employed, we choose $\kappa =0.015$ and a fixed step size of $h={10}^{-4}$. For RESGLD and AIRESGLD, we incorporate two temperatures: a low temperature of ${\tau }_1=1$ and a high temperature of ${\tau }_2=3$ with a swapping intensity of 1. The target multi-modal density is shown in Figure 6a. Figure 6b–f displays the early-stage empirical performance of all the sampling methods with $T={10}^5$ steps. AIRESGLD still maintains relatively stable local sampling capabilities with significantly increasing sampling exploration capabilities. AISGPLD demonstrates superior empirical performance compared to RESGLD and SGLD with standard irreversible perturbations (irr-small SGLD with $J={\tau }_1\times \left[\begin{array}{cc}0& 1\\ -1& 0\end{array}\right]$ and irr-large SGLD with $J={\tau }_2\times \left[\begin{array}{cc}0& 1\\ -1& 0\end{array}\right]$). In Figure 7, we illustrate the KL divergence variations across various sampling methods. Our method, AIRESGLD, demonstrates a marked improvement in sampling performance compared to RESGLD and SGLD with standard irreversible perturbation, evident from the fluctuations in the KL divergence.

4.3. Simulations of Multi-Modal Distribution

This example is consistent with the one utilized in Deng et al. [4]. The target density function is given by $\pi \left(\theta \right)\propto exp(-U(\theta \left)\right)$, $U\left(\theta \right)=0.2\left({\theta }_1^2+{\theta }_2^2\right)-2\left(cos\left(2\pi {\theta }_1\right)+cos\left(2\pi {\theta }_2\right)\right)$. In this experiment, with stochastic gradients computed, we make comparisons between SGLD, irr-small SGLD, irr-large SGLD, RESGLD, and AIRESGLD. We opt for a constant step size of $h={10}^{-4}$. In RESGLD and AIRESGLD, we incorporate two temperatures: ${\tau }_1=1$ for the low temperature and ${\tau }_2=3$ for the high temperature, with a swapping intensity set at 1. For SGLD with standard irreversible perturbation, we set $J={\tau }_1\times \left[\begin{array}{cc}0& 1\\ -1& 0\end{array}\right]$ and $J={\tau }_2\times \left[\begin{array}{cc}0& 1\\ -1& 0\end{array}\right]$ for irr-small SGLD and irr-large SGLD, respectively. In this experiment, we have also shown the performance of contour stochastic gradient Langevin dynamics (CSGLD) [23], specifically designed for simulating multi-modal distributions. In Figure 8, the trajectories of various sampling algorithms are depicted. Relative to SGLD, irr-small SGLD, CSGLD, and irr-large SGLD, AIRESGLD display better global exploration capability. Additionally, when compared to RESGLD, the AIRESGLD approach demonstrates an improved capacity for global exploration. Simultaneously, the AIRESGLD approach offers more focused sampling within local regions, demonstrating superior local exploitation ability.

The target multi-modal density is depicted in Figure 9a. Figure 9b–f illustrate the empirical performance of all the sampling algorithms. Based on the sampling results, the AIRESGLD approach demonstrates the most favorable performance. In Figure 10, we illustrate how the KL divergence changes across sampling iterations. The AIRESGLD approach notably minimizes the KL divergence early on, leveraging its robust global exploration capability. Additionally, thanks to its local exploitation ability, the algorithm consistently converges to the target distribution.

4.4. Bayesian Logistic Regression

Next, we consider a binary regression model, wherein $\mathbf{y}={\left\{y_i\right\}}_{i=1}^N$ denotes a vector containing N binary responses, and $\mathbf{X}$ represents an $N\times d$ matrix of covariates. $\theta$ stands as a d-dimensional vector representing model parameters, the likelihood function for the logistic regression model takes the form:

$$p(\mathbf{y},\mathbf{X}\mid \theta )=\prod _{i=1}^N{\left[\frac{1}{1+exp\left(-{\theta }^{\top }{\mathbf{x}}_i\right)}\right]}^{y_i}{\left[1-\frac{1}{1+exp\left(-{\theta }^{\top }{\mathbf{x}}_i\right)}\right]}^{1-y_i},$$

(15)

where ${\mathbf{x}}_i$ represents a d-dimensional vector corresponding to the ith observation. The prior distribution for $\theta$ assumes a zero-mean Gaussian distribution with a covariance matrix ${\Sigma }_{\theta }=10{\mathbf{I}}_d$, where ${\mathbf{I}}_d$ denotes a $d\times d$ identity matrix.

We simulate $N=$${10}^3$ data points with ${\mathbf{x}}_i\sim N\left(0,{\Sigma }_{\mathbf{x}}\right)$, where ${\Sigma }_{\mathbf{x}}^{(i,j)}=$ U${[-\rho ,\rho ]}^{|i-j|}$ with $\rho =0.4$. After obtaining ${\mathbf{x}}_i$, we obtain ${\mathbf{y}}_i$ by randomly generating $\theta$ and using a binomial distribution with probability $1/\left(1+exp\left(-{\theta }^{\top }{\mathbf{x}}_i\right)\right)$. We partition the dataset into a training set and a test set, with the training set comprising 800 data points and the test set comprising 200 data points. Additionally, we employ the log-loss as the metric for assessing the predictive accuracy of the classifier on test dataset $D_{*}$. In this case, the log-loss is expressed as

$$\begin{array}{cc}\hfill l\left(\theta ,D_{*}\right)=& -\frac{1}{\left|T_{*}\right|}\sum _{\left(y_{*},{\mathbf{x}}_{*}\right)\in D_{*}}{y}_{*}logp\left({\mathbf{x}}_{*},\theta \right)\hfill \\ \hfill & +\left(1-y_{*}\right)log\left(1-p\left({\mathbf{x}}_{*},\theta \right)\right),\hfill \end{array}$$

(16)

where $p\left({\mathbf{x}}_{*},\theta \right)={\left(1+exp\left(-{\theta }^{\top }{\mathbf{x}}_{*}\right)\right)}^{-1}$. In this experiment, we compare the performance of SGLD, irr-small SGLD, irr-large SGLD, RESGLD, and AIRESGLD using log-loss. We fix the parameter dimension of $\theta$ at 4. Additionally, we choose a constant step size of $h={10}^{-4}$. In RESGLD and AIRESGLD, we incorporate two temperatures: ${\tau }_1=1$ for the low temperature and ${\tau }_2=3$ for the high temperature, with a swapping intensity set to 1. We create the skew-symmetric matrix $J_{4\times 4}$ by forming a lower triangular matrix with entries randomly drawn from $\{1,-1\}$ and then subtracting its transpose. For SGLD with standard irreversible perturbation, we set $J_1={\tau }_1\times J_{4\times 4}$ and $J_2={\tau }_2\times J_{4\times 4}$ for irr-small SGLD and irr-large SGLD, respectively.

The outcomes are visualized in Figure 11. In the logistic regression scenario, there is a noticeable potential improvement in the predictive performance of AIRESGLD. Specifically, among the conventional methods including SGLD, irr-small SGLD, irr-large SGLD, and RESGLD, The AIRESGLD approach exhibits the most favorable predictive performance, consistent with the apparent trade-off between exploitation and exploration.

5. Conclusions

The primary contribution of this paper lies in the design of the AIRELD algorithm tailored for simulating multi-modal distributions. This algorithm capitalizes on the insight that constant irreversible perturbations can aid LD in escaping local high-density regions within the target distribution. Simultaneously, by incorporating the swapping mechanism from the RELD approach, which mitigates the divergence issue encountered when artificially modifying the magnitude of irreversible perturbations, the AIRELD approach achieves adaptively changed perturbations. Consequently, the AIRELD approach harnesses the profound exploration capabilities of large constant irreversible perturbations, facilitating frequent transitions between distinct peaks in the target density. Conversely, as the dynamic process transitions to a new peak, small constant irreversible perturbations equip it with local exploitation abilities, enhancing its sampling efficiency in the vicinity of these local regions. This adaptive and versatile approach provides a robust solution for effectively sampling multi-modal distributions.

Across multiple numerical experiments, we have validated that adaptive irreversible perturbations do improve the performance of LD-based sampling algorithms. Our proposed algorithm, AIRELD, leverages the advantages of RELD and LD with standard irreversible perturbations. In all experiments, we compared the performance of AIRELD against RELD and LD with standard irreversible perturbations in terms of empirical sampling results, MSE, and KL divergence. The achieved results indicate that our proposed AIRELD algorithm demonstrates notable improvements compared to the other two methods. In Section 4.1, we calculate both real gradients and stochastic gradients. Then, in Section 4.2, Section 4.3 and Section 4.4, we continue with stochastic gradient computations. These findings highlight the applicability of using stochastic gradients alongside adaptive irreversible perturbations for practical computational purposes. Our numerical results affirm the convergence of AIRELD to the target distribution. Theoretically, a constant irreversible perturbation J can maintain the system’s invariant distribution [17]. Additionally, the swapping rule in the RELD approach [1] does not alter the invariant distribution of the dynamics. Hence, the AIRELD approach converges to the target distribution.

Future work can explore theoretical characterizations of the performance of adaptive irreversible perturbations and compare them with other approaches. Additionally, from a computational perspective, investigating parallel multi-chain AIRELD will be an interesting avenue for future research.