Adaptive MCMC for Bayesian Variable Selection in Generalised Linear Models and Survival Models

Abstract

Developing an efficient computational scheme for high-dimensional Bayesian variable selection in generalised linear models and survival models has always been a challenging problem due to the absence of closed-form solutions to the marginal likelihood. The Reversible Jump Markov Chain Monte Carlo (RJMCMC) approach can be employed to jointly sample models and coefficients, but the effective design of the trans-dimensional jumps of RJMCMC can be challenging, making it hard to implement. Alternatively, the marginal likelihood can be derived conditional on latent variables using a data-augmentation scheme (e.g., Pólya-gamma data augmentation for logistic regression) or using other estimation methods. However, suitable data-augmentation schemes are not available for every generalised linear model and survival model, and estimating the marginal likelihood using a Laplace approximation or a correlated pseudo-marginal method can be computationally expensive. In this paper, three main contributions are presented. Firstly, we present an extended Point-wise implementation of Adaptive Random Neighbourhood Informed proposal (PARNI) to efficiently sample models directly from the marginal posterior distributions of generalised linear models and survival models. Secondly, in light of the recently proposed approximate Laplace approximation, we describe an efficient and accurate estimation method for marginal likelihood that involves adaptive parameters. Additionally, we describe a new method to adapt the algorithmic tuning parameters of the PARNI proposal by replacing Rao-Blackwellised estimates with the combination of a warm-start estimate and the ergodic average. We present numerous numerical results from simulated data and eight high-dimensional genetic mapping data-sets to showcase the efficiency of the novel PARNI proposal compared with the baseline add–delete–swap proposal.

1. Introduction

Variable selection is an automatic method for finding a small subset of covariates that explain most of the variation in the response of interest. In addition to identifying the most predictive covariates, there is a growing interest in exploring the low-rank structure between the covariates and the response, especially in genetic mapping problems where the objective is to find the expressed genes that are associated with a specific disease the most. In the frequentist framework, model selection is based on maximising the penalised log-likelihood [1] or minimising information criteria such as AIC [2] and BIC [3]. Other approaches, such as the deviance information criterion (DIC) [4] and widely applicable information criterion (WAIC) [5], which are generalisations of the AIC, are also popular in model selection.

A natural alternative to these frequentist approaches is Bayesian variable selection (BVS). In the Bayesian approach, a prior is imposed on all candidate models, and the resulting posterior distribution naturally captures model uncertainty. In this work, we consider a spike-and-slab prior [6,7], which introduces indicator variables denoting the inclusion or exclusion of every covariate. Therefore, the spike-and-slab prior leads to a model posterior distribution that lies in a lattice with the same dimension as the number of covariates. We can understand the dependency between the importance of covariates and response using natural measures of the posterior distribution such as posterior model probability (PMP) and marginal posterior inclusion probability (PIP). The computation of the exact posterior distribution requires a full search over the whole model space, which is computationally infeasible when a high-dimensional data-set is analysed. In these settings, Markov Chain Monte Carlo (MCMC) algorithms are often used to explore the model space and estimate the posterior distribution. For “large n, large p” data-sets, which are now often encountered in some problems in genetics/genomics (such as genetic mapping studies), such algorithms must be carefully designed. In this work, we mainly consider Bayesian variable selection in generalised linear models and survival models and focus on three popular models: the logistic regression model [8,9], the Cox proportional hazards model with partial likelihood [10,11,12,13,14] and the Weibull regression model [15]. In each case, we illustrate how carefully designed algorithms can facilitate effective posterior computation.

A natural challenge of Bayesian variable selection methods in the above settings is that the marginal likelihood (or the integrated likelihood in [16]) is not analytically available. One set of solutions are Reversible Jump MCMC schemes (RJMCMC) [17], which sample from the joint space of models and regression coefficients by jointly proposing moves between models and regression coefficients. But it is often difficult to construct efficient proposals for these trans-dimensional jumps and design an MCMC scheme that mixes well [18]. For some specific models, data-augmentation methods [19] are available and result in closed-form marginal likelihood conditioned on latent variables, for instance, Pólya-gamma data augmentation [20] for logistic regression. For other models where no suitable data-augmentation scheme exists, the most popular approaches are the Laplace approximation and the correlated pseudo-marginal method [21], which rely on finding the maximum a posteriori (MAP) estimate of the regression coefficients. A novel scalable estimation method for marginal likelihood, approximate Laplace approximation (ALA), is introduced in [16] and relies on defining an initial value for the coefficient parameters. ALA can save computational time during the optimisation process of finding the MAP estimate, but it does not yield an asymptotically consistent estimate. A detailed discussion of these approaches will be given in Section 3.

Assuming that the marginal likelihood has been estimated, several MCMC algorithms can be used for simulation starting from the posterior distribution of BVS. The widely used add–delete–swap proposal [22] can be employed here. The add–delete–swap proposal generates a new model by randomly selecting one of three possible moves: addition/deletion of a covariate into/from the model or swapping one covariate that is included with another that is not. Although it has been proved in [23] that the add–delete–swap proposal can produce a rapidly mixing Markov Chain, the chains may still converge slowly, particularly when dealing with large-p problems. Adaptive MCMC schemes [24], which involve updating tuning parameters on the fly, are found to be valuable in addressing the issue of poor convergence. Lamnisos et al. [25] describe an adaptive add–delete–swap proposal that allows for simultaneous changes to multiple variables at a time. Griffin et al. [26] introduce the Adaptively Scaled Individual adaptation proposal (ASI), which simulates a new model with probability proportional to the product of PIPs. Wan and Griffin [27] extend the ASI proposal to logistic regression and accelerated failure time models. Other popular MCMC approaches include the Hamming ball sampler (HBS) [28], which proposes a new model within a Hamming neighbourhood using the PMPs as proposal weights, and the tempered Gibbs sampler [29,30], which uses tempering to efficiently sample from the multi-modal posteriors that commonly arise due to highly correlated covariates.

The recent work of [31] provides useful insights into the design of efficient MCMC schemes in discrete spaces. The work introduces the locally informed proposal, which re-weights a given non-informed base kernel with a function of the PMPs. It is shown in [31] that the locally informed proposal constructed with a balancing function that satisfies certain functional properties is asymptotically optimal compared with other choices of function in terms of Peskun ordering. Building upon the idea of locally informed proposal, Zhou et al. [32] show that the locally informed and thresholded (LIT) proposal can achieve dimension-free mixing times under conditions similar to those mentioned in [23] for BVS in linear regression models. Recent work in [33] introduces a Point-wise implementation of the Adaptive Random Neighbourhood Informed proposal (PARNI), which combines the advantages of both adaptive schemes and locally informed proposals. The PARNI proposal outperforms other state-of-the-art algorithms in a wide range of high-dimensional data-sets for BVS in linear regression models.

Other computational approaches are also available for estimating the BVS posterior distribution. Hans et al. [34] introduce a novel Shotgun Stochastic Search (SSS) approach that also explores the “local neighbourhood” idea and targets very high-dimensional model spaces to find high-probability regions. The integrated nested Laplace approximations (INLAs) [35] can solve latent Gaussian models including generalised linear models and approximate the posterior marginals obtained from the continuous priors [36]. Sara et al. [37] view survival models as latent Gaussian models and also approximate the posterior marginals using INLAs. The posterior distribution can also be approximated using Variational Bayes (VB) [38]. Ray et al. [39] describe a scalable mean-field variational family to approximate the posterior distribution of BVS in linear regression and extended this VB approximation to the logistic regression model in [40]. Komodromos et al. [41] apply the Sparse Variational Bayes (SVB) method to approximate the posterior of proportional hazards models with partial likelihood. Other works develop a sampling strategy based on simulating piece-wise deterministic Markov processes (PDMPs) [42,43], which directly target the posterior distribution obtained from a spike-and-slab prior.

In this paper, we extend the PARNI proposal to sampling from the BVS posterior distribution in generalised linear models and survival models. To avoid the overwhelming computational costs of approximating the marginal likelihood in the locally informed proposals and motivated by ALA [16], we introduce an ALA estimate of the marginal likelihood with a novel initial value. In contrast to the suggestion in [16], which initialises ALA at origin, the novel initial value is adaptively updated on the fly using previously sampled models. The new method is computationally less complex than the Laplace approximation or correlated pseudo-marginal scheme as a result of avoiding iterative optimisation and provides a more accurate estimate than the original ALA approach initialised at the origin. We also consider new approaches to adapt the tuning parameters in the PARNI proposal. The new adaptation scheme replaces the Rao-Blackwellised estimates of PIPs using the combination of a warm-start estimate and the ergodic average calculated using previously sampled models.

To illustrate the performance of the new PARNI scheme in real-life high-dimensional problems, we perform BVS on eight genetic mapping data-sets (four for the logistic regression model and four for survival analysis) and compare the output of the PARNI proposal with the add–delete–swap proposal as a baseline. For the logistic model with binary outcome, we consider the problem of finding expressed genes that are related the most to the presence of Systemic Lupus Erythematosus in a case-control study with 10,995 observations and various numbers of SNPs, from 5771 to 42,430, on four different chromosomes. In survival analysis, we consider four cancer-related data-sets (two for breast cancer and two for lung cancer), containing patients ranging from 130 to 1904 and genetic covariates varying from 662 to 54,675.

This paper is organised as follows: In Section 2, we review the model setup and prior specification for BVS in generalised linear models, Cox proportional hazards and Weibull survival models. In Section 3, we introduce four computational methods to estimate the marginal likelihood. Section 4 describes the PARNI proposal, highlighting the novelties in the adaption of algorithmic tuning parameters and the calculation of the accurate and efficient marginal likelihood estimates. We implement these MCMC algorithms in Section 5 and compare their performance with the add–delete–swap proposal on several real data-sets. We include a discussion in Section 6, highlighting some possible future research directions.

2. Bayesian Variable Selection for Generalised Linear Models and Survival Models

2.1. Generic Model Setting

Suppose that p covariates are available in the data. Let $X={(x_1,\cdots ,x_n)}^T\in {\mathbb{R}}^{n\times p}$ be the full data matrix that contains n observations with rows $x_i=(X_{i1},\cdots ,X_{ip})$ and let $Z={(z_1,\cdots ,z_n)}^T\in {\mathbb{R}}^{n\times q}$ be the full data matrix that contains q variables that must be included in every model. Let binary vector $\gamma =({\gamma }_1,\cdots ,{\gamma }_p)\in \mathrm{\Gamma }={\{0,1\}}^p$ be a model indicator, where ${\gamma }_j=1$ if the j-th variable is included in model ${\mathcal{M}}_{\gamma }$ and ${\gamma }_j=0$ otherwise.

Let $y=(y_1,\cdots ,y_n)$ be the vector of responses. The generalised linear model associated with model ${\mathcal{M}}_{\gamma }$ can be specified as

$$\begin{array}{c}\hfill y_i\sim \mathrm{F}({\mu }_{\gamma ,i},\varphi )\end{array}$$

(1)

where $\mathrm{F}(\mu ,\varphi )$ is a distribution that belongs to the exponential family with mean $\mu$ and dispersion $\varphi$. Linear predictor ${\eta }_{\gamma ,i}$ is defined as

$$\begin{array}{c}\hfill {\eta }_{\gamma ,i}=z_i^T\alpha +x_{\gamma ,i}^T{\beta }_{\gamma }\end{array}$$

(2)

where $x_{\gamma ,i}$ contains those variables j for which ${\gamma }_j=1$. In addition, we define the size of model ${\mathcal{M}}_{\gamma }$ as $p_{\gamma }={\sum }_{j=1}^p{\gamma }_j$. Linear predictor ${\eta }_{\gamma ,i}$ is mapped to mean ${\mu }_{\gamma ,i}$ using link function g as

$$\begin{array}{c}\hfill {\eta }_{\gamma ,i}=g\left({\mu }_{\gamma ,i}\right).\end{array}$$

(3)

We consider the following setup of survival models: For the i-th patient, given hazard function $h_i\left(t\right)$ at time t, the probability that an event occurs at time $T_i$ before a certain time $t_i$ can be written as

$$\begin{array}{cc}\hfill F_{T_i}\left(t\right)& =\mathbb{P}(T_i\le t_i)=1-S_{T_i}\left(t_i\right)\hfill \end{array}$$

where $S_{T_i}$ is called the survival function and is defined by

$$\begin{array}{c}\hfill S_{T_i}\left(t\right)=exp\left\{-{\int }_0^t{h}_i\left(u\right)du\right\}.\end{array}$$

Data often involve censoring where the true time to event is not observed. Let t be the vector of the observed times, where each $t_i$ denotes the minimum of censoring time $C_i$ and survival time $T_i$. In the case of “right-censored” data, we define an n-dimensional event indicator vector d to denote, for each patient i, whether the event was observed during their follow-up ($d_i=1$) or was censored ($d_i=0$). In the case where the event was observed for patient i ($d_i=1$), then $t_i$ denotes their time to event; otherwise, we observed the length of their follow-up.

Given a model ${\mathcal{M}}_{\gamma }$ associated with linear predictor ${\eta }_{\gamma ,i}$ as in (2), we consider the exponential hazard, ${\lambda }_{\gamma ,i}=exp\left({\eta }_{\gamma ,i}\right)$, and assume that the hazard function conditioned on model ${\mathcal{M}}_{\gamma }$ has the form

$$\begin{array}{c}\hfill h_{\gamma ,i}\left(t\right)=h(t,{\lambda }_{\gamma ,i},k)\end{array}$$

(4)

where k is an additional shape parameter if needed. We can conclude the following log-likelihood on $y=(t,d)$:

$$\begin{array}{c}\hfill logp\left(y\right|\alpha ,{\beta }_{\gamma },\gamma )=\sum _{i=1}^n{d}_{i}log\left(h_{\gamma ,i}\left(t_i\right)\right)-H_i\left(t_i\right)\end{array}$$

where $H_i$ is the cumulative hazard function for the i-th patient and is defined by

$$\begin{array}{c}\hfill H_i\left(t\right)={\int }_0^t-\frac{dlogS\left(t\right)}{dt}{|}_{t=u}du=-log{S}_{T_i}\left(t\right).\end{array}$$

2.2. Prior Elicitation

Recalling model indicator $\gamma \in {\{0,1\}}^p$, we consider the prior structure

$$\begin{array}{c}\hfill p(\alpha ,{\beta }_{\gamma },\varphi ,\gamma )\propto p\left(\alpha \right)p\left({\beta }_{\gamma }\right|\varphi ,\gamma )p\left(\varphi \right|\gamma )p\left(\gamma \right).\end{array}$$

(5)

For generalised linear models in which the dispersion parameter is known (e.g., the logistic regression model where $\varphi =1$) or some survival models that do not involve a dispersion parameter, the prior specification becomes

$$\begin{array}{c}\hfill p(\alpha ,{\beta }_{\gamma },\gamma )\propto p\left(\alpha \right)p\left({\beta }_{\gamma }\right|\gamma )p\left(\gamma \right),\end{array}$$

(6)

which is equivalent to treating $\varphi$ as a fixed parameter. In this work, we focus on the prior structure described in (6), and we assume that there is no additional dispersion parameter in the model.

We specify the following prior distribution for the coefficient parameters:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \alpha & \sim {\mathrm{N}}_q(0,{\sigma }_{\alpha }^2{I}_q)\hfill \\ \hfill {\beta }_{\gamma }|\gamma & \sim {\mathrm{N}}_{p_{\gamma }}(0,g{I}_{p_{\gamma }})\hfill \end{array}\end{array}$$

(7)

where g is a positive scale parameter, ${\sigma }_{\alpha }^2$ is the prior variance on the coefficients of the fixed covariates, $I_p$ denotes a $p\times p$ identity matrix and $p_{\gamma }={\sum }_{j=1}^p{\gamma }_j$ is the size of model ${\mathcal{M}}_{\gamma }$.

We consider the choice of model prior

$$\begin{array}{c}\hfill p\left(\gamma \right)=h^{p_{\gamma }}{(1-h)}^{p-p_{\gamma }}\end{array}$$

(8)

where hyper-parameter h denotes the prior inclusion probability for each variable.

It is possible to construct a fully Bayesian hierarchical model based on the prior specifications described above. We can impose the following hyper-priors on hyper-parameters g and h:

$$\begin{array}{cc}\hfill \sqrt{g}& \sim {\mathrm{C}}^{+}(0,1)\hfill \\ \hfill h& \sim \mathrm{Beta}\left(a,b\right)\hfill \end{array}$$

where ${\mathrm{C}}^{+}(0,1)$ denotes the standard half-Cauchy distribution and $\mathrm{Beta}(a,b)$ denotes the Beta distribution with parameters $a>0$ and $b>0$. The half-Cauchy hyper-prior is a generalisation of the horseshoe prior [44,45,46], employed on the global-scale parameter of a continuous mixture of normal priors, to BVS problems. Liang et al. [47] note that fixing g can lead to several paradoxes and problems of model mis-specification. For other possible choices of hyper-priors on g, see [47,48]. In the context of prior inclusion probability h, Ley et al. [49] advise against using a fixed h in the absence of strong prior knowledge about the number of important variables. Kohn et al. [50] propose a Beta-binomial model prior in which hyper-parameter h can be integrated out analytically, leading to

$$\begin{array}{c}\hfill p\left(\gamma \right)=\frac{\mathrm{B}\left(a+p_{\gamma },b+p-p_{\gamma }\right)}{\mathrm{B}\left(a,b\right)}\end{array}$$

where $\mathrm{B}(·,·)$ denotes the Beta function.

2.3. Logistic Regression

Assume that $y_i\in \{0,1\}$, with $y_i=1$ indicating the success of an event and $y_i=0$ indicating failure. Logistic regression links the proportion of successes to the linear predictor with a logistic link function g as

$$\begin{array}{c}\hfill {\eta }_{\gamma ,i}=log\left(\frac{{\mu }_{\gamma ,i}}{1-{\mu }_{\gamma ,i}}\right)=z_i^T\alpha +x_{\gamma ,i}^T{\beta }_{\gamma },i=1,\cdots ,n,\end{array}$$

(9)

and the response variable is modelled as $y_i\sim \mathrm{Bern}\left({\mu }_{\gamma ,i}\right)$ under model ${\mathcal{M}}_{\gamma }$.

2.4. Cox Proportional Hazards (PHs) with Partial Likelihood

Starting with exponential hazard function ${\lambda }_{\gamma ,i}=exp\left({\eta }_{\gamma ,i}\right)$ associated with model ${\mathcal{M}}_{\gamma }$, in the Cox proportional hazard function, the hazards are assumed to have the form

$$\begin{array}{c}\hfill h_i\left(t\right)=h_0\left(t\right){\lambda }_{\gamma ,i}\end{array}$$

(10)

where $h_0$ is some baseline hazard function. In this proportional model, all covariate effects are assumed to be multiplicative. The full likelihood is then given as

$$\begin{array}{c}\hfill L(\alpha ,{\beta }_{\gamma },H_0|y,\gamma )\propto \prod _{j=1}^n{\left(exp\left({\eta }_{\gamma ,i}\right)H_0^{\prime }\left(t_i\right)\right)}^{d_i}exp\{-exp\left({\eta }_{\gamma ,i}\right)H_0\left(t_i\right)\}\end{array}$$

(11)

where $H_0\left(t\right)={\int }_0^t{h}_0\left(u\right)du$ is the cumulative baseline hazard function. If we model the prior of $H_0$ with a prior process $p\left(H_0\right)$ on the cumulative hazard function, the resulting posterior distribution of $\alpha$ and ${\beta }_{\gamma }$ is

$$\begin{array}{c}\hfill p(\alpha ,{\beta }_{\gamma }|y,\gamma )\propto \int L(\alpha ,{\beta }_{\gamma },H_0|y,\gamma )\times p\left(\alpha \right)p\left({\beta }_{\gamma }\right|\gamma )p\left(H_0\right)d{H}_0.\end{array}$$

(12)

Alternatively, we can take the partial likelihood of Cox, which is given by

$$\begin{array}{c}\hfill \mathrm{PL}(\alpha ,{\beta }_{\gamma }|y,\gamma )\propto \prod _{i=1}^n{\left\{\frac{exp\left({\eta }_{\gamma ,i}\right)}{\sum _{s\in \mathcal{R}\left(t_i\right)}exp\left({\eta }_{\gamma ,s}\right)}\right\}}^{d_i}\end{array}$$

(13)

where $\mathcal{R}\left(t\right)=\{i:t_i\ge t\}$ is the set of patients at risk at time t. Unlike the full likelihood formulated in (11), partial likelihood does not rely on the specification and estimation of baseline hazard function $h_0$. Partial likelihood and its variants are, therefore, popular alternatives to full likelihood in many survival studies [11,14,51]. It is highlighted in [52,53] that partial likelihood can be obtained by integrating out the baseline hazard function using a Gamma process prior. Bayesian inference with partial likelihood (13) relies on approximate posterior $p_{\mathrm{PL}}(\alpha ,{\beta }_{\gamma }|y,\gamma )$, which can be expressed as

$$\begin{array}{c}\hfill p_{\mathrm{PL}}(\alpha ,{\beta }_{\gamma }|y,\gamma )\propto \mathrm{PL}(\alpha ,{\beta }_{\gamma }|y,\gamma )\times p\left(\alpha \right)p\left({\beta }_{\gamma }\right|\gamma )\end{array}$$

(14)

where baseline hazard function $h_0$ is eliminated.

2.5. Weibull Regression

In addition to the semi-parametric approach of Cox PHs with partial likelihood, we consider another commonly used parametric model for survival analysis, namely, the Weibull model. A Weibull model is obtained by extending the exponential model by raising the survival rate to a positive power k, giving

$$\begin{array}{c}\hfill S_i\left(t\right)=exp\left\{-{\left(t{\lambda }_i\right)}^k\right\}.\end{array}$$

(15)

Parameter k is the shape parameter of a Weibull random variable. When $k<1$, the hazard rate decreases over time. Conversely, when $k>1$, the hazard rate increases over time. It is possible to recover the exponential survival model when $k=1$, and it represents a constant hazard rate over time.

We can derive the hazard function

$$\begin{array}{c}\hfill h_i\left(t\right)=-\frac{d}{dt}log\left(S_i\left(t\right)\right)={\lambda }_{i}k{\left({\lambda }_{i}t\right)}^{k-1}\end{array}$$

(16)

and the log-likelihood for parameters $\alpha$, ${\beta }_{\gamma }$ and k as

$$\begin{array}{c}\hfill log\left(L(\alpha ,{\beta }_{\gamma },k|y,\gamma )\right)=\sum _{i=1}^n{d}_i\left[log\left(k\right)+klog\left({\lambda }_i\right)+(k-1)log\left(t_i\right)\right]-{\left(t_i{\lambda }_i\right)}^k.\end{array}$$

(17)

It should be noted that the Weibull distribution does not belong to the exponential family, unless shape parameter k is assumed to be fixed. In the Bayesian framework, we consider the prior $p(log\left(k\right))=N(0,{\sigma }_k^2)$ for some positive prior variance ${\sigma }_k^2$ as in [15]. To perform MCMC, we alternatively update $\gamma |k$ using the PARNI proposal and $k|\gamma$ using an adaptive random walk proposal as described in Appendix B.

3. Computation of Marginal Likelihood $\mathit{p}\left(\mathit{y}\right|\mathit{\gamma })$

Let ${\theta }_{\gamma }=(\alpha ,{\beta }_{\gamma })$ be the collection of all coefficient parameters associated with model ${\mathcal{M}}_{\gamma }$. We are interested in simulating samples from the posterior distribution $\pi \left(\gamma \right)\propto p\left(y\right|\gamma \left)p\right(\gamma )$, where $p\left(y\right|\gamma )$ represents the marginal likelihood, given by

$$\begin{array}{c}\hfill p\left(y\right|\gamma )\propto \int p\left(y\right|{\theta }_{\gamma },\gamma )p\left({\theta }_{\gamma }\right|\gamma )d{\theta }_{\gamma }.\end{array}$$

(18)

In generalised linear models and survival analysis, a closed-form solution to (18) is typically not analytically available.

Assuming that an estimate of marginal likelihood $\widehat{p}\left(y\right|\gamma )$ can be obtained, we consider MCMC algorithms with random neighbourhood proposals as described in [33], which is a sub-class of Metropolis–Hastings (MH) schemes [54,55]. The random neighbourhood proposal consists of the following three stages:

• Around the current model, $\gamma$, randomly generate a neighbourhood $\mathcal{N}\sim p(·|\gamma )$.
• Propose a new model, ${\gamma }^{\prime }$, within random neighbourhood $\mathcal{N}$ according to $q_{\mathcal{N}}(\gamma ,·)$.
• Accept the new proposal, ${\gamma }^{\prime }$, with the MH acceptance probability

  $$\begin{array}{cc}\hfill \alpha (\gamma ,{\gamma }^{\prime })& =min\left\{1,\frac{\pi \left({\gamma }^{\prime }\right)p\left({\mathcal{N}}^{\prime }\right|{\gamma }^{\prime })q_{{\mathcal{N}}^{\prime }}({\gamma }^{\prime },\gamma )}{\pi \left(\gamma \right)p\left(\mathcal{N}\right|\gamma )q_{\mathcal{N}}(\gamma ,{\gamma }^{\prime })}\right\}\hfill \\ \hfill & =min\left\{1,\frac{\widehat{p}\left(y\right|{\gamma }^{\prime })p\left({\gamma }^{\prime }\right)p\left({\mathcal{N}}^{\prime }\right|{\gamma }^{\prime })q_{{\mathcal{N}}^{\prime }}({\gamma }^{\prime },\gamma )}{\widehat{p}\left(y\right|\gamma )p\left(\gamma \right)p\left(\mathcal{N}\right|\gamma )q_{\mathcal{N}}(\gamma ,{\gamma }^{\prime })}\right\}\hfill \end{array}$$

  (19)

  where ${\mathcal{N}}^{\prime }$ is the neighbourhood used in the reverse move of the MH scheme.

In this section, we will describe four methods commonly used to estimate marginal likelihood $p\left(y\right|\gamma )$: data augmentation, Laplace approximation, correlated pseudo-marginal and approximate Laplace approximation. Before introducing these methods, it is necessary to define the following terms for convenience. Let $J_{\gamma }$ be a $n\times (q+p_{\gamma })$ matrix which contains all necessary covariates for model ${\mathcal{M}}_{\gamma }$ and is given by $J_{\gamma }=\left(Z{X}_{\gamma }\right)$, and let $V_{\gamma }$ be the variance–covariance matrix of the prior distribution of ${\theta }_{\gamma }$, defined by

$$\begin{array}{c}\hfill V_{\gamma }=\left(\begin{array}{cc}{\sigma }_{\alpha }^2{I}_q& 0\\ 0& g{I}_{p_{\gamma }}\end{array}\right).\end{array}$$

(20)

3.1. Data Augmentation

The data-augmentation scheme [19] introduces latent variables $\omega$ into the model such that the posterior distribution of variables of interest becomes analytically tractable given $\omega$. The Pólya-gamma data-augmentation scheme [20] can be utilised for the logistic regression model to evaluate the marginal likelihood. Given real numbers $\psi \in \mathbb{R}$, $a>0$, $b>0$ and a set of latent variables $\omega =({\omega }_1,\cdots ,{\omega }_n)$, in which each individual ${\omega }_i$ follows a Pólya-gamma distribution $\mathrm{PG}(b,0)$, the application of Pólya-gamma data augmentation exploits the following identity:

$$\begin{array}{c}\hfill \frac{{(exp\left(\psi \right))}^a}{{(1+exp\left(\psi \right))}^b}=2^{-b}exp\left(\kappa \psi \right){\int }_0^{\infty }exp\left(-{\omega }_i{\psi }^2/2\right)p\left({\omega }_i\right)d{\omega }_i\end{array}$$

(21)

where $\kappa =a-b/2$. The above identity implies that the posterior distribution of the coefficients can be represented as a multivariate normal distribution:

$$\begin{array}{c}\hfill {\theta }_{\gamma }\sim \mathrm{N}\left({\mathrm{\Lambda }}_{\gamma }^{-1}\xi ,{\mathrm{\Lambda }}_{\gamma }^{-1}\right)\end{array}$$

(22)

where $\xi =J_{\gamma }^T\kappa$, ${\mathrm{\Lambda }}_{\gamma }=J_{\gamma }^{T}W{J}_{\gamma }+V_{\gamma }^{-1}$, $\kappa$ is an n-dimensional vector with entries ${\kappa }_i=y_i-1/2$ and W is a diagonal matrix with $\omega$ appearing along its diagonal. By integrating out coefficient ${\theta }_{\gamma }$, analytically conditioned on Pólya-gamma random variables $\omega$, we obtain the conditional marginal likelihood

$$\begin{array}{c}\hfill p\left(y\right|\gamma ,\omega )\propto |V_{\gamma }{|}^{-\frac{1}{2}}{\left|{\mathrm{\Lambda }}_{\gamma }\right|}^{-\frac{1}{2}}exp\left\{\frac{1}{2}{\xi }^T{\mathrm{\Lambda }}_{\gamma }^{-1}\xi \right\}.\end{array}$$

(23)

In each iteration of the MCMC algorithm, we update $\gamma$ and $\omega$ alternatively. To refresh $\omega$, we can perform a simulation directly from its posterior distribution, which also follows a Pólya-gamma distribution given by

$$\begin{array}{c}\hfill {\omega }_i\sim \mathrm{PG}(1,{\eta }_{\gamma ,i}).\end{array}$$

(24)

where linear predictor ${\eta }_{\gamma ,i}$ involves coefficient ${\theta }_{\gamma }$ simulated from (22). Efficient samples from the Pólya-gamma random variables can be simulated using the R package pgdraw (version 1.1) [56]. In addition, Zens et al. described the ultimate Pólya-gamma sampler [57] to address the slow mixing rate for categorical imbalanced data, as illustrated in [58].

In general, the data-augmentation schemes may not be applicable to all generalised linear models and survival models. Specifically, for the Cox proportional hazards with partial likelihood or the Weibull model, there is currently no suitable data augmentation to directly yield a parametric posterior distribution for the regression coefficients.

3.2. Laplace Approximation

Assuming a unimodal posterior distribution of the regression coefficients, the Laplace approximation estimates the marginal likelihood with a second-order Taylor approximation. This method leads to a Gaussian integral, with the solution of the marginal likelihood being given by

$$\begin{array}{c}\hfill p_{\mathrm{LA}}\left(y\right|\gamma )=p\left(y\right|{\widehat{\theta }}_{\gamma },\gamma )p\left({\widehat{\theta }}_{\gamma }\right|\gamma ){\left|{\widehat{H}}_{\gamma }\right|}^{-\frac{1}{2}}{\left(2\pi \right)}^{\frac{p_{{\theta }_{\gamma }}}{2}}\end{array}$$

(25)

where ${\widehat{\theta }}_{\gamma }$ is the posterior mode of ${\theta }_{\gamma }$ and $H_{\gamma }$ is the negated Hessian of $logp\left(y\right|{\theta }_{\gamma },\gamma )+logp\left({\theta }_{\gamma }\right|\gamma )$ evaluated at mode ${\widehat{\theta }}_{\gamma }$. Additionally, the Laplace approximation provides a normal approximation to the posterior distribution of coefficient ${\theta }_{\gamma }$ as

$$\begin{array}{c}\hfill {\pi }_{\mathrm{LA}}\left({\theta }_{\gamma }\right)={\mathrm{N}}_{p_{{\theta }_{\gamma }}}({\widehat{\theta }}_{\gamma },{\widehat{H}}_{\gamma }^{-1}).\end{array}$$

(26)

To incorporate the Laplace approximation in MH sampling, we replace marginal likelihood $p\left(y\right|\gamma )$ in (19) with the approximate $p_{\mathrm{LA}}\left(y\right|\gamma )$ as described above.

Laplace approximation has been shown to be asymptotically consistent for estimating Bayes factors [59] and Bayesian variable selection on generalised linear models [60]. In finite-sample problems, however, Laplace approximation introduces biases, so $p_{\mathrm{LA}}\left(y\right|\gamma )$ is not an unbiased estimate of true marginal likelihood $p\left(y\right|\gamma )$. The resulting MCMC scheme, which involves the step of Laplace approximation, targets a different distribution compared with the true posterior $\pi \left(\gamma \right)$. Instead, it targets the distribution ${\pi }_{\mathrm{LA}}\left(\gamma \right)\propto p_{\mathrm{LA}}\left(y\right|\gamma )p\left(\gamma \right)$.

3.3. Correlated Pseudo-Marginal Method

We can alternatively make use of normal approximation ${\pi }_{\mathrm{LA}}\left({\theta }_{\gamma }\right)$ to derive an importance sampling estimate of marginal likelihood $p\left(y\right|\gamma )$. This estimator is unbiased and given by

$$\begin{array}{c}\hfill \widehat{p}\left(\gamma \right|y)=\frac{1}{N}\sum _{i=1}^N\frac{p\left(y\right|{\theta }_{\gamma }^{\left(i\right)},\gamma )p\left({\theta }_{\gamma }^{\left(i\right)}\right|\gamma )}{{\pi }_{\mathrm{LA}}\left({\theta }_{\gamma }^{\left(i\right)}\right)}\end{array}$$

(27)

where ${\theta }_{\gamma }^{\left(1\right)},\cdots ,{\theta }_{\gamma }^{\left(N\right)}$ are N samples from ${\pi }_{\mathrm{LA}}\left({\theta }_{\gamma }\right)$. As in Laplace approximation, we can replace marginal likelihood $p\left(y\right|\gamma )$ in (19) with estimated marginal likelihood $\widehat{p}\left(y\right|\gamma )$. This leads to the pseudo-marginal scheme in [61,62]. Andrieu and Roberts [62] show that the resulting Markov Chain preserves $\pi$-reversibility as long as estimated marginal likelihood $\widehat{p}\left(y\right|\gamma )$ is an unbiased estimator of the true marginal likelihood, $p\left(y\right|\gamma )$.

It is possible to extend a pseudo-marginal method to a correlated pseudo-marginal method [21], with the aim of reducing the estimation variance of the ratio of estimated marginal likelihoods $\widehat{p}\left(y\right|{\gamma }^{\prime })/\widehat{p}\left(y\right|\gamma )$. The correlated pseudo-marginal method is applied to Bayesian variable selection for the logistic regression model in [27], which provides an implementation that we also adopt in this work.

3.4. Approximate Laplace Approximation

The above Laplace approximation and correlated pseudo-marginal methods are computationally intensive due to the optimisation process required to obtain the normal approximation in (26), especially for dealing with large-n data. To avoid the overwhelming computational cost associated with the optimisation process, Rossell et al. [16] introduce the approximate Laplace approximation method (ALA), which is more computationally tractable for large-n problems. In this work, we consider the alternative formula described in supplementary material S.1. of [16], as it offers better computational stability when inverting the Hessian under the independent prior in (7).

In ALA, a Taylor expansion of log-posterior density $logp\left(y\right|{\theta }_{\gamma },\gamma )+logp\left({\theta }_{\gamma }\right|\gamma )$ is performed at initial value ${\theta }_{\gamma }^0$. Solving the resulting Gaussian integral leads to

$$\begin{array}{c}\hfill p_{\mathrm{ALA}}\left(y\right|\gamma )=p\left(y\right|{\theta }_{\gamma }^0,\gamma )p\left({\theta }_{\gamma }^0\right|\gamma ){\left(2\pi \right)}^{\frac{d}{2}}{\left|H_{\gamma }^0\right|}^{-\frac{1}{2}}exp\left\{\frac{1}{2}{g}_{\gamma }^{0T}{\left(H_{\gamma }^0\right)}^{-1}{g}_{\gamma }^0\right\}\end{array}$$

(28)

where $g_{\gamma }^0$ and $H_{\gamma }^0$ are the gradient and Hessian of the negative log-posterior density evaluated at ${\theta }_{\gamma }^0$, respectively. It is suggested in [16] to set initial value ${\theta }_{\gamma }^0$ to ${\theta }_{\gamma }^0=0$ for convenience.

By applying ALA to the MH acceptance probability in (19), we obtain an MCMC algorithm that targets the ALA posterior distribution ${\pi }_{\mathrm{ALA}}\left(\gamma \right)\propto p_{\mathrm{ALA}}\left(y\right|\gamma )p\left(\gamma \right)$ as the equilibrium distribution. Although Ref. [16] shows that ALA can recover the optimal model with respect to a mean squared loss, it is important to note that ALA is not consistent with respect to the marginal likelihood (in contrast to the classical Laplace approximation) and $p_{\mathrm{ALA}}\left(y\right|\gamma )$ is not an unbiased estimator of the true marginal likelihood, $p\left(y\right|\gamma )$.

4. Point-Wise Implementation of Adaptive Random Neighbourhood Informed Proposal

4.1. The PARNI Proposal

The PARNI proposal belongs to the class of random neighbourhood informed proposals, which typically involve the following two steps: (i) sampling a neighbourhood $\mathcal{N}\sim p(·|\gamma )$ and then (ii) proposing a model ${\gamma }^{\prime }$ within this neighbourhood $\mathcal{N}$ according to the informed proposal of [31]. In the PARNI proposal, we assume that the randomness in neighbourhood generation is characterised by an auxiliary variable $k\in \mathcal{K}$, with conditional distribution $p\left(k\right|\gamma )$, which leads to neighbourhood $\mathcal{N}=\mathcal{N}(\gamma ,k)$, such that $p\left(k\right|\gamma )=p(\mathcal{N}\left|\gamma \right)$. By defining $\mathcal{K}={\{0,1\}}^p$, the value of k indicates whether the change in the corresponding position in $\gamma$ is included in neighbourhood $\mathcal{N}$. Specifically, for those positions j such that $k_j=1$, the neighbourhood consists of models obtained by varying some or all of these positions in the current model, $\gamma$.

The conditional distribution of k takes the product form $p\left(k\right|\gamma )={\prod }_{j=1}^{p}p\left(k_j\right|{\gamma }_j)$, where each $k_j$ depends on the corresponding component ${\gamma }_j$ in $\gamma$. This probability distribution is driven by a set of tuning parameters $(A_1,\cdots ,A_p,D_1,\cdots ,D_p)$, where $A_j,D_j\in (ϵ,1-ϵ)$ for a small value of $ϵ\in (0,1/2)$. The probabilities of event $k_j=1$ are then defined by

$$\begin{array}{c}\hfill p(k_j=1|{\gamma }_j=0)=A_j,p(k_j=1|{\gamma }_j=1)=D_j,\end{array}$$

(29)

and the consequent neighbourhood is constructed as

$$\begin{array}{c}\hfill \mathcal{N}(\gamma ,k)=\left\{{\gamma }^{*}\in \mathrm{\Gamma }\mid {\gamma }_j^{*}={\gamma }_j\forall j\mathrm{s}.\mathrm{t}.k_j=0\right\}.\end{array}$$

(30)

Neighbourhood $\mathcal{N}(\gamma ,k)$ contains $2^{p_k}$, models where $p_k$ denotes the number of 1s in k. Performing a full enumeration over the entire neighbourhood is, therefore, computationally expensive when $p_k$ is large. In fact, it becomes computationally infeasible to explore the whole neighbourhood when $p_k$ is beyond 30. Liang et al. [33], therefore, consider a point-wise approximate implementation of this algorithm that dramatically reduces the number of model probability evaluations from $\mathcal{O}\left(2^{p_k}\right)$ to $\mathcal{O}\left(2p_k\right)$.

The point-wise implementation proceeds by constructing a sequence of smaller neighbourhoods $\left\{{\mathcal{N}}_r\right\}$ such that each ${\mathcal{N}}_r$ is a subset of $\mathcal{N}$. A proposed model ${\gamma }^{\prime }$ is sequentially simulated from these neighbourhoods $\left\{{\mathcal{N}}_r\right\}$ according to locally informed proposals $q_{{\mathcal{N}}_r}$. This procedure requires us to define a sequence of intermediate models $\gamma =\gamma \left(0\right)\to \gamma \left(1\right)\to \cdots \to \gamma \left(p_k\right)={\gamma }^{\prime }$. We collect positions j such that $k_j=1$ and define them as $j_1,\cdots ,j_{p_k}$ (the order is random). Small neighbourhood ${\mathcal{N}}_r$ is then defined as follows:

$$\begin{array}{c}\hfill {\mathcal{N}}_r=\mathcal{N}(\gamma (r-1),j_r)=\left\{{\gamma }^{*}\in \mathrm{\Gamma }\mid {\gamma }_j^{*}=\gamma {(r-1)}_j\forall j\ne j_r\right\}.\end{array}$$

(31)

Each small neighbourhood ${\mathcal{N}}_r$ only consists of two models, $\gamma (r-1)$ and ${\gamma }^{*}$, which only differ with $\gamma (r-1)$ at position $j_r$. The resulting proposal mass function is

$$\begin{array}{c}\hfill q_k(\gamma ,{\gamma }^{\prime })=\prod _{r=1}^{p_k}{q}_{{\mathcal{N}}_r}(\gamma (r-1),\gamma \left(r\right))\end{array}$$

(32)

where $q_{\mathcal{N}}$ is the locally informed proposal over neighbourhood $\mathcal{N}$ and is defined by

$$\begin{array}{c}\hfill q_{\mathcal{N}}(\gamma ,{\gamma }^{\prime })\propto \left\{\begin{array}{cc}g\left(\frac{\pi \left({\gamma }^{\prime }\right)p\left(k\right|{\gamma }^{\prime })}{\pi \left(\gamma \right)p\left(k\right|\gamma )}\right){\left(\frac{\zeta }{1-\zeta }\right)}^{d_H(\gamma ,{\gamma }^{\prime })},\hfill & \mathrm{if}{\gamma }^{\prime }\in \mathcal{N}\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(33)

Tuning parameter $\zeta \in (ϵ,1-ϵ)$ denotes the non-informative jumping probability. Two different methods for adapting $\zeta$ are provided in [33]. One of the key factors influencing the performance of the informed proposal in (33) is the choice of weighting function g. Given a positive real number $x>0$, a balancing function is defined as a function g that satisfies the condition $g\left(x\right)=xg(1/x)$. The locally informed proposal constructed with a balancing function is locally optimal in terms of Peskun ordering under mild conditions [31]. For the comparisons between different balancing functions, see Supplement B.1.3 of [31]. In this work, we exclusively focus on the Hastings’ choice of balancing function given by $g_H\left(x\right)=min\left\{1,x\right\}$, as $g_H$ has demonstrated better empirical performance in many problems (e.g., [33]).

To construct a $\pi$-reversible chain in the MH scheme, we define a collection of neighbourhoods $\left\{{\mathcal{N}}_r^{\prime }\right\}$ for the reverse moves, where $\left\{{\mathcal{N}}_r^{\prime }\right\}$ are identical to $\left\{{\mathcal{N}}_r\right\}$ but with reverse order. For a more detailed explanation of the PARNI proposal, we refer to Section 4.2.1 of [33]. The MH acceptance probability of the PARNI proposal is given by

$$\begin{array}{cc}\hfill \alpha (\gamma ,{\gamma }^{\prime })& =min\left\{1,\frac{\pi \left({\gamma }^{\prime }\right)p\left(k\right|{\gamma }^{\prime })q_k({\gamma }^{\prime },\gamma )}{\pi \left(\gamma \right)p\left(k\right|\gamma )q_k(\gamma ,{\gamma }^{\prime })}\right\}\hfill \\ \hfill & =min\left\{1,\frac{\widehat{p}\left(y\right|{\gamma }^{\prime })p\left({\gamma }^{\prime }\right)p\left(k\right|{\gamma }^{\prime })q_k({\gamma }^{\prime },\gamma )}{\widehat{p}\left(y\right|\gamma )p\left(\gamma \right)p\left(k\right|\gamma )q_k(\gamma ,{\gamma }^{\prime })}\right\}.\hfill \end{array}$$

(34)

Remark 1. 

The concept of a neighbourhood is also used in other schemes designed to estimate discrete posterior distributions, including the Shotgun Stochastic Search (SSS) approach [34] and Hamming ball sampler (HBS) [28]. The SSS method works on the same neighbourhood as that constructed with the add–delete–swap proposal [22]. Given the current model, γ, SSS constructs a neighbourhood $\mathcal{N}\left(\gamma \right)$ that comprises three disjoint sub-neighbourhoods: ${\mathcal{N}}_a\left(\gamma \right)$, ${\mathcal{N}}_d\left(\gamma \right)$ and ${\mathcal{N}}_s\left(\gamma \right)$. The “addition” neighbourhood, ${\mathcal{N}}_a\left(\gamma \right)$, is formed by adding a covariate into the model, and similarly, the “deletion” neighbourhood, ${\mathcal{N}}_d\left(\gamma \right)$, is formed by deleting a covariate from the model. Lastly, ${\mathcal{N}}_s\left(\gamma \right)$ is obtained by swapping an included covariate with an excluded one. On the contrary, the HBS constructs neighbourhoods based on the Hamming ball, ${\mathcal{H}}_d\left(\gamma \right)$, consisting of models that differ from γ by at most d positions. The typical example is the 1-Hamming ball, denoted by ${\mathcal{H}}_1\left(\gamma \right)$. It is worth mentioning that the SSS and HBS approaches construct neighbourhoods with sizes of $(p_{\gamma }+1)p$ and p, respectively. By contrast, the PARNI proposal constructs neighbourhoods that are typically approximately of size $p_{{\gamma }^{*}}$, where ${\gamma }^{*}$ denotes the true underlying model. Assuming that the size of the true underlying model is much smaller than p, as is typical in many applications, $p_{{\gamma }^{*}}\ll p$, and PARNI exhibits a higher level of scalability in handling the large-p data in comparison to SSS and HBS.

In the remaining parts of this section, we will describe a novel scheme to estimate tuning parameters A and D, and a new method for efficiently estimating the marginal likelihood in the locally informed proposal of (33).

4.2. New Adaptation Scheme on Algorithmic Tuning Parameters

The performance of the PARNI proposal is largely dictated by the choice of algorithmic tuning parameters A and D. Griffin et al. [26] consider the informed proposal of the form

$$\begin{array}{c}\hfill A_j=min\left\{1,\frac{{\pi }_j}{1-{\pi }_j}\right\},D_j=min\left\{1,\frac{1-{\pi }_j}{{\pi }_j}\right\}\end{array}$$

(35)

where ${\pi }_j$ denotes the PIP for the j-th covariate and is defined by ${\pi }_j=\pi ({\gamma }_j=1)$. In their ASI scheme for BVS in the linear regression model, tuning parameters $\pi$ are adaptively updated based on a Rao-Blackwellised estimate of the PIP given in Equation (10) of [26]. Wan and Griffin [27] extend ASI to the logistic regression model, in which they derive Rao-Blackwellised estimates of PIPs conditioned on the Pólya-gamma latent variables. Generalising this method to other generalised linear models and survival models that lack a suitable data-augmentation scheme is challenging. As the analytic marginal likelihood is inaccessible, it becomes intractable to derive Rao-Blackwellised estimates of PIPs. As an alternative, a simple Monte Carlo average over the output ${\left\{{\gamma }^{\left(l\right)}\right\}}_{l=1}^L$ can be taken, where L is the current iteration number. This ergodic average is calculated as

$$\begin{array}{c}\hfill {\tilde{\pi }}_j^{\left(L\right)}=\frac{1}{L}\sum _{l=1}^L\mathbb{I}\left\{{\gamma }_j^{\left(l\right)}=1\right\}.\end{array}$$

(36)

The ergodic average tends to be broad and biased, and it often downweights the importance of highly correlated covariates. Using the ergodic average directly in the PARNI proposal, however, results in a feedback effect, wherein a poor ergodic average leads to inadequate exploration over the sample space, leading to a subsequent bad ergodic average. To combat this phenomenon, we consider the following composition of two measures: a “warm-start” approximation, ${\tilde{\pi }}^{\left(0\right)}$, and the ergodic average, ${\tilde{\pi }}^{\left(L\right)}$, obtained from the first L samples. This composite estimate is adaptively updated using the formula

$$\begin{array}{c}\hfill {\widehat{\pi }}_j^{\left(L\right)}={\varphi }_L{\tilde{\pi }}_j^{\left(0\right)}+(1-{\varphi }_L){\tilde{\pi }}_j^{\left(L\right)}\end{array}$$

(37)

where ${\left\{{\varphi }_l\right\}}_{l=1}^L$ is a set of weights that control the trade-off between the warm-start approximation and the ergodic average.

Warm-start approximation ${\tilde{\pi }}_j^{\left(0\right)}$ is computed in the following way: Given the initial model of the Markov Chain, ${\gamma }^{\left(0\right)}$, and two related models, ${\gamma }^{j\uparrow }=({\gamma }_j=1,{\gamma }_{-j}^{\left(0\right)})$ and ${\gamma }^{j\downarrow }=({\gamma }_j=0,{\gamma }_{-j}^{\left(0\right)})$, for the j-th component, the Rao-Blackwellised estimate of the j-th PIP at model ${\gamma }^{\left(0\right)}$ is given by

$$\begin{array}{cc}\hfill \mathbb{P}({\gamma }_j=1|{\gamma }_{-j}={\gamma }_{-j}^{\left(0\right)},y)& =\frac{\pi \left({\gamma }^{j\uparrow }\right)}{\pi \left({\gamma }^{j\uparrow }\right)+\pi \left({\gamma }^{j\downarrow }\right)}=\frac{\frac{p\left(y\right|{\gamma }^{j\uparrow })p\left({\gamma }^{j\uparrow }\right)}{p\left(y\right|{\gamma }^{j\downarrow })p\left({\gamma }^{j\downarrow }\right)}}{1+\frac{p\left(y\right|{\gamma }^{j\uparrow })p\left({\gamma }^{j\uparrow }\right)}{p\left(y\right|{\gamma }^{j\downarrow })p\left({\gamma }^{j\downarrow }\right)}}.\hfill \end{array}$$

We consider the ALA in (28) initialised at the origin to estimate the intractable Bayes factor, $p\left(y\right|{\gamma }^{j\uparrow })/p\left(y\right|{\gamma }^{j\downarrow })$, for including the j-th covariate. Let ${\eta }_i$ be the i-th linear predictor, ${\eta }_i^0$ be the i-th linear predictor evaluated at the origin (i.e., ${\eta }_i^0=0$), $X_j$ denote the j-th column of data matrix X, $\tilde{y}$ be a vector with i-th component equal to $\partial p\left(y\right|{\theta }_{\gamma },\gamma )/\partial {\eta }_i$ evaluated at ${\eta }_i={\eta }_i^0$ and W be a matrix such that $W_{il}=-{\partial }^{2}p\left(y\right|{\theta }_{\gamma },\gamma )/\partial {\eta }_i\partial {\eta }_l$ evaluated at ${\eta }_i={\eta }_i^0$ and ${\eta }_l={\eta }_l^0$. Thanks to the Schur complement, we can facilitate the computation of p Bayes factors as in [26,27]: when ${\gamma }^{\left(0\right)}={\gamma }^{j\downarrow }$,

$$\begin{array}{c}\hfill \frac{\tilde{p}\left(y\right|{\gamma }^{j\uparrow })}{\tilde{p}\left(y\right|{\gamma }^{j\downarrow })}=d_j^{\uparrow -\frac{1}{2}}{g}^{-\frac{1}{2}}exp\left\{\frac{1}{2d_j^{\uparrow }}({\tilde{y}}^T{X}_{\gamma }{\mathrm{\Lambda }}_{\gamma }^{-1}{X}_{\gamma }^{T}W{X}_j-{\tilde{y}}^T{X}_j)\right\}\end{array}$$

(38)

where ${\mathrm{\Lambda }}_{\gamma }=X_{\gamma }W{X}_{\gamma }+V_{\gamma }^{-1}$ and $d_j^{\uparrow }=X_j^{T}W{X}_j+1/g-X_j^{T}W{X}_{\gamma }{\mathrm{\Lambda }}_{\gamma }^{-1}{X}_{\gamma }^{T}W{X}_j$; when ${\gamma }^{\left(0\right)}={\gamma }^{j\uparrow }$

$$\begin{array}{c}\hfill \frac{\tilde{p}\left(y\right|{\gamma }^{j\uparrow })}{\tilde{p}\left(y\right|{\gamma }^{j\downarrow })}=d_j^{\downarrow -\frac{1}{2}}{g}^{-\frac{1}{2}}exp\left\{-\frac{1}{2}{d}_j^{\downarrow }{\left({\tilde{y}}^T{X}_{\gamma }{\left({\mathrm{\Lambda }}_{\gamma }^{-1}\right)}_{·,q+p_j}\right)}^2\right\}\end{array}$$

(39)

where $d_j^{\downarrow }=1/{\left({\mathrm{\Lambda }}_{\gamma }^{-1}\right)}_{q+p_j,q+p_j}$ and $p_j$ is the ordered position of the j-th variable. When working with data that involve a high level of collinearity, it is also possible to increase the number of the ALA Rao-Blackwellised estimates.

The last building block of adapting ${\widehat{\pi }}_j^{\left(L\right)}$ is defining weight ${\varphi }_l$. We employ a straightforward construction of ${\varphi }_l$ given by

$$\begin{array}{c}\hfill {\varphi }_l=\left\{\begin{array}{cc}1-\frac{1}{2}{\left(N_b-l+1\right)}^{-0.5}\hfill & \mathrm{if}l\le N_b\hfill \\ \frac{1}{2}{\left(l-N_b\right)}^{-0.5}\hfill & \mathrm{if}l>N_b.\hfill \end{array}\right.\end{array}$$

(40)

where $N_b$ denotes the length of the burn-in period. This choice results in a weight that exceeds 1/2 during the period of burn-in and drops below 1/2 afterwards. Consequently, the PARNI proposal initially relies on the warm-start approximation to explore the model space. As the chain converges to the high-probability region and the ergodic average stabilises, the PARNI proposal gradually uses more information from the ergodic average. After running for a longer time, the PARNI proposal completely relies on the ergodic average.

4.3. The Adaptive ALA Informed Proposal

In each iteration of the PARNI proposal, the locally informed proposal in (33) relies on computing the posterior model probabilities. Using the estimates from the Laplace approximation or correlated pseudo-marginal scheme in PARNI can be computationally intractable in “large-p, large-n” situations due to the use of an optimisation algorithm that most run many times in one iteration. It should be noted, however, that the model probabilities in the locally informed proposal do not need to precisely match the true posterior model probabilities, and the PARNI proposal can still generate samples that preserve $\pi$-reversibility as long as the correct (or proper estimate) $\pi$ is used in the MH acceptance probability of (34). In the locally informed proposal, one can incorporate the approximate Laplace approximation initialised at the origin to design the proposal distribution. In the MH acceptance probability, we can then use the estimates obtained from the Laplace approximation or correlated pseudo-marginal method. Based on empirical observations, however, this ALA informed proposal may not always mix well. One reason for this is the phenomenon of downweighting the model probabilities of non-null models in favour of the null model, resulting in an informed proposal that is less informative than the true likelihood. The simulated chain is, therefore, more likely to get stuck and becomes less effective in exploring the model space.

Alternatively, we can note that the ALA estimate coincides with the Laplace approximation when the initial value is chosen to be posterior mode ${\widehat{\theta }}_{\gamma }$ under model ${\mathcal{M}}_{\gamma }$. Therefore, the accuracy of the ALA estimate is crucially influenced by the choice of initial value ${\theta }_{\gamma }^0$. We employ the ALA informed proposal with an adaptive initial value for ALA (adaptive ALA), which aims to reduce the estimation errors and thus improve the overall performance of the MCMC algorithm.

For each model in the neighbourhood, the adaptive ALA starts with an initial guess of linear predictor $\eta$ and proceeds with the following steps:

• Calculate a “guess” estimate from linear predictor $\eta$:

  $$\begin{array}{c}\hfill {\theta }_{\gamma }^0={\left(J_{\gamma }^T{J}_{\gamma }\right)}^{-1}{J}_{\gamma }^T\eta .\end{array}$$

  (41)
• Perform one step of Newton’s method and obtain an updated estimate of the coefficient:

  $$\begin{array}{c}\hfill {\tilde{\theta }}_{\gamma }={\theta }_{\gamma }^0-{\left(H_{\gamma }^0\right)}^{-1}{g}_{\gamma }^0\end{array}$$

  (42)

  where $g_{\gamma }^0$ and $H_{\gamma }^0$ are the gradient and Hessian of the negated log-posterior density evaluated at ${\theta }_{\gamma }^0$, respectively.
• Use the ALA estimate in (28) with ${\tilde{\theta }}_{\gamma }$ as the initial value to estimate marginal likelihood $p\left(y\right|\gamma )$.

Practically speaking, we can skip step 1 with the matrix inverse operation in (41) and obtain ${\tilde{\theta }}_{\gamma }$ directly from the initial guess of linear predictor $\eta$. This simplification is followed by [63] and is given in Appendix C. This approach leads to a coherent computational scheme that is easy to implement. We adaptively update the initial-guess $\eta$ according to

$$\begin{array}{c}\hfill {\widehat{\eta }}_i^{\left(L\right)}=\frac{1}{L}\sum _{l=1}^L{\widehat{\eta }}_{{\gamma }^{\left(l\right)},i}\end{array}$$

(43)

where ${\widehat{\eta }}_{\gamma ,i}=J_{\gamma }{\widehat{\theta }}_{\gamma }$ is the “optimal” i-th linear predictor obtained from MAP estimate ${\widehat{\theta }}_{\gamma }$ under model ${\mathcal{M}}_{\gamma }$. By storing the MAP estimate obtained from the Laplace approximation or correlated pseudo-marginal scheme, we can compute the linear predictor without introducing additional computational costs.

In addition, we experimented with adapting coefficients $\widehat{\beta }$ from the posterior samples and using the coefficients of the covariates selected by $\gamma$ to navigate the ALA. This approach did not work well, however, because the posterior distribution of $\beta$ differs significantly from the posterior distribution of $\beta$ conditioned on model $\gamma$. In contrast, the linear predictors offer more stability, in the sense that they do not vary as much across different models.

Combining all of the above components, we have the PARNI proposal. The complete algorithm is outlined in Algorithm 1.

Algorithm 1 The algorithmic pseudo-code of the Point-wise Adaptive Random Neighbourhood Sampler with Informed proposal (PARNI)

Initialise the chain at ${\gamma }^{\left(0\right)}$ and compute ${\left\{{\tilde{\pi }}_j^{\left(0\right)}\right\}}_{j=1}^p$; for $i=1$ to $i=N$ do   Sample $k\sim p(·|{\gamma }^{(i-1)})$ as in (29);   Set $\gamma \left(0\right)={\gamma }^{(i-1)}$, $p_k={\sum }_{j=1}^p{k}_j$ and define $j_1,\cdots ,j_{p_k}$;   for $r=1$ to $r=p_k$ do     Construct ${\mathcal{N}}_r$ as in (31) and estimate $p\left(y\right|{\gamma }^{*})$ for all ${\gamma }^{*}\in {\mathcal{N}}_r$ as in Section 4.3;     Sample $\gamma \left(r\right)\sim q_{{\mathcal{N}}_r}(\gamma (r-1),·)$ as in (33);   end for   Set ${\gamma }^{\prime }=\gamma \left(p_k\right)$, estimate $p\left(y\right|{\gamma }^{\prime })$ by LA or CPM and sample $U\sim \mathrm{Unif}(0,1)$;   If $U<\alpha ({\gamma }^{(i-1)},{\gamma }^{\prime })$ as in (34), then ${\gamma }^{\left(i\right)}={\gamma }^{\prime }$, else ${\gamma }^{\left(i\right)}={\gamma }^{(i-1)}$;   for $j=1$ to $j=p$ do     Update ${\tilde{\pi }}_j^{\left(i\right)}$ as in (36) and ${\widehat{\pi }}_j^{\left(i\right)}$ as in (37);     Update $A_j^{\left(i\right)}=min\left\{1,{\widehat{\pi }}_j^{\left(i\right)}/(1-{\widehat{\pi }}_j^{\left(i\right)})\right\}$;     Update $D_j^{\left(i\right)}=min\left\{1,(1-{\widehat{\pi }}_j^{\left(i\right)})/{\widehat{\pi }}_j^{\left(i\right)}\right\}$;   end for   Update ${\omega }^{\left(i\right)}$ using the selected adaption scheme and ${\widehat{\eta }}^{\left(i\right)}$ as in (43); end for

5. Experiments

5.1. Simulated Data-Sets with Adaptive ALA Informed Proposal

In this subsection, we study the mixing behaviour of different versions of the PARNI proposal for the logistic regression model, Cox PHs and Weibull survival models. We simulate two data-sets with 500 covariates and 500 observations as described in Appendix D and compare the following four algorithms:

• PARNI-adaptiveALA: The PARNI proposal with adaptive approximate Laplace approximation in the informed proposal and the correlated pseudo-marginal scheme in the MH acceptance probability.
• PARNI-LA: The PARNI proposal with Laplace approximation in the informed proposal and the correlated pseudo-marginal scheme in the MH acceptance probability.
• PARNI-ALA: The PARNI proposal with approximate Laplace approximation in the informed proposal and the correlated pseudo-marginal scheme in the MH acceptance probability.
• ADS (thinned): The PARNI proposal with approximate Laplace approximation in the informed proposal and the correlated pseudo-marginal scheme in the MH acceptance probability.

The first three algorithms were run for 10,000 iterations, with the first 2000 iterations being discarded as burn-in, whereas the ADS (thinned) proposal was run for a CPU time similar to that of PARNI-adaptiveALA and PARNI-ALA, with all collected samples being thinned to 10,000 values.

Figure 1 presents trace plots of the log-posterior model probability and bar plots of CPU time for the PARNI-adaptiveALA, PARNI-LA, PARNI-ALA and ADS (thinned) proposals in the logistic model, and the Cox PHs and Weibull models. In all three models, the PARNI-adaptiveALA proposal mixes as well as the PARNI-LA proposal and performs much better than the PARNI-ALA proposal. The result of the ADS (thinned) proposal provides the benchmark performance of a simple add–delete–swap MCMC scheme on these data-sets for comparison purposes. As illustrated in Figure 1, the adaptive ALA informed proposal is computationally much cheaper than the LA informed proposal. In comparison to the ALA informed proposal, the adaptive ALA informed proposal is also computationally competitive, and it only introduces the additional computational costs of updating linear predictor ${\widehat{\eta }}^{\left(L\right)}$ and computing initial value ${\tilde{\theta }}_{\gamma }$ from the estimate of linear predictor ${\widehat{\eta }}^{\left(L\right)}$ as in (42). In addition, the PARNI-adaptiveALA proposal demonstrates improved mixing behaviour in comparison to the baseline add–delete–swap proposal in all three models with similar CPU time. Therefore, we conclude that the PARNI-adaptiveALA proposal is more computationally efficient than the informed proposals constructed using Laplace approximation or ALA initialised at the origin.

5.2. Logistic Regression: Genetic Mapping for Systemic Lupus Erythematosus

Genetic mapping is a process of locating a specific gene or genetic variant within a particular genomic region and has the objective to find the precise genetic elements responsible for a particular trait or disease phenotype. One common application is to study whether an individual has a particular disease. In this scenario, one can use a logistic regression model with the response variable based on the case/control design and explanatory variables consisting of single-nucleotide polymorphisms (SNPs).

We consider a problem of identifying the SNPs that play a crucial role in predicting Systemic Lupus Erythematosus using a case/control study. It consists of genotypes from a genome-wide genetic case/control association study involving 4035 cases and 6959 controls, where the cases are SLE patients and the controls are from a public repository of European ancestry. These data were previously studied in [64] using step-wise logistic regression in a meta-analysis. In Chapter 5 of [65], Griffin and Steel apply Bayesian variable selection to analyse these SLE data but only focus on exploring the relationship between disease and SNPs on Chromosome 1. In addition to their work, we extend the study by including a total of four chromosomes. We consider a different number of SNPs for each chromosome, with Chromosome 1 having 5771 SNPs, Chromosome 3 having 42,430 SNPs, Chromosome 11 having 32,290 SNPs and Chromosome 21 having 9306 SNPs. We consider the prior specification in Section 2.2 with hyper-parameter $g=1/4$, ${\sigma }_{\alpha }^2=1$ and assume the hyper-prior of $h\sim \mathrm{Beta}(1,(p-5)/5)$, where p denotes the number of SNPs. The full details of the data-set are provided in

Table 1. Details of Systemic Lupus Erythematosus data on Chromosomes 1, 3, 11 and 21.

Data-Set	Observations	Cases	Fixed Covariates	Genetic Covariates
Chromosome 1	10,995	4036	Gender, PC1–PC4	5771
Chromosome 3				42,430
Chromosome 11				32,290
Chromosome 21				9306

, including the five fixed covariates (gender and top four principal components of expressed genes) that are mandatory in all models.

We implement the following four algorithms:

• PARNI-DA: PARNI proposal with Pólya-gamma data augmentation in both the informed proposal and the MH acceptance probability.
• PARNI-CPM: PARNI proposal with adaptive ALA informed proposal and correlated pseudo-marginal method in the MH acceptance probability.
• ADS-DA: Add–delete–swap proposal with Pólya-gamma data augmentation in the MH acceptance probability.
• ADS-CPM: Add–delete–swap proposal with the correlated pseudo-marginal method in the MH acceptance probability.

These MCMC algorithms all simulate samples from the exact posterior distribution, $\pi$. We treat the ADS-DA proposal as the baseline to showcase the rapid mixing of the PARNI proposals. Each algorithm was run for 1 h with 10 repetitions, and we recorded the estimates of PIPs. Firstly, we calculated the mean squared errors of the estimates of p PIPs compared with the “gold standard” estimates taken from the PARNI-CPM proposal, which was run for roughly 12 h. Then, we took the average over p mean squared errors to obtain the average mean squared error (average MSE). To compare the computational efficiency of the PARNI proposals with the baseline ADS-DA proposal, we provide the relative efficiency (in brackets) as the ratio of their average MSE.

The average MSEs and relative efficiency values are presented in

Table 2. Systemic Lupus Erythematosus data: The average mean squared errors of the ADS-DA, ADS-CPM, PARNI-DA and PARNI-CPM proposals in estimating the posterior inclusion probabilities of all SNPs (smaller is better). The relative efficiency as the ratio of the average MSE between algorithm A and the ADS-DA proposal presented in brackets (larger is better). The best performance is presented in bold.

Data-Set	Algorithms
	ADS-DA	ADS-CPM	PARNI-DA	PARNI-CPM
Chromosome 1	$1.84\times {10}^{-5}\left(1\right)$	$4.29\times {10}^{-5}\left(0.43\right)$	$\mathbf{5}.\mathbf{14}\times {\mathbf{10}}^{-\mathbf{6}}(\mathbf{3}.\mathbf{58})$	$7.34\times {10}^{-6}\left(2.51\right)$
Chromosome 3	$2.01\times {10}^{-4}\left(1\right)$	$2.37\times {10}^{-4}\left(0.85\right)$	$8.76\times {10}^{-5}\left(2.30\right)$	$\mathbf{5}.\mathbf{11}\times {\mathbf{10}}^{-\mathbf{5}}(\mathbf{3}.\mathbf{94})$
Chromosome 11	$7.09\times {10}^{-5}\left(1\right)$	$1.07\times {10}^{-4}\left(0.66\right)$	$\mathbf{9}.\mathbf{73}\times {\mathbf{10}}^{-\mathbf{6}}(\mathbf{7}.\mathbf{29})$	$9.89\times {10}^{-6}\left(7.15\right)$
Chromosome 21	$1.18\times {10}^{-5}\left(1\right)$	$1.67\times {10}^{-5}\left(0.71\right)$	$\mathbf{1}.\mathbf{51}\times {\mathbf{10}}^{-\mathbf{7}}(\mathbf{78}.\mathbf{08})$	$1.79\times {10}^{-7}\left(65.93\right)$

. The PARNI proposals consistently outperform the ADS proposals in terms of the average MSE. The PARNI proposals show at least twofold improvements over the add–delete–swap proposal and lead to much larger improvements in most cases, such as in Chromosome 21, where the PARNI proposals perform 78 times better than the add–delete–swap proposal. On the other hand, both the PARNI-DA and ADS-DA proposals consistently result in smaller average MSEs compared with the PARNI-CPM and ADS-CPM proposals due to their computational advantages. Firstly, data augmentation can evaluate the conditional marginal likelihood without finding the posterior mode using iteratively re-weighted least squares. Secondly, the Pólya-gamma latent variables are drawn using the R package pgdraw (version 1.1) [56] implemented using Rcpp (version 1.0.10) [66].

5.3. Survival Analysis: Variable Selection for Five Large Cancer-Related Gene Expression Data-Sets

We consider a total of four cancer-related real data-sets, where the first two data-sets are for breast cancer and the remaining data-sets are for lung cancer. NKI Breast Cancer Data (https://data.world/deviramanan2016/nki-breast-cancer-data, accessed on 4 September 2023) contain patient info, treatment, survival time and the 1554 most varying genes of 272 breast cancer patients. These data were analysed in [67,68] with the aim of reducing the mortality rates from this disease. The METABRIC breast cancer data-set is derived from the Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) database. The METABRIC data-set was analysed in [69,70] and is publicly available in [71] (https://www.cbioportal.org/study/summary?id=brca_metabric, accessed on 4 September 2023). The data contain 1907 patients with the gene expression for 331 genes and mutations for 175 genes. Gene mutation variables are encoded as 1 if a mutation exists and 0 otherwise. For both data-sets, we include some clinical covariates, including the age of the patients and the stage of the cancer, as suggested by [72]. We also consider the treatment variables (such as chemotherapy and surgery type), which also influence survival time. The last two lung cancer data-sets, “GSE31210” and “GSE4573”, were previously studied in [72], and they are publicly available in the Gene Expression Omnibus repository [73]. See Figure 1 in [72] for the estimated survival functions of these three data-sets. We provide the full details of these four real data-sets in

Table 3. Details of 4 real data-sets for survival analysis.

Data-Set	Cancer Type	Observations	Events	Fixed Covariates	Genetic Covariates
NKI	Breast	272	77	Age, chemo, hormone, surgery, stage	1554
METABRIC	Breast	1903	622	Age, chemo, hormone, radio, surgery, stage	662
GSE31210	Lung	226	30	Age, gender, smoker, stage	54,675
GSE4573	Lung	130	63	Age, gender, stage	22,283

.

We consider two computational algorithms used in previous studies for logistic models, the PARNI-CPM and ADS-CPM proposals, as a data augmentation scheme is not available for the Cox PHs or Weibull model. We consider the hyper-prior of $h\sim \mathrm{Beta}(1,(p-5)/5)$, where p denotes the number of genetic covariates, and impose a half-Cauchy hyper-prior on $\sqrt{g}$, where a Gibbs update is taken on g conditioned on the model (see Appendix A for more details). In addition, we assume ${\sigma }_{\alpha }^2={10}^5$ and ${\sigma }_k^2={10}^5$ (only for the Weibull model).

The average MSEs and relative efficiency values of the PARNI-CPM and ADS-CPM proposals on these four survival data-sets are shown in

Table 4. Survival analysis data: The average mean squared errors of the ADS-CPM and PARNI-CPM proposals in estimating posterior inclusion probabilities of all genetic covariates (smaller is better). The relative efficiency as the ratio of the average MSE between algorithm A and the ADS-CPM proposal presented in brackets (larger is better). The best performance is presented in bold.

Data-Set	Cox PHs		Weibull Model
	ADS-CPM	PARNI-CPM	ADS-CPM	PARNI-CPM
NKI	$5.54\times {10}^{-4}\left(1\right)$	$\mathbf{3}.\mathbf{19}\times {\mathbf{10}}^{-\mathbf{4}}(\mathbf{1}.\mathbf{74})$	$3.21\times {10}^{-5}\left(1\right)$	$\mathbf{4}.\mathbf{33}\times {\mathbf{10}}^{-\mathbf{6}}(\mathbf{7}.\mathbf{40})$
METABRIC	$\mathbf{1}.\mathbf{27}\times {\mathbf{10}}^{-\mathbf{3}}\left(\mathbf{1}\right)$	$3.80\times {10}^{-3}\left(0.33\right)$	$1.36\times {10}^{-3}\left(1\right)$	$\mathbf{1}.\mathbf{73}\times {\mathbf{10}}^{-\mathbf{4}}(\mathbf{7}.\mathbf{89})$
GSE31210	$\mathbf{1}.\mathbf{26}\times {\mathbf{10}}^{-\mathbf{6}}\left(\mathbf{1}\right)$	$3.56\times {10}^{-6}\left(0.35\right)$	$3.91\times {10}^{-5}\left(1\right)$	$\mathbf{2}.\mathbf{16}\times {\mathbf{10}}^{-\mathbf{5}}(\mathbf{1}.\mathbf{81})$
GSE4573	$4.57\times {10}^{-5}\left(1\right)$	$\mathbf{2}.\mathbf{54}\times {\mathbf{10}}^{-\mathbf{5}}(\mathbf{1}.\mathbf{83})$	$3.56\times {10}^{-5}\left(1\right)$	$\mathbf{8}.\mathbf{37}\times {\mathbf{10}}^{-\mathbf{6}}(\mathbf{4}.\mathbf{26})$

. For the Weibull model, the PARNI proposal consistently exhibits better computational efficiency compared with add–delete–swap on all four data-sets. In the case of the NKI and METABRIC data-sets, which have a relatively small number of covariates, the PARNI-CPM proposal produces PIP estimates that are seven times more accurate compared with ADS-CPM. For high-dimensional data-sets, we can obtain PIP estimates from the PARNI-CPM proposal that are two times better than ADS-CPM. The lesser improvement observed in the high-dimensional examples can be attributed to the increasing number of unimportant covariates, where both algorithms are good at excluding these unimportant covariates from the models.

The Bayesian variable selection in the Cox PHs model with partial likelihood is generally more challenging compared with the Weibull model. The primary reason is that the inclusion of the non-parametric setup introduces additional complexities in evaluating the log-likelihood functions and its Hessian matrices. The PARNI-CPM proposal provides roughly two times better estimates on the NKI and GSE4573 data-sets compared with ADS-CPM. However, the ADS-CPM proposal shows better performance on the remaining two data-sets. In the “small-p, large-n” METABRIC data, the add–delete–swap proposal shows greater computational efficiency compared with the PARNI proposal, as the informed proposal needs to evaluate many computationally expensive Hessian matrices. In fact, the computation of evaluating the Hessian matrix scales with the order of $\mathcal{O}\left(n^2\right)$, in contrast with parametric models, where the computation of the Hessian matrix scales linearly with n. In the GSE31210 data with few strong signals, the posterior distribution on model space is relatively flat, and both algorithms have smaller average MSEs compared with the other data-sets. In particular, the add–delete–swap proposal can run for more iterations; it is, therefore, more computationally efficient compared with the PARNI-CPM proposal.

6. Discussion

In this work, we apply the PARNI proposal to Bayesian variable selection problems in generalised linear models and survival models. We find that the informed proposal obtained from the approximate Laplace approximation with our new adaptive initial point yields improved efficiency and accuracy in posterior sampling. We compare the performance of the PARNI proposal with the baseline add–delete–swap proposal in numerous “large-p, large-n” real-world data-sets, and the PARNI proposal with the correlated pseudo-marginal method provides PIP estimates with smaller mean squared errors than the add–delete–swap proposal in most of the problems. The numerical results from the Cox PHs also provide useful insights to improve the PARNI proposal in the future. Code to run the PARNI proposal on the logistic regression model, and the Cox PHs and Weibull models is available at https://github.com/XitongLiang/The-PARNI-scheme.git (accessed on 4 September 2023).

In addition to the three models described in the paper, the proposed technique can be extended to other generalised linear models and survival models. Two possible extensions are the Gamma generalised linear model [74] and various Bayesian non-parametric approaches to survival analysis [75]. It is still a challenging problem to reduce the computational cost of simulating samples when a data-set contains a large number of observations. As highlighted in [76], simple sub-sampling strategies may not lead to a substantial improvement in the computational efficiency of posterior sampling. It would be interesting, therefore, to design an efficient PARNI scheme specifically tailored for large-n data-sets.