Bayesian Logistic Regression Model for Sub-Areas

Abstract

Many population-based surveys have binary responses from a large number of individuals in each household within small areas. One example is the Nepal Living Standards Survey (NLSS II), in which health status binary data (good versus poor) for each individual from sampled households (sub-areas) are available in the sampled wards (small areas). To make an inference for the finite population proportion of individuals in each household, we use the sub-area logistic regression model with reliable auxiliary information. The contribution of this model is twofold. First, we extend an area-level model to a sub-area level model. Second, because there are numerous sub-areas, standard Markov chain Monte Carlo (MCMC) methods to find the joint posterior density are very time-consuming. Therefore, we provide a sampling-based method, the integrated nested normal approximation (INNA), which permits fast computation. Our main goal is to describe this hierarchical Bayesian logistic regression model and to show that the computation is much faster than the exact MCMC method and also reasonably accurate. The performance of our method is studied by using NLSS II data. Our model can borrow strength from both areas and sub-areas to obtain more efficient and precise estimates. The hierarchical structure of our model captures the variation in the binary data reasonably well.

1. Introduction

The Nepal Living Standard Survey (NLSS) II (see [1]) is a two-stage stratified sampling. A random sample of wards (areas) were selected from six strata and 12 households (sub-areas) were selected from each sampled ward. All individuals in each sampled household were interviewed. One interest is on health status, a binary variable. To make smooth estimates of the finite population proportion of the individuals with good health in each household, we focus on hierarchical Bayesian (HB) models with sub-area random effects to obtain reliable “indirect” estimates for numerous small areas or sub-areas. Most of the sample surveys are designed to provide reliable “direct” estimates of interests for large areas or domains (e.g., state level, national level). However, direct estimates are not reliable for areas or domains for which only small samples or no samples are available—see [2].

In many applications, some areas, e.g., states and wards, are sampled; in each sampled area, a sample of sub-areas, e.g., counties and households, is further selected. Ref. [3] proposed a one-fold hierarchical Bayesian logistic regression model and applied the model to NLSS II data. The main objective is to make an inference for the finite population proportion of individuals with a specific character for each area. However, the one-fold model ignores the sub-area level structure in the data. As an extension of [3], we are particularly interested in small area models that can capture the hierarchical structure of the NLSS II data in this paper. Although the one-fold basic models are very popular and in common use in producing reliable estimates, the hierarchical structure of the data and the consistency between the estimates for different levels may not hold. In particular, the sampling designs of many population-based surveys were two-stage stratified sampling as NLSS II. But if we use a one-fold unit level model to fit the data, the sub-area level effects will have been ignored. Ref. [4] studied the case that the data follow a normal model with a two-stage (three-stage) hierarchical structure, while the fitted model has a one-stage (two-stage) hierarchical structure using posterior predictive p-values. Ref. [5] discussed the ability to detect a three-stage model when a two-stage model is actually fitted.

Two-fold models are an important extension of basic small area models. Many authors have considered the problems and proposed these kinds of models. Much of the literature focuses on continuous data. Ref. [6] proposed a sub-area level model which provides model-based estimates that account for the hierarchical structure of data. Two-fold sub-area level models were studied by [2,7,8,9], and many others. This type of model is an area-level model which extends the Fay-Harriot model (see [10]) to the sub-area level. Two-fold nested error regression models were considered by [11,12]. On the other hand, some literature focus on the categorical data. Ref. [13] described a HB model to make an inference about the finite population proportion under two-stage clustering sampling. Ref. [14] extended the Beta-Binomial model to the two-fold model and used Gibbs sampling to obtain the posterior estimates. Ref. [15] showed that the two-fold Beta-Binomial model is preferable over the one-fold one if the data have a hierarchical structure. Ref. [16] extended [15] to accommodate heterogeneous correlations. They used a HB model to make a posterior inference about the finite population proportion of each area, accounting for intracluster correlations. Ref. [17] discussed the sub-area Beta-Binomial model and applied the model to estimate the finite population proportion of healthy individuals in each household covered by the NLSS II, assuming no covariate was available.

Bayesian logistic regression models with random effects are suitable for handling binary data with covariates. Ref. [18] discussed discrimination between the logit and the complementary log-log link functions by using the logistic regression model. Roberts, Ref. [19], discussed logistic regression for the sample survey data (not small area estimation). Ref. [20] showed how to accelerate the Gibbs sampler for a model with latent variables introduced earlier by [21] for Bayesian probit analysis. Ref. [22], discussed the logistic regression model by using the empirical Bayesian approach. Ref. [23] showed how to analyze binary data with covariates to maintain conjugacy for both the logistic and Poisson regression models. The analysis of binary data with covariates under nonignorable nonresponse was discussed by [24]. Ref. [3] proposed a hierarchical Bayesian logistic regression model for binary data in a small area estimation. This model is a unit level model without a sub-area effect. Our two-fold sub-area model is an extension of this logistic regression model. We add the sub-area level random effect into the model which can capture the hierarchical structure of the sampled data. At the same time, we add more hyper-parameters into the model, which make the inference more complicated. However, we propose an approximation method called the integrated nested normal approximation (INNA), which solves the difficulties.

The other side of our application is that there are numerous small areas (households and individuals) and MCMC methods, which involve complicated integrals, and cannot handle them efficiently. “Big data” are defined as data that are too big to comfortably process on a single machine [25]. The researchers considered consensus Monte Carlo methods that split the data across several machines. They proposed algorithms that perform distributed approximate Bayesian analyses in order to minimize the communication between computers. The parallel MCMC methods for non-Gaussian posterior distributions were discussed by [26]. Fortunately, in survey sampling, the design generally uses a stratification which is not artificial, and, in this case, consensus Monte Carlo may not be needed; it will be a good idea for a large stratum.

The integration involved in Bayesian inference is usually intractable, which is true for our logistic regression model. The approximation techniques are desired. The procedure we used to approximate the posterior density of the parameters of the logistic regression sub-area model, INNA, is similar to the integrated nested Laplace approximation (INLA) originally proposed by [27], but they are actually different. INLA is a quite popular algorithm and an alternative to MCMC for big data analysis if the joint posterior density is very complicated. It requires posterior modes, and, for numerous small areas, the computation of modes becomes time-consuming and challenging for the logistic regression model or any generalized linear mixed models. Yet, INLA has found many useful applications, such as in Poisson regression by [28], and in spatial point pattern data by [29]. We note that INLA can be problematic, especially for logistic and Poisson hierarchical regression models, even if the modes can be computed. Ref. [30], attempted to improve INLA using a copula-based correction, which adds complexity to INLA. Our approximation method, INNA, which does not require finding posterior modes, uses a sampling-based procedure accommodated by the multiplication rule of probability. Instead of finding the posterior modes, INNA finds the approximate modes in closed form, facilitated by the empirical logistic transform ([31]) and the second-order Taylor series approximation.

On the other hand, two-fold models can capture the heterogeneity between samples within not only areas but also sub-areas. Many model-based estimation techniques for the sampling variances have been considered in the literature, but most of them for the area-level model: see [32,33,34].

In Section 2, a full description of a sub-area HB logistic regression model is given. In Section 3, we describe the integrated nested normal approximation (INNA) computation method and some theoretical results are provided. The exact MCMC method is presented in Appendix A. The exact method refers to MCMC methods without further approximation. In Section 4, we apply the model to the NLSS II data to provide smoothed estimates of the household proportions of members in good health for both sampled and nonsampled households. Some comparisons between INNA and the exact method are presented. Finally, in Section 5, we make concluding remarks and discuss the future work.

2. Sub-Area Logistic Regression Model

In this section, we discuss the sub-area HB logistic regression model at the unit level. In the NLSS II data, we have binary data (good health versus poor health) for each individual within a household, and these households are within wards. The observations are available at the unit level and so is the reliable auxiliary information. However, the model and method we proposed for small areas and sub-areas is not only for this application on NLSS II data. It can be also applied to other population-based surveys with binary responses which contain small areas or/and sub-areas.

Suppose that there are L small areas (wards) in the finite population and that, within the $i^{\mathrm{th}}$ area, there are $N_i$ sub-areas (households). Within the $j^{\mathrm{th}}$ sub-area, there are $M_{ij}$ individuals. We assume that $\ell (<L)$ areas are sampled and a simple random sample of $n_i(<N_i)$ households is taken from the $i^{\mathrm{th}}$ area. All individuals in the sampled households are sampled. Here, we assume the survey weights are the same within all households in each area. Actually, the design is almost self-weighting.

Let $y_{ijk},k=1,\dots ,m_{ij},j=1,\dots ,n_i,i=1,\dots \dots ,\ell$ denote the binary responses. Let $\underset{\tilde{}}{y}={(y_{ijk},k=1,\dots ,m_{ij},j=1,\dots ,n_i,i=1,\dots \dots ,\ell )}^{\prime }$. Let $y_{ij}={\sum }_{k=1}^{m_{ij}}{y}_{ijk}$ be the number with response 1 and $m_{ij}$ be the total number of people who responded. Let ${\underset{\tilde{}}{x}}_{ijk}={(1,x_{ijk1},\dots ,x_{ijkp})}^{\prime }$ be the $(p+1)$ vector with p covariates for individuals and an intercept.

We use P to represent the population proportion and p as the sample proportion. Let $p_{ij}$ be the corresponding sample probability of $y_{ij}$, $j=1,\dots ,n_i,i=1,\dots ,\ell$.

The primary interests are the finite population proportions of the households, which are $P_{ij}=\frac{1}{M_{ij}}{\sum }_{k=1}^{M_{ij}}{y}_{ijk},j=1,\dots ,N_i,i=1,..\dots ,\ell$ and the finite population proportions of the areas, which are $P_i=\frac{1}{N_i}{\sum }_{j=1}^{N_i}{\sum }_{k=1}^{M_{ij}}{y}_{ijk},i=1,\dots \dots ,\ell .$

In the content of the logistic regression model, the two-fold hierarchical Bayesian logistic regression model for the sub-area means, ${\mu }_{ij}$, is

$$y_{ijk}|\underset{\tilde{}}{\beta },{\nu }_i,{\mu }_{ij}\stackrel{ind}{\sim }\mathrm{Bernoulli}\left\{\frac{e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }\underset{\tilde{}}{\beta }+{\nu }_i+{\mu }_{ij}}}{1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }\underset{\tilde{}}{\beta }+{\nu }_i+{\mu }_{ij}}}\right\},k=1,\dots ,m_{ij},$$

$${\mu }_{ij}|{\sigma }^2\stackrel{iid}{\sim }\mathrm{Normal}(0,{\sigma }^2),j=1,\dots ,n_i,$$

(1)

$${\nu }_i|{\delta }^2\stackrel{iid}{\sim }\mathrm{Normal}(0,{\delta }^2),i=1,\dots ,\ell ,$$

$$\pi (\underset{\tilde{}}{\beta },{\delta }^2,{\sigma }^2)\propto \frac{1}{{(1+{\delta }^2)}^2}\frac{1}{{(1+{\sigma }^2)}^2},{\delta }^2>0,{\sigma }^2>0.$$

Here, ${\mu }_{ij},i=1,\dots ,\ell ;j=1,\dots ,n_i$ are the sub-area level random effects, which are not in the area-level model in [3]. ${\nu }_i,i=1,\dots ,\ell$ are the area random effects and $\underset{\tilde{}}{\beta }={({\beta }_0,{\beta }_1,\cdots ,{\beta }_p)}^{\prime }$ are the regression coefficients, with ${\sigma }^2,{\delta }^2$ as the variance of the random effects, respectively.

In order to apply our approximation method and make an inference for posterior distribution, we use an equivalent model.

First, we separate $\underset{\tilde{}}{\beta }$ into ${\beta }_0$ and ${\underset{\tilde{}}{\beta }}_{\left(0\right)}$, where ${\underset{\tilde{}}{\beta }}_{\left(0\right)}={({\beta }_1,{\beta }_2,\dots ,{\beta }_p)}^T$. We set ${\beta }_0$ as the mean of $\underset{\tilde{}}{\nu }$, and then we can omit the intercept term from the covariate ${\underset{\tilde{}}{x}}_{ijk}$. Second, we introduce a new parameter, $w_{ij}={\nu }_i+{\mu }_{ij}$, in order to set ${\nu }_i$ and ${\mu }_{ij}$ independently and make it easy to make an inference for both of them. We have

$$y_{ijk}|w_{ij},{\underset{\tilde{}}{\beta }}_{\left(0\right)}\stackrel{ind}{\sim }\mathrm{Bernoulli}\left\{\frac{e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij}}}{1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij}}}\right\},k=1,\dots ,m_{ij},$$

$$w_{ij}|{\nu }_i,{\sigma }^2\stackrel{ind}{\sim }\mathrm{Normal}({\nu }_i,{\sigma }^2),j=1,\dots ,n_i,$$

(2)

$${\nu }_i|{\beta }_0,{\delta }^2\stackrel{iid}{\sim }\mathrm{Normal}({\beta }_0,{\delta }^2),i=1,\dots ,\ell ,$$

$$\pi (\underset{\tilde{}}{\beta },{\delta }^2,{\sigma }^2)\propto \frac{1}{{(1+{\delta }^2)}^2}\frac{1}{{(1+{\sigma }^2)}^2},{\delta }^2>0,{\sigma }^2>0.$$

The joint posterior density for the parameters is

$$\begin{array}{cc}\hfill \pi (\underset{\tilde{}}{v},\underset{\tilde{}}{w},\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2|\underset{\tilde{}}{y})& \propto \prod _{i=1}^{\ell }\prod _{j=1}^{n_i}\prod _{k=1}^{m_{ij}}\left[\frac{e^{({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij})y_{ijk}}}{1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij}}}\right]\times {\left(\frac{1}{\sqrt{2\pi {\sigma }^2}}\right)}^{n}exp\left\{-\sum _{i=1}^l\sum _{j=1}^{ni}\frac{{(w_{ij}-{\nu }_i)}^2}{2{\sigma }^2}\right\}\hfill \\ & \times {\left(\frac{1}{\sqrt{2\pi {\delta }^2}}\right)}^{l}exp\left\{-\sum _{i=1}^l\frac{{({\nu }_i-{\beta }_0)}^2}{2{\delta }^2}\right\}\frac{1}{{(1+{\sigma }^2)}^2}\frac{1}{{(1+{\delta }^2)}^2}.\hfill \end{array}$$

(3)

The posterior density is a non-standard multivariate density, and there are difficulties in fitting it using MCMC methods, more so when $n_i,m_{ij}$ are large. This motivates our approximate methods.

3. Integrated Nested Normal Approximation Method

In this section, we discuss the INNA method for the sub-area HB logistic regression model. It is an extension of the INNA method in [3]. INNA method is not required to find the posterior modes. Due to the large amount of sub-areas, it would be time-consuming to find all posterior modes, which is why we did not choose the popular INLA method. In detail, we discuss the approximation of the joint posterior density (3).

Notice that the joint posterior density (3) is very complicated and it is the expit part, ${\displaystyle \prod _{i=1}^{\ell }}{\displaystyle \prod _{j=1}^{n_i}}{\displaystyle \prod _{k=1}^{m_{ij}}}\left[\frac{e^{({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij})y_{ijk}}}{1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij}}}\right]$, that causes the difficulties. In the following, we discuss how to approximate this term to normal density functions by using Laplace approximation, the second-order multivariate Taylor-series approximation and the empirical logistic transform (ELT). This is the key contribution in the paper. Then we use the multiplication rule to approximate the joint posterior density,

$${\pi }_a(\underset{\tilde{}}{w},\underset{\tilde{}}{\nu },\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2\mid \underset{\tilde{}}{y})\propto {\pi }_a(\underset{\tilde{}}{w}\mid \underset{\tilde{}}{\nu },{\underset{\tilde{}}{\beta }}_{\left(0\right)},{\sigma }^2,\underset{\tilde{}}{y}){\pi }_a(\underset{\tilde{}}{\nu }\mid {\beta }_0,{\delta }^2,\underset{\tilde{}}{y}){\pi }_a({\underset{\tilde{}}{\beta }}_{\left(0\right)}\mid \underset{\tilde{}}{y}){\pi }_a(\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2\mid \underset{\tilde{}}{y}),$$

where the first three densities on the right-hand side are all multivariate normal densities. Therefore, we can draw samples and make inference through the approximate joint posterior density.

Let $f\left(\underset{\tilde{}}{\tau }\right)=e^{h\left(\underset{\tilde{}}{\tau }\right)}$ denote the density of a vector of parameters $\underset{\tilde{}}{\tau }$. Let $\underset{\tilde{}}{g}$ denote the gradient vector and H the Hessian matrix at some point ${\underset{\tilde{}}{\tau }}^{*}$.

Lemma 1. 

Let $h\left(\underset{\tilde{}}{\tau }\right)$ be a logconcave density function with the parameter $\underset{\tilde{}}{\tau }$. Then, $\underset{\tilde{}}{\tau }$ approximately has a multivariate normal distribution,

$$\underset{\tilde{}}{\tau }\stackrel{}{\sim }Normal({\underset{\tilde{}}{\tau }}^{*}-H^{-1}\underset{\tilde{}}{g},-H^{-1}).$$

Proof. 

Simply applying the second-order multivariate Taylor series of $h\left(\underset{\tilde{}}{\tau }\right)$ at ${\underset{\tilde{}}{\tau }}^{*}$ gives

$$f\left(\underset{\tilde{}}{\tau }\right)\approx f\left({\underset{\tilde{}}{\tau }}^{*}\right)+{(\underset{\tilde{}}{\tau }-{\underset{\tilde{}}{\tau }}^{*})}^{\prime }\underset{\tilde{}}{g}+\frac{1}{2}{(\underset{\tilde{}}{\tau }-{\underset{\tilde{}}{\tau }}^{*})}^{\prime }H(\underset{\tilde{}}{\tau }-{\underset{\tilde{}}{\tau }}^{*}).$$

Due to the logconcavity of $h\left(\underset{\tilde{}}{\tau }\right)$, its Hessian Matrix $-H$ is posit-definite, which can be the covariance matrix. Notice that we are not required to use the mode of $h\left(\underset{\tilde{}}{\tau }\right)$. We do not need to find the solution of the gradient vector $\underset{\tilde{}}{g}=0$. Therefore, ${\underset{\tilde{}}{\tau }}^{*}$ does not have to be the solution but some other point. It is worth noticing that the term, $-H^{-1}\underset{\tilde{}}{g}$, is a correction to ${\underset{\tilde{}}{\tau }}^{*}$. □

To illustrate the approximation steps, we start with a simpler model with flat priors for ${\underset{\tilde{}}{\beta }}_{\left(0\right)}$ and the $\underset{\tilde{}}{w}$, according to model (2). That is,

$$y_{ijk}|w_{ij},{\underset{\tilde{}}{\beta }}_{\left(0\right)}\stackrel{ind}{\sim }\mathrm{Bernoulli}\left\{\frac{e^{{\underset{\tilde{}}{x}}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij}}}{1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij}}}\right\},k=1,\dots ,m_{ij},j=1,\dots ,n_i,i=1,\dots ,\ell ,$$

$$p(\underset{\tilde{}}{w},{\underset{\tilde{}}{\beta }}_{\left(0\right)})=1.$$

(4)

The joint posterior density is

$$\pi (\underset{\tilde{}}{w},{\underset{\tilde{}}{\beta }}_{\left(0\right)}|\underset{\tilde{}}{y})\propto \prod _{i=1}^{\ell }\prod _{j=1}^{n_i}\prod _{k=1}^{m_{ij}}\left\{\frac{e^{({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij})y_{ijk}}}{1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij}}}\right\}.$$

(5)

The logarithm of the joint posterior density (or log likelihood) is

$$\Delta =h\left(\underset{\tilde{}}{\tau }\right)=\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}\left\{({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij})y_{ijk}-log(1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij}})\right\}.$$

Let ${\underset{\tilde{}}{\tau }}^{\prime }=({\underset{\tilde{}}{\mu }}^{\prime },{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{\prime })$. In our method, we find a convenient point to expand the log-likelihood in a second-order multivariate Taylor-series expansion.

To begin with, let ${\overline{y}}_{ij}=\frac{1}{m_{ij}}{\sum }_{k=1}^{m_{ij}}{y}_{ijk}$. We use the empirical logistic transform $z_{ij}$ to get an estimate of $w_{ij}$, where

$${\widehat{w}}_{ij}^{*}=z_{ij}=log\left\{\frac{{\overline{y}}_{ij}+\frac{1}{2m_{ij}}}{1-{\overline{y}}_{ij}+\frac{1}{2m_{ij}}}\right\},i=1,\dots ,\ell ;j=1,\cdots ,n_i.$$

First, we discuss how to find the quasi mode of ${\underset{\tilde{}}{\beta }}_{\left(0\right)}$. We plug ${\widehat{w}}_{ij}^{*}$ into the log likelihood function $\Delta$ and consider it as a function of ${\underset{\tilde{}}{\beta }}_{\left(0\right)}$ only as $q\left({\underset{\tilde{}}{\beta }}_{\left(0\right)}\right)$, and we get

$$q\left({\underset{\tilde{}}{\beta }}_{\left(0\right)}\right)=\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}\left[({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+{\widehat{w}}_{ij}^{*})y_{ijk}-log(1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+{\widehat{w}}_{ij}^{*}})\right].$$

The first derivative of $q\left({\underset{\tilde{}}{\beta }}_{\left(0\right)}\right)$ is

$$\begin{array}{cc}\hfill q^{\prime }\left({\underset{\tilde{}}{\beta }}_{\left(0\right)}\right)& =\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}\left\{{\underset{\tilde{}}{x}}_{ijk}{y}_{ijk}-\frac{{\underset{\tilde{}}{x}}_{ijk}{e}^{({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+{\widehat{w}}_{ij}^{*})}}{1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+{\widehat{w}}_{ij}^{*}}}\right\}\hfill \\ \hfill & =\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}\left\{{\underset{\tilde{}}{x}}_{ijk}{y}_{ijk}-{\underset{\tilde{}}{x}}_{ijk}{\left[1+e^{-({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+{\widehat{w}}_{ij}^{*})}\right]}^{-1}\right\}.\hfill \end{array}$$

Usually we should set $q^{\prime }\left({\underset{\tilde{}}{\beta }}_{\left(0\right)}\right)$ equal to zero and find the modes as the maximum likelihood estimator (MLE) of ${\underset{\tilde{}}{\beta }}_{\left(0\right)}$. But here, it is not easy to solve the equation due to the complexity of $q^{\prime }\left({\underset{\tilde{}}{\beta }}_{\left(0\right)}\right)$. We use the first-order Taylor series to approximate it and then simplify $q^{\prime }\left({\underset{\tilde{}}{\beta }}_{\left(0\right)}\right)$ so that we can get quasi-modes of ${\underset{\tilde{}}{\beta }}_{\left(0\right)}$.

The first-order Taylor expansion of ${(1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+{\widehat{w}}_{ij}^{*}})}^{-1}$ equals $(1-e^{-({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+{\widehat{w}}_{ij}^{*})})$. Notice that by Taylor series, $e^{-({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+{\widehat{w}}_{ij}^{*})}\approx 1-({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+{\widehat{w}}_{ij}^{*})$. Then we can get

$$\begin{array}{cc}\hfill q^{\prime }\left({\underset{\tilde{}}{\beta }}_{\left(0\right)}\right)& \approx \sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}\left\{{\underset{\tilde{}}{x}}_{ijk}{y}_{ijk}-{\underset{\tilde{}}{x}}_{ijk}\left[(1-e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+{\widehat{w}}_{ij}^{*}})\right]\right\}\hfill \\ \hfill & \approx \sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}\left\{{\underset{\tilde{}}{x}}_{ijk}{y}_{ijk}-{\underset{\tilde{}}{x}}_{ijk}\left[(1-1+({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+{\widehat{w}}_{ij}^{*}))\right]\right\}\hfill \\ \hfill & =\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}\left\{{\underset{\tilde{}}{x}}_{ijk}(y_{ijk}-{\widehat{w}}_{ij}^{*})-{\underset{\tilde{}}{x}}_{ijk}{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}\right\}.\hfill \end{array}$$

We can get the quasi-modes of ${\underset{\tilde{}}{\beta }}_{\left(0\right)}$ by solving the equation $q^{\prime }\left({\underset{\tilde{}}{\beta }}_{\left(0\right)}\right)=0$. That is,

$${\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}={\left[\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}{\underset{\tilde{}}{x}}_{ijk}{\underset{\tilde{}}{x}}_{ijk}^{\prime }\right]}^{-1}\left[\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}{\underset{\tilde{}}{x}}_{ijk}(y_{ijk}-{\widehat{w}}_{ij}^{*})\right].$$

Second, we obtain quasi-modes for the $w_{ij}$, a refinement of the $z_i$. Plug ${\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}$ into the likelihood function $\Delta$ and consider it as function $w_{ij}$ only:

$$g\left(w_{ij}\right)=\sum _{k=1}^{m_{ij}}\left[({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij})y_{ijk}-log(1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{{\left(0\right)}^{*}}+w_{ij}})\right].$$

Similarly, after applying Taylor expansion, we get the approximate first derivative of $g\left(w_{ij}\right)$

$$\begin{array}{cc}\hfill g^{\prime }\left(w_{ij}\right)& =\sum _{k=1}^{m_{ij}}\left\{y_{ijk}-{\left[1+e^{-({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij})}\right]}^{-1}\right\}\hfill \\ \hfill & \approx \sum _{k=1}^{m_{ij}}\left\{y_{ijk}-\left(1-e^{-w_{ij}}{e}^{-{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}}\right)\right\}.\hfill \end{array}$$

We can obtain the approximate posterior mode of $w_{ij}$ by solving the equation $g^{\prime }\left(w_{ij}\right)=0$.

$$w_{ij}^{*}=log\left\{\frac{{\sum }_{k=1}^{m_{ij}}{e}^{-{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}}}{m_{ij}(1-{\overline{y}}_{ij})}\right\}.$$

Notice that the term $1-{\overline{y}}_{ij}$ in denominator may cause trouble if ${\overline{y}}_{ij}=1$ for some is and js. Here, we borrow the idea from ELT and make a small adjustment in order to avoid a zero denominator. That is,

$$w_{ij}^{*}\approx log\left\{\frac{{\sum }_{k=1}^{m_{ij}}{e}^{-{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}}}{m_{ij}(1-{\overline{y}}_{ij}+\frac{1}{2m_{ij}})}\right\}i=1,\dots ,\ell ,j=1,\dots ,n_i.$$

Let ${\underset{\tilde{}}{\tau }}^{*\prime }=({\underset{\tilde{}}{\mu }}^{*\prime },{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*\prime })$. Next, we evaluate $\underset{\tilde{}}{g}$ and H at the quasi-modes $\underset{\tilde{}}{\tau }={\underset{\tilde{}}{\tau }}^{*}$ can also be obtained as

$$\underset{\tilde{}}{g}={\left(\begin{array}{c}\hfill \frac{\partial \Delta }{\partial w_{11}}\cdots \frac{\partial \Delta }{\partial w_{\ell n_{\ell }}}\frac{\partial \Delta }{\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}}\end{array}\right)}_{\underset{\tilde{}}{w}={\underset{\tilde{}}{w}}^{*},{\underset{\tilde{}}{\beta }}_{\left(0\right)}={\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}}^T,$$

$$H={\left(\begin{array}{cccc}\frac{{\partial }^2\Delta }{\partial w_{11}^2}& \cdots & \frac{{\partial }^2\Delta }{\partial w_{11}\partial w_{\ell n_{\ell }}}& \frac{{\partial }^2\Delta }{\partial w_{11}\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}}\\ ⋮& ⋮& \ddots & ⋮\\ 0& \cdots & \frac{{\partial }^2\Delta }{\partial w_{\ell n_{\ell }}^2}& \frac{{\partial }^2\Delta }{\partial w_{\ell n_{\ell }}\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}}\\ \frac{{\partial }^2\Delta }{\partial w_{11}\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}}& \cdots & \frac{{\partial }^2\Delta }{\partial w_{\ell n_{\ell }}\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}}& \frac{{\partial }^2\Delta }{\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}^2}\end{array}\right)}_{\underset{\tilde{}}{w}={\underset{\tilde{}}{w}}^{*},{\underset{\tilde{}}{\beta }}_{\left(0\right)}={\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}}.$$

The partial derivatives can be expressed in terms of response $y_{ijk}$ and covariates ${\underset{\tilde{}}{x}}_{ijk}$ as

$$\frac{\partial \Delta }{\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}}=\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}\left\{{\underset{\tilde{}}{x}}_{ijk}{y}_{ijk}-\frac{{\underset{\tilde{}}{x}}_{ijk}{e}^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij}^{*}}}{1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij}^{*}}}\right\},$$

$$\frac{\partial \Delta }{\partial w_{ij}}=\sum _{k=1}^{m_{ij}}(y_{ijk}-\frac{e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij}^{*}}}{1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij}^{*}}}),$$

$$\frac{{\partial }^2\Delta }{\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}^2}=-\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}\frac{{\underset{\tilde{}}{x}}_{ijk}{\underset{\tilde{}}{x}}_{ijk}^{\prime }{e}^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij}^{*}}}{{(1+e^{{\underset{\tilde{}}{x}}_{ij}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij}^{*}})}^2},$$

$$\frac{{\partial }^2\Delta }{\partial w_{ij}^2}=-\sum _{k=1}^{m_{ij}}\frac{e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij}^{*}}}{{(1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij}^{*}})}^2},$$

$$\frac{{\partial }^2\Delta }{\partial {\mu }_i\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}}=-\sum _{k=1}^{m_{ij}}\frac{{\underset{\tilde{}}{x}}_{ijk}{e}^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij}^{*}}}{{(1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}^{*}+w_{ij}^{*}})}^2},$$

where $i=1,\dots ,\ell ,j=1,\dots ,n_i$.

For the convenience of computation, denote $\underset{\tilde{}}{g}=\left(\begin{array}{c}{\underset{\tilde{}}{g}}_1\\ {\underset{\tilde{}}{g}}_2\end{array}\right)$ and $H=-\left(\begin{array}{c}D{C}^{\prime }\\ CB\end{array}\right),$ where

$${\underset{\tilde{}}{g}}_1={\left(\begin{array}{c}\hfill \frac{\partial \Delta }{\partial w_{11}}\cdots \frac{\partial \Delta }{\partial w_{\ell n_{\ell }}}\end{array}\right)}^T,{\underset{\tilde{}}{g}}_2=\frac{\partial \Delta }{\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}},$$

$$B=-\frac{{\partial }^2\Delta }{\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}^2},C=-\left(\begin{array}{c}\hfill \frac{{\partial }^2\Delta }{\partial w_{11}\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}}\cdots \frac{{\partial }^2\Delta }{\partial w_{\ell n_{\ell }}\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}}\end{array}\right),D=-\left(\begin{array}{c}\hfill \frac{{\partial }^2\Delta }{\partial w_{11}^2}\cdots 0\\ \hfill :\ddots :\\ \hfill 0\cdots \frac{{\partial }^2\Delta }{\partial w_{\ell n_{\ell }}^2}\end{array}\right).$$

Let $-H^{-1}={\left(\begin{array}{c}D{C}^{\prime }\\ CB\end{array}\right)}^{-1}=\left(\begin{array}{c}E{F}^{\prime }\\ FG\end{array}\right)$, where

$$E=D^{-1}+D^{-1}{C}^{\prime }{(B-C{D}^{-1}{C}^{\prime })}^{-1}C{D}^{-1},F=-{(B-C{D}^{-1}{C}^{\prime })}^{-1}C{D}^{-1},G={(B-C{D}^{-1}{C}^{\prime })}^{-1}.$$

Lemma 2. 

Assuming that the design matrix is full-rank and $0<{\sum }_{k=1}^{m_{ij}}{y}_{ijk}<m_{ij},j=1,\dots ,n_i;i=1,\dots ,\ell$, the posterior density, $\underset{\tilde{}}{\tau }|\underset{\tilde{}}{y}$ in (5), is logconcave.

Proof. 

If $0<{\sum }_{k=1}^{m_{ij}}{y}_{ijk}<m_{ij},i=1,\dots ,\ell ,j=1,\dots ,n_i$, there are solutions to the gradient vector set to zero.

Let $p_{ijk}=\frac{e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij}}}{1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij}}},k=1,\dots ,m_{i}j,j=1,\dots ,n_i,i=1,\dots ,\ell$. Then, A, B and C of the negative Hessian matrix can be written as,

$$B=-\frac{{\partial }^2\Delta }{\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}^2}=\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}{p}_{ijk}(1-p_{ijk}){\underset{\tilde{}}{x}}_{ijk}{\underset{\tilde{}}{x}}_{ijk}^{\prime },$$

$$D=diagonal\left(d_{ij}\right),d_{ij}=\frac{{\partial }^2\Delta }{\partial w_{ij}^2}=\sum _{k=1}^{m_{ij}}{p}_{ijk}(1-p_{ijk}),$$

$$C=\left({\underset{\tilde{}}{c}}_{ij}\right),{\underset{\tilde{}}{c}}_{ij}=\frac{{\partial }^2\Delta }{\partial w_{ij}\partial {\underset{\tilde{}}{\beta }}_{\left(0\right)}}=\sum _{k=1}^{m_{ij}}{p}_{ijk}(1-p_{ijk}){\underset{\tilde{}}{x}}_{ijk},$$

where $j=1,\dots ,n_i,i=1,\dots ,\ell .$

It is obvious that D is positive-definite. Thus, to show that $-H$ is positive-definite, we need to show that its Schur complement of D, $S=B-C{D}^{-1}{C}^{\prime }$, is positive-definite (e.g., see [35]). Let ${\omega }_{ijk}=p_{ijk}(1-p_{ijk})/{\sum }_{k=1}^{m_{ij}}{p}_{ijk}(1-p_{ijk}),k=1,\dots ,m_{ij},j=1,\dots ,n_i,i=1,\dots ,\ell$. The Schur complement is

$$S=\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}{p}_{ijk}(1-p_{ijk})\sum _{k=1}^{m_{ij}}{\omega }_{ijk}{\underset{\tilde{}}{x}}_{ijk}{\underset{\tilde{}}{x}}_{ijk}^{\prime }-\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}{p}_{ijk}(1-p_{ijk})\sum _{k=1}^{m_{ij}}{\omega }_{ijk}{\underset{\tilde{}}{x}}_{ijk}\sum _{k=1}^{m_{ij}}{\omega }_{ijk}{\underset{\tilde{}}{x}}_{ijk}^{\prime }.$$

It is now easy to show that

$$S=\sum _{i=1}^{\ell }\sum _{j=1}^{n_i}\sum _{k=1}^{m_{ij}}{\omega }_{ijk}({\underset{\tilde{}}{x}}_{ijk}-\sum _{k=1}^{m_{ij}}{\omega }_{ijk}{\underset{\tilde{}}{x}}_{ijk}){({\underset{\tilde{}}{x}}_{ijk}-\sum _{k=1}^{m_{ij}}{\omega }_{ijk}{\underset{\tilde{}}{x}}_{ijk})}^{\prime }.$$

Therefore, $-H$ is positive-definite, and $\underset{\tilde{}}{\tau }|\underset{\tilde{}}{y}$ is logconcave. □

Finally, according to the Lemmas 1 and 2, we can establish the approximation Theorem.

Theorem 1. 

Assuming that the design matrix is full-rank and $0<{\sum }_{k=1}^{m_{ij}}{y}_{ijk}<m_{ij},j=1,\dots ,n_i,i=1,\dots ,\ell$, the posterior density, $\underset{\tilde{}}{\tau }|\underset{\tilde{}}{y}$ in (5) is approximately a multivariate normal density, and the conditional posterior density of $\underset{\tilde{}}{w}|{\underset{\tilde{}}{\beta }}_{\left(0\right)},\underset{\tilde{}}{y}$ and ${\underset{\tilde{}}{\beta }}_{\left(0\right)}|\underset{\tilde{}}{y}$ can also be approximated by multivariate normal distributions.

Proof. 

The proof is given in Appendix B. □

Therefore, we can approximate that logit expit term ${\displaystyle \prod _{i=1}^{\ell }}{\displaystyle \prod _{j=1}^{n_i}}{\displaystyle \prod _{k=1}^{m_{ij}}}\left[\frac{e^{({\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij})y_{ijk}}}{1+e^{{\underset{\tilde{}}{x}}_{ijk}^{\prime }{\underset{\tilde{}}{\beta }}_{\left(0\right)}+w_{ij}}}\right]$ into two multivariate densities by Theorem 1. And then we can get our approximate two-fold Bayesian logistic regression model.

Recall the posterior density of our two-fold logistic model is

$$\pi (\underset{\tilde{}}{w},\underset{\tilde{}}{\nu },\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2\mid \underset{\tilde{}}{y})\propto \pi \left(\underset{\tilde{}}{y}\right|\underset{\tilde{}}{w},{\underset{\tilde{}}{\beta }}_{\left(0\right)})\pi (\underset{\tilde{}}{w}\mid \underset{\tilde{}}{\nu },{\sigma }^2)\pi (\underset{\tilde{}}{\nu }\mid {\beta }_0,{\delta }^2)\pi ({\underset{\tilde{}}{\beta }}_{\left(0\right)},{\beta }_0,{\sigma }^2,{\delta }^2)$$

The likelihood function $\pi \left(\underset{\tilde{}}{y}\right|\underset{\tilde{}}{w},{\underset{\tilde{}}{\beta }}_{\left(0\right)})$ can be approximated by the multivariate normal distribution by Theorem 1. Combining the prior values of $\underset{\tilde{}}{w}$ and $\underset{\tilde{}}{\nu }$ given by our Bayesian Logistic model and the results in Theroem 1, we can obtain our INNA model

$$\underset{\tilde{}}{w}|{\underset{\tilde{}}{\beta }}_{\left(0\right)},\underset{\tilde{}}{y}\stackrel{}{\sim }\mathrm{Normal}\{{\underset{\tilde{}}{\mu }}_w-D^{-1}{C}^{\prime }({\underset{\tilde{}}{\beta }}_{\left(0\right)}-{\underset{\tilde{}}{\mu }}_{\beta }),D^{-1}\}$$

$${\underset{\tilde{}}{\beta }}_{\left(0\right)}|\underset{\tilde{}}{y}\stackrel{}{\sim }\mathrm{Normal}\{{\underset{\tilde{}}{\mu }}_{\beta },G\}$$

$$\underset{\tilde{}}{w}|\underset{\tilde{}}{\nu },{\sigma }^2\stackrel{ind}{\sim }\mathrm{Normal}({\underset{\tilde{}}{\mu }}_{\nu },{\sigma }^{2}I),$$

$$\underset{\tilde{}}{\nu }|{\beta }_0,{\delta }^2\stackrel{iid}{\sim }\mathrm{Normal}({\beta }_0\underset{\tilde{}}{j},{\delta }^{2}I),$$

$$\pi ({\underset{\tilde{}}{\beta }}_{\left(0\right)},{\beta }_0,{\delta }^2,{\sigma }^2)\propto \frac{1}{{(1+{\delta }^2)}^2}\frac{1}{{(1+{\sigma }^2)}^2},{\delta }^2>0,{\sigma }^2>0,$$

where ${\underset{\tilde{}}{\mu }}_{\nu }^{\prime }={\left(\underset{n_1}{\underbrace{{\nu }_1,\dots ,{\nu }_1}}\cdots \underset{n_{\ell }}{\underbrace{{\nu }_{\ell },\dots ,{\nu }_{\ell }}}\right)}^{\prime }$ and $\underset{\tilde{}}{j}$ is a vector of ones.

Using Bayes’ Theorem and the multiplication rule, the posterior density $\pi (\underset{\tilde{}}{w},\underset{\tilde{}}{\nu },\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2\mid \underset{\tilde{}}{y})$ can be approximated as

$$\begin{array}{cc}\hfill {\pi }_a(\underset{\tilde{}}{w},\underset{\tilde{}}{\nu },\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2\mid \underset{\tilde{}}{y})& \propto {\pi }_a(\underset{\tilde{}}{w}\mid \underset{\tilde{}}{\nu },{\underset{\tilde{}}{\beta }}_{\left(0\right)},{\sigma }^2,\underset{\tilde{}}{y}){\pi }_a(\underset{\tilde{}}{\nu }\mid {\beta }_0,{\delta }^2,\underset{\tilde{}}{y}){\pi }_a({\underset{\tilde{}}{\beta }}_{\left(0\right)}\mid \underset{\tilde{}}{y}){\pi }_a(\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2\mid \underset{\tilde{}}{y})\hfill \\ \hfill & =e^{-\frac{1}{2}\left\{{\left[\underset{\tilde{}}{w}-\left({\underset{\tilde{}}{\mu }}_w-D^{-1}{C}^{\prime }({\underset{\tilde{}}{\beta }}_{\left(0\right)}-{\underset{\tilde{}}{\mu }}_{\beta })\right)\right]}^{\prime }D\left[\underset{\tilde{}}{w}-\left({\underset{\tilde{}}{\mu }}_w-D^{-1}{C}^{\prime }({\underset{\tilde{}}{\beta }}_{\left(0\right)}-{\underset{\tilde{}}{\mu }}_{\beta })\right)\right]\right\}}\hfill \\ \hfill & \times e^{-\frac{1}{2}\left\{{\left[\underset{\tilde{}}{w}-{\underset{\tilde{}}{\mu }}_{\nu }\right]}^{\prime }{\left({\sigma }^{2}I\right)}^{-1}\left[\underset{\tilde{}}{w}-{\underset{\tilde{}}{\mu }}_{\nu }\right]+{\left[\underset{\tilde{}}{\nu }-{\beta }_0\underset{\tilde{}}{j}\right]}^{\prime }{\left({\delta }^{2}I\right)}^{-1}\left[\underset{\tilde{}}{\nu }-{\beta }_0\underset{\tilde{}}{j}\right]+{\left[{\underset{\tilde{}}{\beta }}_{\left(0\right)}-{\underset{\tilde{}}{\mu }}_{\beta }\right]}^{\prime }{G}^{-1}\left[{\underset{\tilde{}}{\beta }}_{\left(0\right)}-{\underset{\tilde{}}{\mu }}_{\beta }\right]\right\}}\hfill \\ \hfill & \times \frac{{\left|D\right|}^{1/2}}{|{\delta }^2{I|}^{1/2}|{\sigma }^2{I|}^{1/2}{\left|G\right|}^{1/2}}\frac{1}{{(1+{\sigma }^2)}^2}\frac{1}{{(1+{\delta }^2)}^2}\hfill \end{array}$$

(6)

Therefore, we can get the following key result.

Theorem 2. 

Using the multiplication rule, the joint posterior density, $\pi (\underset{\tilde{}}{w},\underset{\tilde{}}{\nu },\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2\mid \underset{\tilde{}}{y})$ in (6), can be approximated by

$${\pi }_a(\underset{\tilde{}}{w},\underset{\tilde{}}{\nu },\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2\mid \underset{\tilde{}}{y})\propto {\pi }_a(\underset{\tilde{}}{w}\mid \underset{\tilde{}}{\nu },{\underset{\tilde{}}{\beta }}_{\left(0\right)},{\sigma }^2,\underset{\tilde{}}{y}){\pi }_a(\underset{\tilde{}}{\nu }\mid {\beta }_0,{\delta }^2,\underset{\tilde{}}{y}){\pi }_a({\underset{\tilde{}}{\beta }}_{\left(0\right)}\mid \underset{\tilde{}}{y}){\pi }_a(\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2\mid \underset{\tilde{}}{y}),$$

where the first three densities on the right-hand side are all multivariate normal densities.

Proof. 

The proof is given in Appendix C. □

The INNA is actually a random sampler. First, we draw samples for ${\sigma }^2,{\delta }^2$ from $\pi ({\sigma }^2,{\delta }^2|\underset{\tilde{}}{y})$. The posterior distribution of ${\sigma }^2,{\delta }^2|\underset{\tilde{}}{y}$ does not have standardized form. Here, we use the grid method and numerical integration to sample ${\sigma }^2$ and ${\delta }^2$. Since $0<{\sigma }^2<\infty$ and $0<{\delta }^2<\infty$, we make a transformation to ${\varphi }_1=\frac{1}{1+{\sigma }^2}$ and ${\varphi }_2=\frac{1}{1+{\delta }^2}$ so that we get $0<{\varphi }_1<1$ and $0<{\varphi }_2<1$. Then, the posterior density of ${\varphi }_1,{\varphi }_2|\underset{\tilde{}}{y}$ is

$$\begin{array}{cc}\hfill {\pi }_a({\varphi }_1,{\varphi }_2|\underset{\tilde{}}{y})& \propto {\left\{{\left|\begin{array}{c}\begin{array}{c}{\delta }_0^2{\underset{\tilde{}}{\gamma }}^{\prime }\\ \underset{\tilde{}}{\gamma }{\Delta }_{\left(0\right)}\end{array}\end{array}\right|}^{-\frac{1}{2}}\times \prod _{i=1}^l{\left(\frac{1}{{\sum }_{j=1}^{n_i}{\sigma }_{ij}^2}+\frac{1}{{\delta }^2}\right)}^{\frac{1}{2}}\frac{1}{|{\delta }^{2}D+\frac{{\delta }^2}{{\sigma }^2}{I|}^{1/2}}\right\}}_{{\varphi }_1=\frac{1}{1+{\sigma }^2},{\varphi }_2=\frac{1}{1+{\delta }^2}}\hfill \\ \hfill & \times exp\left\{-\frac{1}{2}{({\underset{\tilde{}}{\mu }}_w+D^{-1}{C}^{\prime }{\underset{\tilde{}}{\mu }}_{\beta })}^{\prime }{(D^{-1}+{\sigma }^{2}I+{\delta }^{2}I)}^{-1}({\underset{\tilde{}}{\mu }}_w+D^{-1}{C}^{\prime }{\underset{\tilde{}}{\mu }}_{\beta })+{\underset{\tilde{}}{\mu }}_{\beta }^{\prime }{G}^{-1}{\underset{\tilde{}}{\mu }}_{\beta }\right\}\hfill \\ \hfill & \times exp{\left\{-\frac{1}{2}{\left(\begin{array}{c}{\beta }_0-{\omega }_0\\ {\underset{\tilde{}}{\beta }}_{\left(0\right)}-{\underset{\tilde{}}{\omega }}_{\left(0\right)}\end{array}\right)}^{\prime }\left(\begin{array}{c}{\delta }_0^2{\underset{\tilde{}}{\gamma }}^{\prime }\\ \underset{\tilde{}}{\gamma }{\Delta }_{\left(0\right)}\end{array}\right)\left(\begin{array}{c}{\beta }_0-{\omega }_0\\ {\underset{\tilde{}}{\beta }}_{\left(0\right)}-{\underset{\tilde{}}{\omega }}_{\left(0\right)}\end{array}\right)\right\}}_{{\varphi }_1=\frac{1}{1+{\sigma }^2},{\varphi }_2=\frac{1}{1+{\delta }^2}}.\hfill \end{array}$$

We need to draw ${\varphi }_1,{\varphi }_2$ together. The joint density can be rewritten as

$$\pi ({\varphi }_1,{\varphi }_2|\underset{\tilde{}}{y})=\pi \left({\varphi }_2\right|{\varphi }_1)\pi \left({\varphi }_1\right|\underset{\tilde{}}{y})=\pi \left({\varphi }_2\right|{\varphi }_1){\int }_0^1\pi ({\varphi }_1,{\varphi }_2|\underset{\tilde{}}{y})d{\varphi }_2.$$

We plug each grid of ${\varphi }_1\in (0,1)$ into ${\int }_0^1\pi ({\varphi }_1,{\varphi }_2|\underset{\tilde{}}{y})d{\varphi }_2$ and then use numerical integration to get the density of $\left({\varphi }_1\right|\underset{\tilde{}}{y})$. After we plug all the 100 grids, we can get 100 value of $\pi \left({\varphi }_1\right|\underset{\tilde{}}{y})$ and then draw ${\varphi }_1$ from them, i.e. ${\varphi }_1^{\left(h\right)}$. Next, we plug ${\varphi }_1^{\left(h\right)}$ into $\pi \left({\varphi }_2\right|{\varphi }_1)$ and use grid method to draw ${\varphi }_2^{\left(h\right)}$. We repeat those steps 10,000 times to get the sample of $({\varphi }_1^{\left(h\right)},{\varphi }_2^{\left(h\right)}),h=1,\dots ,\mathrm{10,000}.$ Once we get samples for ${\varphi }_1,{\varphi }_2$, we transform them back to ${\sigma }^2$ and ${\delta }^2$ respectively. Second, given ${\sigma }^2,{\delta }^2$, we can simply draw samples of $\underset{\tilde{}}{\beta }$ from the approximate multivariate normal distribution ${\pi }_a(\underset{\tilde{}}{\beta }\mid {\sigma }^2,{\delta }^2,\underset{\tilde{}}{y})$. Third, we can draw samples of ${\nu }_i$ independently given $\underset{\tilde{}}{\beta },{\delta }^2$ and data from the approximate normal distribution ${\pi }_a\left(\underset{\tilde{}}{\nu }\right|{\beta }_0,{\delta }^2,\underset{\tilde{}}{y})$. Finally, samples of $w_{ij}$ independently given $\underset{\tilde{}}{\nu },\underset{\tilde{}}{\beta },{\sigma }^2$ can be obtained from the approximate normal distribution ${\pi }_a\left(\underset{\tilde{}}{w}\right|\underset{\tilde{}}{\nu },{\underset{\tilde{}}{\beta }}_{\left(0\right)},{\sigma }^2,\underset{\tilde{}}{y})$. Notice that the last three steps are very simple, just drawing samples from normal densities. In addition, $w_{ij}$ and ${\nu }_i$ are all independent so that we can draw them simultaneously. Therefore, those latter steps permit fast computing.

In order to check if INNA method can provide resonal results, we apply the MCMC logistic regression exact method to the sub-area model. The idea of exact method is to get full conditional posterior distributions for all of the parameters in the model, and then get a large number of independent samples of each parameter with its full conditional posterior density. Details are given in Appendix A.

There are two differences between these two methods. First, both methods are sampling-based. The approximate method implements random samples and the exact method uses numerical integration method and Markov chains. Second, ${\pi }_a(\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2\mid \underset{\tilde{}}{y})$ is used for the INNA method. In the exact method, a Metroplis step is used for the $\pi (\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2\mid \underset{\tilde{}}{y})$. This is very time-consuming in the exact method. On the other hand, the exact method actually uses the INNA method. We use M-H sampler draw samples for $\underset{\tilde{}}{\nu }$ and $\underset{\tilde{}}{w}$, respectively. Proposal functions are ${\pi }_a(\underset{\tilde{}}{\nu }\mid {\underset{\tilde{}}{\beta }}_{\left(0\right)},{\beta }_0,{\sigma }^2,\underset{\tilde{}}{y})$ and ${\pi }_a(\underset{\tilde{}}{w}\mid \underset{\tilde{}}{\nu },{\underset{\tilde{}}{\beta }}_{\left(0\right)},{\sigma }^2,\underset{\tilde{}}{y})$, respectively, from the INNA method.

4. Numerical Example

4.1. Nepal Living Standards Survey II

The performance of our method is studied using the Nepal Living Standard Survey (NLSS II), conducted in the years 2003–2004. The main objective of the NLSS II is to track changes in and the progress of national living standards and social indicators of the Nepalese population. It is an integrated survey which covers samples from the whole country and runs throughout the year.

The NLSS II gathers information on a variety of aspects. It has collected data on demographics, housing, education, health, fertility, employment, income, agricultural activity, consumption, and various other areas. The sampling design of NLSS II is two-stage stratified sampling. Nepal is stratified into Primary Sampling Units (PSUs) and, within each PSU, there are a number of households (sub-area) selected. All household members in the sample were interviewed.

In detail, the NLSS II has records for 20,263 individuals from 3912 households (sub-areas) from 326 PSUs (areas) from a population of 60,262 households and about two million Nepalese. A sample of PSUs was selected from the strata using probability proportional to size (PPS) sampling and 12 households were systematically selected from each PSU. The survey is self-weighted and some adjustments were made after conducting the survey for non-responses or missing data. For simplicity, in this paper, we assume all samples have the same weight.

Table 1. Distribution of wards and households in the sample.

Strata	Mountains	Kathemandu	Urban Hill	Rural Hills	Urban Tarai	Rural Tarai	Total
PSU	32	34	28	96	34	102	326
Households	384	408	336	1152	408	1224	3912
Individuals	1949	1954	1467	5755	2104	7034	20,263

shows the distribution of all samples by stratum.

We chose four relevant covariates which can influence health status from the same NLSS II survey for our two-fold logistic regression model. They are age, nativity, sex and religion. We created binary variables for nativity (Indigenous = 1, Non-indigenous = 0) and religion ((Hindu = 1, Non-Hindu = 0), sex (Male = 1, Female = 0).

Table 2. The descriptives of 4 covariates.

Covariates		Frequency	Percentage
Age	0–14	7765	38.32
	15–59	10,951	54.04
	60+	1547	7.64
Gender	Male	9763	48.18
	Female	10,500	51.82
Nativity	Indigeous	11,903	41.25
	Non-Indigous	8,360	58.75
Religion	Hingdu	16,378	80.83
	Non-Hingdu	3385	19.17

shows the details of these four covariates. In the model fitting, we standardized the age covariate. Older age and a child’s age are more vulnerable times than younger age. Indigenous people can have different health statuses from migrated people.

According to the 2001 census data, only about 0.091% of households and only 0.904% of PSU were sampled. The NLSS II was designed to provide reliable estimates only at the stratum level, or even larger areas than the stratum. It cannot give estimates in small areas (PSU or household level) since the sample sizes are too small. Therefore, we need to use statistical models to fit the available data and find reliable estimates in small areas. In our study, we chose the binary variable, health status, from the health section of the questionnaire.

4.2. Numerical Comparison

We used data from NLSS II to illustrate our sub-area logistic regression model. We predicted the household proportions of members in good health for 18,924 households (sampled and non-sampled). Bayesian bootstrap by [36] was applied to get non-sampled auxiliary information. This analysis was based on 1224 sample households from 102 wards (PSUs) in strata 6. Our primary purposes were to show that our model can provide good estimates and to compare the approximate method with the exact method when there are random effects at the household level.

We used Rcpp [37] and RcppArmadillo [38] packages in R [39] to fit the model based on both the approximation, INNA, method and the exact method to this NLSS II dataset. For the INNA method, we began with 10,000 iterations and a burn-in of 1000 and we kept only every ninth sample. Finally, 1000 samples were obtained for constructing the posterior distributions of all the parameters. The exact method was very time-consuming, taking about 30 h to finish. However, the INNA approximation method can get samples in 8 min. When we have a large number of areas or sub-areas, the approximation method will make enormous savings.

Convergence diagnostics were conducted. The convergences of the hyperparameters $(\underset{\tilde{}}{\beta },{\sigma }^2,{\delta }^2)$ were monitored by the Geweke test of stationarity [40] and the effective sample sizes. The p-values and effective sample sizes are shown in

Table 3. Posterior means (PM), associated posterior standard deviations (PSD), Geweke test p-values and effective sample sizes (ESS) of hyperparameters based on the INNA and exact method.

Method	INNA				Exact
Estimator	PM	PSD	p-Value	ESS	PM	PSD	p-Value	ESS
${\beta }_0$	0.802	0.323	0.688	780	0.840	0.405	0.264	1000
${\beta }_1$	0.675	0.015	0.759	857	0.654	0.016	0.156	858
${\beta }_2$	0.302	0.017	0.453	894	0.313	0.019	0.655	1000
${\beta }_3$	−0.862	0.017	0.605	915	−0.824	0.017	0.418	863
${\beta }_4$	−0.338	0.013	0.718	937	−0.360	0.011	0.839	1000
${\sigma }^2$	115.248	35.601	0.408	1000	108.959	38.719	0.615	911
${\delta }^2$	30.002	1.146	0.448	731	29.034	1.697	0.490	780

, resulting in good convergence for both methods. Table 3 also shows the posterior means (PMs) and associated posterior standard deviations (PMs) of the hyperparameters. The PMs are very close between these two methods. The PSDs are slightly larger for the exact method than for the INNA method, but they are reasonably close.

In Figure 1, Figure 2 and Figure 3, we compare the PMs, PSDs and posterior coefficient of variations (PCVs) in the household level as our primary purpose. We can see that the PMs are very close, nearly lying on the 45-degree line through the origin. The PSDs are slightly spread out and thicker, but all points still lie on the 45-degree line and so do the PCVs. Overall, these approximations are acceptable in the data analysis. Figure 4, Figure 5 and Figure 6 we compare respectively to the PMs, PSDs and PCVs at the ward level. The plots of the PMs are still very good. Notice that other two plots of PSDs and PCVs are more spread out than those in the household level. Again, though, the approximate method and the exact method are reasonably close.

We also compare the approximate method with the exact method using the five number summaries (the minimum values, the first quartiles, the median, the third quartiles, and the maximum values) with respect to the PMs, PSDs and PCVs of the finite population proportions at the household level and ward level in

Table 4. Comparison of posterior inference about the finite population proportions using the five-number summaries at the household level.

Households		Min	Q1	Mean	Q3	Max
PM	INNA	0.049	0.541	0.565	0.584	0.981
	Exact	0.050	0.542	0.566	0.584	0.984
PSD	INNA	0.040	0.246	0.441	0.465	0.500
	Exact	0.038	0.248	0.442	0.466	0.501
PCV	INNA	0.041	0.379	0.762	0.852	1.549
	Exact	0.039	0.384	0.768	0.852	1.577

and

Table 5. Comparison of posterior inference about the finite population proportions using five-number summaries at the ward level.

Wards		Min	Q1	Mean	Q3	Max
PM	INNA	0.449	0.537	0.563	0.589	0.684
	Exact	0.450	0.537	0.564	0.590	0.683
PSD	INNA	0.056	0.059	0.063	0.066	0.077
	Exact	0.058	0.061	0.064	0.066	0.077
PCV	INNA	0.095	0.103	0.113	0.121	0.163
	Exact	0.097	0.103	0.113	0.122	0.165

. The PMs from both methods at the household level have larger variations than those at the ward level. The PCVs at the ward level are generally much smaller than at the household level. The summaries of the PMs, PSDs and PCVs within households and wards between the approximate and exact methods are very close.

We conclude that the approximation method at the household level is reasonable. The approximation is desirable because one can perform the computations in real time.

5. Conclusions and Future Work

The sub-area HB logistic regression model can be applied to analyze the binary response variable. This model is an extension of the HB logistic regression area-level model, which ignores the actual hierarchical structure of the data. We propose an approximation method, INNA, to fit the model. For large datasets, it is very unrealistic to use the MCMC method to fit the model. We propose the approximation method, INNA, which saves time significantly because there is no need to compute numerous modes. In the numerical example, we can show that INNA can provide reliable estimates as well. An illustrative example of the NLSS II is presented in order to compare the approximation method and the exact method. It shows that, when there are a large number of areas and sub-areas, the approximation method can be efficient and it can also provide reasonable estimates.

INNA is a method for approximate Bayesian inference based on Laplace’s method, the second-order multivariate Taylor-series approximation and the empirical logistic transform (ELT). It can be applied to all HB logistic regression models, for which it can be a fast and accurate alternative to the Markov chain Monte Carlo methods. The comparison and model results illustrate the performance of the INNA methods based on the sub-area model.

There will be many future works on the two-fold small areas model. First, in this paper, we assume equal survey weights since the NLSS II is a self-weighted sampling. However, after the data are collected, the sampling weights are usually adjusted for various characteristics or based on nonresponse as well. Incorporating those survey weights into the model is also very important. Generally, we need to consider these weights in the model. The NLSS II is a national population-based survey. We should rescale the sample weights to sum to an equivalent sample size. That is, we consider the adjusted weight as $w_{ijk}^{*}=\widehat{n}\left(\frac{w_{ijk}}{{\displaystyle \sum _{i=1}^{\ell }}{\displaystyle \sum _{j=1}^{n_i}}{\displaystyle \sum _{k=1}^{m_{ij}}}{w}_{ijk}}\right)$, where $\widehat{n}=\frac{{\left({\displaystyle \sum _{i=1}^{\ell }}{\displaystyle \sum _{j=1}^{n_i}}{\displaystyle \sum _{k=1}^{m_{ij}}}{w}_{ijk}\right)}^2}{{\displaystyle \sum _{i=1}^{\ell }}{\displaystyle \sum _{j=1}^{n_i}}{\displaystyle \sum _{k=1}^{m_{ij}}}{w}_{ijk}^2}$ as an equivalent sample. Introducing the sampling weights, we can obtain an updated normalized likelihood function. Based on the updated likelihood function and the same prior in the two-fold model, we can have a full Bayesian analysis on the updated model and then project the finite population proportion of the family members with good health in each household.

Second, we focus on the binary data. Actually, there are four options in the health status questionnaire. The Multinomial-Dirichlet model can be an extension of the polychotomous data. Third, the two-fold sub-area level models can also be extended to three-fold models if the data have an additional hierarchical structure; actually, the NLSS II has this structure (households within wards, wards within districts). Fourth, in our models, we consider parametric priors. Introducing the Dirichlet process as a prior might make our method more robust to its specifications.