Explicit Gaussian Variational Approximation for the Poisson Lognormal Mixed Model

Abstract

In recent years, the Poisson lognormal mixed model has been frequently used in modeling count data because it can accommodate both the over-dispersion of the data and the existence of within-subject correlation. Since the likelihood function of this model is expressed in terms of an intractable integral, estimating the parameters and obtaining inference for the parameters are challenging problems. Some approximation procedures have been proposed in the literature; however, they are computationally intensive. Moreover, the existing studies of approximate parameter inference using the Gaussian variational approximation method are usually restricted to models with only one predictor. In this paper, we consider the Poisson lognormal mixed model with more than one predictor. By extending the Gaussian variational approximation method, we derive explicit forms for the estimators of the parameters and examine their properties, including the asymptotic distributions of the estimators of the parameters. Accurate inference for the parameters is also obtained. A real-life example demonstrates the applicability of the proposed method, and simulation studies illustrate the accuracy of the proposed method.

1. Introduction

The variational method of approximating intractable computations has its roots in the calculus of variations and is reviewed in Stephenson [1]. Ormerod and Wang [2] gave an overview of how the variational approximations facilitate approximate inference for the parameters in complex statistical models, and provide a fast and deterministic alternative to the Monte Carlo method. Moreover, Teschendorff et al. [3] and Flandin and Penny [4] applied the variational method to model gene-expression data and fMRI data, respectively. Hall et al. [5] and Hall et al. [6] applied the variational method to approximate the likelihood function for a Poisson linear mixed model with only one predictor and derived the maximum likelihood estimators and their asymptotic distributions.

The main idea of the variational method of approximating the density of a statistic from a complex model is to find a density from a pre-determined family of distributions such that the approximated density and the target density have the smallest Kullback–Leibler divergence. Mathematically, let $q(x)$ be the target density and $p(x)$ be a specific density from a pre-determined family of distributions $\mathcal{G}$. Then, the variational approximation of $q(x)$ is

$$p^{*}(x)={argmin}_{p(x)\in \mathcal{G}}{\int }_{\mathcal{X}}p(x)log\frac{p(x)}{q(x)}dx.$$

$\mathcal{G}$ is the Gaussian family, the variational approximation is referred to as the Gaussian variational approximation (GVA).

Recent advances in GVA include links to Bayesian posterior inference in [7,8] and Bayesian Gaussian graphical model selection [9]; see also other applications to particle inference [10], GVA with a factorial covariance structure [11], and variational approximation in a Poisson-lognormal model [12].

In this paper, we consider the Poisson lognormal mixed model considered in [5] but with more than one predictor in the model. The GVA method is applied to obtain the maximum likelihood estimators of the parameters in the model. The advantages of the proposed method over the method discussed in [5] are:

• The method discussed in [5] can only be applied to the Poisson lognormal mixed model with only one predictor, whereas the proposed method allows the model to have more than one predictor.
• Although the explicit closed forms of the estimators are not available, the structure of the set of estimating equations allows us to easily establish the consistency property of the estimators and derive the limiting distributions of the estimators.

The rest of the paper is organized as follows. Section 2 examines a special member of the exponential family model. For this specific distribution, it is shown that the first three moments can be obtained directly from the normalizing constant. Although the explicit form of the normalizing constant is unavailable, it can be approximated by the GVA method. Section 3 reviews the Poisson lognormal mixed model and shows that the density of the model can be expressed in the form of the specific exponential model introduced in Section 2.

Thus, the likelihood function of the Poisson lognormal mixed model is explicitly available, and the maximum likelihood estimators and their limiting distributions are derived. A real-life example is presented in Section 4 to demonstrate the application of the proposed method. Moreover, results from simulation studies are also presented in Section 4 to illustrate the accuracy of the proposed method. Finally, we give our concluding remarks and discussion in Section 5.

2. A Special Member of the Exponential Family Model

Consider a special exponential family model with density

$$\begin{array}{c}\hfill f(x;\alpha ,\beta ,{\sigma }^2)=C^{-1}(\alpha ,\beta ,{\sigma }^2)exp\left\{\alpha x-\beta e^x-\frac{x^2}{2{\sigma }^2}\right\},-\infty <x<\infty ,\end{array}$$

(1)

where $\alpha ,\beta$ and $\sigma$ are non-negative real numbers, and

$$\begin{array}{c}\hfill C(\alpha ,\beta ,{\sigma }^2)={\int }_{-\infty }^{\infty }exp\left\{\alpha x-\beta e^x-\frac{x^2}{2{\sigma }^2}\right\}dx\end{array}$$

(2)

is the normalizing constant. Then,

$$E(X)=\frac{1}{C(\alpha ,\beta ,{\sigma }^2)}{\int }_{-\infty }^{\infty }xexp\left\{\alpha x-\beta e^x-\frac{x^2}{2{\sigma }^2}\right\}dx=\frac{\partial logC(\alpha ,\beta ,{\sigma }^2)}{\partial \alpha },$$

and

$$E(X^2)=\frac{1}{C(\alpha ,\beta ,{\sigma }^2)}{\int }_{-\infty }^{\infty }{x}^{2}exp\left\{\alpha x-\beta e^x-\frac{x^2}{2{\sigma }^2}\right\}dx=-\frac{\partial logC(\alpha ,\beta ,{\sigma }^2)}{\partial \left[{(2{\sigma }^2)}^{-1}\right]}.$$

The variance of X can be obtained by $Var(X)=E(X^2)-{\left[E(X)\right]}^2$. Following the same argument as above, it can be shown that

$$E(X^3)=-\frac{{\partial }^{2}logC(\alpha ,\beta ,{\sigma }^2)}{\partial \alpha \partial \left[{(2{\sigma }^2)}^{-1}\right]}+E(X)E(X^2)\mathrm{and}E(e^X)=-\frac{\partial logC(\alpha ,\beta ,{\sigma }^2)}{\partial \beta }.$$

Note that, as $\beta \to 0$, the Model (1) converges to the normal model with mean $\alpha {\sigma }^2$ and variance ${\sigma }^2$. Furthermore, as ${\sigma }^2\to \infty$, the Model (1) converges to the log-gamma model with shape $\alpha$ and rate $\beta$.

Although Model (1) is not commonly discussed in the statistics literature, it occurs naturally in information theory. In particular, the concept of cross entropy in information theory, which is generally known as the Kullback–Leibler (KL) divergence in mathematics, is a measure of the closeness of two probability distributions P and Q. If both P and Q are continuous distributions, the cross entropy is defined as

$$\mathrm{KL}(p,q)={\int }_{-\infty }^{\infty }p(x)log\frac{p(x)}{q(x)}dx,$$

where $p(x)$ and $q(x)$ are the density functions of P and Q, respectively. Note that, by Jensen’s inequality, $\mathrm{KL}(p,q)$ is always non-negative.

The variational method is to find $g\in \mathcal{G}$ such that it is closest to Model (1) in terms of having the smallest KL divergence. When $\mathcal{G}$ is restricted to the family of normal distributions with mean ${\mu }_U$ and variance ${\sigma }_U^2$, we have

$$\mathcal{G}=\left\{g(x;{\mu }_U,{\sigma }_U^2)=\frac{1}{\sqrt{2\pi {\sigma }^2}}exp\left\{-\frac{{(x-{\mu }_U)}^2}{2{\sigma }_U^2}\right\},-\infty <{\mu }_U<\infty ,{\sigma }_U^2>0\right\},$$

where

$${\int }_{-\infty }^{\infty }g(x;{\mu }_U,{\sigma }_U^2)dx=1.$$

Mathematically, the variational method gives

$$\begin{array}{ccc}& & \underset{g(x)\in \mathcal{G}}{min}\mathrm{KL}\left(g,f\right)\hfill \\ & =& \underset{({\mu }_U,{\sigma }_U^2)}{min}\mathrm{KL}\left(g(x;{\mu }_U,{\sigma }_U^2),f(x;\alpha ,\beta ,{\sigma }^2)\right)\hfill \\ & =& \underset{({\mu }_U,{\sigma }_U^2)}{min}\left\{{\int }_{-\infty }^{\infty }g(x;{\mu }_U,{\sigma }_U^2)log\frac{g(x;{\mu }_U,{\sigma }_U^2)}{f(x;\alpha ,\beta ,{\sigma }^2)}dx\right\}\hfill \\ & =& \underset{({\mu }_U,{\sigma }_U^2)}{min}\left\{logC(\alpha ,\beta ,{\sigma }^2)-{\int }_{-\infty }^{\infty }g(x;{\mu }_U,{\sigma }_U^2)log\frac{f(x;\alpha ,\beta ,{\sigma }^2)C(\alpha ,\beta ,{\sigma }^2)}{g(x;{\mu }_U,{\sigma }_U^2)}dx\right\}.\hfill \end{array}$$

(3)

Since the KL divergence is always non-negative, the variational method gives

$$\begin{array}{cc}\hfill & arg\underset{({\mu }_U,{\sigma }_U^2)}{max}{E}_U\left\{log\frac{f(U;\alpha ,\beta ,{\sigma }^2)C(\alpha ,\beta ,{\sigma }^2)}{g(U;{\mu }_U,{\sigma }_U^2)}\right\}\hfill \\ \hfill & =arg\underset{({\mu }_U,{\sigma }_U^2)}{max}\alpha {\mu }_U-\beta e^{{\mu }_U+{\sigma }_U^2/2}-\frac{{\sigma }_U^2+{\mu }_U^2}{2{\sigma }^2}+\frac{1}{2}+\frac{1}{2}log(2\pi {\sigma }_U^2),\hfill \end{array}$$

(4)

where $U\sim N({\mu }_U,{\sigma }_U^2)$. Note that Equation (3) indicates that minimizing the KL divergence is the same as minimizing between $logC(\alpha ,\beta ,{\sigma }^2)$ and all its lower bounds by Jensen’s inequality. Moreover, Equation (4) shows that (3) is equivalent to maximizing an expectation with respect to ${\mu }_U$ and ${\sigma }_U^2$.

Since the main idea of GVA is to approximate the log of an intractable integral by a function that is a maximum of all possible lower bounds derived from Jensen’s inequality, and all the possible lower bounds of the log of the normalizing constant of Model (1) can be determined by Jensen’s inequality, with $U\sim N({\mu }_U,{\sigma }_U^2)$, we have

$$\begin{array}{ccc}\hfill logC(\alpha ,\beta ,{\sigma }^2)& =& log\left[{\int }_{-\infty }^{\infty }exp\left\{\alpha u-\beta e^u-\frac{u^2}{2{\sigma }^2}\right\}\frac{exp\{-{(u-{\mu }_U)}^2/(2{\sigma }_U^2)\}/\sqrt{2\pi {\lambda }_U}}{exp\{-{(u-{\mu }_U)}^2/(2{\sigma }_U^2)\}/\sqrt{2\pi {\sigma }_U^2}}\right]du\hfill \\ & =& log{E}_U\left[exp\left\{\alpha U-\beta e^U-\frac{U^2}{2{\sigma }^2}+\frac{{(U-{\mu }_U)}^2}{2{\sigma }_U^2}\right\}\sqrt{2\pi {\sigma }_U^2}\right]\hfill \\ & \ge & E_{U}log\left[exp\left\{\alpha U-\beta e^U-\frac{U^2}{2{\sigma }^2}+\frac{{(U-{\mu }_U)}^2}{2{\sigma }_U^2}\right\}\sqrt{2\pi {\sigma }_U^2}\right]\hfill \\ & =& \alpha {\mu }_U-\beta e^{{\mu }_U+{\sigma }_U^2/2}-\frac{{\sigma }_U^2+{\mu }_U^2}{2{\sigma }^2}+\frac{1}{2}+\frac{1}{2}log(2\pi {\sigma }_U^2),\hfill \end{array}$$

where $-\infty <{\mu }_U<\infty$ and $0<{\sigma }_U^2<\infty$. The Jensen gap between $logC(\alpha ,\beta ,{\sigma }^2)$ and its lower bounds can be further narrowed by maximizing the lower bounds for all values of ${\mu }_U$ and ${\sigma }_U^2$. Finally, we have the following theorem:

Theorem 1.

The log of the normalizing constant $C(\alpha ,\beta ,{\sigma }^2)$ approximated by GVA is

$$\begin{array}{ccc}\hfill \underline{logC(\alpha ,\beta ,{\sigma }^2)}& =& \underset{{\mu }_U,{\sigma }_U^2}{max}\left\{\alpha {\mu }_U-\beta e^{{\mu }_U+{\sigma }_U^2/2}-\frac{{\sigma }_U^2+{\mu }_U^2}{2{\sigma }^2}+\frac{1}{2}+\frac{1}{2}log(2\pi {\sigma }_U^2)\right\}\hfill \\ & =& \alpha {\tilde{\mu }}_U-\beta e^{{\tilde{\mu }}_U+{\tilde{\sigma }}_U^2/2}-\frac{{\tilde{\sigma }}_U^2+{({\tilde{\mu }}_U)}^2}{2{\sigma }^2}+\frac{1}{2}+\frac{1}{2}log(2\pi {\tilde{\sigma }}_U^2)+O({\alpha }^{-2}),\hfill \end{array}$$

(5)

where ${\tilde{\mu }}_U$ and ${\tilde{\sigma }}_U^2$ satisfy

$$\alpha -\beta e^{{\tilde{\mu }}_U+{\tilde{\sigma }}_U^2/2}-\frac{{\tilde{\mu }}_U}{{\sigma }^2}=0\mathrm{and}-\beta e^{{\tilde{\mu }}_U+{\tilde{\sigma }}_U^2/2}-\frac{1}{{\sigma }^2}+\frac{1}{{\tilde{\sigma }}_U^2}=0.$$

In particular, for large α and β with $s=log(\alpha /\beta )=O(1)$,

$$\begin{array}{cc}\hfill \underline{logC(\alpha ,\beta ,{\sigma }^2)}& =\alpha s-\alpha -\frac{1}{2}log\alpha +\frac{1}{2}log(2\pi )-\frac{s^2}{2{\sigma }^2}+\frac{s{\sigma }^2-{\sigma }^2+s^2+({\sigma }^4/6)}{2\alpha {\sigma }^4}+O({\alpha }^{-2}).\hfill \end{array}$$

(6)

It is important to note that the drawback of Equation (5) is that numerical approximation is needed to solve the nonlinear system of the estimated mean and variance. In contrast, Equation (6) has an explicit form with second-order accuracy. A comparison of these two equations is given in Figure 1. The inference for the Poisson lognormal mixed model will be provided in Section 3.

Proof of Theorem 1.

Let

$$\underline{\ell }({\mu }_U,{\sigma }_U^2)=\left\{\alpha {\mu }_U-\beta e^{{\mu }_U+{\sigma }_U^2/2}-\frac{{\sigma }_U^2+{\mu }_U^2}{2{\sigma }^2}+\frac{1}{2}+\frac{1}{2}log(2\pi {\sigma }_U^2)\right\}.$$

If ${\tilde{\mu }}_U$ and ${\tilde{\sigma }}_U^2$ maximize $\underline{\ell }({\mu }_U,{\sigma }_U^2)$, then we have

$${\left.\frac{\partial \underline{\ell }({\mu }_U,{\sigma }_U^2)}{\partial {\mu }_U}\right|}_{({\tilde{\mu }}_U,{\tilde{\sigma }}_U^2)}=0\mathrm{and}{\left.\frac{\partial \underline{\ell }({\mu }_U,{\sigma }_U^2)}{\partial {\sigma }_U^2}\right|}_{({\tilde{\mu }}_U,{\tilde{\sigma }}_U^2)}=0.$$

Hence, ${\tilde{\mu }}_U$ and ${\tilde{\sigma }}_U^2$ satisfy the equations

$$\begin{array}{cc}\hfill \alpha -\beta e^{{\tilde{\mu }}_U+{\tilde{\sigma }}_U^2/2}-\frac{{\tilde{\mu }}_U}{{\sigma }^2}& =0,\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill -\beta e^{{\tilde{\mu }}_U+{\tilde{\sigma }}_U^2/2}-\frac{1}{{\sigma }^2}+\frac{1}{{\tilde{\sigma }}_U^2}& =0.\hfill \end{array}$$

(8)

Therefore, (7) gives $e^{{\tilde{\mu }}_U+{\tilde{\sigma }}_U^2/2}=\frac{\alpha }{\beta }-\frac{{\tilde{\mu }}_U}{\beta {\sigma }^2}$ and ${\tilde{\mu }}_U=O(1)$. Furthermore,

$$\begin{array}{c}\hfill {\tilde{\mu }}_U=-{\tilde{\sigma }}_U^2/2+log\left(\frac{\alpha }{\beta }-\frac{{\tilde{\mu }}_U}{\beta {\sigma }^2}\right)=-{\tilde{\sigma }}_U^2/2+log\left(\frac{\alpha }{\beta }\right)-\left(\frac{{\tilde{\mu }}_U}{\alpha {\sigma }^2}\right)-\left(\frac{{\tilde{\mu }}_U^2}{2{\alpha }^2{\sigma }^4}\right)+O({\alpha }^{-3}).\end{array}$$

(9)

Substituting $e^{{\tilde{\mu }}_U+{\tilde{\sigma }}_U^2/2}$ obtained from (7) into (8) and solving it for ${\tilde{\sigma }}_U^2$, we have

$$\begin{array}{cc}\hfill & \frac{1}{{\tilde{\sigma }}_U^2}=\alpha +\frac{1}{{\sigma }^2}-\frac{{\tilde{\mu }}_U}{{\sigma }^2},\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & {\tilde{\sigma }}_U^2=\frac{1}{\alpha }+\frac{{\tilde{\mu }}_U-1}{{\alpha }^2{\sigma }^2}+O({\alpha }^{-3}),\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & log{\tilde{\sigma }}_U^2=-log\alpha +\frac{{\tilde{\mu }}_U-1}{\alpha {\sigma }^2}+O({\alpha }^{-2}).\hfill \end{array}$$

(12)

By substituting ${\tilde{\sigma }}_U^2$ in (11) back into (9), we have

$${\tilde{\mu }}_U=log\left(\frac{\alpha }{\beta }\right)-\frac{{\sigma }^2+2log(\alpha /\beta )}{2\alpha {\sigma }^2}+O({\alpha }^{-2}).$$

(13)

Finally, replacing ${\tilde{\mu }}_U$ and ${\tilde{\sigma }}_U^2$ in $\underline{logC(a,b,{\sigma }^2)}$ by (9) and (11), respectively, we have

$$\begin{array}{cc}\hfill \underline{logC(\alpha ,\beta ,{\sigma }^2)}& =-\alpha {\tilde{\sigma }}_U^2/2+\alpha log\left(\frac{\alpha }{\beta }\right)-\left(\frac{{\tilde{\mu }}_U}{{\sigma }^2}\right)-\left(\frac{{\tilde{\mu }}_U^2}{2\alpha {\sigma }^4}\right)+O({\alpha }^{-2})\hfill \\ \hfill & =-\frac{1}{2}-\frac{{\tilde{\mu }}_U-1}{2\alpha {\sigma }^2}+\alpha log\left(\frac{\alpha }{\beta }\right)-\left(\frac{{\tilde{\mu }}_U}{{\sigma }^2}\right)-\left(\frac{{\tilde{\mu }}_U^2}{2\alpha {\sigma }^4}\right)+\frac{1}{2}+\frac{1}{2}log(2\pi {\tilde{\sigma }}_U^2)+O({\alpha }^{-2}).\hfill \end{array}$$

(14)

With $\alpha$ and $\beta$ large and $s=log(\alpha /\beta )=O(1)$, by substituting ${\tilde{\mu }}_U$ with (13) and ${\tilde{\sigma }}_U^2$ with (12) into (14), we have

$$\begin{array}{cc}\hfill \underline{logC(\alpha ,\beta ,{\sigma }^2)}& =\alpha s-\alpha -\frac{1}{2}log\alpha +\frac{1}{2}log(2\pi )-\frac{s^2}{2{\sigma }^2}+\frac{s{\sigma }^2-{\sigma }^2+s^2+({\sigma }^4/6)}{2\alpha {\sigma }^4}+O({\alpha }^{-2}).\hfill \end{array}$$

(15)

□

To demonstrate the difference in the numerical accuracy of the GVA, we consider the density $f(x;\alpha ,\beta ,{\sigma }^2)$ as stated in (1) with $\alpha =5,\beta =5,$ and $\sigma =0.5,1,\dots ,10$. The exact $C(\alpha ,\beta ,{\sigma }^2)$ is obtained by numerical integration. The plot of the result is given in Figure 1.

From Figure 1, we observe that the results of GVA by (5) are accurate for small $\sigma$, while GVA by (6) gives a good approximation for large $\sigma$. Overall, Equation (6) is a better explicit approximation compared with the numerical approximation by Equation (5).

3. GVA for Poisson Lognormal Mixed Model

In this section, we apply the results from Theorem 1 to obtain the estimates of the parameters in a Poisson lognormal mixed model. The estimates are simpler to obtain than the ones in Hall et al. [5,6].

Let $Y_{it}$, which takes on non-negative integer values, be the number of occurrences of an event for subject i at time t, where $i=1,\dots ,m$ and $t=1,\dots ,n$. Moreover, let ${\mathit{X}}_{it}=(X_{it}^1,X_{it}^2,\dots ,X_{it}^p)$ where $X_{it}^j$ is the $j^{th}$ power of the covariate for subject i at time t, $X_{it}$. Then, the conditional Poisson model is

$$\begin{array}{c}\hfill Y_{it}|{\mathit{X}}_{it}\sim \mathrm{Poisson}({\lambda }_{it})\end{array}$$

(16)

where ${\lambda }_{it}=exp\left({\sum }_{j=0}^p{\beta }_j{X}_{it}^j\right)$ is the mean of the conditional Poisson model, $X_{it}^0\equiv 1$, and ${\beta }_0,{\beta }_1,\dots ,{\beta }_p$ are the regression coefficients. An unobservable latent random variable $U_i$ for the $i^{th}$ subject is introduced into Model (16) to capture the within-subject correlation. The resulting model, which is known as the Poisson generalized linear mixed model, can be written as

$$\begin{array}{c}\hfill Y_{it}|({\mathit{X}}_{it},U_i)\sim \mathrm{Poisson}({\lambda }_{it})\end{array}$$

(17)

where the mean of the model is ${\lambda }_{it}=exp\left(U_i+{\sum }_{j=0}^p{\beta }_j{X}_{it}^j\right)$, and $U_i$ and $X_{it}$ are assumed to be independently distributed. Furthermore, if $U_1,\dots ,U_m$ are assumed to be identical and independently distributed as $N(0,{\sigma }^2$), the model is referred to as the Poisson lognormal mixed model, though [5] referred to it as the Poisson linear mixed model. McCulloch et al. [13] gave a detailed discussion of the connections between the Poisson lognormal mixed model, and the model is studied in longitudinal data analysis.

It is challenging to obtain the estimators of the regression coefficients, ${\beta }_0,\dots ,{\beta }_p$, and ${\sigma }^2$, because it involves the unobservable latent variables $U_i$. Rue et al. [14] suggested eliminating the effect of the unobservable latent variables by using the marginal log-likelihood function, which can be written as

$$\begin{array}{cc}\hfill \ell (\mathit{\beta },{\sigma }^2)& =\sum _{i=1}^m\sum _{t=1}^n\{Y_{it}({\beta }_0+{\beta }_1{X}_{it}+{\beta }_2{X}_{it}^2+\dots +{\beta }_p{X}_{it}^p)-log(Y_{it}!)\}-\frac{m}{2}log(2\pi {\sigma }^2)\hfill \\ \hfill & +\sum _{i=1}^{m}log{\int }_{-\infty }^{+\infty }exp\left(\sum _{t=1}^n\left(Y_{it}u-e^{{\beta }_0+{\beta }_1{X}_{it}+{\beta }_2{X}_{it}^2+\dots +{\beta }_p{X}_{it}^p}\right)-\frac{u^2}{2{\sigma }^2}\right)du.\hfill \end{array}$$

(18)

The major challenge to developing likelihood-based inferences for the parameters is hindered by the last term in (18), which involves the summation of m integrals. Hall et al. [5] applied the GVA method to overcome such integration problems. They considered only the model with one covariate $(p=1)$ and the resulting estimating equations involved $2m$ nuisance parameters. Due to the complexity of their method, it is difficult, if not impossible, to extend to models with more than one covariate $(p>1)$.

Using the notations introduced in Section 2, the marginal log-likelihood function (18) can be rewritten as

$$\begin{array}{c}\hfill \ell (\mathit{\beta },{\sigma }^2)=\sum _{i=1}^m\left\{logC(a_{in},b_{in},{\sigma }^2)+\sum _{t=1}^n{Y}_{it}\sum _{j=0}^p{\beta }_j{X}_{it}^j-\frac{log({\sigma }^2)}{2}\right\}+{\ell }_0(Y),\end{array}$$

(19)

where

$${\ell }_0(Y)=-\sum _{i=1}^m\sum _{t=1}^{n}log(Y_{it}!)-\frac{m}{2}log(2\pi ),$$

$$a_{in}=\sum _{t=1}^n{Y}_{it}\mathrm{and}{b}_{in}=b_{in}({\beta }_0,\dots ,{\beta }_p)=\sum _{t=1}^{n}exp\left(\sum _{j=0}^p{\beta }_j{X}_{it}^j\right).$$

From (6), we have

$$\sum _{i=1}^{m}logC(a_{in},b_{in},{\sigma }^2)\approx \sum _{i=1}^m\left\{-\frac{{log}^2(a_{in}/b_{in})}{2{\sigma }^2}-a_{in}log(b_{in})\right\}+{\ell }_1(Y),$$

where ${\ell }_1(Y)={\sum }_{i=1}^m\{a_{in}log{a}_{in}-a_{in}-log(a_{in})/2+log(2\pi )/2\}$. Therefore, the log-likelihood function (19) can be approximated by

$$\ell (\mathit{\beta },{\sigma }^2)\approx \sum _{i=1}^m\left\{-\frac{{log}^2(a_{in}/b_{in})}{2{\sigma }^2}-a_{i,n}log(b_{in})+\sum _{t=1}^n{Y}_{it}\sum _{j=0}^p{\beta }_j{X}_{it}^j-\frac{log({\sigma }^2)}{2}\right\}+{\ell }_0(Y)+{\ell }_1(Y).$$

For $j=1,\dots ,p$, denote

$$b_{in}^{(j)}=b_{in}^{(j)}({\beta }_0,\dots ,{\beta }_p)=\frac{\partial b_{in}}{\partial {\beta }_j}=\sum _{t=1}^n{X}_{it}^{j}exp\left(\sum _{k=0}^p{\beta }_k{X}_{it}^k\right),$$

and $b_{in}^{(0)}=b_{in}$. Hence, the partial derivatives of the log-likelihood function with respect to ${\beta }_0,\dots ,{\beta }_p$ and ${\sigma }^2$ are

$$\begin{array}{cc}\hfill \frac{\partial \ell (\mathit{\beta },{\sigma }^2)}{\partial {\beta }_j}& \approx \sum _{i=1}^m\left\{\left(\sum _{t=1}^n{Y}_{it}{X}_{it}^j-a_{in}\frac{b_{in}^{(j)}}{b_{in}}\right)+log\left(\frac{a_{in}}{b_{in}}\right)\frac{b_{in}^{(j)}}{{\sigma }^2{b}_{in}}\right\},j=0,\dots ,p,\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill \frac{\partial \ell (\mathit{\beta },{\sigma }^2)}{\partial {\sigma }^2}& \approx \sum _{i=1}^m\left\{\frac{{log}^2(a_{in}/b_{in})}{2{\sigma }^4}-\frac{1}{2{\sigma }^2}\right\}.\hfill \end{array}$$

(21)

Note that

$$\frac{b_{in}^{(j)}}{b_{in}}=\left\{\begin{array}{ccc}1\hfill & & \mathrm{for}j=0\hfill \\ \hfill & & \\ \frac{{\sum }_{s=1}^n{X}_{is}^{j}exp({\sum }_{k=1}^p{\beta }_k{X}_{is}^k)}{{\sum }_{s=1}^{n}exp({\sum }_{k=1}^p{\beta }_k{X}_{is}^k)}\hfill & & \mathrm{otherwise}\hfill \end{array}\right.$$

and $log(b_{in})={\beta }_0+log\{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)\}$. Since the maximum likelihood estimators $\widehat{\mathit{\beta }}$ and ${\widehat{\sigma }}^2$ must satisfy

$${\left.\frac{\partial \ell (\mathit{\beta },{\sigma }^2)}{\partial {\beta }_j}\right|}_{(\widehat{\mathit{\beta }},{\widehat{\sigma }}^2)}=0\mathrm{and}{\left.\frac{\partial \ell (\mathit{\beta },{\sigma }^2)}{\partial {\sigma }^2}\right|}_{(\widehat{\mathit{\beta }},{\widehat{\sigma }}^2)}=0,$$

we have

$$\begin{array}{cc}\hfill {\widehat{\beta }}_0& =\frac{1}{m}\sum _{i=1}^{m}log\left\{\frac{{\sum }_{t=1}^n{Y}_{it}}{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\widehat{\beta }}_k{X}_{it}^k)}\right\},\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill \sum _{i=1}^m\sum _{t=1}^n{Y}_{it}{X}_{it}^j& =\sum _{i=1}^m\left\{\left(\sum _{t=1}^n{Y}_{it}\right)\frac{{\sum }_{s=1}^n{X}_{is}^{j}exp({\sum }_{k=1}^p{\widehat{\beta }}_k{X}_{is}^k)}{{\sum }_{s=1}^{n}exp({\sum }_{k=1}^p{\widehat{\beta }}_k{X}_{is}^k)}\right\},\hfill \end{array}$$

(23)

and

$$\begin{array}{c}\hfill {\widehat{\sigma }}^2=\frac{1}{m}\sum _{i=1}^m{log}^2\left\{\frac{{\sum }_{t=1}^n{Y}_{it}}{{\sum }_{t=1}^{n}exp({\sum }_{k=0}^p{\widehat{\beta }}_k{X}_{it}^k)}\right\}.\end{array}$$

(24)

Although the maximum likelihood estimators of ${\widehat{\beta }}_k$ do not have a closed form solution, Equation (23) is well-structured enough that numerical procedure, such as the Newton–Raphson method or the EM algorithm can be applied to obtain the numerical solution.

To study the asymptotic properties of the maximum likelihood estimators, we need to make the following three assumptions.

Assumption 1.

Let ${\mathit{\xi }}_{mn}={({\xi }_{mn1},\dots ,{\xi }_{mnp})}^T{\to }_{d}N(\mathbf{0},\mathbf{B})$ as $mn\to \infty$, where

$${\xi }_{mnj}={(mn)}^{-1/2}\sum _{i=1}^m\sum _{t=1}^n(Y_{it}-{\lambda }_{it})(X_{it}^j-r_j),j=1,\dots ,p,$$

$X_{it}$’s are independent and have the same distribution as X whose moment generating function (MGF), $M_X(s)=E\{exp(sX)\}$, is well-defined on the whole real line. ${\lambda }_{it}=E(Y_{it}|X_{it},U_i)$, $r_j$ and $B_{j_1,j_2}$ (entry of $\mathit{B}$) are defined as

$$r_j=r_j({\beta }_1,\dots ,{\beta }_p)=E(Y_{it}{X}_{it}^j)/E(Y_{it})={\varphi }_j/{\varphi }_0,$$

where

$$\begin{array}{ccc}\hfill {\varphi }_0& =& E\{exp(\sum _{k=1}^p{\beta }_k{X}^k)\}\hfill \\ \hfill {\varphi }_j& =& {\varphi }_j({\beta }_1,\dots {\beta }_p)=E\{X^{j}exp(\sum _{k=1}^p{\beta }_k{X}^k)\}\mathrm{for}j=1,\dots ,p\hfill \end{array}$$

and

$$B_{j_1,j_2}=\mathit{cov}\{(Y_{it}-{\lambda }_{it})(X_{it}^{j_1}-r_{j_1}),(Y_{it}-{\lambda }_{it})(X_{it}^{j_2}-r_{j_2})\},1\le j_1,j_2\le p.$$

Assumption 2.

For $j=1,\dots ,p$, ${\sum }_{i=1}^m{\sum }_{t=1}^n(Y_{it}-{\lambda }_{it})(r_j-b_{in}^{(j)}/b_{in})=o_{\mathrm{p}}(\sqrt{mn})$.

Assumption 3.

For $j=1,\dots ,p$,

$${\widehat{b}}_{in}^{(j)}/{\widehat{b}}_{in}-b_{in}^{(j)}/b_{in}=\sum _{k=1}^p{\delta }_{jk}({\widehat{\beta }}_k-{\beta }_k)+o_{\mathrm{p}}(\sum _{k=1}^p|{\widehat{\beta }}_k-{\beta }_k|)$$

where ${\delta }_{jk}=r_{j+k}-r_j{r}_k={\delta }_{jk}({\beta }_1,\dots ,{\beta }_k)$, ${\widehat{b}}_{in}^{(j)}=\partial b_{in}/\partial {\beta }_j=b_{in}^{(j)}(0,{\widehat{\beta }}_1,\dots ,{\widehat{\beta }}_p)$, ${\widehat{b}}_{in}={\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)=b_{in}(0,{\widehat{\beta }}_1,\dots ,{\widehat{\beta }}_p)$, and ${\widehat{\beta }}_1,\dots ,{\widehat{\beta }}_p$ are solutions obtained from (22).

We give a few remarks on each assumption.

Remark 1.

Assumption 1 is implied by the Central Limit Theorem. Thus, we only need to calculate the moments about the expectation and covariance.

$$\begin{array}{cc}\hfill E\left\{(Y_{it}-{\lambda }_{it})(X_{it}^j-r_j)\right\}& =E\left[E\left\{(Y_{it}-{\lambda }_{it})(X_{it}^j-r_j)|X_{it},U_i\right\}\right]\hfill \\ \hfill & =E\left[(X_{it}^j-r_j)E\left\{(Y_{it}-{\lambda }_{it})|X_{it},U_i\right\}\right]\hfill \\ \hfill & =0.\hfill \end{array}$$

Similarly, from the definitions of $r_j$ and ${\varphi }_j$, we have

$$B_{j_1,j_2}=B_{j_1,j_2}({\beta }_0,\dots ,{\beta }_p,{\sigma }^2)=exp({\beta }_0+\frac{1}{2}{\sigma }^2)E\{(X^{j_1}-r_{j_1})(X^{j_2}-r_{j_2})exp(\sum _{k=1}^p{\beta }_k{X}^k)\}.$$

To be more specific, when $p=1$,

$$B_{1,1}=exp({\beta }_0+\frac{1}{2}{\sigma }^2)({\varphi }_2-{\varphi }_1^2/{\varphi }_0).$$

When $p=2$,

$$\begin{array}{cc}\hfill & B_{1,2}=exp({\beta }_0+\frac{1}{2}{\sigma }^2)({\varphi }_3-{\varphi }_1{r}_2-{\varphi }_2{r}_1+r_1{r}_2)=exp({\beta }_0+\frac{1}{2}{\sigma }^2)({\varphi }_3-{\varphi }_1{\varphi }_2/{\varphi }_0),\hfill \\ \hfill & B_{2,2}=exp({\beta }_0+\frac{1}{2}{\sigma }^2)({\varphi }_4-2{\varphi }_2{r}_2+{\varphi }_0{r}_2^2)=exp({\beta }_0+\frac{1}{2}{\sigma }^2)({\varphi }_4-{\varphi }_2^2/{\varphi }_0).\hfill \end{array}$$

Remark 2.

In Assumption 2, $E\left\{(Y_{it}-{\lambda }_{it})(r_j-b_{in}^{(j)}/b_{in})\right\}=0$, and thus the variance is

$$\begin{array}{cc}\hfill & \left\{\sum _{i=1}^m\sum _{t=1}^n(Y_{it}-{\lambda }_{it})(r_j-b_{in}^{(j)}/b_{in})\right\}\hfill \\ \hfill & =m\times \left\{\sum _{t=1}^n(Y_{it}-{\lambda }_{it})(r_j-b_{in}^{(j)}/b_{in})\right\}\hfill \\ \hfill & =m\times E\left[{(r_j-b_{in}^{(j)}/b_{in})}^{2}Var\left\{\sum _{t=1}^n(Y_{it}-{\lambda }_{it})\right\}\right]\hfill \\ \hfill & =mn\times E\left\{{\lambda }_{it}{(r_j-b_{in}^{(j)}/b_{in})}^2\right\}.\hfill \end{array}$$

On account of Markov’s inequality, we need to show $E\left\{{\lambda }_{it}{(r_j-b_{in}^{(j)}/b_{in})}^2\right\}=o(1).$ Note that $r_j=E(b_{in}^{(j)})/E(b_{in})$. Derive the second-order Taylor expansion of the ratio $b_{in}^{(j)}/b_{in}$ about the value $(E(b_{in}^{(j)}),E(b_{in}))$ and use it to obtain $E\left\{{\lambda }_{it}{(r_j-b_{in}^{(j)}/b_{in})}^2\right\}=O(n^{-1})$.

Remark 3.

In Assumption 3, we obtain the $p^{th}$-order Taylor expansion of ${\widehat{b}}_{in}^{(j)}$ and ${\widehat{b}}_{in}$ about the value $({\beta }_1,\dots ,{\beta }_p)$, which yields

$$\begin{array}{cc}\hfill \frac{{\widehat{b}}_{in}^{(j)}}{{\widehat{b}}_{in}}-\frac{b_{in}^{(j)}}{b_{in}}& =\frac{{\sum }_{t=1}^n{X}_{it}^{j}exp({\sum }_{k=1}^p{\widehat{\beta }}_k{X}_{it}^k)}{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\widehat{\beta }}_k{X}_{it}^k)}-\frac{{\sum }_{t=1}^n{X}_{it}^{j}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)}{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)}\hfill \\ \hfill & =\frac{{\sum }_{t=1}^n{X}_{it}^{j}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)+{\sum }_{s=1}^p{\sum }_{t=1}^n{X}_{it}^{j+s}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)({\widehat{\beta }}_s-{\beta }_s)}{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)+{\sum }_{s=1}^p{\sum }_{t=1}^n{X}_{it}^{s}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)({\widehat{\beta }}_s-{\beta }_s)}\hfill \\ \hfill & -\frac{{\sum }_{t=1}^n{X}_{it}^{j}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)}{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)}+o_p(\sum _{k=1}^p|{\widehat{\beta }}_k-{\beta }_k|)\hfill \\ \hfill & =\frac{{\sum }_{s=1}^p{\sum }_{t=1}^n{X}_{it}^{j+s}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)({\widehat{\beta }}_s-{\beta }_s)}{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)}\hfill \\ \hfill & -\frac{{\sum }_{t=1}^n{X}_{it}^{j}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k){\sum }_{s=1}^p{\sum }_{t=1}^n{X}_{it}^{s}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)({\widehat{\beta }}_s-{\beta }_s)}{{\{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\beta }_k{X}_{it}^k)\}}^2}\hfill \\ \hfill & +o_{\mathrm{p}}\left(\sum _{k=1}^p|{\widehat{\beta }}_k-{\beta }_k|\right)\hfill \\ \hfill & =\sum _{s=1}^p(r_{j+s}-r_j{r}_s)({\widehat{\beta }}_s-{\beta }_s)+o_{\mathrm{p}}\left(\sum _{k=1}^p|{\widehat{\beta }}_k-{\beta }_k|\right).\hfill \end{array}$$

If $p=1$, ${\delta }_{11}=r_2-r_1^2$. Furthermore, we have ${\rho }_{11}={\delta }_{11}^{-1}/E(Y_{it})$, where

$$E(Y_{it})=exp({\beta }_0+\frac{1}{2}{\sigma }^2)E\{exp(\sum _{k=1}^p{\beta }_k{X}^k)\},\mathrm{and}{\gamma }_1={\rho }_{11}^2{B}_{11}=\frac{exp(-{\beta }_0-\frac{1}{2}{\sigma }^2)}{r_2-r_1^2}.$$

Note that ${\gamma }_1$ agrees with Equation (3.5) in Theorem 3.1 of Hall et al. [5].

If $p=2$,

$$\Lambda =\left(\begin{array}{cc}{\delta }_{11}& {\delta }_{12}\\ {\delta }_{21}& {\delta }_{22}\end{array}\right)=\left(\begin{array}{cc}{r}_2-r_1^2& r_3-r_1{r}_2\\ r_3-r_1{r}_2& r_4-r_2^2\end{array}\right),$$

$$\frac{{\Lambda }^{-1}}{E(Y_{it})}=\left(\begin{array}{cc}{\rho }_{11}& {\rho }_{12}\\ {\rho }_{21}& {\rho }_{22}\end{array}\right)=\frac{exp(-{\beta }_0-\frac{1}{2}{\sigma }^2){\varphi }_0^{-1}}{(r_2-r_1^2)(r_4-r_2^2)-{(r_3-r_1{r}_2)}^2}\left(\begin{array}{cc}{r}_4-r_2^2& r_1{r}_2-r_3\\ r_1{r}_2-r_3& r_2-r_1^2\end{array}\right).$$

Furthermore, we have

$${\gamma }_1={\rho }_{11}^2{B}_{11}+2{\rho }_{11}{\rho }_{12}{B}_{12}+{\rho }_{12}^2{B}_{22},$$

$${\gamma }_2={\rho }_{21}^2{B}_{11}+2{\rho }_{21}{\rho }_{22}{B}_{21}+{\rho }_{22}^2{B}_{22}.$$

After tedious calculations, we obtain the simplified versions of ${\gamma }_1$ and ${\gamma }_2$:

$$\begin{array}{cc}\hfill & {\gamma }_1=\frac{exp(-{\beta }_0-\frac{1}{2}{\sigma }^2){\varphi }_0({\varphi }_4{\varphi }_0-{\varphi }_2^2)}{({\varphi }_2{\varphi }_0-{\varphi }_1^2)({\varphi }_4{\varphi }_0-{\varphi }_2^2)-{({\varphi }_3{\varphi }_0-{\varphi }_1{\varphi }_2)}^2},\hfill \\ \hfill & {\gamma }_2=\frac{exp(-{\beta }_0-\frac{1}{2}{\sigma }^2){\varphi }_0({\varphi }_2{\varphi }_0-{\varphi }_1^2)}{({\varphi }_2{\varphi }_0-{\varphi }_1^2)({\varphi }_4{\varphi }_0-{\varphi }_2^2)-{({\varphi }_3{\varphi }_0-{\varphi }_1{\varphi }_2)}^2}.\hfill \end{array}$$

Theorem 2.

With Assumptions 1, 2, and 3, for $j=1,\dots ,p$,

$$\sqrt{mn}({\widehat{\beta }}_j-{\beta }_j)=\sum _{k=1}^p{\rho }_{jk}{\xi }_{mnk}+o_{\mathrm{p}}(1),$$

where ${\widehat{\beta }}_j$’s are the solutions obtained from (22), ${\rho }_{jk}={\rho }_{jk}({\beta }_1,\dots ,{\beta }_p)$ is the jth row and kth column of $p\times p$ matrix ${\Lambda }^{-1}/E(Y_{it})$ with $\Lambda =({\delta }_{jk})$, and $E(Y_{it})={\lambda }_{it}=exp\left(U_i+{\sum }_{j=0}^p{\beta }_j{X}_{it}^j\right)$. Thus, as $m,n\to \infty$, ${\widehat{\beta }}_j$ is a consistent estimator with asymptotic distribution of ${\widehat{\beta }}_j$ is

$$\sqrt{mn}({\widehat{\beta }}_j-{\beta }_j){\to }_{\mathrm{d}}N(0,{\gamma }_j)j=1,\dots ,p$$

where ${\gamma }_j={\sum }_{k_1=1}^p{\sum }_{k_2=1}^p{\rho }_{j{k}_1}{\rho }_{j{k}_2}{B}_{k_1,k_2}$. Moreover, ${\widehat{\beta }}_0$ is a consistent estimator with asymptotic distribution

$$\sqrt{m}({\widehat{\beta }}_0-{\beta }_0){\to }_{d}N(0,{\sigma }^2),$$

where ${\widehat{\beta }}_0$ is the solution of ${\beta }_0$ provided in (23). Finally, ${\widehat{\sigma }}^2$ is also a consistent estimator with the asymptotic distribution is

$$\sqrt{m}({\widehat{\sigma }}^2-{\sigma }^2){\to }_{\mathrm{d}}N(0,2{\sigma }^4),$$

where ${\widehat{\sigma }}^2$ is the solution presented in (24).

Proof of Theorem 2.

From (23), we consider the following algebraic manipulation:

$$\begin{array}{cc}\hfill 0& =\sum _{i=1}^m\sum _{t=1}^n(Y_{it}-{\lambda }_{it}+{\lambda }_{it})\left\{X_{it}^j-r_j+r_j-\frac{b_{in}^{(j)}}{b_{in}}+\frac{b_{in}^{(j)}}{b_{in}}-\frac{{\widehat{b}}_{in}^{(j)}}{{\widehat{b}}_{in}}\right\}\hfill \\ \hfill & =\sum _{i=1}^m\sum _{t=1}^n(Y_{it}-{\lambda }_{it})(X_{it}^j-r_j)+\sum _{i=1}^m\sum _{t=1}^n(Y_{it}-{\lambda }_{it})\left\{r_j-\frac{b_{in}^{(j)}}{b_{in}}\right\}+\sum _{i=1}^m\sum _{t=1}^n{Y}_{it}\left\{\frac{b_{in}^{(j)}}{b_{in}}-\frac{{\widehat{b}}_{in}^{(j)}}{{\widehat{b}}_{in}}\right\}\hfill \\ \hfill & +\sum _{i=1}^m\sum _{t=1}^n{\lambda }_{it}\left\{X_{it}^j-\frac{b_{in}^{(j)}}{b_{in}}\right\},\hfill \end{array}$$

where the first term is $\sqrt{mn}{\xi }_{mnj}$, the second term is $o_{\mathrm{p}}(\sqrt{mn})$, and the last term is zero.

Consequently,

$$\begin{array}{c}\hfill \sum _{i=1}^m\sum _{t=1}^n{Y}_{it}\left\{\frac{{\widehat{b}}_{in}^{(j)}}{{\widehat{b}}_{in}}-\frac{b_{in}^{(j)}}{b_{in}}\right\}=\sqrt{mn}{\xi }_{mnj}+o_{\mathrm{p}}(\sqrt{mn}).\end{array}$$

As ${(mn)}^{-1}{\sum }_{i=1}^m{\sum }_{t=1}^n{Y}_{it}{\to }_{\mathrm{p}}E(Y_{it})$ and ${(mn)}^{-1}\{\sqrt{mn}{\xi }_{mnj}+o_{\mathrm{p}}(\sqrt{mn})\}=o_{\mathrm{p}}(1)$, ${\widehat{b}}_{in}^{(j)}/{\widehat{b}}_{in}-b_{in}^{(j)}/b_{in}{\to }_{\mathrm{p}}0$ for each $j>0$. By the continuity, ${\widehat{\beta }}_j{\to }_{\mathrm{p}}{\beta }_j$ for each $j>0$. This shows that the estimators are consistent. In addition,

$$\sqrt{mn}\sum _{k=1}^p{\delta }_{jk}({\widehat{\beta }}_k-{\beta }_k)=\frac{{\xi }_{mnj}}{E(Y_{it})}+o_{\mathrm{p}}(1).$$

This gives the asymptotic result below by the definition of ${\rho }_{jk}$,

$$\sqrt{mn}({\widehat{\beta }}_j-{\beta }_j)=\sum _{k=1}^p{\rho }_{jk}{\xi }_{mnk}+o_{\mathrm{p}}(1).$$

Therefore, by Central Limit Theorem, we have

$$\sqrt{mn}({\widehat{\beta }}_j-{\beta }_j){\to }_{\mathrm{d}}N(0,{\gamma }_j)$$

where ${\gamma }_j={\sum }_{k_1=1}^p{\sum }_{k_2=1}^p{\rho }_{j{k}_1}{\rho }_{j{k}_2}{B}_{k_1,k_2}$.

Moreover, we have

$$\frac{1}{m}\sum _{i=1}^n(U_i+{\beta }_0)={\beta }_0+m^{-1/2}{\zeta }_{m1},$$

where ${\zeta }_{m1}{\to }_{\mathrm{d}}N(0,{\sigma }^2)$. From the estimator of ${\beta }_0$ in (22),

$${\widehat{\beta }}_0=\frac{1}{m}\sum _{i=1}^{m}log\left\{\frac{{\sum }_{t=1}^n{Y}_{it}}{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\widehat{\beta }}_k{X}_{it}^k)}\right\}+\frac{1}{m}\sum _{i=1}^n(U_i+{\beta }_0).$$

Since

$$\begin{array}{cc}\hfill & log\left\{\frac{{\sum }_{t=1}^n{Y}_{it}}{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\widehat{\beta }}_k{X}_{it}^k)}\right\}-(U_i+{\beta }_0)\hfill \\ \hfill & =log\left\{\frac{{\sum }_{t=1}^n{Y}_{it}}{exp(U_i+{\beta }_0){\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\widehat{\beta }}_k{X}_{it}^k)}\right\}=o_{\mathrm{p}}(1),\hfill \end{array}$$

(25)

we have

$$\frac{1}{m}\sum _{i=1}^{m}log\left\{\frac{{\sum }_{t=1}^n{Y}_{it}}{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\widehat{\beta }}_k{X}_{it}^k)}\right\}-\frac{1}{m}\sum _{i=1}^n(U_i+{\beta }_0)=o_{\mathrm{p}}(1).$$

Therefore, ${\widehat{\beta }}_0-{\beta }_0=o_{\mathrm{p}}(1)$, and hence ${\widehat{\beta }}_0$ is a consistent estimator. Moreover, by the Central Limit Theorem, we have

$$\sqrt{m}({\widehat{\beta }}_0-{\beta }_0){\to }_{d}N(0,{\sigma }^2).$$

From the estimator of ${\sigma }^2$ in (24), combing the consistence of ${\widehat{\beta }}_0$ and (25), we have

$$\begin{array}{cc}\hfill {\widehat{\sigma }}^2& =\frac{1}{m}\sum _{i=1}^m{\left[log\left\{\frac{{\sum }_{t=1}^n{Y}_{it}}{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\widehat{\beta }}_k{X}_{it}^k)}\right\}-{\widehat{\beta }}_0\right]}^2\hfill \\ \hfill & =\frac{1}{m}\sum _{i=1}^m{\left[log\left\{\frac{{\sum }_{t=1}^n{Y}_{it}}{{\sum }_{t=1}^{n}exp({\sum }_{k=1}^p{\widehat{\beta }}_k{X}_{it}^k)}\right\}-(U_i+{\beta }_0)+({\beta }_0-{\widehat{\beta }}_0)+\{(U_i+{\beta }_0)-{\beta }_0\}\right]}^2\hfill \\ \hfill & =\frac{1}{m}\sum _{i=1}^m{\left\{(U_i+{\beta }_0)-{\beta }_0\right\}}^2+o_{\mathrm{p}}(1)\hfill \\ \hfill & =\frac{1}{m}\sum _{i=1}^m{U}_i^2+o_{\mathrm{p}}(1)\hfill \\ \hfill & ={\sigma }^2+m^{-1/2}{\zeta }_{m2}+o_{\mathrm{p}}(1),\hfill \end{array}$$

where ${\zeta }_{m2}{\to }_{\mathrm{d}}N(0,2{\sigma }^4)$. Hence, we have ${\widehat{\sigma }}^2-{\sigma }^2=o_{\mathrm{p}}(1)$, and, thus, ${\widehat{\sigma }}^2$ is a consistent estimator. Again, by the Central Limit Theorem, we have

$$\sqrt{m}({\widehat{\sigma }}^2-{\sigma }^2){\to }_{d}N(0,2{\sigma }^4).$$

□

Remark 4.

It is natural to consider independent covariates in which the ${\lambda }_{it}$ can be modeled as ${\lambda }_{it}=exp(U_i+{\sum }_{j=0}^p{\beta }_j{X}_{it}^{(j)})$, where $X_{it}^{(0)}\equiv 1$ and $X_{it}^{(j_1)}$ and $X_{it}^{(j_2)}$ are independent for $0<j_1\ne j_2\le p$. We still have

$${\xi }_{mnj}={(mn)}^{-1/2}\sum _{i=1}^m\sum _{t=1}^n(Y_{it}-{\lambda }_{it})(X_{it}^{(j)}-r_j),$$

where

$$r_j=E(Y_{it}{X}_{it}^{(j)})/E(Y_{it})=E\{X_{it}^{(j)}exp({\beta }_j{X}_{it}^{(j)})\}/E\{exp({\beta }_j{X}_{it}^{(j)})\}={\varphi }_{j,1}/{\varphi }_{j,0}$$

$$B_{j,j}=exp({\beta }_0+\frac{1}{2}{\sigma }^2)\{{\varphi }_{j,2}-{\varphi }_{j,1}^2{\varphi }_{j,0}^{-1}\}\prod _{k\ne j}{\varphi }_{k,0},$$

where ${\varphi }_{j,2}=E\{{(X_{it}^{(j)})}^{2}exp({\beta }_j{X}_{it}^{(j)})\}$, and $B_{j_1,j_2}=0$ if $j_1\ne j_2$. Furthermore, ${\delta }_{jj}={\varphi }_{j,2}{\varphi }_{j,0}^{-1}-{\varphi }_{j,1}^2{\varphi }_{j,0}^{-2}$ and ${\delta }_{jk}=0$ if $k\ne j$. Other modifications on $B_{j,j}$ can be similarly done.

Remark 5.

According to Theorem 2, we can construct an approximate confidence interval (CI) for parameters based on the limiting distributions, and they take the form:

Parameter	$(1-\alpha )100\%$ CI
${\beta }_0$	${\widehat{\beta }}_0\pm \mathrm{\Phi }(1-\frac{1}{2}\alpha )\sqrt{\frac{{\widehat{\sigma }}^2}{m}}$
${\beta }_j(j=1,\dots ,p)$	${\widehat{\beta }}_j\pm \mathrm{\Phi }(1-\frac{1}{2}\alpha )\sqrt{\frac{{\widehat{\gamma }}_j}{mn}}$
${\sigma }^2$	${\widehat{\sigma }}^2\pm \mathrm{\Phi }(1-\frac{1}{2}\alpha ){\widehat{\sigma }}^2\sqrt{\frac{2}{m}}$

where Φ denotes the distribution function of $N(0,1)$. Note that, when $p=1$, these confidence intervals agree with those given in Hall, Pham, Wand, and Wang (2011).

4. Numerical Studies

The proposed method is applied to a study by the Philippine Statistics Authority to illustrate how the methodology works. The simulation studies are performed to demonstrate the accuracy of the proposed method.

4.1. Real Data Analysis

The Philippine Statistics Authority spearheads the nationwide Family and Expenditure Survey. One of the objectives of this survey is to identify the age of the head of the household that is likely to have the maximum number of family members in the household. This would allow the government of the Philippines to plan and set policies as well as psychological help for Filipinos to adjust to the possibility of loneliness in old age.

The results of the nationwide Family and Expenditure Survey were published in Philippine Statistics Authority (2015). For each of the 17 regions $(m=17)$ reported in [15], a sample of 1249 households $(n=1249)$ was randomly selected. Let $Y_{it}$ be the count of the number of family members in the household in the $t^{th}$ sample of the region i, $X_{it}$ is the age of the head of the $t^{th}$ household of region i, and $U_i$ is the random effect in the $i^{th}$ region. The model considered is

$$\begin{array}{c}\hfill Y_{it}|({\mathit{X}}_{it},U_i)\sim \mathrm{Poisson}({\lambda }_{it})i=1,\dots ,17\mathrm{and}t=1,\dots ,1249\end{array}$$

(26)

where

$${\lambda }_{it}=exp\left(U_i+\sum _{j=0}^2{\beta }_j{X}_{it}^j\right).$$

Figure 2 plots the log of the averaged counts of the number of family members in the household versus the age of the head of the household. At age 80 years or below, there is a clear maximum of the log of the average counts of the number of family members in the household. After age 80, this number increased because of the grand and great grand members joining the household. In this case, the method in Hall et al. [6] is not applicable because we have more than one predictor. Figure 2 also suggested that $X_{it}^2$ should be included in Model (26). Based on the proposed method, we have the estimates and confidence intervals of the parameters in

Table 1. Estimates and confidence intervals of the parameters.

Parameter	Estimates	95% Confidence Interval
${\beta }_0$	0.55262	(0.52585, 0.57939)
${\beta }_1$	0.04403	(0.04363, 0.04443)
${\beta }_2$	$-0.00045$	($-0.00046,-0.00045$)
${\sigma }^2$	0.00317	(0.00104, 0.0053)

.

Moreover, the age of the head of the household that maximized the model is 48.4 years, which is indicated by the red line in Figure 2.

4.2. Simulations

Simulation studies are used to demonstrate the accuracy of the proposed method. In all studies, we set $m=10,20,\dots ,100$ and $n=10m$.

For the first study, we consider Model (16) with only one predictor $(p=1)$ and two sets of parameter values:

$$({\beta }_0,{\beta }_1,{\sigma }^2)=(2.2,-0.1,0.16)\mathrm{and}(1.2,0.4,0.3).$$

Let $X_{it}$ be generated from $N(0,1)$, where $i=1,\dots ,m$ and $t=1,\dots ,n$. For each simulated sample, we obtain the 95% confidence interval for ${\beta }_0$ and record if the true ${\beta }_0$ is within the 95% confidence interval. Figure 3 shows the coverage proportion, where the nominal value is 0.95. Moreover, we have only one predictor, and thus the method in Hall et al. [6] can be applied as well. The results are also plotted in Figure 3 for comparison. We repeated the same procedure to obtain the coverage proportions for ${\beta }_1$ and ${\sigma }^2$, which are reported in Figure 3, respectively. As shown in Figure 3, for variance ${\sigma }^2=0.16$, both methods are comparable for larger variance ${\sigma }^2=0.3$. However, the proposed method is slightly better for small m.

For the second study, we consider Model (16) with only two predictors $(p=2)$ and four sets of parameter values:

Set	${\mathit{\beta }}_{\mathbf{0}}$	${\mathit{\beta }}_{\mathbf{1}}$	${\mathit{\beta }}_{\mathbf{2}}$	${\mathit{\sigma }}^{\mathbf{2}}$
1	2.2	0.1	−0.1	0.16
2	1.2	0.4	−0.2	0.1
3	2.2	−0.1	0.1	0.16
4	1.2	0.4	0.1	0.1

Let $X_{it}$ be generated from $N(0,1)$ where $i=1,\dots ,m$ and $t=1,\dots ,n$. The 95% coverage probabilities are plotted in Figure 4. In this case, the method in Hall et al. [6] is not applicable because $p=2$. As shown in Figure 4, the convergence rates for estimates of ${\beta }_1$ and ${\beta }_2$ are faster than the ones for ${\beta }_0$ and ${\sigma }^2$. That may explain the differences in coverage percentage among them.

Our last study is similar to the previous study, except the predictors are independent. In other words, we consider the model given in Remark 4 after Theorem 2. The same parameter settings as in the second study were used, and the two predictors were generated from an independent standard normal distribution. The 95% coverage probabilities are plotted in Figure 5. In this case, again the method in Hall et al. [6] is not applicable because $p=2$. The results are similar to those obtained in our second study even for different generation processes of predictors as shown in Figure 5.

Other parameter settings were also considered, and similar results were obtained. Thus, they are not reported here. They are available from the authors upon request.

5. Conclusions and Discussion

We proposed a special exponential family model and studied the GVA of the normalized constant by connecting the KL divergence and Jensen’s inequality. A closed form of expansion of the normalized constant was given up to the second order, which simplified the calculation and theoretical justification in inference of the Poisson lognormal mixed model. Real data analysis justified the importance of extension to multiple predictors. Simulation studies showed that the coverage percentage was consistent.

The proposed method was implemented into R and is available from the authors upon request. The flowchart of the proposed algorithm by GVA is shown in Figure 6.

Note that the glmer function in R package and lme4 perform similar calculations; however, our proposed method has two advantages over glmer—namely,

• glmer does not provide the standard deviation of the estimate of variance ${\sigma }^2$, and hence asymptotic inference for ${\sigma }^2$ is not available.
• glmer is slow. Figure 7 compares the required computational times for different m. The computational time for glmer increases dramatically as the sample size m or n increases. For example, for $m=10,000$, our proposed method was about 56-times faster than glmer.

In summary, the Gaussian variational approximation method was developed for the Poisson lognormal mixed model with one or more predictors in this paper. The explicit forms of the estimators of the parameters and the corresponding limiting distributions were derived. Hence, inference for the parameters was obtained. A real-life example demonstrated the applicability of the proposed method, and simulation studies illustrated the accuracy of the proposed method. Although the same problem has been studied in Hall et al. [5] and Hall et al. [6], their studies were restricted to one-predictor models and are difficult to generalize to multiple-predictor models. The proposed method gives results that are comparable to their results but are applicable to multiple-predictor models.