Variable Selection and Allocation in Joint Models via Gradient Boosting Techniques

Abstract

Modeling longitudinal data (e.g., biomarkers) and the risk for events separately leads to a loss of information and bias, even though the underlying processes are related to each other. Hence, the popularity of joint models for longitudinal and time-to-event-data has grown rapidly in the last few decades. However, it is quite a practical challenge to specify which part of a joint model the single covariates should be assigned to as this decision usually has to be made based on background knowledge. In this work, we combined recent developments from the field of gradient boosting for distributional regression in order to construct an allocation routine allowing researchers to automatically assign covariates to the single sub-predictors of a joint model. The procedure provides several well-known advantages of model-based statistical learning tools, as well as a fast-performing allocation mechanism for joint models, which is illustrated via empirical results from a simulation study and a biomedical application.

1. Introduction

Joint models for longitudinal and time-to-event data, first introduced in [1], are a powerful tool for analyzing data where event times are recorded alongside a longitudinal outcome. If the research interest lies in the association between these two outcomes, joint modeling avoids potential bias arising from separate analyses by combining two sub-models in one single modeling framework. A thorough introduction to the concept of joint models can be found in [2], and various well-established R packages are available covering frequentist [3,4] and Bayesian [5] approaches.

Like many regression models, joint models suffer from the usual drawbacks, where proper tools for variable selection are not immediately available and computation becomes more and more infeasible in higher dimensions. In addition, joint models also raise the question of which sub-model a variable should be assigned to, i.e., should a variable x have a direct impact on the survival outcome T, or should the potential influence be modeled indirectly by an impact of x on the longitudinal outcome y, which then might affect T? This choice gets exponentially more complex with an increasing amount of covariates, and usually has to be made by researchers based on background knowledge. Boosting techniques from the field of statistical learning, however, are well-known for addressing these exact issues. Originally emerging from the machine learning community as an approach to classification problems [6,7], boosting algorithms have been adapted to regression models [8] and, by now, cover a wide range of statistical models. For an introduction and overview of model-based boosting, we recommend [9,10].

The formulation of boosting routines for joint models is, to date, still a little-developed field. The foundations were made by [11], where generalized additive models for location scale and shape (GAMLSS) were fitted using boosting techniques. Due to the multiple predictors for each single distributional parameter, these models consist of a similar structure to joint models and thus the boosting concept for GAMLSS could be adapted to joint models by [12]. Furthermore, in [13], joint models were estimated using likelihood-based boosting techniques, and [14,15] compare boosting routines for joint models with various other estimation approaches.

In the last few years, several additional developments have been made in order to enable a variable selection for joint models, usually by applying different shrinkage techniques. In [16], an adaptive LASSO estimator was constructed that estimates $L1$-penalized likelihoods in a two-stage fashion. This approach was later extended to multivariate longitudinal outcomes in [17] and to time-varying coefficients in [18]. In [19,20], Bayesian shrinkage estimators were applied to achieve either variable or model selection for various classes of joint models and, recently, ref. [21] applied Monte Carlo methods to enable a variable selection for joint models with an interval-censored survival outcome. However, all of these mentioned approaches are only capable of selecting and estimating effects into predefined predictor functions. To the best of our knowledge, no methods exist that allocate single features to the given sub-models in a data-driven way.

The aim of the present work is to combine recent developments from the field of model-based gradient boosting in order to develop a new routine, JMalct, that is able to allocate the single candidate variables to the specific sub-models. Therefore, the initial boosting approach by [12] was equipped with a non-cyclical updating scheme proposed by [22] and adaptive step-lengths as investigated in [23]. These two preliminary works are of high importance and their combination is the foundation of our proposed allocation procedure. Furthermore, the JMalct algorithm makes use of a recent random effects correction [24] providing an unbiased estimation of the random effects using gradient boosting and tuning based on probing [25] for faster computation and improved selection properties.

The remainder of this article is structured as follows. In Section 2, the underlying joint model as well as the JMalct boosting algorithm are formulated. Section 3 then applies the proposed method to simulated data with varying amounts of candidate variables. Several real-world applications are presented in Section 4 and the final section gives a brief summary and outlook.

2. Methods

This section first formulates the considered joint model as well as the basics of the underlying JMboost approach. Afterwards, the new JMalct routine and a thorough discussion of its computational details are provided.

2.1. Model Specification

A joint model consists of two sub-models modeling the longitudinal and time-to-event outcome, respectively. The longitudinal sub-model is specified as a linear mixed model

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill y_{ij}& ={\eta }_{\mathrm{long}}^{\phantom{l}}(t_{ij},{\mathit{x}}_{\mathrm{long}i}^{\phantom{l}})+{\epsilon }_{ij}\hfill \\ \hfill & ={\beta }_0+{\beta }_t{t}_{ij}+{\mathbf{\beta }}_{\mathrm{long}}^T{\mathit{x}}_{\mathrm{long}i}^{\phantom{l}}+{\gamma }_{0i}+{\gamma }_{ti}{t}_{ij}+{\epsilon }_{ij},\hfill \end{array}\end{array}$$

(1)

with individuals $i=1,\cdots ,n$ and corresponding measurements $j=1,\cdots ,n_i$. Here, ${\mathit{x}}_{\mathrm{long}i}^{\phantom{l}}\in {\mathbb{R}}^{p_{\mathrm{long}}}$ denotes a set of longitudinal time-independent covariates, and $t_{ij}$ the specific measurement times and normal distributed error components, i.e., $({\gamma }_{0i},{\gamma }_{ti})\sim {\mathcal{N}}^{\otimes 2}(\mathbf{0},\mathit{Q})$ and ${\epsilon }_{ij}\sim \mathcal{N}(0,{\sigma }^2)$ are assumed.

In the survival sub-model, the individual hazard is modeled by

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\lambda }_i\left(t\right)& ={\lambda }_0\left(t\right)exp\left({\eta }_{\mathrm{surv}}\left({\mathit{x}}_{\mathrm{surv}i}^{\phantom{l}}\right)+\alpha {\eta }_{\mathrm{long}}^{\phantom{l}}(t,{\mathit{x}}_{\mathrm{long}i}^{\phantom{l}})\right)\hfill \end{array}\end{array}$$

(2)

with the survival predictor ${\eta }_{\mathrm{surv}}^{\phantom{l}}\left({\mathit{x}}_{\mathrm{surv}i}^{\phantom{l}}\right)={\mathbf{\beta }}_{\mathrm{surv}}^T{\mathit{x}}_{\mathrm{surv}i}^{\phantom{l}}$ including baseline covariates ${\mathit{x}}_{\mathrm{surv}i}^{\phantom{l}}\in {\mathbb{R}}^{p_{\mathrm{surv}}}$ and the longitudinal predictor ${\eta }_{\mathrm{long}}^{\phantom{l}}$ reappearing in the survival sub-model, this time scaled by the association parameter $\alpha$. The baseline hazard ${\lambda }_0\left(t\right):={\lambda }_0>0$ is chosen to be constant as conventional gradient boosting methods tend to struggle with a proper estimation of time-varying baseline hazard functions [26].

Given the sub-models (1) and (2) and assuming independence between the random components, the joint log-likelihood is

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \ell ({\eta }_{\mathrm{long}}^{\phantom{l}},{\eta }_{\mathrm{surv}}^{\phantom{l}},& \alpha ,{\lambda }_0,{\sigma }^2|\mathit{y},\mathit{T},\delta )=\sum _{i=1}^n\{\sum _{j=1}^{n_i}log\varphi \left(y_{ij}|{\eta }_{\mathrm{long}}^{\phantom{l}}(t_{ij},{\mathit{x}}_{\mathrm{long}i}^{\phantom{l}}),{\sigma }^2\right)\hfill \\ \hfill & +{\delta }_{i}log{\lambda }_i\left(T_i\right|{\eta }_{\mathrm{long}}^{\phantom{l}},{\eta }_{\mathrm{surv}}^{\phantom{l}},\alpha ,{\lambda }_0)-{\int }_0^{T_i}exp\left({\lambda }_i\left(t\right|{\eta }_{\mathrm{long}}^{\phantom{l}},{\eta }_{\mathrm{surv}}^{\phantom{l}},\alpha ,{\lambda }_0)\right)dt\},\hfill \end{array}\end{array}$$

(3)

where, in the longitudinal part, $\varphi (·|m,v)$ denotes the density of a normal distribution with mean m and variance v. In this context, we considered the complete data log-likelihood as it is used solely for allocation purposes. The random effects will be estimated in a less time-consuming way based on a fixed penalization integrated in the random effects base-learner discussed in Section 2.4.

2.2. The JMboost Concept

In [12], joint models were estimated for the first time using a boosting algorithm, although they addressed a slightly different model to the one described above. The original concept in this publication was based on an alternating technique that used two loops: one outer loop circling through the two sub-predictors and two inner loops that circle through the single base-learners. In a very simple manner, the boosting algorithm can hence be summarized as follows:

• Initialize ${\eta }_{\mathrm{long}}^{\phantom{l}}$, ${\eta }_{\mathrm{surv}}^{\phantom{l}}$, $\alpha$, ${\lambda }_0$ and ${\sigma }^2$;
• While $m\le max(m_{\mathrm{stop},\mathrm{l}},m_{\mathrm{stop},\mathrm{s}})$;

  -

  If $m\le m_{\mathrm{stop},\mathrm{l}}$: perform one boosting cycle to update ${\eta }_{\mathrm{long}}^{\phantom{l}}$;

  -

  If $m\le m_{\mathrm{stop},\mathrm{s}}$: perform one boosting cycle to update ${\eta }_{\mathrm{surv}}^{\phantom{l}}$;

  -

  If $m\le m_{\mathrm{stop},\mathrm{l}}$: update ${\sigma }^2$;

  -

  If $m\le m_{\mathrm{stop},\mathrm{s}}$: update ${\lambda }_0$ and $\alpha$.

Both sub-predictors have their own stopping iteration $m_{\mathrm{stop},\mathrm{l}}$ and $m_{\mathrm{stop},\mathrm{s}}$, which need to be optimized via a grid search. The latter is computationally quite burdensome, particularly for high numbers of candidate variables.

2.3. The JMalct Boosting Algorithm

The central JMalct algorithm is depicted in Algorithm 1.

Algorithm 1: JMalct

Initialize predictors ${\widehat{\eta }}_{\mathrm{long}}^{\left[0\right]}$ and ${\widehat{\eta }}_{\mathrm{surv}}^{\left[0\right]}$. Specify base-learners $h_{\mathrm{long}1},\cdots ,h_{\mathrm{long}p}$ and $h_{\mathrm{surv}1},\cdots ,h_{\mathrm{surv}p}$, as well as $h_{\gamma }$. Initialize association ${\widehat{\alpha }}^{\left[0\right]}$ and baseline hazard ${\widehat{\lambda }}_0^{\left[0\right]}$. Choose iteration limit $m_{\mathrm{stop}}$ and learning rate c, and define the sets ${\mathcal{S}}_{\mathrm{long}}^{\left[0\right]}={\mathcal{S}}_{\mathrm{surv}}^{\left[0\right]}:=\{1,\cdots ,p\}$. for$m=1$ to $m_{\mathrm{stop}}$ do step1: Allocation step Compute the gradients $${\mathit{u}}_{\mathrm{long}}^{\left[m\right]}={\left(u_{\mathrm{long}ij}^{\left[m\right]}\right)}_{i\le n,j\le n_i}={\left(y_{ij}-{\widehat{\eta }}_{\mathrm{long}ij}^{[m-1]}\right)}_{i\le n,j\le n_i}$$ (4) and $${\mathit{u}}_{\mathrm{surv}}^{\left[m\right]}={\left(u_{\mathrm{surv}i}^{\left[m\right]}\right)}_{i\le n}={\left({\delta }_i-{\int }_0^{T_i}{\widehat{\lambda }}_i^{[m-1]}(t,·)dt\right)}_{i\le n}.$$ (5) Fit both gradients separately to the base-learners $$\begin{array}{cc}\hfill {\mathit{u}}_{\mathrm{long}}^{\left[m\right]}& \stackrel{\mathrm{base}-\mathrm{learner}}{\to }{\widehat{h}}_{\mathrm{long}r}^{\left[m\right]},r\in {\mathcal{S}}_{\mathrm{long}}^{[m-1]},\hfill \end{array}$$ (6) $$\begin{array}{cc}\hfill {\mathit{u}}_{\mathrm{surv}}^{\left[m\right]}& \stackrel{\mathrm{base}-\mathrm{learner}}{\to }{\widehat{h}}_{\mathrm{surv}r}^{\left[m\right]},r\in {\mathcal{S}}_{\mathrm{surv}}^{[m-1]},\hfill \end{array}$$ (7) and select the best performing component for each predictor: $$r_{\mathrm{long}}^{*}=\underset{r\le p}{arg\; min}\sum _{ij}{\left(u_{\mathrm{long}ij}^{\left[m\right]}-{\widehat{h}}_{\mathrm{long}ij}^{\left[m\right]}\right)}^2,r_{\mathrm{surv}}^{*}=\underset{r\le p}{arg\; min}\sum _i{\left(u_{\mathrm{surv}i}^{\left[m\right]}-{\widehat{h}}_{\mathrm{surv}i}^{\left[m\right]}\right)}^2$$ (8) Compute the optimal step lengths ${\nu }_{\mathrm{long}{r}^{*}}$, ${\nu }_{\mathrm{surv}{r}^{*}}$ with corresponding likelihood values ${\ell }_{\mathrm{long}{r}^{*}}$, ${\ell }_{\mathrm{surv}{r}^{*}}$ and only update the component resulting in the best joint likelihood improvement: $$\begin{array}{cc}\hfill {\widehat{\eta }}_{\mathrm{long}}^{\left[m\right]}& ={\widehat{\eta }}_{\mathrm{long}}^{[m-1]}+c{\nu }_{\mathrm{long}{r}^{*}}{\widehat{h}}_{\mathrm{long}{r}^{*}}^{\left[m\right]},\mathrm{if}{\ell }_{\mathrm{long}{r}^{*}}>{\ell }_{\mathrm{surv}{r}^{*}},\hfill \end{array}$$ (9) $$\begin{array}{cc}\hfill {\widehat{\eta }}_{\mathrm{surv}}^{\left[m\right]}& ={\widehat{\eta }}_{\mathrm{surv}}^{[m-1]}+c{\nu }_{\mathrm{surv}{r}^{*}}{\widehat{h}}_{\mathrm{surv}{r}^{*}}^{\left[m\right]},\mathrm{if}{\ell }_{\mathrm{long}{r}^{*}}<{\ell }_{\mathrm{surv}{r}^{*}}\hfill \end{array}$$ (10) Update the active sets $$\begin{array}{cc}\hfill {\mathcal{S}}_{\mathrm{long}}^{\left[m\right]}={\mathcal{S}}_{\mathrm{long}}^{[m-1]}\setminus \left\{r_{\mathrm{surv}}^{*}\right\},& \mathrm{if}{\ell }_{\mathrm{surv}{r}^{*}}>{\ell }_{\mathrm{long}{r}^{*}},\hfill \end{array}$$ (11) $$\begin{array}{cc}\hfill {\mathcal{S}}_{\mathrm{surv}}^{\left[m\right]}={\mathcal{S}}_{\mathrm{surv}}^{[m-1]}\setminus \left\{r_{\mathrm{long}}^{*}\right\},& \mathrm{if}{\ell }_{\mathrm{long}{r}^{*}}>{\ell }_{\mathrm{surv}{r}^{*}}.\hfill \end{array}$$ (12) step2: Update remaining parameters Perform an additional longitudinal boosting update regarding the random structure: $${\mathit{u}}_{\mathrm{long}}^{\left[m\right]}\stackrel{\mathrm{base}-\mathrm{learner}}{\to }{\widehat{h}}_{\gamma }^{\left[m\right]}\Rightarrow {\widehat{\eta }}_{\mathrm{long}}^{\left[m\right]}={\widehat{\eta }}_{\mathrm{long}}^{\left[m\right]}+c{\widehat{h}}_{\gamma }^{\left[m\right]}$$ (13) Obtain updates for the association by maximizing the joint likelihood: $${\widehat{\alpha }}^{\left[m\right]}=\underset{\alpha \in \mathbb{R}}{arg\; max}\ell (\alpha ,·)$$ (14) end for Stop the algorithm early based on probing, i.e., when a phantom variable would get selected.

2.4. Computational Details of the JMalct Algorithm

In the new JMalct algorithm, we only have one cycle. This cycle consists of three steps: in the first one, the base-learner with the best fitting gradient for the longitudinal predictor ${\eta }_{\mathrm{long}}$ is chosen and the corresponding step length ${\nu }_{\mathrm{long}}$ is calculated. In the second step, the base-learner with the best fitting gradient for the time-to-event submodel ${\eta }_{\mathrm{surv}}$ is chosen and the corresponding step length ${\nu }_{\mathrm{surv}}$ is calculated. These first two steps will be referred to as the G-steps (gradient-steps) in the following. In the third step, referred to as the L-step (likelihood step), the likelihood is calculated for both the best longitudinal base-learner, weighted with the step length ${\nu }_{\mathrm{long}}$, and the best survival base-learner, weighted with the step length ${\nu }_{\mathrm{surv}}$. The base-learner performing better in the L-step is then chosen to be updated. The algorithm is summarized in the following overview and depicted in Figure 1. A detailed description is provided below.

• while $m\le m_{\mathrm{stop}}$:

• G-step 1

  -

  Fit all base-learners to the longitudinal gradient with regard to ${\widehat{\eta }}_{\mathrm{long}}^{\left[m\right]}$;

  -

  Find the best-performer, ${\beta }_{\mathrm{long}}^{*}$ and corresponding step-length ${\nu }_{long}$.

• G-step 2

  -

  Fit all base-learners to gradient with regard to ${\widehat{\eta }}_{\mathrm{surv}}^{\left[m\right]}$;

  -

  Find the best-performer, ${\beta }_{\mathrm{surv}}^{*}$ and corresponding step-length ${\nu }_{\phantom{l}surv}$.

• L-step

  -

  Fit likelihood for ${\eta }_{\mathrm{long}}^{*}$ and ${\eta }_{\mathrm{surv}}^{*}$ with updates from G1 and G2;

  -

  Select the best-performer and update corresponding sub-predictor;

  -

  Remove the selected candidate variable from options to choose for the other predictor (if not performed already).

• Step 4

  -

  Update ${\widehat{\alpha }}^{\left[m\right]},{\widehat{\sigma }}^{2\left[m\right]}$ based on the current fit.

The baseline covariates that enter the allocation process are not assigned to a sub-model in the beginning and therefore have to be considered in two forms. ${\mathit{X}}_{\mathrm{long}}\in {\mathrm{Mat}}_{\mathbb{R}}(N,p)$, where $N={\sum }_i{n}_i$, denotes the set of candidate variables resembled as longitudinal covariates, i.e., measurements assigned to the same individual i contain the same cluster-constant measurement $n_i$ times. On the other hand, ${\mathit{X}}_{\mathrm{surv}}\in {\mathrm{Mat}}_{\mathbb{R}}(n,p)$ contains the exact same variables as ${\mathit{X}}_{\mathrm{long}}$ but reduced to just one representative of each individual in order to fit the corresponding base-learner to the survival gradient. The measurements of one specific covariate r are denoted by ${\mathit{x}}_{\mathrm{long}r}$ and ${\mathit{x}}_{\mathrm{surv}r}$, which matches the rth column of the corresponding matrix.

Starting values. The regression coefficients underlying the allocation mechanism are necessarily set to zero, i.e., ${\mathbf{\beta }}_{\mathrm{long}}^{\left[0\right]}={\mathbf{\beta }}_{\mathrm{surv}}^{\left[0\right]}=\mathbf{0}$. The remaining longitudinal parameters are extracted by an initial linear mixed model fit

$$y={\beta }_0+{\beta }_t·t+{\gamma }_0+{\gamma }_t·t$$

(15)

containing only the intercept as well as time and random effects. For the remaining survival parameters, we chose ${\alpha }^{\left[0\right]}=0$ and ${\widehat{\lambda }}^{\left[0\right]}={\sum }_i{\delta }_i/{\sum }_i{T}_i$.

Computing the gradients. The gradients ${\mathit{u}}_{\mathrm{long}}$ and ${\mathit{u}}_{\mathrm{surv}}$ are a crucial component of the JMalct algorithm. For the longitudinal part, we considered the quadratic loss $\rho (y,\eta )=\frac{1}{2}{(y-\eta )}^2$ and calculated

$${\mathit{u}}_{\mathrm{long}}=-\frac{\partial \rho }{\partial {\eta }_{\mathrm{long}}^{\phantom{l}}}(y,{\eta }_{\mathrm{long}}^{\phantom{l}})=y-{\eta }_{\mathrm{long}}^{\phantom{l}}$$

(16)

as the regular residuals of the longitudinal sub-model, following [9]. The survival gradient was obtained by differentiating the likelihood (3) with respect to ${\eta }_{\mathrm{surv}}^{\phantom{l}}$, yielding

$${\mathit{u}}_{\mathrm{surv}}=-\frac{\partial \rho }{\partial {\eta }_{\mathrm{surv}}^{\phantom{l}}}({\eta }_{\mathrm{surv}}^{\phantom{l}},·)={\left({\delta }_i-{\int }_0^{T_i}{\widehat{\lambda }}_i^{[m-1]}(t,·)dt\right)}_{i\le n},$$

(17)

as the longitudinal part vanishes. This is a nice analogy to the longitudinal gradient, as ${\mathit{u}}_{\mathrm{surv}}$ represents the martingale residuals of the survival sub-model.

Fitting the longitudinal base-learners. The possible fixed effects estimates were obtained by fitting the pre-specified base-learners to the longitudinal and survival gradient. In the longitudinal case, the fixed effects base-learners $h_{\mathrm{long}1},\cdots ,h_{\mathrm{long}p}$ were equipped with an additional effect estimate for the time coefficient ${\beta }_t$ as this variable shall not be subject to the selection and allocation mechanism. Fitting the base-learners is achieved by

$${\widehat{h}}_{\mathrm{long}r}=S_{\mathrm{long}r}{\mathit{u}}_{\mathrm{long}},r=1,\cdots ,p,$$

(18)

with the projection matrix

$$S_{\mathrm{long}r}={\tilde{\mathit{x}}}_{\mathrm{long}r}{\left({\tilde{\mathit{x}}}_{\mathrm{long}r}^T{\tilde{\mathit{x}}}_{\mathrm{long}r}\right)}^{-1}{\tilde{\mathit{x}}}_{\mathrm{long}r}^T,r=1,\cdots ,p,$$

(19)

where ${\tilde{\mathit{x}}}_{\mathrm{long}r}=(\mathit{1},\mathit{t},{\mathit{x}}_{\mathrm{long}r})$ and $\mathit{t}$ denotes the collection of longitudinal measurement times. If the base-learner actually gets selected, estimates ${\widehat{\beta }}_0$ for the intercept and ${\widehat{\beta }}_t$ for the time effect receive the corresponding updates computed in the fitting process.

Fitting the survival base-learners. Similar to the longitudinal part, the survival base-learner was fitted by applying the corresponding projection matrix to the survival gradient, i.e.,

$${\widehat{h}}_{\mathrm{surv}r}=S_{\mathrm{surv}r}{\mathit{u}}_{\mathrm{surv}},r=1,\cdots ,p,$$

(20)

where the survival gradient ${\mathit{u}}_{\mathrm{surv}}$ represents the martingale residuals of the time-to-event model. The projection matrix takes the form

$$S_{\mathrm{surv}r}={\tilde{\mathit{x}}}_{\mathrm{surv}r}{\left({\tilde{\mathit{x}}}_{\mathrm{surv}r}^T{\tilde{\mathit{x}}}_{\mathrm{surv}r}\right)}^{-1}{\tilde{\mathit{x}}}_{\mathrm{surv}r}^T,r=1,\cdots ,p,$$

(21)

with ${\tilde{\mathit{x}}}_{\mathrm{surv}r}=(\mathbf{1},{\mathit{x}}_{\mathrm{surv}r})$. This means that, if the base-learner actually gets selected, the estimate ${\widehat{\lambda }}_0$ for the constant baseline hazard receives the corresponding update computed in the fitting process.

Adaptive step lengths. As the two distinct sub-models affect different parts of the joint likelihood, it may not be sufficient to stick to a fixed learning rate, e.g., ${\nu }_{\mathrm{long}}={\nu }_{\mathrm{surv}}=0.1$. To ensure that the comparison of potential likelihood improvements is fair, for each selected component, the optimal step length was computed using a basic line search by finding

$${\nu }_{\mathrm{long}}=\underset{\nu \in {\mathbb{R}}^{+}}{arg\; max}\ell ({\widehat{\eta }}_{\mathrm{long}}+\nu {\widehat{h}}_{\mathrm{long}{r}_{\mathrm{long}}^{*}},·),{\nu }_{\mathrm{surv}}=\underset{\nu \in {\mathbb{R}}^{+}}{arg\; max}\ell ({\widehat{\eta }}_{\mathrm{surv}}+\nu {\widehat{h}}_{\mathrm{surv}{r}_{\mathrm{surv}}^{*}},·).$$

(22)

following [23]. The corresponding maximal likelihood values are denoted by ${\ell }_{\mathrm{long}}^{*}$ and ${\ell }_{\mathrm{surv}}^{*}$, which were used to determine the overall best-performing sub-model of each iteration. When this is achieved, the learning rate for the actual update was then again scaled by a constant $c<1$—here, $c=0.1$—in order to ensure small updates with weak base-learners.

Fitting the random effects base-learner. In general, the random effects base-learner is similar to the formulation found in the appendix of [27]. One major difference is that it is fixed for all iterations and not updated based on the current covariance structure. It is defined through its projection matrix

$$S_{\gamma }=\mathit{Z}\mathit{C}{({\mathit{Z}}^T\mathit{Z}+{\lambda }_{\mathrm{df}})}^{-1}{\mathit{Z}}^T$$

(23)

where ${\lambda }_{\mathrm{df}}$ was chosen so that $\mathrm{tr}\left(S_{\gamma }\right)=\mathrm{df}$ holds, which fixes the degrees of freedom for the random effects update. In the simulation study, we used $\mathrm{df}=10$ and determined the corresponding ${\lambda }_{\mathrm{df}}$ with the internal function mboost:::df2lambda().

The matrix $\mathit{Z}$ denotes the conventional random effects design matrix for intercepts and slopes, i.e.,

$$\mathit{Z}=\mathrm{diag}({\mathit{Z}}_1,\cdots ,{\mathit{Z}}_n),{\mathit{Z}}_i=\left(\begin{array}{cc}1& t_{i1}\\ ⋮& ⋮\\ 1& t_{i{n}_i}\end{array}\right),i=1,\cdots ,n,$$

(24)

and $\mathit{C}$ is a correction matrix introduced in [28] correcting the random effects update for the candidate variables ${\mathit{x}}_{\mathrm{long}1},\cdots ,{\mathit{x}}_{\mathrm{long}p}$, which are baseline covariates and thus cluster-constant. A derivation of the correction matrix $\mathit{C}$ can also be found in Appendix A.

Tuning the hyperparameter m based on probing. Both the step length as well as the number of iterations can be considered as hyperparameters of the boosting algorithm. Since the step length is usually set as constant or, like in this work, determined by an adaptive line search, the number of overall iterations m states the main tuning parameter of the algorithm. While this hyperparameter is usually tuned in a computationally more extensive way by considering out-of-bag loss, we determined the optimal amount $m^{*}$ with the help of probing. Probing for gradient boosting was introduced by [25]. The pragmatic idea avoids more time-consuming procedures such as cross validation or bootstrapping, which rely on a re-fitting of the model. For each covariate ${\mathit{x}}_r$, another variable ${\overline{\mathit{x}}}_r$ was added to the set of candidate variables, where ${\overline{\mathit{x}}}_r$ is a random permutation of the observations contained in ${\mathit{x}}_r$. These additional variables were artificially created to be non-informative and called probes or shadow variables. Instead of finding the best-performing number of iterations based on a computationally burdensome cross validation, the boosting routine was simply stopped as soon as one of the shadow variables ${\overline{\mathit{x}}}_r$, i.e., a known-to-be non-informative variable, would get selected. The focus is hence shifted from tuning the algorithm purely based on prediction accuracy (with regard to the test risk) towards a reasonable variable selection.

Computational complexity and asymptotic behavior. Due to the artificial construction of the algorithm and the comparatively complex model class, theoretical analysis regarding complexity and asymptotic behavior is quite a challenging task. The model-based boosting related literature is still little-developed with respect to theoretical investigations, but thorough analyses in simpler cases were carried out in [29] for the quadratic loss, where exponentially fast bias reduction could be proven, as well as for more general settings in [30,31]. Consistency properties for very-high-dimensional linear models were obtained in [32] and, regarding JMalct, we refer to the following section, where further insights with respect to the algorithm’s complexity are given based on numerical evaluations. In addition, we experienced no convergence issues in simulations and applications.

3. Simulation Study

The JMalct algorithm was evaluated by conducting a simulation study where data according to the assumed generating process specified in Section 2.1 were simulated and models were subsequently fitted using JMalct and, if sensible, JM [3] as a benchmark and well-established approach. In addition, we considered the combination JMalct+JM, where JMalct was used solely for allocating the variables, which were then refitted by JM according to the allocation obtained from JMalct. After briefly highlighting the single scenarios, the simulation section evaluates allocation properties and the accuracy of estimates, as well as the quality of the prediction and the computational burden.

3.1. Setup

We simulated data according to the model specification in Section 2.1 with $n=500$ and $n_i=5$ using inversion sampling. The pre-specified true parameter values are

$${\beta }_0=1,{\beta }_t=1.5,{\beta }_{\mathrm{long}}^T=(1,2,1,2),{\beta }_{\mathrm{surv}}^T=(0.3,0.5,0.3,0.5),\alpha =0.1$$

(25)

with variance components

$$\sigma =0.1,\mathit{Q}=\left(\begin{array}{cc}{\tau }_0^2& 0\\ 0& {\tau }_t^2\end{array}\right),{\tau }_0=2,{\tau }_t=0.3.$$

(26)

The entries of the covariate vectors ${\mathit{x}}_{\mathrm{long}i}^{\phantom{l}}$ and ${\mathit{x}}_{\mathrm{surv}i}^{\phantom{l}}$ were drawn independently from the uniform distribution $\mathcal{U}\left(\right[-0.1,0.1\left]\right)$. In addition to the informative covariates with effects ${\beta }_{\mathrm{long}}^{\phantom{l}}$ and ${\beta }_{\mathrm{surv}}^{\phantom{l}}$, the total set of covariates was expanded with a varying number $p_{\mathrm{non}-\inf }$ of non-informative noise variables. The baseline hazard was chosen as ${\lambda }_0\left(t\right)\equiv 1$ and given the censoring mechanism described in Algorithm A1 depicted in Appendix B. The chosen parameter values result in an average censoring rate of $\approx 50\%$. All of the parameters were specified in a way to obtain reasonably distributed event times $\mathit{T}$.

Overall, we considered four scenarios with varying numbers of additional noise variables $p_{\mathrm{non}-\inf }$ yielding overall dimensions $P\in \{10,25,50,100\}$. In each scenario, 100 independent data sets were generated and models were fitted using the various routines. The results were then summarized over all 100 independent simulation runs.

3.2. Selection and Allocation

In order to address allocation, we considered the criteria of correctly allocated (CA) and incorrectly allocated (IA) variables per predictor, as well as the share of false positives (FPs). Precisely, CA_long is the share of longitudinal variables, which are correctly assigned to the longitudinal predictor, and IA_long is the share of survival variables, which are falsely assigned to the longitudinal predictor and CA_surv, IA_surv analogously. FPs, on the other hand, denote the share of wrongly selected noise variables regardless of which predictor they are assigned to.

Table 1. Share of correctly allocated (CA) and incorrectly allocated (IA) variables for each predictor as well, as false positive rate. Values are averaged over 100 independent simulation runs of each scenario.

P	CAlong	IAlong	CAsurv	IAsurv	FP
10	1.00	0.20	0.76	0.00	0.58
25	1.00	0.15	0.78	0.00	0.12
50	1.00	0.14	0.77	0.00	0.05
100	1.00	0.10	0.76	0.00	0.02

depicts allocation and selection properties obtained for the different simulation scenarios. While, for the longitudinal predictor, variables get allocated perfectly, the survival part shows less ideal but still satisfactory results. There are various possible explanations for this behavior. On the one hand, the simulated signal is less strong for the survival effects due to the chosen parameter values, which, in general, increases the chance of false negatives. On the other hand, the longitudinal part of the likelihood carries more information, as there are more longitudinal measurements available than event times, which increases the risk of incorrect allocations. Finally, survival variables being incorrectly allocated to the longitudinal predictor is inherently more probable than vice versa as the longitudinal predictor also appears in the survival sub-model and the model therefore still accounts for the variables’ impact on the time-to-event outcome. While the allocation properties are roughly constant with a varying number of dimensions, the false positives rate clearly diminishes with more and more noise variables.

3.3. Estimation Accuracy

The accuracy of coefficient estimation is shown in

Table 2. Mean squared error for longitudinal (mse_long) and survival (mse_surv) coefficients averaged over 100 independent simulation runs. Regular parameter estimates are indicated by $\theta$, whereas ${\theta }_{-\mathrm{n}.\inf }$ denotes the second half, where non-informative effects are neglected.

	P	JMalct		JM		JMalct+JM
		mselong	msesurv	mselong	msesurv	mselong	msesurv
$\theta$	10	0.497	0.343	0.713	0.489	0.342	0.453
	25	0.479	0.296	1.888	1.359	0.309	0.417
	50	0.585	0.303	4.212	3.375	0.374	0.409
	100	0.563	0.295	9.532	9.483	0.320	0.393
${\theta }_{-\mathrm{n}.\inf }$	10	0.486	0.323	0.302	0.254	0.288	0.423
	25	0.470	0.292	0.247	0.291	0.239	0.398
	50	0.578	0.297	0.362	0.367	0.306	0.384
	100	0.557	0.290	0.315	0.558	0.254	0.373

, separated for each sub-model. We considered the mean squared error (mse) computed as

$${\mathrm{mse}}_{\mathrm{long}}=\parallel {\theta }_{\mathrm{long}}-{\widehat{\theta }}_{\mathrm{long}}{\parallel }^2,{\mathrm{mse}}_{\mathrm{surv}}={\parallel {\theta }_{\mathrm{surv}}-{\widehat{\theta }}_{\mathrm{surv}}\parallel }^2,$$

(27)

where ${\theta }_{\mathrm{long}}={({\beta }_0,{\beta }_t,{\mathbf{\beta }}_{\mathrm{long}}^T)}^T$ and ${\theta }_{\mathrm{surv}}={(\lambda ,\alpha ,{\mathbf{\beta }}_{\mathrm{surv}}^T)}^T$. The lower half of the table discards all entries of the estimates ${\widehat{\mathbf{\beta }}}_{\mathrm{long}}$ and ${\widehat{\mathbf{\beta }}}_{\mathrm{surv}}$ referring to non-informative variables and thus only measures the accuracy of the effects that are known to be informative.

It is evident that the accuracy of JM is heavily influenced by the number of noise variables, whereas the routines relying on the allocation and selection mechanism by JMalct stay fairly robust. As usual for regularization techniques, JMalct’s estimates for informative effects are slightly biased due to the early stopping of the algorithm. The combination JMalct+JM, however, stays unaffected by the number of noise variables and is, at least for the longitudinal predictor, the most accurate. The main hindrance of this approach is that the estimation accuracy of survival effects is slightly more influenced by false negatives occurring in the selection mechanism, which is why the combination lags behind its two competitors regarding precision for the survival sub-model.

3.4. Predictive Performance

Boosting is a tool primarily designed for prediction, and thus the predictive performance of JMalct and how it compares to established routines are of interest. Since our underlying joint model focuses on the time-to-event outcome as the main endpoint, we evaluated the prediction accuracy regarding the predicted and actual event time based on additional test data with $n_{\mathrm{test}}=1000$ individuals and $n_i=5$. We considered the loss

$$L(T,\widehat{T})=|logT-log\widehat{T}|,\widehat{T}=\mathbb{E}\left[T\right],$$

(28)

as the absolute deviation between the predicted and actual event time $\widehat{T}$ and T, respectively, on a log-scale [33].

Figure 2 depicts the values of L over the varying numbers of additional noise variables. The prediction is comparable among the three routines in low-dimensional settings. However, as expected, it worsens for JM when the dimensions increase. Both JMalct and the combination JMalct+JM rely on the selection conducted by JMalct and hence produce sparse models, which is why their quality of prediction stays fairly equal even in higher dimensions.

3.5. Computational Effort

Table 3. Average computation times of the three approaches measured in seconds.

P	JMalct	JM	JMalct+JM
10	92.69	97.44	149.19
25	88.80	339.41	151.65
50	86.92	1009.52	152.59
100	85.06	3517.76	146.26

shows the elapsed computation time measured in seconds, where each simulation run was carried out on a 2 x 2.66 GHz-6-Core Intel Xeon CPU (64GB RAM). Most obviously, JM becomes tremendously more burdensome as the dimensions increase. The constant or even a little decreasing computation times for JMalct over various dimensions might be surprising at first, as component-wise procedures such as gradient boosting tend to increase at least linearly in computation time with additional covariates. However, as the overall stopping criterion is based on probing, the algorithm tends to stop earlier in high-dimensional settings since more non-informative probes are available, increasing the probability that one might get selected earlier in the process. Due to the sparsity obtained by JMalct, the combination JMalct+JM also profits from the allocation and selection mechanism regarding the computational effort, as JM runs considerably faster again.

3.6. Complexity

While a formulation of explicit complexity results for the JMalct routine is quite technical in general, like that stated in Section 2.4, simulations can give insights toward how the algorithm scales up with varying numbers of observations and covariates. Therefore, we considered the same setup as above with different values for n and p and ran the JMalct routine 100 times independently for $m_{\mathrm{stop}}=100$ iterations without early stopping. Figure 3 depicts the averaged computation times for increasing values of n and p.

The left figure depicts square root computation times with $p=3$ as fixed, and thus reveals quadratically growing run times for increasing observations. On the other hand, the run times clearly expose a linear relationship with the amount of total covariates and $n=100$ as fixed. This is to be expected, as further candidate variables simply add to the inner loops of univariate base-learner fits and, thus, the algorithm is capable of fitting data sets with almost arbitrary high dimensions.

4. 1994 AIDS Study

The 1994 AIDS data [34] were originally collected in order to compare two antiretroviral drugs based on a collective of HIV-positive patients. They include 1405 longitudinal observations of 467 individuals, from which, 188 unfortunately died during the course of the study. The main longitudinal outcome is each patient’s repeatedly measured CD4 cell counts. CD4 cells decline in HIV-positive patients and are a well-known proxy for disease progression, and are therefore of high interest. Apart from the CD4 cell count as the longitudinal outcome, death as the time-to-event outcome and time t itself, the four additional baseline variables—drug (treatment group), gender, AZT (indicator of whether a previous AZT therapy failed) and AIDS (indicator of whether AIDS is diagnosed)—were observed. The structure of the data is depicted in

Table 4. Structure of the data with primary outcomes for the joint analysis in the three columns on the left.

y	T	$\mathit{\delta }$	t	Drug	Gender	AZT	prevOI	id
10.67	16.97	0	0	ddC	male	intolerance	AIDS	1
8.43	16.97	0	6	ddC	male	intolerance	AIDS	1
9.43	16.97	0	12	ddC	male	intolerance	AIDS	1
6.32	19.00	0	0	ddI	male	intolerance	noAIDS	2
8.12	19.00	0	6	ddI	male	intolerance	noAIDS	2
4.58	19.00	0	12	ddI	male	intolerance	noAIDS	2
5.00	19.00	0	18	ddI	male	intolerance	noAIDS	2
3.46	18.53	0	0	ddI	female	intolerance	AIDS	3
3.61	18.53	0	2	ddI	female	intolerance	AIDS	3
6.16	18.53	1	6	ddI	female	intolerance	AIDS	3
⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮

.

Figure 4 depicts the coefficient paths computed by the JMalct algorithm and the corresponding allocation process. The variable AIDS is selected into the longitudinal sub-model right away and frequently updated. This is not surprising, as the diagnosis of AIDS is by definition partly linked to the CD4 cell count. The remaining variables drug and gender were also allocated to the longitudinal sub-model by a smaller amount, whereas AZT was selected into the survival predictor, indicating an increased risk of death for patients with failed AZT therapy.

5. Discussion and Outlook

Finding adequate data-driven allocation mechanisms for joint models is a very important task, as modeling possibilities increase exponentially with a growing number of covariates. Until today, decisions about the specific model choice have to be made based on background knowledge or by conducting a preliminary analysis, and both of these approaches can be seen as rather unsatisfactory.

The JMalct algorithm combines recent findings from the field of gradient boosting to construct a fast-performing allocation and selection mechanism for a joint model focusing on time-to-event data as the primary outcome. A simulation study revealed that the selection and allocation mechanism yields promising results while preserving the well-known advantages from gradient boosting. Therefore, it is advised to use the JMalct algorithm in its current form in advance of the actual analysis in order to determine an allocation of covariates, which is then fitted using convenient frameworks such as JM.

Possible ways of improving the accuracy of estimates and allocation properties regarding the survival sub-model could be based on additional weighting rules. As the longitudinal part contributes substantially more to the likelihood due the higher number of observations, weighting the two sub-models solely by different step lengths may not be sufficient. Promising ideas are initial weightings of the sub-models using various maximum likelihood estimations or focusing on a relative likelihood improvement in the selection step.

Another aspects focuses on variable selection and tuning the algorithm via probing. Although probing leads to fast runtimes and good selection properties, the procedure comes with disadvantages. Especially in higher dimensions, the probability that one shadow variable is informative simply by chance increases, leading to very early stopping. An alternative could rely on stability selection [35,36], as shown to be helpful in other cases [37].

Furthermore, the difference in the proportion of falsely selected variables between the longitudinal and survival sub-predictor could be an inherent joint modeling problem and should also be subject of future analysis. Further research is also warranted on theoretical insights, as it remains unclear if the existing findings on the consistency of boosting algorithms [32,38] also hold for the adapted versions for boosting joint models.

In conclusion, the JMalct algorithm represents a promising statistical inference scheme for joint models that also provides a starting point for a much wider framework of boosting joint models, covering a great range of potential models and types of predictor effects.