On the Relation between Prediction and Imputation Accuracy under Missing Covariates

Abstract

Missing covariates in regression or classification problems can prohibit the direct use of advanced tools for further analysis. Recent research has realized an increasing trend towards the use of modern Machine-Learning algorithms for imputation. This originates from their capability of showing favorable prediction accuracy in different learning problems. In this work, we analyze through simulation the interaction between imputation accuracy and prediction accuracy in regression learning problems with missing covariates when Machine-Learning-based methods for both imputation and prediction are used. We see that even a slight decrease in imputation accuracy can seriously affect the prediction accuracy. In addition, we explore imputation performance when using statistical inference procedures in prediction settings, such as the coverage rates of (valid) prediction intervals. Our analysis is based on empirical datasets provided by the UCI Machine Learning repository and an extensive simulation study.

1. Introduction

The presence of missing values in data preparation and data analysis makes the use of state-of-the art statistical methods difficult to apply. Seeking a universal answer to such problems was the main idea of [1], who introduced (multiple) imputation. Through imputation, one provides data analysts (sequences) of completed datasets, based on which, various data analysis procedures can be conducted. An alternative to imputation is the use of so-called data adjustment methods: statistical methods that directly treat missing instances during training or parameter estimation, such as the full-information-maximum-likelihood method (see, e.g., [2]) or the expectation-maximization algorithm (cf. [3]).

A large disadvantage of these methods is the expertise knowledge on theoretical model construction, where the likelihood function of parameters of interest needs to be adopted appropriately in order to account for missing information. Such examples can be found in [4,5,6], where whole statistical testing procedures were adjusted to account for missing values. It is already well-known that more naive methods, such as list-wise deletion or mean imputation can lead to severe estimation bias, see, e.g., [1,7,8,9,10]. Therefore, we do not discuss these approaches further.

In the current paper, we focus on regression problems, where we do not have complete information on the set of covariates. Missing covariates in supervised regression learning have been part in a variety of theoretical and applicative research fields. In [10], for example, a theoretical analysis based on maximum semiparametric likelihood for constructing consistent regression estimates was conducted. While in [11,12] or [13], for example, multiple imputation is used as a tool in medical research for variable selection or bias-reduction in parameter estimation. More recent research has focused on Machine-Learning (ML)-based imputation.

In [14,15,16,17], for example, the Random Forest method was used to impute missing values in various datasets with mixed variable scales under the assumption of independent measurements. Other ML-based methods, such as the k-nearest neighbor method, boosting machines and Bayesian Regression in combination with classification and regression trees, have been part of (multiple) imputation, see, e.g., [18,19,20,21,22].

Modeling dependencies in imputation methods for multi-block time series data or repeated measurement designs is, however, a non-trivial underpinning. Imputation methods for such time series data can be found in, e.g., [23,24] or [25], for example. Therein, a special focus has been placed on (data) matrix factorization methods, such as the singular value decomposition. In our setting, however, a focus is placed on matrix completion methods using ML-approaches, such as tree-based algorithms.

Choosing an appropriate imputation method for missing data problems usually depends on several aspects, such as the capability to treat mixed-type data, to impute reasonable values in variables of, e.g., bounded support, and to provide a fast imputation solution. Imputation accuracy can usually be assessed through the consideration of performance measures. Here, depending on the subsequent application, one may focus on the data reproducibility measured through the normalized root mean squared error ($NRMSE$) and proportion of false classification ($PFC$) or distribution preserving measures, such as Kolmogorov–Smirnov based statistics, see, e.g., [14,21,26].

It is important to realize that these two classes of performance measures for the evaluation of imputation do not always agree, see, e.g., [26,27]. In fact, one could be provided by an imputation scheme with comparably low $NRMSE$ values, for which subsequent statistical inference can have highly-inflated type-I error rates. Therefore, choosing imputation methods by solely focusing on data reproducibility will not always lead to correct inference procedures. To control for the latter, [1] coined the term proper imputation: a property that guarantees correct statistical inference under the (multiple) imputation scheme.

While imputation accuracy can be accessed through data reproducibility or distributional recovery, the prediction performance of subsequently applied ML methods (i.e., after imputation) is often evaluated using the mean-squared error ($MSE$) or the misclassification error ($MCE$). Under missing covariates, however, the sole focus on these measures is not sufficient, as the disagreement between data reproducibility and statistical inference has shown. In fact, beyond point-prediction, we may also be interested in uncertainty quantifiation in the form of prediction intervals. The effect of missing covariates on the latter remains mostly unknown. In this paper, we aim to close this gap in empirical and simulation-based scenarios.

We thereby place a special focus on the relation between data reproducibility and correct statistical inference for post-imputation prediction. While the latter could also be measured by distributional discrepancies as in [26], we aim to place a special focus on coverage rates and prediction interval lengths in post-imputation prediction instead. The reason for this is the rise in ML-based imputation methods and their competitive predictive performance for both imputation and prediction. Taking into account that data reproducibility and statistical inference are not always in harmony under the missing framework, an interesting question remains whether appropriate data reproducible imputation schemes lead to favourable prediction results after imputation.

Furthermore, it is unknown whether imputation schemes with comparably low $NRMSE$ values will lead to accurate predictive machines in terms of delivering appropriate point predictions for future outcomes while also correctly quantifying their uncertainty. Therefore, based on different ML methods in supervised regression learning problems, we

(i)

aim to clarify the interaction between $NRMSE$ and $MSE$ as measures for data reproducibility and predictive post-imputation ability. We estimate both measures on various missing rates, imputation methods and prediction models to account for potential interactions between the $NRMSE$ and $MSE$.

(ii)

Furthermore, we aim to enlighten the general issue whether imputation methods with comparably low imputation errors also lead to correct predictive inference. That is, we analyze the impact of accurate data-reproducible imputation schemes on predictive, post-imputation statistical inference in terms of correct uncertainty quantification. To measure the latter, we take into account correct coverage rates and (narrow) interval lengths of point-wise prediction intervals in post-imputation settings obtained through ML methods.

2. Measuring Accuracy

Measuring imputation accuracy can happen in many ways. A great deal of research has been focused on the general idea of reconstructing missing instances and being as close as possible to the true underlying data (cf. [14,21]).

Although this approach seems reasonable, several disadvantages have been discovered when using data reproducible measures, such as squared error loss, especially for later statistical inference; see, e.g., [1,26,27]. Therefore, we discuss several suitable measures for assessing prediction accuracy in regression learning problems with missing covariates. In the sequel, we assume that we have access to iid data collected in ${\mathcal{D}}_n=\{{[{\mathit{X}}_i^{\top },Y_i]}^{\top }\in {\mathbb{R}}^{p+1}:i=1,\dots ,n\}$, where

$$\begin{array}{c}\hfill Y_i=m\left({\mathit{X}}_i\right)+{ϵ}_i.\end{array}$$

(1)

Here, $m\left(\mathit{x}\right)=\mathbb{E}\left[Y_1\right|{\mathit{X}}_1=\mathit{x}]$ is the regression function, ${\left\{{ϵ}_i\right\}}_{i=1}^n$ is a sequence of iid random variables with $\mathbb{E}\left[{ϵ}_1\right]=0$ and $Var\left({ϵ}_1\right)={\sigma }^2\in (0,\infty )$ and we assume continuous covariates ${\mathit{X}}_i$. Missing positions in the features ${\left\{{\mathit{X}}_i\right\}}_{i=1}^n$ are modeled by the indicator matrix $\mathit{R}={\left\{R_{ij}\right\}}_{ij}\in {\{0,1\}}^{n\times p}$, where $R_{ij}=0$ indicates that the i-th observation of feature $j\in \{1,\dots ,p\}$ is not observable. Focusing on the general issue of predicting outcomes in regression learning for new feature outcomes, we restrict our attention to data reproducible accuracy measures and model prediction accuracy measures. In order to cover statistical inference correctness for prediction, we use ML-based prediction intervals as proposed in [28,29,30] to account for coverage rates and interval lengths.

2.1. Imputation and Prediction Accuracy

In our setting, (missing) covariates are continuously scaled leading to the use of accuracy measures for continuous random variables. Regarding imputation accuracy, we consider the $NRMSE$ formally given by

$$\begin{array}{cc}\hfill NRMSE& =\frac{\sqrt{{\displaystyle \sum _{(i,j)\in N_{mis}}}{(X_{ij}^{imp}-X_{ij}^{mis})}^2}}{\sqrt{{\displaystyle \sum _{(i,j)\in N_{mis}}}{(X_{ij}^{imp}-{\overline{X}}_{··}^{mis})}^2}},\hfill \end{array}$$

(2)

where $N_{mis}=\{(i,j)\in \{1,\dots ,n\}\times \{1,\dots ,p\}|R_{ij}=0\}$ is the set of all observations and features with missing entries in those positions. Here, $X_{ij}^{imp}$ denotes the imputed value of observation i for variable j, while $X_{ij}^{mis}$ is the true, unobserved component of those positions. $X_{··}^{mis}$ is the mean of the sequence $\{X_{ij}:R_{ij}=0\}$. Regarding the overall model performance on prediction, we make use of the mean squared error

$$\begin{array}{cc}\hfill MSE& =\mathbb{E}\left[{(Y-{\widehat{m}}_n\left(\mathit{X}\right))}^2\right],\hfill \end{array}$$

(3)

where ${\widehat{m}}_n$ is an ML-based estimator of m on ${\mathcal{D}}_n$ and ${[{\mathit{X}}^{\top },Y]}^{\top }$ is an independent copy of ${[{\mathit{X}}_1^{\top },Y_1]}^{\top }$. Note that, in the missing framework, m is estimated on the imputed dataset ${\mathcal{D}}_n^{imp}$, while the $MSE$ is (usually) estimated using cross-validation procedures.

2.2. Prediction Intervals

Based on the methods for uncertainty quantification proposed in [28,29,30,31], we make use of Random-Forest-based prediction intervals. In an extensive simulation study in [30], it could be seen that other, ML-based prediction intervals, such as the (stochastic) gradient tree boosting (cf. [32]) or the XGBoost method (cf. [33]) did not perform well under completely observed covariates. Therefore, we restrict our attention to those already indicating accurate coverage rates in completely observed settings. Meinshausen’s Quantile Regression Forest (see [28]), for example, delivers a point-wise prediction interval for an unseen feature point $\mathit{X}=\mathit{x}$, which is formally given by

$$\begin{array}{cc}\hfill P{I}_{QRF,n}& =[{\widehat{Q}}_{n,\alpha /2}\left(\mathit{x}\right);{\widehat{Q}}_{n,1-\alpha /2}\left(\mathit{x}\right)],\hfill \end{array}$$

(4)

where ${\widehat{Q}}_{n,\alpha /2}\left(\mathit{x}\right)=inf\left\{y\right|{\widehat{F}}_n\left(y\right|\mathit{x})\ge \alpha /2\}$ and ${\widehat{F}}_n\left(y\right|\mathit{x})$ is a Random-Forest-based estimator for the conditional distribution function $F\left(y\right|\mathit{x})$ of $Y|\mathit{X}=\mathit{x}$. Other prediction intervals based on the Random Forest are, e.g., given in [29,30]. Following the same notation as in [30], we refer with ${\widehat{m}}_{n,M}\left(\mathit{x}\right)$ to a Random Forest prediction at $\mathit{x}$, trained on ${\mathcal{D}}_n$ using M decision trees, while $z_{1-\alpha }$ is the corresponding quantile of the standard normal distribution. We consider the same residual variance estimators as in [30], where ${\widehat{\sigma }}_{n,M}$ is the trivial residual variance estimate, ${\widehat{\sigma }}_{n,Mcorrect}$ is the residual variance estimate with finite-M bias correction and ${\widehat{\sigma }}_{n,M;W}$ is a weighted residual variance estimator, see also [31]. Moreover, we denote with ${\widehat{D}}_{n,\alpha /2}^{RF}$ the empirical quantile of the Random Forest Out-of-Bag residuals. With this notation, we obtain four more prediction intervals:

$$\begin{array}{cc}\hfill P{I}_{n,empQ}\left(\mathit{x}\right)& =[{\widehat{m}}_{n,M}\left(\mathit{x}\right)+{\widehat{D}}_{n,\alpha /2}^{RF};{\widehat{m}}_{n,M}\left(\mathit{x}\right)+{\widehat{D}}_{n,1-\alpha /2}^{RF}],\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill P{I}_{n,ResVar}\left(\mathit{x}\right)& =[{\widehat{m}}_{n,M}\left(\mathit{x}\right)-z_{1-\alpha /2}·{\widehat{\sigma }}_{n,M};{\widehat{m}}_{n,M}\left(\mathit{x}\right)+z_{1-\alpha /2}·{\widehat{\sigma }}_{n,M}],\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill P{I}_{n,Mcorrect}\left(\mathit{x}\right)& =[{\widehat{m}}_{n,M}\left(\mathit{x}\right)-z_{1-\alpha /2}·{\widehat{\sigma }}_{n,Mcorrect};{\widehat{m}}_{n,M}\left(\mathit{x}\right)+z_{1-\alpha /2}·{\widehat{\sigma }}_{n,Mcorrect}],\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill P{I}_{n,weighted}\left(\mathit{x}\right)& =[{\widehat{m}}_{n,M}\left(\mathit{x}\right)-z_{1-\alpha /2}·{\widehat{\sigma }}_{n,M;W};{\widehat{m}}_{n,M}\left(\mathit{x}\right)+z_{1-\alpha /2}·{\widehat{\sigma }}_{n,M;W}].\hfill \end{array}$$

(8)

For benchmarking, we additionally consider a prediction interval obtained under the linear model assumption. Imputation accuracy in inferential prediction under missing covariates is then assessed by considering Monte-Carlo estimated coverage rates and interval lengths.

3. Imputation and Prediction Models

We made use of the following state-of-the-art ML regression models for prediction

• the Random Forest as implemented in the R-package ranger (c.f [17]),
• the (stochastic) gradient tree boosting (SGB) method from the R-package gbm (cf. [32]) and
• the XGBoost method, also known as Queen of ML (cf. [34]), as implemented in the R-package xgboost.

For each of them, we fit a prediction model to the (imputed) data. Both boosting methods rely on additive regression trees that are fitted sequentially using the principles of gradient descent for loss minimization. XGBoost, however, is slightly different by introducing extra randomization in tree construction, a proportional shrinkage on the leaf nodes and a clever penalization of trees. We refer to [32,33,35] for details on the concrete algorithms. For benchmarking, a linear model is trained as well.

Although several imputation models are available on various (statistical) software packages, we place a special focus on Random-Forest-based imputation schemes and the multivariate imputation using chained equations (MICE) procedure (cf. [11,14,21]). The reasons for this are twofold, but both have roots in the same theoretical issue called congeniality, see [36] for a formal definition. In its core, congeniality in (multiple) imputation refers to the existence of a Bayesian model such that

• the posterior mean and posterior variance of the parameter of interest agrees with the point estimator resp. its variance estimator calculated under the analysis model and
• the conditional distribution of the missing observations given the observed points under the considered Bayesian model agrees with the imputation model.

Therefore, congeniality builds a bridge between the imputation and analysis procedure by assuming the existence of a larger model that is compatible with both—the analysis and imputation model. If ML methods are used during the analysis phase, the compatibility in terms of congeniality is non-trivial. Using the same methods during the imputation and analysis phase, however, can ease the verification through the use of the same model during imputation and analysis.

Hence, a potential disagreement of imputation and prediction models, however, can result in uncongenial (multiple) imputation methods. The latter yields to invalid (multiple) imputation inference, as can be seen in [37] or [36], for example. Secondly, focusing on Bayesian models for imputation, such as the MICE procedure, is in line with the general framework of congeniality and the idea of (multiple) imputation. Although we do not directly compute point-estimates during the analysis phase, interesting quantities in our framework are Random-Forest-based prediction intervals and estimators of the $MSE$.

missForest in R is an iterative algorithm developed by [14] that imputes continuous and discrete random variables using trained Random Forests on complete subsets of the data and imputes missing values through prediction with the trained Random-Forest model. The process iterates in imputing missing values until a pre-defined stopping criterion is met. Similar to the missForest algorithm, we substituted the core learning method with other ML-based methods, such as the SGB method (in the sequel referred to as the gbm for the algorithmic implementation) and the XGBoost method (in the sequel referred to with xgboost for the algorithmic implementation).

Both methods are implemented in R using the same algorithmic framework as missForest, while substituting the Random Forest method with the SGB resp. XGBoost. That means that we train the SGB resp. XGBoost on (complete) subsets of the data and impute missing values through the prediction of the trained model in an iterative fashion.

MICE is a family of Bayesian imputation models developed in [38,39]. Under the normality assumption (i.e., MICE NORM), the method assumes a (Bayesian) linear regression model, where every parameter in that model is drawn from suitable priors. The predictive mean matching approach (MICE PMM) is similar to MICE NORM but does not impute missing values through the prediction of those points using the Bayesian linear model and instead randomly selects among observed points that are closest to the same model prediction as MICE NORM. In addition to these methods, MICE enables the implementation of Random-Forest-based methods, referred to as MICE RF, see, e.g., [40].

The latter assumes a modified Random Forest, where additional randomization is applied compared to the missForest. For example, instead of simply predicting missing values through averaging observations in leaf nodes, the method randomly selects them. In addition, in the complete subset of the data determined for training the Random Forest, potential missing values are not initially imputed by mean or mode values but by random draws among observed values. In the sequel, we refer to the algorithmic implementation in R of all these methods using the terms mice_norm, mice_pmm and mice_rf.

4. Simulation Design

Our simulation design is separated in two parts. In the first part, empirical data from the UCI Machine Learning Repository covering regression learning problems are considered for the purpose of measuring imputation and prediction accuracy. We focused on selecting datasets from the repository that reveal a high amount of continous variables, while reflecting both time series data and observations measured as independent and identical realizations of random variables with different dimensions. Summary statistics of every dataset can be found in Appendix A. The following five datasets are considered:

• The Airfoil Data consists of $(p+1)=6$ variables measured in $n=1503$ observations, where the target variable is the scaled sound pressure level measured in decibels. The aim of this study conducted by NASA was to detect the impact of physical shapes of airfoils on the produced noise. The data consists of several blades measured in different experimental scenarios, such as various wind attack angles, free-stream velocities and frequencies. We may assume iid observations ${[{\mathit{X}}_i^{\top },Y_i]}^{\top }$, $i=1,\dots ,n$ for every experimental setting.
• In the Concrete Data, $(p+1)=7$ variables are measured in $n=1030$ observations. The target variable is the concrete compressive strength measured in MPa units. Different mixing components, such as cement, water and fly ash, for example, are used to measure the concrete strength. It is reasonable to assume iid realizations from ${[{\mathit{X}}_i^{\top },Y_i]}^{\top }$ for $i=1,\dots ,n$.
• The aim in the QSAR Data is to predict aquatic toxicity for a certain fish species. It consists of $(p+1)=9$ variables measured in $n=546$ observations. The $n=546$ observations can be considered as iid realizations of ${[{\mathit{X}}_i^{\top },Y_i]}^{\top }$ for $i=1,\dots ,n$, while all features and the response are continuous.
• The Real Estate Data has $(p+1)=7$ variables and $n=413$ observations. The aim is to build a prediction model for house price developments in the area of New Taipei City in Taiwan. Different features, such as the house age or the location measured as a bivariate coordinate vector, for example, are measured for building a prediction model. In our simulation, we dropped the variable transaction date and assumed an row-wise iid structure. The dataset, however, can also be considered as time series data.
• The Power Plant Data consists of $n=9568$ observations with $(p+1)=5$ variables. The actual dataset is much larger in terms of observations; however, only the first 9568 are selected to speed the computations. The aim of this dataset is to predict the electric power generation of a water power-plant in Turkey. This dataset is different from the previous ones due to its time series structure. The dataset can be considered as multiple time series measured in five different variables.

For each dataset, missing values under the MCAR scheme were inserted to the $(n\times p)$-dimensional dataset with $r\in \{0.1,0.2,0.3,0.5,0.6,0.8\}$ missing rates. Hence, missing values are randomly spread across cases and variables in the dataset. Then, missing values were (once) imputed with the imputation methods mentioned in Section 3.

Although multiple imputation can be very beneficial when analyzing coverage rates for prediction intervals (see, e.g., [41] or [1]) in terms of more accurate uncertainty reflection of the missing mechanism itself, our considered methods, however, are partly limited to be applied within the multiple imputation framework. In [42], for example, the missForest procedure was shown to be not multiple imputation proper making its direct usage in the multiple imputation scheme limited. Once missing values are imputed, the whole process is then iterated using $M{C}_{imp}=500$ Monte-Carlo iterates.

Based on each imputed dataset, all of the above mentioned prediction models are trained and their prediction accuracy is measured using a five-fold cross-validated $MSE$. Regarding hyper-parameter tuning of the various prediction models, we conducted a grid-search using a ten-fold cross-validation procedure with ten replications on the completely observed data, prior to the generation of missing values. This was conducted using the R-function trainControl of the caret-package [43].

In the second part of our simulation study, synthetic data was generated with missing covariates to detect the effect of imputation accuracy on prediction interval coverage rates. Here, we have focused on point-wise prediction intervals. For sample sizes $n\in \{100,500,1000\}$, regression learning problems of the form ${\left\{{[{\mathit{X}}_i^{\top },Y_i]}^{\top }\right\}}_{i=1}^n$ were generated using a $p=10$ dimensional covariate space and model (1), where ${\mathit{X}}_i\stackrel{\mathrm{iid}}{\sim }{\mathcal{N}}_{p=10}(0,\Sigma )$ and ${ϵ}_i\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,{\sigma }^2)$ were simulated independent of each other. Missing values were inserted under the MCAR scheme using various missing rates $r_{PI}\in \{0.1,0.2,0.3\}$. Regarding the functional relationship between features and response, different regression functions with coefficient ${\mathbf{\beta }}_0={[2,4,2,-3,1,7,-4,0,0,0]}^{\top }$ were used, such as

• a linear model: $m\left({\mathbf{x}}_i\right)={\mathbf{x}}_i^{\top }{\mathbf{\beta }}_0$,
• a polynomial model: $m\left({\mathbf{x}}_i\right)=\sum _{j=1}^p{\beta }_{0,j}{x}_{i,j}^j$,
• a trigonometric model: $m\left({\mathbf{x}}_i\right)=2·sin({\mathbf{x}}_i^{\top }{\mathbf{\beta }}_0+2)$ and
• a non-continuous model:

  $$\begin{array}{cc}\hfill m\left({\mathbf{x}}_i\right)& =\left\{\begin{array}{cc}{\beta }_{0,1}{x}_{i,1}+{\beta }_{0,2}{x}_{i,2}+{\beta }_{0,3}{x}_{i,3},\hfill & \mathrm{if}{x}_{i,3}>0.5,\hfill \\ {\beta }_{0,4}{x}_{i,4}+{\beta }_{0,5}{x}_{i,5}+3\hfill & \mathrm{if}{x}_{i,3}\le 0.5.\hfill \end{array}\right.\hfill \end{array}$$

In order to capture potential dependencies among the features, various choices for the covariance matrix $\Sigma$ were considered: a positive auto-regressive, negative auto-regressive, compound symmetric, Toeplitz and the scaled identity structure. In addition, we aimed to take care of the systematic variation originating from $m\left({\mathit{X}}_1\right)$, and the noise ${ϵ}_1$, by choosing ${\sigma }^2$ in such a way that the signal-to-noise ratio $SN:=Var\left(m\left({\mathit{X}}_1\right)\right)/{\sigma }^2=1$. Finally, using $M{C}_{PI}=1000$ Monte-Carlo iterations, prediction interval performance of the intervals proposed in Section 2 are evaluated by approximating coverage rates and (average) interval lengths over the Monte-Carlo iterates.

5. Simulation Results

In the sequel, the simulation results for both parts, the empirical datasets obtained through the UCI Machine Learning repository and the simulation study are presented. Note that additional results can be found in Appendix A and in the supplement of [44].

5.1. Results on Imputation Accuracy and Model Prediction Accuracy

In this section, we present the results for the empirical data analysis based on the Airfoil dataset using the imputation and prediction accuracy measures described in Section 2 for evaluation. We thereby focus on the Random Forest and the XGBoost prediction model. The results of the linear and the SGB model as well as the results for all other datasets are given in the supplement in [44] (see Figures 1–19 therein) and summarized at the end of this section.

Random Forest as Prediction Model.Figure 1 and Figure 2 summarize for each imputation method the imputation error ($NRMSE$) and the model prediction error ($MSE$) over $M{C}_{imp}=500$ Monte-Carlo iterates using the Random Forest method for prediction on the imputed dataset. On average, the smallest imputation error measured with the $NRMSE$ could be attained when using missForest and the gbm imputation method. In addition, these methods yielded low variations in $NRMSE$ across the Monte-Carlo iterates. In contrast, the mice_norm, mice_pmm and mice_rf behaved similarly resulting into largest $NRMSE$ values across the different imputation schemes with an increased variation in $NRMSE$ values.

The xgboost method performed slightly worse than missForest and gbm, when focusing on imputation accuracy. In addition, all methods seemed to be more or less robust towards an increased missing rate. Interesting is the fact that volatility decreases, as missing rates increase for the MICE procedures. The prediction accuracy measured in terms of cross-validated $MSE$ using the Random-Forest model was the lowest under the missForest, xgboost and gbm, which corresponds with the $NRMSE$ results.

As expected, the estimated $MSE$ suffered from missing covariates and the effect became worse with an increased missing rate. For example, an increase in the missing rate from 10% to 50% yielded an increase of the $NRMSE$ by 8.4%, while the $MSE$ realized an increase of 127.6%. Hence, model prediction accuracy heavily suffered from an increased amount of missing values, independent of the used imputation scheme. In addition, if the $NMRSE$ increases by $0.1$ units, it is expected that the $MSE$ will increase by 122.1%. Although congeniality was defined for valid statistical inference procedures, the effect of using the same method for imputation and prediction seemed to also have a positive effect on model prediction accuracy.

XGBoost as Prediction Model. In switching the prediction method to XGBoost, we realized an increase in model prediction accuracy for missing rates up to $20\%$ as can be seen in Figure 3. In addition, for those missing rates, the xgboost imputation was competitive to the missForest method but lost in accuracy for larger missing rates compared to the missForest. Different from the Random Forest, the XGBoost prediction method was more sensitive towards an increased missing rate.

For example, an increase of the missing rate from 10% to 50% yields to an increase of the $NRMSE$ by 8.1%, while the $MSE$ suffered by an increase of 300% on average. In addition, an increase of the imputation error by $0.1$ points, can yield an average increase of prediction error by 189.9% Although under the completely observed framework the XGBoost method performed best in terms of estimated $MSE$, the results indicate that missing covariates can disturb the ranking. In fact, for missing rates $r\ge 30\%$, the Random Forest exhibited a better prediction accuracy.

Other Prediction models. Using the linear model as the prediction model resulted in worse prediction accuracy with $MSE$ values ranging from 25 ($r=10\%$) to 45 ($r=50\%$). For all missing scenarios, using the missForest or the gbm method for imputation before prediction with the linear model resulted in the lowest $MSE$. The results for the SGB method were even worse with $MSE$ values between 80 and 99.

As a surprising result, the prediction accuracy measured in terms of cross-validated $MSE$ decreased with an increasing missing rate. A potential source of this effect could be the general weakness of the SGB in the Airfoil dataset without any missing values. After inserting and imputing missing values, which can yield to distributional changes of the data, it seems that the SGB method benefits from these effects. However, model prediction accuracy is still not satisfactory, see Figure 2 in the supplement in [44].

Other Datasets. For the other datasets, similar effects were obtained. The Random Forest and the XGBoost showed the best prediction accuracy, see Figures 4–19 in the supplement in [44]. Again, larger missing rates affected model prediction accuracy for the XGBoost method, but the Random Forest was more robust to them overcoming XGBoost prediction performance measured in cross-validated $MSE$ for larger missing rates. Overall, $NRMSE$ and cross-validated $MSE$ seem to be positively associated to each other. Hence, more accurate imputation models seemed to yield better model prediction measured by $MSE$.

5.2. Results on Prediction Coverage and Length

Using the prediction intervals in Section 2, we present coverage rates and interval lengths of point-wise prediction intervals in simulated data. Both quantities were computed using 1000 Monte-Carlo iterations with sample sizes $n\in \{100,500,1000\}$. The boxplots presented here (see Figure 4, Figure 5, Figure 6 and Figure 7) and in the supplement in [44] (see Figures 19–30 therein) spread over the different covariance structures used during the simulation. Every row corresponds to one of the simulated missing rates $r\in \{0.1,0.2,0.3\}$, while the columns reflect the different Random-Forest-based prediction intervals.

The left column summarizes the results for the Random-Forest-based prediction interval using empirical quantiles ($P{I}_{n,empQ}$), the center column reflects the Random Forest prediction interval using the simple residual variance estimator on Out-of-Bag errors ($P{I}_{n,ResVar}$), while the right column summarizes the Random-Forest-based prediction interval using the weighted residual variance estimator ($P{I}_{n,weighted}$). We shifted the results of $P{I}_{QRF,n}$, $P{I}_{n,MCorrect}$ and the prediction interval based on the linear model to the supplement in [44] (see Figures 19–22, 24, 26, 28 and 30 therein). Under the complete case scenarios, the latter methods did not show comparably well coverage rates as $P{I}_{n,emQ}$, $P{I}_{n,ResVar}$ and $P{I}_{n,weighted}$. For imputed missing covariates, the methods performed with less accuracy in terms of correct coverage rates, when comparing them with $P{I}_{n,emQ}$, $P{I}_{n,ResVar}$ and $P{I}_{n,weighted}$.

Although the interval lengths of $P{I}_{QRF,n}$, $P{I}_{n,MCorrect}$ and the linear model were, on average smaller, the coverage rate was not sufficient to make them competitive with $P{I}_{n,emQ}$, $P{I}_{n,ResVar}$ and $P{I}_{n,weighted}$. For prediction intervals that underestimated the $0.95$ threshold in the complete case scenario, we observed more accurate coverage rates for larger missing rates. It seems that larger missing rates increase coverage rates for the $P{I}_{QRF,n}$ and $P{I}_{n,MCorrect}$ methods, independent of the used imputation scheme.

In Figure 4, the boxplots of the linear regression model are presented. In general, the use of Random-Forest-based prediction intervals with empirical quantiles ($P{I}_{n,empQ}$) or simple variance estimation ($P{I}_{n,ResVar}$) show competitive behavior in the complete case scenario. When considering the various imputation schemes, under the different missing rates, it can be seen that coverage rate slightly suffered compared to the complete case. To be more precise, larger missing rates lead to slightly larger coverage rates for $P{I}_{n,emQ}$, $P{I}_{n,ResVar}$ and $P{I}_{n,weighted}$.

For the Random-Forest-based prediction interval with weighted residual variance, this effect seems to be positive, i.e., larger missing rates will lead to better coverage rates for $P{I}_{n,weighted}$. Comparing the results with the previous findings, we see that the xgboost yields, on average, the best coverage results across the different imputation schemes. While the MICE procedures did not reveal competitive performance in model prediction accuracy, the mice_norm method under the linear model performed similar to the missForest procedure when comparing coverage rates.

Figure 5 summarizes coverage rates of point-wise prediction intervals under the trigonometric model. Similar to the linear case, all three methods $P{I}_{n,empQ}$, $P{I}_{n,ResVar}$ and $P{I}_{n,weighted}$ yielded accurate coverage rates showing better approximation to the $0.95$ threshold when the sample size increase under the complete observation case. On average, the xgboost imputation method remains competitive compared to the other imputation methods.

Slightly different from the linear case, the mice_norm approach gains in correct coverage rate approximation compared to the missForest, together with the mice_pmm approach. Nevertheless, the approximations between mice_norm, mice_pmm and missForest are close to each other. As mentioned earlier, $P{I}_{n,weighted}$ turns more accurate in terms of correct coverage rates, when the missing rate increases. Similar results compared to the linear and trigonometric case could be obtained for the polynomial model and the non-continuous model. Boxplots of the coverage rates can be found in Figures 24 and 28 of the supplement in [44].

Regarding the length of the intervals for the linear model (see Figure 6), the prediction interval $P{I}_{n,empQ}$ and the parametric interval $P{I}_{n,ResVar}$ yielded similar interval lengths. Under imputed missing covariates, however, the $P{I}_{n,empQ}$ interval led to slightly smaller intervals than $P{I}_{n,ResVar}$. Nevertheless, the prediction interval based on the weighted residual variance estimator $P{I}_{n,wighted}$ had the smallest intervals on average. This comes with the cost of less accurate coverage rates as can be seen in Figure 4.

In addition, independent of the used prediction interval, an increased missing rate yielded larger intervals making the learning methods, such as Random Forest, more insecure about future predictions. Regarding the used imputation method, almost all imputation methods resulted in similar interval lengths. On average, the missForest method had slightly smaller intervals comparable to the xgboost imputation.

Similar results on prediction lengths were obtained with other models. Considering the trigonometric function as in Figure 7, it can be seen that $P{I}_{n,empQ}$ results in slightly smaller intervals than $P{I}_{n,ResVar}$. However, the interval lengths for the empirical quantiles under the trigonometric model were more robust towards dependent covariates.

Comparably to the linear case, $P{I}_{n,weighted}$ results in the smallest interval lengths, but suffers from less accurate coverage. Furthermore, all imputation methods behave similar with respect to prediction interval lengths under the trigonometric case and other models (see Figures 21 and 22 in the supplement in [44]). It can be seen that Random-Forest-based prediction intervals are, more or less, universally applicable to the different imputation schemes used in this scenario yielding similar interval lengths.

In summary, Random-Forest-based prediction intervals with imputed missing covariates yielded slightly wider intervals compared to the regression framework without missing values. For prediction intervals that underestimated the true coverage rate, such as $P{I}_{QRF,n}$, $P{I}_{n,MCorrect}$ and $P{I}_{n,weighted}$, an increased missing rate had positive effects on the coverage rate. Overall, missForest and xgboost were competitive imputation schemes when considering accurate coverage rates and interval lengths using the $P{I}_{n,empQ}$ and $P{I}_{n,ResVar}$ intervals. mice_norm resulted in similar, but slightly less accurate, coverage compared to missForest and xgboost.

6. Conclusions

Missing covariates in regression learning problems are common issues in data analysis problems. Providing a general approach for enabling the application of various analysis models is often obtained through imputation. The use of ML-based methods in this framework has obtained increased attention over the last decade, since fast and easy to use ML methods, such as the Random Forest, can provide us with quick and accurate fixes for data analysis problems.

In our work, we placed a special focus on variants of ML-based methods for imputation, which mainly rely on decision trees as base learners, and their aggregation is conducted in a Random-Forest-based fashion or a boosting approach. We aimed to shed light into the general issue when and which imputation method should be used for missing covariates in regression learning problems that provide accurate point predictions with correct uncertainty ranges. To provide an answer to this, we conducted empirical analyses and simulations, which were led by the following questions:

Does an imputation scheme with a low imputation error (measured with the $NRMSE$) automatically provide us with accurate model prediction performance (in terms of cross-validated $MSE$)? How do ML-based imputation methods perform in estimating uncertainty ranges for future prediction points in form of point-wise prediction intervals? Are the results in harmony; that is, does an accurate imputation method with a low $NRMSE$ provide us with good model prediction accuracy measured in $MSE$ while delivering accurate and narrow prediction interval lengths?

By analyzing empirical data from the UCI Machine Learning repository, we found that imputation methods with low imputation error measured with the $NRMSE$ yielded better model prediction measured by cross-validated $MSE$. In our analysis, we could see that an increased missing rate had a negative effect on both the $MSE$ and the $NRMSE$, while on the latter, the effect was less expressive. Particularly, for larger missing rates, the use of the same ML method for both imputation and prediction was beneficial. This is in line with the congeniality assumption; a theoretical term that (partly) guarantees correct inference after (multiple) imputation.

In particular, the missForest and our modified xgboost method for imputation yielded preferable results in terms of a low imputation error and good model prediction. It is expected that ML methods with accurate model prediction capabilities measured in $MSE$ can be transformed to be used as an imputation method yielding low imputation errors as well. Regarding statistical inference procedures in prediction settings, such as the construction of valid prediction intervals, Random-Forest-based imputation schemes, such as the missForest and the xgboost method, yielded competitive coverage rates and interval lengths.

In addition, the MICE procedure with a Bayesian linear regression and normal assumption was under the aspect of correct coverage rates and interval lengths competitive as well. However, the method did not reveal low imputation error and overall good model prediction.

Hence, based on our findings, the missForest and the xgboost method in combination with Random-Forest-based prediction intervals using empirical quantiles resp. Out-of-Bag estimated residual variances are competitive in three aspects: providing low imputation errors measured with the $NRMSE$, yielding comparably low model prediction errors measured by cross-validated $MSE$ and providing comparably accurate prediction interval coverage rates and narrow widths using Random-Forest-based intervals. Regarding the latter, our results also indicate that these intervals are competitively applicable to a wide range of imputation schemes.

In summary, data analysts that fully rely on prediction accuracy after imputing missing data should focus on imputation schemes with comparably low $NRMSE$ as a prior indicator, especially when using tree-based ML methods. In addition, the same or more general imputation methods should be used. However, when moving to predictive statistical inference in terms of accurate prediction coverage rates, the $NRMSE$ is not a direct measure indicating good coverage results.

Future research will focus on a theoretical exploration of the interaction between the $NRMSE$ and $MSE$ and the effect of the considered imputation methods on uncertainty estimators in multiple imputation scenarios. The aim is to discover the type of impact several factors have on the interactions between both measures, such as the missing rate, the missing structure and the used prediction method on more general imputation schemes accounting for multiple imputation as well. Insights into their theoretical interaction will provide additional information to the general issue that imputation is not only prediction.