Survival Prediction of Glioma Patients from Integrated Radiology and Pathology Images Using Machine Learning Ensemble Regression Methods

Abstract

Gliomas are tumors of the central nervous system, which usually start within the glial cells of the brain or the spinal cord. These are extremely migratory and diffusive tumors, which quickly expand to the surrounding regions in the brain. There are different grades of gliomas, hinting about their growth patterns and aggressiveness and potential response to the treatment. As part of routine clinical procedure for gliomas, both radiology images (rad), such as multiparametric MR images, and digital pathology images (path) from tissue samples are acquired. Each of these data streams are used separately for prediction of the survival outcome of gliomas, however, these images provide complimentary information, which can be used in an integrated way for better prediction. There is a need to develop an image-based method that can utilise the information extracted from these imaging sequences in a synergistic way to predict patients’ outcome and to potentially assist in building comprehensive and patient-centric treatment plans. The objective of this study is to improve survival prediction outcomes of gliomas by integrating radiology and pathology imaging. Multiparametric magnetic resonance imaging (MRI), rad images, and path images of glioma patients were acquired from The Cancer Imaging Archive. Quantitative imaging features were extracted from tumor regions in rad and path images. The features were given as input to an ensemble regression machine learning pipeline, including support vector regression, AdaBoost, gradient boost, and random forest. The performance of the model was evaluated in several configurations, including leave-one-out, five-fold cross-validation, and split-train-test. Moreover, the quantitative performance evaluations were conducted separately in the complete cohort (n = 171), high-grade gliomas (HGGs), n = 75, and low-grade gliomas (LGGs), n = 96. The combined rad and path features outperformed individual feature types in all the configurations and datasets. In leave-one-out configuration, the model comprising both rad and path features was successfully validated on the complete dataset comprising HGFs and LGGs ($R=0.84 p=2.2\times {10}^{-16}$). The Kaplan–Meier curves generated on the predictions of the proposed model yielded a hazard ratio of $3.314 \left[95\%CI:1.718-6.394\right], log-rank\left(P\right)=2\times {10}^{-4}$ on combined rad and path features. Conclusion: The proposed approach emphasizes radiology experts and pathology experts’ clinical workflows by creating prognosticators upon ‘rad’ radiology images and digital pathology ‘path’ images independently, as well as combining the power of both, also through delivering integrated analysis, that can contribute to a collaborative attempt between different departments for administration of patients with gliomas.

1. Introduction

Glioma, comprising around 50% of all intracranial primary tumors, is the most common brain tumor. This condition can induce a variety of neurological symptoms, primarily reflecting the function of the tumor site in the brain. This can result in motor dysfunction, cognitive problems, and change in personality. It is important to note that Motor dysfunction relates to schizophrenia, with a preceding clinical symptom in recent longitudinal investigations. The increased occurrence of involuntary motions that are evident from childhood and reflect subcortical brain disorders is of special interest. Cure is now uncommon, and any treatment aims for minimizing the symptoms delay the progression of disease and maximise the length of time of survival with the highest possible quality of life.

The fact that glioma exhibits a wide range of clinical behavior suggests that our present concept of high-grade glioma as a stand-alone disease entity is oversimplified. In addition to host characteristics, such as age and gender, biological variables, including MGMT promoter methylation status, 1p/19q deletion, and IDH1 gene mutation status, were demonstrated to explain some of the observed variance. This also can be highlighted by the fact that molecular subtyping is used to separate.

Gliomas have a dismal prognosis, with recurrence or progression being unavoidable. Its progression patterns can have a wide range of phenotypes, which could alter the treatment approach and clinical result. The inability of current therapeutic choices (chemotherapy, radiation, and surgical resection) to prevent disease development (known as relapse/recurrence) is responsible for bad prognosis. Gliomas infiltrate the underlying brain parenchyma and go far beyond the radiological or surgical brain tumor barrier on a microscopic level. There exist differences in the regional patterns of progression in gliomas, as well as differences in ‘overall survival’ or ‘progression-free-survival’ (PFS) on a clinical level. Such a diverse landscape of these tumors suggests the need to develop imaging-based markers that are prognostic of patient outcome at the initial presentation of the disease to assist in building patient-centric treatment plans [1,2].

Several old analyses show the usefulness of MRI features obtained from different compartments of the gliomas in survival prediction in gliomas [3,4,5]. Macyzyn et al. [6]. extracted intensity, volumetric, and shape features from various subregions of gliomas and applied traditional machine learning models for prediction of OS. Baid et al. [7] also used radiomic features extracted from several MR images to predict disease prognosis, thus providing useful knowledge for personalized care. This work uses FLAIR and T1CE data to extract intensity-based length, first-order textural radiomic, shape-based and characteristics. The region of interest (ROI) is then decomposed using a stationary wavelet transform (WT) with low-pass (LP) and high-pass (HP) filtering. Furthermore, radiomic characteristics are recovered from these deconstructed pictures, assisting in the gathering of directional information. The proposed approach was evaluated using brain tumor segmentation (BraTS) challenge training, validation, and test datasets, providing 0.695, 0.571, and 0.558 on training, validation, and test datasets, respectively. Sun et al. [8] in their survial prediction study, extracted 4524 radiomic features which are segmented from tumor regions, the important features are selected using cross validation and decision tree. Finally, they used the random forest model to achieve 61% classification success in prediction of short verssu long-term survivors. Recently, the deep learning methods have been utilized for survival prediction. Zhao et al. [9] proposed a deep learning architecture, named segmentation then prediction, which was based on 3D U-Net. The proposed architecture comprised a segmentation component, which delineates the tumor into the three main subregions (i.e., whole tumor, enhancing tumor and tumor core), and a feature extractor, by computing features from (i) whole tumor only, and (ii) from final layer of segmentation component. Global and local features were used together to predict survival in gliomas, yielding the validation accuracy of 65.5%. In yet another study on survival prediction of gliomas by Asma et al. [10], authors used multi-regional radiomics analysis by employing the three-subregions radiomics, six-subregions radiomics and twenty one-subregions radiomics model. The top performance was obtained using 3-subregions radiomics model with (mean AUC = 0.73). Wankhede et al. [11] proposed a model that is multilevel layer and dynamic architecture in a faster R-CNN (MLL-CNN) approach. This model utilizes the relative description model based on feature weight factor to build selected features to predict glioblastoma survival. Some researchers also show the effectiveness of combining traditional features extracted from MRI images and the deep learning features extracted by the deep learning models. Hunag et al. [12] extracted intensity, wavelet, texture, shape along with radiological features taken from tumor area, and a CNN network based deep features. They used the random forest regression model for prediction of OS in gliomas and reported a root mean square error of 311.5. Suter [2], used 4 MR images (T1, T2, T1c and FLAIR), extracted an initial set of 1353 features. They used deep learning CNN1 and CNN2 strategies. Their method performed well, but they used only radiomics sets and their characteristics and features.

Some researchers used pathology features obtained from the regions demarcated on pathology images [13] Iftikhar et al. [13] trained a deep neural network on 663 pathology images acquired from TCIA to estimate parameters of microvascular proliferation, nuclear atypia, mitotic activity and the presence of necrosis, and eventually predict OS. They reported a c-statistic of 0.82 (p-value = 4.8 × 10⁻⁵) between overall survival and risk scores of proposed deep learning model. Rathore et al. [14] also confirmed these findings using the same dataset. Similarly, Mobadersany et al. [15] used a survival neural network on 769 pathology images acquired from TCIA to predict OS in gliomas. They reported a c-statistic of 0.77 between OS deep learning model with risk score.

Despite the various studies showing the potential of MRI and pathology images alone in the prediction of OS, the previous studies lack in exploiting the integrated potential of radiology and pathology images, which provide diagnostically relevant information. In these prioir studies, only radiology or pathology images commonly employed, in contrast to synergistically to combine both imaging modalities to build more robust predictive and most powerful model.

To overcome shortcomings, we analyze and integrate entirety of pathology and radiology data and leverage the power of advanced pattern analysis methods to achieve better prognosis of gliomas. We hypothesise in this study that the integration of MRI and digital pathology images can be utilized for better prognosis of gliomas compared to that achieved with either of these imaging modalities. The proposed method utilizes the MRI images, as well as digital pathology images, as input data, and the combination thereof to achieve better prognosis of gliomas. The anticipated technique was assessed on a diverse set of glioma patients, including low-grade gliomas (LGG) and high-grade gliomas (HGG). Ensemble regression models are opted to predict patients’ survival outcome owing to the better performance that ensembles offer over traditional weak learners.

The main contributions of our study are:

• Exploit the rich phenotypic information present in radiology and histology images through multivariate pattern analysis methods for prediction survival in gliomas.
• Perform robust statistical analysis on individual radiology and pathology features and their combinations via multiple permuted iterations.
• Validate the model on a diverse multi-institutional cohort of images by using a complete glioma dataset and also individual cancer grades.

The remining paper is divided in following parts: material and method is described in Section 2. The results are detailed in Section 3, discussion in Section 4, and Section 5 concludes the paper.

2. Materials and Methods

2.1. Dataset

The dataset utilized in this study comprises 171 glioma patients (gender: 91 male and 80 female, tumor grade: 96 LGG and 75 HGG) collected from The Cancer Image Archive (TCIA) and The Cancer Genomic Atlas (TCGA) repository. The data comprised (1) preoperative MRI images, including T1 post contrast gadolinium (T1-GD), T1-wegihted (T1), T2-wegihted (T2), and T2-weighted fluid-attenuated inversion recovery (T2-FLAIR), (2) digital pathology images collected from paraffin-embedded H&E-stained tissue sections, and (3) survival information. Survival outcome was collected from medical records and was defined as the time from the initial presentation of the disease to death or last follow-up time, whatever happened earlier. In our study, some patients’ data was censored because the last follow-up interval preceded the end of the study as well as the death. As such, the survival time can be stated conclusively and is just known to be longer than the latest follow-up.

Table 1. Glioma patients’ characteristics.

Genetic, Clinical Characteristics	Complete Set (n = 171)	Training Group (n = 103, ~60%)	Testing Group (n = 68, ~40%)	p-Value
Median overall survival	467	467	489.5	0.999
No. of uncensored patients	104	63	39	0.619
Age	51.22 ± 15.35	52.50 ± 15.66	49.27 ± 14.76	0.436
Sex				0.493
Male	91	57	34
Female	80	46	34
Tumor grade				0.956
High grade glioma	75	45	30
Low grade glioma	96	58	38
IDH Mutation				0.691
Wildtype	83	50	33
Mutant	77	44	33
NA	11	9	2
1p19q mutation				0.887
Wildtype	53	30	23
Mutant	24	14	10
NA	94	59	35

Note.—Except where otherwise noted, data are numbers of patients with percentages in parentheses. p values were computed by employing Student’s t-test for continuous variables and χ2 test for computing categorical variables. Percentage for 1p/19q computed by utilizing only number of patients having 1p/19q status available. IDH = isocitrate dehydrogenase, 1p/19q = deletion of chromosome arm 1p and/or 19q, NA = not applicable.

provides more details on the dataset.

Figure 1 provides a schematic overview of the proposed ensemble regression-based survival prediction method. Several pre-processing steps are applied both on MRI and digital pathology images. The MRI images were smoothed using the adaptive nonlocal means algorithm [7,8,9] followed by the registration of T1, T2, and T2-FALIR images to T1-CE images using the FLIRT registration method [10], and brain extraction tool was used using skull stripping. The N4 biased field correction method [11] was also applied on the images to remove intensity inhomogeneities happening because of image acquisition. The digital pathology images were converted to grayscale and were calibrated to a reference digital pathology image to correct for color inconsistencies in images acquired across multiple institutions. Feature extraction methods were then applied on the pre-processed images to quantify the imaging profiles of these glioma patients into meaningful quantitative measures. In this study, features extracting techniques, such as GLCM, GLSZM, GLRLM, LBP, and NGTDM, are used to extract relevant information from rad and path images. The final step of the pipeline involves an ensemble of various regression techniques to predict survival outcome of glioma patients. These features were given as input to the final step of the pipeline, which is an ensemble of various regression techniques, including support vector regression (SVR), SVR with RBF kernel and linear kernel, AdaBoost, gradient boost, and random forest, for prediction of survival outcome of glioma patients.

The Table 1 presents the Glioma patients’ characteristics with clinical information of complete set, training and test group along with the p-Value.

2.2. Feature Extraction

This is an important step to develop a conventional machine learning model is to extract features and to select the most relevant features based on the type of the problem at hand. Features play an important role in image processing. Extracted features provide a convenient way in converting an input image to a set of quantitative values [12]. Different features have special supremacy over diagnosis criterion [13]. In this study, we employed different texture feature extraction methods, such as neighbourhood gray-tone difference matrix features, gray length size zone matrix, gray-level co-occurrence matrix (GLCM), and gray-level run-length matrix (GLRM), in order to extract appropriate and relevant information from an input set of images. These features are shown to be very effective in the past in differentiating between the cancerous and healthy cases [16] and predicting survival outcome not only in the pathology domain, but also in MRI images [4,17,18].

Local binary pattern (LBP) features: The original LBP [19] was proposed by the University of Oulu’s computer vision group in Finland. It was initially used to classify texture images and since gained popularity in the domains of computer vision and pattern recognition. The original LBP was defined in a 3 × 3 rectangular region, which significantly constrained texture information extraction and made it impossible to produce rotation invariant features. As a result, Ref. [20] transformed LBP’s rectangular neighbourhood structure into a circular neighbourhood structure and used uniform sampling of p neighbourhood points in a circular local structure with radius R. However, the size of LBP characteristics grows exponentially with the number of neighbourhoods, severely limiting computation performance and storage space.

Neighbourhood grey-tone difference matrix (NGTDM) features: texture features obtained from an NGTDM neighbourhood gray-tone difference matrix to classify cancerous nodules in a patient could be used as predictors; the value of those texture features expressed the intensity variations between a voxel and its neighbouring voxels, according to the definition of those NGTDM textural features [21].

Grey level size-zone matrix (GLSZM) features: A standardized texture is made up of vast regions with the same intensity, rather than smaller groupings of pixels or segments in any given direction. To account for this feature, the size of each region containing pixels of the same intensity level has to be taken into account in a matrix. This matrix was created using run length matrix principle: value of the matrix’s $\left(s,n\right)$ square equals the number of regions of size s and grey level $n$ [22]. Grey level run length matrix (GLRLM) features: The grey level run length matrix is a statistical texture characterisation technique [23]. This approach involves to count pixel segments number with same strength in a particular direction and then displaying the results in a matrix [22]. The grey level run length matrix (GLRLM) is a matrix that may be used to extract texture characteristics for texture analysis [24]. The GLRLM approach is a technique applied to obtained the higher order statistical texture characteristics. A grey level run is defined as a line of pixels with the same intensity value that runs in a certain direction [24]. The grey level run length is defined by number of such pixels, and number of occurrences is referred to as run length value. In this context, a run length is defined as number of neighbouring pixels with same grey intensity in a certain direction. Grey level co-occurrence matrix (GLCM) features: GLCMs are often referred to as spatial grey level dependence matrices. It is a popular second-order statistical method for extracting texture information from images. The GLCM functions are utilized to determine an image’s texture features by computing pixel occurrence with certain values and in a specific spatial relationship. GLCM may be created in four different directions $\left(0°, 45°, 90° \mathrm{and} 135\right)$ [24]. This method is old and most effective statistical texture representation methods [22].

2.3. Ensemble Regression

Regression [25] methods, analogous to machine learning (ML), are not only used for forecasting and prediction, but are also employed to determine the causal relationships between the variables in particular instances [26]. Regression analysis determines value of dependent y variable based on range of independent variable values ‘$x$’ [27]. Linear regression [28] is a frequently used mathematical research approach in which the projected effects may be measured and modelled against various input variables. Ensemble regression combines a group of weak learner models and data used to train these learners. By aggregating predictions from its weak learners, it can anticipate ensemble responses for new data. In this study, we trained several individual weak learners, the survival outcomes of which were averaged to calculate the final survival outcome of each test patient. We used the following weak learners:

Support vector regression (SVR): The SVR [29] models are employed to solve nonlinear regression problems of estimation. SVR demonstrated great performance in nonlinear regression analysis, the research on using SVR to solve reliability prediction issues is still restricted.

AdaBoost: AdaBoost algorithm was proposed by Freund and Schapire [30] was first practical boosting algorithm which is now most widely used in many applications. Schapire [31] defines dignified indulgent of nature of learning at its core. It is a decision tree algorithm by generating a forest of stumps (i.e., simple trees with one node and two leaves). Stumps having ineffective regressors are poor learners. Each created stump has its own vote; whereas, the votes are distributed based on the error of various vote weights $\left(w\right)$ [32].

Gradient boost: A machine learning method which is used for both classification and regression problems. A basic parameterized function is used by building the additive regression model to current “pseudo”-residuals using least squares. The pseudo-residuals are gradient of loss function that is being minimised in relation to model values at each training data point assessed in current phase [33].

Random forest (RF): In 2001, Breiman introduced a random forest technique to categorise machine learning [34]. The random forest approach had a substantial influence on artificial intelligence, computer development and machine learning [35].

3. Results

The statistical analysis was performed using R version 3.2.4 developed by R Foundation for Statistical Computing, Vienna, Austria. In this section, the obtained results and discussions of ensemble regression methods are presented for the prediction of survival of glioma patients. The results gained from extracted features LBP, GLCM, GLSZM, GLRLM, and NGTDM from radiology (rad) images, pathology (path) images, and combination (ad-path) images are processed through the ensemble regression machine learning method, comprising support vector regression (SVR), AdaBoost, gradient boost, random forest (RF).

To confirm the robustness of the method, while avoiding overfitting on the same patient sets, we evaluated the ensemble regression method in all 171 patients using leave-one-out (LOO) cross-validation (configuration-I), repeated 5-fold cross-validation (configuration-II), and repeated the split-train-test (3:2 ratio) (configuration-III). In all the configurations, rad, path, and rad-path features were separately used to evaluate the performance on all the cancer grades (i.e., HGGs + LGGs, HGGs, and LGGs). The outputs of the models were compared against the actual survival to calculate the performance in terms of Pearson’s correlation coefficient (PCC). The value of PCC ranges from −1 to +1, where values lower and higher than zero, respectively, show negative and positive correlation, with −1 and +1 being the highest values of correlation. Performance is also summarized in terms of mean absolute error.

Survival prediction results of the proposed method are given in Figure 2, where the x-axis shows the actual survival of the patients and the y-axis shows the predicted survival in terms of the number of days. In split-train-test configuration, the proposed method yielded robust performance, as shown by the calculated Pearson’s correlation coefficients for rad-path. $\left(R=0.83, p=8.4\times {10}^{-11}\right),$ followed by pathology (path) images with $\left(R=0.72, p=2.6\times {10}^{-7}\right)$ and radiology (Rad) images $\left(R=0.72, p=3.7\times {10}^{-7}\right)$. The survival prediction of HGGs patients (second row) yielded with test-train split validation for rad-path images, producing PCC of $\left(R=0.79, p=1.8\times {10}^{-7}\right)$ followed by pathology (path) images with $\left(R=0.70, p=1.6\times {10}^{-5}\right)$ and radiology (rad) images $\left(R=0.71, p=9.6\times {10}^{-6}\right)$. The survival prediction combined sample LGGs and HGGs (first row) patients se yielded with test-train split validation for rad-path images producing PCC of $\left(R=0.81, p=2.2\times {10}^{-16}\right)$ followed by pathology (path) images with $\left(R=0.71, p=1.1\times {10}^{-11}\right)$ and radiology (rad) images with $\left(R=0.70, p=3.2\times {10}^{-11}\right)$.

The second column in Figure 2 presents the results of 5-fold cross-validation, the survival prediction of sample LGGs patients set (third row) yielded with 5-fold cross-validation for rad-path images producing PCC of $\left(R=0.80, p=2.2\times {10}^{-16}\right),$ followed by pathology (path) images with $\left(R=0.74, p=2.2\times {10}^{-16}\right)$ and radiology (rad) images $\left(R=0.74, p=2.2\times {10}^{-16}\right)$. The survival prediction of the sample HGGs patient’s dataset (second row) yielded with 5-fold cross-validation for rad-path images producing PCC of $\left(R=0.81, p=2.2\times {10}^{-16}\right)$ followed by pathology (path) images with $\left(R=0.69, p=5.6\times {10}^{-12}\right)$ and radiology (rad) images $\left(R=0.71, p=6\times {10}^{-13}\right)$. The survival prediction of the sample LGGs and HGGs patients combined dataset first row) yielded with 5-fold cross-validation for rad-path images producing Pearson’s correlation coefficient of $\left(R=0.80, p=2.2\times {10}^{-16}\right),$ followed by pathology (path.) images with $\left(R=0.74, p=2.2\times {10}^{-16}\right)$ and radiology (rad.) images $\left(R=0.73, p=2.2\times {10}^{-16}\right)$.

The first column in Figure 2 presents LOO configuration, the survival prediction of sample LGGs patient’s dataset (third row) yielded with leave-one-out cross-validation for rad-path images producing PCC of $\left(R=0.83, p=2.2\times {10}^{-16}\right),$ followed by pathology (path) images with $\left(R=0.76, p=2.2\times {10}^{-16}\right)$ and radiology (rad) images $\left(R=0.72, p=2.2\times {10}^{-16}\right)$. The survival prediction sample HGGs patient’s dataset (second row) yielded with leave-one-out cross-validation for rad-path images producing PCC of $\left(R=0.78, p=4.6\times {10}^{-16}\right),$ followed by pathology (path) images with $\left(R=0.73, p=8.3\times {10}^{-14}\right)$ and radiology (rad.) images $\left(R=0.70, p=1.8\times {10}^{-12}\right)$. The survival prediction of sample the LGGs and HGGs patients combined dataset (first row) yielded with leave-one-out cross-validation for rad-path images producing PCC of $\left(R=0.84, p=2.2\times {10}^{-16}\right),$ followed by pathology (path) images with $\left(R=0.73, p=2.2\times {10}^{-16}\right)$ and radiology (rad) images $\left(R=0.72, p=2.2\times {10}^{-16}\right)$.

Survival Analysis

Based on the predictions of the proposed models by utilizing Kaplan–Meier survival curves, the predictions were computed for both short and long term survival. The performance was also computed in entire and validation data. The model accurately predict the long and short survivors, higher and lower actual survivors respectively. The trained models were applied on the test dataset set, and the scores predicted by the regression model were noted. A cutoff that maximized the survival difference was developed on the scores of training data to identify the relatively long and short survivors. This cutoff was then applied on the scores of the test dataset to determine long and short survivor groups and overall survival was then compared between the predicted long and short survivor groups using the log-rank test. Hazard ratios (HR) with 95% confidence intervals (CI) and Kaplan–Meier curves on the predicted groups were calculated with level of statistical significance for a two-sided comparison set at $P<0.05$. The x-axis shows actual in days and y-axis shows predictions generated by proposed model.

The model trained on rad-path features divided the patients of the replication set (configuration-III) into long and short survivor groups (HGGs + LGGs: $HR=3.314 \left[95\%CI:1.718-6.394\right], log-rank\left(P\right)=2\times {10}^{-4}$; HGGs: $HR=2.172 \left[95\%CI:1.019-4.627\right], log-rank\left(P\right)=0.04$; LGGs: $HR=5.028 \left[95\%CI:0.9728-25.99\right], log-rank\left(P\right)=0.04$]). Patients were also stratified into long and short survivor groups in configuration-I (HGGs + LGGs: $HR=4.910 \left[95\%CI:3.176-7.593\right], log-rank\left(P\right)=2\times {10}^{-14}$; HGGs: $HR=5.197 \left[95\%CI:2.922-9.240\right], log-rank\left(P\right)=2\times {10}^{-9}$; LGGs: $HR=2.735 \left[95\%CI:1.186-6.306\right], log-rank\left(P\right)=0.01$ and configuration-II (HGGs + LGGs: $HR=7.118 \left[95\%CI:4.591-11.03\right], log-rank\left(P\right)<2\times {10}^{-16}$; HGGs: $HR=3.895 \left[95\%CI:2.317-6.547\right], log-rank\left(P\right)=6\times {10}^{-8}$; LGGs: $HR=8.155 \left[95\%CI:3.393-19.6\right], log-rank\left(P\right)=1\times {10}^{-7}$).

Ensemble methods supply significant amounts of information, when the combination of both datasets and both joint rad-path groups are gathered collectively. Applying LOO cross-validation yielded a best correlation coefficient score of $R=0.84$ on rad-path features, which is highest performance achieved in this work and is also consistent with very low $p=2.2\times {10}^{-16}$ in this experiment. While discussing the situation in which another LOOCV performed robustness for LGGs dataset includes only path images holding the highest value $R=0.76$ and other minimum value that is $p=2.2\times {10}^{-16}$, the rad images utilized form LGGs dataset obtains the highest value $R=0.74$ and minimum $p=2.2\times {10}^{-16},$ while performing 5-fold cross-validation as reflected in Figure 3. The alone HGGs dataset did not perform well throughout the work. The individual features are not sustainable enough for better results. The average performance is shown to increase using a LGGs and HGGs combined dataset for all of three groups: (i) radiology rad images (ii) pathology Path images, and (iii) rad-path images.

The model evaluations in terms of MAE to predict survival is reflected in

Table 2. Quantitative evaluation of model in terms of mean absolute error (MAE) in predicting survival using rad, path, and rad-path feature sets in leave-one-out cross-validation, 5-fold cross-validation, and split-train-test at 60–40%. The results are separately shown for combined dataset (HGGs + LGGs) and separate cancer grades (HGGs and LGGs).

	Configuration-I (Leave-One-Out)	Configuration-II (5-fold)	Configuration-III (Split-Train-Test)
High-grade gliomas + Low-grade gliomas
Rad	548.04	444.13	359.45
Path	407.15	479.83	338.60
Rad-path	399.32	379.66	259.19
High-grade gliomas
Rad	321.33	269.47	254.08
Path	225.62	226.99	237.50
Rad-path	212.72	190.13	193.65
Low-grade gliomas
Rad	757.97	741.48	590.54
Path	621.49	670.48	543.00
Rad-path	502.18	609.69	498.64

.

4. Discussion

The powerful and robust ensemble machine learning methods, such as support vector regression (SVR), SVR with linear kernel, SVR with RBF kernel, AdaBoost adaptive algorithm, gradient boost boosting method, and random forest approaches, are employed in this research study. These are used to obtain maximum accuracy and significance for finding the better prediction of survival outcomes of glioma patients. Different high-resolution images show higher complexity and nonlinear dynamics. These require multi-dimensional feature extraction techniques for prediction from an image due to large variations in shape and size required in multi-dimension feature extraction techniques to effectively and significantly performance. Therefore, the feature extracting strategies, such as LBP, GLCM, GLSZM, GLRLM, and NGTDM, are employed to overcome the hurdles faced by the prediction problems. To achieve robustness in survival prediction, the novel ensemble regression machine learning methodologies, such as SVR, SVR (with various kernels), AdaBoost, gradient boost, and random forest approaches are developed in Python version 3.9.5. The ensemble regression methods were applied in different configurations, including split-train-test validation, 5-fold cross-validation, and leave-one-out cross-validation to evaluate the generalizability of the model. Pearson’s correlation coefficient was used as a performance measure to evaluate the performance of the model.

The datasets involved in this work are a low-grade glioma LGGS dataset, high-grade glioma dataset, and are grouped by combining both these LGGs and HGGs datasets. The radiology (rad) images, digital pathology (path) images, and the combination of image sets, namely rad-path images, are utilized for better performance and to obtain salient features for better prognoses of glioma patients. In past research works, the working is upon either the radiographic images or upon digital pathology images for the survival prediction. However, in this study, the obtained results reportedly show that the current feature extraction strategies (LBP, GLCM, GLSZM, GLRLM, and NGTDM) deliver the most promising results. The novelty in this study is that the combining of both types of images of radiology and pathology domains outbursts the results by employing ensemble machine learning regression approaches. The results are more effective and can play a very important and more significant role by predicting more accurately and can be helpful in clinical diagnosis and treatments.

In previous work, authors suggested that T1-Gd intensity is a measure of breakdown of blood–brain barrier and serves as an important predictor of overall survival in gliomas. In our regression survival model, patients predicted with higher score, and lower volumes of enhancing tumor, higher intensities in FLAIR and T2 images, reflecting reduced cell density and increased concentration of fluid. The entropy and homogeneity are other vital features, which quantify randomness in an image. The invasion and diffused nature of the tumor were measured in terms of volume and shape features. Consistent with previous studies, we observed irregular and larger edema in the high-risk group. Moreover, low-risk tumors were found mostly in the left temporal and frontal lobes.

4.1. Performance Comparison with Existing Methods

This section provides a performance overview of the proposed method with existing literature. It is worth mentioning that most of the existing methods used either radiology, pathology, or genetic data for the prediction of survival in gliomas, however, in the proposed method, we innovatively combined the features extracted from pathology and radiology images for the prediction of survival in gliomas. Due to the lack of literature or existing methods in the field of radiopathomics, we provided a comparison and performance overview of our method in the light of other existing radiology, pathology, or genomics-based methods. It is also important to note that different papers differ significantly in terms of the type and the size of the dataset and reported performance of the models in terms of different performance metrics; however, the table below provides a rough estimate of the performance of our method compared to existing literature.

The results are compared with existing studies as reflected in

Table 3. Comparison of the proposed method with existing survival prediction techniques.

Author (Year)	Patients	Dataset Type	Features	Machine Learning Model	Performance
Radiology data
Macyszyn et al. (2015) [6]	129	Radiology	Intensity, shape and location features from the tumor area.	Support vector classification	Accuracy = 80%
Baid et al. (2020) [7]	NA	Radiology	First-order, intensity-based length, shape-based, and textural radiomic characteristics.	Neural Network	AUC = 0.57
Sun et al. (2019) [8]	285 + 191	Radiology	Radiomic features from segmented tumor regions.	Random forest model	Accuracy = 61%
Zhao et al. (2020) [9]	NA		Local feature extractor and global extractor utilizing the final layer of segmentation model.	Deep learning model	Accuracy = 65.5%
Asma et al. (2022) [10]	178	Radiology	Shape and location features	Random forest classifier	AUC = 0.73
Huang et al. (2021) [12]	369+ 125+ 236	Radiology	Intensity, texture, wavelet, shape and other radiological features from the tumor area, and CNN network based deep features.	Random forest regression model	RMSE= 311.5
Suter et al. (2018) [36]	NA	Radiology	Intensity, shape, location and deep features.	Support vector regression	57.1%
Gene expression data
Wijethilake et al. (2020) [37]	315 + 252	Genetic expressions	13,094 gene	Probabilistic programming	Accuracy = 74%
Pathology data
Mobadersany et al. (2018) [15]	769	Pathology	No features were extracted. Images were given directly as input to the model.	Survival convolutional neural networks	c-index = 0.77
Iftikhar et al. (2019) [13]	663	Pathology	No features were extracted. Images were given directly as input to the model.	Deep neural network based	c-index = 0.82
Proposed model
---	171	MRI+Pathology	Radiomic features extracted from MRI and pathology images.	Support vector regression	Correlation coefficient = 0.84

.

4.2. Strengths and Limitations

The main strength of the proposed rad-path model is that it provides a way for radiologists and pathologists to create coherent, correlated, and integrated diagnostic summaries. The rich set of radiology/pathology characteristics, as well as the machine learning signatures produced from them, will improve our understanding of gliomas, and can help with precision diagnoses.

There are several important limitations to this study. The most important is that the pathology images were outlined in a semiautomatic fashion, which is a user-dependent and laborious task. Delineation of the region of interest in a full-automated fashion would facilitate automation of the analysis, reduce user bias, and empower larger-scale analysis. Another important limitation is that a retrospective dataset was used in this study, whereas the comparison of our methods to clinically measure OS by using a prospective dataset would further validate our models.

5. Conclusions and Future Directions

In summary, we developed a radiopathomic signature to predict overall survival and stratification of patients with glioma. This signature proves enhanced performance in comparison to radiology and pathology-based features. Several possible future directions this study could take include evaluation of the methods in larger cohorts, and evaluation of sophisticated deep learning methods for prediction of overall survival, though such methods need considerably larger datasets. Even though the texture features used here provided strong survival predictions, future experiments would benefit from incorporating more advanced radiomic and shape features extracted in the vicinity of the tumor to emphasize the biologic nature of the gliomas. Along these lines, using machine learning methods to perform both a supervised and unsupervised selection of features may help the models to account for a possibly complex and nonlinear relation between radiographic and histopathologic features. This work may be further expanded upon by utilising numerous strategies, by using advance deep learning methodologies, by utilizing the combination of both imaging characteristics, and by using most discriminative histology and radiology features through employing histology and pathology correlation analysis. Various other features of both imaging sequences, which are not the part of this study, can be used with other machine learning techniques for optimum predictions. Neural networks can also use morphological, textural, and density of pixels-based features for the same prediction purpose. We also plan to use the machine learning models specifically developed for survival data in the future.