Survival Prediction of Glioma Patients from Integrated Radiology and Pathology Images Using Machine Learning Ensemble Regression Methods

Round 1

Reviewer 1 Report

This paper integrated radiology and pathology imaging to improve survival prediction outcomes of gliomas. The authors extracted imaging features from tumor regions in Rad and Path images, and applied an ensemble regression machine learning pipeline to study these features. The performance was evaluated in several configurations, and they found that the combined Rad and Path features outperformed individual feature types in all the configurations and datasets. This manuscript is interesting. Some points that need to revise are below:

 

1.     The authors can do a literature review in the Introduction Section about the background, dataset, and approach used in this topic. Only a small number of citations cannot enable readers to understand this manuscript well. In addition, the authors have not adequately surveyed this field, and there are only a few papers published after 2020.

2.     How can this proposed model help doctors and patients, especially doctors? The author needs to give more discussion about this.

3.     This model seems to be valid only for the dataset used in this manuscript, so the authors need to explain if the model is overfitting and if it works on other datasets.

4.     It is not clear to me why the authors applied the classification and regression algorithm, as well as the feature extraction method listed in this manuscript. Would reinforcement learning or deep learning algorithms produce better results on this dataset? The authors can add more illustrations.

5.     Has this issue been studied before? This model can be compared with others (especially these papers published recently, after 2020) based on numerical simulation results, which can further highlight the advantages of this model.

 

In short, this work is interesting, but it needs a MAJOR revision before reconsideration.

Author Response

Review Report Form 

Open Review

(x) I would not like to sign my review report  
( ) I would like to sign my review report  

English language and style

( ) Extensive editing of English language and style required  
( ) Moderate English changes required  
(x) English language and style are fine/minor spell check required  
( ) I don't feel qualified to judge about the English language and style  

	Yes	Can be improved	Must be improved	Not applicable
Does the introduction provide sufficient background and include all relevant references?	( )	( )	(x)	( )
Are all the cited references relevant to the research?	( )	(x)	( )	( )
Is the research design appropriate?	( )	( )	(x)	( )
Are the methods adequately described?	(x)	( )	( )	( )
Are the results clearly presented?	(x)	( )	( )	( )
Are the conclusions supported by the results?	( )	(x)	( )	( )

Comments and Suggestions for Authors

This paper integrated radiology and pathology imaging to improve survival prediction outcomes of gliomas. The authors extracted imaging features from tumor regions in Rad and Path images, and applied an ensemble regression machine learning pipeline to study these features. The performance was evaluated in several configurations, and they found that the combined Rad and Path features outperformed individual feature types in all the configurations and datasets. This manuscript is interesting. Some points that need to revise are below:

 

We thank the reviewer for detailed, competent, and constructive feedback. We revised the manuscript and detail the responses to the individual points in this response letter. All changes in the revised version of the manuscript have been highlighted in yellow.

 

1. The authors can do a literature review in the Introduction Section about the background, dataset, and approach used in this topic. Only a small number of citations cannot enable readers to understand this manuscript well. In addition, the authors have not adequately surveyed this field, and there are only a few papers published after 2020.

 

Ans: We thank the reviewer for raising this concern. Several past studies have shown the effectiveness of both MRI features extracted from different subregions of the tumors gliomas^1–4 and histopathology pathology^5–7 features extracted from the region of interest demarcated in the prediction of survival (or recurrence) in glioma patients. Lately, several studies have also shown that texture measures when combined with clinical and genomic markers^8,9 offer better prediction of survival, underscoring the complementary nature of the datasets. Moreover, histopathology features when evaluated in combination with the genomic features offer better survival prediction¹⁰. We have now summarized these prior studies in the introduction section of the revised version of the manuscript.

2. How can this proposed model help doctors and patients, especially doctors? The author needs to give more discussion about this.

 

Ans: This model may facilitate patient selection for targeted therapies, stratification of patients into clinical trials, and power to observe treatment effects after extensive evaluation in prospective setting. Furthermore, as the field of medicine is growing and the drift towards team-based management of diseases is continuously evolving, improved exchange of information between different departments such as radiology and pathology is needed particularly. The proposed radiopathomics model has the potential to provide a mechanism to generate coherent, correlated, and integrated diagnostic summaries with minor additional effort from radiologists and pathologists. We consider that a comprehensive set of radiology and pathology features, and the regression-based signatures derived from them will help us understand the mechanistic complexity of gliomas and may contribute to precision diagnostics. We have not provided these details in the Discussion section of the revised version of the manuscript.

3. This model seems to be valid only for the dataset used in this manuscript, so the authors need to explain if the model is overfitting and if it works on other datasets.

 

Ans: The dataset that we have used in this analysis has been collected from various institutions, and the overall reasonable performance of our model in various configurations including 5-fold cross validation, leave-one-out, and split-train-test suggests that the model performs well on unseen populations as well.

4. It is not clear to me why the authors applied the classification and regression algorithm, as well as the feature extraction method listed in this manuscript. Would reinforcement learning or deep learning algorithms produce better results on this dataset? The authors can add more illustrations.

 

Ans: Several deep learning (DL)-based approaches have gained popularity in the field of radiomics. Even though these methods have shown promise in the detection of survival, they tend to suffer with the problem of huge computational complexity. Also, these methods have a very large number of parameters and are therefore notorious for overfitting the data and suffering from poor reproducibility. The problem gets more severe when we have smaller sample size like the ones that we have in this study. On the other hand, simple machine learning algorithm captures complete heterogeneity of the tumor and provides almost similar performance on totally unseen test datasets with much lesser complexity and lesser chances of overfitting, therefore, we did settle with a standard pipeline of radiomic feature extraction, and regression.

 

5. Has this issue been studied before? This model can be compared with others (especially these papers published recently, after 2020) based on numerical simulation results, which can further highlight the advantages of this model.

 

Ans: We thank the reviewer for the valuable suggestion. There have been multiple explorative recent studies (6-10) which have attempted to develop prognostic models from the tumor habitat on MRI and from the tissue region on Path images to predict OS in gliomas, however, the integrated potential of Rad and Path images in gliomas is still unexplored. Therefore, we aim to develop a radiopathomics model that can predict survival using a multi-institutional cohort and across various glioma grades.

 

In short, this work is interesting, but it needs a MAJOR revision before reconsideration.

 

Submission Date

16 July 2022

Date of this review

31 Jul 2022 19:05:11

 

 

 

 

1. Kickingereder, P. et al. Radiomic Profiling of Glioblastoma: Identifying an Imaging Predictor of Patient Survival with Improved Performance over Established Clinical and Radiologic Risk Models. https://doi.org/10.1148/radiol.2016160845 280, 880–889 (2016).
2. Bae, S. et al. Radiomic MRI phenotyping of glioblastoma: Improving survival prediction. Radiology 289, 797–806 (2018).
3. Rathore, S. et al. Radiomic MRI signature reveals three distinct subtypes of glioblastoma with different clinical and molecular characteristics, offering prognostic value beyond IDH1. Scientific Reports 2018 8:1 8, 1–12 (2018).
4. Rathore, S., Chaddad, A., Iftikhar, M. A., Bilello, M. & Abdulkadir, A. Combining MRI and Histologic Imaging Features for Predicting Overall Survival in Patients with Glioma. Radiology: Imaging Cancer 3, e200108 (2021).
5. Mobadersany, P. et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A 115, E2970–E2979 (2018).
6. Iftikhar, M., Rathore, S. & Nasrallah, M. Analysis of microscopic images via deep neural networks can predict outcome and IDH and 1p/19q codeletion status in gliomas. in Journal of Neuropathology and experimental neurology vol. 78 553 (OXFORD UNIV PRESS INC JOURNALS DEPT, 2001 EVANS RD, CARY, NC 27513 USA, 2019).
7. Rathore, S., Iftikhar, M., MacLean Nasrallah, M. G., Rajpoot, N. & Mourelatos, Z. TMOD-35. PREDICTION OF OVERALL SURVIVAL, AND MOLECULAR MARKERS IN GLIOMAS VIA ANALYSIS OF DIGITAL PATHOLOGY IMAGES USING DEEP LEARNING. Neuro-Oncology21, vi270 (2019).
8. Bae, S. et al. Radiomic MRI phenotyping of glioblastoma: Improving survival prediction. Radiology 289, 797–806 (2018).
9. Chaddad, A., Daniel, P., Desrosiers, C., Toews, M. & Abdulkarim, B. Novel Radiomic Features Based on Joint Intensity Matrices for Predicting Glioblastoma Patient Survival Time. IEEE Journal of Biomedical and Health Informatics 23, 795–804 (2019).
10. Chaddad, A., Daniel, P., Sabri, S., Desrosiers, C. & Abdulkarim, B. Integration of Radiomic and Multi-omic Analyses Predicts Survival of Newly Diagnosed IDH1 Wild-Type Glioblastoma. (2019) doi:10.3390/cancers11081148.

 

 

 

 

 

Reviewer 2 Report

The authors used Machine Learning Ensemble Regression Methods to predict Survival  of Glioma Patients from Integrated Radiology and Pathology Images. Although there are some merits in the subject and the work, it suffers from critical issues that makes it hard to accept it in the present form. Some issues are as follows: 

1- the survival outcome has not been defined clearly. The authors must define the outcome of interest as time from special event like disease diagnosis to a special outcome like death or remission.

2- Also the censorship must be defined.

3- the most important issue is that the authors have considered the time to event as a usual response variable and they fitted machine learning and regression methods for the usual continuous outcomes, while in the case of survival data this is a wrong method. The ML models have been well-developed for survival data so that the authors can use the survival version of these models to analyze their data. 

4- Moreover, the performance criteria like R^2 that the authors have used are not correct. They should use other criteria like Brier score or C-index.

5- The software or package that the authors used must be provided in the method section.

 

Author Response

Review Report Form 

Open Review

( ) I would not like to sign my review report  
(x) I would like to sign my review report  

English language and style

( ) Extensive editing of English language and style required  
( ) Moderate English changes required  
( ) English language and style are fine/minor spell check required  
(x) I don't feel qualified to judge about the English language and style  

	Yes	Can be improved	Must be improved	Not applicable
Does the introduction provide sufficient background and include all relevant references?	(x)	( )	( )	( )
Are all the cited references relevant to the research?	( )	(x)	( )	( )
Is the research design appropriate?	( )	( )	(x)	( )
Are the methods adequately described?	( )	( )	(x)	( )
Are the results clearly presented?	( )	( )	(x)	( )
Are the conclusions supported by the results?	( )	( )	( )	( )

 

Comments and Suggestions for Authors

The authors used Machine Learning Ensemble Regression Methods to predict Survival  of Glioma Patients from Integrated Radiology and Pathology Images. Although there are some merits in the subject and the work, it suffers from critical issues that makes it hard to accept it in the present form. Some issues are as follows: 

 

We thank the reviewer for detailed, competent, and constructive feedback. We revised the manuscript and detail the responses to the individual points in this response letter. All changes in the revised version of the manuscript have been highlighted in yellow.

 

1- the survival outcome has not been defined clearly. The authors must define the outcome of interest as time from special event like disease diagnosis to a special outcome like death or remission.

 

Ans. Survival was defined as the time from the initial presentation of the disease to death or last follow-up time, whatever happened earlier. We have now mentioned this in the manuscript.

2- Also the censorship must be defined.

 

Ans. In our study, censored means, that the last follow-up interval preceded the end of the study as well as the death. As such, the survival time can be stated conclusively and is just known to be longer than the latest follow-up. We have explained this in the revised version of the manuscript.

3- the most important issue is that the authors have considered the time to event as a usual response variable and they fitted machine learning and regression methods for the usual continuous outcomes, while in the case of survival data this is a wrong method. The ML models have been well-developed for survival data so that the authors can use the survival version of these models to analyze their data. 

 

Ans. We agree with the reviewer that such methods exist, however, to keep the model simple, we have used normal regression models. The effectiveness of the method can be accessed by the fact that it performs very well even with ordinary prediction models. We plan to use the ML models specifically developed for survival data in future. We have also mentioned this as a future work in the Discussion section.

4- Moreover, the performance criteria like R^2 that the authors have used are not correct. They should use other criteria like Brier score or C-index.

 

Ans. We thank the reviewer for the valuable suggestion. We have now calculated the mean absolute error and have provided in Table – in the revised version of the manuscript.

5- The software or package that the authors used must be provided in the method section.

 

Ans. All statistical analyses, including machine learning, were performed by using R, version 3.2.4 (R Foundation for Statistical Computing, Vienna, Austria). We have now mentioned this in the revised version of the manuscript.

 

Submission Date

16 July 2022

Date of this review

11 Aug 2022 15:50:56

 

Reviewer 3 Report

There are two major downsides to this experiment, specifically because it has been performed on a sensitive and critical topic which is health care. 

1 - The use of Pearson’s correlation coefficient as the metric for goodness-of-fit for regression models is weak and misleading in general. The correlation between predicted vs actual tells us almost nothing about the strength and statistical power of the model in predicted values. The model can have systemic bias but still maintain a high correlation and statistically significant p value. There needs to be a metric for accuracy such as R squared, MSE, etc. between predicted vs actual values to indicate the bias and variance of errors quantitatively. Once these metrics are in, it needs to be compared with the latest state-of-the-art model in this area to see if the results of this new approach is statistically significant or not. 

 

2- In general, high model interpretability in critical and sensitive topics are preferred over more complex and black box approaches. It would be ideal to include discussions about the feature importance and effect or at least provide references how it can be done in ensemble of machine learning models. 

Author Response

Review Report Form 

Open Review

( ) I would not like to sign my review report  
(x) I would like to sign my review report  

English language and style

( ) Extensive editing of English language and style required  
(x) Moderate English changes required  
( ) English language and style are fine/minor spell check required  
( ) I don't feel qualified to judge about the English language and style  

	Yes	Can be improved	Must be improved	Not Applicable
Does the introduction provide sufficient background and include all relevant references?	(x)	( )	( )	( )
Are all the cited references relevant to the research?	(x)	( )	( )	( )
Is the research design appropriate?	( )	( )	(x)	( )
Are the methods adequately described?	( )	( )	(x)	( )
Are the results clearly presented?	( )	( )	(x)	( )
Are the conclusions supported by the results?	( )	( )	(x)	( )

 

Comments and Suggestions for Authors

There are two major downsides to this experiment, specifically because it has been performed on a sensitive and critical topic which is health care. 

 

 We thank the reviewers and editor for their detailed, competent, and constructive feedback. We revised the manuscript and detail the responses to the individual points in this response letter. All changes in the revised version of the manuscript have been highlighted in yellow.

 

1 - The use of Pearson’s correlation coefficient as the metric for goodness-of-fit for regression models is weak and misleading in general. The correlation between predicted vs actual tells us almost nothing about the strength and statistical power of the model in predicted values. The model can have systemic bias but still maintain a high correlation and statistically significant p value. There needs to be a metric for accuracy such as R squared, MSE, etc. between predicted vs actual values to indicate the bias and variance of errors quantitatively. Once these metrics are in, it needs to be compared with the latest state-of-the-art model in this area to see if the results of this new approach is statistically significant or not. 

 

Ans. We thank the reviewer for the valuable suggestion. We have now calculated the mean absolute error and have provided in Table – in the revised version of the manuscript.

 

2- In general, high model interpretability in critical and sensitive topics are preferred over more complex and black box approaches. It would be ideal to include discussions about the feature importance and effect or at least provide references how it can be done in ensemble of machine learning models. 

 

Ans. In our regression survival model, patients who were predicted to have higher score had lower volumes of enhancing tumor, higher intensities in FLAIR and T2 images, reflecting reduced cell-density and increased concentration of fluid. Other important features of our model were texture measures of entropy and homogeneity, which quantify randomness in an image. The invasion and diffused nature of the tumor were measured in terms of volume and shape features. Consistent with previous studies, we observed irregular and larger edema in high-risk group. Moreover, low-risk tumors were found mostly in the left temporal and frontal lobes. On the other hand, high-risk tumors were found both in the left and right temporal lobes. We have now provided these details in the Discussion section of the revised version of the manuscript.

 

 

Submission Date

16 July 2022

Date of this review

03 Aug 2022 03:20:16

 

Reviewer 4 Report

In this manuscript, the authors aimed to improve survival prediction outcomes of gliomas by integrating radiology and pathology imaging. Although the manuscript is interesting, the following comments should be considered. 

1. The abstract is too long it should be shortened and should contain answers to the following questions: What problem was studied and why is it important? What methods were used? What are the important results? What conclusions can be drawn from the results? What is the novelty of the work and where does it go beyond previous efforts in the literature?

2. The Introduction should make a compelling case for why the study is useful along with a clear statement of its novelty or originality by providing relevant information and providing answers to basic questions such as: What is already known in the open literature? What is missing (i.e., research gaps)? What needs to be done, why, and how? Clear statements of the novelty of the work should also appear briefly in the Abstract and Conclusions sections.

3. The literature review could be greatly improved. In the current version, all related works are just introduced in sequential order. The authors first need to make comparisons of them and then draw the motivation of the paper. Neither the comparison of references and this work nor the corresponding conclusion is made in the paper. Thus, it is difficult for me to know the novelty and advantages of this paper over other works.

4. The authors should point out the major contributions of this paper by using 3 to 5 brief bullet points at the end of the Introduction section, right before the last paragraph.

5. The structure of arguments needs to be improved. At the end of the introduction part, you should have a section plan (for example section 2 discusses... and section 3 gives...).

6. The most important problem is the technical one. There is no clear explanation of what the novelty of the proposed method is, given that, as the authors point out.

7. The performance of the proposed method should be more analyzed, commented and studied.

8. The experimental results need to be improved in the following aspects, (i) Insights should be presented regarding why the proposed algorithm performed much better than the existing methods. (ii) The experiments can be designed in a more elaborate way to cover all important aspects of the proposed algorithm.

9. Superiority of the proposed method should be addressed perfectly in comparison with similar established ones in the literature.

9. The conclusion should write taking into account one or more of the aspects: strengths and weaknesses of research, assessment, and implications of the work results or findings, projection of possible applications, recommendations, or suggestions.

10. The authors cited only three papers from 2020-2022. Try to discuss and cite more recent papers from 2020-2022 including the following papers: Overall Survival Prediction of Glioma Patients With Multiregional Radiomics (2022), https://doi.org/10.3389/fnins.2022.911065, Object pose estimation using mid-level visual representations (2022) arXiv preprint arXiv:2203.01449, Survival prediction of glioblastoma patients—are we there yet? A systematic review of prognostic modeling for glioblastoma and its clinical potential. Neurosurg Rev 44, 2047–2057 (2021). https://doi.org/10.1007/s10143-020-01430-z, Dynamic architecture based deep learning approach for glioblastoma brain tumor survival prediction, Neuroscience Informatics, 2(4), (2022) 100062, https://doi.org/10.1016/j.neuri.2022.100062, and Survival prediction in gliomas: Current state and novel approaches, Exon Publications (2021): 151-169, https://doi.org/10.36255/exonpublications.gliomas.2021.chapter9.

There are some typos and grammatical errors in some parts of this text, especially in the introduction section. Please double-check all sentences and correct all sentences that need to be corrected grammatically.

Author Response

1. The abstract is too long it should be shortened and should contain answers to the following questions: What problem was studied and why is it important? What methods were used? What are the important results? What conclusions can be drawn from the results? What is the novelty of the work and where does it go beyond previous efforts in the literature?
2. The Introduction should make a compelling case for why the study is useful along with a clear statement of its novelty or originality by providing relevant information and providing answers to basic questions such as: What is already known in the open literature? What is missing (i.e., research gaps)? What needs to be done, why, and how? Clear statements of the novelty of the work should also appear briefly in the Abstract and Conclusions sections.
3. The literature review could be greatly improved. In the current version, all related works are just introduced in sequential order. The authors first need to make comparisons of them and then draw the motivation of the paper. Neither the comparison of references and this work nor the corresponding conclusion is made in the paper. Thus, it is difficult for me to know the novelty and advantages of this paper over other works.
4. The authors should point out the major contributions of this paper by using 3 to 5 brief bullet points at the end of the Introduction section, right before the last paragraph.
5. The structure of arguments needs to be improved. At the end of the introduction part, you should have a section plan (for example section 2 discusses... and section 3 gives...).
6. The most important problem is the technical one. There is no clear explanation of what the novelty of the proposed method is, given that, as the authors point out.
7. The performance of the proposed method should be more analyzed, commented and studied.
8. The experimental results need to be improved in the following aspects, (i) Insights should be presented regarding why the proposed algorithm performed much better than the existing methods. (ii) The experiments can be designed in a more elaborate way to cover all important aspects of the proposed algorithm.
9. Superiority of the proposed method should be addressed perfectly in comparison with similar established ones in the literature.
10. The conclusion should write taking into account one or more of the aspects: strengths and weaknesses of research, assessment, and implications of the work results or findings, projection of possible applications, recommendations, or suggestions.
11. The authors cited only three papers from 2020-2022. Try to discuss and cite more recent papers from 2020-2022 including the following papers: Overall Survival Prediction of Glioma Patients With Multiregional Radiomics (2022), https://doi.org/10.3389/fnins.2022.911065, Object pose estimation using mid-level visual representations (2022) arXiv preprint arXiv:2203.01449, Survival prediction of glioblastoma patients—are we there yet? A systematic review of prognostic modeling for glioblastoma and its clinical potential. Neurosurg Rev 44, 2047–2057 (2021). https://doi.org/10.1007/s10143-020-01430-z, Dynamic architecture based deep learning approach for glioblastoma brain tumor survival prediction, Neuroscience Informatics, 2(4), (2022) 100062, https://doi.org/10.1016/j.neuri.2022.100062, and Survival prediction in gliomas: Current state and novel approaches, Exon Publications (2021): 151-169, https://doi.org/10.36255/exonpublications.gliomas.2021.chapter9.

There are some typos and grammatical errors in some parts of this text, especially in the introduction section. Please double-check all sentences and correct all sentences that need to be corrected grammatically.

Round 2

Reviewer 1 Report

I think this manuscript still needs a MAJOR revision to be accepted. As stated in my last review report, a sufficient survey of the existing literature is needed to allow readers to know this field, and experimental comparisons with existing methods are needed to demonstrate the effectiveness of this method. None of these are shown in the revised manuscript well. The performance of a model is mainly illustrated by the quantitative experimental results, which are not fully demonstrated in this manuscript, especially the lack of comparison with existing methods.

Author Response

Review Report Form

Open Review

(x) I would not like to sign my review report
( ) I would like to sign my review report

English language and style

( ) Extensive editing of English language and style required
( ) Moderate English changes required
(x) English language and style are fine/minor spell check required
( ) I don't feel qualified to judge about the English language and style

	Yes	Can be improved	Must be improved	Not applicable
Does the introduction provide sufficient background and include all relevant references?	( )	( )	(x)	( )
Are all the cited references relevant to the research?	( )	(x)	( )	( )
Is the research design appropriate?	( )	( )	(x)	( )
Are the methods adequately described?	( )	(x)	( )	( )
Are the results clearly presented?	( )	(x)	( )	( )
Are the conclusions supported by the results?	( )	(x)	( )	( )

Comments and Suggestions for Authors

 

I think this manuscript still needs a MAJOR revision to be accepted. As stated in my last review report, a sufficient survey of the existing literature is needed to allow readers to know this field, and experimental comparisons with existing methods are needed to demonstrate the effectiveness of this method. None of these are shown in the revised manuscript well. The performance of a model is mainly illustrated by the quantitative experimental results, which are not fully demonstrated in this manuscript, especially the lack of comparison with existing methods.

Ans. We thank the reviewers and editor for their detailed, competent, and constructive feedback. We revised the manuscript and detail the responses to the individual points in this response letter. All changes in the revised version of the manuscript have been highlighted in yellow. We believe these changes have resulted in a substantially stronger paper. In response to this particular concern of the reviewer, we have now revised the introduction section to provide a more sufficient survey of the existing literature. Paragraph 4 and 5 in the introduction section provides a detailed account of existing glioma survival prediction techniques using MRI and pathology datasets, respectively. We have cited following relevant research in these paragraphs. 

1. Luke Macyszyn, Hamed Akbari, Jared M Pisapia, et al., Imaging patterns predict patient survival and molecular subtype in glioblastoma via machine learning techniques, Neuro-oncology, 18(3), 417-425, 2015.
2. Asma Shaheen, Syed Talha Bukhari, Maria Nadeem, Stefano Burigat, Ulas Bagci and Hassan Mohy-ud-Din, Overall Survival Prediction of Glioma Patients With Multiregional Radiomics, Frontiers in neuroscience, 2022, https://doi.org/10.3389/fnins.2022.911065
3. Ishaan Ashwini Tewarie, Joeky T Senders, Stijn Kremer, Sharmila Devi, William B Gormley, Omar Arnaout, Timothy R Smith, Marike L D Broekman, Survival prediction of glioblastoma patients—are we there yet? A systematic review of prognostic modeling for glioblastoma and its clinical potential. Neurosurg Rev 44 (4), 2047–2057, 2021, https://doi.org/10.1007/s10143-020-01430-z
4. Disha Sushant Wankhede, R.Selvarani Dynamic architecture based deep learning approach for glioblastoma brain tumor survival prediction, Neuroscience Informatics, 2(4), (2022) 100062, https://doi.org/10.1016/j.neuri.2022.100062
5. Rachel Zhao, Andra Valentina Krauze , Waldemar Debinski, Survival prediction in gliomas: Current state and novel approaches, Exon Publications, 151-169, 2021, https://doi.org/10.36255/exonpublications.gliomas.2021.chapter9.
6. Baid, U. et al. Overall Survival Prediction in Glioblastoma With Radiomic Features Using Machine Learning. Frontiers in Computational Neuroscience 14, 1–9 (2020).
7. Iftikhar Muhammad, Rathore Saima & Nasrallah MacLean. Analysis of microscopic images via deep neural networks can predict outcome and IDH and 1p/19q codeletion status in gliomas. Journal of Neuropathology and experimental neurology 78, 553–553 (2019).
8. Rathore, S. et al. PREDICTION OF OVERALL SURVIVAL, AND MOLECULAR MARKERS IN GLIOMAS VIA ANALYSIS OF DIGITAL PATHOLOGY IMAGES USING DEEP LEARNING. Neuro-Oncology 21, vi270–vi270 (2019).
9. Rathore, S., Iftikhar, M. A., Gurcan, M. N. & Mourelatos, Z. Radiopathomics: Integration of radiographic and histologic characteristics for prognostication in glioblastoma. arXiv (2019).
10. Mobadersany P, Yousefi S, Amgad M, Gutman DA, Barnholtz-Sloan JS, Velázquez Vega JE, Brat DJ, Cooper LAD. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A. 2018 Mar 27;115(13):E2970-E2979. doi: 10.1073/pnas.1717139115. Epub 2018 Mar 12. PMID: 29531073; PMCID: PMC5879673.
11. Sun Li, Zhang Songtao, Chen Hang, Luo Lin, Brain Tumor Segmentation and Survival Prediction Using Multimodal MRI Scans With Deep Learning, Frontiers in Neuroscience, 13, 2019, DOI=10.3389/fnins.2019.00810
12. Zhao, G., Jiang, B., Zhang, J., and Xia, Y. (2020). “Segmentation then Prediction: A Multi-task Solution to Brain Tumor Segmentation and Survival Prediction,” in International MICCAI Brainlesion Workshop, (Berlin: Springer), 492–502. doi: 10.1007/978-3-030-72084-1_44
13. Huang H, Zhang W, Fang Y, Hong J, Su S, Lai X. Overall Survival Prediction for Gliomas Using a Novel Compound Approach. Front Oncol. 2021 Aug 18;11:724191. doi: 10.3389/fonc.2021.724191. PMID: 34490121; PMCID: PMC8416476.
14. Suter, Y. et al. Deep learning versus classical regression for brain tumor patient survival prediction. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) vol. 11384 LNCS 429–440 (Springer Verlag, 2019).

We thank the reviewer for the valuable suggestion about comparing the performance of the model with existing approaches. We want to mention here that most of existing methods have used either radiology, pathology, or genetics data for the prediction of survival in gliomas, however, in the proposed method, we have innovatively combined the features extracted from pathology and radiology images for the prediction of survival in gliomas. Due to the lack of literature or existing methods in the field of radiopathomics, we have provided comparison and performance overview of our method in the light of other existing radiology, pathology, or genomics-based methods. We also want to mention that different papers differ significantly in terms of the type and the size of the dataset and have reported performance of the models in terms of different performance metrics, however, the table below provides a rough estimate of the performance of our method compared to existing literature. Table below provides a comparative performance review of our method with existing literature. We have also added a section named ‘Performance comparison with existing methods’ in the revised version of the manuscript that provides these details.

 

Author (year)	Patients	Dataset type	Features	Machine learning model	Performance
Radiology data
Macyszyn et al. (2015)	129	Radiology	Intensity, shape and location features from the tumor area	Support vector classification	Accuracy=80%
Baid et al. (2020)	NA	Radiology	First-order, intensity-based length, shape-based, and textural radiomic characteristics.	NA	AUC=0.57
Sun et al. (2019)	285+191	Radiology	Radiomic features from segmented tumor regions	Random forest model	Accuracy = 61%
Zhao et al. (2020)	NA		Local feature extractor and global extractor utilizing the final layer of segmentation model	Deep learning model	Accuracy=65.5%
Asma et al. (2022)	178	Radiology	Shape and location features	Random forest classifier	AUC = 0.73
Huang et al. (2021)	369+ 125+ 236	Radiology	Intensity, texture, wavelet, shape and other radiological features from the tumor area, and CNN network based deep features	Random forest regression model	RMSE= 311.5
Suter et al. (2018)	NA	Radiology	Intensity, shape, location and deep features	Support vector regression	57.1%
Gene expression data
Wijethilake et al. (2020)	315+252	Genetic expressions	13094 gene	Probabilistic programming	Accuracy = 74%
Pathology data
Mobadersany et al. (2018)	769	Pathology	No features were extracted. Images were given directly as input to the model.	Survival convolutional neural networks	c-index=0.77
Iftikhar et al. (2019)	663	Pathology	No features were extracted. Images were given directly as input to the model.	Deep neural network based	c-index=0.82
Proposed model
---	171	MRI+ Pathology	Radiomic features extracted from MRI and pathology images	Support vector regression	Correlation =0.84

 

 

 

 

 

 

Author Response File: Author Response.docx

Reviewer 2 Report

Unfortunately the authors did not handle the comments so as I have previously mentioned it is not possible to use linear regression instead of survival models. So all the analysis are wrong. 

Author Response

Review Report Form 

Open Review

( ) I would not like to sign my review report  
(x) I would like to sign my review report  

English language and style

( ) Extensive editing of English language and style required  
( ) Moderate English changes required  
( ) English language and style are fine/minor spell check required  
(x) I don't feel qualified to judge about the English language and style  

	Yes	Can be improved	Must be improved	Not applicable
Does the introduction provide sufficient background and include all relevant references?	(x)	( )	( )	( )
Are all the cited references relevant to the research?	( )	(x)	( )	( )
Is the research design appropriate?	( )	( )	(x)	( )
Are the methods adequately described?	( )	( )	(x)	( )
Are the results clearly presented?	( )	( )	(x)	( )
Are the conclusions supported by the results?	( )	( )	( )	( )

 

Comments and Suggestions for Authors

The authors used Machine Learning Ensemble Regression Methods to predict Survival  of Glioma Patients from Integrated Radiology and Pathology Images. Although there are some merits in the subject and the work, it suffers from critical issues that makes it hard to accept it in the present form. Some issues are as follows: 

 

We thank the reviewer for detailed, competent, and constructive feedback. We revised the manuscript and detail the responses to the individual points in this response letter. All changes in the revised version of the manuscript have been highlighted in yellow.

 

1- the survival outcome has not been defined clearly. The authors must define the outcome of interest as time from special event like disease diagnosis to a special outcome like death or remission.

 

Ans. Survival was defined as the time from the initial presentation of the disease to death or last follow-up time, whatever happened earlier. We have now mentioned this in the manuscript.

2- Also the censorship must be defined.

 

Ans. In our study, censored means, that the last follow-up interval preceded the end of the study as well as the death. As such, the survival time can be stated conclusively and is just known to be longer than the latest follow-up. We have explained this in the revised version of the manuscript.

3- the most important issue is that the authors have considered the time to event as a usual response variable and they fitted machine learning and regression methods for the usual continuous outcomes, while in the case of survival data this is a wrong method. The ML models have been well-developed for survival data so that the authors can use the survival version of these models to analyze their data. 

 

Ans. We agree with the reviewer that such methods exist, however, to keep the model simple, we have used normal regression models. The effectiveness of the method can be accessed by the fact that it performs very well even with ordinary prediction models. We plan to use the ML models specifically developed for survival data in future. We have also mentioned this as a future work in the Discussion section.

4- Moreover, the performance criteria like R^2 that the authors have used are not correct. They should use other criteria like Brier score or C-index.

 

Ans. We thank the reviewer for the valuable suggestion. We have now calculated the mean absolute error and have provided in Table – in the revised version of the manuscript.

5- The software or package that the authors used must be provided in the method section.

 

Ans. All statistical analyses, including machine learning, were performed by using R, version 3.2.4 (R Foundation for Statistical Computing, Vienna, Austria). We have now mentioned this in the revised version of the manuscript.

 

Submission Date

16 July 2022

Date of this review

11 Aug 2022 15:50:56

 

Reviewer 3 Report

On the merit that the study's methodology is detailed and the results are clearly defined which allows other researchers to conduct and compare their results with this approach, it is a helpful paper. 

Author Response

Review Report Form 

Open Review

( ) I would not like to sign my review report  
(x) I would like to sign my review report  

English language and style

( ) Extensive editing of English language and style required  
(x) Moderate English changes required  
( ) English language and style are fine/minor spell check required  
( ) I don't feel qualified to judge about the English language and style  

	Yes	Can be improved	Must be improved	Not Applicable
Does the introduction provide sufficient background and include all relevant references?	(x)	( )	( )	( )
Are all the cited references relevant to the research?	(x)	( )	( )	( )
Is the research design appropriate?	( )	( )	(x)	( )
Are the methods adequately described?	( )	( )	(x)	( )
Are the results clearly presented?	( )	( )	(x)	( )
Are the conclusions supported by the results?	( )	( )	(x)	( )

 

Comments and Suggestions for Authors

There are two major downsides to this experiment, specifically because it has been performed on a sensitive and critical topic which is health care. 

 

 We thank the reviewers and editor for their detailed, competent, and constructive feedback. We revised the manuscript and detail the responses to the individual points in this response letter. All changes in the revised version of the manuscript have been highlighted in yellow.

 

1 - The use of Pearson’s correlation coefficient as the metric for goodness-of-fit for regression models is weak and misleading in general. The correlation between predicted vs actual tells us almost nothing about the strength and statistical power of the model in predicted values. The model can have systemic bias but still maintain a high correlation and statistically significant p value. There needs to be a metric for accuracy such as R squared, MSE, etc. between predicted vs actual values to indicate the bias and variance of errors quantitatively. Once these metrics are in, it needs to be compared with the latest state-of-the-art model in this area to see if the results of this new approach is statistically significant or not. 

 

Ans. We thank the reviewer for the valuable suggestion. We have now calculated the mean absolute error and have provided in Table – in the revised version of the manuscript.

 

2- In general, high model interpretability in critical and sensitive topics are preferred over more complex and black box approaches. It would be ideal to include discussions about the feature importance and effect or at least provide references how it can be done in ensemble of machine learning models. 

 

Ans. In our regression survival model, patients who were predicted to have higher score had lower volumes of enhancing tumor, higher intensities in FLAIR and T2 images, reflecting reduced cell-density and increased concentration of fluid. Other important features of our model were texture measures of entropy and homogeneity, which quantify randomness in an image. The invasion and diffused nature of the tumor were measured in terms of volume and shape features. Consistent with previous studies, we observed irregular and larger edema in high-risk group. Moreover, low-risk tumors were found mostly in the left temporal and frontal lobes. On the other hand, high-risk tumors were found both in the left and right temporal lobes. We have now provided these details in the Discussion section of the revised version of the manuscript.

 

 

Submission Date

16 July 2022

Date of this review

03 Aug 2022 03:20:16

 

Reviewer 4 Report

Most of my comments have not been addressed, therefore the following should be addressed.

1. The Introduction should make a compelling case for why the study is useful along with a clear statement of its novelty or originality by providing relevant information and providing answers to basic questions such as: What is already known in the open literature? What is missing (i.e., research gaps)? What needs to be done, why, and how? Clear statements of the novelty of the work should also appear briefly in the Abstract and Conclusions sections.

2. The literature review could be greatly improved. In the current version, all related works are just introduced in sequential order. The authors first need to compare them and then draw the paper's motivation. Neither the comparison of references and this work nor the corresponding conclusion is made in the paper. Thus, it is difficult for me to know the novelty and advantages of this paper over other works.

3. The authors should point out the major contributions of this paper by using 3 to 5 brief bullet points at the end of the Introduction section, right before the last paragraph.

4. The structure of arguments needs to be improved. At the end of the introduction part, you should have a section plan (for example section 2 discusses... and section 3 gives...).

5. The conclusion should write taking into account one or more of the aspects: strengths and weaknesses of research, assessment, and implications of the work results or findings, projection of possible applications, recommendations, or suggestions.

6. The authors cited only four papers from 2020-2022. Try to discuss and cite more recent papers from 2020-2022 including the following papers: Overall Survival Prediction of Glioma Patients With Multiregional Radiomics (2022), https://doi.org/10.3389/fnins.2022.911065, Object pose estimation using mid-level visual representations (2022) arXiv preprint arXiv:2203.01449, Survival prediction of glioblastoma patients—are we there yet? A systematic review of prognostic modeling for glioblastoma and its clinical potential. Neurosurg Rev 44, 2047–2057 (2021). https://doi.org/10.1007/s10143-020-01430-z, Dynamic architecture based deep learning approach for glioblastoma brain tumor survival prediction, Neuroscience Informatics, 2(4), (2022) 100062, https://doi.org/10.1016/j.neuri.2022.100062, and Survival prediction in gliomas: Current state and novel approaches, Exon Publications (2021): 151-169, https://doi.org/10.36255/exonpublications.gliomas.2021.chapter9.

 

 

Author Response

Review Report Form

Open Review

(x) I would not like to sign my review report
( ) I would like to sign my review report

English language and style

( ) Extensive editing of English language and style required
( ) Moderate English changes required
(x) English language and style are fine/minor spell check required
( ) I don't feel qualified to judge about the English language and style

	Yes	Can be improved	Must be improved	Not applicable
Does the introduction provide sufficient background and include all relevant references?	( )	( )	(x)	( )
Are all the cited references relevant to the research?	( )	( )	(x)	( )
Is the research design appropriate?	( )	(x)	( )	( )
Are the methods adequately described?	( )	(x)	( )	( )
Are the results clearly presented?	( )	(x)	( )	( )
Are the conclusions supported by the results?	( )	( )	(x)	( )

Comments and Suggestions for Authors

Most of my comments have not been addressed, therefore the following should be addressed.

1. The Introduction should make a compelling case for why the study is useful along with a clear statement of its novelty or originality by providing relevant information and providing answers to basic questions such as: What is already known in the open literature? What is missing (i.e., research gaps)? What needs to be done, why, and how? Clear statements of the novelty of the work should also appear briefly in the Abstract and Conclusions sections.

Ans. We thank the reviewers and editor for their detailed, competent, and constructive feedback. We revised the manuscript and detail the responses to the individual points in this response letter. All changes in the revised version of the manuscript have been highlighted in yellow. We believe these changes have resulted in a substantially stronger paper. In response to this particular concern of the reviewer, we want to mention that the main limitation of the existing studies that we have mentioned in our paper is that they have used only one imaging modalities (either radiology or pathology) to predict survival, whereas both the imaging modalities provide complementary information. In this study, we overcome this limitation by integrating the imaging features extracted from both the modalities and hence achieve better survival prediction compared to the studies using only one imaging modality. We have provided this motivation in the introduction section.  

 

2. The literature review could be greatly improved. In the current version, all related works are just introduced in sequential order. The authors first need to compare them and then draw the paper's motivation. Neither the comparison of references and this work nor the corresponding conclusion is made in the paper. Thus, it is difficult for me to know the novelty and advantages of this paper over other works.

Ans. We have now revised the introduction section to provide a more sufficient survey of the existing literature. Paragraph 4 and 5 in the introduction section provide a detailed account of existing glioma survival prediction techniques using MRI and pathology datasets, respectively. We have cited relevant research (list given below) in these paragraphs.

 

1. Luke Macyszyn, Hamed Akbari, Jared M Pisapia, et al., Imaging patterns predict patient survival and molecular subtype in glioblastoma via machine learning techniques, Neuro-oncology, 18(3), 417-425, 2015.
2. Asma Shaheen, Syed Talha Bukhari, Maria Nadeem, Stefano Burigat, Ulas Bagci and Hassan Mohy-ud-Din, Overall Survival Prediction of Glioma Patients With Multiregional Radiomics, Frontiers in neuroscience, 2022, https://doi.org/10.3389/fnins.2022.911065
3. Ishaan Ashwini Tewarie, Joeky T Senders, Stijn Kremer, Sharmila Devi, William B Gormley, Omar Arnaout, Timothy R Smith, Marike L D Broekman, Survival prediction of glioblastoma patients—are we there yet? A systematic review of prognostic modeling for glioblastoma and its clinical potential. Neurosurg Rev 44 (4), 2047–2057, 2021, https://doi.org/10.1007/s10143-020-01430-z
4. Disha Sushant Wankhede, R.Selvarani Dynamic architecture based deep learning approach for glioblastoma brain tumor survival prediction, Neuroscience Informatics, 2(4), (2022) 100062, https://doi.org/10.1016/j.neuri.2022.100062
5. Rachel Zhao, Andra Valentina Krauze , Waldemar Debinski, Survival prediction in gliomas: Current state and novel approaches, Exon Publications, 151-169, 2021, https://doi.org/10.36255/exonpublications.gliomas.2021.chapter9.
6. Baid, U. et al. Overall Survival Prediction in Glioblastoma With Radiomic Features Using Machine Learning. Frontiers in Computational Neuroscience 14, 1–9 (2020).
7. Iftikhar Muhammad, Rathore Saima & Nasrallah MacLean. Analysis of microscopic images via deep neural networks can predict outcome and IDH and 1p/19q codeletion status in gliomas. Journal of Neuropathology and experimental neurology 78, 553–553 (2019).
8. Rathore, S. et al. PREDICTION OF OVERALL SURVIVAL, AND MOLECULAR MARKERS IN GLIOMAS VIA ANALYSIS OF DIGITAL PATHOLOGY IMAGES USING DEEP LEARNING. Neuro-Oncology 21, vi270–vi270 (2019).
9. Rathore, S., Iftikhar, M. A., Gurcan, M. N. & Mourelatos, Z. Radiopathomics: Integration of radiographic and histologic characteristics for prognostication in glioblastoma. arXiv (2019).
10. Mobadersany P, Yousefi S, Amgad M, Gutman DA, Barnholtz-Sloan JS, Velázquez Vega JE, Brat DJ, Cooper LAD. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A. 2018 Mar 27;115(13):E2970-E2979. doi: 10.1073/pnas.1717139115. Epub 2018 Mar 12. PMID: 29531073; PMCID: PMC5879673.
11. Sun Li, Zhang Songtao, Chen Hang, Luo Lin, Brain Tumor Segmentation and Survival Prediction Using Multimodal MRI Scans With Deep Learning, Frontiers in Neuroscience, 13, 2019, DOI=10.3389/fnins.2019.00810
12. Zhao, G., Jiang, B., Zhang, J., and Xia, Y. (2020). “Segmentation then Prediction: A Multi-task Solution to Brain Tumor Segmentation and Survival Prediction,” in International MICCAI Brainlesion Workshop, (Berlin: Springer), 492–502. doi: 10.1007/978-3-030-72084-1_44
13. Huang H, Zhang W, Fang Y, Hong J, Su S, Lai X. Overall Survival Prediction for Gliomas Using a Novel Compound Approach. Front Oncol. 2021 Aug 18;11:724191. doi: 10.3389/fonc.2021.724191. PMID: 34490121; PMCID: PMC8416476.
14. Suter, Y. et al. Deep learning versus classical regression for brain tumor patient survival prediction. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) vol. 11384 LNCS 429–440 (Springer Verlag, 2019).

 

3. The authors should point out the major contributions of this paper by using 3 to 5 brief bullet points at the end of the Introduction section, right before the last paragraph.

Ans. The main contributions of our study are: (i) to exploit the rich phenotypic information present both in radiology and histology images through multivariate pattern analysis methods for prediction survival in gliomas; (ii) to perform robust statistical analysis on individual radiology and pathology features and their combinations via multiple permuted iterations; and (iii) to validate the model on a diverse multi-institutional cohort of images. We have now provided these numbered points in the introduction section, right before the last paragraph. 

 

4. The structure of arguments needs to be improved. At the end of the introduction part, you should have a section plan (for example section 2 discusses... and section 3 gives...).

Ans. In the light of the reviewer’s comment, we have now provided a section plan at the end of Introduction Section.

 

5. The conclusion should write taking into account one or more of the aspects: strengths and weaknesses of research, assessment, and implications of the work results or findings, projection of possible applications, recommendations, or suggestions.

Ans. We thank the reviewer for the valuable suggestion. We have now added two sections at the end of Discussion section to discuss these aspects. These sections named ‘Strengths and Limitations’ and ‘Conclusion and Future Directions’, respectively provide an account of strengths/weaknesses of this research work, and possible future directions, and recommendations.

 

6. The authors cited only four papers from 2020-2022. Try to discuss and cite more recent papers from 2020-2022 including the following papers: Overall Survival Prediction of Glioma Patients With Multiregional Radiomics (2022), https://doi.org/10.3389/fnins.2022.911065, Object pose estimation using mid-level visual representations (2022) arXiv preprint arXiv:2203.01449, Survival prediction of glioblastoma patients—are we there yet? A systematic review of prognostic modeling for glioblastoma and its clinical potential. Neurosurg Rev 44, 2047–2057 (2021). https://doi.org/10.1007/s10143-020-01430-z, Dynamic architecture based deep learning approach for glioblastoma brain tumor survival prediction, Neuroscience Informatics, 2(4), (2022) 100062, https://doi.org/10.1016/j.neuri.2022.100062, and Survival prediction in gliomas: Current state and novel approaches, Exon Publications (2021): 151-169, https://doi.org/10.36255/exonpublications.gliomas.2021.chapter9.

Ans. We thank the reviewers for the valuable suggestion. We have now cited the latest papers from 2020-2022, including the ones suggested by the reviewer.

 

 

 

Round 3

Reviewer 1 Report

The authors well addressed my comments, so I recommend accepting this manuscript in the current version.

Reviewer 2 Report

No further comments.

Reviewer 4 Report

Since most of my comments are addressed, I have no further comments.