Recognition and Classification of Martian Chaos Terrains Using Imagery Machine Learning: A Global Distribution of Chaos Linked to Groundwater Circulation, Catastrophic Flooding, and Magmatism on Mars
Round 1
Reviewer 1 Report
Summary:
The paper implements CNN classifiers for different image modalities to classify images of terrains as AHO, AP, non-chaos classes. In addition to training and testing images with known labels, images from terrains with potential chaos-like features are also used as target set. The terrains are identified using Google Mars app.
VGG architecture is used for classification. Two different image scales and two different batch sizes are considered. Both the original data as well as a balanced version of it is considered. Results suggest these three classes can be classified (especially using CTX and Themis image types) with high precision and recall. However, there is one major concern about the splitting of the train and test datasets that may have led to overestimation of the test accuracy. There are also several minor concerns.
Major Concern:
Train/test split is done randomly. That means images from the same terrain are shared in both train and test sets. Testing accuracy may overestimate accuracy in this case because the network is trained and tested on spatially contiguous images of same terrains. For a more accurate evaluation of the classifier performance the split should be done such that all images from the same terrain are assigned to either train or test sets but not both.
Minor Concerns:
The authors generate balanced data with only 92 images from each of the three classes and show that the network trained with the balanced set has worse predictive performance than the one trained with the original imbalanced data. It is not clear what loss function was used when training these networks. A weighted cross-entropy loss function would have more effectively tackled class imbalance. 4 Division performing better than 16 Division might have to do with the test/train split randomly done without paying attention to terrains as described above.
What is the significance of using machine learning for global survey of chaos like features if Google Mars already provides this information? Google Mars is used to identify chaos-like features. These images were then classified with the trained networks. This could potentially bias the target data. Evaluation is only done on terrains in which Google Mars shows chaos-like features. How accurate is Google Mars in this task? If it is very accurate then it diminishes the potential impact of proposed machine learning algorithms as people can use Google Mars software instead of machine learning. If it is not very accurate then that would suggest there are other terrains with chaos-like features, and it would be interesting to see how machine learning performs in identifying such features.
Why are only batch sizes of 32 and 64 considered? If the goal is to investigate the effect of batch size on classifier performance then it would have been more useful if significantly different batch sizes were considered such as 8/16 vs 64/128.
Figure 6 is being used to justify that overfitting is not happening. This is a premature conclusion due to train/test split not being done properly.
Related work in automatic mapping and classification in planetary geology can be extended with following recent work.
Rajaneesh, A., Vishnu, C. L., Oommen, T., Rajesh, V. J., & Sajinkumar, K. S. (2022). Machine learning as a tool to classify extra-terrestrial landslides: A dossier from Valles Marineris, Mars. Icarus, 376, 114886.
Plebani, E., Ehlmann, B. L., Leask, E. K., Fox, V. K., & Dundar, M. M. (2022). A machine learning toolkit for CRISM image analysis. Icarus, 376, 114849.
Rubanenko, L., Pérez-López, S., Schull, J., & Lapôtre, M. G. (2021). Automatic Detection and Segmentation of Barchan Dunes on Mars and Earth Using a Convolutional Neural Network. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14, 9364-9371.
Typos:
Is it "class unbalance" or "class imbalance"?
The sentence in lines 428-429 is not clear. ... CNN focuses on modeling. CNN focuses on "learning" would be a better word choice.
Line 480: Test accuracies are computed three times.
What causes the variation in test accuracies? If the network is fixed it will always give the same test accuracy. Is training repeated three times and test accuracy is computed after each repetition? This needs clarification.
Line 485: What is full chance level performance? I assume this is the accuracy achieved by randomly assigning the test sample to one of three classes. I would call this "Random classification accuracy".
Line 512: The second MOLA should be THEMIS
Author Response
We are grateful to the reviewer #1 for the constructive comments and helpful suggestions.
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Knowledge of the global distribution and activity of groundwater/ground ice is critical to understand the evolution of hydrology, habitability, and in situ water resource use for future crewed missions to Mars. In this study, the authors perform recognition and classification of Martian chaos using imagery machine learning. Via the developed neural network models, the block landforms commonly found in chaos terrains and non-chaos surface features are classified with 93.9% ± 0.3% accuracy. By applying the developed models to ~3150 images of block landforms of chaos-like features, they identified two types of chaos terrain and presented the global distributions of different types of chaos terrains presumably formed by water activity and volcano-tectonic activity on Mars. Overall, I’d say the paper is well written and informative.
Minor comments：
1. Lines 130-137: “three distinct image sources of the Martian surface” are used in this work, but their spatial resolutions are different. How to process the data in the classification? Correspondingly, in Lines 204-219, the expressions such as “224 × 224 RGB (3 colors) image” are difficult to understand. I also recommend to give a brief introduction to the used datasets and the classification results.
2. The 4 and 16 Division data are significant in the paper, please give a detailed description about the definition and application of them.
Author Response
We are grateful to the reviewer #2 for the constructive comments and helpful suggestions.
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
The paper ist very well written and shows an important application of neural networks for classification of geological features.
I would suggest investigating the dependence of the classification quality w.r.t. to latitude and longitude, since this would also shed some light on sub-surface geological activities. I do not suggest this for the paper here but maybe for a follow on publication.
Minor: The figures on page 16 are a little bit small. Better split and distribute on two pages.
Author Response
We are grateful to the reviewer #3 for the constructive comments and helpful suggestions.
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
Authors have done additional experiments showing that region-based splitting of the train/test data does not change results much compared to random splitting and have chosen to proceed with random splitting, which is understandable.
Authors also repeated experiments with weighted cross entropy loss function and have shown that the differences are negligible and have chosen to proceed with standard cross entropy loss function which is acceptable given that results for the two loss functions are similar.
These revisions address my previous concerns. However, there are still grammar errors in the text that need to be corrected. For example, in the newly added section "we randomly chosen ..." should be "we have randomly chosen".
Author Response
We are grateful to the reviewer #1 for helpful suggestions.
Please see the attachment.
Author Response File: Author Response.pdf
