Dual-Branch-AttentionNet: A Novel Deep-Learning-Based Spatial-Spectral Attention Methodology for Hyperspectral Data Analysis
Round 1
Reviewer 1 Report
In this paper, the author focuses on solving the core problem of hyperspectral classification research-------How to balance the contribution of spatial and spectral information. The author ' s motivation is relatively novel, and the author clearly describes the problem and related research status, but the author ' s experiment is not sufficient, so I suggest the author make the following modify to obtain a better manuscript.
(1). In the abstract and title section, the authors describe that this method is used for hyperspectral data analysis, but I think that the authors’ model maybe more suitable for the hyperspectral image classification.
(2). In the introduction part, the authors summarize the research status that solving the inconsistent contribution of spatial-spectral information as “Attention”, but the deformable convolution or center position coding can also balance the spatial-spectral information. In addition, the author did not give a complete summary of the current research status of attention based, such as the line based on self-attention and some existing advanced methods based on spatial-spectral attention.
(3). In the contribution part, the author lists five contributions, for the principal component analysis, the author said that “An effective PCA based dimensionality reduction and spectral feature extraction113 technique was employed to bring down the dimensionality of the input HSI data114 cube and conserve spatial-spectral details in lower dimensional subspace”, but according to the later introduced method, the author may not make the own improvements or contributions to the principal component.
(4). In the Approach Overview part, the author mainly describes the theory of the proposed model, but the concepts of residual network and forward network are relatively basic. If it takes so much space to introduce, it may be redundant.
(5). The authors spend too much space to introduce the comparison methods, which may be unnecessary. The authors can simply summarize these methods and their experimental settings as an experimental setting.
(6). In the experimental part, the author’s experiment is more like an ablation experiment rather than a comparison experiment. The author only compares three existing SOTA methods on two datasets. I suggest that the authors add an additional dataset to avoid the data bias, and most importantly, the authors should compare some newer SOTA methods to verify the advantages of the model.
Author Response
We thank the reviewer for their valuable comments and have edited the manuscript to address the reviewer’s concerns. Below is an item-by-item response to all the comments made during the review process.
Comment 1: In the abstract and title section, the authors describe that this method is used for hyperspectral data analysis, but I think that the authors’ model may be more suitable for the hyperspectral image classification.
Response : The terms hyperspectral data analysis and hyperspectral data classification have been interchangeably and synonymously used in the remote sensing community to a great extent. However, taking the reviewer’s comment into consideration, the term hyperspectral data analysis has been replaced by hyperspectral image classification in the manuscript wherever apt.
Changes made in the manuscript : The term hyperspectral data analysis has been replaced by hyperspectral data classification wherever appropriate throughout the manuscript.
Comment 2 : In the introduction part, the authors summarize the research status of solving the inconsistent contribution of spatial-spectral information as “Attention”, but the deformable convolution or center position coding can also balance the spatial-spectral information. In addition, the author did not give a complete summary of the current research status of attention based, such as the line based on self-attention and some existing advanced methods based on spatial-spectral attention.
Response : As the reviewer mentioned, inconsistent information contribution of spatial-spectral information could possibly be balanced by deformable convolution or center position coding. However, in our work, solving the issue of inconsistent information contribution isn’t being called “Attention”. It is clearly explained in our manuscript that devising a functional methodology to efficiently emphasize more informative pixels in a spatial neighborhood and spectral bands in the temporal domain to aid in boosting the efficacy of a HSI data classification framework is referred to as the “Attention” concept in this particular research. This is clarified in the lines 69 and 87 in our manuscript. Additionally, the authors appreciate the reviewer’s concern about the manuscript not including a complete summary on the current research of attention based hyperspectral classification techniques. Taking this into account, the manuscript has been revised and now includes a brief summary on the state-of-the-art attention methodologies related to hyperspectral image classification.
Changes made in the manuscript : A brief summary of history and current research status related to attention methodologies in hyperspectral remote sensing has been included between lines 91 and 106. Additional references [30-33] were also added for clarifications and appropriate discussions.
Comment 3 : In the contribution part, the author lists five contributions, for the principal component analysis, the author said that “An effective PCA based dimensionality reduction and spectral feature extraction technique was employed to bring down the dimensionality of the input HSI data cube and conserve spatial-spectral details in lower dimensional subspace”, but according to the later introduced method, the author may not make the own improvements or contributions to the principal component.
Response : The proposed work does not claim or intend to make our own improvements or contributions to the Principal Component Analysis (PCA) as an algorithm. Instead, PCA is used as a statistical technique or tool that allows identifying underlying linear patterns in a data set, so that the original dataset can be expressed in a significantly lower dimensional subspace in comparison with the dimensionality of the original dataset without much loss of information. This technique is beneficial for processing data sets with hundreds of features (like the ones in our case) while, at the same time, maintaining/retaining most of the information from the original data set. This reduction in the dimensionality of feature space of the input dataset helps the underlying deep learning framework converge quicker than usual (where no dimensionality reduction technique has been used) as it now has to only generalize/converge based on lesser number of data features which makes the deep learning framework used more time and memory efficient. Most importantly, the framework that has been proposed in our work (SPAT-SPEC-HYP-ATTN) is entirely built on top of a PCA block in which the input data is represented in a reduced dimensional feature space which not only aids in boosting the performance of our approach but also brings down the overall computational cost of our technique fairly.
Comment 4 : In the Approach Overview part, the author mainly describes the theory of the proposed model, but the concepts of residual network and forward network are relatively basic. If it takes so much space to introduce, it may be redundant.
Response: While we appreciate the reviewer’s feedback, we respectfully disagree. We believe that the concepts described in this study makes a valuable contribution to the field in general. It is our goal to make our scientific research pursued in this work to be expressed fairly, easily reachable and understandable to all researchers, not only from the remote sensing community, but also from other relevant broad areas of research interests. Hence, we strongly believe that the ‘Approach Overview’ part of our paper is relevant and needs to be published in its current form without any modifications.
Comment 5 : The authors spend too much space to introduce the comparison methods, which may be unnecessary. The authors can simply summarize these methods and their experimental settings as an experimental setting.
Response : While we appreciate the reviewer’s feedback, we respectfully disagree. We think this study makes a valuable contribution to the field and it is our goal to make the content written in our work fairly easily reachable and understandable and reproducible to researchers, not only from the remote sensing community, but also from other relevant broad areas of research. Hence, we strongly believe that the ‘Methodologies for Comparison’ section of our paper needs to be published in its current form without any modifications.
Comment 6 : In the experimental part, the author’s experiment is more like an ablation experiment rather than a comparison experiment. The author only compares three existing SOTA methods on two datasets. I suggest that the authors add an additional dataset to avoid the data bias, and most importantly, the authors should compare some newer SOTA methods to verify the advantages of the model.
Response : We recognize that the reviewer is concerned about any data bias that may have been introduced. Hence, to address this concern, new experimental results have been collected on a third dataset – ‘Indian Pines’. The corresponding model performance evaluation on the new Indian Pines data has been clearly documented in the updated manuscript.
In addition, the authors wish to respectfully decline the reviewer’s comment that our experimental and comparison is more of an ablation process and not a fair comparison with other state-of-the-art HSI classification approaches. The effectiveness of our technique has been exhaustively compared against a bi-directional LSTM-based spectral attention and hyperspectral data classification methodology SPEC-HYP-ATTN, a ResNet architecture-based spatial attention methodology SPAT-HYP-ATTN, a spatial-spectral feature construction and HSI classification framework GAB-RP-S-3DCNN, a comprehensive spatial-spectral attention technique for pixelwise HSI data classification SSAN and the well-known composite kernel SVM-based feature extraction and classification technique which produce exceptional results and are state-of-the-art methodologies with many of them being recently published. Hence, it is with great confidence that we say that our methodology is on par, in fact, even better when compared to many of these compared approaches in terms of classification results.
Changes made in the manuscript : Addition of new results related to the Indian Pines dataset in Figure 6, Table 3, Table 6, Figure 9 and Figure 10 have been added in the updated manuscript. Also, Table 7 and Table 8 have been updated to include the additional empirical results related to the newly added Indian Pines dataset in our work.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
This paper presents a method for extracting informative features from hyperspectral remote sensing data. The novelty lies in utilizing 3D convolution and long short-term memory to extract spatial and temporal (or spectral) features. After scrutinizing the manuscript, the reviewer has some comments as follows:
1. The introduction is somewhat lengthy, while it should focus on the paper's main contributions. For example, the first and fifth points of the summary list (lines 113-139) are deemed relatively basic in image processing. Thus, it is unnecessary to mention them as new contributions. Please revise the introduction.
2. Mathematical expressions should be revised thoroughly to ensure that they are consistent with definitions in the main text. Also, please double-check Equation (8) because dimensions are not agreed (n-by-2h (matrix multiplication) h-by-q).
3. It appears that an equation was supposed to be inserted between lines 321-322.
4. The run-time comparison in Table 5 is not sensible because algorithms have not been executed on the same computer.
5. The experimental results are astounding. However, it is essential to provide cross-validation results to ensure that the model does not suffer overfitting. In addition, the testing data in Tables 1 and 2 are deemed imbalanced, which may affect the accuracy. Hence, it is highly advisable to provide the cross-validation results and utilize balanced testing data to re-check the model's performance.
There are also some minor comments below:
1. Abbreviations should be defined once on their first appearance.
2. On line 161, please provide correct definitions of n and d (which one denotes rows and which one denotes columns).
Author Response
We thank the reviewer for their valuable comments on our manuscript and have edited the manuscript to address the reviewer’s concerns. Below is an item-by-item response to all the comments made during the review process.
Comment 1 : The introduction is somewhat lengthy, while it should focus on the paper's main contributions. For example, the first and fifth points of the summary list (lines 113-139) are deemed relatively basic in image processing. Thus, it is unnecessary to mention them as new contributions. Please revise the introduction.
Response : We appreciate the reviewer’s feedback, and have removed PCA as the first new contribution. However, the authors argue that the proposed methodology in this work as a whole (with the inclusion of PCA) is still novel. In this work, using PCA based dimensionality reduction and spectral feature extraction technique to reduce the dimensionality of the input HSI data and conserve spatial-spectral details in lower dimensional subspace has effectively aided in better data representation but also has brought down the computational complexity (gauged in terms of overall execution time) of our approach greatly. Hence, we feel that it is of great importance that these factors are discussed in the introduction. In addition, the fourth (previously fifth) contribution states that our work introduces a number of unique spatial-only and spectral-only attention-based approaches to compare and validate the efficacy of our methodology. We believe that it is essential that we add this to the contribution summary to clearly justify what our work offers to the readers.
Changes made in the manuscript: The statement which introduces the summary in the manuscript between lines 130 and 131 has been structurally revised and now justifies the content in the summary part of the introduction better.
Comment 2 : Mathematical expressions should be revised thoroughly to ensure that they are consistent with definitions in the main text. Also, please double-check Equation (8) because dimensions are not agreed (n-by-2h (matrix multiplication) h-by-q).
Response : We thank the reviewer for their careful observations and constructive remarks pertinent to the manuscript. It is true that a matrix multiplication between H_t and W_hq is involved in equation 8. However, the shape of H_t is determined to be (n-by-2h) and the shape of W_hq is determined to be (2h-by-q) and hence compatible for matrix multiplication. We have made necessary changes in the revised manuscript to clarify the equations.
Changes made in the manuscript : The manuscript is updated with detailed information about inputs of the matrix multiplication and their respective shapes involved in equation 8 between lines 247 and 249.
Comment 3 : It appears that an equation was supposed to be inserted between lines 321-322.
Response: The previously defined equations (6-11) in sections 2.3 and 2.4 have been referred to on line 331 in the updated manuscript. We have added clarification to this on line 331.
Comment 4: The run-time comparison in Table 5 is not sensible because algorithms have not been executed on the same computer.
Response : We understand that all our experiments related to SPAT-SPEC-HYP-ATTN, SPAT-HYP-ATTN, SPEC-HYP-ATTN, GAB-RP-S-3DCNN and SVM-CK were conducted on a workstation with an Intel(R) Core(TM) i7-7700HQ processor and 16 GB RAM, whereas SSAN was executed on a workstation with an Intel Core i7-5930K processor with 64 GB RAM and a GeForce GTX Titan X graphics card, which is a much powerful machine. However, it was not our goal to pit one methodology versus another discussed in our work based on the machines they were executed on. Instead, we are more inclined towards proving that our proposed approach produced exceptional classification performance with a relative trade-off between classification efficacy and computational complexity irrespective of the machine these techniques were executed on. Nonetheless, it is noteworthy that most of our proposed methods yielded superior classification and computational performance than SSAN.
Comment 5: The experimental results are astounding. However, it is essential to provide cross-validation results to ensure that the model does not suffer overfitting. In addition, the testing data in Tables 1 and 2 are deemed imbalanced, which may affect the accuracy. Hence, it is highly advisable to provide the cross-validation results and utilize balanced testing data to re-check the model's performance.
Response : We fear the reviewer may have misunderstood us here. We find it unnecessary to check the performance of our approach on balanced testing data, because that leads to a whole another research problem domain of class imbalance issue. Our goal is not to solve the class imbalance problem but to develop a robust hyperspectral data analysis framework that can provide an exceptional hyperspectral image classification performance despite the class imbalance issues. Nonetheless, to address the reviewer’s concern about possible over-fitting, we have included detailed analysis plots for training loss versus validation loss during the training phase of experimentation in Figure 10 for all the three datasets used in our work. From figure 10, it is clearly evident that the training and validation losses converge to their optimal values without any model over-fitting issues.
Changes made in the manuscript : Updated manuscript includes plots of training loss versus validation loss for all the three datasets used during experimentation in Figure 10.
Comment 6 : Abbreviations should be defined once on their first appearance.
Response : We thank the reviewer for this valuable suggestion. Any discrepancies related to abbreviations have been carefully looked at and solved in the updated version of our manuscript.
Comment 7 : On line 161, please provide correct definitions of n and d (which one denotes rows and which one denotes columns).
Response : We thank the reviewer for catching the error in the definition of terms n and d on line 161 in our original manuscript. It has been corrected in this version of our manuscript (n denotes the rows and d denotes the columns).
Changes made in the manuscript : Necessary changes have been made after line 177 in this version of our manuscript.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
The author responded carefully to our comments and revised them accordingly, which greatly improved the manuscript, but there are two details that need to be fixed.
(1). To solve the imbalance of spatial-spectral contribution, the following two articles may be more relevant to the motivation of this paper.
1. "Central Attention Network for Hyperspectral Imagery Classification", IEEE Trans Neural Netw Learn Syst 2022
2."Superpixel Guided Deformable Convolution Network for Hyperspectral Image Classification", IEEE Trans Image Process 2022
(2). There are some format errors of some references, such as
Ji, S., Xu,W., Yang, M. and Yu, K. 3D convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 2012, 35(1), 221-231.
Schuster, M. and Paliwal, K.K. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 1997, 45(11), 2673-2681.
……
Author Response
We thank the reviewer for their valuable comments and have edited the manuscript to address the reviewer’s concerns. Below is an item-by-item response to all the comments made during the review process.
Comment 1: To solve the imbalance of spatial-spectral contribution, the following two articles may be more relevant to the motivation of this paper.
- "Central Attention Network for Hyperspectral Imagery Classification", IEEE Trans Neural Netw Learn Syst 2022
2."Superpixel Guided Deformable Convolution Network for Hyperspectral Image Classification", IEEE Trans Image Process 2022
Response : We appreciate the reviewer’s feedback and have cited the first reference in our work as it is relevant to the motivation of our work. However, we strongly believe that the second reference suggested by the reviewer is directly related to superpixels, which fall out of scope of our work, and hence feel that citing this paper would be irrelevant.
Changes made in the manuscript : A new citation for the first paper suggested by the reviewer has been added in line 53 in our updated manuscript.
Comment 2: There are some format errors of some references, such as,
Ji, S., Xu,W., Yang, M. and Yu, K. 3D convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 2012, 35(1), 221-231.
Schuster, M. and Paliwal, K.K. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 1997, 45(11), 2673-2681.
Response : The authors appreciate the reviewer’s concern about such inconsistencies present in the reference section of our paper. All such inconsistencies have been corrected to the best of our knowledge.
Author Response File: Author Response.pdf
Reviewer 2 Report
The reviewer highly appreciates the authors' effort in revising the paper. Many comments have been raised, and the readability has been improved. However, there are still some points that require clarification.
1. The paper is relatively lengthy, which may reduce the overall readability. It is deemed unnecessary to provide too many details because interested readers generally have a good background in the field. Let take the paragraph pertinent to the attention mechanism (on lines 83-111) as an example. The reviewer thinks that it is unnecessary to detail how this mechanism have been utilized in other fields. Instead, its popularity in image processing can be summarized in one or two sentences. This will help readers focus on the role of attention mechanism in the proposed method.
2. The reviewer has not been convinced that the fourth point (on lines 153-155) is a novel contribution. In the reviewer's opinion, it is an essential requirement for comparative evaluation. And the reviewer would like to suggest the following.
The spatial-only and spectral-only variants should be excluded from the comparative evaluation. It would be better to compare to benchmark methods rather than these two variants of the proposed method. Instead, the reviewer highly suggests the authors create a new section called "Ablation study" for evaluating the proposed method against those two variants. This would verify the contribution of spatial features and spectral features to the overall performance, as well as justifying the significance of adopting both spatial and spectral features.
To sum up, it would be better to change the fourth point to "We present ablation studies by considering two variants of the proposed method to verify ..." and add an "Ablation study" section.
3. It is highly advisable to scrutinize the entire paper for the correctness and consistency of equations. The notations adopted by the authors are very confused because it is unclear which ones are matrices, vectors, or scalars.
Generally, boldfaced uppercase letters denote matrices, boldfaced lowercase letters denote vectors, and plain lowercase letters denote scalars.
Representations of row and column vectors should be also taken into consideration. For example, it would be better to denote u(k) = [u1, ..., ud]T (a column vector) and p(i) = [p1, ..., pd] (a row vector), so that it can be easily noticed that tki = p(i)u(k) is a scalar.
4. In the abstract, the authors claimed that the proposed method can selectively accentuate cardinal spatial and spectral features while suppressing less useful ones. Thus, please provide evidence (e.g., maybe visualization of features) to support that claim.
5. Evaluation metrics (overall accuracy and kappa) are not introduced in the paper. Please introduce them and their formulas briefly.
6. The analysis on execution time should be double-checked.
6.1. According to Table 8, execution time on Salinas is longer than execution time on Pavia for 3×3, 5×5, and 7×7 window sizes. However, the opposite is observed for 9×9, 11×11, and 13×13 window sizes. This requires a respective analysis.
6.2. Also from Table 8, it can be observed that performance increases as window size increases on Salinas and Pavia, and this makes sense. However, on Indian, a sharp drop is observed when window size changes from 3×3 to 5×5. First, please double-check for experimental errors, and then provide respective analysis.
Author Response
We thank the reviewer for their valuable comments and have edited the manuscript to address the reviewer’s concerns. Below is an item-by-item response to all the comments made during the review process.
Comment 1: The paper is relatively lengthy, which may reduce the overall readability. It is deemed unnecessary to provide too many details because interested readers generally have a good background in the field. Let's take the paragraph pertinent to the attention mechanism (on lines 83-111) as an example. The reviewer thinks that it is unnecessary to detail how this mechanism have been utilized in other fields. Instead, its popularity in image processing can be summarized in one or two sentences. This will help readers focus on the role of attention mechanism in the proposed method.
Response : While we appreciate the reviewer’s feedback, we respectfully disagree. We think this study makes a valuable contribution to the field and it is our goal to make the content written in our work fairly easily reachable and understandable and reproducible to researchers, not only from the remote sensing community, but also from other relevant broad areas of research. Hence, we strongly believe that the section of our paper that provides a brief summary on the current research status and history of attention technique in hyperspectral data analysis needs to be published in its current form without any modifications.
Comment 2: The reviewer has not been convinced that the fourth point (on lines 153-155) is a novel contribution. In the reviewer's opinion, it is an essential requirement for comparative evaluation. And the reviewer would like to suggest the following.
The spatial-only and spectral-only variants should be excluded from the comparative evaluation. It would be better to compare to benchmark methods rather than these two variants of the proposed method. Instead, the reviewer highly suggests the authors create a new section called "Ablation study" for evaluating the proposed method against those two variants. This would verify the contribution of spatial features and spectral features to the overall performance, as well as justifying the significance of adopting both spatial and spectral features.
To sum up, it would be better to change the fourth point to "We present ablation studies by considering two variants of the proposed method to verify ..." and add an "Ablation study" section.
Response : While we appreciate the reviewer’s feedback, we respectfully disagree. The goal of this paper is not to conduct an ablation study but to provide a more comprehensive perspective of the approach to our readers. Hence, the comment made by the reviewer is not relevant to this study. The effectiveness of our technique has been exhaustively compared against a bi-directional LSTM-based spectral attention and hyperspectral data classification methodology SPEC-HYP-ATTN, a ResNet architecture-based spatial attention methodology SPAT-HYP-ATTN, a spatial-spectral feature construction and HSI classification framework GAB-RP-S-3DCNN, a comprehensive spatial-spectral attention technique for pixelwise HSI data classification SSAN and the well-known composite kernel SVM-based feature extraction and classification technique which produce exceptional results and are state-of-the-art methodologies.
Comment 3: It is highly advisable to scrutinize the entire paper for the correctness and consistency of equations. The notations adopted by the authors are very confused because it is unclear which ones are matrices, vectors, or scalars.
Generally, boldfaced uppercase letters denote matrices, boldfaced lowercase letters denote vectors, and plain lowercase letters denote scalars.
Representations of row and column vectors should be also taken into consideration. For example, it would be better to denote u(k) = [u1, ..., ud]T (a column vector) and p(i) = [p1, ..., pd] (a row vector), so that it can be easily noticed that tki = p(i)u(k) is a scalar.
Response : Taking the reviewer’s suggestion into consideration, necessary changes in the representation of equations have been made and have been checked for consistency in our updated manuscript.
Comment 4: In the abstract, the authors claimed that the proposed method can selectively accentuate cardinal spatial and spectral features while suppressing less useful ones. Thus, please provide evidence (e.g., maybe visualization of features) to support that claim.
Response : We fear the reviewer might have misunderstood the concept of ‘attention’ here. Hyperspectral data generally has more than 100 features per data point effectively encoded in the spectral/temporal dimension of the HSI data cube. It is not a common practice in the hyperspectral data analysis and image classification community to visualize hundreds of features as it is impossible, and hence, innovative dimensionality reduction methods are used to project hyperspectral data from its original dimensional space to a lower dimensional subspace just for the purpose of visualization. Thus, we ask the reviewer to refer to the attention methodologies and papers in literature to gain further clarity and knowledge on the general methodology of ‘attention’.
Comment 5: Evaluation metrics (overall accuracy and kappa) are not introduced in the paper. Please introduce them and their formulas briefly.
Response : We thank the reviewer for this comment. However, we strongly believe that accuracy and kappa are commonly used metrics in machine learning and deep learning and do not need to be defined in general in the community.
Comment 6: The analysis on execution time should be double-checked.
6.1. According to Table 8, execution time on Salinas is longer than execution time on Pavia for 3×3, 5×5, and 7×7 window sizes. However, the opposite is observed for 9×9, 11×11, and 13×13 window sizes. This requires a respective analysis.
6.2. Also from Table 8, it can be observed that performance increases as window size increases on Salinas and Pavia, and this makes sense. However, on Indian, a sharp drop is observed when window size changes from 3×3 to 5×5. First, please double-check for experimental errors, and then provide respective analysis.
Response : The overall execution time for varying window sizes on all the three datasets in this work was determined empirically, averaged over three trials and carefully documented. However, slight fluctuations in overall execution time might be because artificial neural network (ANN) algorithms are non-deterministic in nature, where we arrive at different positive/negative outputs at every layer, each time we run the algorithm on the same inputs. This indirectly might affect the complexity of operations that are needed to be completed during the back-propagation phase of optimizing the underlying ANN-based algorithm.
However, after carefully verifying the overall classification accuracy of Indian Pines dataset for a window size of 3x3, it was realized that the accuracy was documented incorrectly in the previous version of our manuscript which has been corrected in the latest version.
Changes made in the manuscript : The overall classification accuracy for Indian Pines dataset for a window size of 3x3 has been updated in Table 8.
Author Response File: Author Response.pdf
