Anomaly Detection of Zero-Day Attacks Based on CNN and Regularization Techniques

Round 1

Reviewer 1 Report

The authors propose a method for detecting zero-day attacks. The method`s performance is evaluated using precision, recall, F measure, and accuracy and the results demonstrate that the proposed method performs better in terms of accuracy. Some concerns need to be addressed before the paper is published.

o   It is not clear what the real research contributions are. There are many such techniques available in the literature. The authors need to clarify the differences from the related works, and contributions of this article.

o   The authors need to give more detailed explanations of the main point of the proposal. What is a proposal?

o   The authors need to give detailed explanations of how the authors train the models.

o   The authors need to clarify the experimental environments and conditions. In the article, the authors do not include any explanations and conditions of the experiment. The section should cover enough details of the proposed work.

o   The presentation of the paper needs to be improved e.g., equations should be part of the proposed methodology instead of the introduction.

o   What is the purpose of including figure 1?

o   There should be consistency in terminologies used i.e., in some occurrences the authors use zero-day attack for the proposal, in others they use back door attack whereas DDoS attack is also used in some occurrences. 

o   The authors need to add discussions of the evaluation results. In the article, the authors only show the results of different classifiers. The authors need to add findings from the evaluation results. Comparative results with other relevant methods should be included to validate the results of the proposed method. The authors employ four evaluation metrics, Accuracy, Precision, Recall, and F1-score. From these evaluation metrics, what do the authors want to say?

 

Author Response

Reviewer #1:

o   It is not clear what the real research contributions are. There are many such techniques available in the literature. The authors need to clarify the differences from the related works, and contributions of this article.

[Authors]: We extended the usage and evaluation of regularization techniques in CNN to include more zero-day attacks and a new dataset to be validated on. We added more illustrations in Abstract [lines 11, 14 -> 17], and introduction [lines: 67 -> 81].

 

o   The authors need to give more detailed explanations of the main point of the proposal. What is a proposal?

[Authors]: Same answer above.

 

o   The authors need to give detailed explanations of how the authors train the models.

[Authors]: Section 3 now has all the details related to the training phase: the dataset used, data preparation, features selection, amount of used data, used classifiers, and the used hardware and software. We re-ordered some subsections, and added a figure in page 8 to wrap all training phase steps in one figure.

 

 

o   The authors need to clarify the experimental environments and conditions. In the article, the authors do not include any explanations and conditions of the experiment. The section should cover enough details of the proposed work.

[Authors]: We moved the subsection of the used hardware and software to be in section 3 instead of 4.

 

o   The presentation of the paper needs to be improved e.g., equations should be part of the proposed methodology instead of the introduction.

[Authors]: As requested, we moved this paragraph to a subsection in section 3.

 

o   What is the purpose of including figure 1?

[Authors]: To demonstrate the used CNN architecture, we changed the figure style to be more clear.

 

o   There should be consistency in terminologies used i.e., in some occurrences the authors use zero-day attack for the proposal, in others they use back door attack whereas DDoS attack is also used in some occurrences.

[Authors]: A zero-day attack is a concept for unknown attack for the detection system, regardless the type/name of the attack. In our case, the tested zero-day attacks were Backdoor, and Scanning. We added elaborations about this in lines: 71 -> 73

o   The authors need to add discussions of the evaluation results. In the article, the authors only show the results of different classifiers. The authors need to add findings from the evaluation results. Comparative results with other relevant methods should be included to validate the results of the proposed method. The authors employ four evaluation metrics, Accuracy, Precision, Recall, and F1-score. From these evaluation metrics, what do the authors want to say?

[Authors] The mentioned parameters are standard metrics that are usually used in all anomaly-based methods evaluations. We’ve added a reference in line 308 that explains them in details.

Reviewer 2 Report

This paper proves that the regularized CNN classifiers have better prediction quality regarding the Zero-Day attacks rather than the classical ML based methods. Overall, this article is well organized and its presentation is good. This paper has many experiments and a lot of works. However, some major issues still need to be improved:

1. Suggested that the introduction should be simplified. Lines 41-67: This part should focus on CNN-based detection methods and reduce the introduction of signature-based detection methods. Because the article mainly introduces the detection method of CNN.

2. Lines 69-75: Suggested that this part only describes the reasons of combining CNN and regularization techniques rather than gives the L1 and L2 technique formulas.

3. Lines 186-192: This part should add a summary of the existing literature and the reasons for the proposed method should be given at the end of the related work.

4. Lines 348-371: Suggested that the conclusion should be rewritted. The conclusion is not clear and not concise. Please this part is a summary of the full article. The description of TON-IoT dataset is too many about how to collect data. This mainly write proposed method, used security dataset, experimental results and future directions briefly and concisely.

Author Response

Reviewer #2:

 

1. Suggested that the introduction should be simplified. Lines 41-67: This part should focus on CNN-based detection methods and reduce the introduction of signature-based detection methods. Because the article mainly introduces the detection method of CNN.

[Authors]: As suggested, we shrank the signature-based detection lines, but kept some details just to elaborate the differences with anomaly detection based methods.

 

2. Lines 69-75: Suggested that this part only describes the reasons of combining CNN and regularization techniques rather than gives the L1 and L2 technique formulas.

[Authors]: We elaborated the reasons of using regularization techniques in introduction lines: 67 -> 70 and moved the mathematical details to a subsection in section 3.

 

3. Lines 186-192: This part should add a summary of the existing literature and the reasons for the proposed method should be given at the end of the related work.

[Authors]: we added a state of art comparison table in section 2 page 5.

 

4. Lines 348-371: Suggested that the conclusion should be rewritted. The conclusion is not clear and not concise. Please this part is a summary of the full article. The description of TON-IoT dataset is too many about how to collect data. This mainly write proposed method, used security dataset, experimental results and future directions briefly and concisely.

[Authors]: we rewrote the conclusion with some shrinking of not needed details.

Reviewer 3 Report

1.       This paper is about Cyber-attacks (Zero-Day attacks) and claims that CNN has a better quality, as compared to classical ML-based anomaly detection methods, for the detection of Zero-Day attacks. However, this has already been proved in the literature. Therefore, this paper's novelty or contribution must be highlighted. (in the abstract, introduction as well as literature review sections). Particularly speaking, the evaluation of CNN models, regularized by L1 and L2 techniques, regarding unknown attacks has already been discussed in the literature. Therefore, the existing techniques, addressing the same issue,  must be briefly overviewed in the introduction. Consequently, the research gap is required to be highlighted.

2.       It is recommended that the introduction should be rewritten. The following information should be provided in an appropriate way.

a.       The background on Anomaly detection of zero-day attacks.

b.       An overview of existing techniques (particularly CNN and Regularization techniques) for the anomaly detection of zero-day attacks.

c.       What are the limitations of existing CNN and Regularization techniques and therefore highlight the research gap

d.       What is being proposed? How the proposal will solve the identified research gap ?

e.       How the proposal is being validated?

f.        What are the achieved results? What is the significance of achieved results ?

3.       In section 2, a descriptive summary of existing methods is provided. However, a comprehensive analytical discussion (comparing the works with others and the proposed work is clearly missing). Therefore, the outcome of this section is to highlight the strengths and limitations of existing CNN and Regularization techniques. How the proposed work in this paper is different ? How it will address the limitations ?

4.       Section 2 is followed by section 3. This section is providing the implementation details. However, before providing the implementation details, a dedicated section is needed which should describe:

a.       The structure of proposed method

b.       Behavioral description of the proposed method

5.       Finally, the last major limitation of this work is the lack of performance comparison with state-of-the-art. The achieved results must be compared with state-of-the-art CNN and Regularization techniques in terms of certain performance attributes.

     The quality of article from English language point of view should be improved.

Author Response

Reviewer #3:

1. This paper is about Cyber-attacks (Zero-Day attacks) and claims that CNN has a better quality, as compared to classical ML-based anomaly detection methods, for the detection of Zero-Day attacks. However, this has already been proved in the literature. Therefore, this paper's novelty or contribution must be highlighted. (in the abstract, introduction as well as literature review sections). Particularly speaking, the evaluation of CNN models, regularized by L1 and L2 techniques, regarding unknown attacks has already been discussed in the literature. Therefore, the existing techniques, addressing the same issue, must be briefly overviewed in the introduction. Consequently, the research gap is required to be highlighted.

[Authors]: We added some modifications to highlight the contributions in abstract, introduction, and related work section by adding a state-of-art comparison.

 

2. It is recommended that the introduction should be rewritten. The following information should be provided in an appropriate way.

 

1. The background on Anomaly detection of zero-day attacks.

 

1. An overview of existing techniques (particularly CNN and Regularization techniques) for the anomaly detection of zero-day attacks.

 

1. What are the limitations of existing CNN and Regularization techniques and therefore highlight the research gap

 

1. What is being proposed? How the proposal will solve the identified research gap ?

 

1. How the proposal is being validated?

 

1. What are the achieved results? What is the significance of achieved results ?

 

[Authors]: for points a, b, c, d, and e: The details are mentioned in the introduction now with more highlighting to the research gap and elaborations of the regularization techniques. While for point e: it’s mentioned that the validation of the required method is by training and testing the classifiers on data from TON-IoT dataset.

For point f: we think that mentioning the results with their discussion are more fit in their sections 4, and 5 where all of the results are presented and visualized in details.

 

3. In section 2, a descriptive summary of existing methods is provided. However, a comprehensive analytical discussion (comparing the works with others and the proposed work is clearly missing). Therefore, the outcome of this section is to highlight the strengths and limitations of existing CNN and Regularization techniques. How the proposed work in this paper is different ? How it will address the limitations ?

[Authors]: As in response to comment #1, we added a state-of-art comparison table to the section 2 of related work.

 

4. Section 2 is followed by section 3. This section is providing the implementation details. However, before providing the implementation details, a dedicated section is needed which should describe:

 

1. The structure of proposed method

 

1. Behavioral description of the proposed method

[Authors]: Section 3 is dedicated to present all the details regarding the experimental work details of implementation and the experiment phases and stages inputs and outputs.

 

5. Finally, the last major limitation of this work is the lack of performance comparison with state-of-the-art. The achieved results must be compared with state-of-the-art CNN and Regularization techniques in terms of certain performance attributes.

[Authors]: As in response to comment #1, we added a state-of-art comparison table to the section 2 of related work.

 

Reviewer 4 Report

The paper presents some techniques for anomaly detection of zero-day attacks based on CNN.

But there are no original elements, the paper has no novelty, it is only a study. There is nothing new.

A proofreading is required.

Author Response

Reviewer #4:

The paper presents some techniques for anomaly detection of zero-day attacks based on CNN.

 

But there are no original elements, the paper has no novelty, it is only a study. There is nothing new.

 

A proofreading is required.

 

We agree with the reviewer that further work is needed to improve the paper contribution and raise its novelty. In this article, we evaluate a new Deep Learning approach based on the convolutional neural network (CNN) to classify the flow traffic into normal or attack classes. While several related works deployed ML/DL for NIDSs, most of these approaches ignore the influence of the overfitting problem during the implementation of such algorithms. In addition, the current solution have a low ability on network flow analysis. The network traffic is very dynamic and the relationships between input and output data can change over time (i.e., concept drift). These changes can result in poor and degrading predictive performance in predictive models. In this work, we investigate the current network intrusion detection systems (NIDSs) and their limitations. Different regularization techniques i.e. L1 and L2 have been used to address the problem of overfitting and to improve the capability of NIDSs in detection of unseen intrusion events. The CNN has been used to decrease the number of training parameters, and therefore, we can generate a new model capable of detecting network intrusions with a low computational cost. The evaluation results indicate that the regularization methods outperform the standard CNN algorithm and the classical ML techniques.  Furthermore, several experiments are performed to verify the performance of the proposed CNN models for unknown attacks. We believe the current version solved one significant obstacle in the development of lightweight intrusion detection by training the DL based CNN model using a less number of features without causing a significant drop in the model performance.

Thanks for considering our work.

Round 2

Reviewer 1 Report

My previous concerns are addressed in the revised version.

Author Response

We thank the reviewer for accepting our paper. 

Reviewer 2 Report

I am glad that authors revised this paper based on comments. This paper can be accepted. However, some issues still need to be improved:

1. It is necessary to introduce TON-IoT datasets [10–15] and the reason to use TON-IoT. In Table 1, only this paper use TON-IoT.

2. The used classifiers are Logistic Regression (LR), Naive Bayes (NB), AdaBoost, regular CNN, CNN L1, and CNN L2. However, these methods are not compared in table 2. The CNN L1 and CNN L2 should be defined definitely in section 3.

Author Response

Reviewer #2:
1. It is necessary to introduce TON-IoT datasets [10–15] and the reason to use
TON-IoT. In Table 1, only this paper use TON-IoT.
[Authors]: The article has a description paragraph explains the TON-IoT dataset in section 3.2,
we also followed your suggestion by adding few lines in the Introduction section (lines: 74 ->
77) to introduce the dataset to the reader before diving into details.
2. The used classifiers are Logistic Regression (LR), Naive Bayes (NB),
AdaBoost, regular CNN, CNN L1, and CNN L2. However, these methods are not
compared in table 2. The CNN L1 and CNN L2 should be defined definitely in
section 3.
[Authors]: We are not sure that we quite understood this concern. However, kindly note the
following:
 If the required to theoretically define those classifiers, you can check section 3.1 (lines:
208 -> 216) as we theoretically defined CNN L1, and CNN L2 here.
 They are also defined practically in section 3.5 and figure 2 where we state that the
regularization factor for both methods is 0.005.
 We added new references for Naïve Bayes, Logistic Regression, and AdaBoost that
explains the background of those classifiers (lines: 199, and 200).
 Table 2 is concerned about comparing other articles in detecting attacks in IoT. That
means that not all of them have the same used the same classifiers we use. Our paper
measures the impact of using regularization techniques in CNN in detecting zero-day
attacks in comparison with standard CNN and classical ML methods.
Thanks for considering our work.

Reviewer 3 Report

The articles have addressed the raised concerns. 

The revised article is in much better condition as compared to the original submission. 

 

 

Author Response

We thank to the reviewer for accepting our paper