Detecting Data Anomalies from Their Formal Specifications: A Case Study in IoT Systems

Round 1

Reviewer 1 Report

The paper aims to present a novel method to detect anomalies in datasets by comparison from a data set with normal behavior and the data sets under attack.

 

The proposed method was explained using formal language. However, some points in the text need to be reviewed:

 

-       Who are “we”? Once there is only one author.

-       Normally, in the MDPI papers, there are the “Author Contributions” that describe the work of each author in the paper, such as conceptualization, methodology, and so on … This section is missing in this paper. 

-       In the text, “we find that the majority of IoT literature is mostly concerned with either investigating such systems as sources of data and information, often to be used as validation datasets for machine learning algorithms that detect different classes of properties in these datasets, or with the classical software engineering aspect of these systems that aims to achieve specific IoT based solutions to all sorts of problems in different domain of applications.” The author should provide the reference of the studies that are referenced. The same for the end of this paragraph.

-       There are missing Figures 4, 5, 7, and 9. Please review that.

-       The statement, “We highlight that this method is more direct and robust than fuzzy methods that use machine learning algorithms as it relies directly on formal specification and verification methods in detecting anomalous data.” How to prove it? The author should add a reference from studies that prove this statement or add another section with tests and comparisons made by the author. 

-       Another question is about the term “delivery semantics.” The MQTT uses the Quality of Service to deliver messages that correspond to the confirmation message between to publisher or subscriber to the broker. The term delivery semantics comes from Apache Kafka, a technology different from MQTT. Apache Kafka stores a large amount of data that will be delivered to its subscribers. I suggest reviewing the MQTT and Apache Kafka to see what technology feet better on the method proposed.

-    Most of the references are from the author. Is it an extension of the work previously developed? Are there related works with this? A related works section should be added as a recommendation with papers published in the last five years.

Author Response

-       Who are “we”? Once there is only one author.
The standard literature way of writing a paper is to always use second/third person, never first person, regardless of whether or not there are multiple authors in a paper.

-       Normally, in the MDPI papers, there are the “Author Contributions” that describe the work of each author in the paper, such as conceptualization, methodology, and so on … This section is missing in this paper.
The paper has only one author, hence, naturally, the paper as a whole is the sole author's work!

-       In the text, “we find that the majority of IoT literature is mostly concerned with either investigating such systems as sources of data and information, often to be used as validation datasets for machine learning algorithms that detect different classes of properties in these datasets, or with the classical software engineering aspect of these systems that aims to achieve specific IoT based solutions to all sorts of problems in different domain of applications.” The author should provide the reference of the studies that are referenced. The same for the end of this paragraph.
This is done now.

-       There are missing Figures 4, 5, 7, and 9. Please review that.
These were referring to figures in the case study from literature, however, I have now removed the figure reference to these and used each dataset directly citing the specific section it appears in, in the case study it is adopted from.

-       The statement, “We highlight that this method is more direct and robust than fuzzy methods that use machine learning algorithms as it relies directly on formal specification and verification methods in detecting anomalous data.” How to prove it? The author should add a reference from studies that prove this statement or add another section with tests and comparisons made by the author. 
We have removed the word "more direct", and replaced with the word "different."

-       Another question is about the term “delivery semantics.” The MQTT uses the Quality of Service to deliver messages that correspond to the confirmation message between to publisher or subscriber to the broker. The term delivery semantics comes from Apache Kafka, a technology different from MQTT. Apache Kafka stores a large amount of data that will be delivered to its subscribers. I suggest reviewing the MQTT and Apache Kafka to see what technology feet better on the method proposed.
Delivery semantics is a generic term, and in the case of MQTT, it refers to QoS level.  I suggest the reviewer familiarise herself/himself with the MQTT OASIS specification.  Apache Kafka is totally irrelevant to this work, therefore, it makes no sense to reference this platform.

-    Most of the references are from the author. Is it an extension of the work previously developed? Are there related works with this? A related works section should be added as a recommendation with papers published in the last five years.
A related works section has been added.

Reviewer 2 Report

It is obvious that the technical description of the article is not sufficient.

Here are a few suggestions I would like to make.

1. The provision on page three needs to be changed.

2. How to complete the project content in Figure 1, the descriptions in lines 47 to 91 are mainly descriptions, which are not clear enough.

3. The article contains the machine learning referenced in the conclusion, which should be more fully described.

4. The experimental findings cannot be understood in the article, and only the reasoning and explanation of the number theory are given. Readers have difficulty understanding abstract experimental findings.

Author Response

1. The provision on page three needs to be changed.
Sorry, but this comment is unclear.

2. How to complete the project content in Figure 1, the descriptions in lines 47 to 91 are mainly descriptions, which are not clear enough.
This is standard theory section, which gives an introduction to the pi-calculus and its sos semantics.  This would normally be clear to anyone familiar with this language or any other process algebraic languages.

3. The article contains the machine learning referenced in the conclusion, which should be more fully described.
The reference is described in earlier sections.

4. The experimental findings cannot be understood in the article, and only the reasoning and explanation of the number theory are given. Readers have difficulty understanding abstract experimental findings.
The short paper presents only some initial validation of the idea, which is that formal specifications and their analysis can determine at earlier stages the presence of anomalies, which is more direct than fuzzy approaches.  We plan to expand the paper in the future with more detail.

Round 2

Reviewer 1 Report

The comments were addressed. There are no more comments.

Reviewer 2 Report

The author responded with an update of the article. Having examined the article, there are no further questions.