A Typed Iteration Approach for Spoken Language Understanding
Round 1
Reviewer 1 Report
Spoken language understanding (SLU) is a practical and timely research topic. The authors of this manuscript studied the SLU problem aiming at improving the intent detection performance. More specifically, they proposed a typed abstraction mechanism that utilizes the encoded information of slot filling tasks to enhance the performance of intent detection. They designed a typed iteration approach to achieve the bidirectional connection of the encoded information and mitigate the negative effects of error propagation. The experimental evaluation was conducted based on two benchmarks ATIS and SNIPS. The results present the superiority of their approach.
Overall, this paper is well prepared and easy to follow. The two examples provided in Section 1 are very helpful for readers to understand this research. The topic studied is interesting, timely, and practical. The approach proposed is effective. The experimental evaluation is comprehensive and convincing.
There are only a few typoes to be addressed. For example:
1) in Section 4.3, "BERT-Base-Uncased is pre-trained by Bookcorpus", the "by" should be "based on".
2) the citations are incorrect in Table 3 (Section 4.4).
3) to the reviewer's knowledge, the "Data Availability Statement" is to disclose the data provided by authors rather than third parties. Those publicly available datasets can be provided by citations or footnotes.
Author Response
Thank you very much for your careful review. According to your review comments, the modifications are as follows:
1) in Section 4.3, "BERT-Base-Uncased is pre-trained by Bookcorpus", the "by" should be "based on".
Modified.
2) the citations are incorrect in Table 3 (Section 4.4).
Sorry, there was a problem with cross reference when saving the original manuscript. Modified.
3) to the reviewer's knowledge, the "Data Availability Statement" is to disclose the data provided by authors rather than third parties. Those publicly available datasets can be provided by citations or footnotes.
Modified. The references which provide the two publicly availabe datasets are cited.
Reviewer 2 Report
Good quality paper which looks to be is in aim and scope of journal. The title of the article corresponds to the presented material. The topic is relevant for computer science. The introduction and related work are convincing and complete. The obtained results have practical value. The used methodology of scientific research deserves attention. The presented material has elements of novelty and can have practical application in computer engineering.
Author Response
Dear reviewer,
Thank you for your careful review.
Reviewer 3 Report
In this paper, the authors have proposed a typed abstraction mechanism that can explicitly enhance the intent detection task. The paper is well-written, but I have the following comments.
1. The authors need to provide proper motivation for selecting BERT as the encoding layer.
2. The authors add up the losses for the joint training. Can the scaling of the ID and SF loss before addition improve the model?
3. The authors need to provide the motivation behind selection of the different evaluation metrics.
4. The authors need to provide detailed discussion why their model performed better than the other existing models in the literature.
Author Response
Dear reviewer,
Thank you very much for your careful and in-depth review.
The following is our revision and answer to your review comments：
1. The authors need to provide proper motivation for selecting BERT as the encoding layer.
Due to the popularity of pre-trained language models such as BERT [21] and some research having already shown the utility of pre-trained language models on the SLU and other natural language processing task [22, 23], we also use BERT as the encoding layer and use the result of the BERT-joint method as the baseline in this paper.
We explained the reason for choosing BERT as the encoding layer from line 135 to 200 of the paper, but it may not be clear enough, so we made some modifications to the paper.
2. The authors add up the losses for the joint training. Can the scaling of the ID and SF loss before addition improve the model?
We have done some preliminary experiments and did not find that the scaling of the ID and SF loss before addition can improve the model. In future research, we will conduct more in-depth analysis and experiments.
3. The authors need to provide the motivation behind selection of the different evaluation metrics.
In spoken langauge understanding task, Intent detection can be seen as a classification problem to decide the intent label oI of an utterance, and Slot filling is generally seen as a sequence labeling task. In order to compare the performance of our model with other related work conveniently, we use evaluation metrics in the experiments following Qin et al. (2019)[6] and Goo et al. (2018)[15], and evaluate the SLU performance of slot filling using F1 score and the performance of intent prediction using accuracy, and sentence-level semantic frame parsing using overall accuracy.
In view of the reviewer's opinion, we added more explanation of selecting different evaluation metrics in the paper to facilitate readers' understanding.
4. The authors need to provide detailed discussion why their model performed better than the other existing models in the literature.
Terribly sorry that we feel there is some explanation in the paper. The existing joint models cannot explicitly use the encoded information of the two subtasks, Intent detection and Slot filling, to realize their mutual interaction, nor can they achieve the bidirectional connection between them. So we designed a typed abstraction mechanism to enhance the performance of intent detection by utilizing the encoded information of SF tasks, and the experimental results show that this mechanism does have certain effects.
