A Typed Iteration Approach for Spoken Language Understanding

Abstract

A spoken language understanding (SLU) system usually involves two subtasks: intent detection (ID) and slot filling (SF). Recently, joint modeling of ID and SF has been empirically demonstrated to lead to improved performance. However, the existing joint models cannot explicitly use the encoded information of the two subtasks to realize mutual interaction, nor can they achieve the bidirectional connection between them. In this paper, we propose a typed abstraction mechanism to enhance the performance of intent detection by utilizing the encoded information of SF tasks. In addition, we design a typed iteration approach, which can achieve the bidirectional connection of the encoded information and mitigate the negative effects of error propagation. The experimental results on two public datasets ATIS and SNIPS present the superiority of our proposed approach over other baseline methods, indicating the effectiveness of the typed iteration approach.

1. Introduction

Spoken language understanding is the core module in a task-oriented dialog system. It typically consists of two subtasks: intent detection (ID) and slot filling (SF) [1,2]. For the sentence given by the user, the SLU module identifies the user intent and the utterance slot labels and then uses them to execute user commands or continue a conversation with the user through the dialog management module. For instance,

Table 1. An example sentence.

Sentence	Flights	from	Beijing	to	Shanghai
Slot	O	O	B-fromloc	O	B-toloc
Intent	atis_flight

shows an example sentence “Flights from Beijing to Shanghai” extracted from the ATIS corpus. Each word in the sentence corresponds to a slot label, and the entire sentence has a specific intent. Correctly identifying the intent of the sentence and each word’s slot label are essential for downstream tasks.

Traditionally, the ID and SF tasks are implemented separately by pipeline models. Because the independent pipeline models ignore the characteristics that these two tasks are always co-occurring and highly correlated, the recent trend is to design joint models, which can be seen as a multi-task learning framework for ID and SF. Some studies have proven that joint models are effective in SLU [3,4,5]. However, the link methods of these joint models are implicit, such as applying a joint loss function or shared embedding. A more effective way is to use the encoded information to inter-act and enhance explicitly [6,7]. In theory, there are two different ways to achieve this explicit improvement. The first is to enhance the SF task by the encoded information of the ID task (I2S), and the second is to enhance the ID task by the encoded information of the slot filling task (S2I). Due to the poor performance of the S2I method, existing research mostly uses the I2S method, and the reasons for the poor performance of the S2I method have not been explored. In this paper, we believe that the main reason for the poor performance of the S2I method is that part of the slot encoded information can mislead the ID task. If this information can be eliminated, the S2I method is also effective. The main reasons for misleading information are as follows:

• Due to the variety of slot labels, there are many out-of-vocabulary (OOV) words in some slot categories (contact names, song names, etc.), such as in Case 1, shown in

  Table 2. Two examples of misleading information; different colors correspond to different slots. It is still easy to identify the intent only by the Result of Typed Abstraction section, containing only slot categories and not specific slot words.

  Case	Sentence	Result of Typed Abstraction	Intent
  1	Play Adele’s Rolling in the deep.	Play the artist’s song	Play music
  	Play Taylor Swift’s Sparks Fly.
  2	Send a message to Mr. Jack saying tonight Miss White has a date with him.	Send a message to name saying message_text	Send message
  	Send a message to Mrs. Tom saying remember to book two Taylor Swift concert tickets.

  : the joint model may not know what Adele means. The encoded information of the slot filling task is inaccurate at this time, and in-accurately encoded information can negatively affect the ID task.
• The specific meanings of the corresponding words in the slot can mislead the ID task (message text, recording content, etc.), as Case 2 shows in Table 2. The meaning of the message text slot in the two example sentences is entirely different, and the encoded information of the slot filling task is also different. However, their slot labels are the same and should obtain the exact representation for the ID task. Understanding the specific meaning of slot words cannot help the ID task but will mislead it.

Aiming at the problem of misleading information in the S2I method, we propose a typed abstraction mechanism. This mechanism abstracts slot words into type words through the result of the slot filling task and generates an abstract sentence. In this way, the misleading information contained in the slot words can be removed, and only the information that is meaningful to the ID task is retained. Through this mechanism, we can achieve the S2I method and avoid the negative impact of misleading information on the ID task. The Result of the Typed Abstraction section in Table 2 shows the abstract sentences after the typed abstraction mechanism.

Now, we can achieve the S2I method by the typed abstraction mechanism, but this mechanism brings a new problem. Because the mechanism is based on the SF task, it is as vulnerable to error propagation as the pipeline model. To solve this problem, we designed a typed iteration approach. Because the information interaction process is iterative in this approach, the adverse effects caused by error propagation can be mitigated. In addition, the typed iteration approach is not like the previous model that only used a single encoded information interaction method. We used both I2S and S2I methods to achieve interaction explicitly and accomplish a two-way connection of two subtasks.

To summarize, our main contributions are as follows:

3.

We propose a typed abstraction mechanism. Through this mechanism, we can use the encoded information of the slot filling task to explicitly enhance the ID task while avoiding the negative impact of misleading information on the ID task.

4.

We designed a typed iteration approach. This approach uses the typed abstraction mechanism and feature fusion mechanism to realize the bidirectional connection of the encoded information in the joint model. At the same time, it reduces the error propagation problem that the typed abstraction mechanism may bring through the iterative process. To the best of our knowledge, this is the first attempt to use the encoded information of two subtasks for simultaneous explicit enhancement and interaction.

5.

We performed experiments on two public corpora, ATIS and SNIPS, and our approach achieved state-of-the-art performance on both corpora. The experimental results prove the effectiveness of our model.

2. Related Work

SLU generally includes two subtasks: ID and SF. The traditional implementation method of SLU is to carry out ID subtask first and then SF independently in a pipeline way. Among them, the ID subtask is generally regarded as a classification problem, which is implemented by machine learning approaches such as support vector machines (SVMs) [8] and recurrent neural networks (RNNs) [9]. The SF task is regarded as a sequence annotation problem, which is realized by machine learning methods such as conditional random fields (CRF) [10], convolutional neural net-works (CNNs) [11], and recurrent neural networks (RNNs) [12]. Wang et al. [13] proposed a dual-model learning method for slot filling problem to leverage the large amounts of unlabeled data in a weakly supervised learning manner. The advantage of these methods is that the two subtasks are completed independently, and the goal of the subtasks is clear and easy to implement. The disadvantage is that the interaction and correlation between the two subtasks are not considered.

Therefore, to give full play to the interaction and promotion of the two subtasks, some studies have proposed various joint learning models to simultaneously train and implement ID and SF subtasks. Zhang et al. [4] proposed an RNN-based collaborative model to integrate the correlation between intention and slots. Hakkani et al. [14] proposed a joint RNN model of SF and ID. Liu et al. [5] presented a joint learning model of SF and ID based on the attention mechanism. Goo et al. [15] designed a slot gate mechanism to model the internal relationship between ID and SF. Li et al. [16] proposed an intention-enhanced gate mechanism to fuse the related semantic information between slots and intent. Want et al. [17] proposed a bidirectional model to cross-impact between the user intent and utterance slots. Zhang et al. [18] presented a capsule neural network model to encode hierarchical semantic relations between the slots and the intent of the utterance. E et al. [7] presented an SF-ID neural net-work association model to combine SF and ID subtasks. Hui et al. [19] proposed a continual learning interrelated model to utilize semantic information with different characteristics and balance the performance between ID and SF. Wang et al. [20] presented a transformer-based architecture for ID and SF, which encodes syntactical knowledge in it.

However, the methods mentioned above in general utilize the correlation of the two subtasks mainly based on the joint learning loss of two subtasks and cannot explicitly use the encoded information of the two subtasks to employ mutual interaction. Therefore, we utilized the encoded information of the SF task to augment ID performance. Due to the popularity of pre-trained language models such as BERT [21] and some research having already shown the utility of pre-trained language models on the SLU and other natural language-processing tasks [22,23], we also used BERT as the encoding layer and used the result of the BERT-joint method as the baseline in this paper.

3. Approach

In this section, we introduce our approach in detail. Figure 1 gives an overview of our approach. First, we present the typed abstraction mechanism; this part is the implementation of the S2I method. Second, we introduce our typed iteration approach. Finally, we introduce the joint training method used in our approach.

3.1. Typed Abstraction Mechanism

For a given sentence $X=(w_1,w_2,\dots ,w_n)$, we abstract the sentence based on the results of the SF task $S=(s_1,s_2,\dots ,s_n)$. The specific approach is to use slot filling result S to replace the actual slot words in the sentence with abstract type words and obtain the typed sentence X′. Since the slot words in the typed sentence X′ are replaced by type words, they will not be affected by misleading or redundant information when used for intent detection. In order for the two subtasks to interact directly with the encoded information in the approach, we align the sentence during the type abstraction to ensure that X and X′ have the same length. The specific process of the type abstraction mechanism is shown in Figure 2.

3.2. Typed Iteration Approach

The typed iteration approach we propose, as shown in Figure 1, is mainly composed of the following four modules: SF module, typed abstract module, intent-detection module, and feature fusion module. First, in view of the outstanding effect of the pre-trained model BERT [21] on many other natural language-processing tasks, it is used to encode the sentence $X=(w_1,w_2,\dots ,w_n)$. We take the last hidden layer vector $H=(h_1,h_2,\dots ,h_n)\in R^{n\times d}$ (n is the length of the sentence, and d is the dimension of the hidden layer vector) calculated by BERT as an encoding representation of a sentence:

$$H=BERT(w_1,w_2,\dots ,w_n)$$

(1)

Then, we perform feature fusion with H and the vector $H^{\prime }=(h_1^{\prime },h_2^{\prime },\dots ,h_n^{\prime })\in R^{n\times d}$ obtained in the previous iteration (H′ in the first iteration is a randomly initialized vector), to obtain a new hidden layer vector $H^S\in R^{n\times d}$. Then, the vector is used to obtain the SF task result $S=(s_1,s_2,\dots ,s_n)$:

$$H^S=H\oplus H^{\prime }$$

(2)

$$Y^S=\mathrm{softmax}(w^S{H}^S+b^S)$$

(3)

$$S=\mathrm{argmax}(Y^S)$$

(4)

After that, the SF result S is used to perform type abstraction mechanism on the input sentence X to obtain the typed sentence $X^{\prime }=(w_1^{\prime },w_2^{\prime },\dots ,w_n^{\prime })$ and then use it for the intent-detection task. We use the same BERT model to encode X′ and obtain the encoded representation $H^{\prime }=(h_1^{\prime },h_2^{\prime },\dots ,h_n^{\prime })\in R^{n\times d}$ of the typed sentence X′. We processed the vector H′ by mean pooling and $H^l$ and then used it to obtain the ID results:

$$X^{\prime }=\mathrm{TypedAbstraction}(X,S)$$

(5)

$$H^{\prime }=\mathrm{BERT}(w_1^{\prime },w_2^{\prime },\dots ,w_n^{\prime })$$

(6)

$$H^l=\frac{1}{n}{\displaystyle {\sum }_{x\in n}{h}_x^{\prime }}$$

(7)

$$Y^l=\mathrm{softmax}(w^l{H}^l+b^l)$$

(8)

The ID result $I=\arg \max (Y^l)$ and the SF result S are then compared with the results of the previous iteration. If the results are the same, or the preset maximum iteration round is reached, the iteration is terminated, and the result of the current round of ID and SF is used as outputs; otherwise, the above process is repeated. The specific description of the typed iteration algorithm is as shown in Algorithm 1.

Algorithm 1 Typed iteration algorithm

Input: Input sentence X; Parameter: A random vector $H^{\prime }$ has the same shape with H; Max iteration number m; t: 0; A vector $S_{t-1}$ initialized to 0; A vector $I_{t-1}$ initialized to 0 Output: S, I flag←True H = BERT(X) while flag do $S_t=\mathrm{SlotFilling}([H\prime H^{\prime }])$ $X^{\prime }=\mathrm{TypedAbstraction}(X,S_t)$ $H^{\prime }=BERT(X^{\prime })$      $H^l=\mathrm{MeanPooling}(H^{\prime })$ $I_t=\mathrm{IntentDection}(H^l)$     if ($S_t⩵S_{t-1}$ and $I_t⩵I_{t-1}$ or $t⩵m$ then      $flag\leftarrow False$     else      $S_{t-1}\leftarrow S_t$      $I_{t-1}\leftarrow I_t$       $t\leftarrow t+1$    end if end while return S; I

The ID result I and the SF result S are then compared with the results of the previous iteration. If the results are the same, or the preset maximum iteration round is reached, the iteration is terminated, and the result of the current round of ID and SF is used as outputs; otherwise, the above process is repeated. The specific description of the typed iteration algorithm is shown in Algorithm 1.

3.3. Joint Training

Because the typed abstraction mechanism interrupts the continuity of the gradient, we calculate the loss for each iteration in the typed iteration approach and finally add the losses for joint training. ID and SF loss are calculated as follows:

$$L_S\triangleq -{\displaystyle {\sum }_{x=1}^t{\displaystyle {\sum }_{z=1}^n{\widehat{y}}_z^S\log (y_z^{S,x})}}$$

(9)

$$L_I\triangleq -{\displaystyle {\sum }_{x=1}^t{\widehat{y}}^I\log (y^{I,x})},$$

(10)

where ${\widehat{y}}^I$ and ${\widehat{y}}_z^S$ are the gold intent label and gold slot label, respectively; n is the number of slot labels, and t is the number of iterations. To jointly model the intent classification and SF tasks, the objective is formulated as follows:

$$p(y^S,y^I|X)=p(y^I|X){\displaystyle {\prod }_{t=1}^{T}p(y_t^S|X)}$$

(11)

Compared with pipeline models [4], the shared representations learned through joint training by the encoder will combine two subtasks jointly and further ease the error propagation.

4. Experiments

4.1. Datasets

We performed experiments on two public datasets, ATIS [24] and SNIPS [25]. ATIS is a dataset related to the aviation field that is widely used in the task of spoken language understanding. SNIPS is a dataset collected from Snips Voice Assistant, and the distribution of categories is relatively balanced. The specific statistics of the two datasets are shown in

Table 3. Specific statistics of the two datasets.

Dataset	ATIS	SNIPS
Vocabulary size	722	11,241
#Intents	21	7
#Slots	120	72
#Training samples	4478	13,084
#Validation samples	500	700

. The format and division of datasets used in this paper are the same [15].

4.2. Evaluation Metrics

In order to compare the performance of our model with other related work, conveniently, we used three evaluation metrics in the experiments following [6,15]. For the ID task, we used the accuracy rate. For the SF task, the F1-score was applied. In addition, we used the sentence accuracy rate as the overall assessment of the sentence level; that is, the sentence is correct when the intent detection and all slots in a sentence are correctly identified. The calculation method of the evaluation metrics in this paper is the same as [6,15].

4.3. Experimental Settings

We used BERT-Base-Uncased (https://github.com/google-research/bert, accessed on 16 August 2022), released by Google, as the pre-training parameters of our BERT encoding layer. BERT-Base-Uncased is pre-trained based on Bookcorpus (800M words) and English Wikipedia (2500M words), the hidden layer dimension is 768, and the parameter size is 110M. The maximum number of iterations of the typed iteration approach is set to 4, the learning rate is preset to 10⁻⁵, and adaptive adjustment is made according to the loss of the validation set.

4.4. Overall Results

Table 4. Comparison of experimental results of different methods.

Dataset	SNIPS			ATIS
	Intent (acc)	Slot (F1)	Sentence (acc)	Intent (acc)	Slot (F1)	Sentence (acc)
Joint Seq (Dilek et al., 2016) [14]	96.9	87.3	73.2	92.6	94.3	80.7
Attention BiRNN (Liu et al., 2016) [14]	96.7	87.8	74.1	91.1	94.2	78.9
Slot-Gated Full Atten (Goo et al., 2018) [15]	97.0	88.8	75.5	93.6	94.8	82.2
Slot-Gated Intent Atten (Goo et al., 2018) [15]	96.8	88.3	74.6	94.1	95.2	82.6
Self-Attentive Model (Li et al., 2018) [16]	97.5	90.0	81.0	96.8	95.1	82.2
Bi-Model (Wang et al., 2018) [17]	97.2	93.5	83.8	96.4	95.5	85.7
CAPSULE-NLU (Zhang et al., 2019) [18]	97.3	91.8	80.9	95.0	95.2	83.4
SF-ID Network (Haihong et al., 2019) [7]	97.0	90.5	78.4	96.6	95.6	86.0
Stack-propagation (Qin et al., 2019) [6]	98.0	94.2	86.9	96.9	95.9	86.5
BERT-baseline(Chen et al., 2019) [22]	98.1	96.3	90.4	97.1	85.9	87.9
Stack-propagation + BERT (Qin et al., 2019) [6]	98.6	96.7	91.8	97.3	96.1	88.2
Our model	98.9	96.7	92.2	98.1	96.2	88.7

shows the experimental results of the proposed models on the SNIPS and ATIS datasets. It can be seen that our approach achieves state-of-the-art performance in the three evaluation metrics of both datasets. To prove that the improvement of our approach does not depend on using BERT, we used the same pre-training parameters to reimplement two other models using BERT as the coding layer. From the table, we can see that compared with the best prior stack-propagation + BERT, in the SNIPS dataset, we obtained a 0.3% improvement on intent (Acc), the same performance on slot (F1) score, and 0.4% improvement on sentence (Acc). In the ATIS dataset, we obtained 0.8% improvement on intent (Acc), 0.1% improvement on slot (F1) score and 0.4% improvement on sentence (Acc).

In addition, from Table 4, we can see that our model performs better on intent-detection tasks due to the role of typed abstraction mechanisms. The performance of the SF task proves that our typed iteration approach avoids the impact of error propagation on the SF task. The result also verifies our assumptions that part of the slot encoded information can mislead the intent-detection task. If this information can be eliminated, the S2I method is also effective.

4.5. Ablation Experiment

In this section, we prove the effectiveness of each part of our method by comparing adding separate modules to the BERT-baseline method, as shown in

Table 5. Results of the ablation experiment.

Dataset	SNIPS			ATIS
	Intent (acc)	Slot (F1)	Sentence (acc)	Intent (acc)	Slot (F1)	Sentence (acc)
BERT-baseline	98.1	96.3	90.4	97.1	95.9	87.9
+Typed Abstraction(TA)	98.4	96.3	90.6	97.3	95.8	88.0
+Feature Fusion(FF)	98.0	96.1	89.9	97.1	95.4	87.5
+TA&FF(M = 1)	98.4	96.5	91.1	97.5	95.9	88.1
M = 3	98.7	96.5	91.5	97.8	95.9	88.4
M = 5	98.9	96.6	92.2	98.1	96.1	88.6
FF by Mean Pooling	98.6	96.4	91.8	97.8	95.9	88.4
FF by Max Pooling	98.6	96.5	92.0	97.7	95.6	86.1
FF by Attention	98.8	96.7	92.2	98.1	96.1	88.7
Our model	98.9	96.7	92.2	98.1	96.2	88.7

.

The first part of the table shows the effect of adding two interactive methods separately. It can be seen that using only the type abstraction mechanism will slightly increase the intent-detection task, but using the feature fusion mechanism alone will not improve the performance. Using two information interaction methods simultaneously can achieve performance improvement on both subtasks. In the second part, we compared the performance of the approach under different maximum iterations. It can be seen that different maximum iteration numbers will affect the performance of the approach. The selection of the maximum number of iterations will be further described below. In the third part, we compared the effect of different feature fusion mechanisms on performance. It can be seen that the pooling method is worse than the attention mechanism and concatenate method, and the performance gap between the attention mechanism and the concatenate method is not significant. However, the attention mechanism will take more time and memory space, so we chose the concatenate method to achieve feature fusion.

5. Analysis

5.1. Selection of the Maximum Iteration Number

When choosing the maximum iteration number, we do not treat it as a hyperparameter and determine it by tuning the parameters on the validation set. We believe that a correct maximum iteration number M should ensure that as much data as possible can reach convergence within M times.

Table 6. Details of different maximum iterations.

Max Iteration Number M	$\mathit{t} \le \mathit{M}$	$\mathit{t} > \mathit{M}$	Time for Training One Epoch
M = 3	72.49%	27.51%	2 m 04 s
M = 4	91.94%	7.06%	3 m 11 s
M = 5	92.33%	7.06%	3 m 58 s

shows the details of the training set on the ATIS dataset when selecting different maximum iteration numbers. It can be seen that when the maximum iteration number M = 3, 27.51% of the data still fails to converge, indicating that the iteration process is not complete at this time; when M = 4, it can ensure that most of the data reach convergence; when M = 5, the convergent data growth is not apparent, and the performance is not improved, but the time spent training will increase significantly. Therefore, the maximum iteration number should be performed in multiple experiments on datasets and determined by observing the data convergence on the training set.

5.2. Effect of Typed Abstraction Mechanism

To prove the effectiveness of the typed abstraction mechanism, we used a set of heatmaps for comparison and illustration, as shown in Figure 3. We used our model and the BERT-baseline model for comparison and used these two models to predict two contrasting sentences that have the same intent but different slot words. We drew the heat map by the last level of the attention distribution matrix in the intent-detection task.

From Figure 3a,b, it can be seen that although the meaning of the two sentences is the same, the attention distribution of the BERT-baseline method is not the same, and it can be seen that the model treats the date as an essential feature, but many sentences with a different intent in the ATIS dataset contain date features. This proves that the implicit joint model will choose some wrong slot types as the classification features, which will affect the model’s performance. Figure 3c is a heat map obtained when using our typed iterative approach to predict the comparative sentences. In our approach, two sentences have the same encoding information in the intent-detection task, and it can be seen that dates and location names are no longer essential features for the intent-detection task. Through the comparison of the heat map, we can see that our approach can filter the impact of the specific meaning of slot words on the intent-detection task and help the model choose the accurate features.

5.3. Case Study

Figure 4 shows the results of the two models on an artificially constructed sentence to compare and illustrate how the typed iteration approach can correct errors through the iterative process. We made two virtual location names, loc_A and loc_B, to simulate the model’s performance when OOV words appear. It can be seen that although the BERT-baseline model can correctly identify the slot labels and intent, the result of time slots produces errors. Our method also makes errors during the first iteration, but it was gradually revised through the iterative process and finally obtained the correct results. This proves the iterative process’s necessity and shows that the typed iterative approach is effective when dealing with OOV words.

6. Conclusions

This paper proposes a typed abstraction mechanism that can explicitly enhance the intent-detection task by the encoded information of the SF task. We also designed a typed iterative approach based on the pre-trained language model BERT. The joint model constructed by this approach can achieve a bidirectional connection of the encoded information of the two subtasks and mitigate the negative effects of error propagation through iteration. We performed experiments on two public datasets, ATIS and SNIPS, and obtained state-of-the-art performance. In addition, through experiments and analysis, we verified the effectiveness of the typed abstraction mechanism and the typed iteration approach.