Margin and Shared Proxies: Advanced Proxy Anchor Loss for Out-of-Domain Intent Classification

Abstract

Out-of-Domain (OOD) intent classification is an important task for a dialog system, as it allows for appropriate responses to be generated. Previous studies aiming to solve the OOD intent classification task have generally adopted metric learning methods to generate decision boundaries in the embedding space. However, these existing methods struggle to capture the high-dimensional semantic features of data, as they learn decision boundary using scalar distances. They also use generated OOD samples for learning. However, such OOD samples are biased, and they cannot include all real-world OOD intents, thus representing a limitation. In the current paper, we attempt to overcome these challenges by using Advanced Proxy-Anchor loss, which introduces a margin proxy and shared proxy. First, to generate a decision boundary that has the high-dimensional semantic features of training data, we use a margin proxy for learnable embedding vectors. Next, the shared proxy, which is shared by all In-Domain (IND) samples, is introduced to make it possible to learn the discriminative feature between IND intents and OOD intent, ultimately leading to the improved classification of OOD samples. We conduct evaluations of the proposed method using three benchmark datasets. The experimental results demonstrate that our method achieved an improved performance compared to the methods described in previous studies.

1. Introduction

Recently, research examining conversation systems has been actively conducted, which has made the accurate classification of user utterance intent increasingly important [1,2,3,4]. In particular, a task-oriented dialog system classifies the utterances that are input by the user into pre-defined intents; this is an essential function for a conversation process [5,6,7]. For example, as shown in Figure 1, a banking system classifies each user utterances as a banking intention such as a deposit, withdrawal, and balance check, and then proceeds with the follow-up task, accordingly. However, even if a user inputs an utterance with an intention that is difficult to classify as one of the pre-defined intents (e.g., asking a banking chatbot to make a hotel reservation) is entered, it is necessarily classified as one of the pre-defined banking words. As mentioned above, intents that are not pre-defined in the dialog system are referred to as an OOD intent. It is essential to use an OOD intent classification technique for a dialog system to prepare for the situation when a user inputs OOD intent utterances [8].

To solve this problem, many researchers regard the OOD intent classification task as a (k + 1)-class classification task. For example, Scheirer et al. [9] propose the concept of open space risk as an evaluation metric for OOD classification. Intuitively, open space risk means that data that have not been experienced during the training process can be located anywhere in the embedding space. Zhang et al. [10] propose an Adaptive Decision Boundary (ADB) method that optimizes the decision boundary. To reduce the open space risk, that method limits the space in which OOD samples can exist inside the decision boundary by making the intra-class variance small and the inter-classes variance large. However, since the OOD intent can be located anywhere in the embedding space, this method is vulnerable in cases where OOD samples are embedded inside the In-Domain (IND) decision boundary. Moreover, the decision boundaries generated through ADB face difficulty when attempting to learning the high-dimensional semantic feature of data embeddings. To solve this problem, Zhou et al. [11] propose the KNN-Contrastive learning method. This method is designed to avoid the risk of OOD samples being located inside the IND decision boundary. However, it experiences instability when sampling random k neighbors. Lang et al. [12] generate OOD samples by utilizing a self-supervised learning method and propose an Adaptive Soft pseudo labeling (ASoul) method that labels each data with a soft labeling technique rather than one hot encoding. However, the OOD samples generated in this way are biased, and there is a limitation in that they cannot include all real-world OOD intents [11,13,14].

In this paper, we propose Advanced Proxy-Anchor loss, which captures the high-dimensional semantic features of data and learns the discriminative features between OOD intents and IND intents without needing to use OOD intent samples in training. Our method performs an OOD intent classification task by introducing a margin proxy and shared proxy to the Proxy-Anchor loss [15], which is an existing distance-based classification approach. The proxy, which is a concept that was introduced by Movshovitz-Attias et al. [16] in the metric learning approach, is an embedding vector that learns the distances from samples and represents each class of the training dataset.

In previous studies [10,17], the decision boundary has been created by directly calculating the scalar distance from the center of the IND intents when learning the decision boundary. However, since these methods only learn whether each sample is located inside the decision boundary, it is difficult for them to capture high-dimensional features. To overcome this limitation, we utilize the margin proxy in the proposed model. The margin proxy is an embedding vector that determines the decision boundary and learns a high-dimensional semantic distance by adaptively adjusting the distance to the centroid proxy as well as the distance to the positive sample. Margin proxies are created in an one-to-one correspondence to the centroid proxies representing each IND intent (like the moon orbiting the earth).

Moreover, to learn the discriminative features between IND intents and the OOD intent, we introduce a shared proxy that represents the class to which all IND samples belong. Existing methods learn using single-label samples, but our method utilizes a simple multi-label method in which each sample belongs to both the positive class and the shared class. IND samples are learned to reduce the distance from the positive IND centroid proxy and shared proxy, and OOD samples can be clearly classified in the embedding space, as only the distance from the shared proxy is reduced. The proposed method does not require the use of any additional classifiers or any changes to the model structure.

In this paper, through experiments using three intent classification benchmark datasets, we confirmed that the proposed method achieves an improved performance compared to previous works. The contributions of our study can be summarized as follows:

• We propose an Advanced Proxy-Anchor loss to conduct the OOD intent classification task.
• Our method, which introduces new concepts of the margin proxy and shared proxy, effectively captures high-dimensional features to determine decision boundaries and learns discriminative features between IND intents and the OOD intent.
• The results of experiments performed on three benchmark datasets demonstrate that our method achieves significant improvements compared to baseline outcomes.

2. Related Work

2.1. OOD Detection

Many recent works have attempted to solve the problem of OOD detection. For example, MSP [18] uses the maximum Softmax probability to detect OOD intents. GOT [19] detects OOD intents by utilizing energy scores aligned with the density of inputs. SCL [20] and LMCL [21] use Supervised Contrastive Learning and margin loss to minimize intra-class variance and maximize inter-classes variance, respectively. ADB [10] detects OOD by learning the adaptive decision boundaries of IND intents and ODD intent. Outlier [22] generates OOD samples using a pre-trained model to learn the difference between IND and OOD. ODIST [23] creates pseudo OOD samples using pre-trained language models that are included in the learning stage. KNN-Contrastive [11] uses KNN-contrastive learning to extract discriminative semantic features that can aid the OOD classification. ASoul [12] performs (k + 1) classification using the semi-supervised learning method that uses the adapted soft pseudo label (ASoul) for the ODD intent classification. That method also uses a feature generator to generate pseudo OOD samples to be used for training.

2.2. Proxy-Anchor Loss

Proxy-Anchor loss is a method proposed by Kim et al. [15], and it is one of the proxy-based metric learning methods [16,24] that introduce proxy, a learnable parameter. It is mainly used for image retrieval, face recognition, and few-shot learning tasks [25]. This method creates as many proxies as needed to match the number of classes in the training dataset. In the embedding space, each proxy is trained to have a short distance from positive samples and a long distance from negative samples, thus allowing the model to classify effectively. Proxy-Anchor loss can overcome the disadvantages of previous proxy-based methods [26,27,28,29] because it can effectively learn the semantic relationship between each data point by learning the relationship between proxies and all data in a batch. Moreover, since it does not involve a pair sampling process, the computational complexity is low, and the problem of the pair-based method [30,31], where the learning process can be hindered when too-easy negative data are sampled (i.e., when the loss easily becomes 0), is solved.

3. Proposed Method

To improve the OOD classification performance, we propose Advanced Proxy-Anchor loss, which captures high-dimensional semantic features to determine decision boundaries and then learns the discriminative features between IND intents and the OOD intent. The proposed method achieves the above two goals by introducing new concepts into Proxy-Anchor loss [15], margin proxy and shared proxy, which are both learnable embedding vectors. The margin proxy learns not only the distance from the center of the IND intents, but also the distance from the samples to generate a decision boundary. The proposed method also has the ability to effectively distinguish between IND intents and OOD intents through a multi-label method using a shared proxy.

This section first describes the method used to create an intent representation. Next, after the reviewing Proxy-Anchor loss [15], the basis of the proposed method—Advanced Proxy-Anchor loss—is explained. Finally, we present the method we use to classify intents in the embedding space based on the generated decision boundaries.

3.1. Intent Representation

To extract the intent representation of each sample, we use BERT [32], a pre-trained language model. While following the method deployed by Lin et al. [33], we obtain the embedding vector $e_i=[\left[CLS\right],T_1,\dots ,T_N]$ of the i-th IND sentence through BERT. We then perform mean-pooling to obtain intent representation $x_i$ as expressed in the following equation:

$$x_i=\mathit{Mean}\text{-}\mathit{pooling}\left([\left[CLS\right],T_1,\dots ,T_N]\right),$$

(1)

where $x_i\in {\mathcal{R}}^H$, N is the sequence length and H is the hidden size of BERT.

3.2. Classic Proxy-Anchor Loss

Proxy-Anchor loss randomly initializes one proxy for each class and uses the created proxy as an anchor to learn the distance to all samples within a batch. The equation of Proxy-Anchor loss is as follows:

$$\mathcal{L}=\frac{1}{|P^{+}|}\sum _{p\in P^{+}}log\left(1+\sum _{x\in X_p^{+}}{e}^{-\alpha \left(s\right(x,p)-\delta )}\right)+\frac{1}{\left|P\right|}\sum _{p\in P}log\left(1+\sum _{x\in X_p^{-}}{e}^{\alpha \left(s\right(x,p)+\delta )}\right),$$

(2)

where $\delta >0$ and $\alpha >0$ are hyperparameters that respectively refer to the margin and scaling factor. P is a set of all proxies, and $P^{+}$ is a set of the positive proxies of the samples in a batch. Moreover, X is a set of the embedding vectors of all samples in a batch, $X_p^{+}$ is a set of the positive samples of each proxy p, and $X_p^{-}=X-X_p^{+}$. The similarity function s denotes the cosine similarity function.

The first term of Proxy-Anchor loss, which consists of two terms in total, intuitively brings the distance of positive samples closer to each proxy in the embedding space, while the second term makes negative samples further away. Here, based on the calculation of $s(x,p)-\delta$, the cosine similarity between the positive sample and the proxy is made larger than the margin (i.e., the sample is made to be located inside the decision boundary). The margin of Proxy-Anchor loss is set directly by the researcher to be the same for all proxies, which has limitations in terms of reflecting the specific characteristics of each IND intents.

3.3. Advanced Proxy-Anchor Loss

As implied by the name, Advanced Proxy-Anchor loss is designed to overcome the limitations of previous OOD intent classification studies by achieving advancements over Proxy-Anchor loss [15]. Our proposed method utilizes three types of proxies: a centroid proxy, a margin proxy, and a shared proxy.

Centroid proxy p is a proxy representing each class, as is used in the Proxy-Anchor loss [15]. $p_k\in {\mathcal{R}}^H$ is an embedding vector that can be learned and which represents the centroid proxy of the k-th IND intent.

Margin proxy $m_k\in {\mathcal{R}}^H$ determines the decision boundary by calculating the distance to the corresponding centroid proxy $p_k$. The concept of the margin also exists in Proxy-Anchor loss. However, since the margin of the proxy-anchor loss is a hyperparameter, the researchers set the margin, and there is a limitation in that several experiments must be conducted to find the optimal value [17]. Moreover, there have been various studies aiming to learn the margin that determines the decision boundary in the OOD Detection task [10,34,35], but since the low-dimensional distance itself is set as a learning parameter, these approaches have been limited in terms of their ability to capture the features of the high-dimensional embedding vector. Since each margin proxy of our method is a vector of the same dimension as the data embedding, our method captures the semantic features of the sample more effectively than previous studies.

Only one shared proxy exists in the embedding space, and one margin proxy is assigned the same as the centroid proxy. The shared proxy $p_{shared}\in {\mathcal{R}}^H$ is treated as a positive proxy for all IND samples in the training process. That is, all samples are in a multi-label state belonging to both the original centroid proxy and the shared proxy. Chun et al. [36] use the method of intentionally overfitting the model when training data are sparse. Inspired by this method, we overfit the model so that all samples are located inside the decision boundary of the shared proxy. However, since the model is trained using multi-label samples, it is possible to overfit with a shared proxy and learn the semantic features of the intent to which each sample originally belongs. In other words, the model is weakly overfitted. With this intentional overfitting method, IND samples are located inside the decision boundaries of both the shared proxy and the original proxy. Further, since OOD samples do not include the semantic features of IND intents, they are only located only inside the decision boundary of the shared proxy.

Advanced Proxy-Anchor loss is formulated as follows:

$$\mathcal{L}=\frac{1}{|P^{+}|}\sum _{p\in P^{+}}log\left(1+\sum _{x\in X_p^{+}}{e}^{\alpha M_{pos}}\right)+\frac{1}{\left|P\right|}\sum _{p\in P}log\left(1+\sum _{x\in X_p^{-}}{e}^{\alpha M_{neg}}\right),$$

(3)

where the notation of each variable is the same as that of the Proxy-Anchor loss, and $M_{pos}$ and $M_{neg}$ are, respectively, defined as follows:

$$M_{pos}=\left\{\begin{array}{cc}-s(x,m),\hfill & \hfill \mathrm{if}d>0\hfill \\ -d,\hfill & \hfill \mathrm{if}d\le 0,\hfill \end{array}\right.$$

(4)

$$M_{neg}=\left\{\begin{array}{cc}d,\hfill & \hfill \mathrm{if}d>0\hfill \\ s(x,m),\hfill & \hfill \mathrm{if}d\le 0,\hfill \end{array}\right.$$

(5)

where m is a margin proxy corresponding to the centroid proxy p. The d is defined as:

$$d=s(x,p)-s(m,p),$$

(6)

where d is the difference of the distance between the sample and the centroid proxy and the distance between the sample and the margin proxy. This value determines whether the sample is located inside the decision boundary. Advanced Proxy-Anchor loss not only makes it so that each sample is located inside the decision boundary of the positive proxy, but it also limits the space in which OOD samples can be located inside by making it smaller. Here, the decision boundary is made small but still large enough to contain the samples inside. We divide each sample into intra-samples and inter-samples according to whether the sample exists inside each decision boundary in the embedding space. Figure 2 shows the process used to train inter-samples and intra-samples in our proposed method.

First, since the positive intra-sample ($d>0$ in Equation (4)) is already inside the decision boundary, it does not need to be moved further. In this case, the space in which OOD samples can exist is limited, as the distance between the margin proxy and samples has been made small but is still large enough to include the intra-sample. By contrast, the negative intra-sample ($d>0$ in Equation (5)) is in the wrong location and should be moved outside the decision boundary. Thus, Advanced Proxy-Anchor loss makes the distance between the sample and the centroid proxy greater than the distance between the sample and margin proxy.

We now describe the inter-sample case where each sample exists outside the positive or negative decision boundary. Since the positive inter-sample ($d\le 0$ in Equation (4)) is outside the decision boundary, it is desirable for the sample to move inside the positive decision boundary. Therefore, the distance between the sample and the centroid proxy should be closer than the distance between the sample and the margin proxy. In contrast, the negative inter-sample ($d\le 0$ in Equation (5)) already exists outside the decision boundary. In this case, we increase the distance between the sample and margin proxy to ensure that each decision boundary is distinct. If all the mentioned equations exist separately, the decision boundary becomes extremely large or small; thus, the OOD intent classification cannot be performed properly. However, in our method, since all equations are computed together, it is possible to obtain an appropriate decision boundary.

As shown in the gray decision boundary in Figure 2b, the decision boundary of the shared proxy is positive for all samples, so it becomes large enough to include samples of all embedding spaces at the end of the training. As mentioned earlier, since all IND samples are in a multi-label state that is positive for each centroid proxy and shared proxy, the two decision boundaries overlap with each other. However, since the embedding vector of the OOD sample does not have the features of the IND intent, it exists outside the decision boundary of all IND proxies, and it is only located inside the decision boundary of the shared proxy or outside all decision boundaries. Since the embedding space is created as detailed above, our proposed method is effective in learning the discriminative features between IND intents and the OOD intent as well as distinguishing them in the embedding space.

3.4. Classification with Decision Boundaries

As described in Section 3.3, after training the decision boundary through our proposed method, the classification was finally performed. The classification process to obtain the predicted intent $\widehat{y}$ for each sample follows the equation:

$$\widehat{y}=\left\{\begin{array}{cc}argma{x}_{k\in Y}s(x,p),\hfill & \mathrm{if}d>0\hfill \\ OOD,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(7)

where Y is the entire set of IND intents. If the sample is located inside the decision boundary of intent k, it is classified as intent k. If the sample exists in the space where the decision boundaries of two intents overlap, it is predicted as the intent of the closer centroid proxy. If it is not located inside any of the decision boundaries, it is predicted as an OOD intent.

4. Experiments

4.1. Datasets

To verify the proposed method, we conduct experiments on the following three benchmark datasets: CLINC [37] is a dataset created for intent classification which includes 150 intents. It includes 10 domains in total, including banking, airlines, hotels, credit cards, insurance, loans, and restaurants. We follow the settings used by Lang et al. [12]. StackOverflow [38] is a dataset published on Kaggle.com. It consists of 20 technical question intents. BANKING [39] is a dataset made up of the banking domain. It consists of 77 intents. The detailed statistics are presented in

Table 1. Statistics of CLINC, StackOverflow(SO), and BANKING datasets.

Dataset	Train	Valid	Test	Intent
CLINC	15,000	3000	5700	150
SO	12,000	2000	6000	20
BANKING	9003	1000	3080	77

.

4.2. Baselines

We compare Advanced Proxy-Anchor loss with the following previously reported OOD intent classification methods described in Section 2.1: MSP [18], DOC [40], OpenMax [41], LMCL [21], ADB [10], Outlier [22], SCL [20], GOT [19], ODIST [23] and ASoul [12].

4.3. Evaluation Metrics

Following many prior studies, we employ four metrics [10,11,12]. First, we utilize the accuracy and F1-score of all IND and OOD intent to check the overall performance of the trained model (Acc-ALL, F1-ALL). Next, to separately check the performance of the model for IND intents’ classification and OOD intent classification, we use the average F1-score of all IND intents (F1-IND) and the F1-score of the OOD intent (F1-OOD). All F1-scores are macro average F1-scores.

4.4. Experimental Settings

Following the settings used in the previous study [10,11,12], we randomly sampled only 25%, 50%, and 75% of the IND intents and used them for training. Then, we regarded the remaining unsampled intents as OOD, which are used with sampled intents for validation and testing. We selected the model with the highest F1-ALL performance in the validation set and conducted the test accordingly.

We utilized the BERT-base model (bert-base-uncased) to generate the intent representation. The max sequence length of BERT is 50, the hidden size is 768, and the scaling factor $\alpha$ is 64. Moreover, we adapt the AdamW Optimizer, and we set the learning rate to 2 $\times {10}^{-5}$ for each sample, 2 $\times {10}^{-4}$ for margin proxies, and 2 $\times {10}^{-3}$ for centroid and shared proxies. The results of all experiments are expressed as the average of 10 repeated experiments by applying different random seeds in the intent sampling process.

4.5. Results

Table 2. The performance of the Advanced Proxy-Anchor loss model and baselines models. The best performances are indicated in bold and the second-best performances are underlined.

		CLINC				StackOverflow				BANKING
	Methods	Acc-ALL	F1-ALL	F1-OOD	F1-IND	Acc-ALL	F1-ALL	F1-OOD	F1-IND	Acc-ALL	F1-ALL	F1-OOD	F1-IND
25%	MSP	47.02	47.62	50.88	47.53	28.67	37.85	13.03	42.82	43.67	50.09	41.43	50.55
	DOC	74.97	66.37	81.98	65.96	42.74	47.73	41.25	49.02	56.99	58.03	61.42	57.85
	OpenMax	68.50	61.99	75.76	61.62	40.28	45.98	36.41	47.89	49.94	54.14	51.32	54.28
	SCL	75.01	65.45	81.92	65.01	62.08	61.01	67.99	59.61	70.82	64.82	77.28	64.17
	GOT	72.63	64.01	72.63	64.01	65.02	62.26	65.02	62.26	63.05	63.49	63.05	63.49
	LMCL	81.43	71.16	79.45	63.60	47.84	52.05	68.58	61.00	64.21	61.36	68.61	63.22
	ADB	87.59	77.19	91.84	76.80	86.72	80.83	90.88	78.82	78.85	71.62	84.56	70.94
	Outlier	88.44	80.73	92.35	80.43	68.74	65.64	74.86	63.80	74.11	69.93	80.12	69.39
	ODIST	89.79	80.04	93.42	79.69	91.53	85.05	94.41	83.18	81.69	73.44	87.11	72.72
	FM + ASoul	92.71	84.11	95.42	83.81	92.04	86.03	94.76	84.29	87.41	78.39	91.52	77.70
	Ours	92.90	85.38	95.51	85.12	93.26	87.54	95.58	85.93	88.36	80.27	92.16	79.64
50%	MSP	62.96	70.41	57.62	70.58	52.42	63.01	23.99	66.91	59.73	71.18	41.19	71.97
	DOC	77.16	78.26	79.00	78.25	52.53	62.84	25.44	66.58	64.81	73.12	55.14	73.59
	OpenMax	80.11	80.56	81.89	80.54	60.35	68.18	45.00	70.49	65.31	74.24	54.33	74.76
	SCL	71.14	75.03	70.81	75.09	76.16	78.95	74.42	79.40	74.81	78.04	72.45	78.19
	GOT	67.06	73.15	63.48	73.28	65.56	72.19	55.53	73.86	69.97	76.37	63.03	76.72
	LMCL	83.35	82.16	85.85	82.11	58.98	68.01	43.01	70.51	72.73	77.53	69.53	77.74
	ADB	86.54	85.05	88.65	85.00	86.40	85.83	87.34	85.68	78.86	80.90	78.44	80.96
	Outlier	88.33	86.67	90.30	86.54	75.08	78.55	71.88	79.22	72.69	79.21	67.26	79.52
	ODIST	88.61	86.57	90.62	86.52	88.52	87.35	89.57	87.13	80.90	81.78	81.32	81.79
	FM + ASoul	89.96	88.20	91.72	88.15	88.92	88.24	89.69	88.09	81.98	83.96	81.65	84.03
	Ours	90.38	88.62	92.08	88.57	89.14	88.44	90.02	88.28	83.70	84.93	83.87	84.96
75%	MSP	74.07	82.38	59.08	82.59	72.17	77.95	33.96	80.88	75.89	83.60	39.23	84.36
	DOC	78.73	83.59	72.87	83.69	68.91	75.06	16.76	78.95	76.77	83.34	50.60	83.91
	OpenMax	76.80	73.16	76.35	73.13	74.42	79.78	44.87	82.11	77.45	84.07	50.85	84.64
	SCL	76.50	82.65	66.90	82.79	79.91	84.41	63.79	85.78	78.45	84.09	56.19	84.57
	GOT	72.65	81.49	54.11	81.73	77.76	81.85	52.80	83.79	77.11	83.36	48.30	83.97
	LMCL	83.71	86.23	81.15	86.27	72.33	78.28	37.59	81.00	78.52	84.31	58.54	84.75
	ADB	86.32	88.53	83.92	88.58	82.78	85.99	73.86	86.80	81.08	85.96	66.47	86.29
	Outlier	88.08	89.43	86.28	89.46	81.71	85.85	65.44	87.22	81.07	86.98	60.71	87.47
	ODIST	87.70	89.30	85.86	89.33	83.75	86.88	75.21	87.66	82.79	86.94	71.95	87.20
	FM + ASoul	89.88	91.38	88.21	91.41	85.00	87.90	75.76	88.71	84.47	88.39	72.64	88.66
	Ours	89.41	91.13	87.49	91.16	86.68	89.12	78.68	89.82	84.62	88.67	73.00	88.94

presents our experimental results. The performance of all baselines was extracted from Zhang et al. [10], Zhou et al. [11], Lang et al. [12]. Based on the results of our experiment, we can make the following conclusions: First, the performance of our method is improved significantly over ADB, which is similar to our method in that it focused on decision boundary learning. This suggests that the decision boundary created by Advanced Proxy-Anchor loss is effective for OOD intent classification; Second, our method achieves a state-of-the-art performance, surpassing the performance of ODIST and FM (Feature Mixup) + ASoul, both of which directly utilize OOD samples for training, in all settings except for one case. This indicates that our method has successfully learned the discriminative features between IND intents and OOD intents without needing to use additional data. FM + ASoul, which shows the best performance at the CLINC dataset’s known intent ratio setting of 75%, forcibly distorts the IND samples through Mixup [42] to generate OOD samples and use them for learning. However, since the above method only generates samples that are different from the existing IND samples, the data are biased, and the method is limited in terms of real-world applicability [11,13,14]; Finally, in OOD intent classification, since the size of IND samples decreases because the known intent ratio is low, it is common for the F1-IND performance to decrease compared to the setting of the high known intent ratio. However, our method is relatively robust to changes in the known intent ratio compared to baseline models, and F1-IND metrics have the largest improvement when compared to baselines. This shows that our method can better classify IND intents and OOD intent, as well as better classify between IND intents, than the baseline models.

4.6. Ablation Studies

We conduct ablation studies to verify the effectiveness of the margin proxy and the shared proxy, which are components of the Advanced Proxy-Anchor loss. Each ablation model is as follows:

• w/o Shared is a model in which the shared proxy is removed from the proposed method. That is, in this setting, all samples are learned in a single-label state, not a multi-label state;
• w/o margin is a model without the margin proxy, and it is a variant used to check the difference between setting the margin as a vector and as a scalar. This model uses the margin as a trainable scalar value, not as an embedding vector.
• w/o ALL is a setting where all the components proposed in this study are removed, and where OOD intent classification is performed with the Classic Proxy-Anchor loss setting [15].

As can be observed in

Table 3. The performances of the proposed method and ablation models. The best performances are indicated in bold, and the second-highest performances are underlined.

		CLINC				StackOverflow				BANKING
	Methods	Acc-ALL	F1-ALL	F1-OOD	F1-IND	Acc-ALL	F1-ALL	F1-OOD	F1-IND	Acc-ALL	F1-ALL	F1-OOD	F1-IND
25%	w/o Shared	87.24	78.31	91.53	77.96	89.19	81.78	92.72	79.6	76.79	72.75	82.51	72.23
	w/o Margin	83.85	29.71	90.82	28.1	93.31	87.97	95.58	86.45	82.86	48.89	89.66	46.75
	w/o ALL	91.74	83.19	94.76	82.89	74.57	70.46	80.6	68.43	79.47	71.86	85.04	71.16
	Ours	92.90	85.38	95.51	85.12	93.26	87.54	95.58	85.93	88.36	80.27	92.16	79.64
50%	w/o Shared	80.88	82.52	82.36	82.52	76.78	80.85	74.82	81.46	77.74	81.94	75.53	82.11
	w/o Margin	64.27	15.16	77.19	14.33	70.98	56.3	77.40	54.18	57.12	21.57	70.23	20.29
	w/o ALL	88.45	87.44	90.20	87.40	83.97	84.45	84.66	84.43	80.88	83.43	80.19	83.52
	Ours	90.38	88.62	92.08	88.57	89.14	88.44	90.02	88.28	83.70	84.93	83.87	84.96
75%	w/o Shared	85.35	88.89	81.35	88.95	83.35	86.86	69.66	88.01	82.54	87.72	65.27	88.1
	w/o Margin	44.03	8.38	59.45	7.92	37.56	24.26	44.45	22.91	29.73	11.04	41.23	10.52
	w/o ALL	88.01	90.50	85.33	90.55	84.76	87.37	76.63	88.09	82.43	87.93	64.78	88.33
	Ours	89.41	91.13	87.49	91.16	86.68	89.12	78.68	89.82	84.62	88.67	73.00	88.94

, our method outperforms in all settings. We can summarize the results of the ablation studies as follows: First, the w/o Margin model shows very poor results in the datasets aside from the StackOverflow dataset. This means that the w/o Margin model face difficulty in determining the decision boundary for OOD intent classification, suggesting that it is effective to initialize the margin with a high-dimensional vector. Compared to the proposed model, the w/o Shared model exhibits an overall performance degradation, and in particular, the F1-OOD has a significant decline. These results demonstrate that the newly introduced multi-label method using a shared proxy is effective in detecting OOD intent. Comparing w/o ALL and baselines through Table 2 and Table 3, we can observe that the w/o ALL model outperforms many of the baseline models. Altogether, these findings suggest that the Proxy-Anchor loss framework itself is effective in the OOD intent classification task.

4.7. Cluster Analysis

The analysis involves embedding the test set through the trained model and examining the distribution of these vectors in the embedding space. As the model was trained with distance-based functions, clusters are expected to form in the embedding space for each intent. To evaluate the quality of these clusters, a Silhouette analysis [43] and the average separation [44] of clusters are utilized. The silhouette coefficient used in the silhouette analysis ranges from $-1$ to 1, where values closer to 1 indicate more successful clustering. The separation metric, which also ranges between 0 and 1, is considered to be successful when it is close to 0.

Table 4. Clustering performances of proposed method and ablation models. The best performances are indicated in bold letters and the second-highest performances are underlined.

		CLINC		StackOverflow		BANKING
	Methods	Silhouette	Separation	Silhouette	Separation	Silhouette	Separation
25%	w/o Shared	−0.1315	0.2847	−0.1240	0.4318	−0.1682	0.3024
	w/o Margin	−0.1526	0.6618	−0.0729	0.7193	−0.1324	0.6441
	w/o ALL	−0.1395	0.6251	0.0535	0.4834	−0.1358	0.5926
	Ours	−0.1137	0.2736	0.0770	0.3132	−0.0735	0.3273
50%	w/o Shared	0.0419	0.3509	0.2494	0.4749	0.1028	0.3306
	w/o Margin	0.0471	0.6876	0.1887	0.6814	0.0152	0.6704
	w/o ALL	−0.0191	0.6457	0.2377	0.4417	−0.0290	0.5934
	Ours	0.0689	0.3096	0.2813	0.4267	0.1246	0.3277
75%	w/o Shared	0.2024	0.4327	0.4515	0.4251	0.2974	0.4289
	w/o Margin	0.2563	0.6993	0.3783	0.6556	0.2798	0.7062
	w/o ALL	0.1122	0.6502	0.3428	0.3927	0.1184	0.5968
	Ours	0.2759	0.3702	0.4748	0.4173	0.3655	0.3629

presents the silhouette coefficients and separation values for the proposed method and variant models. Bolded values mark indicate the highest performance, while underlined values mark represent the second-highest performance.

The evaluation results show that the proposed method outperforms other models in silhouette coefficients and separation metrics across most configurations. This suggests that the proposed method is effective in embedding data samples based on semantic distances. Moreover, models without margin proxies (w/o Margin) and model without all proxies (w/o ALL) exhibit significantly lower separation values compared to models utilizing margin proxies, thus indicating the effectiveness of margin proxies in separating intents. Despite the fact that the w/o Margin model shows a much lower separation than the w/o Shared model, there are cases where it surpasses it in the silhouette coefficient. This implies that shared proxies can help distinguish OOD and enhance cohesion within OOD classes.

Figure 3 visualizes the embedding vectors of the StackOverflow test set with a 25% sampling rate. The dimensionality reduction to 2D using t-SNE is applied, and each color represents an intent class (red for Cocoa, orange for Oracle, green for Hibernate, blue for Scala, purple for OSX, and brown for OOD intent).

By qualitatively examining the visualizations, it can be observed that the proposed method generates the most appropriate vectors. The w/o Shared model faces difficulties in distinguishing between OOD and IND compared to other models. It can also be confirmed that the embedding vectors belonging to one class in w/o ALL and w/o Margin that do not use the margin proxy are separated compared to the model that uses the margin proxy.

5. Conclusions

In this paper, we propose the Advanced Proxy-Anchor loss, which additionally introduces the concepts of margin proxy and shared proxy, which are high-dimensional embedding vectors, to Proxy-Anchor loss for OOD intent classification. In doing so, we overcome the problems encountered by existing studies. The prior method could not sufficiently learn the semantic features of samples because they used a low-dimensional margin. They also had to use biased OOD samples for training to learn the discriminative semantic feature between IND intents and the OOD intent. The proposed method generates a decision boundary that reflects the semantic features of samples, while previous studies did so through a margin proxy, which is a learnable high-dimensional embedding vector. Moreover, through the multi-label method using a shared proxy, it is possible to learn the discriminative features between IND intents and the OOD intent without using OOD samples in training. The results of experiments on three benchmark datasets demonstrate that our proposed method overcomes the aforementioned limitations of prior methods and achieves a SOTA performance in the OOD intent classification task. In the future, we plan to further validate our method using datasets from various domains such as DialoGLUE [45] or CoLA [46] and apply our method to various metric learning frameworks that can utilize concepts of proxy. Furthermore, our method, as a metric learning approach using proxies, will benefit from lower computational costs compared to pair-based methods like contrastive learning. Therefore, we plan to explore the superiority of the proposed method through experiments comparing its computational cost performance with various baselines.