REKP: Refined External Knowledge into Prompt-Tuning for Few-Shot Text Classification

Abstract

Text classification is a machine learning technique employed to assign a given text to predefined categories, facilitating the automatic analysis and processing of textual data. However, an important problem is that the number of new text categories is growing faster than that of human annotation data, which makes many new categories of text data lack a lot of annotation data. As a result, the conventional deep neural network is forced to over-fit, which damages the application in the real world. As a solution to this problem, academics recommend addressing data scarcity through few-shot learning. One of the efficient methods is prompt-tuning, which transforms the input text into a mask prediction problem featuring [MASK]. By utilizing descriptors, the model maps output words to labels, enabling accurate prediction. Nevertheless, the previous prompt-based adaption approaches often relied on manually produced verbalizers or a single label to represent the entire label vocabulary, which makes the mapping granularity low, resulting in words not being accurately mapped to their label. To address these issues, we propose to enhance the verbalizer and construct the refined external knowledge into a prompt-tuning (REKP) model. We employ the external knowledge bases to increase the mapping space of tagged terms and design three refinement methods to remove noise data. We conduct comprehensive experiments on four benchmark datasets, namely AG’s News, Yahoo, IMDB, and Amazon. The results demonstrate that REKP can outperform the state-of-the-art baselines in terms of Micro-F1 on knowledge-enhanced text classification. In addition, we conduct an ablation study to ascertain the functionality of each module in our model, revealing that the refinement module significantly contributes to enhancing classification accuracy.

1. Introduction

Text classification is a classic natural language processing task, and its purpose is to divide the text into preset categories, so as to better serve downstream tasks, such as information retrieval, sentinel analysis, and spam detection. Most conventional supervised learning methods heavily rely on a substantial amount of labeled data.

With the development of large language models (LLMs), pre-training language models (PLMs) have become widely adopted for text classification tasks due to their exceptional performance [1,2,3]. The PLMs can transfer the knowledge learned from other data to the target task. At present, the mainstream way is based on fine-tuning [4,5,6]. This method entails adding a classifier to the pre-trained model and adjusting the model’s parameters using a well-labeled training set. However, obtaining a large amount of accurately labeled training data can be costly in terms of both time and resources, especially when facing newly emerging categories. Consequently, a common challenge arises in scenarios where only a limited number of samples are available for certain categories, known as the few-shot scenario. The vanilla fine-tuning approach is not well-suited for tackling tasks in the few-shot scenario. This is primarily due to the fact that a limited number of samples are insufficient for optimizing the numerous parameters of PLMs, resulting in overfitting. To address this issue, recent models such as GPT-3 [7], PET [8], and other models [9,10] have put forward the method of prompt-tuning, which has narrowed the gap between the PLMs and the downstream target tasks and has provided a novel idea to imitate the effect of the few-shot scene. In essence, the goal of prompt-tuning is to reduce semantic differences and mitigate overfitting, thereby improving model performance on both the training and test sets.

The primary concept underlying prompt-tuning is the insertion of a concise textual template containing a [MASK] token into the original input sentence. This transformation converts the label prediction task into a [MASK] filling problem. A crucial aspect of prompt-tuning involves mapping the prediction probabilities of words to their respective labels. It is essential to establish a verbalizer capable of generating a thesaurus and corresponding labels to facilitate this process. Previous works mainly use manual verbalizers [7,8]. For instance, the word “mathematics” corresponds with the category “MATHEMATICS” and typically, a category is denoted by a tag word. However, we contend that such an approach is flawed. Specifically, words such as “formula” and “numeracy” should also be classified under the category “MATHEMATICS”, as they possess similar semantic relationships. Moreover, the prediction accuracy is significantly influenced by human subjectivity. Recently, some researchers have attempted to learn tag words through the discrete clustering method [9]. However, this approach lacks human prior knowledge and often falls short of producing accurate word categories. It typically only generates words with similar semantics or vectors in the embedding space. To address these issues, we constructed the refined external knowledge into a prompt-tuning (REKP) model. To be more precise, the proposed model introduces external knowledge to assist in prompt learning, builds a verbalizer, enhances the tag thesaurus, and refines them based on specific tasks, thereby enabling improved adaptation to target tasks.

To evaluate the efficacy of our proposition, we conducted comprehensive experiments in two application scenarios: the Ag’news and Yahoo topic categorization datasets [11], as well as the IMDB and Amazon sentiment categorization datasets. The experimental results clearly illustrate that our scheme outperforms the competitive baseline, showing a significant improvement. Furthermore, the ablation study confirms the effectiveness of each module in our proposal, with the results highlighting the crucial role of the refinement module in enhancing classification accuracy.

We summarize the main contributions in this paper as follows:

• We introduce external knowledge to enhance the verbalizer by expanding related words, which improves the granularity of the tag thesaurus and enriches the mapping range of tags.
• We propose three methods to refine the enhanced verbalizer and remove the noise words to optimize the model.
• Extensive experiments conducted on four benchmark datasets, namely AG’s News, Yahoo, IMDB, and Amazon, demonstrate the superiority of REKP over the competitive baselines in terms of Micro-F1 on knowledge-enhanced text classification tasks.

The rest of the paper is organized as follows: Section 2 presents the related work. Section 3 presents the overall framework of our method and the implementation details of the REKP model. Section 4 describes the research questions, evaluation metrics, and baselines. Section 5 presents and analyzes the experimental results of our system on the AG’s News, Yahoo, IMDB, and Amazon benchmark datasets. Section 6 gives the summary and outlook for this work.

2. Related Work

In this section, we first review previous work on few-shot text classification in Section 2.1, and subsequently provide a summary of the related work on prompt tuning for few-shot learning in Section 2.2.

2.1. Few-Shot Text Classification

Processing text classification problems with a small number of labeled samples is the topic of few-shot text classification. The primary objective is to enhance the generalization capability of pre-trained models for rapid adaptation to new tasks [12,13,14] while acquiring sufficient prior knowledge. The three prevailing techniques employed for few-shot text categorization include data augmentation, transfer learning, and meta-learning.

Data augmentation is a straightforward concept that directly addresses the issue of insufficient training samples in tasks with small sample sizes, which can lead to overfitting. The diversity of data structures is increased by the use of data augmentation. In our research, we classify data enhancement technologies into three categories according to the direction of enhancing features: those utilizing unlabeled data [15,16,17], those employing data synthesis [18,19,20], and those focusing on feature enhancement [21]. However, increasing the generalization capabilities of few-shot learning models through data augmentation still presents challenges as the semantics of the synthesized training samples closely resemble those of the original samples. Typically, transfer learning pre-trains encoders in large-scale corpora and then immediately apply the taught encoders to a few-shot tasks [1]. Nakamura et al. [3] use the Universal Language Model Fine-tuning [22] where parameters are gradually adjusted by modifying the learning rate, and set different learning rates for each layer of the network, continuously learning and adjusting the learning strategy to obtain the best results. However, due to the small sample data of each category in the training set, the global fine-tuning method may destroy the information learned by the feature extractor, resulting in overfitting. To solve this problem, Chen [23] proposed the ContrastNet for few-shot text classification, which learned differentiated text representations belonging to different categories through supervised contrast learning, and introduced unsupervised contrast regularization at task and instance levels to prevent overfitting. Meta-learning is a cutting-edge field in machine learning, which aims to endow machines with the ability to learn in a manner analogous to human learning processes [24]. Meta-learning can be divided into optimization-based methods and metric-based methods. A notable example of an optimization-based model is MAML [25]. This approach involves learning an appropriate initialization of model parameters from a set of base classes and transferring these parameters to new classes using a few gradient steps. Hong [26] recently created a dictionary of meta-level attention characteristics and identified the top k most pertinent attention elements for few-shot learning using pre-trained models. On the other hand, metric-based meta-learning approaches determine the category of a query by assessing the dissimilarity between the query vector and the prototype vector of each class. One example is the prototypical network [27], which computes the prototypes by averaging the support sets and determines the category by calculating the Euclidean distance between the query and each prototype. Gao et al. [28] introduced a prototype network framework with a hybrid attention mechanism, which effectively accomplishes denoising and mitigates the issue of feature sparsity.

Despite the significant progress made in the field of few-shot text classification, there still exists a notable gap compared to human classification accuracy. Data augmentation techniques, while commonly used, are susceptible to introducing noisy data. Transfer learning methods rely on strong correlations between the old and new tasks. Meta-learning approaches, although promising, often suffer from higher complexity and limited knowledge acquisition when only a few samples are available.

2.2. Prompt Tuning for Few-Shot Learning

Although PLMs [29,30] have shown remarkable success in various natural language processing tasks, closing the gap between pre-training and target tasks using few-shot learning remains challenging. However, the concept of “in-context learning” has emerged since the introduction of GPT-3 [7]. By incorporating timely adjustments and text-free learning, large-scale language models have demonstrated excellent performance in few-shot scenarios. Furthermore, the utilization of prompt templates to activate model knowledge has gained significant attention. Subsequently, researchers have developed a series of prompt-tuning approaches, such as text classification [28], natural language understanding [31], relation extraction [32], and entity typing [33]. These works have demonstrated that even small-scale language models [34] can achieve impressive performance through prompt adjustments.

As mentioned in Section 1, the performance of prompt-tuning is strongly influenced by the verbalizer, an essential component in the prompt-tuning process. For specific tasks, artificially constructed verbalizer have been widely used and have proven to be effective [8], which makes use of prompt learning to annotate a sizable dataset that is unlabeled for upcoming training. However, this approach requires significant manual labor and relies heavily on expert knowledge, limiting its ability to fully exploit the deep semantics of the language model. To address this limitation, a search-based verbalizer device [35] has been proposed, which is a model that does not use the manual definition of class names. By choosing the words with the most information in the PLMs vocabulary as tag words. However, due to the limited amount of data, it is difficult to find suitable words. The other direction is to build a soft verbalizer [10], using continuous vectors to represent classes, and multiplying the output of Masked Language Model (MLM) with class vectors to obtain the prediction probability of each class. However, the effectiveness of the soft verbalizer heavily relies on abundant data for optimal performance, which may not be feasible in few-shot learning scenarios.

Unlike previous approaches, we propose the incorporation of external knowledge to enhance the verbalizer through associative extensions. This refinement allows for an improved granularity in the tag lexicon and expands the mapping range of tags, thereby enhancing its adaptability in few-shot learning scenarios.

3. Proposed Approach

In this section, we describe the overall REKP framework in Section 3.1. In particular, we introduce the template conversion module and detail the verbalizer construction module in Section 3.2 and Section 3.3, respectively. Finally, we discuss the process of verbalizer utilization in Section 3.4.

3.1. Overall Framework

The proposed framework in Figure 1 consists of three essential components: template conversion, verbalizer construction, and verbalizer utilization.

Firstly, the input sequence $\mathbf{x}=(x_0,x_1,\cdots ,x_n)$ is transformed into a template $\mathcal{T}$, which is a natural language text. This conversion results in the formation of a sentence with a [MASK] token. Subsequently, the entire sentence is passed as input to a masked language model (RoBERTa [30]) in order to predict the tag words $v\in \mathcal{V}$.

During the process of converting tag words into tags, we employ open-source external knowledge to construct a tag thesaurus for each tag. This thesaurus consists of synonyms with varying levels of granularity. However, directly incorporating the external knowledge base can introduce noise into the target task. To address this, we propose three refining approaches in this paper: Label Word Refinement, Correlation Refinement, and Importance Refinement. These approaches optimize the tag thesaurus specifically for the target task. Finally, to obtain the prediction score of the tag, the weighted average of the tag words is used to get the prediction score of the tag.

3.2. Template Conversion

The template, denoted as $\mathcal{T}$, is a concise sentence with the [MASK] token. Currently, there exist five established methods for constructing templates, each with distinct characteristics. Further details regarding these methods are presented in

Table 1. Existing template construction methods.

Category	Method	Introduction
Discrepancy	Manual Template	Manual template design based on the nature of specific tasks and prior knowledge.
	Heuristic-based Template	Build templates by heuristic search, etc.
	Generation	According to the given training data with few samples, an appropriate template is generated.
Continuity	Word Embedding	Initialize discrete templates and train them by gradient descent method.
	Pseudo Token	Use the template as a trainable parameter.

.

Among them, the first three methods are referred to as Hard Templates. In these methods, the text is directly combined with a short sentence containing [MASK]. Furthermore, the Word Embedding of discrete characters remains unaltered throughout the training process. The latter two methods are called Soft Template, which is able to dynamically adjust its parameters based on the contextual semantics of the text and the objectives of the task throughout the training process.

Typically, a sentence is equipped with a single Template. However, constructing multiple templates can yield diverse samples, which effectively serves as a form of disguised data augmentation. The Manual Template, owing to its inclusion of extensive expert knowledge, has shown better performance compared to the generative template. For our experiments, we meticulously designed four templates for each dataset using manually constructed templates. Additional information regarding the prompt templates can be found in

Table 2. Prompt templates for four datasets.

Ag’news	${\mathit{T}}_1\left(\mathbf{x}\right)=$ A [MASK] news : x
	$T_2\left(\mathbf{x}\right)=$$\mathbf{x}$ This topic is about [MASK] .
	$T_3\left(\mathbf{x}\right)=$ [Category : [MASK]] $\mathbf{x}$
	$T_4\left(\mathbf{x}\right)=$[Topic : [MASK]]$\mathbf{x}$
IMDB	$T_1\left(\mathbf{x}\right)=$ A [MASK] question : $\mathbf{x}$
	$T_2\left(\mathbf{x}\right)=$ x This topic is about [MASK] .
	$T_3\left(\mathbf{x}\right)=$ [Category : [MASK]] $\mathbf{x}$
	$T_4\left(\mathbf{x}\right)=$[Topic : [MASK]]$\mathbf{x}$
Yahoo, Amazon	$T_1\left(\mathbf{x}\right)=$ It was [MASK] . $\mathbf{x}$
	$T_2\left(\mathbf{x}\right)=$ Just [mask]! $\mathbf{x}$
	$T_3\left(\mathbf{x}\right)=$$\mathbf{x}$ All in all, it was [MASK] .
	$T_4\left(\mathbf{x}\right)=$$\mathbf{x}$ In summary, it was [MASK] .

.

3.3. Verbalizer Construction

The task of predicting masked words based on contextual semantics does not resemble a multiple-choice question with a single correct answer. Instead, it involves determining the probability of predicting each word from a given set of alternatives. We should consider the following two issues when constructing the tag thesaurus of the verbalizer: the coverage should be wide and the subjective influence should be small. These requirements can be simultaneously fulfilled by leveraging the structural knowledge of external molding, as discussed in KPT [36]. To address these requirements effectively, we select relevant external knowledge sources for topic classification and emotion classification tasks, respectively.

Regarding topic classification, the primary concern lies in ensuring the comprehensiveness of the tag thesaurus and avoiding excessive granularity. To address this concern, we adopt Related Words, which is a knowledge map that aggregates information from multiple resources. This choice allows us to mitigate the aforementioned problem effectively. The algorithm employs word embedding to convert words into multi-dimensional vectors, representing their meanings. It proceeds by searching for words similar to the provided word vectors in extensive word vector libraries. Additionally, it traverses the concept network to identify words associated with the query, based on certain semantic criteria. In the experimental evaluation, we used a threshold of 0 and selected all words corresponding to the labels. For emotion classification, our goal is to expand binary emotions (positive and negative) to emotions with wider coverage and larger granularity. To accomplish this, we use the emotional dictionaries POSITIVEwords and NEGATIVEwords summarized by previous researchers to form our emotional classification tag thesaurus.

Although we have introduced the external knowledge base into our prompt-learning verbalizer, enriching it with a wide range of tag words, we acknowledge that these words might not be specifically tailored for the pre-training model. As a result, we put forward three refinement methods to save high-quality words and solve the noise problem of words in the external knowledge base.

3.3.1. Label Word Refinement (WR)

In the context of Pre-trained Language Models (PLMs), there exist unfamiliar words within the external knowledge base that can lead to significant errors in prediction. To address this challenge, we propose a method called WR to handle such unfamiliar words. Specifically, we utilize the task prior probability of tag words to filter out and remove these unfamiliar words from consideration. The visual representation of the image is illustrated in Figure 2, where the shaded area denotes the removed words.

In our approach, we consider a text classification task where we represent the sentence distribution $\mathcal{D}$ of a given sample $\mathbf{x}$. Initially, we transform each sentence into a predefined template and compute the probability $P_{\mathcal{M}}(\left[\mathrm{MASK}\right]=v|{\mathbf{x}}_t)$ for each tag word in the mask position. By estimating the overall distribution probability, we can derive the prior probability $P_{\mathcal{D}}\left(v\right)$ of the tag word in the mask position:

$$P_{\mathcal{D}}\left(v\right)={\mathbf{E}}_{\mathbf{x}\sim \mathcal{D}}{P}_{\mathcal{M}}\left(\left[\mathrm{MASK}\right]=v\left|{\mathbf{x}}_t\right.\right),$$

(1)

In the experiment, we extract a small part from the training set to form the support set S, and remove the sentence tags of the support set as the above expectation estimation. We assume that $\mathbf{x}\in S$ has a uniform prior probability, then the task prior probability of tag words can be approximately calculated as follows:

$$P_{\mathcal{D}}\left(v\right)\approx \frac{1}{\left|S\right|}\sum _{\mathbf{x}\in S}{P}_{\mathcal{M}}\left(\left[\mathrm{MASK}\right]=v\left|{\mathbf{x}}_t\right.\right).$$

(2)

According to the calculated prior probability, we remove the tag words less than the threshold. Because the distribution of task prior probability is different for each task, we do not have a predefined numerical value for the threshold in our experiments. Instead, we employ a ranking-based removal method that excludes the bottom half of the label terms based on their prior probabilities.

3.3.2. Correlation Refinement (CR)

The mapping intensity between different tag words and tags varies, with certain tag words exhibiting a stronger sense of belonging to their respective tags. Taking this into consideration, we propose CR as a solution. The architecture is illustrated in Figure 3.

Firstly, we compute the prediction probability of tag words on the support set S as the vector representation ${\mathbf{q}}_i^v$ of tag words, where ${\mathbf{x}}_{it}$ denotes the sentence composed of ${\mathbf{x}}_i$ and template t.

$${\mathbf{q}}_i^v=P_{\mathcal{M}}\left(\left[\mathrm{MASK}\right]=v\left|{\mathbf{x}}_{it}\right.\right),{\mathbf{x}}_i\in S$$

(3)

To denote the vector representation of class labels, we assume that the name of each class label is $v_0$; for instance, “MATHEMATICS” is “mathematics”. Despite the limited coverage of a single word, it exhibits the highest similarity to its corresponding label. Consequently, we employ the vector representation ${\mathbf{q}}^{v_0}$ of these words as the vector representation ${\mathbf{q}}^y$ for the class label y. In this way, we can quantify the correlation between the tag word v and the tag y, utilizing cosine similarity as the metric.

$$r\left(v,y\right)=cos\left({\mathbf{q}}^v,{\mathbf{q}}^y\right)=cos\left({\mathbf{q}}^v,{\mathbf{q}}^{v_0}\right).$$

(4)

Furthermore, we take into account the possibility of a word appearing in the tag library of multiple class tags, resulting in mapping ambiguity. For instance, the tag word “calculator” under the “MATHEMATICS” category may also be present under the “TECHNOLOGY” category. To address this issue, we propose a calculation method that assigns a high degree of relevance to a specific category of tags while maintaining a low degree of relevance to other categories of tags, effectively filtering out these multi-category words.

$$R\left(v\right)=r\left(v,f\left(v\right)\right)\frac{\left|\mathcal{Y}\right|-1}{{\displaystyle \sum _{y\in \mathcal{Y},y\ne f\left(v\right)}}r\left(v,y\right)},$$

(5)

where $f\left(v\right)$ denotes the class associated with v. Essentially, the concept of Correlation Refinement in CR bears similarity to the TF-IDF algorithm, where class tags can be viewed as documents and tag words as words.

In an ideal scenario, a reliable keyword exhibits a stronger correlation with a specific type of tag compared to other types of tags. Therefore, the tag words we have omitted have correlations with their respective categories that fall below the average correlation with other categories.

3.3.3. Importance Refinement (IR)

The tag thesaurus underwent denoising for specific tasks following WR and CR. Taking into account the varying impacts of tag words on tags, we propose an importance attention module that discerns the significance of tag words. This module aims to diminish the influence of insignificant tag word data and extract tag words that have a greater contribution to tags. The architecture is illustrated in Figure 4.

We posit that the extended tag words vary in their significance to the tags, thus necessitating the assignment of a weight $W_v$ to each tag word v in consideration.

$$w_v=sum\left\{\sigma \left(g\left(\mathbf{u}\in {\mathcal{V}}_y\right)\otimes g\left(\mathbf{y}\right)\right)\right\},$$

(6)

where, $\mathbf{u}$ represents the feature vector of the tag words, $g(·)$ denotes the linear layer responsible for the linear transformation between $\mathbf{u}$ and $\mathbf{y}$, ⊗ represents the dot product operation, and $\sigma (·)$ refers to the activation function. In this study, we employ the tanh activation function to map the dot product result to the interval $[-1,1]$. Additionally, the $sum(·)$ operation signifies the computation of the summation of vector elements ${\alpha }_j$ using the Softmax functions, which serve as the weights for the tag words.

Finally, we normalize the tag words within each tag:

$${\alpha }_v=\frac{exp\left(w_v\right)}{{\displaystyle \sum _{u\in {\mathcal{V}}_y}}exp\left(w_u\right)}.$$

(7)

By adding IR, the weight of tag words that are more similar to tags is improved. In other words, by introducing certain incorrect words into the tag words, the weight assigned to them is adjusted using this module in order to mitigate their influence on the tag.

3.4. Verbalizer Utilization

The three refinements designed in this study eliminate less probable tag words, address the issue of multiple mappings in tag words, and enhance the attention to highly important tag words. In the final estimation of tag prediction probability, this study employs the prediction score computed through a weighted average of the tag words.

$$\widehat{y}=arg\underset{y\in \mathcal{Y}}{max}\frac{exp\left(s\left(y\left|{\mathbf{x}}_{\mathrm{p}}\right.\right)\right)}{{\displaystyle \sum _{y^{\prime }}}exp\left(s\left(y^{\prime }\left|{\mathbf{x}}_{\mathrm{p}}\right.\right)\right)},$$

(8)

$$s\left(y\left|{\mathbf{x}}_{\mathrm{p}}\right.\right)=\sum _{v\in {\mathcal{V}}_y}{\alpha }_{v}log{P}_{\mathcal{M}}\left(\left[\mathrm{MASK}\right]=v\left|{\mathbf{x}}_{\mathrm{p}}\right.\right).$$

(9)

The objective function in this study employs the CrossEntropyLoss (CEL) function as the loss function, enabling the model to obtain optimal parameters during the training process and consequently improving the accuracy of tag prediction. The calculation method is as follows:

$$loss\left(\mathbf{Y},y\right)=-log\left(\frac{e^{{\mathbf{Y}}_y}}{{\sum }_i{e}^{{\mathbf{Y}}_i}}\right)=-{\mathbf{Y}}_y+log\left({\sum }_i{e}^{{\mathbf{Y}}_i}\right),$$

(10)

where $\mathbf{Y}$ is the prediction result and is the score vector $\mathbf{Y}=[y_1,y_2,\cdots y_n]$ of each category, and y represents the real category label of this sample.

4. Experiments

4.1. Research Questions

We prove the effectiveness of REKP by addressing the following four research questions:

(RQ1)

Can our proposed REKP outperform the baselines in terms of performance?

(RQ2)

How does each refinement in verbalizer contribute to REKP?

(RQ3)

How our PTEK model and baselines perform in terms of classification accuracy depends on the number of training samples?

(RQ4)

What is the particular impact of refinement on the number of label thesaurus?

4.2. Datasets and Evaluation Metrics

The effectiveness of REKP and the baseline models is evaluated on publicly available topic categorization datasets, namely, Ag’news and Yahoo [11], as well as sentiment categorization datasets, namely, IMDB and Amazon. They are the public classification datasets. AG’s News is a dataset for news categorization tasks, containing text samples and their corresponding labels. The dataset comprises over 2000 distinct news items. Yahoo is a topic-categorized dataset derived from user-generated questions on the Yahoo website. IMDB is a binary sentiment categorization dataset of movie reviews, encompassing both positive and negative sentiments. It is widely used for sentiment analysis tasks in the domain of movies. Similarly, the Amazon dataset is sourced from the Amazon website and comprises reviews and ratings aimed at anticipating consumer feedback for various products.

Accuracy, Precision, or Recall are commonly employed as evaluation metrics for binary classification tasks, but they cannot be directly applied to multiclassification problems. To address this, we adopt the Micro-F1 metric as the comprehensive evaluation criterion for all our experiments.

First, calculate the total precision $Precisio{n}_{micro}$ and recall $Recal{l}_{micro}$ for all categories.

$$Precisio{n}_{micro}=\frac{{\displaystyle \sum _{i=1}^n}T{P}_i}{{\displaystyle \sum _{i=1}^n}T{P}_i+{\displaystyle \sum _{i=1}^n}F{P}_i},$$

(11)

$$Recal{l}_{micro}=\frac{{\displaystyle \sum _{i=1}^n}T{P}_i}{{\displaystyle \sum _{i=1}^n}T{P}_i+{\displaystyle \sum _{i=1}^n}F{N}_i}.$$

(12)

The final Micro-F1 metric value is calculated using the F1 metric formula as follows:

$$F{1}_{micro}=2·\frac{Precisio{n}_{micro}·Recal{l}_{micro}}{Precisio{n}_{micro}+Recal{l}_{micro}}.$$

(13)

4.3. Baselines

We assess the effectiveness of our proposed model by comparing it with the following state-of-the-art baselines:

• FT inputs the [CLS] hidden embedding of the pre-trained model into the classification layer for label prediction.
• PT [8] is a classical model for text classification with few-shot using prompt learning. It inserts a short text template with the [MASK] token, utilizing the name of the category as an unusual label, and predicting the results of the labeled words in direct response to the category results.
• AUTO [9] is a model that automatically chooses the most informative terms from the PLM vocabulary as tag words, eliminating the need for manually established class names.
• SOFT [10] uses consecutive vectors to represent the classes, and the output of the MLM is dot-produced with the class vectors to obtain the predicted probabilities for each class. The method is carried on by REKP, which uses the class vectors that are the word embeddings of the class names.

4.4. Model Configuration

We consider four types of meta tasks, following the common practice in few-shot learning experiments [27]: “1-shot”, “5-shot”, “10-shot”, and “15-shot”, where the numbers indicate the available number of training samples. To construct the task training set and the validation set, we randomly sampled k instances from each category in the original training dataset. To mitigate experiment errors and result in randomness, we created four templates for each dataset (in Table 2) and conducted three independent runs using different random seeds. Consequently, the final score for each dataset is the average of 12 experiments.

We implement our model in PyTorch with Transformer Library. For the PLMs, we choose RoBERTa [30], which is a modified version of BERT [29]. For model training, we use the AdamW optimizer with a batch size of 16. The learning rate is 0.00003 for parameters of RoBERTa and the margin in contrastive loss is set to 1. The entire experiment is run on a Linux server with two RTX 3090 GPUs.

5. Results and Discussion

5.1. Overall Performance

To answer RQ1, we put PTEK with four other baselines on four datasets based on the Micro-F1 metrics. The overall text classification performances of all discussed models are shown in

Table 3. Overall performance in terms of Micro-F1 (%) and 95% confidence interval on the test set for four types of shot tasks. The outcomes achieved by each column’s top performance are boldfaced. The sections marked with horizontal lines represent the results obtained from the optimal baseline.

Shot	Method	AG’s News	Yahoo	IMDB	Amazon
	FT	68.6 ± 8.9	35.1 ± 6.5	75.3 ± 5.3	71.2 ± 4.7
	PT	79.7 ± 6.3	54.3 ± 5.8	90.5 ± 4.3	92.3 ± 3.4
1	AUTO	70.3 ± 7.2	45.2 ± 6.2	79.7 ± 4.9	85.6 ± 4.0
	SOFT	79.8 ± 6.1	54.6 ± 5.6	89.6 ± 4.3	91.2 ± 3.5
	REKP	83.9 ± 4.1	64.4 ± 3.5	91.1 ± 3.3	93.5 ± 3.1
	FT	70.1 ± 8.2	39.3 ± 6.3	79.6 ± 4.2	77.5 ± 3.1
	PT	82.3 ± 6.2	62.6 ± 4.2	91.1 ± 1.6	93.2 ± 1.9
5	AUTO	76.5 ± 7.2	50.2 ± 5.9	87.0 ± 3.4	88.9 ± 2.9
	SOFT	82.3 ± 5.9	62.1 ± 4.8	90.8 ± 1.8	92.5 ± 1.5
	REKP	85.8 ± 3.8	66.3 ± 2.8	91.7 ± 2.7	94.1 ± 2.5
	FT	79.2 ± 5.2	50.1 ± 5.1	87.1 ± 3.8	87.8 ± 2.5
	PT	84.7 ± 3.2	64.5 ± 4.5	92.0 ± 2.1	93.6 ± 2.1
10	AUTO	81.2 ± 4.1	59.9 ± 5.1	91.5 ± 2.8	93 ± 2.5
	SOFT	84.8 ± 2.9	64.7 ± 3.8	91.8 ± 2.7	93.2 ± 2.6
	REKP	86.1 ± 2.5	66.8 ± 1.9	91.3 ± 1.5	94.5 ± 1.7
	FT	85.1 ± 2.5	59.8 ± 3.1	89.3 ± 2.4	90.2 ± 1.8
	PT	86.2 ± 1.5	66.9 ± 2.4	92.1 ± 1.9	93.9 ± 1.4
15	AUTO	85.3 ± 2.1	64.7 ± 2.6	92.1 ± 2.1	93.8 ± 1.2
	SOFT	86.1 ± 1.4	67.1 ± 0.9	92.5 ± 1.0	94.3 ± 1.1
	REKP	86.9 ± 0.8	67.8 ± 0.3	92.2 ± 0.5	94.7 ± 0.6

.

We discover that the baselines’ non-prompt tuning model, FT, clearly lags behind other prompt tuning models. This further demonstrates the clear benefits of using prompt-tuning in the task of model fine-tuning text classification. SOFT consistently outperforms PT in models based on prompt-tuning, suggesting that continuous vector representation may be preferable to discrete representation. The effectiveness of AUTO, however, falls considerably short of these two models. Due to the small number of tag samples, the vocabulary’s choice of tag terms becomes erroneous. The results are disappointing, especially in experiments with small sample sizes, because AUTO can only choose tag words based on sample data from coefficients. The performance of this model, however, improves the quickest as the number of samples increases.

Then, we compared the performance of different baselines with REKP. We capture that the Micro-F1 performance of REKP on each dataset is significantly superior to that of the baselines, which proves that our idea of expanding the tag thesaurus of external knowledge is effective. On average, in the 1-shot, 5-shot, and 10-shot experiments, compared with the best baseline, our Micro-F1 index increased by 16.8%, 8.7%, and 5.6% respectively. For the 15-shot experiment, we can see that with the increase of training samples, the gap between different methods gradually decreases, but REKP still maintains its advantages and outperforms other benchmark models. It is worth noting that in the IMDB dataset, SOFT outperformed PTEK in the 15-shot scenario. We hypothesize that the continuous vector representation may demonstrate stronger advantages as the number of samples increases, and future research can explore optimizing the model using vector representation of tag words.

5.2. Ablation Study

For RQ2, to explore the effectiveness of the three refinement methods, we further analyze the influence of the three refinement methods on the classification performance, and carry out the ablation experiment of the model by deleting specific modules.

• w/o WR removes Label Word Refinement from REKP;
• w/o CR removes Correlation Refinement from REKP;
• w/o IR removes Importance Refinement from REKP;
• w/o RRR removes three refinement modules of REKP.

Due to the long running time of the model, we chose the AG’s News dataset with topic classification and the IMDB dataset with emotion classification. The specific experimental results are shown in

Table 4. Results of ablation experiment (↑: elevation, ↓: descent, =: basically equal), Better than the optimal baseline model in W/O is appended *.

Dataset	Method	1	5	10	15
AG’s News	REKP	83.9	85.8	86.1	86.9
	w/o WR	83.7 ↓	85.2 ↓	86.1 =	86.8 ↓
	w/o CR	83.0 ↓	85.6 ↓	86.5 ↑	86.9 =
	w/o IR	83.9 =	85.9 =	86.7 ↑	87.4 ↑
	w/o RRR	82.5 * ↓	83.6 * ↓	85.6 * ↓	86.6 * ↓
IMDB	REKP	91.1	91.7	91.3	92.2
	w/o WR	91.0 ↓	91.7 =	91.3 =	92.2 =
	w/o CR	90.8 ↓	91.6 ↓	90.5 ↓	92.4 =
	w/o IR	90.8 ↓	91.8 =	90.5 ↓	92.1 ↓
	w/o RRR	90.7${}^{*}$ ↓	91.5${}^{*}$ ↓	90.2 ↓	91.9 ↓

.

For Variant 1 (w/o WR): For topic classification, WR proves to be more effective when working with a smaller number of samples, aligning with our initial intent of leveraging prior knowledge. As for emotion classification, since there is less noisy data associated with emotion tag words, the number of tag words removed in this module is relatively lower.

For Variant 2 (w/o CR): It is evident that both topic classification and emotion classification CR have yielded satisfactory results. However, when dealing with a large volume of samples, the omission of certain tag words may lead to a decrease in the granularity of the tag lexicon, consequently resulting in a decline in model performance.

For variant 3 (w/o IR): We observe that this module is not conducive to topic classification, possibly due to the presence of numerous tag categories in this task. In this scenario, a tag word may belong to a distinct tag lexicon, and the lack of consideration for cross-information during importance calculation may be a contributing factor.

When comparing with the optimal baseline, we observe that in the majority of cases, the model’s performance remains superior even when excluding the implementation of WR, CR, and IR.

5.3. Impact of Sample Number

To address RQ3 and examine the impact of different sample quantities on classification performance, we conducted 1-shot, 5-shot, 10-shot, and 15-shot experiments. The baseline models employed were FT, PT, AUTO, and SOFT. The results are presented in Figure 5. It can be observed that our proposed model, PTEK, outperforms the four baseline models in most cases, validating the effectiveness and robustness of our model. Furthermore, it is evident that as the sample quantity increases, the classification performance of all models improves, albeit at a slower rate.

In comparison to topic classification, the model demonstrates superior performance in the task of sentiment classification, achieving a Micro-F1 score of 0.925 even in the 1-shot scenario. This can be attributed to the fewer number of label categories and inherently lower difficulty level associated with sentiment classification when contrasted with topic classification. Importantly, our model maintains high classification performance even with a limited number of samples, underscoring its compatibility with few-shot learning conditions.

5.4. Refinement Visualization

To answer RQ4, we count and visualize the number of tag words remaining after Label Word Refinement (WR) and Correlation Refinement (CR) processes, and we can intuitively observe the denoising process.

As we can see in Figure 6, these refinement processes delete a large part of the tag words while retaining the tags with the largest amount of information and the most important to the tag in the target task. Moreover, the remaining number of tag words at least exceeds 100, which far exceeds the number of tag words in the previous methods. The universality of tag words is helpful to improve the performance of the prompt tuning model. In addition, it is not difficult to see that the number of tag words deleted in emotion classification is minimal, which also verifies that the refinement processes analyzed in Section 5.2 have little influence on emotion classification. The emotion dictionary has almost no noise.

In summary, the process of refinements plays a pivotal role in effectively reducing noise within the label word repository.

6. Conclusions and Future Work

In this work, we proposed a refined external knowledge into a prompt-tuning (REKP) model for few-shot text classification. We enhanced the verbalizer by adding an external knowledge-expanded label vocabulary, building on prompt learning. In addition, to remove noise from the label vocabulary for a more positive impact on the results, we design three refinement methods: Label Word Refinement, Correlation Refinement, and Importance Refinement. These approaches take into account the noise impact of the vocabulary in various tasks. The efficacy and reliability of our technique are demonstrated by experimental findings on two different dataset types, namely topic classification and sentiment classification compared to all baselines. Furthermore, thorough ablation research revealed that refinements are a crucial element for our model. Our approach enhances the accuracy of text classification in scenarios with limited new sample quantities, such as healthcare and finance, thereby enabling more precise downstream tasks such as decision analysis, sentiment analysis, and intention recognition.

In future work, we will like to incorporate more social network information to further enhance knowledge expansion. Moreover, while this paper utilizes an enhanced verbalizer, it still relies on manually constructed “hard templates”. Although these “hard templates” conform to human language conventions, they may not be easily interpretable for machine learning algorithms. Looking ahead, we envision initializing the templates as vectorized “soft templates” that can adapt to the task during the training process, thus improving the adaptability of hint learning.