ELM-Based Active Learning via Asymmetric Samplers: Constructing a Multi-Class Text Corpus for Emotion Classification

Abstract

A high-quality annotated text corpus is vital when training a deep learning model. However, it is insurmountable to acquire absolute abundant label-balanced data because of the huge labor and time costs needed in the labeling stage. To alleviate this situation, a novel active learning (AL) method is proposed in this paper, which is designed to scratch samples to construct multi-class and multi-label Chinese emotional text corpora. This work shrewdly leverages the superiorities, i.e., less learning time and generating parameters randomly possessed by extreme learning machines (ELMs), to initially measure textual emotion features. In addition, we designed a novel combined query strategy called an asymmetric sampler (which simultaneously considers uncertainty and representativeness) to verify and extract ideal samples. Furthermore, this model progressively modulates state-of-the-art prescriptions through cross-entropy, Kullback–Leibler, and Earth Mover’s distance. Finally, through stepwise-assessing the experimental results, the updated corpora present more enriched label distributions and have a higher weight of correlative emotional information. Likewise, in emotion classification experiments by ELM, the precision, recall, and F1 scores obtained 7.17%, 6.31%, and 6.71% improvements, respectively. Extensive emotion classification experiments were conducted by two widely used classifiers—SVM and LR—and their results also prove our method’s effectiveness in scratch emotional texts through comparisons.

1. Introduction

Large amounts of electronic textual data are produced by media or mobile platforms every day (due to the rapid development of social media). By deeply mining the raw data collected from the internet, a large number of valuable threads and rules can be found and used, promoting various research developments, such as patient emotion state detecting [1], bot detection [2,3,4], emergency response [5], and disease diagnosis [6]. Information recognition study is usually conducted on various deep neural networks (DNNs), which need large amounts of labeled data. However, data labeling is extremely tedious, as well as labor-, cost-, and time-consuming, and requires professional knowledge by human experts. Moreover, keeping labels balanced while tagging also requires expert resources. Thus, labeled data corpora are expensive [7,8] and hard to acquire, which restricts the sharing and spreading of high-level corpora worldwide. Moreover, the labeled data shortage has evolved as a ‘bottleneck’ to the development of artificial intelligence (AI) [9]. As an important branch of machine learning, active learning (AL) has the ability to alleviate the shortage of labeled data. AL aims to (1) acquire labeled data through the cooperation between computer model selections and human expert labeling, or (2) complete the challenging task via tagging pseudo labels based on accurate label predictions.

However, in real applications, the existing AL approaches still have some problems. On the one hand, traditional machine learning methods have embarrassing performances when recognizing language semantics, even with the help of pre-trained word vectors (based on the bag of words proposed by Markov [10]), or when looking for precise pre-trained word vectors. On the other hand, more deep active learning (DAL) [11,12] models are introduced into AL tasks to generate multi-dimension feature representations with long training times, such as TextCNN [13], BERT [14], LSTM [15], and their variants. In NLP tasks, they usually achieve better performances when compared to traditional machine learning regarding homologous data features; however, their long training times involve unaffordability, conflicts with the original intentions of AL, and are inharmonious when it comes to cooperating with human experts. Therefore, we need to build a kind of AL method with a lightweight neural network that is (1) highly explainable, (2) effective in extracting emotional information, and (3) takes up less training time.

In this work, ELM (as proposed in 2006 by Huang et al. [16] with a single-layer feed-forward network (SLFN)), considering its previous excellent performance and less training time, was employed to (instruct to) build the AL framework in a novel method. In the proposed AL framework, the widely-used vectorization tool TF-IDF was initially utilized to provide textual representation for the instances. Then, through ELM’s particular processing, textual representations of the features were transformed as burgeoning feature maps before being executed via sampling criteria for probability comparisons in the AL iteration progress. In this work, we argue that asymmetric samplers constructed by uncertainty [17,18] and representativeness [19,20] can select important and needed instances; these two strategies are normally successively reflected by cross-entropy, Kullback–Leibler divergence, and earth mover’s distance. The selected samples were professionally annotated by Human Oracle, progressively updating the training corpus. Furthermore, to verify the efficiency of the designed AL method, it was examined on sentence-level Chinese text extracted from a social media platform; extensive emotion classification experiments were conducted simultaneously. Moreover, the referred data from the experiments were divided into eight emotion classes referred from Ren Lab’s emotion taxonomy and an additional neutral class. Moreover, the recurrent emotion classification experiments verified the selected samples by our method and could improve the model’s learning ability progressively.

In this work, there are three main contributions involved in constructing a high-quality sentence-level Chinese textual emotion corpus: (1) A novel AL model that employs SLFN (ELM) as a textual feature processor, replenishing the method to build the Chinese text corpus. (2) A novel combined query strategy—called asymmetric samplers—that simultaneously considers two asymmetric factors (i.e., uncertainty and representativeness) and achieve state-of-the-art performances among the mentioned query strategies. (3) The learning efficiency and ceiling performances for text emotion classification are significantly improved with the proposed method.

The remainder of this paper is organized as follows. Section 2 reviews the working procedure of ELM, traditional AL, and the development of emotion taxonomy. The model’s methodology is described in Section 3. We illustrate the experimental results in detail and compare the differences among the multi-query strategies in Section 4. Finally, Section 5 concludes this study.

2. Related Works

The research of AL helps to alleviate the pressure of obtaining the labeled text in mining and analyzing the emotion polarity described by the targeted text [21,22]. While designing the AL algorithm, two vital parts are attracting the attention of researchers—the precise probability predictor and excellent sampling criterion [17,18,19,20]. In this section, we briefly review the related research on ELM and highlight the merits and achievements. The existing AL technologies are briefly introduced, which use different feature processors and query strategies. Later, the developing history of emotion taxonomy is described concisely.

2.1. Extreme Learning Machine

A single-layer feed-forward network (SLFN), different from the deep learning model (with many learning layers), has a more brief and bright structure with only one hidden layer, which means higher computing speeds and lower computing resource costs when dealing with enormous amounts of data. In the past decades, ELM, as the representative model of SLFN, except for the advantages mentioned above, has two advantages: (1) generating hidden layer unit parameters randomly, and (2) admirable interpretability.

To cast off the limitations of parameter tuning in general DNNs while training, in ELM, parameters in the hidden layer are exhaustive and generated randomly, which does not need manual tuning or control. Moreover, this operation can save parameter fine-tuning time. Moreover, ELM has other specialties—activation functions can be bounded, non-constant piecewise continuous functions as well as integrable piecewise continuous functions, which means we can adjust ELM’s activation functions to fit our own tasks, which is how we can explain its working mechanism in a detailed manner. With these two traits, ELM can be transferred to many tasks that concern the fields of visual [23,24], audio [25,26], natural language [27,28], and physiological signals [29,30], comfortably and flexibly. Thus, many works have been conducted on the deuterogenic ELM, achieving decent results. For example, Yang et al., in [31], offered an ELM-based learning method that could improve its performance by pulling back the residual network error to the hidden layer. Moreover, due to fewer hidden nodes employed in the model, the learning speed was one hundred times faster than other ELMs. Migel et al. [32] also proposed stacking multi-ELM modules together to construct a DNN, and they controlled the inputting data dimensions by a weighted matrix between two ELM modules, which reduced the computing complexity. In [33], Deng et al. continued the research by shortening the learning times by fusing the singular value decomposition (SVD) with the hidden ELM layer, which reduced learning costs in the task and improved the classifier’s performance of high-dimensional data. Normally, ELM is used in classification and regression tasks, and the learning data are in an imbalanced state. To adjust ELM (to adapt to this situation), Li et al. [34] used a modified AdaBoost, which could provide weighted training samples to weighted ELM. Follow-up studies focused on assembling ELM with a novel learning structure for different aims as well as on optimizing the inner loss functions to obtain a reduction of errors. In [35], Li et al. abandoned the conventional modus operandi for optimizing the output layer’s error in training. They reduced the calculating error via controlling hidden units based on the number of training samples before being sent to the hidden layer. The above description shows that ELM has indomitable vitality in classification and regression tasks. However, to our knowledge, ELM is seldom reported or employed in the AL field. Thus, we decided to utilize ELM’s extraordinary ability in recognizing instance information and transfer it into our AL work.

2.2. Traditional Active Learning

One challenge in machine learning is how to acquire an appreciable quantity of labeled data to ‘learn’ an outstanding model in a variety of tasks [21,22,36]. With the hope of decreasing the sampling times and minimizing annotated data costs, the investigator wields various information distance evaluating measurements to classify or cluster unlabeled samples via training on small labeled data corpora, discarding the redundant parts according to the designed sampling rules. Then, the selected samples are transferred into a training set after being labeled by Human Oracle; this selecting and labeling process continues until they stop being triggered. Overall, the above sampling procedure is called AL.

Normally, the core of AL is the query strategy to investigate the similarity of samples and complete the selection of wanted samples [37]. Typically, we summarize various strategies into two types: uncertainty [38] and representativeness [19,20]. With uncertainty, if the candidate samples are far away from the baseline data in the probability map and cannot confirm their classes based on current extracted features, there is adequate evidence to suggest that such samples with high uncertainty scores should be selected to enrich the previous training set and be annotated by Human Oracle. However, unitary sampling hurts the AL model when filtering information. Therefore, combining representativeness with uncertainty to form asymmetric samplers can make the selection procedure more justifiable. Moreover, the hybrid model can help reduce the costs of eliminating the samples with less information [39]. Nowadays, it is hard to say which query strategy has universal relevance in various AL tasks (in the background of the big data era). However, researchers are attempting to optimize typical methods or recommend renewable ideas to progressively improve the model’s performance. Through verification—cross-entropy, Kullback–Leibler divergence and earth mover’s distance contribute to the research on measuring sample distances when they are employed in enlarging data sets. In recent years, earth mover’s distance, as a burgeoning difference assessing method, has been pursued by many researchers when measuring information similarities. However, previous AL models declared that they were effective in only one certain research realm, which means many targets should be overcome. Thus, this paper proposes a novel AL model to alleviate the pressure of labeled corpora shortages.

By adjusting the format of data inputting as a decisive standard, we can roughly categorize information-oriented AL into three main scenarios—pool-based, stream-based, and membership query synthesis. Moreover, due to text sets usually being closed in classifications, we followed the format of pool-based AL in this work. Therefore, to bridge the gap between the need for labeled data and labeling costs, this work proposes asymmetric sample AL fused with label-balanced mechanisms.

2.3. Emotion Taxonomy

Miller T.L. et al. [40] enhanced Ekman’s emotion category, i.e., human emotions could be categorized into six basic classes from the view of psychology—anger, disgust, fear, happiness, sadness, and surprise, regardless of race, culture, and language. Since then, more elaborate emotion taxonomies have been nominated according to the needs of different research tasks, such as psychology, face emotion recognition, mental diseases, and others. For example, Plutchik and Kellerman [41] employed four bipolar axes—joy vs. sadness, anger vs. fear, trust vs. disgust, surprise vs. anticipation for modeling emotions in a multi-dimensional space. Moreover, a tree-structured model was proposed by Shaver et al. [42], in which anger, fear, joy, love, sadness, and surprise were the emotions of the main branches, each of which had subordinate categories, such as affection, lust, and longing.

More recently, Quan et al. [43] from Ren Lab, after deeply studying human communication habits in the text (based on Ekman’s six basic emotions), proposed a peculiar emotion sorting technique suitable for natural language processing, for anxiety, anger, sorrow, hate, joy, love, expect, and surprise. Moreover, the fine-grained emotion distribution is closer to ‘feeling’ expressions in daily life. Furthermore, Ren Lab released a large and thoroughly annotated emotion corpus Ren CECps (Available online: http://a1-www.is.tokushima-u.ac.jp/member/ren/Ren-CECps1.0/DocumentforRen-CECps1.0.html (accessed on 16 October 2018)), this corpus has attracted many related research studies [44,45,46,47,48]. Therefore, this is why Ren Lab’s emotion taxonomy was adopted in this work.

A lightweight network is remarkable in data mining. For further excavating its potency, in our research, ELM, as an outstanding SLFN, was first employed to predict the pseudo-emotion labels for expanding the Chinese textual emotion corpus. Moreover, asymmetric samplers were constructed to evaluate textual emotion distributions based on the emotion category proposed by Ren Lab.

3. Methodology

In this section, the proposed AL via asymmetric samplers is introduced in detail Algorithm 1. First, ELM is described from its inner structure and its working procedure on multi-class Chinese textual tasks. Then, detailed descriptions of query strategies in asymmetric samplers are presented in mathematical derivations.

Algorithm 1. ELM-based active learning algorithm

Input: Training set ${\mathcal{D}}_{train}\in R^l$, Raw set ${\mathcal{D}}_{sample\_pool}\in R^{len\left({\mathcal{D}}_{sample\_pool}\right)}$

Output: Updated training set ${\mathcal{D}}_{updated\_train}$

1: repeat

2:    Learn a multi-label emotion classifier ELM on X;

3:    repeat

4:       i means the similarity measurements $CE$, $KL$, $EM$

5:       ${\mathcal{D}}_{sample\_pool}^u\leftarrow maxpartition({\mathcal{D}}_{sample\_pool},{\mathbb{D}}_i)$ (Figure A1a)

6:    until stop criterion 1

7:    repeat

8:       j means the similarity measurements $CE$, $C{E}_b$, $E{M}_b$

9:       ${\mathcal{D}}_{sample\_pool}^r\leftarrow minpartition({\mathcal{D}}_{sample\_pool}^u,Sim{\left(x^{\prime }\right)}_j)$ (Figure A1b)

10:    until stop criterion 2

11:    obtain ground truth label from Human Oracle $y_t$ for ${\mathcal{D}}_{sample\_pool}^r$ (Figure A2)

12:    ${\mathcal{D}}_{updated\_train}={\mathcal{D}}_{train}+({\mathcal{D}}_{sample\_pool}^r,y_t)$

13: until stop criterion 3

14: return ${\mathcal{D}}_{updated\_train}$

3.1. Working Procedure of ELM

Firstly, a brief explanation is given of ELM and how it is used in the proposed model. Generally, within the realm of machine learning, fewer layers in the model mean fewer time–costs in the training. As the classical representative of SLFN, ELM has a single hidden layer that can adjust and generate hidden unit numbers randomly. With this property, our ELM-based AL method saves a lot of training time.

As a single-layer neural network, compared to other traditional SLFNs, ELM can promise learning accuracy while having a faster learning speed [49]. While training, in the hidden layer, ELM employs random weights w and biases b. Moreover, training data are denoted as ${\mathcal{D}}_{train}\in R^l$, which are already encoded by TF-IDF in the form of $\left\{X,Y\right\}=\left\{\left\{x_1,y_1\right\},\dots ,\left\{x_i,y_i\right\},\dots ,\left\{x_l,y_l\right\}\right\}$, where xi means the instance and yi is the ground truth label. The activation function in the hidden layer (h in Figure 1) is $g\left(*\right)$ (sigmoid, in this paper). Thus, the single-layer network transformation can be expressed as:

$$F\left(x\right)=\beta g(X;w,b)$$

(1)

where $\beta$ is the weight matrix, w is the weight between the hidden nodes and input nodes, and b is the bias.

Compared with the ordinary SLFN, the most soul-stirring innovation in ELM is that all the initial parameters in $\beta$ are generated randomly after fixing the number of hidden nodes and activation functions. Then, when training on ${\mathcal{D}}_{train}$, the least square (LS) can be formulated compactly as:

$$L(X,Y;\beta )={∥Y-\beta H∥}^2\to 0$$

(2)

$$H=g(X;w,b)$$

(3)

where X is the text, Y is the true textual emotion label, and H is the matrix of the collection about the activation functions.

While setting the deviation between the predicted probability and true value being 0 (Equation (2)), ELM finds the value of the simulation, which is extremely close to the ground truth value. Thus, parameter $\beta$ is determined directly by

$$\widehat{\beta }=H^{*}Y,$$

(4)

where $H^{*}$ is the Moore–Penrose generalized inverse of H. Moreover, the learned parameter $\widehat{\beta }$ is a key role in the procedure of predicting the raw Chinese textual emotion label. In theory, due to LS being preset as 0, the emotion probability should be trusted with a high confidence score.

By reviewing ELM, we separate its architecture into two parts: learning and predicting. In learning, the goal is to learn the parameter $\widehat{\beta }$. Moreover, in predicting, with the help of learned $\widehat{\beta }$, ELM predicts the raw Chinese textual emotion state precisely, which enhances the instance selection work in the query strategy phase.

3.2. Proposed Active Learning Method

The core idea of AL is to extract the wanted samples to alleviate the current pressure in mining and labeling texts with both high uncertainty and representativeness [50,51,52]. Due to our raw data set being a closed set, we executed our experiments as a pool-based batch-mode scenario as mentioned in Section 2.

3.2.1. Proposed Textual Label Predicting Mechanism

In this paper, we employed ELM as the representation executor to nominate text features on the presupposed eight emotion classes. The predicted probabilities on candidate texts were provided by basic ELM. According to the learning parameters $\widehat{\beta }$, the text emotion probability can be noted as:

$$\begin{array}{cc}\hfill P& =\widehat{\beta }g(T;w,b)\hfill \\ \hfill & =\widehat{\beta }H\hfill \end{array}$$

(5)

where $T\subseteq {\mathcal{D}}_{sample\_pool}=\left\{t_1,\dots ,t_j,\dots ,t_n\right\}\in R^n$, and $H=g(T;w,b)$. To simulate our daily communication convention more closely, we fully believe that being ‘neutral’, as the common emotion state in life, should be introduced in our work. We denote its probability as:

$$p_9=1-Max\left(P\right)$$

(6)

The sigmoid function is used to normalize the ELM output $p_m\subseteq P_M=\left\{p_1,p_2,···,p_9\right\}$ $\in R^M$ within the interval $[0,1]$, and the sum of the nine emotion probabilities is equal to 1 (see details in Figure 2), which is written as:

$${\mathbb{P}}_{\left(\sigma \right)}=\frac{1}{1+e^{-\left(P_M\right)}}$$

(7)

where ${\mathbb{P}}_{\left(\sigma \right)}$ denotes the finally textual feature representations (denoted in the format of probabilities). Then the textual independently identical distributions on the nine emotion states are obtained. Since 0 and 1 are included in the range of probability, the ultimate endpoint is hardly reached. Hereto, we already completed the preparation work, giving the estimation of each raw text by ELM.

From the above description of traditional AL in Section 2, researchers usually construct AL models from two aspects: uncertainty and representativeness. Thus, to fully investigate text information, the proposed AL via asymmetric samplers integrates the above two aspects simultaneously. In this work, to make a strong and direct comparison, we constructed baselines on the support of these three similar measurements: cross-entropy (CE), earth mover’s distance (EM), and Kullback–Leibler divergence (KL), which are distinguished query strategies in information theory. The detailed descriptions of the computational processes between ${\mathbb{P}}_{\left(\sigma \right)}$ and Y are as follows.

CE, known as the most popular loss function, plays a vital role in information theory. In this paper, it is rewritten as:

$${\mathbb{D}}_{CE}({\mathbb{P}}_{\left(\sigma \right)},Y)=Max(-\sum _{m=1}^{M}Ylog\left({\mathbb{P}}_{\left(\sigma \right)}\right)).$$

(8)

Observing the above function, the texts with maximum loss are far away from the training data; it is difficult to distinguish their classes. In the other words, they should be selected and annotated by human experts carefully.

From the definition of EM described in [53], the distance from one distribution to another distribution can be measured based on the given approach. Moreover, from its name, this method’s ‘mathematics’ means to find the shortest way to deliver one distribution to another. The lower the score achieved, the higher the similarity between two distributions. If we scratch the maximum distribution, it means the corresponding text has a higher opportunity to be extracted. Moreover, EM evaluates the distance between ${\mathbb{P}}_{\left(\sigma \right)}$ and Y via the formula given by

$${\mathbb{D}}_{EM}({\mathbb{P}}_{\left(\sigma \right)},Y)=Max(\frac{{\sum }_{m=1}^M{\sum }_{j=1}^n{\rho }_{mj}{\eta }_{mj}}{{\sum }_{m=1}^M{\sum }_{j=1}^n{\rho }_{mj}}).$$

(9)

EM is also known as the optimal transport (OT) distance. In this paper, it is used to measure the distance from one textual distribution ${\mathbb{P}}_{\left(\sigma \right)}$ to another textual distribution Y through an optimized process, until ${\mathbb{P}}_{\left(\sigma \right)}$ is geometrically reshaped as Y. Moreover, ${\rho }_{mj}$ indicates the cost of the transferred probability from the mth stack of ${\mathbb{P}}_{\left(\sigma \right)}$ to the jth stack of Y, and ${\eta }_{mj}$ indicates the quantity of transferred probability from the mth stack of ${\mathbb{P}}_{\left(\sigma \right)}$ to the jth stack of Y. Considering the important properties of EM, such as symmetry and triangle inequality, this paper also adopts it as a sample query strategy.

Specifically, KL divergence is a widely-used similarity measurement in sampling work, which evaluates the log difference between ${\mathbb{P}}_{\left(\sigma \right)}$ and Y under the expectation of ${\mathbb{P}}_{\left(\sigma \right)}$. Moreover, it is also employed in this paper as an instance query strategy; we rewrote it in the format as,

$${\mathbb{D}}_{KL}({\mathbb{P}}_{\left(\sigma \right)},Y)=Max(\sum _{m=1}^M{P}_{(\sigma )m}log(\frac{Y_m}{P_{\left(\sigma \right)m}})).$$

(10)

3.2.2. Query Strategies used in Asymmetric Samplers

Based on the above similarity evaluation methods and related experiments (shown in Section 4), the proposed methodology for the uncertainty criterion is CE. In uncertainty, AL wants to select the instances that are furthest from the training data. However, regarding representativeness, the model expects to find the samples with the shortest distance in the matrix space. For a better self-comparison between the instances, we rewrote cross-entropy as:

$$H\left({\mathbb{P}}_{\left(\sigma \right)}\right)=Min(-{\mathbb{P}}_{\left(\sigma \right)}log{\mathbb{P}}_{\left(\sigma \right)}-(1-{\mathbb{P}}_{\left(\sigma \right)})log(1-{\mathbb{P}}_{\left(\sigma \right)}).$$

(11)

Through the method posted by Equation (11), the texts with the most expectations of uncertainty were extracted. These instances were selected and treated as seed players by another criterion (representativeness). In representativeness, the sampler employs CE (Equation (12)), CE with a balancing strategy (Equation (13)), and EM with a balancing strategy (Equation (14)) to instruct sampling work from three aspects, respectively. In each sampler, AL minimizes the distance scores to find the nearest sample point to strengthen data representativeness. Therefore, this process is described as follows.

For cross-entropy (DS1),

$$Sim{\left({\mathit{x}}^{\prime }\right)}_{CE}=Min\{{\mathbb{D}}_{CE}({\mathbb{P}}_{\left(\sigma \right)},\widehat{Y}),\widehat{Y}\in Y\}.$$

(12)

For cross-entropy with a balancing strategy (DS2),

$$Sim{\left({\mathit{x}}^{\prime }\right)}_{C{E}_b}=Min\{{\mathbb{D}}_{C{E}_b}({\mathbb{P}}_{\left(\sigma \right)},\widehat{Y}),\widehat{Y}\in Y\cup {\widehat{y}}_{x^{\prime }}\}.$$

(13)

For the earth mover’s distance with the balancing strategy (DS3),

$$Sim{\left({\mathit{x}}^{\prime }\right)}_{E{M}_b}=Min\{{\mathbb{D}}_{E{M}_b}({\mathbb{P}}_{\left(\sigma \right)},\widehat{Y}),\widehat{Y}\in Y\cup {\widehat{y}}_{x^{\prime }}\},$$

(14)

where x means the seeds in the raw data pool and ${\widehat{y}}_{x^{\prime }}$ are the predicted labels of the selected texts by Equation (15).

Moreover, inspired by paper [54], asymmetric samplers refer to (and modify) the label balanced strategies. While balancing, the emotion class with the largest predicting probability is directly marked as 1. The balancing strategy is given by

$$\begin{array}{c}\hfill {\widehat{y}}_{x^{\prime }}=\left\{\begin{array}{c}1,\{Max\left\{p_{\left(\sigma \right)m}^{x^{\prime }}\right\},m\in M\}\hfill \\ 0,others\hfill \end{array}\right.\\ \hfill s.t:x^{\prime }\in {\mathcal{D}}_{sample\_seeds}\end{array}$$

(15)

where ${\mathcal{D}}_{sample\_seeds}$ is the selected unlabeled sample set by the first sampler, each of which is a candidate influencing the emotion label balancing state in the training set.

Based on the above textual information similarity measurements, to excavate the ELM module’s potential capacity in AL, asymmetric samplers simultaneously mine textual emotion information from uncertainty and representativeness and successively utilize three mainstream distance strategies (CE, EM, KL) as sampling strategies in this work. Furthermore, the conceptual graph of asymmetric samplers is shown in Figure 2. In the design, the first sampler is utilized to measure the uncertainty of texts and the second sampler is employed to measure textual representativeness. Moreover, a novel label balancing method is proposed in this section, which strengthens the label balance exploration.

4. Experiments and Discussion

In this section, AL tasks and extensive classification experiments were conducted successively. The comparisons are presented on the experimental results, not only from traditional key indicators, such as micro precision, recall, and F1 but also from current popular visual evaluating methods, such as the WordCloud graph for emotion words. Based on these descriptions and analyses of the referred experimental results, this work also provides discussions and unscrambles the unique expressing phenomena in Chinese short-texts.

4.1. Evaluations

Choosing applicable evaluations is extremely important to reflect the model’s efficiency and is necessary to present the rules of the selected texts. In this paper, we mainly evaluate asymmetric samplers from three aspects: the improvement of the emotion classifiers’ learning abilities depending on the updated training set, the distribution of textual emotion labels, and the emotion word rates. To check the ability of the updated text set in learning classifiers, this work constructs three multi-label emotion classifiers based on ELM, logistic regression (LR), and the support vector machine (SVM), which provide compared experiments on updated training corpora annotated by Human Oracle. Moreover, the main evaluation methods employed in this work are micro precision (Equation (16)), recall (Equation (17)), and F1 (Equation (18)), which are widely used in the study of machine learning [55,56]. Moreover, these three indicators are formulated as:

$$P^{micro}=\frac{{\sum }_{m=1}^{M}T{P}_m}{{\sum }_{m=1}^{M}T{P}_m+F{P}_m},$$

(16)

$$R^{micro}=\frac{{\sum }_{m=1}^{M}T{P}_m}{{\sum }_{m=1}^{M}T{P}_m+F{N}_m},$$

(17)

$$F{1}^{micro}=\frac{2\times P^{micro}\times R^{micro}}{P^{micro}+R^{micro}}.$$

(18)

Furthermore, the second aspect involves the ground truth rate of each emotion percentage in the updated set, which denotes the model’s capacity in capturing the useful text in this work. Moreover, the third aspect is presented by WordCloud, which can be seen as the information percentage for each emotion class in the selected instances by our methods.

4.2. Text Preprocessing

Previous research [57] divided texts into three classes—document, sentence, and word (or tokens), depending on the averaged length of the text. Specifically, sentence-level texts express the users’ emotions, intensively and directly, without redundant textual information. Thus, we conducted AL experiments on sentence-level Chinese text from Weibo (Available online: https://weibo.com/ (accessed on 23 March 2013)) in this work. Before experiments, data cleaning is important and necessary in NLP tasks, due to various noises mixed in the raw text resources.

Moreover, in this paper, text pre-processing included filtering, segmenting, and duplicate removal. Firstly, a tiny noisy text set was constructed, which contained 1907 noise sentences, including advertisements, English text, the promotion of games, recipes, etc. By learning these noise examples and filtering the highly similar texts, the number of raw texts in each ${\mathcal{D}}_{sample\_pool}$ can be cleaned and reduced from nearly 20,000 to 7000 obviously. This operation provides a more clear target for the sampling criteria. Secondly, Chinese text has its own traits in syntax rules, which are significantly different from English, with a blank space token between two words and beginning with the capital letter in every independent sentence. However, in Chinese, there are no more marks between characters, and it takes more attention to semantics while expressing. Thus, this work employed the segment module THULAC (Available online: http://thulac.thunlp.org/ (accessed on 11 November 2019)) developed by Tsinghua University, publicly, which is widely used in Chinese text segment work. With the help of THULAC, whole sentences can be split into separate words and characters. In this way, the separated words or characters can be easily encoded by AL algorithm. Thirdly, with the specialized tiny module, duplicated texts are removed from the raw text set. Reasonably, they hurt indiscriminate learning regarding ELM weights. Finally, through these three pre-processing steps, text data are largely purified. Moreover, large amounts of computing resources are saved, improving the efficiency of human–computer interactions. As a result, the sampling procedures of AL are accelerated widely.

4.3. Experimental Setting

In SLFN, the most important parameter is the number of hidden nodes, which largely influence the computational complexities as well as the accuracies in recognizing objects. Referring to papers [28,58], this work sets the model’s hidden nodes as 1000, which is appropriate to our middleweight raw text set.

Generally, the experiment is split into two stages in this work. In the first stage, text sampling work was executed by a single sampler named ‘uncertainty’, which was constituted by CE, EM, and KL. The experimental results in this stage are regarded as baselines or foundations ({(CE: B1), (KL: B2), (EM: B3)}). In the second stage, AL via asymmetric samplers, including uncertainty and representativeness, was constructed based on B1, and denoted as {(double CE: DS1), (CE with balancing strategy: DS2), (EM with balancing strategy: DS3).}

In this work, AL stop criteria were 60 iterations on the AL model (stop criterion 3 in Algorithm 1), 30 sentence-level texts were extracted from ${\mathcal{D}}_{sample\_pool}$ in each iteration (stop criterion 2 in Algorithm 1). In contrast, in AL via asymmetric samplers, stop criterion 1 (in Algorithm 1) selected 500 texts by uncertainty in each iteration.

The initial ${\mathcal{D}}_{train}$ largely impacted AL via the asymmetric sampler model’s learning ability. In this work, Ren Lab collected and labeled 2022 and 1589 sentence-level texts on nine emotion states for ${\mathcal{D}}_{train}$ and ${\mathcal{D}}_{test}$, respectively. In ${\mathcal{D}}_{train}$, the emotion label distribution was nearly kept at a balanced state, except $Neutral$, which was treated as a control role to the eight other emotion categories via manual intervention. Correspondingly, in ${\mathcal{D}}_{test}$, the quantity of each emotion label in the category was nearly 150.

4.4. Experimental Results

The classification performance is a key indicator of corpus quality. Concretely, micro precision, recall, and the F1 score can present the textual information from three different sides. In this section, two widely-used methods provide the comparisons of classification results, and three state-of-the-art query strategies were selected as the baselines.

4.4.1. Results of Baselines

In this section, we mainly discuss the choice of the first sampler depending on the performance of classification tasks and label distribution.

Firstly,

Table 1. The results of the emotion classification experiments on the final corpora of baselines by ELM. (↑ means the improvement, and ↓ notes the decrement).

	Precision			recall			F1
	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)
B1	51.23	57.33	6.10↑	45.05	50.42	5.37↑	47.94	53.65	5.71↑
B2	51.23	54.81	3.58↑	45.05	48.20	3.15↑	47.94	51.30	3.36↑
B3	51.23	61.04	9.18↑	45.05	53.68	8.63↑	47.94	57.13	9.19↑

,

Table 2. The results of emotion classification experiments on the final corpora of baselines by LR. (↑ means the improvement, and ↓ notes the decrement).

	Precision			Recall			F1
	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)
B1	57.32	61.63	4.31↑	65.19	68.46	3.27↑	61.00	64.87	3.87↑
B2	57.32	61.75	4.43↑	65.19	68.62	3.43↑	61.00	65.01	4.01↑
B3	57.32	60.14	2.82↑	65.19	67.29	2.10↑	61.00	63.52	2.52↑

and

Table 3. The results of emotion classification experiments on the final corpora of baselines by SVM. (↑ means the improvement, and ↓ notes the decrement).

	Precision			Recall			F1
	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)
B1	57.17	59.02	1.85↑	56.72	57.06	0.34↑	56.94	58.02	1.08↑
B2	57.17	55.86	1.31↓	56.72	53.79	2.93↓	56.94	54.81	2.13↓
B3	57.17	58.76	1.59↑	56.72	56.45	0.27↓	56.94	57.58	0.64↑

present the initial and final emotion classification experimental results on ${\mathcal{D}}_{train}$, conducted by ELM, LR, and SVM, respectively.

After 60 loops, updated text corpora based on B1 improve the performances of the classifiers and the improvements of classification experiments are {(precision, 6.10%), (recall, 5.37%), (F1, 5.71%)}, {(precision, 4.31%), (recall, 3.27%), (F1, 3.87%)}, and {(precision, 1.85%), (recall, 0.34%), (F1, 1.08%)} on ELM, LR, and SVM, respectively. Meanwhile, corpora built by B2 do not train competent SVM classifiers with the improvements as {(precision, −1.31%), (recall, −2.93%), (F1, −2.13%)}, and corpora built by B3 also have the same commonplace performances in classification experiments conducted by SVM, which are {(precision, 1.59%), (recall, −0.27%), (F1, 0.64%)}. It is clearly shown that the corpora constructed by B1 improve the three classifier learning abilities the most, which indirectly demonstrate that texts extracted by the B1 sampler contain more information on emotions. Figure 3 presents the increments of micro precision, recall, and F1 scores in the experiments. We observed that B1 performed better than B2 and B3 in all three aspects, and its score distributions were more compact, which meant emotional information in the corpus improved smoothly and progressively. Moreover, through comparing the final results, ELM was the suboptimal classifier in Table 1 and Table 3. ELM had a hidden layer in the network as the representative of SLFN, and it required a large amount of data to fine-tune its state, while SVM and LR did not need as many samples to learn due to their formalistic mechanisms.

Secondly,

Table 4. The quantity of each emotion label in the final baselines corpus.

	Anxiety	Anger	Sorrow	Hate	Joy	Love	Expect	Surprise	Neutral
Initial	301	300	304	300	301	300	306	308	99
B1	338	352	486	352	491	415	418	387	1153
B2	316	311	319	311	1794	431	410	369	232
B3	584	774	686	414	335	308	366	338	626

and Figure 4a–c present the emotion label distributions of the final ${\mathcal{D}}_{train}$ from the three baselines. The expectations of each emotion class show the differences from the ideal state to reality. The experimental results for the expectations are presented in Table 4, and their scores are observed at approximately {(B1, 1336), (B2, 2589.56), and (B3, 1397.56)}, respectively, which means B1 treats emotion texts more equally. Additionally, the tight-binding label rate is achieved in Figure 4a–c and it proves that B1 outperforms the other two baselines in controlling emotional text increments. Furthermore, the Hamming loss training (of B1, B2, and B3) is shown in Figure 5a. B1-based ${\mathcal{D}}_{train}$ learns the classifier better with the same amount of emotional texts.

In conclusion, despite the classification performance or quantity of textual emotion labels observed from the above results, it is proper to construct B1 (CE) as the basic sampler. Moreover, to show the improvements in emotion classification performances within groups in a detailed manner, we present the results achieved by ELM on the final corpus of B1 in

Table 5. The emotion classification performances within groups on the final corpus of B1 by ELM. (↑ means the improvement, and ↓ notes the decrement).

	Precision			Recall			F1
	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)
Anxiety	49.76	64.90	15.14↑	57.30	55.06	2.24↓	53.26	59.57	6.31↑
Anger	38.15	61.19	23.04↑	32.84	40.80	7.96↑	35.29	48.96	13.67↑
Sorrow	51.63	65.05	13.43↑	43.18	55.00	11.82↑	47.03	59.61	12.58↑
Hate	38.89	53.19	14.30↑	43.26	28.09	15.17↓	40.96	36.76	4.20↓
Joy	70.00	84.02	14.02↑	58.88	66.36	7.48↑	63.96	74.15	10.19↑
Love	68.47	79.33	10.83↑	65.57	56.13	9.44↓	66.99	65.75	1.24↓
Expect	52.83	65.73	12.90↑	50.00	52.23	2.23↑	51.38	58.21	6.83↑
Surprise	43.72	56.70	12.98↑	42.55	29.26	13.29↓	43.13	38.60	4.53↓
Neutral	33.33	29.53	3.80↓	8.85	66.15	57.30↑	13.99	40.84	26.85↑

. Moreover, the table shows that the consequences also enhance the conclusions obtained from the above analysis, in that B1 (CE) is excellent at extracting emotional texts.

4.4.2. Results of AL via Asymmetric Samplers

In this section, extended instance selection experiments were conducted by AL via the asymmetric sampler model, designed to verify the effectiveness of representativeness criteria in sampling. Firstly, to find the effective query strategy for the second sampler, we conducted extensive experiments; the results can be seen in

Table 6. The results of the emotion classification experiments on the final asymmetric sampler corpus by ELM. (↑ means the improvement, and ↓ notes the decrement).

	Precision			Recall			F1
	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)
DS1	51.23	58.40	7.17↑	45.05	51.36	6.31↑	47.94	54.65	6.71↑
DS2	51.23	59.60	8.37↑	45.05	52.41	7.36↑	47.94	55.77	7.83↑
DS3	51.23	58.84	3.61↑	45.05	51.74	6.69↑	47.94	55.06	7.12↑

,

Table 7. The results of the emotion classification experiments on the final asymmetric sampler corpus by LR. (↑ means the improvement, and ↓ notes the decrement).

	Precision			Recall			F1
	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)
DS1	57.32	61.87	4.55↑	65.19	66.63	1.44↑	61.00	64.16	3.16↑
DS2	57.32	61.15	3.83↑	65.19	65.41	0.22↑	61.00	63.21	2.21↑
DS3	57.32	60.63	3.31↑	65.19	67.24	2.05↑	61.00	63.76	2.76↑

and

Table 8. The results of the emotion classification experiments on the final asymmetric sampler corpus by SVM. (↑ means the improvement, and ↓ notes the decrement).

	Precision			Recall			F1
	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)
DS1	57.17	59.57	2.40↑	56.72	57.33	0.61↑	56.94	58.43	1.49↑
DS2	57.17	58.13	0.94↑	56.72	55.62	1.10↓	56.94	56.84	0.10↓
DS3	57.17	59.38	2.21↑	56.72	56.78	0.06↑	56.94	58.05	1.11↑

and Figure 6. Moreover, the results show that, regardless of what indicators are utilized to evaluate the performances of the models, double CE always outperforms the baselines (DS2 and DS3).

Table 6, Table 7 and Table 8 and Figure 6 demonstrate the changing of micro precision, recall, and F1 in the classification experiments. Table 6, Table 7 and Table 8 show that DS1 has a better performance on LR and SVM than ELM. Moreover, DS2 has higher improvements in {(precision, 8.37%), (recall, 7.36%), and (F1, 7.83%)} on the classifier ELM, but it only achieves {(precision, 3.83%), (recall, 0.22%), (F1, 2.21%)}, and {(precision, 0.94%), (recall, −1.10%), and (F1, −0.10%)} in tasks conducted by LR and SVM, respectively. Meanwhile, DS1 achieves all positive improvements in the three classification experiments, which are {(precision, 7.17%), (recall, 6.31%), (F1, 6.71%)}, {(precision, 4.55%) on ELM, (recall, 1.44%), (F1, 3.16%)} on LR and {(precision, 2.40%), (recall, 0.61%), (F1, 1.11%)} on SVM, respectively. Furthermore, from the tendency of precision, recall, and F1 scores shown in Figure 6, scores achieved by DS1 are smoother and more compact than the other two. Hereto, the usefulness of AL via asymmetric samplers is proved directly and strongly.

Secondly, label distributions are also measured in this section. From Figure 4d,e,f, we observe that the differences in the fluctuations of the emotion label percentages between DS1 and DS2 are difficult to distinguish. Therefore, we took the implementations of the direct quantities of each class emotion label to analyze the tiny distinctions. Based on the quantities of the emotion labels in

Table 9. The quantity of each emotion label in the final asymmetric sampler corpus.

	Anxiety	Anger	Sorrow	Hate	Joy	Love	Expect	Surprise	Neutral
Initial	301	300	304	300	301	300	306	308	99
DS1	372	382	560	381	695	541	482	421	633
DS2	360	399	561	371	662	500	469	396	739
DS3	643	848	611	449	335	310	340	344	559

, except for the numbers, the expectations on every final ${\mathcal{D}}_{train}$ were easily achieved via mathematical computing, and the results were {(DS1, 996.44), (DS2, 962.22), and (DS3, 1245.33),} which depicts the manner of DS2 utilizing the balancing strategy, but its superiority is tiny to DS1.

Thirdly, the Hamming loss measures the importance of text corpora to classifiers from another perspective. In this task, the Hamming loss (based on corpora updated by DS1, DS2, and DS3) is shown in Figure 5b, and it is clear that the curves of the Hamming loss converged rapidly while training on the basis of DS1. Conclusively, ${\mathcal{D}}_{train}$ based on DS1 could provide more ‘emotion’ features to ‘learn’ more proper distributed weights in ELM; improves the classification performance with the same number of labeled texts. Additionally, the improvements within groups of emotion classification on the final corpus based on DS1 by ELM are shown in

Table 10. The emotion classification performances within groups on the final corpus of DS1 by ELM.

	Precision			Recall			F1
	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)	Initial (%)	Final (%)	$\uparrow \mathit{or}\downarrow$ (%)
Anxiety	49.76	57.39	7.63↑	57.30	56.74	0.56↓	53.26	57.06	3.80↑
Anger	38.15	55.28	17.13↑	32.84	44.28	11.44↑	35.29	49.17	13.88↑
Sorrow	51.63	61.72	10.09↑	43.18	58.64	15.46↑	47.03	60.14	13.11↑
Hate	38.89	55.38	16.49↑	43.26	40.45	2.81↓	40.96	46.75	5.79↑
Joy	70.00	74.23	4.23↑	58.88	67.29	8.41↑	63.96	70.59	6.63↑
Love	68.47	76.00	7.53↑	65.57	62.74	2.83↓	66.99	68.73	1.74↑
Expect	52.83	61.11	8.28↑	50.00	54.02	4.02↑	51.38	57.35	5.97↑
Surprise	43.72	48.59	4.87↑	42.55	36.70	5.85↓	43.13	41.82	1.31↓
Neutral	33.33	34.31	0.98↑	8.85	36.46	27.61↑	13.99	35.35	21.36↑

. It is clear that the performances of the overwhelming majority of groups increased visibly in precision, recall, and F1, which proved the proposed method’s efficiency in recognizing emotional tokens.

4.5. Emotional Information in Corpus Built by DS2

From the above description, the proposed AL via asymmetric samplers presents an excellent performance in extracting sentence-level text to enrich the emotional information in the text sets. To further mine emotional lexicons and information in ${\mathcal{D}}_{train}$ built by DS1, this work roughly explores Chinese expressing traditional customs and shows high-frequency emotion words via WordCloud.

By observing the texts in ${\mathcal{D}}_{train}$, we believe that two significant points from empirical evidence impact the model’s learning ability most. Firstly, due to its traditional culture (different from the direct expression manner in English), Chinese is implicit while showing its inner true feelings in some certain emotions. For example, “愿为梁祝”. There are only four characters—and its literal meaning is “want to be Liang and Zhu”, with an ‘emotion’ state of “expect”. However, the publisher presents his “love” in the manner of a love story. Therefore, more accurate text linguistic representation is needed to prevent misunderstanding the integrated emotional information.

Secondly, textual emotion probing is still a research hot topic, but it is a hard point in NLP, especially in multi-class and multi-label tasks. Figure 7 shows the WordCloud drawn from ${\mathcal{D}}_{train}$, built by DS1. For more distinct details, we split emotion into two groups: positive and negative. The emotions joy, love, and expect are arranged positively; the other five emotion labels are believed to be negative. While expressing emotional text randomly, it is obvious that some special words are used with high frequency. In the positive part, “joy” is usually represented by “嘻嘻” (Mimetic word, Xi Xi), “偷笑” (Titter), while “love” is more likely to be expressed by “爱, 喜欢” (both mean love in Chinese). In “expect”, “希望” (expect) and “想” (want) are the most frequent tokens. Moreover, it was found that “泪” (Tears) has a large font size in “expect”. Through an analysis, we observed that the Chinese prefer to express “expect” when they are in an awful situation; it encourages them to overcome the difficulties in their lives due to nationality characteristics.

In contrast, in the negative part, “anxiety” is always represented by words such as “抓狂” (Crazy) and “烦躁” (anxiety). The “anger” expression in Chinese is single, and the most frequent character is “怒” (Anger). In “sorrow”, “泪” (Tears), “伤心” (Sad), and “悲伤” (Sorrow) usually appear together. Compared to the latter-day obscure expression style, it means that the directly emotive expression is the tendency in Chinese. While Chinese shows the feeling “讨厌” (Hate), the token “怒” (Anger) is often used together to intensify people’s feelings. When it comes to “surprise”, it is complex, with its definition being something to happen or messages beyond one’s control or thinking in Chinese. Thus, too many aspects/intentions can be summarized in this class, such as “惊喜” (surprise with happy), “惊吓” (Scare), “惊扰” (Alarm), and so on. However, the above statement does not mean “surprise” texts are collected easily because of the low percentage of this emotion in daily life. Thus, an effective solution is urgently needed to prevent diverting the correct meaning of the input sentence during the process of embedding learning in multi-label and multi-class tasks.

Overall, from the above analyses, we can see that it is important to grasp Chinese from its textual semantics and syntax; this will be our (continued) research point in future work.

5. Conclusions

In this paper, to alleviate the pressure from the shortage of labeled data, we proposed a novel AL via an asymmetric sampler model based on ELM for sentence-level Chinese texts. This model integrates information measurement and a label-balanced strategy to capture emotional texts efficiently, to build a Chinese sentence-level emotion corpus and guide the textual learning of ELM. To begin with, this paper proposes using the highly explainable SLFN (ELM) as the textual feature generator in AL research; our research findings show that it can be transferred into the AL domain conveniently and efficiently, providing a new method to explore the AL field. Furthermore, in the proposed AL model, a novel combined query strategy, known as asymmetric samplers, which consider two asymmetric factors (i.e., uncertainty and representativeness) simultaneously, was utilized to update the text corpus; the updated corpus achieved a state-of-the-art performance among the mentioned query strategies. Finally, the experiments found that ELM-based AL via asymmetric samplers based on double CE achieved state-of-the-art classification performances, and greatly reduced human labor in selecting the raw text, which could largely alleviate the pressure of shortages in acquiring the emotion labels of Chinese texts.

In the future, we plan to study more effective implementations of the query strategy based on ELM. Furthermore, more lightweight neural networks will be employed in our AL research. Moreover, we will introduce an auto-labeling module to the AL study, which would provide the exact predicted labeled data to the text and further reduce labeled data pressure in NLP tasks. Moreover, we will conduct more relative research on correlations among different emotion groups and probe the influences of cross-performances in AL sampling tasks.