Intuitionistic Fuzzy-Based Three-Way Label Enhancement for Multi-Label Classification

Abstract

Multi-label classification deals with the determination of instance-label associations for unseen instances. Although many margin-based approaches are delicately developed, the uncertainty classifications for those with smaller separation margins remain unsolved. The intuitionistic fuzzy set is an effective tool to characterize the concept of uncertainty, yet it has not been examined for multi-label cases. This paper proposed a novel model called intuitionistic fuzzy three-way label enhancement (IFTWLE) for multi-label classification. The IFTWLE combines label enhancement with an intuitionistic fuzzy set under the framework of three-way decisions. For unseen instances, we generated the pseudo-label for label uncertainty evaluation from a logical label-based model. An intuitionistic fuzzy set-based instance selection principle seamlessly bridges logical label learning and numerical label learning. The principle is hierarchically developed. At the label level, membership and non-membership functions are pair-wisely defined to measure the local uncertainty and generate candidate uncertain instances. After upgrading to the instance level, we select instances from the candidates for label enhancement, whereas they remained unchanged for the remaining. To the best of our knowledge, this is the first attempt to combine logical label learning with numerical label learning into a unified framework for minimizing classification uncertainty. Extensive experiments demonstrate that, with the selectively reconstructed label importance, IFTWLE achieves statistically superior over the state-of-the-art multi-label classification algorithms in terms of classification accuracy. The computational complexity of this algorithm is $O\left(n^{2}mk\right)$, where n, m, and k denote the unseen instances count, label count, and average label-specific feature size, respectively.

1. Introduction

In multi-label settings [1,2,3], an instance is associated with multiple labels simultaneously. For example, a picture may be relevant to tags, such as sky, ocean, and seagull; a book may cover topics, such as sports and art. The goal of multi-label classification is to learn mapping from a feature space to a label space, such that the label associations of unseen instances can be determined. It widely applies in domains involving smart grid management [4], disease diagnosis [5], and image classification [6].

Traditionally, multi-label classification schema builds upon logical labels and provides qualitative associations between instances and labels. For robustness, many researchers focus on the extensions of linear or hyperplane-based models, and their work can be roughly categorized as problem transformation and algorithm adaptation, respectively. The previous transforms multi-label classification into a collection of simplified learning scenarios, such as single-label classification. Representative work involves binary relevance [7], random k-labelsets [8], and learning label-specific feature (LLSF) [9]. Comparatively, the latter extends the existing algorithms to simultaneously generate multi-outputs. Representative work includes multi-label forest (ML-Forest) [10] and multi-label twin support vector machine (MLTSVM) [11]. However, both strategies suffered from the calibrated threshold problem. Intuitively, the uncertainty of the label association is larger if the output is close to the threshold, and is smaller otherwise. Hence, we need labels with stronger supervision to boost the model performance.

Regarding a numerical label–this quantifies how much information a label has in describing an instance. For example, a facial expression can be interpreted as the combination of slight sadness, some anger, and a bit of disgust. The fitting of the numerical label is defined as label distribution learning [12]. Although numerical labels offer more discriminative information than logical labels, it is expensive to annotate all numerical labels manually. One feasible solution is to learn numerical labels from logical labels, also known as label enhancement [13,14,15,16]. However, the research studies have indistinguishably employed label enhancements for all instances, regardless of the uncertainty differences.

Three-way decisions [17], also known as trisecting–acting–outcome (TAO) model [18,19,20], is an emerging decision theory for problem solving with uncertainty [21,22,23]. It originates from the semantic explanations for three regions induced by rough sets and has become an emerging theory in the soft computing community [24,25,26,27]. The sequential three steps are trisecting, acting and outcome characterizing the routines on uncertainty. Typically, the trisecting step divides the information into three non-overlapping regions, with two regions as certain and the others as uncertain; the acting step takes the positive/negative strategy w.r.t. of certain objects whereas takes the deferment strategy w.r.t. of uncertain objects; the outcome step evaluates the performance induced by trisecting and acting. Such routines may continue until the uncertainty is negligible and the classification performance is improved. Existing three-way-based multi-label classifications [28,29,30,31,32] deal with label uncertainty either from ensemble features or ensemble algorithms, whereas the ensemble on logical and numerical labels remains untouched.

1.1. Motivation

The semantic uncertainty analysis on multi-label classification has limitations. The fuzzy set shows the effectiveness in measuring the membership degree towards multi-label [33,34,35], but it fails to consider the non-membership degree. As a generalization of the fuzzy set, an intuitionistic fuzzy number (IFN) is effective at quantifying the vagueness of a qualitative instance-label assignment [36], which offers heuristic information to select instances for label enhancement. This paper presents an intuitionistic fuzzy-based three-way label enhancement (IFTWLE) model. It implements three-way decisions by ”trisecting” unseen instances from label-specific learning and by ”acting” label enhancement of uncertain instances. Inspired by empirical studies on single-label [37,38,39], we employed an intuitionistic fuzzy number to search instances with uncertainty classifications. Concretely speaking, IFTWLE applies IFN at the label level and defines a pair of membership and non-membership functions for every unseen instance based on the generated pseudo-labels. The membership functions evaluate the weighted closeness of instances belonging to the specified class, whereas the non-membership functions evaluate the possibility of instances belonging to the complementary class. The ultimate uncertainty instances are determined via an aggregation function. Consequently, we preserve the predicted labels if the classifications are plausible and exploit the numerical label otherwise.

1.2. Contribution

Compared with the existing multi-label classifications models, our contributions are as follows:

(1)

For the first time, cascade learning of logical labels and numerical labels of multi-labels are presented and unified under the semantics of classification uncertainty. Determination of the label association for unseen instances broadens the application of the three-way decisions theory.

(2)

A novel instance selection principle in two stages is presented to integrate logical label learning with numerical label learning. In this way, instances that exhibit significant uncertainty across most labels are enhanced with better discrimination (regarding label importance).

(3)

This is the first attempt to employ an intuitionistic fuzzy set on multi-label classification. It addresses the issue of identifying potentially uncertain instances on each label without stipulating many hyperparameters.

(4)

The proposed IFTWLE has demonstrated effectiveness across many domains. The computational complexity is proportional to the quadratic instances count and linear to the label scale and label-specific features count.

The remainder of the paper is organized as follows. Section 2 reviews some preliminaries on label-specific learning, label enhancement, and the intuitionistic fuzzy set, Section 3 presents our proposed model for multi-label classification; experimental results are reported in Section 4. Section 5 concludes this work and identifies our future directions.

2. Preliminaries

In this section, we briefly review some preliminaries regarding label-specific feature learning and label enhancement.

2.1. Label-Specific Feature Learning

Label-specific feature [9,40,41,42] assumes that each label has unique characteristics and can be described by different feature subsets. For the simplicity of computation, the learning label-specific feature (LLSF) [9] considers second-order label correlation (a.k.a. pairwise, one label is at most dependent on another) and assumes three hypotheses:

(1)

Discrimination: the set of i-th label-specific feature should be most pertinent to the corresponding label ($l_i$), and the included components should be different from other label-specific features.

(2)

Sparsity: the label-specific features should be sparse as compared to the feature space.

(3)

Shareability: the cardinality of common features of two label-specific features with strong label correlations should be larger than those with weak label correlations.

Based on the previous hypotheses, the objective function is formulated as:

$$\underset{\mathbf{W}}{min}\frac{1}{2}{∥\mathbf{XW}-\mathbf{Y}∥}_F^2+\frac{\delta }{2}\mathit{tr}\left({\mathbf{RW}}^{\top }\mathbf{W}\right)+\eta {∥\mathbf{W}∥}_1$$

(1)

where $\mathbf{X}$ and $\mathbf{Y}$ represent the features and labels of multi-label, $\mathbf{W}=\left[{\mathbf{w}}_1,{\mathbf{w}}_2,\dots ,{\mathbf{w}}_i,\dots ,{\mathbf{w}}_m\right]$, and the basic element ${\mathbf{w}}_i$ represents the weight for the label-specific feature of $l_i$, which is composed of the non-zero terms in ${\mathbf{w}}_i$. ${∥·∥}_F^2$ denotes the Frobenius norm. $\mathbf{R}=\left[r_{ij}\right]$ represents a matrix composed of the second-order label relevance. $r_{ij}=1-c_{ij}$, where $c_{ij}$ measures the correlation between label $l_i$ and label $l_j$. The label correlation is computed with cosine similarity. $\mathit{tr}\left(·\right)$ denotes the matrix trace. Symbols $\delta$ and $\eta$ are the balance factors. The inner product of ${\mathbf{w}}_i$ and ${\mathbf{w}}_j$ describes the correlation between label $l_i$ and label $l_j$ from the feature view. The stronger the correlation is, the larger the inner product becomes, and vice versa.

For the prediction of unseen instances, LLSF employs logistic regression and can be denoted as:

$$\widehat{\mathbf{Y}}=sgn\left(\mathbf{X}\mathbf{W}-\tau \right)$$

(2)

where $sgn(·)$ returns 1 if the condition holds, and returns 0 otherwise.

2.2. Label Enhancement

Label enhancement assumes that each instance is intrinsically represented by real-valued labels and, thus, can be recovered from qualitative logical label ($\mathbf{Y}$) to quantitative numerical label ($\mathbf{U}$) via the instance-level or label-level smoothness [15,16,43,44]. The distribution of numerical labels describes the relative importance of different labels in describing a given instance.

To guarantee the effectiveness, three hypotheses are presented in label enhancement multi-label learning (LEMLL) [16].

(1)

Linear relevance: the mapping from feature space to numerical label $g:\mathbf{X}\to \mathbf{U}$ follows a linear model.

(2)

Label similarity: the values of learnt numerical labels should be approximated to the original logical labels.

(3)

Topology consistency: the instances with similar features share similar numerical label values.

Based on the previous assumptions, the objective function is given as:

$$\begin{array}{c}\underset{\Theta ,\mathbf{b},\mathbf{U}}{min}\sum _{i=1}^n{L}_R\left(R_i\right)+\alpha {∥\Theta ∥}_F^2+\beta {∥\mathbf{U}-\mathbf{Y}∥}_F^2+\gamma \mathit{tr}\left({\mathbf{U}}^{\top }\mathbf{MU}\right)\hfill \\ s.t.R_i={∥{\xi }_i∥}_2=\sqrt{{\xi }_i^{\top }{\xi }_i}\hfill \\ {\xi }_i={\mathbf{u}}_i-\Theta \phi \left({\mathbf{x}}_i\right)-\mathbf{b}\hfill \\ L_R\left(R_i\right)=\left\{\begin{array}{cc}0& R_i<\epsilon \\ R_i^2-2R_i\epsilon +{\epsilon }^2& R_i\ge \epsilon \end{array}\right.\hfill \end{array}$$

(3)

where ${\displaystyle \sum _{i=1}^n}{L}_R\left(R_i\right)$ is the loss function from the feature space to the numerical labels, with the regularizer as $R_i={∥{\xi }_i∥}_2=\sqrt{{\xi }_i^{\top }{\xi }_i}$, where ${\xi }_i={\mathbf{u}}_i-\Theta \phi \left({\mathbf{x}}_i\right)-\mathbf{b}$, and $\phi \left({\mathbf{x}}_i\right)$ is a mapping from instance ${\mathbf{x}}_i$ to a high-dimensional space ${\mathbb{R}}^{\mathbb{H}}$; $\Theta$ and $\mathbf{b}$ are the parameters in the linear regression model. ${∥\mathbf{U}-\mathbf{Y}∥}_F^2$ is the implementation of the label similarity assumption measured by Frobenius norm (abbreviated as F). $\alpha ,\beta ,\gamma$ are all balance factors. $\mathit{tr}\left({\mathbf{U}}^{\top }\mathbf{MU}\right)={∥\mathbf{U}-\mathbf{WU}∥}_F^2$ is the implementation of topology consistency, where $\mathit{tr}\left(·\right)$ represents the matrix trace, and $\mathbf{M}={\left(\mathbf{I}-\mathbf{W}\right)}^{\top }\left(\mathbf{I}-\mathbf{W}\right)$, with $\mathbf{I}$ being an identity matrix and $\mathbf{W}$ being the weight matrix constructed by the fully connected graph $G=\left(\mathbf{V},\mathbf{E},\mathbf{W}\right)$, which describes the closeness among the arbitrary instances.

For predictions on unseen instances, LEMLL leverages a kernel logistic regression, which is denoted as:

$$\widehat{\mathbf{Y}}=sgn\left(\Theta \phi \left(\mathbf{X}\right)+\mathbf{b}-\tau \right)$$

(4)

2.3. Intuitionistic Fuzzy Set

Suppose an arbitrary instance ${\mathbf{x}}_i\in \mathbf{X}$ has the corresponding label ${\mathbf{y}}_i$, then for a nonempty set $\mathbf{X}$, the intuitionistic fuzzy set is defined as follows:

$$\tilde{A}=\left\{({\mathbf{x}}_i,{\mu }_{\tilde{A}}\left({\mathbf{x}}_i\right),{\nu }_{\tilde{A}}\left({\mathbf{x}}_i\right))\right|{\mathbf{x}}_i\in X\}$$

(5)

where ${\mu }_{\tilde{A}}\left({\mathbf{x}}_i\right)$ is the membership of instance ${\mathbf{x}}_i$ expressing the chances of instance ${\mathbf{x}}_i$ being a particular class A, ${\nu }_{\tilde{A}}\left({\mathbf{x}}_i\right)$ is the non-membership of instance ${\mathbf{x}}_i\in \mathbf{X}$ representing the possibility of instance ${\mathbf{x}}_i$ not related to class A. $\mu$ and $\nu$ are functions mapping from $\mathbf{X}$ to [0,1]; the following two conditions are satisfied:

1.

${\mu }_{\tilde{A}}\left({\mathbf{x}}_i\right)\in [0,1],{\nu }_{\tilde{A}}\left({\mathbf{x}}_i\right)\in [0,1]$;

2.

$0\le {\mu }_{\tilde{A}}\left({\mathbf{x}}_i\right)+{\nu }_{\tilde{A}}\left({\mathbf{x}}_i\right)\le 1$.

The hesitation of instance ${\mathbf{x}}_i$ is defined as:

$${\pi }_{\tilde{A}}\left({\mathbf{x}}_i\right)=1-{\mu }_{\tilde{A}}\left({\mathbf{x}}_i\right)-{\nu }_{\tilde{A}}\left({\mathbf{x}}_i\right)$$

(6)

which measures the hesitation of instance ${\mathbf{x}}_i$ related to class A. ${\mu }_{\tilde{A}}$ and ${\nu }_{\tilde{A}}$ construct the intuitionistic fuzzy number IFN: $\alpha =({\mu }_{\tilde{A}},{\nu }_{\tilde{A}})$, $S\left(\alpha \right)$ is used to compare two intuitionistic fuzzy numbers:

$$S\left(\alpha \right)={\mu }_{\tilde{A}}-{\nu }_{\tilde{A}}$$

(7)

which could compare instances $({\mathbf{x}}_i,{\mathbf{y}}_i,{\mu }_{\tilde{A}}\left({\mathbf{x}}_i\right),{\nu }_{\tilde{A}}\left({\mathbf{x}}_i\right))$ with $({\mathbf{x}}_j,{\mathbf{y}}_j,{\mu }_{\tilde{A}}\left({\mathbf{x}}_j\right),{\nu }_{\tilde{A}}\left({\mathbf{x}}_j\right))$, and through the value of $S\left({\alpha }_i\right)$ and $S\left({\alpha }_j\right)$, we could determine whether ${\mathbf{x}}_i$ or ${\mathbf{x}}_j$ is more likely to be associated with class A.

3. Proposed Model

3.1. Notations

For ease of reference, we present a nomenclature including the major notations and mathematical meanings in

Table 1. Notation of IFTWLE.

Notation	Mathematical Meanings
$Card(·)$	set cardinality
${\mathbf{X}}_1$	multi-label instances set
${\mathbf{Y}}_1$	logical label set
${\mathbf{X}}_2$	unseen instances set
${\mathbf{x}}_i$	an instance
${\mathbf{y}}_i$	logical label set of the instance ${\mathbf{x}}_i$
${\widehat{\mathbf{y}}}_i$	pseudo-label set of the instance ${\mathbf{x}}_i$ learnt by $f_1$
${\mathbf{u}}_i$	numerical label set of the instance ${\mathbf{x}}_i$
${\mu }_c\left({\mathbf{x}}_i\right)$	membership degree of ${\mathbf{x}}_i$ on label $l_c$
${\nu }_c\left({\mathbf{x}}_i\right)$	non-membership degree of ${\mathbf{x}}_i$ on label $l_c$
${\pi }_c\left({\mathbf{x}}_i\right)$	hesitation degree of ${\mathbf{x}}_i$ on label $l_c$
${\mathbf{X}}_2^{(\mu ,\nu )}$	uncertain instances set on all labels
$\neg {\mathbf{X}}_2^{(\mu ,\nu )}$	certain instances set on all labels
$f_1\left(·\right)$	function of logical label-based learning
$f_2\left(·\right)$	function of numerical label-based learning
${\widehat{\mathbf{Y}}}_2^{*}$	final predicted multi-label set of ${\mathbf{X}}_2$
$l_c$	label c
${\mathbf{Y}}_2^{(\mu ,\nu )}$	the label set of ${\mathbf{X}}_2^{(\mu ,\nu )}$ learnt by $f_1$
$\neg {\mathbf{Y}}_2^{(\mu ,\nu )}$	the label set of $\neg {\mathbf{X}}_2^{(\mu ,\nu )}$ learnt by $f_1$
${\mathbf{X}}_2^{({\mu }_c,{\nu }_c)}$	uncertain instance set on label $l_c$
$\neg {\mathbf{X}}_2^{({\mu }_c,{\nu }_c)}$	certain instance set on label $l_c$
${\widehat{\mathbf{y}}}_{ic}$	pseudo-label of instance ${\mathbf{x}}_i$ on label $l_c$ learnt by $f_1$
${\varphi }_c\left({\mathbf{x}}_i\right)$	high-dimensional representation of instance ${\mathbf{x}}_i$ gives a label-specific feature on label $l_c$
${\widehat{C}}_c^{+}$	class center of the pseudo-positive class on label $l_c$
${\widehat{C}}_c^{-}$	class center of the pseudo-negative class on label $l_c$
$D(·,·)$	Euclidean distance
$r_c^{+}$	radius of the pseudo-positive class on label $l_c$
$r_c^{-}$	radius of the pseudo-negative class on label $l_c$
${\mathbf{x}}_i^c$	label-specific feature of instance ${\mathbf{x}}_i$ on label $l_c$
$p_c^{+}$	pseudo-positive instance weight on label $l_c$
$p_c^{-}$	pseudo-negative instance weight on label $l_c$
${\rho }_c$	weighted neighborhood difference of ${\mathbf{x}}_i$ on label $l_c$
r	instance neighborhood size measured by Euclidean distance $D(·,·)$
$N_c{\left({\mathbf{x}}_i\right)}^{+}$	pseudo-positive instance count in the neighborhood of ${\mathbf{x}}_i$ on label $l_c$
$N_c{\left({\mathbf{x}}_i\right)}^{-}$	pseudo-negative instance count in the neighborhood of ${\mathbf{x}}_i$ on label $l_c$
$U{C}_i$	the number of times that instance ${\mathbf{x}}_i$ is regarded as the uncertain instance on all label sets
$\overline{U{C}_j}$	mean of times that instance ${\mathbf{x}}_j$ is regarded as the uncertain instance on all label sets

.

3.2. Problem Formulation

Let ${\mathbf{X}}_1=\left\{{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_n\right\}$ denotes a set of multi-label instances with the union of known logical label information ${\mathbf{Y}}_1=\left\{{\mathbf{y}}_1,{\mathbf{y}}_2,\dots ,{\mathbf{y}}_n\right\}$; ${\mathbf{X}}_2$ denotes the unseen instances. The logical label of ${\mathbf{x}}_i$ to label set $\{l_1,\dots ,l_c,\dots ,l_m\}$ is denoted as ${\mathbf{y}}_i=\{y_{i1},\dots ,y_{ic},\dots ,y_{im}\}$ where $y_{ic}=1$ holds (positive class) if ${\mathbf{x}}_i$ is associated with label $l_c$, and $y_{ic}=0$ (negative class) otherwise. The numerical label of instance ${\mathbf{x}}_i$ is denoted by ${\mathbf{u}}_i=\left\{u_{i1},\dots ,u_{ic},\dots ,u_{im}\right\}\in {\left[0,1\right]}^m$, which satisfies ${\sum }_c{u}_{ic}=1$. For an arbitrary label $l_c$, we denote the degree of the membership function related to $l_c$ and non-membership functions unrelated to $l_c$ as $IF{N}_c\left({\mathbf{x}}_i\right)=({\mu }_c\left({\mathbf{x}}_i\right),{\nu }_c\left({\mathbf{x}}_i\right))$, respectively, where ${\mu }_c\left({\mathbf{x}}_i\right)\in \left[0,1\right]$, ${\nu }_c\left({\mathbf{x}}_i\right)\in \left[0,1\right]$, and $0⩽{\mu }_c\left({\mathbf{x}}_i\right)+{\nu }_c\left({\mathbf{x}}_i\right)⩽1$. The uncertain instances set on the c-th label is denoted as ${\mathbf{X}}_2^{({\mu }_c,{\nu }_c)}$. Uncertain instances set on all labels are denoted as ${\mathbf{X}}_2^{(\mu ,\nu )}$, and certain instances set on all labels are complementary of ${\mathbf{X}}_2^{(\mu ,\nu )}$, denoted as $\neg {\mathbf{X}}_2^{(\mu ,\nu )}$ (i.e., ${\mathbf{X}}_2={\mathbf{X}}_2^{(\mu ,\nu )}\cup \neg {\mathbf{X}}_2^{(\mu ,\nu )}$). Our goal was to identify uncertain classifications from logical label-based learning and improve the performance with numerical label-based learning.

3.3. Basic Idea

IFTWLE is an implementation of the TAO model for unseen instances ${\mathbf{X}}_2$ (see Figure 1). Firstly, the trisecting was realized by a logical label-based model (denoted as $f_1\left(·\right)$) with an intuitionistic fuzzy number (denoted as $\left(\mu ,\nu \right)$). An instance selection principle was developed to identify the uncertain instances. Secondly, the acting was realized on all instances with different strategies, with label enhancements on uncertain instances (denoted as $f_2\left(·\right)$), adopting the classifications otherwise. Finally, we conducted outcome evaluations on the deduced classifications.

For a non-trivial solution, the three-way classification result of the predicted multi-label set (i.e., ${\widehat{\mathbf{Y}}}_2^{*}$) on ${\mathbf{X}}_2$ is defined as:

$${\widehat{\mathbf{Y}}}_2^{*}=\left\{\begin{array}{cc}{f}_1(\neg {\mathbf{X}}_2^{(\mu ,\nu )}),\hfill & \mathbf{x}\in \neg {\mathbf{X}}_2^{(\mu ,\nu )}\hfill \\ f_2\left({\mathbf{X}}_2^{(\mu ,\nu )}\right),\hfill & \mathbf{x}\in {\mathbf{X}}_2^{(\mu ,\nu )}\hfill \end{array}\right.$$

(8)

where $f_2\left({\mathbf{X}}_2^{(\mu ,\nu )}\right)$ refers to the predicted multi-label sets of deferred instances with large label uncertainty degree and $f_1(\neg {\mathbf{X}}_2^{(\mu ,\nu )})$ refers to the predicted multi-label sets of certain instances with little label uncertainty degrees.

Figure 2 describes the pipeline on the instance selection principle. By taking a problem transformation view, we assigned the membership function (${\mu }_c\left({\mathbf{x}}_i\right)$) and non-membership function (${\nu }_c\left({\mathbf{x}}_i\right)$) for all unseen instances on an arbitrary label $l_c$, which was then incorporated to search for the candidate uncertain instances denoted as ${\mathbf{X}}_2^{({\mu }_c,{\nu }_c)}$. Final uncertain instances (${\mathbf{X}}_2^{(\mu ,\nu )}$) were aggregated by considering the distributions of candidate uncertain instances across all labels. In what follows, we will elaborate the details from Section 3.4, Section 3.5 and Section 3.6.

3.4. Intuitionistic Fuzzy Membership Assignment

Although fuzzy membership assignment is capable of measuring the concept vagueness, it has the following drawbacks for multi-label classification:

• Regardless of the concrete membership function definition, it can only describe the closeness of an instance belonging to a concept. The distribution of the heterogeneous class is thus neglected, which is of great importance in finding uncertain instances.
• Multi-label data have some specialized characteristics, such as an imbalanced class. With the membership function only, the model cannot utilize such information effectively, which leads to the degeneration of model generality.

In [45], the intuitionistic fuzzy set figures out the support vectors from instances and improves the generalization of the support vector machine. In our case, instances have obtained the pseudo-label by conducting label-specific learning (i.e., $f_1(·)$). Therefore we assume a desirable hyperplane is available. This means the possibility of instances with noisy labels is less likely to occur, and misclassified instances are both far from the class center and circled by heterogeneous instances. Therefore, from the perspective of labels, such instances are compatible with the intuitionistic fuzzy set. The degrees of membership and non-membership functions for each unseen instance take the problem transformation view and are determined in a label-specific way. Without losing generality, we consider the construction of ${\mu }_c\left({\mathbf{x}}_i\right)$ and ${\nu }_c\left({\mathbf{x}}_i\right)$ on label $l_c$.

3.4.1. Membership Function ${\mu }_c\left({\mathbf{x}}_i\right)$

Let ${\widehat{\mathbf{Y}}}_2$ be the pseudo-label set of the unseen instance set ${\mathbf{X}}_2$ learnt from the LLSF model ($f_1$); the membership of instances is determined by the relative similarity against the predicted class and other classes. In other words, the membership function of an instance is larger if both the relative distance to the predicted class is smaller and the relative distance to the other classes is larger. Using the class center as a representative, for two instances that are both pseudo-positively associated (i.e., ${\widehat{y}}_{ic}={\widehat{y}}_{jc}=1$), our goal can be formally written as:

$$\begin{array}{cc}\hfill {\mu }_c\left({\mathbf{x}}_i\right)>{\mu }_c\left({\mathbf{x}}_j\right)if& \frac{D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c^{+}\right)}{r_c^{+}+ϵ}-\frac{D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c^{-}\right)}{r_c^{-}+ϵ}\hfill \\ \hfill <& \frac{D\left({\varphi }_c\left({\mathbf{x}}_j\right),{\widehat{\mathbf{C}}}_c^{+}\right)}{r_c^{+}+ϵ}-\frac{D\left({\varphi }_c\left({\mathbf{x}}_j\right),{\widehat{\mathbf{C}}}_c^{-}\right)}{r_c^{-}+ϵ}\hfill \end{array}$$

(9)

where $D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c^{+}\right)$ and $D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c^{-}\right)$ are the distances of instance ${\mathbf{x}}_i$ to the pseudo-positive class and pseudo-negative class, respectively. They are defined as:

$$D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c^{+}\right)=∥{\varphi }_c\left({\mathbf{x}}_i\right)-{\widehat{\mathbf{C}}}_c^{+}∥$$

(10)

$$D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c^{-}\right)=∥{\varphi }_c\left({\mathbf{x}}_i\right)-{\widehat{\mathbf{C}}}_c^{-}∥$$

(11)

where ${\varphi }_c\left({\mathbf{x}}_i\right)$ represents the high-dimensional representation of instance ${\mathbf{x}}_i$ given the label-specific feature w.r.t. $l_c$, and $∥·∥$ is the distance between the instance and the corresponding pseudo-class center. Suppose $K({\mathbf{x}}_i^c,{\mathbf{x}}_j^c)$ denotes a kernel function on the label-specific feature w.r.t. $l_c$, the inner product distance is expanded as:

$$∥{\varphi }_c\left({\mathbf{x}}_i\right)-{\varphi }_c\left({\mathbf{x}}_j\right)∥=\sqrt{K({\mathbf{x}}_i^c,{\mathbf{x}}_i^c)-2K({\mathbf{x}}_i^c,{\mathbf{x}}_j^c)+K({\mathbf{x}}_j^c,{\mathbf{x}}_j^c)}$$

(12)

${\widehat{\mathbf{C}}}_c^{+}$ and ${\widehat{\mathbf{C}}}_c^{-}$ are the class centers of the pseudo-positive and pseudo-negative classes on label $l_c$. The average value is measured by all the pseudo-positive instances that predicted the pseudo-label was 1 on label $l_c$, denoted as ${\widehat{\mathbf{C}}}_c^{+}$; the average value is measured by all the pseudo-negative instances that predicted pseudo-label was 0 on label $l_c$, denoted as ${\widehat{\mathbf{C}}}_c^{-}$.

$${\widehat{\mathbf{C}}}_c^{+}=\frac{1}{l_c^{+}}\sum _{{\widehat{y}}_{ic}=1}{\varphi }_c\left({\mathbf{x}}_i\right)$$

(13)

$${\widehat{\mathbf{C}}}_c^{-}=\frac{1}{l_c^{-}}\sum _{{\widehat{y}}_{ic}=0}{\varphi }_c\left({\mathbf{x}}_i\right)$$

(14)

where $l_c^{+}=\left|\left\{{\mathbf{x}}_i\right|{\widehat{y}}_{ic}=1\}\right|$ and $l_c^{-}=\left|\left\{{\mathbf{x}}_i\right|{\widehat{y}}_{ic}=0\}\right|$ denote the pseudo-positive instance count and pseudo-negative instance count w.r.t. label $l_c$, respectively.

$r_c^{+}$ and $r_c^{-}$ are the radii of the pseudo-positive and pseudo-negative class on label $l_c$, which can be quantified as:

$$r_c^{+}=\underset{y_{ic}=1}{max}∥{\varphi }_c\left({\mathbf{x}}_i\right)-{\widehat{\mathbf{C}}}_c^{+}∥$$

(15)

$$r_c^{-}=\underset{y_{ic}=0}{max}∥{\varphi }_c\left({\mathbf{x}}_i\right)-{\widehat{\mathbf{C}}}_c^{-}∥$$

(16)

By substituting Equations (13) and (14) into Equations (15) and (16), based on Equation (12), we have:

$$r_c^{+}=\underset{{\widehat{y}}_{ic}=1}{max}\sqrt{K({\mathbf{x}}_i^c,{\mathbf{x}}_i^c)-\frac{2}{l_c^{+}}{\displaystyle \sum _{{\widehat{y}}_{jc}=1}}K({\mathbf{x}}_i^c,{\mathbf{x}}_j^c)+\frac{1}{{l_c^{+}}^2}\sum _{{\widehat{y}}_{mc}=1}\sum _{{\widehat{y}}_{nc}=1}K({\mathbf{x}}_m^c,{\mathbf{x}}_n^c)}$$

(17)

$$r_c^{-}=\underset{{\widehat{y}}_{ic}=0}{max}\sqrt{K({\mathbf{x}}_i^c,{\mathbf{x}}_i^c)-\frac{2}{l_c^{-}}\sum _{{\widehat{y}}_{jc}=0}K({\mathbf{x}}_i^c,{\mathbf{x}}_j^c)+\frac{1}{{l_c^{-}}^2}\sum _{{\widehat{y}}_{mc}=0}\sum _{{\widehat{y}}_{nc}=0}K({\mathbf{x}}_m^c,{\mathbf{x}}_n^c)}$$

(18)

one can infer that calculation of $D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c^{-}\right)$ is costly if ${\mathbf{x}}_i$ is pseudo-positive on $l_c$. For simplicity, we use $D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c\right)$ instead. In other words, we examine the dissimilarity of an instance to the class center of all instances.

For multi-label cases, the positive class is the minority class, whereas the negative class is the majority class. The imbalanced class distribution results in the different contributions to the membership function. The rationality is that the location of the class center of the negative class has a much lower empirical risk than the positive class, and the empirical risk for the center of the positive class becomes higher as the count for instances with the positive class becomes smaller. Therefore, we introduce symbol $p_c^{+}$ and $p_c^{-}$ to represent the pseudo-positive instance weight and pseudo-negative instance weight w.r.t. label $l_c$. Any $l_c$, is defined as:

$$p_c^{+}=\frac{2\times l_c^{+}}{l_c^{+}+Card\left({\mathbf{X}}_2\right)}$$

(19)

$$p_c^{-}=\frac{2\times l_c^{-}}{l_c^{-}+Card\left({\mathbf{X}}_2\right)}$$

(20)

where $Card\left({\mathbf{X}}_2\right)$ = $\left|{\mathbf{X}}_2\right|$ refers to the instance number in the instance set ${\mathbf{X}}_2$. For each unseen instance ${\mathbf{x}}_i$, the degree of membership ${\mu }_c\left({\mathbf{x}}_i\right)$ is defined as:

$${\mu }_c\left({\mathbf{x}}_i\right)=\left\{\begin{array}{cc}1-\left(p_c^{+}\times \frac{D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c^{+}\right)}{r_c^{+}+ϵ}+(1-p_c^{+})\times \frac{D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c\right)}{r_c+ϵ}\right)\hfill & {\widehat{y}}_{ic}=1\hfill \\ 1-\left(p_c^{-}\times \frac{D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c^{-}\right)}{r_c^{-}+ϵ}+(1-p_c^{-})\times \frac{D\left({\varphi }_c\left({\mathbf{x}}_i\right),{\widehat{\mathbf{C}}}_c\right)}{r_c+ϵ}\right)\hfill & {\widehat{y}}_{ic}=0\hfill \end{array}\right.$$

(21)

where $ϵ\to 0^{+}$ is an adjustable parameter, $r_c^{+}$, $r_c^{-}$, $r_c$ and ${\widehat{\mathbf{C}}}_c^{+}$, ${\widehat{\mathbf{C}}}_c^{-}$, ${\widehat{\mathbf{C}}}_c$ are the radius and class centers of the pseudo-positive class, pseudo-negative class, and all unseen instances on label $l_c$ determined by $f_1^c\left(·\right)$. Figure 3 shows an example describing the effect of Equation (21), where both ${\mu }_c\left({\mathbf{x}}_{\mathit{A}}\right)>{\mu }_c\left({\mathbf{x}}_{\mathit{C}}\right)$ and ${\mu }_c\left({\mathbf{x}}_{\mathit{D}}\right)>{\mu }_c\left({\mathbf{x}}_{\mathit{B}}\right)$ hold.

3.4.2. Non-Membership Function ${\nu }_c\left({\mathbf{x}}_i\right)$

The non-membership of instances is determined by the following two factors. Firstly, it should be impacted by the dissimilarity and similarity of the instance to the predicted class and other classes. It means the non-membership is larger if an instance is located in the region where the probability of being other classes is larger. Secondly, it should be impacted by the heterogeneous instance distribution within the neighborhood. One can infer that the non-membership is larger if an instance is surrounded by instances from other classes. We assume membership is inversely proportional to non-membership, denoted as:

$${\nu }_c\left({\mathbf{x}}_i\right)\propto 1-{\mu }_c\left({\mathbf{x}}_i\right)$$

(22)

The imbalanced class distribution [46] implies that the contributions of heterogeneous instances are label-dependent. In other words, for two instances, ${\mathbf{x}}_i$, and ${\mathbf{x}}_j$, with the same number of heterogeneous instances in their corresponding neighbours, the non-membership of ${\mathbf{x}}_i$ is larger than ${\mathbf{x}}_j$ if ${\mathbf{x}}_i$ belongs to the pseudo-negative class whereas ${\mathbf{x}}_j$ belongs to the pseudo-positive class. To implement our assumption, we introduced two symbols, $n_c^{+}$ and $n_c^{-}$, to represent the prior probability of being pseudo-positive and pseudo-negative, respectively.

$$n_c^{+}=\frac{l_c^{+}}{Card\left({\mathbf{X}}_2\right)}$$

(23)

$$n_c^{-}=\frac{l_c^{-}}{Card\left({\mathbf{X}}_2\right)}$$

(24)

The two prior probabilities are incorporated to quantify the contribution of the heterogeneous instance within the neighborhood. From the perspective of the neighborhood, the non-membership degree is larger if more heterogeneous instances are included, and smaller if more homogeneous instances are included. Therefore, the weighted neighborhood difference of ${\mathbf{x}}_i$ on label $l_c$ (i.e., $\left({\rho }_c\left({\mathbf{x}}_i\right)\right)$) is defined as:

$${\rho }_c\left({\mathbf{x}}_i\right)=\left\{\begin{array}{cc}\frac{n_c^{+}{N}_c{\left({\mathbf{x}}_i\right)}^{-}}{n_c^{+}{N}_c{\left({\mathbf{x}}_i\right)}^{-}+n_c^{-}{N}_c{\left({\mathbf{x}}_i\right)}^{+}}\hfill & {\widehat{y}}_{ic}=1\hfill \\ \frac{n_c^{-}{N}_c{\left({\mathbf{x}}_i\right)}^{+}}{n_c^{+}{N}_c{\left({\mathbf{x}}_i\right)}^{-}+n_c^{-}{N}_c{\left({\mathbf{x}}_i\right)}^{+}}\hfill & {\widehat{y}}_{ic}=0\hfill \end{array}\right.$$

(25)

where $N_c{\left({\mathbf{x}}_i\right)}^{+}=\left|\left\{{\mathbf{x}}_j\left|{\mathbf{x}}_j\in N_c\left({\mathbf{x}}_i\right)\wedge {\widehat{y}}_{jc}=1\right.\right\}\right|$ denotes the pseudo-positive instance count in the r-neighborhood of ${\mathbf{x}}_i$ on label $l_c$ ($r>0$). $r>0$ is an adjustable parameter and $N_c{\left({\mathbf{x}}_i\right)}^{-}=\left|\left\{{\mathbf{x}}_j\left|{\mathbf{x}}_j\in N_c\left({\mathbf{x}}_i\right)\wedge {\widehat{y}}_{jc}=0\right.\right\}\right|$ denotes the pseudo-negative instance count in the $\gamma$-neighborhood of ${\mathbf{x}}_i$ on label $l_c$. We assume the non-membership degree is proportional to the weighted neighborhood difference, denoted as:

$$\nu \left({\mathbf{x}}_i\right)\propto \rho \left({\mathbf{x}}_i\right)$$

(26)

Based on assumptions (22) and (26), we define the non-membership degree as:

$${\nu }_c\left({\mathbf{x}}_i\right)=(1-{\mu }_c\left({\mathbf{x}}_i\right)){\rho }_c\left({\mathbf{x}}_i\right)$$

(27)

It is easy to validate $0⩽{\mu }_c\left({\mathbf{x}}_i\right)+{\nu }_c\left({\mathbf{x}}_i\right)⩽1$ holds. Here, we offer some explanations. The ${\mu }_c\left({\mathbf{x}}_i\right)$ receives the largest value when it is the center of the plausible pseudo-class, and it reaches 0 only when it is the farthest instance to the class center of all instances simultaneously. The ${\nu }_c\left({\mathbf{x}}_i\right)$ itself is smaller than 1, as the two components are all less than 1. As ${\rho }_c\left({\mathbf{x}}_i\right)$ is smaller than 1, the ${\nu }_c\left({\mathbf{x}}_i\right)$ is no larger than $1-{\mu }_c\left({\mathbf{x}}_i\right)$, which means $0⩽{\mu }_c\left({\mathbf{x}}_i\right)+{\nu }_c\left({\mathbf{x}}_i\right)⩽1$ holds.

By referring to Equation (6), we define the hesitation degree as:

$${\pi }_c\left({\mathbf{x}}_i\right)=1-{\mu }_c\left({\mathbf{x}}_i\right)-{\nu }_c\left({\mathbf{x}}_i\right)$$

(28)

3.5. Label-Level Instance Selection

Having the membership/non-membership degrees, the unseen instances (${\mathbf{X}}_2$) with the intuitionistic fuzzy membership (${\mathbf{IFX}}_2$) are denoted as:

$${\mathbf{IFX}}_2=\{({\mathbf{x}}_1,{\widehat{\mathbf{y}}}_1,{\mu }_1,{\nu }_1),({\mathbf{x}}_2,{\widehat{\mathbf{y}}}_2,{\mu }_2,{\nu }_2),\dots ,({\mathbf{x}}_k,{\widehat{\mathbf{y}}}_k,{\mu }_k,{\nu }_k)\}$$

where ${\mu }_i=\left({\mu }_1\left({\mathbf{x}}_i\right),{\mu }_2\left({\mathbf{x}}_i\right),\dots ,{\mu }_m\left({\mathbf{x}}_i\right)\right)$ and ${\nu }_i=\left({\nu }_1\left({\mathbf{x}}_i\right),{\nu }_2\left({\mathbf{x}}_i\right),\dots ,{\nu }_m\left({\mathbf{x}}_i\right)\right)$ denote the degrees of membership and non-membership functions w.r.t. ${\mathbf{x}}_i$ across all labels, respectively.

In terms of each label, we can select the instances with uncertain classifications (denoted as ${\mathbf{X}}_2^{({\mu }_c,{\nu }_c)}$) as:

$${\mathbf{X}}_2^{({\mu }_c,{\nu }_c)}=\left\{{\mathbf{x}}_i\left|{\mu }_c\left({\mathbf{x}}_i\right)>{\nu }_c\left({\mathbf{x}}_i\right)\wedge {\nu }_c\left({\mathbf{x}}_i\right)>0\right.\right\}$$

(29)

${\mathbf{X}}_2^{({\mu }_c,{\nu }_c)}$ can be interpreted as candidate uncertain instances. For example, assume four unseen samples with pseudo-labels, i.e., ${\mathbf{x}}_{\mathit{A}}$, ${\mathbf{x}}_{\mathit{B}}$, ${\mathbf{x}}_{\mathit{C}}$ and ${\mathbf{x}}_{\mathit{D}}$ in Figure 4. It is more likely that ${\mathbf{x}}_{\mathit{B}}$ and ${\mathbf{x}}_{\mathit{C}}$ belong to ${\mathbf{X}}_2^{({\mu }_c,{\nu }_c)}$ than ${\mathbf{x}}_{\mathit{A}}$ and ${\mathbf{x}}_{\mathit{D}}$, and they are more close to the hyperplane. As instances with small separation margins tend to be more uncertain, label enhancement on ${\mathbf{x}}_{\mathit{C}}$ and ${\mathbf{x}}_{\mathit{D}}$ will be more informative than on ${\mathbf{x}}_{\mathit{A}}$ and ${\mathbf{x}}_{\mathit{B}}$, given the pseudo-label distribution on label $l_c$. One should be aware that ${\mathbf{x}}_{\mathit{C}}$ is less likely to be considered if we only apply the traditional fuzzy set model, as the affiliation degree to the positive class is much larger than that of the negative class. However, it will be conducive to enhancing generality if ${\mathbf{x}}_{\mathit{C}}$ is selected, as there are two instances with negative classes in its neighbours. Hence, we can select instance ${\mathbf{x}}_{\mathit{C}}$ via the intuitionistic fuzzy set.

3.6. Instance-Level Instance Selection

Now we determine which instances should be selected for the label enhancement. Since label enhancement works at the instance level, a straightforward notion is to select the instances that are uncertain across most labels. In this paper, we introduce the symbol $U{C}_i$ to represent the number of accumulated times that satisfy ${\mathbf{x}}_i\in {\mathbf{X}}_2^{({\mu }_c,{\nu }_c)}$ for any $l_c$. It is defined as:

$$U{C}_i=\sum _c\left[\left[{\mathbf{x}}_i\in {\mathbf{X}}_2^{({\mu }_c,{\nu }_c)}\right]\right]$$

(30)

where the notation $\left[\left[·\right]\right]$ equals 1 if the condition meets and 0 otherwise. Based on the $UC$, the instances to be enhanced (${\mathbf{X}}_2^{(\mu ,\nu )}$) can be computed as:

$${\mathbf{X}}_2^{(\mu ,\nu )}=\left\{{\mathbf{x}}_i\left|U{C}_i⩾\overline{U{C}_j}\wedge {\mathbf{x}}_j\in {\mathbf{X}}_2\wedge U{C}_j>0\right.\right\}$$

(31)

where the notation $\overline{U{C}_j}$ refers to the average non-zero $U{C}_j$. This means that an instance will be enhanced only when it is recognized as uncertain in most labels.

3.7. Complexity Analysis

We summarize the procedures of IFTWLE in Algorithm 1. The most time-consuming step is the calculation of pair-wise distances within all unseen instances (i.e., $r_c$). Let k be the average length of label-specific features, n be the unseen instance count, and m be the label count, then the calculation for $r_c$ requires $O\left(n^{2}mk\right)$. Therefore, the computational complexity of IFTWLE is much lower than that of training $f_1\left(·\right)$ [9] and $f_2\left(·\right)$ [16]. However, as the complexity is proportional to the quadratic of the instance count, it is inappropriate to employ this algorithm for large-scale multi-labels. The local estimation of the inner product should be considered to accelerate the instance selection.

Algorithm 1: IFTWLE

4. Experiments

4.1. Dataset Characteristics

To demonstrate the effectiveness and efficiency of the proposed model, we compared the classification performances on eight multi-label benchmarks from Mulan (http://mulan.sourceforge.net/datasets.html) (accessed on 1 March 2022) and Meka (http://waikato.github.io/meka/datasets/) (accessed on 1 March 2022) on domains including audio, music, text, biology, and images. The selected benchmarks were either small or moderate and were intensively referenced due to their limited baseline performances. In

Table 2. Characteristics of data sets.

Data Set	# Instances	# Features	# Labels	# Cardinality	Domain
birds	645	260	19	1.014	audio
emotions	593	72	6	1.869	music
enron	1702	1001	53	3.378	text
genbase	662	1185	27	1.252	biology
languagelog	1460	1004	75	1.18	text
medical	978	1449	45	1.245	text
scene	2407	294	6	1.074	image
yeast	2417	103	14	4.237	biology

, for each dataset, “# Instances” means the number of instances, “# Features” means the number of features, “# Labels” means the total number of class labels, and “# Cardinality” means the average number of labels per instance of a dataset, where notation “#” means the number of.

4.2. Evaluation Metrics

We adopted six evaluation metrics (Hamming Loss, Ranking Loss, One Error, Coverage, Average Precision, and Micro F1) [47] to evaluate the classification performance. Except for the last two, which achieve better performance when the values are larger, the remaining metrics obtain better performances when values are smaller. Let ${\mathbf{Y}}_{\mathbf{i}}$ and $\neg {\mathbf{Y}}_{\mathbf{i}}$ denote the relevant and irrelevant label sets in ground-truth, and n be the unseen instances count, then the formulas of metrics are enumerated as:

(1)

Hamming Loss (abbreviated as Hl) evaluates the average difference between predictions and ground-truth (see Formula (32)). The smaller the value of the Hamming Loss, the better the performance of the algorithm.

$$Hl=\frac{1}{n}\sum _{i=1}^n\frac{1}{l}Card\left(f\left({\mathbf{x}}_i\right)\Delta {\mathbf{Y}}_i\right)$$

(32)

where $\Delta$ is the set symmetric difference and $Card(·)$ is the set cardinality.

(2)

Ranking Loss (abbreviated as Rkl), evaluates the fraction that the irrelevant label ranks before the relevant label in the label predictions (see Formula (33)). The smaller the value of the Ranking Loss, the better the performance of the algorithm.

$$Rkl=\frac{1}{n}\sum _{i=1}^n\frac{1}{Card\left({\mathbf{Y}}_i\right)Card\left(\neg {\mathbf{Y}}_i\right)}\times Card\left(\left\{\left(l_a,l_b\right)\left|ran{k}_i\left(l_a\right)>ran{k}_i\left(l_b\right)\wedge \left(l_a,l_b\right)\in {\mathbf{Y}}_i\times \neg {\mathbf{Y}}_i\right.\right\}\right)$$

(33)

where $ran{k}_i\left(l_j\right)$ denotes the ranking position in ascending order for the j-th label on the i-th instance. $Card(·)$ is the set cardinality.

(3)

One Error (abbreviated as Oe) evaluates the average fraction that the label ranking—first in prediction—is actually the irrelevant label (see Formula (34)). The smaller the value of One Error, the better the performance of the algorithm.

$$Oe=\frac{1}{n}\sum _{i=1}^n\left[\left[\left(\underset{l_j}{argminran{k}_i\left(l_j\right)}\right)\notin {\mathbf{Y}}_i\right]\right]$$

(34)

where $\left[\left[·\right]\right]$ equals 1 if the condition holds and equals 0 otherwise. The operator $ran{k}_i\left(l_j\right)$ denotes the ranking position in ascending order for the j-th label on the i-th instance.

(4)

Coverage (abbreviated as Cvg) evaluates the average fraction for inclusion of all ground-truth labels in the ranking of label predictions (see Formula (35)). The smaller the value of Coverage, the better the performance of the algorithm.

$$Cvg=\frac{1}{n}\sum _{i=1}^n\underset{l_j\in {\mathbf{Y}}_i}{max}ran{k}_i\left(l_j\right)-1$$

(35)

where $ran{k}_i\left(l_j\right)$ denotes the ranking position in ascending order for the j-th label on the i-th instance.

(5)

Average Precision (abbreviated as Ap) evaluates the average precision of the actual relevant label rankings before relevant labels are examined by the label predictions (see Formula (36)). The larger the value of Average Precision, the better the performance of the algorithm.

$$Ap=\frac{1}{n}\sum _{i=1}^n\frac{1}{Card\left({\mathbf{Y}}_i\right)}\sum _{l_j\in {\mathbf{Y}}_i}\frac{Card\left(\{l_s\in {\mathbf{Y}}_i\left|ran{k}_i\left(l_s\right)⩽ran{k}_i\left(l_j\right)\right.\right\})}{ran{k}_i\left(l_j\right)}$$

(36)

where $Card(·)$ is the set cardinality.

4.3. Experimental Settings

We examined whether IFTWLE gained superiority over state-of-the-art algorithms learnt on logical labels only. For this reason, we compared IFTWLE with LLSF, multi-label k-nearest neighbour (MLkNN), multi-label learning with label-specific features (LIFT), MLTSVM, multi-label learning with global and local label correlation (Glocal), and active k-label set ensembles (ACkEL). Detailed settings are as follows.

• LLSF (code available at https://jiunhwang.github.io/ (accessed on 1 March 2022)) [9]: This method learns label-specific feature representations for all labels based on logical labels. It shares an identical structure with $f_1\left(·\right)$. Comparing this method, we could examine the contributions of label enhancement. Parameters $\delta$, $\eta$ are tuned in $\{2^{-10},2^{-9},\dots ,2^9,2^{10}\}$. The calibrated threshold ${\tau }_1$ is fixed as 0.5.
• MLkNN (code available at http://www.lamda.nju.edu.cn/code_MLkNN.ashx (accessed on 1 March 2022)) [48]: It learns a conditional probability distribution on all features within the adapted k-neighborhood. The introduction of the neighborhood has some similarities with the components in the non-membership function $\nu \left({\mathbf{x}}_i\right)$. However, we took one further step and enhanced the results of the uncertain instances. The value k took the empirical value 10.
• LIFT (code available at http://cse.seu.edu.cn/PersonalPage/zhangml/index.htm (accessed on 1 March 2022)) [40]: It learned different feature representations to determine the label association. It was the first trial of label-specific learning for multi-label classification. By comparing with this method, we could verify whether label enhancement improves label-specific learning. The ratio parameter is searched in $\{0.1,0.2,\dots ,0.5\}$.
• MLTSVM (code available at http://www.optimal-group.org/Resource/MLTSVM.html (accessed on 1 March 2022)) [11]: It learns distance difference based on multiple nonparallel hyperplanes. The twin support vector machine is a variant of the support vector machine; we trained the enhanced model by a linear support vector machine. The penalty and kernel parameter were searched in $\{2^{-6},2^{-5},\dots ,2^5,2^6\}$ and $\{2^{-4},2^{-3},\dots ,2^3,2^4\}$, respectively.
• Glocal (code available at http://www.lamda.nju.edu.cn/code_Glocal.ashx (accessed on 1 March 2022)) [49]: It learns a mapping from the feature space to latent labels via a low-rank decomposition. It is the initial attempt to simultaneously leverage both global and local label correlations. The similarity is that we also consider global label correlation in a pairwise fashion. By comparing with this work, we can examine whether label importance is superior to local label correlation. The penalty $\lambda$ takes the empirical value 1.
• ACkEL (code available at https://github.com/xuwangfmc/AkEL (accessed on 1 March 2022)) [50]: It takes an ensemble strategy on the k-label set optimized by class separability and class uncertainty simultaneously. It is a revised version of the classic k-label set algorithm, which is assumed to gain a robust performance. By comparing with this work, we can examine whether the strategy, combining the pairwise label correlation and label importance, could gain superiority over a high-order label correlation. A one-versus-all multi-class strategy was conducted on each label set; parameters $\sigma$, $\beta$ were searched in $\{2^{-3},2^{-2},\dots ,2^{10},2^{11}\}$ and $\{0.1,0.3,0.5,0.7,0.9\}$, respectively. The size for each label set was fixed as 3.
• Proposed method. There are three groups of parameters. The parameters in constructing $f_1(·)$ and $f_2(·)$ take the recommended settings, as declared in [9] and [16]. For an intuitionistic fuzzy membership assignment, all components were automatically determined by data characteristics except for neighborhood radius r. It was searched in $\{0.01,0.1\}$ via five-fold cross-validation.

We considered five evaluations [47] including Hamming Loss, One Error, Coverage, Ranking Loss, and Average Precision. Except for Average Precision, which obtains better performance if the metric becomes larger, the others achieve better performances if the metrics become smaller. The experiments were implemented using Matlab R2017b on a desktop PC with an Intel(R) Core i7 processor (2.60 GHz) and 8 GB of RAM. All parameters were selected via five-fold cross-validation.

4.4. Results

We evaluated the classification performances, considering algorithms on five evaluation metrics; we report them in Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9. The down arrow ↓ means the smaller the metrics is, the better the algorithm performance becomes. In contrast, the up arrow ↑ means the larger the metrics is, the better the algorithm performance becomes.

From the metric view, IFTWLE ranks in first place in 60% of cases ($\frac{3}{5}$) and in second place in 40% of cases. From the dataset view, IFTWLE ranks in first place in 42.5% of cases ($\frac{17}{40}$) and in second place in 17.5% of cases ($\frac{7}{40}$). It receives the best performance on the Coverage metric (with first place in 87.5% cases) and worst on the One Error metric (with second place in 75% of cases).

The Friedman test [51] was employed to calculate the relative performances among multiple algorithms over selected datasets. Given k comparing algorithms and N datasets, let $Ran{k}_j=\left(1/N\right){\sum }_{i=1}^N{r}_i^j$ denote the average rank for the j-th algorithm. With the null hypothesis (i.e., $H_0$) of all algorithms obtaining identical performance, the Friedman statistic $F_F$ is distributed according to the F-distribution with the $k-1$ degree of freedom as the numerator and $(k-1)(N-1)$ degree of freedom as the denominator, denoted as:

$$F_F=\frac{\left(N-1\right){\chi }_F^2}{N\left(k-1\right)-{\chi }_F^2}$$

(37)

where

$${\chi }_F^2=\frac{12N}{k\left(k+1\right)}\left[\sum _{j}Ran{k}_j^2-\frac{k{\left(k+1\right)}^2}{4}\right]$$

(38)

Table 3. Summary of the Friedman statistics $F_F$ ($k=7$, $N=8$) and critical values in terms of each evaluation measure (k:# comparing algorithms; N:# data sets).

Metric	${\mathit{F}}_{\mathit{F}}$	Critical Value ($\mathit{\alpha }=0.05$)
Hamming Loss	9.991071	2.3239
Ranking Loss	27.816964
One Error	33.214286
Coverage	41.852679
Average Precision	19.473214

presents the Friedman statistics $F_F$ and the corresponding critical value for all evaluation metrics in this setting. The results clearly support that, at the significance level, $\alpha =0.05$, the null hypothesis (i.e., $H_0$) of the statistically indistinguishable performance of all algorithms on the considered metrics is rejected. It implies that it is feasible to examine if IFTWLE gains statistical superiority over other comparing algorithms by conducting a post hoc test, such as the Holm [51].

Furthermore, by regarding IFTWLE as the control algorithm, we employed the Holm procedure [51] to explore whether IFTWLE gains significant a performance difference against each of the considered algorithms. Without losing generality, we nominate $A_1$ as IFTWLE. For the other $k-1$ comparing algorithms (i.e., $A_j\left(2⩽j⩽k\right)$), we stipulate $A_j$ as the one that has the $j-1$-th largest average ranking over all datasets on a specific evaluation metric. Consequently, we have the test statistic for comparing $A_1$ (i.e., IFTWLE) with $A_j$ as:

$$z_j=\left(Ran{k}_1-Ran{k}_j\right)/\phantom{\left(Ran{k}_1-Ran{k}_j\right)\sqrt{\frac{k\left(k+1\right)}{6N}}}\sqrt{\frac{k\left(k+1\right)}{6N}}\left(2\le j\le k\right)$$

(39)

Let $p_j$ denote the p-value of $z_j$ under a normal distribution. Given the significance level $\alpha =0.05$, the Holm procedure works in a stepwise manner by checking whether the statistics $p_j$ are smaller than $\alpha /\left(k-j+1\right)$ in ascending order of j. Specifically, the Holm procedure continues until there exists a $j^{*}$-th step, and $j^{*}$ denotes the first j, such that $p_j⩾\alpha /\left(k-j+1\right)$ holds (If $p_j<\alpha /\left(k-j+1\right)$ holds for all j, $j^{*}$ takes the value of $k+1$).

It can be seen in

Table 4. Comparison of IFTWLE (control algorithm) against other comparing algorithms (with the Holm procedure as the post hoc test) at the significant level $\alpha =0.05$ on the Hamming Loss metric.

j	Algorithm	${\mathit{z}}_{\mathit{j}}$	p	Holm
2	Glocal	−1.9674	0.0491	0.008
3	LLSF	−1.4466	0.1480	0.010
4	MLTSVM	−0.4629	0.6434	0.013
5	MLkNN	−0.3472	0.7284	0.017
6	ACkEL	−0.1736	0.8622	0.025
7	LIFT	0.7522	1.0000	0.050

,

Table 5. Comparison of IFTWLE (control algorithm) against other comparing algorithms (with the Holm procedure as the post hoc test) at the significance level $\alpha =0.05$ on the Ranking Loss metric. Algorithms that are statistically inferior to IFTWLE are in bold.

j	Algorithm	${\mathit{z}}_{\mathit{j}}$	p	Holm
2	AC$\mathit{k}$EL	−4.281918	0.000019	0.008
3	MLTSVM	−3.587553	0.000334	0.010
4	LLSF	−1.562321	0.118212	0.013
5	Glocal	−1.446594	0.148011	0.017
6	MLkNN	−1.041548	0.297621	0.025
7	LIFT	−1.041548	0.297621	0.050

,

Table 6. Comparison of IFTWLE (control algorithm) against other comparing algorithms (with the Holm procedure as the post hoc test) at the significance level $\alpha =0.05$ on the One Error metric. Algorithms that are statistically inferior to IFTWLE are in bold.

j	Algorithm	${\mathit{z}}_{\mathit{j}}$	p	Holm
2	ACkEL	−3.1246	0.0018	0.008
3	LLSF	−0.9258	0.3545	0.010
4	MLkNN	−0.8101	0.4179	0.013
5	Glocal	−0.6944	0.4874	0.017
6	LIFT	0.9258	1.0000	0.025
7	MLTSVM	2.1988	1.0000	0.050

,

Table 7. Comparison of IFTWLE (control algorithm) against other comparing algorithms (with the Holm procedure as the post hoc test) at the significance level $\alpha =0.05$ on the Coverage metric. Algorithms that are statistically inferior to IFTWLE are in bold.

j	Algorithm	${\mathit{z}}_{\mathit{j}}$	p	Holm
2	MLTSVM	−4.802692	0.000002	0.008
3	Glocal	−4.339782	0.000014	0.010
4	LIFT	−3.529689	0.000416	0.013
5	ACkEL	−3.2982345	0.000973	0.017
6	MLkNN	−1.678049	0.093338	0.025
7	LLSF	−0.578638	0.562834	0.050

and

Table 8. Comparison of IFTWLE (control algorithm) against other comparing algorithms (with the Holm procedure as the post hoc test) at the significance level $\alpha =0.05$ on the Average Precision metric. Algorithms that are statistically inferior to IFTWLE are in bold.

j	Algorithm	${\mathit{z}}_{\mathit{j}}$	p	Holm
2	ACkEL	−3.992599	0.000134	0.008
3	MLTSVM	−1.85164	0.064078	0.010
4	LLSF	−1.504458	0.132464	0.013
5	MLkNN	−1.273003	0.203017	0.017
6	LIFT	−0.752229	0.451913	0.025
7	Glocal	−0.752229	0.451913	0.050

that IFTWLE statistically outperforms ACkEL on the Ranking Loss, Coverage, and Average Precision metrics, and statistically outperform MLTSVM on the Ranking Loss and Coverage metrics. IFTWLE achieves the most dominance on Coverage, which is statistically superior over all algorithms except MLkNN and LLSF. By finding instances with the most uncertain labels, it is more likely to revise a large proportion of misclassified labels, gaining more improvements in the Coverage metric. However, this strategy does not discriminate concerning the relative importance of labels in different instances, which lead to limited improvements in the One Error and Average Precision metrics.

5. Conclusions

This paper proposes a novel model called IFTWLE to deal with multi-label classification. Unlike conventional multi-label learning algorithms, which learn models on either logical labels or numerical labels, we integrated the two forms of labels under the three-way decisions umbrella by exploring classification uncertainty. The intuitionistic fuzzy set provides an insightful view into quantifying the label level uncertainty and it determines the uncertain instances in a group decision-making fashion. Comparisons on benchmarks demonstrate that IFTWLE significantly improves classification performance.

In the future, we will examine more combinations of label-specific algorithms and label enhancement algorithms to see whether some guidelines exist. Meanwhile, we will develop advanced instance selection principles by resorting to the optimization theory.