A Quadratic Surface Minimax Probability Machine for Imbalanced Classification

Abstract

In this paper, a kernel-free minimax probability machine model for imbalanced classification is proposed. In this model, a quadratic surface is adopted directly for separating the data points into two classes. By using two symmetry constraints to define the two worst-case classification accuracy rates, the model of maximizing both the $F_1$ value of the minority class and the classification accuracy rate of all the data points is proposed. The proposed model corresponds to a fractional programming problem. Since the two worst-case classification accuracy rates are the symmetry, the proposed model can be further simplified. After this, the alternating descent algorithm is adopted for efficiently solving. The proposed method reduces the computational costs by both using the kernel-free technique and adopting the efficient algorithm. Some numerical tests on benchmark datasets are conducted to investigate the classification performance of the proposed method. The numerical results demonstrate that the proposed method performs better when compared with the other state-of-the-art methods, especially for classifying the imbalanced datasets. The better performance for the imbalanced classification is also demonstrated on a Wholesale customers dataset. This method can provide methodological support for the research in areas such as customer segmentation.

1. Introduction

Binary classification is an important branch in the field of machine learning. Minimax probability machine (MPM), which is first proposed in [1], is a competitive machine learning method for binary classification problems. It directly maximizes the correct classification probability of the worst-case with a given mean and covariance matrix [1]. In other words, MPM provides a clear lower bound on prediction accuracy without making any assumptions on data distribution. The robustness of the MPM-based methods have been shown with successful applications in the fields such as medical diagnosis [2], anomaly detection [3], and customer churn prediction [4].

Since the role of two classes are often symmetry, the same kind of modeling technique can be used. In the traditional MPM model, the lower bounds of correct classification probability for two symmetry categories are defined to be the same [1]. It may not be effective enough when a certain class of samples needs more attention. The minimum error minimax probability machine (MEMPM) [5] learns the classifier by maximizing the lower bound of the weighted correct classification probability of two categories, which provides different approaches to different classes of samples. After setting the weight to be $1/2$, a biased minimax probability machine is proposed in [6] to deal with the problem of imbalanced learning. Although the balanced parameter in MEMPM can deal with the different importance of two classes to some extent, only the accuracy rate is adopted as the objective function. Luo et al. [7] developed a new minimax probability machine for the $F_{\beta }$ measure (MPMF), which provided a new perspective for the classification on imbalanced datasets. By using the lower bound of the correct classification probability of the two classes, the $F_{\beta }$ measure is defined as the objective function. To solve the MPMF model effectively, its equivalent form, which can be solved by an alternating descent method to learn a linear classifier, is derived. The kernel trick is employed to derive a nonlinear MPMF model to learn a nonlinear classifier. Although MPMF provides a good perspective for dealing with imbalanced datasets, the performance also depends on the choice of kernel functions and kernel parameters, a process that consumes a lot of time.

Some other MPM-based models have also been studied. Nath et al. proposed a maximum margin classifier [8] by considering the structural risk of the model. Maldonado et al. [9] proposed the regularized MPM and regularized MEMPM models, which minimized both the $l_2$ norm of the weight vector and the upper bound on the classification error rate. Drawing on the idea of twin support vector machines [10], Yang et al. [11] proposed the twin minimax probability machine model, which attempts to construct a pair of non-parallel decision hyperplanes such that each hyperplane separates one class of samples with a lower bound on the classification accuracy and is far from the other classes of samples. Although these methods show their good performance for classification, kernel functions are all needed to deal with nonlinear classification problems. This process may run into the efficiency issues of choosing kernel functions and their corresponding parameters.

For dealing with the issues of choosing a suitable kernel function and its optimal parameter, Dagher proposed a kernel-free quadratic surface support vector machine model [12] for binary classification. Luo et. al. extended it to a soft-margin quadratic surface SVM (SQSSVM) model [13]. Furthermore, more kernel-free SVM models are proposed in the literature. Bai et. al. proposed a new quadratic kernel-free least squares support vector machine for a binary classification problem [14]. Yan et. al. proposed a kernel-free semi-supervised quadratic surface support vector machine model [15]. Tian et. al. proposed a new approach based on the kernel-free quadratic surface support vector machine model to handle a binary classification problem with mislabeled information [16]. Luo et. al. proposed an unsupervised quadratic surface support vector machine and applied it to credit risk assessment [17]. Gao et. al. proposed a kernel-free soft quartic surface SVM model by utilizing the double well potential function for highly nonlinear binary classification [18]. Yan et. al. analyzed the equivalent form of the soft quadratic surface support vector machine and proposed a novel support vector machine model with feature mapping [19]. Based on the idea of a soft quadratic surface support vector machine [13], Wang et al. [20] proposed a kernel-free quadratic surface minimax probability machine (QSMPM), which attempts to find a quadratic surface to separate two classes of samples. QSMPM does not require use of the nonlinear mapping for dealing with nonlinear separable datasets, which saves a lot of time for tuning the parameters. Nevertheless, the QSMPM model still aims to maximize the lower bound of the correct classification probability of two classes, which is not suitable enough for dealing with imbalanced classification.

The existing minimax probability machines only focus on kernel-free schemes or imbalanced classification. Meanwhile, the existing optimization goal considers only one certain classification performance measure. For efficiently dealing with imbalanced classification problems, a new quadratic surface minimax probability machine model with the optimization goal that considers two measures (QSMPMFA) is proposed. The optimization goal consists of both the $F_1$ value of the minority class and the weight classification accuracy rate of the two classes. In this model, QSMPMFA tries to find a quadratic surface to separate the samples into two classes. Two symmetry constraints are used to define the lower bound of the probability of different classes of samples being correctly classified, i.e., worst-case classification accuracy rates. Since two worst-case classification accuracy rates are symmetry in the constraint, the proposed model can be further simplified. The original optimization problem of the QSMPMFA can then be transformed into an equivalent form, which can be solved by the alternating descent method. The advantages can be summarized as follows.

First, QSMPMFA is optimized by using the combination of the $F_1$ value and accuracy rate in the objective function, instead of considering only one certain classification performance measure, causing it to be better for handling imbalanced datasets.

Second, the proposed model can be further simplified by using the symmetry structure in the constraint. The alternating descent algorithm is adopted for efficiently solving.

Third, the numerical tests of QSMPMFA on several benchmark datasets show that QSMPMFA outperforms the other state-of-the-art methods in terms of both the $F_1$ value and accuracy rate, especially for the imbalanced datasets.

This paper is structured as follows. In Section 2, the related studies are briefly reviewed, including MPM, SQSSVM, and the formulation of the $F_1$ value. Section 3 provides the details of the proposed QSMPMFA model and the corresponding algorithm. Section 4 shows the numerical tests on different kinds of datasets and illustrates the comparison performance of the proposed method with other methods. Section 5 provides some conclusions and future work.

2. Related Work

This section provides a brief introduction to the minimax probability machine, soft quadratic surface support vector machine, and the performance measures.

Consider the dataset of $T=\{({\mathit{x}}_1,y_1),({\mathit{x}}_2,y_2),\dots ,({\mathit{x}}_l,y_l)\}$, where ${\mathit{x}}_i\in {\mathbb{R}}^d$ refers to the feature vector of the i-th sample and $y_i\in \{+1,-1\}$ represents this sample’s class label for $i=1,2,\dots ,l$.

2.1. Minimax Probability Machine (MPM)

MPM attempts to separate the two classes of data points with the goal of maximizing a lower bound on the classification accuracy rate given the mean and covariance matrix of each class of data.

Let $\mathit{x}\sim (\mathbf{\mu },\Sigma )$ be the random variable $\mathit{x}$ with mean $\mathbf{\mu }$ and covariance matrix $\Sigma$. Suppose the positive class sample ${\mathit{x}}_{+}\sim ({\mathbf{\mu }}_{+},{\Sigma }_{+})$, the negative class sample ${\mathit{x}}_{-}\sim ({\mathbf{\mu }}_{-},{\Sigma }_{-})$, ${\mathbf{\mu }}_{+}$, ${\mathbf{\mu }}_{-}\in {\mathbb{R}}^d$, and ${\Sigma }_{+}$, ${\Sigma }_{-}\in {\mathbb{S}}_{+}^d$. Using MPM tries to find the hyperplane:

$$\begin{array}{c}\hfill g\left(\mathit{x}\right)={\mathit{w}}^T\mathit{x}-b=0,\end{array}$$

(1)

where $\mathit{w}\in {\mathbb{R}}^d$, $b\in \mathbb{R}$. The optimization problem of MPM can be formulated as following:

$$\begin{array}{cc}\hfill \underset{\mathit{w},b,\alpha }{max}& \alpha \hfill \\ \hfill s.t.& \underset{{\mathit{x}}_{+}\sim ({\mathbf{\mu }}_{+},{\Sigma }_{+})}{inf}Pr\{{\mathit{w}}^T{\mathit{x}}_{+}\ge b\}\ge \alpha ,\hfill \\ \hfill & \underset{{\mathit{x}}_{-}\sim ({\mathbf{\mu }}_{-},{\Sigma }_{-})}{inf}Pr\{{\mathit{w}}^T{\mathit{x}}_{-}\le b\}\ge \alpha .\hfill \end{array}$$

(2)

In Problem (2), $\alpha \in (0,1)$ is the lower bound on the probability of the two classes of samples being correctly classified, which can also be explained as the worst-case classification accuracy rate. The structure of the two constraints are symmetry. By maximizing $\alpha$ in the objective function, the hyperplane with the best lower bound can be chosen.

Problem (2) is an optimization problem with probability constraints, which causes the problem to be difficult to deal with. The following two Lemmas are introduced for transformation.

Lemma 1. 

Suppose $\mathit{x}$ is a random vector, $\mathit{S}$ is a given convex set, and $\mathit{x}\sim (\mathbf{\mu },\Sigma )$, then the supremum on the probability of $\mathit{x}\in S$ is equivalent to:

$$\begin{array}{c}\hfill \underset{\mathit{x}\sim (\mathbf{\mu },\Sigma )}{sup}Pr\{\mathit{x}\in \mathit{S}\}=\frac{1}{1+d^2},d^2=\underset{\mathit{x}\in \mathit{S}}{inf}{(\mathit{x}-\mathbf{\mu })}^T{\Sigma }^{-1}(\mathit{x}-\mathbf{\mu }).\end{array}$$

(3)

Lemma 2. 

Given $\mathit{w}\ne \mathbf{0},b,\mathbf{\mu },\Sigma$, the following condition:

$$\begin{array}{c}\hfill \underset{\mathit{x}\sim (\mathbf{\mu },\Sigma )}{inf}Pr\{{\mathit{w}}^T\mathit{x}\le b\}\ge \alpha \end{array}$$

(4)

holds if and only if:

$$\begin{array}{c}\hfill b-{\mathit{w}}^T\mathit{x}\ge k\left(\alpha \right)\sqrt{{\mathit{w}}^T\Sigma \mathit{w}},\end{array}$$

(5)

where $k\left(\alpha \right)=\sqrt{\frac{\alpha }{1-\alpha }}$.

It is obvious that $k\left(\alpha \right)$ is a monotonically increasing function of $\alpha$, so the original optimization problem of MPM can be transformed into the following formulation according to the above two lemmas.

$$\begin{array}{cc}\hfill \underset{\mathit{w},b,\alpha }{max}& k\left(\alpha \right)\hfill \\ \hfill s.t.& -b+{\mathit{w}}^T{\mathbf{\mu }}_{+}\ge k\left(\alpha \right)\sqrt{{\mathit{w}}^T{\Sigma }_{+}\mathit{w}},\hfill \\ \hfill & b-{\mathit{w}}^T{\mathbf{\mu }}_{-}\ge k\left(\alpha \right)\sqrt{{\mathit{w}}^T{\Sigma }_{-}\mathit{w}}.\hfill \end{array}$$

(6)

Problem (6) can be formulated as a second-order cone programming problem as the following:

$$\begin{array}{cc}\hfill \underset{\mathit{w}}{min}& \left|\right|{\Sigma }_{+}^{\frac{1}{2}}{\mathit{w}\left|\right|}_2+\left|\right|{\Sigma }_{-}^{\frac{1}{2}}\mathit{w}{\left|\right|}_2\hfill \\ \hfill s.t.& {\mathit{w}}^T({\mathbf{\mu }}_{+}-{\mathbf{\mu }}_{-})=1,\hfill \end{array}$$

(7)

which can be solved by the interior point method. With the optimal value of ${\mathit{w}}^{*}$, the optimal value of $b^{*}$ can be obtained by:

$$\begin{array}{cc}\hfill b^{*}& ={\left({\mathit{w}}^{*}\right)}^T{\mathbf{\mu }}_{-}+\frac{\parallel {\Sigma }_{-}^{\frac{1}{2}}{\mathit{w}}^{*}{\parallel }_2}{\parallel {\Sigma }_{-}^{\frac{1}{2}}{\mathit{w}}^{*}{\parallel }_2+{\parallel {\Sigma }_{+}^{\frac{1}{2}}{\mathit{w}}^{*}\parallel }_2}\hfill \\ \hfill & ={\left({\mathit{w}}^{*}\right)}^T{\mathbf{\mu }}_{+}-\frac{\parallel {\Sigma }_{+}^{\frac{1}{2}}{\mathit{w}}^{*}{\parallel }_2}{\parallel {\Sigma }_{+}^{\frac{1}{2}}{\mathit{w}}^{*}{\parallel }_2+{\parallel {\Sigma }_{-}^{\frac{1}{2}}{\mathit{w}}^{*}\parallel }_2}.\hfill \end{array}$$

(8)

A new data point $\mathit{x}\in {\mathbb{R}}^d$ can be assigned to positive class or negative class by calculating:

$$\begin{array}{c}\hfill f\left(\mathit{x}\right)=sign\left({\left({\mathit{w}}^{*}\right)}^T\mathit{x}+b^{*}\right).\end{array}$$

(9)

For more details, please refer to [21].

2.2. Soft Quadratic Surface Support Vector Machine (SQSSVM)

The SQSSVM model intends to find the optimal parameter set $(\mathit{W},\mathit{b},c)$ of a quadratic surface:

$$\frac{1}{2}{\mathit{x}}^T\mathit{W}\mathit{x}+{\mathit{b}}^T\mathit{x}+c=0,$$

where $\mathit{W}={\mathit{W}}^T\in {\mathbb{R}}^{d\times d},\mathit{b}\in {\mathbb{R}}^d$, and $c\in \mathbb{R}$, which separates the training points into two classes by maximizing the distance from each data point to the separating surface. The SQSSVM model is as follows:

$$\begin{array}{cc}\hfill & \underset{\mathit{W},\mathit{b},c}{min}\sum _{i=1}^l{\parallel \mathit{W}{\mathit{x}}_{\mathbf{i}}+\mathit{b}\parallel }_2^2+C\sum _{i=1}^l{\xi }_i\hfill \\ \hfill & s.t.y_i(\frac{1}{2}{\mathit{x}}_i^T\mathit{W}{\mathit{x}}_i+{\mathit{b}}^T{\mathit{x}}_i+c)\ge 1-{\xi }_i,,i=1,\dots ,l,\hfill \\ \hfill & {\xi }_i\ge 0,i=1,\dots ,l,\hfill \end{array}$$

(10)

where $C>0$ is the penalty parameter.

The matrix $\mathit{W}=\left({\omega }_{ij}\right)$ can be vectorized to

$$\begin{array}{c}\hfill \overline{\mathit{W}}={[{\omega }_{11}{\omega }_{12}\dots {\omega }_{1d}{\omega }_{22}{\omega }_{23}\dots {\omega }_{2d}\dots {\omega }_{dd}]}^T\in {\mathbb{R}}^{\frac{d^2+d}{2}}.\end{array}$$

(11)

Denote $\mathit{z}={({\overline{\mathit{W}}}^T,{\mathit{b}}^T)}^T$. Problem (10) can be reformulated as follows:

$$\begin{array}{cc}\hfill \underset{\mathit{z},c,\mathbf{\xi }}{min}& {\mathit{z}}^T\widehat{\mathit{G}}\mathit{z}+C\sum _{i=1}^l{\xi }_i\hfill \\ \hfill s.t.& y_i({\mathit{s}}_i^T\mathit{z}+c)\ge 1-{\xi }_i,i=1,\dots ,l,\hfill \\ \hfill & {\xi }_i\ge 0,i=1,\dots ,l.\hfill \end{array}$$

(12)

In Problem (12), $\widehat{\mathit{G}}={\sum }_{i=1}^n{\left([{\mathit{M}}^i,\mathit{I}]\right)}^T[{\mathit{M}}^i,\mathit{I}]\in {\mathbb{R}}^{\left(\frac{d^2+3d}{2}\right)\times \left(\frac{d^2+3d}{2}\right)}$, where $\mathit{I}$ is the m-dimensional identity matrix. ${\mathit{M}}^i\in {\mathbb{R}}^{d\times \left(\frac{d^2+d}{2}\right)}$ is constructed by using the training point ${\mathit{x}}_i=[x_{i1},x_{i2},\dots ,$$x_{id}{]}^T\in {\mathbb{R}}^d$ as follows. For the j-th row of ${\mathit{M}}^i$ in ${\mathbb{R}}^{\frac{d^2+d}{2}},j=1,\dots ,d,$ check the elements of $\overline{\mathit{W}}$ one by one. If the p-th element of $\overline{\mathit{W}}$ is ${\omega }_{jk}$ or ${\omega }_{kj}$ for some $k=1,2,\dots ,d$, then assign the p-th element of the j-th row of ${\mathit{M}}^i$ to be $x_{ik}$. Otherwise, assign it to be 0. Besides,

$$\begin{array}{cc}\hfill {\mathit{s}}_i=& [\frac{1}{2}{x}_{i1}{x}_{i1},\dots ,x_{i1}{x}_{id},\frac{1}{2}{x}_{i2}{x}_{i2},\dots ,x_{i2}{x}_{id},\dots ,\frac{1}{2}{x}_{i,d-1}{x}_{i,d-1},x_{i,d-1}{x}_{id},\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & \frac{1}{2}{x}_{id}{x}_{id},x_{i1},x_{i2},\dots ,x_{id}]\in {\mathbb{R}}^{\frac{d^2+d}{2}+d}.\hfill \end{array}$$

(14)

Problem (12) is a convex quadratic programming problem. With the optimal value of ${\mathit{z}}^{*}$ and $c^{*}$, the parameter set of $({\mathit{W}}^{*},{\mathit{b}}^{*},c^{*})$ can then be obtained. A new data point $\mathit{x}\in {\mathbb{R}}^n$ can be assigned to positive class or negative class by calculating

$$\begin{array}{c}\hfill f\left(\mathit{x}\right)=sign\left(\frac{1}{2}{\mathit{x}}^T{\mathit{W}}^{*}\mathit{x}+{\left({\mathit{b}}^{*}\right)}^T\mathit{x}+c^{*}\right).\end{array}$$

(15)

For more details, please refer to [13].

2.3. $F_1$ Value

In this subsection, some performance measures are introduced. The true positive rate ($TPR$) and true negative rate ($TNR$) can be defined as follows.

$$\begin{array}{cc}\hfill & TPR=1-FNR=\mathit{P}(\widehat{y}=1\mid y=1),\hfill \\ \hfill & TNR=1-FPR=\mathit{P}(\widehat{y}=-1\mid y=-1),\hfill \end{array}$$

(16)

where y is the true value of the label of the data, $\widehat{y}$ is the predicted value of the label of the data, $FNR$ is the false negative rate, and $FPR$ is the false positive rate. Then the precision rate can be defined as:

$$\begin{array}{c}\hfill Precision=\mathit{P}(y=1\mid \widehat{y}=1)=\frac{p·TPR}{p·TPR+(1-p)·FPR},\end{array}$$

(17)

where p denotes the proportion of positive class samples.

$TPR$ is more concerned with the comprehensiveness for the recognition of positive class samples, while $Precision$ is more concerned with the accuracy rate of the recognition of positive class samples. Since the $TPR$ and $Precision$ are two contradictory performance measures, some combinations are proposed. The $F_1$ value is the harmonic mean of $TPR$ and $Precision$, i.e.:

$$\begin{array}{cc}\hfill F_1& =\frac{2·TPR·Precision}{Precision+TPR}.\hfill \end{array}$$

(18)

Since the classification of positive class samples has a great influence on the $F_1$ value, so the $F_1$ value is commonly used to evaluate the classification effect of imbalanced datasets. For more details, please refer to [7].

3. A Quadratic Surface Minimax Probability Machine with a Mixed Performance Measure

In this section, a quadratic surface minimax probability machine with a mixed performance measure (QSMPMFA) is proposed. The mixed performance measure considers both the $F_1$ value and the weighted accuracy rate. The $F_1$ measure is adopted for paying more attention on the class with less points. The weighted accuracy rate is adopted for balancing the importance between two classes. For better understanding, the proposed QSMPMFA is first introduced. After this, the alternating descend method for solving this problem is constructed.

3.1. The Proposed Model

Suppose the positive class sample ${\mathit{x}}_{+}\sim ({\mathbf{\mu }}_{+},{\Sigma }_{+})$ and the negative class sample ${\mathit{x}}_{-}\sim ({\mathbf{\mu }}_{-},{\Sigma }_{-})$, ${\mathbf{\mu }}_{+}$, ${\mathbf{\mu }}_{-}\in {\mathbb{R}}^d$, ${\Sigma }_{+}$, ${\Sigma }_{-}\in {\mathbb{S}}_{+}^d$. Assume that the mean and covariance matrix of the positive and negative samples are all reliable.

For obtaining a robust classifier for imbalanced datasets, the improved minimax probability machine based on the soft quadratic surface support vector machine is proposed first as follows.

$$\begin{array}{cc}\hfill \underset{\mathit{W},\mathit{b},c,{\alpha }_1,{\alpha }_2}{min}& \frac{1}{F_1}+C((1-p){\alpha }_1+p{\alpha }_2)\hfill \\ \hfill s.t.& \underset{{\mathit{x}}_{+}\sim ({\mathbf{\mu }}_{+},{\Sigma }_{+})}{inf}Pr\{\frac{1}{2}{\mathit{x}}_{+}^T\mathit{W}{\mathit{x}}_{+}+{\mathit{b}}^T{\mathit{x}}_{+}-c\ge 0\}\ge 1-{\alpha }_1,\hfill \\ \hfill & \underset{{\mathit{x}}_{-}\sim ({\mathbf{\mu }}_{-},{\Sigma }_{-})}{inf}Pr\{\frac{1}{2}{\mathit{x}}_{-}^T\mathbf{A}{\mathit{x}}_{-}+{\mathit{b}}^T{\mathit{x}}_{-}-c\le 0\}\ge 1-{\alpha }_2,\hfill \end{array}$$

(19)

where ${\alpha }_1$ denotes the classification error rate of the positive class, ${\alpha }_2$ denotes the classification error rate of negative class, C is the penalty parameter, and p denotes the proportion of positive class points. It can be seen that ${\alpha }_1$ is the false negative rate (FNR), while ${\alpha }_2$ is the false positive rate (FPR) mentioned in Section 2.3. The first constraint in Problem (19) guaranteed that the infimum of the probability that the positive class sample ${\mathit{x}}_{+}$ is above the separating quadratic surface was not less than $1-{\alpha }_1$. The second constraint in Problem (19) guaranteed that the infimum of the probability that the positive class sample ${\mathit{x}}_{-}$ behind the separating quadratic surface was not less than $1-{\alpha }_2$.

Since ${\alpha }_1=FNR$ and ${\alpha }_2=FPR$, the first item in the objective function can be represented as follows:

$$\begin{array}{c}\hfill \frac{1}{F_1}=\frac{(1-p){\alpha }_2+p}{2p(1-{\alpha }_1)}+\frac{1}{2}.\end{array}$$

(20)

By omitting the constant, Problem (19) can be transformed to the following problem:

$$\begin{array}{cc}\hfill \underset{\mathit{W},\mathit{b},c,{\alpha }_1,{\alpha }_2}{min}& \frac{(1-p){\alpha }_2+p}{2p(1-{\alpha }_1)}+C((1-p){\alpha }_1+p{\alpha }_2)\hfill \\ \hfill s.t.& \underset{{\mathit{x}}_{+}\sim ({\mathbf{\mu }}_{+},{\Sigma }_{+})}{inf}Pr\{\frac{1}{2}{\mathit{x}}_{+}^T\mathit{W}{\mathit{x}}_{+}+{\mathit{b}}^T{\mathit{x}}_{+}-c\ge 0\}\ge 1-{\alpha }_1,\hfill \\ \hfill & \underset{{\mathit{x}}_{-}\sim ({\mathbf{\mu }}_{-},{\Sigma }_{-})}{inf}Pr\{\frac{1}{2}{\mathit{x}}_{-}^T\mathbf{A}{\mathit{x}}_{-}+{\mathit{b}}^T{\mathit{x}}_{-}-c\le 0\}\ge 1-{\alpha }_2.\hfill \end{array}$$

(21)

Use the definition of $\overline{\mathit{W}}$ in formulation (11) and denote $\mathit{z}={({\overline{\mathit{W}}}^T,{\mathit{b}}^T)}^T$. Problem (21) can be reformulated as follows:    

$$\begin{array}{cc}\hfill \underset{\mathit{z},c,{\alpha }_1,{\alpha }_2}{min}& \frac{(1-p){\alpha }_2+p}{2p(1-{\alpha }_1)}+C((1-p){\alpha }_1+p{\alpha }_2)\hfill \\ \hfill s.t.& \underset{{\mathit{s}}_{+}\sim ({\mathbf{\mu }}_p,{\Sigma }_p)}{inf}Pr\{{\mathit{z}}^T{\mathit{s}}_{+}-c\ge 0\}\ge 1-{\alpha }_1,\hfill \\ \hfill & \underset{{\mathit{s}}_{-}\sim ({\mathbf{\mu }}_n,{\Sigma }_n)}{inf}Pr\{{\mathit{z}}^T{\mathit{s}}_{-}-c\le 0\}\ge 1-{\alpha }_2,\hfill \end{array}$$

(22)

where

$$\begin{array}{cc}\hfill {\mathit{s}}_{+}=& [\frac{1}{2}{x}_{+1}{x}_{+1},\dots ,x_{+1}{x}_{+d},\frac{1}{2}{x}_{+2}{x}_{+2},\dots ,x_{+2}{x}_{+d},\dots ,\frac{1}{2}{x}_{+,d-1}{x}_{+,d-1},\hfill \\ \hfill & x_{+,d-1}{x}_{+d},\frac{1}{2}{x}_{+d}{x}_{+d},x_{+1},x_{+2},\dots ,x_{+d}]\in {\mathbb{R}}^{\frac{d^2+d}{2}+d},\hfill \end{array}$$

(23)

and

$$\begin{array}{cc}\hfill {\mathit{s}}_{-}=& [\frac{1}{2}{x}_{-1}{x}_{-1},\dots ,x_{-1}{x}_{-d},\frac{1}{2}{x}_{-2}{x}_{-2},\dots ,x_{-2}{x}_{-d},\dots ,\frac{1}{2}{x}_{-,d-1}{x}_{-,d-1},\hfill \\ \hfill & x_{-,d-1}{x}_{-d},\frac{1}{2}{x}_{-d}{x}_{-d},x_{-1},x_{-2},\dots ,x_{-d}]\in {\mathbb{R}}^{\frac{d^2+d}{2}+d}.\hfill \end{array}$$

(24)

${\mathbf{\mu }}_p$ represents the mean of the dataset of ${\mathit{s}}_{+}$, ${\Sigma }_p$ represents the covariance matrix of the dataset of ${\mathit{s}}_{+}$, ${\mathbf{\mu }}_n$ represents the mean of the dataset of ${\mathit{s}}_{-}$, and ${\Sigma }_n$ represents the covariance matrix of the dataset of ${\mathit{s}}_{-}$. The separation obtained in Problem (22) can be interpreted as follows. Each data point $\mathit{x}\in {\mathbb{R}}^d$ in the original dataset of T is first mapped into a new space, i.e., $\mathit{s}\in {\mathbb{R}}^{\frac{d^2+d}{2}+d}$. Next, a linear classifier is adopted for the separation.

Since the mean and covariance matrix are often unknown, $\widehat{\mathbf{\mu }}$ and $\widehat{\Sigma }$ are used to estimate of the true value, i.e.:

$$\begin{array}{cc}\hfill & {\widehat{\mathbf{\mu }}}_{p\left(n\right)}=\frac{1}{N_{+(-)}}\sum _{i=1}^{N_{+(-)}}{\mathit{s}}_i\in {\mathbb{R}}^{\frac{d^2+3d}{2}},\hfill \\ \hfill & {\widehat{\Sigma }}_{p\left(n\right)}=\frac{1}{N_{+(-)}}\sum _{i=1}^{N_{+(-)}}({\mathit{s}}_i-{\widehat{\mathbf{\mu }}}_{p\left(n\right)}){({\mathit{s}}_i-{\widehat{\mathbf{\mu }}}_{p\left(n\right)})}^T\in {\mathbb{R}}^{\frac{d^2+3d}{2}},\hfill \end{array}$$

(25)

where $N_{+}$ represents the number of positive points in the dataset and $N_{-}$ represents the number of negative points in the data set.

By using Lemma 2, the probability inequalities in Problem (22) can be equivalently replaced by two inequalities. Then Problem (22) can be formulated as follows:

$$\begin{array}{cc}\hfill \underset{\mathit{z},c,{\alpha }_1,{\alpha }_2}{min}& \frac{(1-p){\alpha }_2+p}{2p(1-{\alpha }_1)}+C((1-p){\alpha }_1+p{\alpha }_2)\hfill \\ \hfill s.t.& {\mathit{z}}^T{\mathbf{\mu }}_p-c\ge \pi \left({\alpha }_1\right)\sqrt{{\mathit{z}}^T{\Sigma }_p\mathit{z}},\hfill \\ \hfill & c-{\mathit{z}}^T{\mathbf{\mu }}_n\ge \pi \left({\alpha }_2\right)\sqrt{{\mathit{z}}^T{\Sigma }_n\mathit{z}},\hfill \end{array}$$

(26)

where $\pi \left({\alpha }_1\right)=\sqrt{\frac{1-{\alpha }_1}{{\alpha }_1}}$, $\pi \left({\alpha }_2\right)=\sqrt{\frac{1-{\alpha }_2}{{\alpha }_2}}$.

Since the objective function is an increasing function with respect to ${\alpha }_1$ or ${\alpha }_2$, Problem (26) can be further formulated as the following problem:

$$\begin{array}{cc}\hfill \underset{\mathit{z},c,{\alpha }_1,{\alpha }_2}{min}& \frac{(1-p){\alpha }_2+p}{2p(1-{\alpha }_1)}+C((1-p){\alpha }_1+p{\alpha }_2)\hfill \\ \hfill s.t.& {\mathit{z}}^T{\mathbf{\mu }}_p-c=\pi \left({\alpha }_1\right)\sqrt{{\mathit{z}}^T{\Sigma }_p\mathit{z}},\hfill \\ \hfill & c-{\mathit{w}}^T{\mathbf{\mu }}_n=\pi \left({\alpha }_2\right)\sqrt{{\mathit{z}}^T{\Sigma }_n\mathit{z}}.\hfill \end{array}$$

(27)

The equivalence of Problem (26) and Problem (27) can be derived similarly to Lemma 4 in [7]. By summarizing the two constraints, the variable c can be omitted, i.e.:    

$$\begin{array}{cc}\hfill \underset{\mathit{z},{\alpha }_1,{\alpha }_2}{min}& \frac{(1-p){\alpha }_2+p}{2p(1-{\alpha }_1)}+C((1-p){\alpha }_1+p{\alpha }_2)\hfill \\ \hfill s.t.& \pi \left({\alpha }_1\right)\sqrt{{\mathit{z}}^T{\Sigma }_p\mathit{z}}+\pi \left({\alpha }_2\right)\sqrt{{\mathit{z}}^T{\Sigma }_n\mathit{z}}={\mathit{z}}^T({\mathbf{\mu }}_p-{\mathbf{\mu }}_n).\hfill \end{array}$$

(28)

Furthermore, as shown in (28), ${\alpha }_1$ and ${\alpha }_2$ play the symmetry role in the equality constraint. Then, ${\alpha }_2$ can be represented by ${\alpha }_1$ and $\mathit{z}$ as follows.

$$\begin{array}{c}\hfill {\alpha }_2=\frac{\sqrt{{\mathit{z}}^T{\Sigma }_n\mathit{z}}}{\sqrt{{\mathit{z}}^T{\Sigma }_n\mathit{z}}+{({\mathit{z}}^T({\mathbf{\mu }}_p-{\mathbf{\mu }}_n)-\pi \left({\alpha }_1\right){\mathit{z}}^T{\Sigma }_p\mathit{z})}^2}.\end{array}$$

(29)

Finally, by using this formulation, Problem (28) can be transformed to the following problem with respect to $\mathit{z}$ and ${\alpha }_1$.

$$\begin{array}{cc}\hfill \underset{\mathit{z},{\alpha }_1}{min}& M(\mathit{z},{\alpha }_1)=\frac{(\frac{1-p}{2p(1-{\alpha }_1)}+Cp)\sqrt{{\mathit{z}}^T{\Sigma }_n\mathit{z}}}{\sqrt{{\mathit{z}}^T{\Sigma }_n\mathit{z}}+{({\mathit{z}}^T({\mathbf{\mu }}_p-{\mathbf{\mu }}_n)-\pi \left({\alpha }_1\right){\mathit{z}}^T{\Sigma }_p\mathit{z})}^2}\hfill \\ \hfill & +\frac{1}{2(1-{\alpha }_1)}+C(1-p){\alpha }_1\hfill \\ \hfill s.t.& {\mathit{z}}^T({\mathbf{\mu }}_p-{\mathbf{\mu }}_n)-\pi \left({\alpha }_1\right){\mathit{z}}^T{\Sigma }_p\mathit{z}\ge 0,\hfill \\ \hfill & \parallel \mathit{z}\parallel =1.\hfill \end{array}$$

(30)

The constraint $\parallel \mathit{z}\parallel =1$ is added to ensure the uniqueness of the solution. Problem (30) is a nonconvex optimization problem with a fractional objective function.

3.2. Algorithm for QSMPMFA

For efficiently solving this problem, an alternating descent algorithm [7] is adopted. The details of the algorithm are described as follows.

In the t-th iteration, suppose that the value of ${\mathit{z}}_t$ satisfies the constraint of $\parallel {\mathit{z}}_t\parallel =1$, then ${\alpha }_1^t$ can be updated by solving the following problem:

$$\begin{array}{cc}\hfill {\alpha }_1^t=\underset{{\alpha }_1}{argmin}& \frac{(\frac{1-p}{2p(1-{\alpha }_1)}+Cp)\sqrt{{\mathit{z}}^T{\Sigma }_n\mathit{z}}}{\sqrt{{\mathit{z}}^T{\Sigma }_n\mathit{z}}+{({\mathit{z}}^T({\mathbf{\mu }}_p-{\mathbf{\mu }}_n)-\pi \left({\alpha }_1\right){\mathit{z}}^T{\Sigma }_p\mathit{z})}^2}+\frac{1}{2(1-{\alpha }_1)}\hfill \\ \hfill & +C(1-p){\alpha }_1\hfill \\ \hfill s.t.& {\alpha }_1\ge {\mathit{z}}_t^T{\Sigma }_p{\mathit{z}}_t/({\mathit{z}}_t^T{\Sigma }_p{\mathit{z}}_t+{({\mathbf{\mu }}_p-{\mathbf{\mu }}_n)}^2).\hfill \end{array}$$

(31)

Since this problem is a univariate optimization problem, it can be solved by the line search method.

Now, $\mathit{z}$ is updated by solving Problem (30) with ${\alpha }_1={\alpha }_1^t$, i.e.,:

$$\begin{array}{cc}\hfill \underset{\mathit{z}}{max}& {\lambda }_t\left(\mathit{z}\right)=\frac{{\mathit{z}}^T({\mathbf{\mu }}_p-{\mathbf{\mu }}_n)-\pi \left({\alpha }_1^t\right)\sqrt{{\mathit{z}}^T{\Sigma }_p\mathit{z}}}{\sqrt{{\mathit{z}}^T{\Sigma }_n\mathit{z}}}\hfill \\ \hfill s.t.& \parallel \mathit{z}\parallel =1.\hfill \end{array}$$

(32)

To find a ${\mathit{z}}_{t+1}$ that satisfies ${\lambda }_t\left({\mathit{z}}_{t+1}\right)>{\lambda }_t\left({\mathit{z}}_t\right)$, define the following function:

$$\begin{array}{c}\hfill f_t\left(\mathit{z}\right)={\mathit{z}}^T({\mathbf{\mu }}_p-{\mathbf{\mu }}_n)-\pi \left({\alpha }_1^t\right)\sqrt{{\mathit{z}}^T{\Sigma }_p\mathit{z}}-{\lambda }_t\left({\mathit{z}}_t\right)\sqrt{{\mathit{z}}^T{\Sigma }_n\mathit{z}}.\end{array}$$

(33)

The $f_t\left({\mathit{z}}_t\right)=0$. A ${\mathit{z}}_{t+1}$ that satisfies $f_t\left({\mathit{z}}_{t+1}\right)>0$ should be found. The following nonlinear programming problem can be solved for obtaining the inexact solution of Problem (32).

$$\begin{array}{cc}\hfill \underset{\mathit{z}}{max}& f_t\left(\mathit{z}\right)\hfill \\ \hfill s.t.& \parallel \mathit{z}\parallel =1.\hfill \end{array}$$

(34)

The gradient ascent method is used to solve Problem (34). Let ${\mathit{v}}_1={\mathit{z}}_t$, step ${\gamma }_k=1/k,k=1,2,\dots$, and, at the k-th step, ${\mathit{v}}_k$ is updated according to the following equation:

$${\mathit{u}}_k={\mathit{v}}_k+{\gamma }_k(({\mathbf{\mu }}_p-{\mathbf{\mu }}_n)-\frac{\pi \left({\alpha }_1^t\right){\Sigma }_p{\mathit{v}}_k}{\sqrt{{\mathit{v}}_k^T{\Sigma }_p{\mathit{v}}_k}}-\frac{{\lambda }_t\left({\mathit{z}}_t\right){\Sigma }_n{\mathit{v}}_k}{\sqrt{{\mathit{v}}_k^T{\Sigma }_n{\mathit{v}}_k}}),{\mathit{v}}_{k+1}=\frac{{\mathit{u}}_k}{\parallel {\mathit{u}}_k\parallel }.$$

(35)

Set the stopping condition of the above iterative process as:

$$\begin{array}{c}\hfill f_t\left({\mathit{v}}_{k+1}\right)\ge 0,f_t\left({\mathit{v}}_k\right)\ge 0,\frac{f_t\left({\mathit{v}}_{k+1}\right)-f_t\left({\mathit{v}}_k\right)}{f_t\left({\mathit{v}}_{k+1}\right)}\le ϵ.\end{array}$$

(36)

The $ϵ$ in the above equation is the stopping error. The $v^{*}$ that satisfied conditions (36) can be set to the updated ${\mathit{z}}_{t+1}$.

After $t_0$ rounds of the alternating descent algorithm, the value of ${\mathit{z}}_{t_0}$ and $b_{t_0}$ can be obtained. Correspondingly, $c^{*}$ in the quadratic surface parameters are represented as:

$$\begin{array}{c}\hfill c^{*}={\mathit{z}}_{t_0}^T{\mathbf{\mu }}_p-\pi \left({\alpha }_1^T\right)\sqrt{{\mathit{z}}_{t_0}^T{\Sigma }_p{\mathit{z}}_{t_0}}.\end{array}$$

(37)

Furthermore, the parameters ${\mathit{W}}^{*}$ and ${\mathit{b}}^{*}$ can be determined. For a new sample $\mathit{x}$, the label can be further determined according to the following decision function:

$$\begin{array}{c}\hfill f\left(\mathit{x}\right)=sign(\frac{1}{2}{\mathit{x}}^T{\mathit{W}}^{*}\mathit{x}+{\left({\mathit{b}}^{*}\right)}^T\mathit{x}+c^{*}),\end{array}$$

(38)

where $sign(·)$ is a sign function.

The algorithm has been summarized in Algorithm 1. The convergence of this algorithm can be illustrated by using the analyze in [7].

Algorithm 1: Training process of QSMPMFA

Input: Training set $T=\{({\mathit{x}}_1,y_1),({\mathit{x}}_2,y_2),\dots ,({\mathit{x}}_{m_1+m_2},y_{m_1+m_2})\}$, C, $t=0$. Output: ${\mathit{W}}^{*}$, ${\mathit{b}}^{*}$, $c^{*}$. 1. Transform the training set T to $\tilde{T}=\{({\mathit{s}}_1,y_1),({\mathit{s}}_2,y_2),\dots ,({\mathit{s}}_{m_{+}+m_{-}},$$y_{m_{+}+m_{-}}\left)\right\}$ by using the formulations (23) and (24). 2. Calculate the mean and covariance matrices $({\widehat{\mathbf{\mu }}}_p,{\widehat{\Sigma }}_p)$, $({\widehat{\mathbf{\mu }}}_n,{\widehat{\Sigma }}_n)$ according to formula (25). 3. Denote ${\mathit{z}}_t=({\widehat{\mathbf{\mu }}}_p-{\widehat{\mathbf{\mu }}}_n)/\parallel {\widehat{\mathbf{\mu }}}_p-{\widehat{\mathbf{\mu }}}_n\parallel$. 4. for t =1, 2, ⋯ do 5.    Calculate ${\alpha }_1^t$ by solving Problem (31). 6.    Calculate the objective function of Problem (30), i.e., $M({\mathit{z}}_t,{\alpha }_1^t)$. 7.    Let ${\mathit{v}}_k={\mathit{z}}_t$, $k=1$. 8.    While the condition (36) is not met 9.         update ${\mathit{v}}_{k+1}$ by using formula (35). 10.    End 11.    Let ${\mathit{z}}_{t+1}={\mathit{v}}_{k+1}$, calculate the objective function of Problem (30), i.e., $M({\mathit{z}}_{t+1},{\alpha }_1^t)$. 12.    If $(M({\mathit{z}}_t,{\alpha }_1^t)-M({\mathit{z}}_{t+1},{\alpha }_1^t))/M({\mathit{z}}_t,{\alpha }_1^t)\le ϵ$ 13.       terminate 14.    End 15. End 16. Calculate ${\mathit{W}}^{*}$, ${\mathit{b}}^{*}$, $c^{*}$ according to Equations (11) and (37).

4. Numerical Tests

In this section, the classification performance of the QSMPMFA model is investigated. For comparison, a minimax probability machine with a linear kernel (MPM-l), polynomial kernel (MPM-p), and Gaussian kernel (MPM-g) [1]; a minimax probability machine for F measure with a linear kernel (MPMF-l), polynomial kernel (MPMF-p), and Gaussian kernel (MPMF-g) [7]; and a quadratic surface minimax probability machine (QSMPM) [22] are tested.

All the experiments are implemented in MATLAB R2018a on a PC with 2.1 GHz CPU and 16 GB RAM. Both the $F_1$ value of the minority class samples and accuracy rate are considered for measuring the classification performance. The $F_1$ value is defined in Section 2.3. The accuracy rate is defined as the ratio of correctly classified numbers to the total number. The penalty parameter C is chosen in the range of $\{2^{-10},2^{-5},2^0,2^5\}$. The kernel function parameter of the kernel-based methods is chosen in the range of $\{2^{-5},2^{-6},\dots ,2^8\}$. They are all trained by using a 5-fold cross-validation [23]. All the methods are tested ten times by randomly selecting points in two classes to form training sets. For each dataset, a random selection of 70% of the samples in the dataset is used for training and the remaining 30% are used for testing.

In order to illustrate the classification performance of the proposed QSMPMFA, comparative experiments on 12 benchmark datasets extracted from the UCI Machine Learning Repository [24] are conducted. The information of all the datasets is shown in

Table 1. Information of the benchmark datasets.

Data Set	Positive #	Negative #	Total #	Feature #	Imbalanced Ratio
Iris	50	50	100	4	1
Seeds	70	70	140	7	1
Glass	51	163	214	9	3.20
Wholesale customers	142	298	440	7	2.10
WDBC	212	357	569	30	1.68
Balance	288	288	576	4	1
Breast cancer	241	458	699	9	1.90
Banknote	610	762	1372	4	1.25
Segment	330	1980	2310	19	6
Rice	1630	2180	3810	7	1.34
Dry Bean	1322	12289	13,611	16	9.30
Skin	50,859	194,198	245,057	3	3.82

. In this table, the number of positive samples, negative samples, total samples, and features are listed. The last column in Table 1 records the imbalanced ratio, which is calculated by the ratio of the negative data numbers and positive data numbers. The larger the imbalanced ratio, the higher the imbalanced degree.

Table 2. $F_1$ value for each method on each data set.

Data Set	MPM-l	MPM-g	MPM-p	MPMF-l	MPMF-g	PMPF-p	QSMPM	QSMPMFA
Iris	1.0000± 0.0000	1.0000 ± 0.0000	1.0000 ± 0.0000	1.0000 ± 0.0000	0.9968 ± 0.0102	1.0000 ± 0.0000	1.0000 ± 0.0000	1.0000 ± 0.0000
Seeds	0.9530 ± 0.0309	0.9907 ± 0.0120	0.9887 ± 0.0130	0.9839 ± 0.0188	0.9643 ± 0.0454	0.9497 ± 0.0291	0.9699 ± 0.295	0.9862 ± 0.0160
Glass	0.7728 ± 0.0702	0.8456 ± 0.0472	0.8722 ± 0.0558	0.8158 ± 0.0452	0.8795 ± 0.0478	0.8604 ± 0.0504	0.8259 ± 0.0463	0.8534 ± 0.0564
Wholesale customers	0.8158 ± 0.0413	0.8452 ± 0.0421	0.8425 ± 0.0164	0.8017 ± 0.1277	0.8451 ± 0.0430	0.7256 ± 0.1361	0.8608 ± 0.0327	0.8660 ± 0.0379
WDBC	0.9357 ± 0.0172	0.9495 ± 0.0192	0.9524 ± 0.0171	0.9495 ± 0.0195	0.9548 ± 0.0194	0.9519 ± 0.0108	0.9598 ± 0.0162	0.9603 ± 0.0116
Balance	0.9448 ± 0.0147	0.9901 ± 0.0087	0.9849 ± 0.0126	0.9605 ± 0.0105	0.9885 ± 0.0066	0.9596 ± 0.0099	0.9878 ± 0.0080	0.9818 ± 0.0105
Breast cancer	0.9427 ± 0.0158	0.9558 ± 0.0156	0.9503 ± 0.0210	0.9500 ± 0.0200	0.9556 ± 0.0142	0.9471 ± 0.0294	0.9527 ± 0.0158	0.9567 ± 0.0117
Banknote	0.8619 ± 0.0253	0.9895 ± 0.0042	0.9820 ± 0.0050	0.9850 ± 0.0060	0.9908 ± 0.0071	0.9823 ± 0.0069	0.9773 ± 0.0070	0.9957 ± 0.0057
Segment	0.5791 ± 0.0164	0.6265 ± 0.0714	0.7106 ± 0.0521	0.9660 ± 0.0120	0.9771 ± 0.0312	0.9110 ± 0.0359	0.9302 ± 0.0209	0.9814 ± 0.0153
Rice	0.9047 ± 0.0094	0.9142 ± 0.0237	0.9073 ± 0.0313	0.9098 ± 0.0074	0.9063 ± 0.0117	0.9131 ± 0.0078	0.9135 ± 0.0074	0.9142 ± 0.0080
Dry Bean	0.5263 ± 0.0121	0.8361 ± 0.0436	0.8117 ± 0.0393	0.8790 ± 0.0109	0.8639 ± 0.0078	0.8617 ± 0.0082	0.8353 ± 0.0135	0.8867 ± 0.0097
Skin	0.8098 ± 0.0017	-	-	0.8793 ± 0.0013	-	-	0.8562 ± 0.0012	0.8824 ± 0.0178
Average rank	7.08	3.83	4.79	4.79	3.96	5.46	4.08	2.00

shows the $F_1$ value of each method on all the datasets. The bold ones are the best results among all the methods. “-” indicates that the memory exceeds the limit in the training process. From Table 2, it can be seen that QSMPMFA outperforms the comparison methods for classifying most datasets, especially on datasets with higher imbalance ratios such as Segment, Dry Bean, Skin, and so on. For the other datasets that QSMPMFA performs not optimally on, it can be seen that the best results often occur in the methods with Gaussian kernel functions. The reason may be that the Gaussian kernel is more proper for dealing with these datasets. For showing the results more intuitively, the boxplot of the $F_1$ value of the positive class of each method for classifying all the datasets is illustrated in Figure 1.

For comparison, the classification accuracy rate of each method is also listed in

Table 3. Accuracy rate for each method on each dataset.

Data Set	MPM-l	MPM-g	MPM-p	MPMF-l	MPMF-g	PMPF-p	QSMPM	QSMPMFA
Iris	1.0000 ± 0.0000	1.0000 ± 0.0000	1.0000 ± 0.0000	1.0000 ± 0.0000	0.9967 ± 0.0105	1.0000 ± 0.0000	1.0000 ± 0.0000	1.0000 ± 0.0000
Seeds	0.9524 ± 0.0317	0.9905 ± 0.0123	0.9893 ± 0.0144	0.9833 ± 0.0196	0.9619 ± 0.0504	0.9476 ± 0.0310	0.9690 ± 0.0298	0.9857 ± 0.0166
Glass	0.8938 ± 0.0328	0.9281 ± 0.0223	0.9406 ± 0.0280	0.9125 ± 0.0235	0.9406 ± 0.0242	0.9375 ± 0.0156	0.9141 ± 0.0247	0.9328 ± 0.0256
Wholesale customers	0.8864 ± 0.0250	0.8992 ± 0.0305	0.8909 ± 0.0148	0.8341 ± 0.1572	0.8962 ± 0.0297	0.7561 ± 0.1686	0.9076 ± 0.0220	0.9099 ± 0.0221
WDBC	0.9520 ± 0.0132	0.9626 ± 0.0144	0.9709 ± 0.0137	0.9632 ± 0.0138	0.9661 ± 0.0148	0.9637 ± 0.0087	0.9696 ± 0.0126	0.9708 ± 0.0087
Balance	0.9453 ± 0.0143	0.9901 ± 0.0087	0.9849 ± 0.0127	0.9593 ± 0.0110	0.9884 ± 0.0067	0.9581 ± 0.0104	0.9878 ± 0.0080	0.9814 ± 0.0109
Breast cancer	0.9603 ± 0.0108	0.9689 ± 0.0111	0.9656 ± 0.0141	0.9646 ± 0.0138	0.9689 ± 0.0099	0.9636 ± 0.0199	0.9670 ± 0.0109	0.9694 ± 0.0082
Banknote	0.8789 ± 0.0214	0.9914 ± 0.0021	0.9872 ± 0.0042	0.9864 ± 0.0055	0.9917 ± 0.0064	0.9840 ± 0.0063	0.9794 ± 0.0065	0.9961 ± 0.0051
Segment	0.7924 ± 0.0144	0.8658 ± 0.0347	0.8831 ± 0.0305	0.9902 ± 0.0036	0.9953 ± 0.0095	0.9729 ± 0.0116	0.9788 ± 0.0066	0.9947 ± 0.0044
Rice	0.9182 ± 0.0082	0.9248 ± 0.0127	0.9094 ± 0.0216	0.9190 ± 0.0068	0.9158 ± 0.0113	0.9227 ± 0.0068	0.9241 ± 0.0067	0.9253 ± 0.0071
Dry Bean	0.8412 ± 0.0065	0.9490 ± 0.0208	0.9571 ± 0.0202	0.9761 ± 0.0022	0.9771 ± 0.0032	0.9705 ± 0.0047	0.9634 ± 0.0034	0.9775 ± 0.0022
Skin	0.9096 ± 0.0008	-	-	0.9430 ± 0.0007	-	-	0.9342 ± 0.0006	0.9448 ± 0.0089
Average rank	6.92	4.08	4.46	4.83	3.83	5.63	4.08	2.17

. Except for the WDBC and Segment datasets, the performance of QSMPMFA is similar to that in Table 2. For the WDBC dataset, QSMPMFA performs almost the same as the best method. The reasons for explaining these results may be that the $F_1$ value is adopted in the cross-validation of choosing parameters. It can be seen that the accuracy rates are often larger than the $F_1$ value on most of the datasets. The boxplot of the accuracy rates of all the methods on each dataset are illustrated in Figure 2.

From Table 2 and Table 3, it can be seen that QSMPMFA performs better than QSMPM for classifying each dataset except for the Balance dataset. QSMPM only aims to maximize the lower bound of the correct classification probability of two classes. Since both QSMPM and QSMPMFA are kernel-free methods, the better performance of the proposed method demonstrates the effectiveness of setting both the $F_1$ value and accuracy rate as objective functions. MPMF with different kernel functions adopts $F_{\beta }$ as the optimization goal, which focuses on the classification of imbalanced datasets. When compared with MPMF, the better performance also illustrates the effectiveness of using two measures and the kernel-free setting.

For illustrating the performance of the proposed method on imbalanced datasets, more tests on the wholesale customers dataset are conducted. The wholesale customer dataset refers to the information of clients of a wholesale distributor. It includes the annual spending in monetary units on diverse product categories. Some positive samples are chosen with a different ratio of 1, 0.9, 0.6, and 0.2 randomly. Next, the chosen samples with different ratios and all the negative samples are combined to construct four datasets. A similar training process is utilized for testing these datasets.

The comparison results of MPM-l, MPM-g, MPM-p, MPMF-l, MPMF-g, MPM-p, QSMPM, and QSMPMFA on the four datasets are illustrated in Figure 3 and Figure 4. It can be seen that the proposed QSMPMFA performs the best for classifying all the datasets. As shown in Figure 3, the $F_1$ values of each method on each dataset show a decreasing trend overall when the ratio of positive points decreases (i.e., the imbalance degree increases). The variation of the $F_1$ values of the proposed QSMPMFA is relatively stable. The other methods, especially MPM-g, MPM-p, and QSMPM, decrease more significantly when the ratio is small. From Figure 4, it can be seen that the accuracy rate of each method on each dataset increase overall when the ratio of positive points decreases (i.e., the imbalance degree increases). The performance of QSMPMFA is also relatively stable. Although the accuracy rate is not a proper measure for analyzing the imbalanced dataset, it still demonstrates the better performance of the proposed method. When compared with QSMPM, the relatively large growth rate shows the better performance of the proposed method, which further demonstrates the superior performance for classifying imbalanced datasets.

5. Conclusions

In this paper, a new quadratic surface minimax probability machine model considering both the $F_1$ value and accuracy rate as the optimization goal is proposed. On the one hand, the proposed QSMPMFA combines the kernel-free technique with the MPM model, which causes it to be more efficient to deal with large-scale datasets. On the other hand, by adopting two performance measures in the objective function of the proposed model, QSMPMFA is more flexible for dealing with the imbalanced dataset. Some public benchmark datasets are adopted for validating the effectiveness of the proposed method. From the numerical tests, it can be seen that QSMPMFA performs the best when compared with other state-of-the-art MPM methods. It can also be concluded that the proposed QSMPMFA enjoys the superiority on dealing with imbalanced datasets.

In the proposed model, both the $F_1$ value of the minority class and the weighted classification accuracy rate of the two classes are considered as the optimization goal. In our settings, the parameter for balancing the importance between the two items in the objective function is tuned by using cross-validation. The parameter in the weighted classification accuracy rate is fixed by using the ratio of the minority class. Actually, the choice of these parameters affects the classification performance and efficiency of the proposed model. In the future, more research on how to choose proper parameters should be performed. More applications of this method can be explored in other areas [25]. Besides, methods with prior information will be considered.