An Effective Multiclass Twin Hypersphere Support Vector Machine and Its Practical Engineering Applications

Abstract

Twin-KSVC (Twin Support Vector Classification for K class) is a novel and efficient multiclass twin support vector machine. However, Twin-KSVC has the following disadvantages. (1) Each pair of binary sub-classifiers has to calculate inverse matrices. (2) For nonlinear problems, a pair of additional primal problems needs to be constructed in each pair of binary sub-classifiers. For these disadvantages, a new multi-class twin hypersphere support vector machine, named Twin Hypersphere-KSVC, is proposed in this paper. Twin Hypersphere-KSVC also evaluates each sample into 1-vs-1-vs-rest structure, as in Twin-KSVC. However, our Twin Hypersphere-KSVC does not seek two nonparallel hyperplanes in each pair of binary sub-classifiers as in Twin-KSVC, but a pair of hyperspheres. Compared with Twin-KSVC, Twin Hypersphere-KSVC avoids computing inverse matrices, and for nonlinear problems, can apply the kernel trick to linear case directly. A large number of comparisons of Twin Hypersphere-KSVC with Twin-KSVC on a set of benchmark datasets from the UCI repository and several real engineering applications, show that the proposed algorithm has higher training speed and better generalization performance.

1. Introduction

Support vector machine (SVM) [1,2], as a computationally powerful tool for classification, have already applied in wide engineering problems [3,4,5,6,7,8]. The SVM has three elements that make it so successful, including structural risk minimization (SRM) principle, kernel trick and dual theory. However, SVM has to solve a large-sized quadratic programming problem (QPP), which greatly limits its applications. To improve learning complexity of SVM, Jayadeva et al. proposed twin SVM (TSVM) [9]. Unlike SVM that seeks an optimal separating hyperplane which maximizes the margin of two classes of samples, TSVM constructs two nonparallel proximal hyperplanes, each of which is close to the corresponding class as possible, and keeps as far away as possibly from the opposite class. The strategy makes TSVM only need to solve two smaller QPPs, instead of one larger QPP as in SVM. Due to its high learning speed, TSVM has attracted interest in recent years. Many improvements have also been proposed [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28], where the twin hypersphere SVM (THSVM) [25,26,27,28] is an excellent improvement of TSVM. Unlike TSVM, THSVM seeks a pair of hyperspheres, instead of two nonparallel hyperplanes, to depict two classes of samples. Compared with TSVM, THSVM has better classification performance.

The SVM and TSVM can only solve the binary classification problems, however, many practical engineering problems are often related to multi-class classification in the real world. Currently, we usually use 1-vs-rest and 1-vs-1 strategy to resolve multi-class classification problems. In the 1-vs-1 SVM, $K(K-1)/2$ binary SVM sub-classifiers are constructed. Each sub-classifier can be trained by using two classes of samples. Because only two classes are considered for each sub-classifier in the 1-vs-1 SVM and the rest samples are not involved, the 1-vs-1 SVM may get unfavorable classification results. The 1-vs-rest SVM needs to construct K binary SVM sub-classifiers. Each sub-classifier can be trained by using all the samples; thus, the 1-vs-rest SVM may lead to class imbalance problems. For above drawbacks of 1-vs-1 SVM and 1-vs-rest SVM, Angulo et al. proposed a new support vector classification-regression machine for K class (K-SVCR) [29]. $K(K-1)/2$ binary SVM sub-classifiers are constructed, each of which is trained with all the samples and evaluates each sample into 1-vs-1-vs-rest structure. K-SVCR avoids the class imbalance problems and information loss. Compared with 1-vs-1 SVM and 1-vs-rest SVM, K-SVCR achieves better generalization performance. Twin-KSVC [30,31], being an effective extension of K-SVCR, is based on TSVM and also evaluates each sample into 1-vs-1-vs-rest structure. Twin-KSVC achieves higher learning speed in comparison with K-SVCR. However, Twin-KSVC has the following disadvantages:

• Each pair of sub-classifiers has to calculate inverse matrices, which is extraordinarily time-consuming for the large-scale engineering problems.
• For nonlinear problems, each pair of sub-classifiers needs to construct a pair of additional primal problems, instead of directly applying the kernel trick to linear case as in SVM.

For the disadvantages of Twin-KSVC, in this paper, we propose a Twin Hypersphere-KSVC, inspired by THSVM. The Twin Hypersphere-KSVC also evaluates each sample into 1-vs-1-vs-rest structure, as in Twin-KSVC. However, our Twin Hypersphere-KSVC does not seek two nonparallel hyperplanes in each pair of binary sub-classifiers as in Twin-KSVC, but a pair of hyperspheres. Compared with Twin-KSVC, Twin Hypersphere-KSVC avoids computing inverse matrices, and for nonlinear problems, can apply the kernel trick to linear case directly.

This paper is outlined as follows. We briefly review the related multi-class classification algorithms in Section 2. In Section 3, the Twin Hypersphere-KSVC is proposed in detail. The experimental results on a set of benchmark datasets and several real engineering problems are presented in Section 4 and the conclusions are drawn in last section.

2. Related Works

In this paper, we consider a multi-class classification problem with a training dataset $D=\{x_p^k\in R^d|k=1,\cdots ,K,p=1,\cdots ,n_k\}$, where K is the number of classes and $n_k$ is the number of the samples of the k-th class. The size of training dataset is $n=n_1+\cdots +n_K$. Denote, for convenience, by $X_k$ the sets of samples of the k-th class, i.e., $X_k=\left\{x_p^k\right|p=1,\cdots ,n_k\}$.

2.1. Review of K-SVCR Multi-Classifier

The K-SVCR multi-classifier [29] is based on decomposition-reconstruction strategy. $K(K-1)/2$ binary SVM sub-classifiers are constructed, each of which evaluates each sample into 1-vs-1-vs-rest structure. The classification result of K-SVCR is shown in Figure 1a intuitively.

The sub-classifier $f^{ij}(x)$ for two focused classes i and j in K-SVCR seeks an optimal hyperplane

$$w^{ij}·x+b^{ij}=0,$$

(1)

here $w^{ij}\in R^d$ is the normal vector and $b^{ij}\in R$ is the bias term. The optimal hyperplane can be obtained by resolving the following QPP:

$$\begin{array}{c}min\frac{1}{2}{∥w^{ij}∥}^2+c_1({\displaystyle \sum _{p=1}^{n_i}}{\eta }_p^{ij}+{\displaystyle \sum _{p=1}^{n_j}}{\eta }_p^{ij*})+c_2{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}({\xi }_p^{ij}+{\xi }_p^{ij*}),\hfill \\ s.t.w^{ij}·x_p^i+b^{ij}\ge 1-{\eta }_p^{ij},p=1,\cdots ,n_i,\hfill \\ w^{ij}·x_p^j+b^{ij}\le -1+{\eta }_p^{ij*},p=1,\cdots ,n_j,\hfill \\ -\epsilon -{\xi }_p^{ij*}<w^{ij}·x_p^{\overline{ij}}+b^{ij}\le \epsilon +{\xi }_p^{ij},p=1,\cdots ,n_{\overline{ij}},\hfill \\ {\eta }_p^{ij}\ge 0,p=1,\cdots ,n_i,{\eta }_p^{ij*}\ge 0,p=1,\cdots ,n_j,\hfill \\ {\xi }_p^{ij}\ge 0,{\xi }_p^{ij*}\ge 0,p=1,\cdots ,n_{\overline{ij}},\hfill \end{array}$$

(2)

where $x_p^{\overline{ij}}\in D-X_i-X_j$, $n_{\overline{ij}}=n-n_i-n_j$, ${\xi }_p^{ij}$, ${\xi }_p^{ij*}$ and ${\eta }_p^{ij}$, ${\eta }_p^{ij*}$ are slack variables, the parameter $\epsilon$ is restricted to $[0,1)$.

For a testing sample x, the sub-classifier $f^{ij}(x)=w^{ij}·x+b^{ij}$ determines its class by

$$\begin{array}{c}{F}^{ij}(x)=\left\{\begin{array}{cc}-1,& if{f}^{ij}(x)<\epsilon ,\\ 1,& if{f}^{ij}(x)>\epsilon ,\\ 0,& else.\end{array}\right.\hfill \end{array}$$

(3)

For the testing sample x, the final label can be determined by vote rule.

2.2. Review of Twin-KSVC Multi-Classifier

Twin-KSVC [30,31] is an improvement of K-SVCR. The Twin-KSVC constructs $K(K-1)/2$ pairs of binary TSVM sub-classifiers, which evaluates each sample into 1-vs-1-vs-rest structure. The classification result is intuitively presented in Figure 1b.

The sub-classifiers $f^i(x)$ and $f^j(x)$ for two focused classes i and j in Twin-KSVC seek a pair of hyperplane

$$w^i·x+b^i=0and{w}^j·x+b^j=0$$

(4)

where $w^{i(j)}\in R^d$ and $b^{i(j)}\in R$ are the normal vector and the bias term of the corresponding hyperplane, respectively. The two hyperplanes can be obtained by resolving the QPPs as follows:

$$\begin{array}{c}min\frac{1}{2}{\displaystyle \sum _{p=1}^{n_i}}{(w^i·x_p^i+b^i)}^2+c_1{\displaystyle \sum _{p=1}^{n_j}}{\eta }_p^i+c_2{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\xi }_p^i,\hfill \\ s.t.-(w^i·x_p^j+b^i)+{\eta }_p^i\ge 1,p=1,\cdots ,n_j,\hfill \\ -(w^i·x_p^{\overline{ij}}+b^i)+{\xi }_p^i\ge 1-\epsilon ,p=1,\cdots ,n_{\overline{ij}},\hfill \\ {\eta }_p^i\ge 0,p=1,\cdots ,n_j,\hfill \\ {\xi }_p^i\ge 0,p=1,\cdots ,n_{\overline{ij}},\hfill \end{array}$$

(5)

$$\begin{array}{c}min\frac{1}{2}{\displaystyle \sum _{p=1}^{n_j}}{(w^j·x_p^j+b^j)}^2+c_3{\displaystyle \sum _{p=1}^{n_i}}{\eta }_p^j+c_4{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\xi }_p^j,\hfill \\ s.t.(w^j·x_p^i+b^j)+{\eta }_p^j\ge 1,p=1,\cdots ,n_i,\hfill \\ (w^j·x_p^{\overline{ij}}+b^j)+{\xi }_p^j\ge 1-\epsilon ,p=1,\cdots ,n_{\overline{ij}},\hfill \\ {\eta }_p^j\ge 0,p=1,\cdots ,n_i,\hfill \\ {\xi }_p^j\ge 0,p=1,\cdots ,n_{\overline{ij}},\hfill \end{array}$$

(6)

where ${\eta }_p^{i(orj)}$ and ${\xi }_p^{i(orj)}$ are slack variables.

For a testing sample x, the sub-classifiers $f^i(x)=x^T{w}^i+b^i$ and $f^j(x)=x^T{w}^j+b^j$ assign its class by

$$\begin{array}{c}{F}^{ij}(x)=\left\{\begin{array}{cc}-1,& if{f}^j(x)<1-\epsilon ,\\ 1,& if{f}^i(x)>-1+\epsilon ,\\ 0,& else.\end{array}\right.\hfill \end{array}$$

(7)

For the testing sample x, the final label can be also determined by vote rule.

2.3. Review of THKSVM Multi-Classifier

THKSVM (Twin Hypersphere Multiclass Support Vector Machine) [32] integrates THSVM and 1-vs-rest structure. THKSVM constructs K hyperspheres in the training stage, whose classification result is intuitively shown in Figure 1c.

The sub-classifier for the focused classes i in THKSVM seeks a hypersphere

$${∥x-a_i∥}^2=R_i^2$$

(8)

where $a_i\in R^d$ and $R_i\in R$ are the center and the radius of the corresponding hypersphere, respectively. The hypersphere can be constructed by resolving the following QPP:

$$\begin{array}{c}min\frac{1}{2}{\displaystyle \sum _{p=1}^{n_{\overline{i}}}}{∥x_p^{\overline{i}}-a_i∥}^2-v_1{R}_i^2+c_1{\displaystyle \sum _{p=1}^{n_i}}{\eta }_p^i,\hfill \\ s.t.{∥x_p^i-a_i∥}^2\ge R_i^2-{\eta }_p^i,\hfill \\ R_i^2\ge 0,{\eta }_p^i\ge 0,p=1,\cdots ,n_i,\hfill \end{array}$$

(9)

where $x_p^{\overline{i}}\in D-X_i$, $n_{\overline{i}}=n-n_i$ and ${\eta }_p^i\ge 0$ are slack variables.

The class of a testing sample x can be determined by

$$Classk=arg\underset{i=1,\cdots ,K}{max}\frac{{∥x-a_i∥}^2}{R_i^2}.$$

(10)

3. Twin Hypersphere-KSVC

Twin Hypersphere-KSVC, inspired by THSVM and 1-vs-1-vs-rest structure, constructs $K(K-1)/2$ pairs of hyperspheres in the training stage. For two focused classes i and j, Twin Hypersphere-KSVC seeks a pair of hypersphere $(a_i,R_i)$ and $(a_j,R_j)$, where $a_i$ ($R_i$) and $a_j$ ($R_j$) are respectively the centers (radii) of the corresponding hyperspheres. Each hypersphere covers the corresponding class as many as possibly, keeps as far away as possibly from another class, contains the rest of samples as little as possibly, and the radius of the hypersphere is as small as possible. The Twin Hypersphere-KSVC are intuitively presented in Figure 1d.

3.1. Linear Case

For the linear case, each pair of hyperspheres $(a_i,R_i)$ and $(a_j,R_j)$ for two focused classes i and j in Twin Hypersphere-KSVC is constructed by resolving the following QPPs:

$$\begin{array}{c}min{R}_i^2-\frac{v_1}{n_j}{\displaystyle \sum _{p=1}^{n_j}}{∥x_p^j-a_i∥}^2+\frac{c_1}{n_i}{\displaystyle \sum _{p=1}^{n_i}}{\eta }_p^i+\frac{c_2}{n_{\overline{ij}}}{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\xi }_p^i,\hfill \\ s.t.{∥x_p^i-a_i∥}^2\le R_i^2+{\eta }_p^i,p=1,\cdots ,n_i,\hfill \\ {∥x_p^{\overline{ij}}-a_i∥}^2\ge R_i^2-{\xi }_p^i,p=1,\cdots ,n_{\overline{ij}},\hfill \\ {\eta }_p^i\ge 0,p=1,\cdots ,n_i,\hfill \\ {\xi }_p^i\ge 0,p=1,\cdots ,n_{\overline{ij}},\hfill \\ R_i^2\ge 0,\hfill \end{array}$$

(11)

$$\begin{array}{c}min{R}_j^2-\frac{v_2}{n_i}{\displaystyle \sum _{p=1}^{n_i}}{∥x_p^i-a_j∥}^2+\frac{c_3}{n_j}{\displaystyle \sum _{p=1}^{n_j}}{\eta }_p^j+\frac{c_4}{n_{\overline{ij}}}{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\xi }_p^j,\hfill \\ s.t.{∥x_p^j-a_j∥}^2\le R_j^2+{\eta }_p^j,p=1,\cdots ,n_j,\hfill \\ {∥x_p^{\overline{ij}}-a_j∥}^2\ge R_j^2-{\xi }_p^j,p=1,\cdots ,n_{\overline{ij}},\hfill \\ {\eta }_p^j\ge 0,p=1,\cdots ,n_j,\hfill \\ {\xi }_p^j\ge 0,p=1,\cdots ,n_{\overline{ij}},\hfill \\ R_j^2\ge 0.\hfill \end{array}$$

(12)

where ${\eta }_p^{i(j)}$ and ${\xi }_p^{i(j)}$ are slack variables.

The Lagrangian function L for the QPP (11) is given by:

$$\begin{array}{c}L=R_i^2-\frac{v_1}{n_j}{\displaystyle \sum _{p=1}^{n_j}}{∥x_p^j-a_i∥}^2+\frac{c_1}{n_i}{\displaystyle \sum _{p=1}^{n_i}}{\eta }_p^i+\frac{c_2}{n_{\overline{ij}}}{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\xi }_p^i+{\displaystyle \sum _{p=1}^{n_i}}{\alpha }_p({∥x_p^i-a_i∥}^2-R_i^2-{\eta }_p^i)\hfill \\ -{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\beta }_p({∥x_p^{\overline{ij}}-a_i∥}^2-R_i^2+{\xi }_p^i)-{\displaystyle \sum _{p=1}^{n_i}}{s}_p{\eta }_p^i-{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{q}_p{\xi }_p^i-\lambda R_i^2.\hfill \end{array}$$

(13)

The Karush-Kuhn-Tucker (KKT) conditions are satisfied as follows:

$$2\frac{v_1}{n_j}{\displaystyle \sum _{p=1}^{n_j}}(x_p^j-a_i)-2{\displaystyle \sum _{p=1}^{n_i}}{\alpha }_p(x_p^i-a_i)+2{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\beta }_p(x_p^{\overline{ij}}-a_i)=0,$$

(14)

$$1-{\displaystyle \sum _{p=1}^{n_i}}{\alpha }_p+{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\beta }_p-\lambda =0,$$

(15)

$$\frac{c_1}{n_i}-{\alpha }_p-s_p=0,p=1,\cdots ,n_i,$$

(16)

$$\frac{c_2}{n_{\overline{ij}}}-{\beta }_p-q_p=0,p=1,\cdots ,n_{\overline{ij}},$$

(17)

$${\alpha }_p({∥x_p^i-a_i∥}^2-R_i^2-{\eta }_p^i)=0,p=1,\cdots ,n_i,$$

(18)

$${\beta }_p({∥x_p^{\overline{ij}}-a_i∥}^2-R_i^2+{\xi }_p^i)=0,p=1,\cdots ,n_{\overline{ij}},$$

(19)

$$s_p{\eta }_p^i=0,p=1,\cdots ,n_i,$$

(20)

$$q_p{\xi }_p^i=0,p=1,\cdots ,n_{\overline{ij}},$$

(21)

$$\lambda R_i^2=0.$$

(22)

From (14), (15) and (22), we can obtain

$$a_i=\frac{1}{1-v_1}({\displaystyle \sum _{p=1}^{n_i}}{\alpha }_p{x}_p^i-\frac{v_1}{n_j}{\displaystyle \sum _{p=1}^{n_j}}{x}_p^j-{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\beta }_p{x}_p^{\overline{ij}})$$

(23)

By denoting

$$v_i=\frac{1}{1-v_1}$$

(24)

and substituting (16)–(23) into (13), the dual optimal problem of (11) is obtained as follows:

$$\begin{array}{c}max{\displaystyle \sum _{p=1}^{n_i}}{\alpha }_p{x}_p^i·x_p^i-{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\beta }_p{x}_p^{\overline{ij}}·x_p^{\overline{ij}}+2v_i\frac{v_1}{n_j}{\displaystyle \sum _{p_1=1}^{n_j}}{\displaystyle \sum _{p_2=1}^{n_i}}{\alpha }_{p_2}{x}_{p_1}^j·x_{p_2}^i-2v_i\frac{v_1}{n_j}{\displaystyle \sum _{p_1=1}^{n_j}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\beta }_{p_2}{x}_{p_1}^j·x_{p_2}^{\overline{ij}}\hfill \\ -v_i{\displaystyle \sum _{p_1=1}^{n_i}}{\displaystyle \sum _{p_2=1}^{n_i}}{\alpha }_{p_1}{\alpha }_{p_2}{x}_{p_1}^i·x_{p_2}^i+2v_i{\displaystyle \sum _{p_1=1}^{n_i}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\alpha }_{p_1}{\beta }_{p_2}{x}_{p_1}^i·x_{p_2}^{\overline{ij}}-v_i{\displaystyle \sum _{p_1=1}^{n_{\overline{ij}}}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\beta }_{p_1}{\beta }_{p_2}{x}_{p_1}^{\overline{ij}}·x_{p_2}^{\overline{ij}}\hfill \\ s.t.1-{\displaystyle \sum _{p=1}^{n_i}}{\alpha }_p+{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\beta }_p=0,\hfill \\ 0\le {\alpha }_p\le \frac{c_1}{n_i},p=1,\cdots ,n_i,\hfill \\ 0\le {\beta }_p\le \frac{c_2}{n_{\overline{ij}}},p=1,\cdots ,n_{\overline{ij}}.\hfill \end{array}$$

(25)

By defining $\alpha =\left(\begin{array}{c}{\alpha }_1\\ ⋮\\ {\alpha }_{n_i}\end{array}\right)$ and $\beta =\left(\begin{array}{c}{\beta }_1\\ ⋮\\ {\beta }_{n_{\overline{ij}}}\end{array}\right)$, the optimal problem (25) can be reformed as

$$\begin{array}{c}max-v_i(\begin{array}{cc}{\alpha }^T& {\beta }^T\end{array})\left(\begin{array}{cc}{X}_i^T{X}_i& -X_i^T{X}_{\overline{ij}}\\ -X_{\overline{ij}}^T{X}_i& X_{\overline{ij}}^T{X}_{\overline{ij}}\end{array}\right)\left(\begin{array}{c}\alpha \\ \beta \end{array}\right)+{\left(\begin{array}{c}diag(X_i^T{X}_i)+2v_i\frac{v_1}{n_j}{X}_i^T{X}_j{e}_j\\ -diag(X_{\overline{ij}}^T{X}_{\overline{ij}})-2v_i\frac{v_1}{n_j}{X}_{\overline{ij}}^T{X}_j{e}_j\end{array}\right)}^T\left(\begin{array}{c}\alpha \\ \beta \end{array}\right),\hfill \\ s.t.\left(\begin{array}{cc}{e}_i^T& -e_{\overline{ij}}^T\end{array}\right)\left(\begin{array}{c}\alpha \\ \beta \end{array}\right)=1,\hfill \\ 0e_i\le \alpha \le \frac{c_1}{n_i}{e}_i,\hfill \\ 0e_{\overline{ij}}\le \beta \le \frac{c_2}{n_{\overline{ij}}}{e}_{\overline{ij}}.\hfill \end{array}$$

(26)

According to the KKT conditions (16)–(21), we can obtain $R_i^2$ by the following formula:

$$R_i^2={∥x^{*}-a_i∥}^2,$$

(27)

where $x^{*}\in S_{i1}\cup S_{i2}$, $S_{i1}=\left\{x_p^i\right|0<{\alpha }_p<\frac{c_1}{n_i}\}$ and $S_{i2}=\left\{x_p^{\overline{ij}}\right|0<{\beta }_p<\frac{c_2}{n_{\overline{ij}}}\}$.

By denoting $v_j=\frac{1}{1-v_2}$, the dual optimal problem of (12) can be obtained as follows:

$$\begin{array}{c}max{\displaystyle \sum _{p=1}^{n_j}}{\theta }_p{x}_p^j·x_p^j-{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\gamma }_p{x}_p^{\overline{ij}}·x_p^{\overline{ij}}+2v_j\frac{v_2}{n_i}{\displaystyle \sum _{p_1=1}^{n_i}}{\displaystyle \sum _{p_2=1}^{n_j}}{\theta }_{p_2}{x}_{p_1}^i·x_{p_2}^j-2v_j\frac{v_2}{n_i}{\displaystyle \sum _{p_1=1}^{n_i}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\gamma }_{p_2}{x}_{p_1}^i·x_{p_2}^{\overline{ij}}\hfill \\ -v_j{\displaystyle \sum _{p_1=1}^{n_j}}{\displaystyle \sum _{p_2=1}^{n_j}}{\theta }_{p_1}{\theta }_{p_2}{x}_{p_1}^j·x_{p_2}^j+2v_j{\displaystyle \sum _{p_1=1}^{n_j}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\theta }_{p_1}{\gamma }_{p_2}{x}_{p_1}^j·x_{p_2}^{\overline{ij}}-v_j{\displaystyle \sum _{p_1=1}^{n_{\overline{ij}}}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\gamma }_{p_1}{\gamma }_{p_2}{x}_{p_1}^{\overline{ij}}{x}_{p_2}^{\overline{ij}},\hfill \\ s.t.1-{\displaystyle \sum _{p=1}^{n_j}}{\theta }_p+{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\gamma }_p=0,\hfill \\ 0\le {\theta }_p\le \frac{c_3}{n_j},p=1,\cdots ,n_j,\hfill \\ 0\le {\gamma }_p\le \frac{c_4}{n_{\overline{ij}}},p=1,\cdots ,n_{\overline{ij}}.\hfill \end{array}$$

(28)

By defining $\theta =\left(\begin{array}{c}{\theta }_1\\ ⋮\\ {\theta }_{n_j}\end{array}\right)$ and $\gamma =\left(\begin{array}{c}{\gamma }_1\\ ⋮\\ {\gamma }_{n_{\overline{ij}}}\end{array}\right)$, the (28) can be reformulated as

$$\begin{array}{c}max-v_j(\begin{array}{cc}{\theta }^T& {\gamma }^T\end{array})\left(\begin{array}{cc}{X}_j^T{X}_j& -X_j^T{X}_{\overline{ij}}\\ -X_{\overline{ij}}^T{X}_j& X_{\overline{ij}}^T{X}_{\overline{ij}}\end{array}\right)\left(\begin{array}{c}\theta \\ \gamma \end{array}\right)+{\left(\begin{array}{c}diag(X_j^T{X}_j)+2v_j\frac{v_2}{n_i}{X}_j^T{X}_i{e}_i\\ -diag(X_{\overline{ij}}^T{X}_{\overline{ij}})-2v_j\frac{v_2}{n_i}{X}_{\overline{ij}}^T{X}_i{e}_i\end{array}\right)}^T\left(\begin{array}{c}\theta \\ \gamma \end{array}\right),\hfill \\ s.t.\left(\begin{array}{cc}{e}_j^T& -e_{\overline{ij}}^T\end{array}\right)\left(\begin{array}{c}\theta \\ \gamma \end{array}\right)=1,\hfill \\ 0e_j\le \theta \le \frac{c_3}{n_j}{e}_j,\hfill \\ 0e_{\overline{ij}}\le \gamma \le \frac{c_4}{n_{\overline{ij}}}{e}_{\overline{ij}}.\hfill \end{array}$$

(29)

We can compute $R_j^2$ by the following formula:

$$R_j^2={∥x^{*}-a_j∥}^2,$$

(30)

where $x^{*}\in S_{j1}\cup S_{j2}$, $S_{j1}=\left\{x_p^j\right|0<{\theta }_p<\frac{c_3}{n_j}\}$ and $S_{j2}=\left\{x_p^{\overline{ij}}\right|0<{\gamma }_p<\frac{c_4}{n_{\overline{ij}}}\}$.

3.2. Nonlinear Case

We extend the linear Twin Hypersphere-KSVC to the nonlinear case by directly considering the nonlinear map $\phi :R^d\to H$ (H is a high-dimensional Hilbert space), instead of the kernel generated surfaces in Twin-KSVC.

$$\begin{array}{c}min{R}_i^2-\frac{v_1}{n_j}{\displaystyle \sum _{p=1}^{n_j}}{∥\phi (x_p^j)-a_i∥}^2+\frac{c_1}{n_i}{\displaystyle \sum _{p=1}^{n_i}}{\eta }_p^i+\frac{c_2}{n_{\overline{ij}}}{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\xi }_p^i,\hfill \\ s.t.{∥\phi (x_p^i)-a_i∥}^2\le R_i^2+{\eta }_p^i,p=1,\cdots ,n_i,\hfill \\ {∥\phi (x_p^{\overline{ij}})-a_i∥}^2\ge R_i^2-{\xi }_p^i,p=1,\cdots ,n_{\overline{ij}},\hfill \\ {\eta }_p^i\ge 0,p=1,\cdots ,n_i,\hfill \\ {\xi }_p^i\ge 0,p=1,\cdots ,n_{\overline{ij}},\hfill \\ R_i^2\ge 0,\hfill \end{array}$$

(31)

$$\begin{array}{c}min{R}_j^2-\frac{v_2}{n_i}{\displaystyle \sum _{p=1}^{n_i}}{∥\phi (x_p^i)-a_j∥}^2+\frac{c_3}{n_j}{\displaystyle \sum _{p=1}^{n_j}}{\eta }_p^j+\frac{c_4}{n_{\overline{ij}}}{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\xi }_p^j,\hfill \\ s.t.{∥\phi (x_p^j)-a_j∥}^2\le R_j^2+{\eta }_p^j,p=1,\cdots ,n_j,\hfill \\ {∥\phi (x_p^{\overline{ij}})-a_j∥}^2\ge R_j^2-{\xi }_p^j,p=1,\cdots ,n_{\overline{ij}},\hfill \\ {\eta }_p^j\ge 0,p=1,\cdots ,n_j,\hfill \\ {\xi }_p^j\ge 0,p=1,\cdots ,n_{\overline{ij}},\hfill \\ R_j^2\ge 0.\hfill \end{array}$$

(32)

According to the dual theory, one can get the dual optimal problems of (31) and (32) as follows:

$$\begin{array}{c}max-v_i(\begin{array}{cc}{\alpha }^T& {\beta }^T\end{array})\left(\begin{array}{cc}K(X_i,X_i)& -K(X_i,X_{\overline{ij}})\\ -K(X_{\overline{ij}},X_i)& K(X_{\overline{ij}},X_{\overline{ij}})\end{array}\right)\left(\begin{array}{c}\alpha \\ \beta \end{array}\right)\hfill \\ +{\left(\begin{array}{c}diag(K(X_i,X_i))+2v_i\frac{v_1}{n_j}K(X_i,X_j)e_j\\ -diag(K(X_{\overline{ij}},X_{\overline{ij}}))-2v_i\frac{v_1}{n_j}K(X_{\overline{ij}},X_j)e_j\end{array}\right)}^T\left(\begin{array}{c}\alpha \\ \beta \end{array}\right),\hfill \\ s.t.\left(\begin{array}{cc}{e}_i^T& -e_{\overline{ij}}^T\end{array}\right)\left(\begin{array}{c}\alpha \\ \beta \end{array}\right)=1,\hfill \\ 0e_i\le \alpha \le \frac{c_1}{n_i}{e}_i,\hfill \\ 0e_{\overline{ij}}\le \beta \le \frac{c_2}{n_{\overline{ij}}}{e}_{\overline{ij}},\hfill \end{array}$$

(33)

$$\begin{array}{c}max-v_j(\begin{array}{cc}{\theta }^T& {\gamma }^T\end{array})\left(\begin{array}{cc}K(X_j,X_j)& -K(X_j,X_{\overline{ij}})\\ -K(X_{\overline{ij}},X_j)& K(X_{\overline{ij}},X_{\overline{ij}})\end{array}\right)\left(\begin{array}{c}\theta \\ \gamma \end{array}\right)\hfill \\ +{\left(\begin{array}{c}diag(K(X_j,X_j))+2v_j\frac{v_2}{n_i}K(X_j,X_i)e_i\\ -diag(K(X_{\overline{ij}},X_{\overline{ij}}))-2v_j\frac{v_2}{n_i}K(X_{\overline{ij}},X_i)e_i\end{array}\right)}^T\left(\begin{array}{c}\theta \\ \gamma \end{array}\right),\hfill \\ s.t.\left(\begin{array}{cc}{e}_j^T& -e_{\overline{ij}}^T\end{array}\right)\left(\begin{array}{c}\theta \\ \gamma \end{array}\right)=1,\hfill \\ 0e_j\le \theta \le \frac{c_3}{n_j}{e}_j,\hfill \\ 0e_{\overline{ij}}\le \gamma \le \frac{c_4}{n_{\overline{ij}}}{e}_{\overline{ij}},\hfill \end{array}$$

(34)

where $K(.,.)$ is a kernel matrix.

3.3. Decision Rule

For a testing sample x, each pair of sub-classifiers determines its label by

$$F_{ij}(x)=\left\{\begin{array}{cc}1& {∥\phi (x)-a_i∥}^2/R_i^2\le {∥\phi (x)-a_j∥}^2/R_j^2\\ -1& {∥\phi (x)-a_i∥}^2/R_i^2>{∥\phi (x)-a_j∥}^2/R_j^2\end{array}\right.$$

(35)

wher

$$\begin{array}{c}{∥\phi (x)-a_i∥}^2=K(x,x)+2v_i({\displaystyle \sum _{p=1}^{n_i}}{\alpha }_{p}K(x_p^i,x)-\frac{v_1}{n_j}{\displaystyle \sum _{p=1}^{n_j}}K(x_p^j,x)-{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\beta }_{p}K(x_p^{\overline{ij}},x))\hfill \\ +v_i^2({\displaystyle \sum _{p_1=1}^{n_i}}{\displaystyle \sum _{p_2=1}^{n_i}}{\alpha }_{p_1}{\alpha }_{p_2}K(x_{p_1}^i,x_{p_2}^i)+{(\frac{v_1}{n_j})}^2{\displaystyle \sum _{p_1=1}^{n_j}}{\displaystyle \sum _{p_2=1}^{n_j}}K(x_{p_1}^j,x_{p_2}^j)+{\displaystyle \sum _{p_1=1}^{n_{\overline{ij}}}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\beta }_{p_1}{\beta }_{p_2}K(x_{p_1}^{\overline{ij}},x_{p_2}^{\overline{ij}})\hfill \\ -\frac{2v_1}{n_j}{\displaystyle \sum _{p_1=1}^{n_i}}{\displaystyle \sum _{p_2=1}^{n_j}}{\alpha }_{p_1}K(x_{p_1}^i,x_{p_2}^j)-2{\displaystyle \sum _{p_1=1}^{n_i}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\alpha }_{p_1}{\beta }_{p_2}K(x_{p_1}^i,x_{p_2}^{\overline{ij}})+\frac{2v_1}{n_j}{\displaystyle \sum _{p_1=1}^{n_j}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\beta }_{p_2}K(x_{p_1}^i,x_{p_2}^{\overline{ij}}))\hfill \end{array}$$

(36)

and

$$\begin{array}{c}{∥\phi (x)-a_j∥}^2=K(x,x)+2v_j({\displaystyle \sum _{p=1}^{n_j}}{\theta }_{p}K(x_p^j,x)-\frac{v_2}{n_i}{\displaystyle \sum _{p=1}^{n_i}}K(x_p^i,x)-{\displaystyle \sum _{p=1}^{n_{\overline{ij}}}}{\gamma }_{p}K(x_p^{\overline{ij}},x))\hfill \\ +v_j^2({\displaystyle \sum _{p_1=1}^{n_j}}{\displaystyle \sum _{p_2=1}^{n_j}}{\theta }_{p_1}{\theta }_{p_2}K(x_{p_1}^j,x_{p_2}^j)+{(\frac{v_2}{n_i})}^2{\displaystyle \sum _{p_1=1}^{n_i}}{\displaystyle \sum _{p_2=1}^{n_i}}K(x_{p_1}^i,x_{p_2}^i)+{\displaystyle \sum _{p_1=1}^{n_{\overline{ij}}}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\gamma }_{p_1}{\gamma }_{p_2}K(x_{p_1}^{\overline{ij}},x_{p_2}^{\overline{ij}})\hfill \\ -\frac{2v_2}{n_i}{\displaystyle \sum _{p_1=1}^{n_j}}{\displaystyle \sum _{p_2=1}^{n_i}}{\theta }_{p_1}K(x_{p_1}^j,x_{p_2}^i)-2{\displaystyle \sum _{p_1=1}^{n_j}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\theta }_{p_1}{\gamma }_{p_2}K(x_{p_1}^j,x_{p_2}^{\overline{ij}})+\frac{2v_2}{n_i}{\displaystyle \sum _{p_1=1}^{n_i}}{\displaystyle \sum _{p_2=1}^{n_{\overline{ij}}}}{\gamma }_{p_2}K(x_{p_1}^i,x_{p_2}^{\overline{ij}})).\hfill \end{array}$$

(37)

For the testing sample x, the final label can be also determined by vote rule.

3.4. Analysis of Learning Complexity

Next, the learning complexity of the proposed Twin Hypersphere-KSVC will be discussed. We take the 4-class classification as an example, suppose samples of 4 classes are approximately equal, and present the learning complexity of K-SVCR, Twin-KSVC and Twin Hypersphere-KSVC in

Table 1. The learning complexities of Twin-KSVC, K-SVCR and Twin Hypersphere-KSVC.

Kernel	Twin-KSVC	K-SVCR	Twin Hypersphere-KSVC
linear kernel	$K(K-1)O(d^3)+O({(\frac{3}{4}n)}^3))$	$\frac{K(K-1)}{2}O({(\frac{3}{2}n)}^3)$	$K(K-1)O({(\frac{3}{4}n)}^3)$
nonlinear kernel	$K(K-1)(O(n^3)+O({(\frac{3}{4}n)}^3))$	$\frac{K(K-1)}{2}O({(\frac{3}{2}n)}^3)$	$K(K-1)$$O({(\frac{3}{4}n)}^3)$

. The main calculating burden in Twin-KSVC includes solving QPPs and calculating inverse matrices. Therefore the learning complexities of linear and nonlinear Twin-KSVC are respectively $K(K-1)$($O(d^3)$+$O({(\frac{3}{4}n)}^3)$) and $K(K-1)$($O(n^3)$+$O({(\frac{3}{4}n)}^3)$). However, K-SVCR and Twin Hypersphere-KSVC avoid computing inverse matrices, the learning complexities of linear and nonlinear K-SVCR are both $\frac{K(K-1)}{2}O({(\frac{3}{2}n)}^3)$ while the learning complexities of linear and nonlinear Twin Hypersphere-KSVC are both $K(K-1)$$O({(\frac{3}{4}n)}^3)$. From the above analysis, we can see that our Twin Hypersphere-KSVC requires less learning time.

4. Experiments

In this section, we investigate classification performance of our Twin Hypersphere-KSVC, ITKSVC [31], THKSVM, Twin-KSVC and K-SVCR on a set of benchmark datasets from the UCI repository and several real engineering problems. The above algorithms are implemented in Matlab R2012a, and we use the “quadprog.m” function to solve the QPP and the “inv.m” function to calculate matrix inversion.

The parameter selection directly affects the classification performance of the above algorithms. We use the most popular exhaustive search to determine the parameters in this section. The K-SVCR includes two penalty parameters $c_i(i=1,2)$ and bandwidth parameter $\epsilon$. The Twin-KSVC has five parameters, including four penalty parameters $c_i(i=1,2,3,4)$ and bandwidth parameter $\epsilon$. The THKSVM holds the penalty parameters $c_1$ and $v_1$. The ITKSVC contains seven parameters which are six penalty parameters $c_i(i=1,2,3,4,5,6)$ and bandwidth parameter $\epsilon$. There exist six penalty parameters $c_i(i=1,2,3,4)$ and $v_i(i=1,2)$ in our Twin Hypersphere-KSVC. The optimal values of penalty parameters $c_i$ are searched from set $\{2^{-7},\cdots ,2^7\}$, penalty parameters $v_i$ from $\{0.1,\cdots ,0.9\}$ and bandwidth parameter $\epsilon$ from set $\{0,\cdots ,0.5\}$.

4.1. Benchmark Datasets

We compare Twin Hypersphere-KSVC with ITKSVC, THKSVM, Twin-KSVC and K-SVCR on a set of benchmark datasets from the UCI repository in this subsection. The benchmark datasets are presented in

Table 2. The statistics of benchmark datasets.

Dataset	#Attributes	#Samples	#Classes
Wine	13	178	3
Iris	4	150	3
Ecoli	7	327	5
Soybean	35	47	4
Hayes-roth	5	132	3
Teaching-evaluation	5	151	3
Dermatology	34	358	6
Balance	4	625	3

. The 5-fold cross validation is used to estimate the testing accuracy and we use radial basis function $K(x,y)=e^{-{∥x-y∥}^2/{\sigma }^2}$ in this subsection, where the parameter $\sigma$ are selected from $\{2^{-7},\cdots ,2^7\}$.

The predicting accuracy and learning time of five algorithms on UCI benchmark data sets are respectively shown in

Table 3. Classification accuracy of THKSVM, ITKSVC, Twin-KSVC, K-SVCR and Twin Hypersphere-KSVC on UCI benchmark datasets.

Dataset	Twin-KSVC(%)	THKSVM(%)	K-SVCR(%)	ITKSVC(%)	Twin Hypersphere-KSVC(%)
Wine	97.29 ± 2.28	92.24 ± 10.22	97.96 ± 2.47	97.53 ± 2.20	98.30 ± 2.55
Iris	93.47 ± 3.67	91.20 ± 4.52	95.47 ± 3.07	95.53 ± 4.03	96.00 ± 3.06
Ecoli	84.33 ± 4.60	77.75 ± 2.80	87.34 ± 3.07	85.63 ± 3.70	83.04 ± 4.21
Soybean	98.00 ± 4.47	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0	100.0 ± 0.0
Hayes-roth	61.07 ± 7.90	39.40 ± 2.05	54.02 ± 8.31	71.28 ± 9.92	52.50 ± 8.21
Teaching-evaluation	56.43 ± 8.23	50.28 ± 11.71	51.91 ± 8.34	52.02 ± 6.27	59.44 ± 7.52
Dermatology	95.43 ± 2.19	70.18 ± 3.16	97.66 ± 1.82	94.19 ± 2.74	96.64 ± 2.39
Balance	96.92 ± 1.29	90.05 ± 2.94	94.37 ± 1.96	97.73 ± 1.19	90.91 ± 1.34

The bolded classification accuracy is the highest one in all multi-classifiers.

and

Table 4. Learning time of THKSVM, ITKSVC, Twin-KSVC, K-SVCR and Twin Hypersphere-KSVC on UCI benchmark datasets.

Dataset	Twin-KSVC(s)	THKSVM(s)	K-SVCR(s)	ITKSVC(s)	Twin Hypersphere-KSVC(s)
Wine	2.1985	0.2749	2.5362	1.8673	1.1457
Iris	1.7260	0.2533	2.0569	1.5347	0.7429
Ecoli	38.7993	1.0329	62.2968	27.2987	15.1017
Soybean	1.1527	0.1787	0.4666	0.6355	0.5374
Hayes-roth	2.4358	0.1848	1.7696	1.1364	0.5671
Teaching-evaluation	3.6512	0.1849	1.7180	1.2330	1.0899
Dermatology	114.424	1.0811	89.2057	41.0467	19.6405
Balance	45.2907	3.7804	68.7381	43.7113	13.5046

. By observing Table 3 and Table 4, one can come to the following conclusions.

(1)

The Twin Hypersphere-KSVC, ITKSVC, Twin-KSVC and K-SVCR obtain better test accuracy than THKSVM. This is mainly because the sub-classifiers in Twin Hypersphere-KSVC, ITKSVC, Twin-KSVC and K-SVCR avoid the class imbalance problem appearing in THKSVM.

(2)

The Twin Hypersphere-KSVC works faster than Twin-KSVC, K-SVCR and ITKSVC. It is mainly because, compared with Twin-KSVC, the sub-classifiers in Twin Hypersphere-KSVC avoid calculating inverse matrices which appear in Twin-KSVC, while compared with K-SVCR and ITKSVC, each pair of sub-classifiers of our Twin Hypersphere-KSVC only needs to resolve two smaller QPPs.

(3)

For predicting accuracy of Twin-KSVC, K-SVCR, ITKSVC and Twin Hypersphere-KSVC, we can observe that not any method is superior to others for all data sets. We can apply Friedman test to analyze the test accuracy of these classifiers statistically [33,34]. The ranks of five algorithms for all data set in the light of test accuracy are presented in

Table 5. The rank of five algorithms on UCI benchmark datasets in the light of classification accuracy.

Dataset	Twin-KSVC	THKSVM	K-SVCR	ITKSVC	Twin Hypersphere-KSVC
Wine	4	5	2	3	1
Iris	4	5	3	2	1
Ecoli	3	5	1	2	4
Soybean	5	2.5	2.5	2.5	2.5
Hayes-roth	2	5	3	1	4
Teaching-evaluation	2	5	4	3	1
Dermatology	3	5	1	4	2
Balance	2	5	3	1	4
Average	3.13	4.69	2.81	2.31	2.44

. The Friedman statistic ${\chi }_F^2$ can be calculated by

$${\chi }_F^2=\frac{12m_2}{m_1(m_1+1)}[{\displaystyle \sum _{k_1=1}^{m_1}}ran{k}_{k_1}^2-\frac{m_1{(m_1+1)}^2}{4}],$$

(38)

where $ran{k}_{k_1}=\frac{1}{m_2}{\displaystyle \sum _{k_2=1}^{m_2}}{r}_{k_1}^{k_2}$, $r_{k_1}^{k_2}$ denotes the rank of the $k_1$-th of $m_1$ classifiers on the $k_2$-th of $m_2$ data sets. Because ${\chi }_F^2$ is undesirably conservative, we can use the other statistic

$$F_F=\frac{(m_2-1){\chi }_F^2}{m_2(m_1-1)-{\chi }_F^2},$$

(39)

which is distributed by the $F(m_1-1,(m_1-1)(m_2-1))$.

We can calculate the statistic $F_F=10.41$, where $F_F\sim F(4,28)$. For the level of significance $\alpha =0.05$, $F(4,28)=2.95$ is smaller than $F_F$, which means there are significant differences among these classifiers. We can see from Table 5 that the average rank of Twin Hypersphere-KSVC is lower than ITKSVC, and is higher than Twin-KSVC and K-SVCR. It implies that classification accuracy of Twin Hypersphere-KSVC is slightly lower than ITKSVC, however, is much higher than Twin-KSVC and K-SVCR.

4.2. Handwritten Digits Recognition

We use Twin Hypersphere-KSVC to recognize handwritten digits. The USPS database is used to compare our Twin Hypersphere-KSVC with Twin-KSVC, THKSVM, ITKSVC and K-SVCR. The USPS dataset consists of 8-bit grayscale images of handwritten digits from 0 to 9, as presented in Figure 2. We choose 55 images for each handwritten digit in the USPS database, 550 images in total. We only consider linear kernel function $K(x,y)=x^{T}y$, and also use 5-fold cross validation to estimate the testing accuracy.

The handwritten digits recognition results of five algorithms are presented in

Table 6. Classification results of Twin Hypersphere-KSVC, Twin-KSVC, THKSVM, K-SVCR and ITKSVC for handwritten digits recognition on the USPS dataset.

Algorithms	Twin-KSVC	THKSVM	K-SVCR	ITKSVC	Twin Hypersphere-KSVC
Accuracy (%)	53.13 ± 5.38	72.55 ± 3.55	85.09 ± 2.99	77.18 ± 4.24	85.75 ± 2.15
Learning time (s)	377.5463	2.1101	641.946	348.6982	342.983

. From Table 6, we can observe that, the proposed Twin Hypersphere-KSVC obtains better accuracy among all algorithms. In terms of learning time, our Twin Hypersphere-KSVC costs shorter learning time, compared with ITKSVC, Twin-KSVC and K-SVCR.

4.3. Text Classification

We evaluate our Twin Hypersphere-KSVC to text classification in this subsection and compare it with the other algorithms on the Reuters21578 dataset. We choose 6 classes from the Reuters21578 dataset, 708 documents in total, which are presented in

Table 7. The statistics of text classification dataset.

Label	Cocoa	Coffee	Corn	Cice	Cubber	Soybean
Training dataset	52	101	176	46	35	83
Test dataset	23	44	76	21	15	36

. We also consider linear kernel function in this subsection.

The experimental results of five algorithms for text classification are presented in

Table 8. Classification results of Twin Hypersphere-KSVC, Twin-KSVC, THKSVM, K-SVCR and ITKSVC for text classification on the Reuters21578 dataset.

Algorithms	Twin-KSVC	THKSVM	K-SVCR	ITKSVC	Twin Hypersphere-KSVC
F1	0.4474	0.5932	0.7076	0.7078	0.7414
Learning time (s)	36.9486	0.4758	68.7947	26.4216	11.9575

. By observing Table 8, we can notice that, our Twin Hypersphere-KSVC can get better accuracy among all multi-classifiers. The proposed Twin Hypersphere-KSVC runs faster in comparison with ITKSVC, Twin-KSVC and K-SVCR.

5. Conclusions

In this paper, we propose a novel multi-class classification algorithm, named Twin Hypersphere-KSVC. The Twin Hypersphere-KSVC evaluates each training sample into 1-vs-1-vs-rest structure, as in Twin-KSVC and K-SVCR, and constructs two hyperspheres in each pair of sub-classifiers, instead of two nonparallel hyperplanes as in Twin-KSVC. Compared with Twin-KSVC, the sub-classifiers in Twin Hypersphere-KSVC avoid computing inverse matrices, and for nonlinear problems, can apply the kernel trick to linear case directly. The classification results on a set of benchmark datasets from UCI repository, handwritten digits recognition and text classification, show that the Twin Hypersphere-KSVC gets better classification performance in comparison with the other classical multi-classifiers.