An Improved Three-Way Clustering Based on Ensemble Strategy

Abstract

As a powerful data analysis technique, clustering plays an important role in data mining. Traditional hard clustering uses one set with a crisp boundary to represent a cluster, which cannot solve the problem of inaccurate decision-making caused by inaccurate information or insufficient data. In order to solve this problem, three-way clustering was presented to show the uncertainty information in the dataset by adding the concept of fringe region. In this paper, we present an improved three-way clustering algorithm based on an ensemble strategy. Different to the existing clustering ensemble methods by using various clustering algorithms to produce the base clustering results, the proposed algorithm randomly extracts a feature subset of samples and uses the traditional clustering algorithm to obtain the diverse base clustering results. Based on the base clustering results, labels matching is used to align all clustering results in a given order and voting method is used to obtain the core region and the fringe region of the three way clustering. The proposed algorithm can be applied on the top of any existing hard clustering algorithm to generate the base clustering results. As examples for demonstration, we apply the proposed algorithm on the top of K-means and spectral clustering, respectively. The experimental results show that the proposed algorithm is effective in revealing cluster structures.

1. Introduction

Clustering aims to classify similar elements into the same cluster and dissimilar elements into different clusters by calculating the certain similarity between all elements, where the elements in the same cluster must have a high similarity and the elements in the different clusters have a low similarity [1,2]. As a key technology of machine learning, clustering is widely used in different fields such as information granulation [3,4,5], information fusion [6,7,8], attribute reduction [9,10,11,12], feature selection [13,14,15], etc. Many clustering methods have been developed over the past decades. Most of the existing clustering algorithms can be divided into five categories: the partition-based method [16,17], hierarchy-based method [18,19,20], density-based method [21,22,23,24], grid-based method [25] and model-based method [26,27,28].

Although there are many different clustering algorithms, the lack of prior knowledge means that clustering analysis remains a very challenging problem. It has been accepted that a single clustering approach cannot always describe the structural characteristics accurately, even if the same clustering approach cannot obtain good clusters because of the different initial parameters as well. To avoid this problem, ensemble clustering [29,30,31] has been developed to improve the robustness, stability, and quality of a clustering solution. Compared with a single clustering approach, the clustering results obtained by ensemble clustering approaches have better stability, robustness and accuracy. In recent years, ensemble clustering has received more attention and many new ensemble clustering approaches have been developed [32,33,34,35].

The above works have obtained good performance in solving a clustering ensemble problem. However, most of the existing ensemble clustering results are hard clustering, in which an element belongs to one cluster or does not belong to one cluster, and there is a clear boundary between different clusters. Hard clustering algorithms often bring a higher decision risk when the sample’s information is insufficient. In order to solve this problem, three-way decision [36] was proposed to describe the uncertainty of the information. The main idea of three-way decision is to divide a universe into three disjoint regions and make different strategies for different regions [37,38,39]. Many soft computing models for learning uncertain concepts, such as rough sets [40], fuzzy sets [41], shadowed sets [42] and concept learning [43,44] can be reinvestigated within the framework of three-way decision. By integrating three-way decision with clustering, Yu [45,46,47] proposed three-way clustering, which uses a core region and the fringe region to represent a cluster. These two sets divide the universe into three parts, where there are three types of relationships between the objects and cluster, namely objects belonging to the cluster, objects not belonging to the cluster, and objects partially belonging to the cluster. Recently, three-way clustering has attracted much research, and many three-way clustering algorithms have been developed [48,49,50,51,52,53,54,55,56].

This paper aims at presenting an improved three-way clustering algorithm based on an ensemble strategy to solve the problem of inaccurate decision-making caused by inaccurate information or insufficient data. Different to the existing clustering ensemble methods by using various clustering algorithms to produce the base clustering results, the proposed algorithm randomly extracts a feature subset of samples and uses the traditional clustering algorithm to obtain the diverse base clustering results. Based on the base clustering results, we develop a three-way clustering method by using the voting method. The main process of the proposed algorithm has two steps. Firstly, we use parts of the sample’s features to obtain the base clustering results. Secondly, we use label matching to align all clustering results to a given order and the voting method to obtain the core region and the fringe region of three-way clustering. The sample is assigned into the core region of a corresponding cluster when the frequency of the sample in the same cluster is more than the given threshold. The difference between the union of the cluster with the same labels and the core region are regarded as the fringe region of the specific cluster. Therefore, a three-way clustering is naturally formed. The proposed strategy can be applied on the top of any existing hard clustering algorithm. Three-way ensemble K-means and NJW are given as examples for demonstration in this paper.

The remainder of this paper is organized as follows. In Section 2, we mainly introduce the concepts of ensemble clustering and three-way clustering. The process of the proposed algorithm is presented in Section 3. The performances of the proposed three-way ensemble clustering algorithm are illustrated through some UCI datasets in Section 4. Conclusions and future works are given in Section 5.

2. Related Work

In this section, we review some concepts and related works about ensemble clustering and three-way clustering.

2.1. Ensemble Clustering

Each clustering algorithm has its unique method for discovering the data structure. Different clustering algorithms can obtain different clustering results even if the data are the same. A single clustering algorithm cannot deal with all types of data structure. It is also difficult to choose a specialized clustering algorithm because of the insufficient prior class information. Hence, researchers are devoted to integrating multiple clustering results to one clustering result, which is called ensemble clustering. Compared with a single clustering algorithm, ensemble clustering algorithm can obtain a better clustering result with a higher performance of robustness, stability and quality.

The concept of ensemble clustering was first proposed by Strehl and Ghosh [29], who combined cluster labels without accessing the original features. Wang et al. [57] developed an ensemble clustering method for probabilities accumulation by considering factors such as cluster size, sample dimension and density. Punera and Ghosh [58] proposed several consensus algorithms that were suitable for soft clustering by extending the relatively hard clustering approaches. Sevillano et al. [59] presented a set of fuzzy consensus functions which can combine multiple soft clustering results into a final soft clustering result, using the application of positional and confidence voting techniques. Li et al. [60] developed an ensemble clustering algorithm based on a sample’s stability. Yu et al. [55] presented a framework of three-way ensemble clustering based on Spark and proposed a consensus clustering algorithm based on cluster units.

In general, ensemble clustering can be roughly divided into two stages: base cluster generation and base cluster aggregation. Base cluster generation is the first step of the ensemble clustering algorithm, and a set of clusters which will be combined should be generated. There are no restrictions on how to achieve base clusters, so we may have many approaches, such as using different clustering algorithms or using the same clustering algorithm with different parameters. In this paper, we mainly research the process of base cluster aggregation and how to convert hard clustering results into a soft clustering result. The process of ensemble clustering is shown in Figure 1.

2.2. Three-Way Clustering

Three-way decision  [36] is an extension of two-way decision, in which a definite decision is given to the elements with definite information and a deferment decision is adopted when the elements’ information is insufficient to avoid decision risk. Three-way decision uses three disjointed regions to represent a set, namely the positive region, the negative region and the boundary region, which correspond to the acceptance decision, the rejection decision and the delayed decision. Inspired by the idea of three-way decision, Yu [45] presented the framework of three-way clustering by combining the clustering approach and three-way decision.

We introduce some basic knowledge about three-way clustering. Given a set $U=\{x_1,x_2,\cdots ,x_n\}$, which has n elements, the clustering result can be denoted as $\mathbb{C}=\{C^1,C^2,\cdots ,C^K\}$ by using the hard clustering algorithm. In contrast to the hard clustering representation, the three-way cluster $C^i$ is represented as a pair of sets:

$$\begin{array}{c}\hfill C^i=\{Co\left(C^i\right),Fr\left(C^i\right)\},\end{array}$$

(1)

where $Co\left(C^i\right)$ is the core region of cluster $C^i$ and $Fr\left(C^i\right)$ is the fringe region of cluster $C^i$. The third region is trivial region $Tr\left(C^i\right)=U-Co\left(C^i\right)-Fr\left(C^i\right)$, which can be expressed as the complement of the union of $Co\left(C^i\right)$ and $Fr\left(C^i\right)$.

We summarize some concepts of three-way clustering. If the element $v\in Co\left(C^i\right)$, v must belong to the cluster $C^i$; if the element $v\in Fr\left(C^i\right)$, v might belong to the cluster $C^i$; if the element $v\in Tr\left(C^i\right)$, v does not belong to the cluster $C^i$. The above three subsets obey the following properties:

$$\begin{array}{cc}& Co\left(C^i\right)\cap Fr\left(C^i\right)=⌀,\end{array}$$

(2)

$$\begin{array}{cc}& Co\left(C^i\right)\cap Tr\left(C^i\right)=⌀,\end{array}$$

(3)

$$\begin{array}{cc}& Fr\left(C^i\right)\cap Tr\left(C^i\right)=⌀,\end{array}$$

(4)

$$\begin{array}{cc}& Tr\left(C^i\right)\cup Co\left(C^i\right)\cup Fr\left(C^i\right)=U.\end{array}$$

(5)

When $Fr\left(C^i\right)=⌀$, it is obvious to find that the cluster $C^i$ can be only represented by the core region $Co\left(C^i\right)$, and it is a hard clustering result. So the hard clustering result is a special case of a three-way clustering result in a certain situation.

In this paper, we adopt the following three conditions:

(1)

$Co\left(C^i\right)\ne ⌀,i=1,2,\cdots$, k,

(2)

$\bigcup _{i=1}^K(Co\left(C^i\right)\cup Fr\left(C^i\right))=U$,

(3)

$Co\left(C^i\right)\cap Co\left(C^j\right)=⌀,i\ne j$.

Condition (1) means that each cluster must have elements. Condition (2) indicates that an element v belongs to more than one cluster. Condition (3) demands that the core region of each cluster must be disjointed. Therefore, we can represent the three-way clustering results as:

$$\mathbb{C}=\{(Co\left(C^i\right),Fr\left(C^i\right)),(Co\left(C^i\right),Fr\left(C^i\right)),\cdots ,(Co\left(C^K\right),Fr\left(C^K\right))\}.$$

Some three-way clustering algorithms were developed since three-way clustering was proposed. Wang [49] proposed a three-way clustering framework by combining ideas of three-way decision and erosion and dilation from mathematical morphology. Wang et al. [50] improved the K-means algorithm and then developed a three-way k-means method. Zhang [61] presented a three-way c-means clustering algorithm by integrating the three-way weight and three-way assignment. Yu et al. [47] proposed an efficient three-way clustering algorithm based on the idea of universal gravitation, which can adjust the thresholds automatically in the process of clustering. Jia et al. [62] developed an automatic three-way clustering approach by combining the proposed threshold selection based on the roughness degree using sample’s similarity and the cluster number selection method.

3. The Proposed Three-Way Clustering Based on Ensemble Strategy

Three-way clustering was presented to show the uncertainty information in the dataset by adding the concept of a fringe region. Although many three-way clustering algorithms have obtained good performances, there is still much room for improving the methods. We present an improved three-way clustering algorithm based on an ensemble strategy (TWCE for short) in this section. Compared with the existing algorithms, the proposed algorithm randomly extracts a feature subset of samples and uses the traditional clustering algorithm to obtain the diverse base clustering results. The computational complexity is lower than that of the existing clustering ensemble methods by using all the sample’s features. In addition, the proposed strategy can be applied on the top of any existing hard clustering algorithm. The process of the proposed algorithm has three steps: generation of base clustering, labels matching and results of three-way clustering.

3.1. Generation of Base Clustering Results

The first task in the clustering ensemble is to obtain a set of base clustering results. There are many approaches to generating base clustering clusters, among which utilizing different clustering algorithms is the most commonly used strategy. Each clustering algorithm has its own specific view on how to discover the underlying structure of a dataset. Therefore, multiple clustering algorithms can be used to generate diverse base clustering results. Another commonly used method to obtain the base clustering results is to use one clustering algorithm with different parameters. For example, base clustering results can be obtained by setting different numbers of clusters and the initial centers of the K-means type of algorithms. The above methods use all the sample’s features during the process of clustering, which will take a lot of time for a multi-dimensional dataset.

Different from the existing clustering ensemble method, the proposed TWCE algorithm uses parts of the sample’s features to obtain the base clustering results. For a multi-dimensional dataset, different subsets of features try to describe the dataset from different views. Thus, a set of diverse clustering results will be obtained when distinguishing subsets of features are utilized. Suppose U is a dataset with m features, we randomly extract parts of the features and use the traditional clustering algorithm to obtain one clustering result. Repeat the above process t times and obtain various clustering results ${\mathbb{C}}_1,{\mathbb{C}}_2,\cdots ,{\mathbb{C}}_t$. The generation process of base clustering results can be depicted as Algorithm 1.

Algorithm 1: Generation of base clustering results.

Input: Data set $U=\{v_1,v_2,\cdots ,v_n\}$, ensemble size t, cluster number K Output: ${\mathbb{C}}_1^{*},{\mathbb{C}}_2^{*},\cdots ,{\mathbb{C}}_t^{*}$

3.2. Labels Matching

Based on the base clustering results, we use the voting method to obtain the core region and the fringe region of three way clustering. The base clustering results ${\mathbb{C}}_1^{*},{\mathbb{C}}_2^{*},\cdots ,{\mathbb{C}}_t^{*}$ cannot be directly used for voting due to the lack of a priori category information. As an example, we consider the dataset $U=(v_1,v_2,v_3,v_4,v_5,v_6)$ and let ${\mathbb{C}}_1,{\mathbb{C}}_2$ and ${\mathbb{C}}_3$ be three clustering results of V, which are shown in

Table 1. Different represents of the same clustering results.

	${\mathbb{C}}_1$	${\mathbb{C}}_2$	${\mathbb{C}}_3$
$v_1$	1	2	3
$v_2$	1	2	3
$v_3$	2	3	2
$v_4$	2	3	2
$v_5$	3	1	1
$v_6$	3	1	1

.

Although the clustering results are expressed in different orders, they represent the same clustering result. In order to combine the clustering results, the cluster labels must be matched to establish the correspondence between each other. Zhou and Tang [31] pointed out that the number of common elements covered by two clusters with the corresponding relationship should be the largest. Therefore, for two base clustering results ${\mathbb{C}}_i=\{C_i^1,\cdots ,C_i^k,\cdots ,C_i^K\}$, ${\mathbb{C}}_j=\{C_j^1,\cdots ,C_j^k,\cdots ,C_j^K\}$, $1\le i,j\le t$, t is the number of base clusters; we record the number of identical elements covered by each cluster $C_i^{k_1}$, $C_j^{k_2}$$(1\le k_1,k_2\le K)$ in the two partitions in the OVERLAP matrix of $K\times K$. Then select the cluster label that covers the largest number of identical elements to establish a corresponding relationship, and delete the result from the matrix OVERLAP. Repeat the above process until all cluster labels have established corresponding relationships. This process is defined as label matching. When there are $t(t>2)$ clustering results, we can select the first clustering results as the matching criterion and match the other clustering results with the first one. The procedure of label matching can be shown as Algorithm 2.   

Algorithm 2: Label matching.

3.3. Results of Three-Way Clustering

After the process of label matching, updated cluster results ${\mathbb{C}}_1,{\mathbb{C}}_2,\cdots ,{\mathbb{C}}_t$ can be obtained. Then we use the voting method to achieve the core region and fringe region of three-way clustering. In the voting process, each clustering result can be regarded as a voter, and they will vote once and only once for each data point. Let $C^j={\cup }_{i=1}^t{C}_i^j$, $(j=1,2,\cdots$, $K)$. For any $v\in C^j$, we count the votes of the element $v\in C_i^j$, $(i=1,2,\cdots$, $t)$ and denote the votes of the element v as count(v). Suppose p is one given threshold. If count(v) $\ge p$, we assign the element v to the core region of the cluster $C^j$, otherwise, we assign the element v to the fringe region of the cluster $C^j$. Finally, we can obtain three-way clustering results. The process of finding the core region and the fringe region can be depicted as Algorithm 3.   

Algorithm 3: Finding core region and fringe region of TWCE.

    Based on Algorithms 1–3, a three-way clustering result that describes the data distribution will be generated. Sequentially executing Algorithms 1–3 forms the framework of the proposed TWCE, which is shown as Algorithm 4.   

Algorithm 4: Three way clustering based on ensemble strategy (TWCE).

Input: Data set $U=\{v_1,v_2,\cdots ,v_n\}$, ensemble size t, cluster number K, threshold p   Output: $\mathbb{C}=\{(Co\left(C^1\right),Fr\left(C^1\right)),(Co\left(C^2\right),Fr\left(C^2\right)),\cdots ,(Co\left(C^K\right),Fr\left(C^K\right))\}$ 1 ${\mathbb{C}}_1^{*},{\mathbb{C}}_2^{*},\cdots ,{\mathbb{C}}_t^{*}\leftarrow$ Algorithm 1; 2 ${\mathbb{C}}_1,{\mathbb{C}}_2,\cdots ,{\mathbb{C}}_t\leftarrow$ Algorithm 2; 3 $\{(Co\left(C^1\right),Fr\left(C^1\right)),(Co\left(C^2\right),Fr\left(C^2\right)),\cdots ,(Co\left(C^K\right),Fr\left(C^K\right))\}\leftarrow$ Algorithm 3; 4 Return $\mathbb{C}=\{(Co\left(C^1\right),Fr\left(C^1\right)),(Co\left(C^2\right),Fr\left(C^2\right)),\cdots ,(Co\left(C^K\right),Fr\left(C^K\right))\}$.

4. Experimental Analyses

In this section, we verify the performances of the proposed TWCE strategy. In the process of the experiments, K-means [16] and NJW [63] are used to generate the base clustering results, respectively, and the percentage of the selected feature subsets are randomly 50%, 60%, 70%, 80% or 90%. This section consists of three parts. In the first part, we introduce some popular clustering evaluation indices. In the second part, 12 datasets from the UCI Machine Learning repository are employed to show the working mechanism of the TWCE strategy. The relation between the clustering performances and the percentage of the selected feature subsets is discussed in the third part.

4.1. Evaluation Indices

The evaluation of clustering is an effective process for assessing the performance of clustering algorithms. We compare the proposed algorithm with other existing clustering algorithms by calculating some evaluation indices such as Normalized Mutual Information ($NMI$) [64], Adjusted Rand Index ($ARI$) [65] and Accuracy ($Acc$) [50]. The three validity metrics $NMI$, $ARI$ and $Acc$ are all positive indices, that is, the larger the value, the better the clustering effect.

• Normalized Mutual Information ($NMI$)

  $$NMI=\frac{I(X,Y)}{\sqrt{H\left(X\right)H\left(Y\right)},}$$

  (6)

  where X is the test label, and Y is the real label. $H\left(X\right)$ and $H\left(Y\right)$ represent the entropy of X and Y, respectively. $I(X,Y)$ is the mutual information between X and Y.
• Adjusted Rand Index($ARI$)

  $$ARI=\frac{2(ad-bc)}{(a+b)(b+d)+(a+c)(c+d)},$$

  (7)

  where a is the number of data points in a pair that belong to the same cluster in both real and experimental situations; b is the number of data points in a pair that belong to the same cluster in real but not in experimental situations; c is the number of data points in a pair that belong to the same cluster in experimental but not in real situations; d is the number of data points in a pair that do not belong to the same cluster in both real and experimental situations.
• Accuracy ($Acc$)

  $$Acc=\frac{1}{N}\sum _{i=1}^k{n}_i,$$

  (8)

  where N is the total number of elements, $n_i$ is the number of elements which are correctly divided into corresponding cluster i, and k is cluster number. $Acc$ represents the ratio between the number of correctly partitioned elements and the total number. A higher value of $Acc$ means the clustering result is better.

4.2. Performances of TWCE Strategy

To test the performances of our proposed TWCE strategy, we employ 12 datasets from the UCI Machine Learning repository, which are Cardiotocography, Congressional voting, Dermatology, Forrest, Landsat, Optical recognition, Synthetic, Urban Land Cover, Vehicle, Waveform, Wdbc and Wine.

Table 2. A description of dataset used.

ID	Data Sets	Samples	Attributes	Classes
1	Cardiotocography	2126	21	10
2	Congressional voting	435	16	2
3	Dermatology	366	34	6
4	Forrest	523	27	4
5	Landsat	6435	36	6
6	Optical recognition	5620	64	10
7	Synthetic	600	60	6
8	Urban Land Cover	675	147	9
9	Vehicle	846	18	4
10	Waveform	5000	21	3
11	Wdbc	569	30	2
12	Wine	178	13	3

shows the details of these datasets. The first step of the TWCE strategy is to obtain base clustering results. Different clustering algorithms may obtain different clustering results. We use the K-means algorithm and the NJW algorithm to generate base clusters in our experiments, which randomly select 50%, 60%, 70%, 80% or 90% feature subsets to clustering, respectively.

Because the evaluation indices $NMI$, $ARI$ and $Acc$ are only adopted to the hard clustering results, a three-way clustering results cannot calculate these values directly. In order to present the performances of our proposed TWCE algorithm, we use the core regions to form a clustering result, then calculate the $NMI$, $ARI$ and $Acc$ valuse by using the core region to represent the corresponding cluster. The average $NMI$ value, $ARI$ value and $Acc$ value are achieved by running 30 times on all datasets and the ensemble size is 50.

Table 3. The performances of TWCE using K-means.

ID	Data Sets	NMI	ARI	Acc
1	Cardiotocography	0.6795	0.5746	0.7611
2	Congressional voting	0.5648	0.6475	0.9026
3	Dermatology	0.9777	0.9776	0.9748
4	Forrest	0.6040	0.5709	0.8221
5	Landsat	0.7151	0.6491	0.7829
6	Optical recognition	0.9573	0.9654	0.9789
7	Synthetic	0.9008	0.8761	0.8947
8	Urban Land Cover	0.6851	0.6543	0.8233
9	Vehicle	0.2812	0.1888	0.4792
10	Waveform	0.4478	0.3108	0.5574
11	Wdbc	0.6875	0.7763	0.9412
12	Wine	0.9219	0.9418	0.9800

and

Table 4. The performances of TWCE using NJW.

ID	Data Sets	NMI	ARI	Acc
1	Cardiotocography	0.6320	0.5135	0.7721
2	Congressional voting	0.5619	0.6430	0.9012
3	Dermatology	0.9850	0.9902	0.9914
4	Forrest	0.6446	0.6410	0.8525
5	Landsat	0.7776	0.7444	0.8440
6	Optical recognition	0.9623	0.9700	0.9798
7	Synthetic	0.9121	0.8885	0.9053
8	Urban Land Cover	0.6870	0.6380	0.8059
9	Vehicle	0.2648	0.1874	0.5081
10	Waveform	0.3326	0.1462	0.5361
11	Wdbc	0.6835	0.7920	0.9452
12	Wine	0.9230	0.9437	0.9812

show the performances of the TWCE strategy based on the K-means and NJW algorithm, respectively. For comparing the clustering effect, the performances of K-means, NJW, Voting [31] and CSPA [29] are also presented in

Table 5. The performances of average $NMI$ value.

ID	Data Sets	K-Means	NJW	Voting	CSPA
1	Cardiotocography	0.3305	0.3314	0.3772	0.3256
2	Congressional voting	0.4574	0.4462	0.4541	0.4640
3	Dermatology	0.8130	0.8226	0.9133	0.8614
4	Forrest	0.5424	0.5428	0.5439	0.4898
5	Landsat	0.6125	0.6028	0.6482	0.6134
6	Optical recognition	0.7335	0.7312	0.7750	0.7385
7	Synthetic	0.7418	0.7587	0.8035	0.7292
8	Urban Land Cover	0.5770	0.5239	0.6029	0.5491
9	Vehicle	0.1126	0.0844	0.1418	0.1058
10	Waveform	0.3642	0.3668	0.2682	0.3639
11	Wdbc	0.6232	0.6075	0.6190	0.6175
12	Wine	0.8249	0.8782	0.8369	0.8354

,

Table 6. The performances of average $ARI$ value.

ID	Data Sets	K-Means	NJW	Voting	CSPA
1	Cardiotocography	0.1611	0.1547	0.2700	0.1567
2	Congressional voting	0.5287	0.5263	0.5381	0.5368
3	Dermatology	0.6544	0.7487	0.8446	0.7685
4	Forrest	0.4956	0.4904	0.4952	0.4299
5	Landsat	0.5264	0.4961	0.5725	0.5276
6	Optical recognition	0.6358	0.6583	0.7215	0.6525
7	Synthetic	0.5902	0.6336	0.7135	0.5730
8	Urban Land Cover	0.4587	0.4021	0.5245	0.4181
9	Vehicle	0.0800	0.0657	0.0965	0.0829
10	Waveform	0.2535	0.2496	0.2178	0.2524
11	Wdbc	0.7302	0.7251	0.7259	0.7242
12	Wine	0.8367	0.8992	0.8564	0.8449

and

Table 7. The performances of average $Acc$ value.

ID	Data Sets	K-Means	NJW	Voting	CSPA
1	Cardiotocography	0.3701	0.3384	0.5484	0.3653
2	Congressional voting	0.8610	0.8631	0.8671	0.8667
3	Dermatology	0.6996	0.8077	0.8508	0.8117
4	Forrest	0.7795	0.7540	0.7807	0.7130
5	Landsat	0.6682	0.6827	0.7429	0.6707
6	Optical recognition	0.7556	0.7716	0.8450	0.7654
7	Synthetic	0.6685	0.7475	0.8337	0.6697
8	Urban Land Cover	0.6119	0.5838	0.7333	0.5910
9	Vehicle	0.3691	0.3738	0.4243	0.3845
10	Waveform	0.5011	0.5049	0.5607	0.5013
11	Wdbc	0.9279	0.9262	0.9267	0.9262
12	Wine	0.9409	0.9663	0.9517	0.9292

.

From the experimental results in Table 3, Table 4, Table 5, Table 6 and Table 7, we can make the following conclusions.

(1)

It is obvious that the $NMI$ and $ARI$ performances of the TWCE strategy based on K-means and NJW are better than only using K-means and NJW. So, the TWCE strategy can indeed obtain a better clustering result than a single clustering algorithm. Compared to the other two ensemble clustering algorithms, the $NMI$ value and $ARI$ value obtained by the TWCE algorithm outperform on most of the 12 datasets. Specifically, the performances of TWCE based on K-means are always superior to the other algorithms on all datasets, and the performances of TWCE based on NJW are only slightly worse than the voting algorithms on Waveform. The improvement can be attributed to the fact that the clustering result of the proposed TWCE algorithm is represented by core regions when calculating the $NMI$ value and the $ARI$ value, which can increase the degree of separation between clusters and reduce the degree of dispersion within clusters.

(2)

TWCE based on K-means or NJW both have a better $Acc$ value on most datasets compared to other algorithms, except Waveform. The increase of $Acc$ value can be attributed to each cluster being represented only by a core region when we calculate the $Acc$ value, which means that the total number is to exclude elements in the fringe region, that is, $n_i$ and N both become smaller in Equation (3).

(3)

Comparing the performances of TWCE using K-means and TWCE using NJW, we can find that using different clustering algorithms to generate base clusters has little effect on the performances of the TWCE algorithm.

4.3. The Influences of the Selected Feature Percentage

The TWCE strategy uses a traditional clustering algorithm on parts of the features to obtain the base clustering results. The percentage of the selected feature subsets will have an impact on the performances of the experiment. In this subsection, we discuss the relation between the clustering performances and the percentage of the selected feature subsets. We also use the datasets in the previous subsection and the percentages of the feature subset are set to 50%, 60%, 70%, 80% and 90% of a dataset when the ensemble size is 50. The average $NMI$ value, $ARI$ value and $Acc$ value are achieved by running the TWCE algorithm 30 times, which uses the K-means algorithm and the NJW algorithm to generate base clusters on all datasets, respectively. Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 list the $NMI$ value, $ARI$ value and $Acc$ value of twelve datasets from the TWCE strategy based on K-means when the percentage of the feature subset takes different values. Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20, Figure 21, Figure 22, Figure 23, Figure 24 and Figure 25 show the relation between the clustering performances and the percentage of the feature subsets selected by the TWCE strategy based on the NJW algorithm.

From the experimental results recorded in Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20, Figure 21, Figure 22, Figure 23, Figure 24 and Figure 25, we can find that different datasets achieve the best performances at different percentages. For example, Dermatology achieves the best performances with the TWCE strategy based on K-means when the percentage is 60%, while most of the other datasets achieve the best performances with the TWCE strategy based on K-means when the percentage is 50%. Even for the same dataset, the TWCE strategy on different clustering algorithms achieves the best performances at different percentages. For example, Synthetic achieves the best performance with the TWCE strategy based on K-means when the percentage is 70%, while the best performance by the TWCE strategy based on NJW is obtained at a different percentage value. Though different datasets achieve the best performances at different percentages, the experimental performances of most of the datasets become worse when the percentage is 90%. This is because the diversity of the base clustering results will become smaller when the percentage becomes larger and low diversity limits the improvement of the ensemble performance. The issue of choosing a reasonable percentage needs to be further explored in future research.

5. Conclusions and Future Work

It has been recognized that a single clustering algorithm cannot identify all types of data structures. Ensemble clustering is an effective approach to solving the problem that a single clustering algorithm may not obtain good clustering results for all datasets. The three-way clustering method uses a core region and a fringe region to solve the problem of inaccurate decision-making caused by inaccurate information or insufficient data. Integrating the idea of three-way clustering and ensemble clustering methods, we propose a new ensemble three-way clustering strategy in this paper. In the proposed strategy, we randomly extract part of the features and use the traditional clustering algorithm to obtain one clustering result. A different feature subset will lead to different clustering results. Diverse base clustering results can be obtained by using different feature subsets. Based on the base clustering results, we use label matching to align all clustering results in a given order and the voting method to obtain the core region and the fringe region of the three way clustering. The sample is assigned to the core region of the corresponding cluster when the frequency of the sample in the same cluster is more than the given threshold. The difference between the union of the cluster with the same labels and the core region is regarded as the fringe region of the specific cluster. Therefore, a three-way clustering strategy is obtained. As examples for demonstration, we apply the proposed strategy on the top of K-means and spectral clustering, respectively. The experimental results on UCI datasets demonstrate the effectiveness in revealing data structures.

The following topics will deserve further investigation:

(1)

In this paper, the cluster number K is set to be constant during the process of generating base cluster members. Due to clustering, the algorithm is an unsupervised method, so how to apply the proposed algorithm to the different K is our next future work.

(2)

The base clusters generated by different feature subsets may be of low quality which may effect the final ensemble clustering result. We can evaluate the quality of base clusters by setting the evaluation function to remove some low-quality members of base clusters. This will be a good research direction.

(3)

In the process of three-way decision, the strategy needs to obtain more details on the division guidelines, so that the proposed algorithm can achieve a clustering result with a higher performance.