A Unified Formulation of k-Means, Fuzzy c-Means and Gaussian Mixture Model by the Kolmogorov–Nagumo Average

Abstract

Clustering is a major unsupervised learning algorithm and is widely applied in data mining and statistical data analyses. Typical examples include k-means, fuzzy c-means, and Gaussian mixture models, which are categorized into hard, soft, and model-based clusterings, respectively. We propose a new clustering, called Pareto clustering, based on the Kolmogorov–Nagumo average, which is defined by a survival function of the Pareto distribution. The proposed algorithm incorporates all the aforementioned clusterings plus maximum-entropy clustering. We introduce a probabilistic framework for the proposed method, in which the underlying distribution to give consistency is discussed. We build the minorize-maximization algorithm to estimate the parameters in Pareto clustering. We compare the performance with existing methods in simulation studies and in benchmark dataset analyses to demonstrate its highly practical utilities.

1. Introduction

In data analysis or data mining, there are two fundamental types of methodologies: clustering and classification [1]. Clustering, which is categorized as an exploratory paradigm, detects the underlying structure behind the data and grasps the rough image before proceeding to more intensive and comprehensive data analysis [2,3]. On the other hand, classification predicts unknown class labels of test data based on models constructed by training data with known class labels. The former is called supervised learning, while the latter is called unsupervised learning in pattern recognition [4].

Clustering algorithms fall roughly into three categories: hierarchical, partitioning, and mixture model-based algorithms [5]. In hierarchical clustering, each observation is considered as one cluster in the initial setting. Then clusters are merged recursively based on a similarity matrix defined beforehand. The resultant clusters are expressed as a dendrogram. The partition algorithm starts with a fixed number of clusters and searches for the cluster centers to minimize an objective function such as the squared distances between the centers and observations. It finds the centers simultaneously. The model-based algorithm assumes a mixture of probability distributions, which generates the observations and assigns the distributions to one of the mixture components. A Gaussian-mixture distribution-based approach is widely used in this context.

In this paper, we propose a new clustering, called Pareto clustering in the framework of quasilinear modeling [6,7,8]. It combines the cluster components by the Kolmogorov–Nagumo average [9] in a flexible way. We consider a generalized energy function as an objective function to estimate cluster parameters, which is an extension of the energy function proposed by [10]. The objective function consists of a survival function of the Pareto distribution, which is widely used in extreme value theory [11]. We investigate the consistency of the parameters, resulting in the underlying probability distribution of the generalized energy function. We find that k-means [12,13] and fuzzy c-means [14] have the underlying probability distributions with singular points at the cluster centers. This fact shows a clear difference from the model-based clustering such as a Gaussian-mixture modeling. Moreover, we show that the quasilinear modeling based on the Kolmogorov–Nagumo average connects k-means, fuzzy c-means, and a Gaussian-mixture modeling using the hyperparameters of the generalized energy function. See [15,16] for the discussion of the relation between k-means and fuzzy c-means.

The paper is organized as follows. In Section 2, we introduce the generalized energy function as the objective function of the Pareto clustering and discuss the consistency of the parameters. Moreover, we show that k-means, fuzzy c-means, and a Gaussian-mixture are all derived from the generalized energy function as special cases. This fact leads to the fact that the parameters can be estimated in a unified manner by the the minorize-maximization (MM) algorithm [17], where the monotone decrease of generalized energy function is guaranteed. In Section 3, we demonstrate the performance of the Pareto clustering based on simulation studies and benchmark datasets and show its practical utilities. We summarize the results of the Pareto clustering and discuss the extensions and applications in various scientific fields.

2. Materials and Methods

2.1. Generalized Energy Function

Let T be a non-negative random variable with a probability density function $f(t)$. The survival function of T is defined as

$$\begin{array}{c}\hfill S(t)=P(T>t),t\ge 0.\end{array}$$

(1)

Then for d-dimensional random variables $x_1,\dots ,x_n$, we define a generalized energy function to be minimized with respect to a parameter $\mu$ for clustering

$$\begin{array}{ccc}\hfill L_S(\mu )& =& \frac{1}{\tau }\sum _{i=1}^n{S}^{-1}\left(\frac{1}{K}\sum _{k=1}^{K}S(\tau \parallel x_i-{\mu }_k{\parallel }^2)\right),\hfill \end{array}$$

(2)

where $\mu =({\mu }_1,\dots ,{\mu }_K)$ is a set of centers and $\tau >0$ is the shape parameter. If we take $S(t)=exp(t)$, the function corresponds to the energy function proposed by [10], where $\tau$ can be interpreted as the temperature in physics. The formulation in (2) is called the Kolmogorov–Nagumo average [9,18] and is widely applied to bioinformatics, ecology, fisheries, etc. [6,8,19].

In Equation (2), we express an average of probabilities that $x_i$ belongs to the kth cluster over all K clusters using $1/K{\sum }_{k=1}^{K}S(\parallel x_i-{\mu }_k{\parallel }^2)$, where $\parallel x_i-{\mu }_k{\parallel }^2$ is the energy of $x_i$ associated with ${\mu }_k$. Hence we view $S^{-1}\left(1/K{\sum }_{k=1}^{K}S(\parallel x_i-{\mu }_k{\parallel }^2)\right)$ as the Kolmogorov–Nagumo average of the energy of $x_i$ with the probabilistic meanings. In effect, we take summation of the Kolmogorov-Nagumo average over the observations $\{x_1,...,x_n\}$.

Remark 1.

The generalized energy function (2) has a relation with the Archimedean copula defined by

$$\begin{array}{c}\hfill 1-S\left(\sum _{k=1}^K{S}^{-1}(1-u_k)\right)\end{array}$$

(3)

for ${\{u_k\}}_{k=1}^K$ in $(0,1)$, cf. [20] for an introductory discussion. In principle, the generalized energy function is a function from a vector of K cluster energy functions to a integrated energy function. The Archimedean copula is that of K marginal cumulative distribution functions to the joint cumulative distribution function. In this way, the generalized energy function expresses an interactive relation for cluster energy functions analogous with the Archimedean copula expressing the correlation among variables.

We consider an estimator of the generalized energy function as

$$\begin{array}{c}\hfill \widehat{\mu }=\underset{\mu }{\mathrm{argmin}}{L}_S(\mu ).\end{array}$$

(4)

If we assume that $x_i(i=1,\dots ,n)$ is distributed according to a probability density function $p(x,{\mu }^{*})$, the expected generalized energy function is given by

$$\begin{array}{c}\hfill {\mathbb{L}}_S(\mu )=\frac{1}{\tau }\int S^{-1}\left(\frac{1}{K}\sum _{k=1}^{K}S(\tau \parallel x-{\mu }_k{\parallel }^2)\right)p(x,{\mu }^{*})dx.\end{array}$$

(5)

Here we define a function for a set of cluster centers as

$$\begin{array}{c}\hfill {\mathcal{E}}_{\mu }(x)=\frac{1}{K}\sum _{k=1}^{K}S(\tau \parallel x-{\mu }_k{\parallel }^2).\end{array}$$

(6)

Thus, we note that

$$\begin{array}{c}\hfill \int {\mathcal{E}}_{\mu }(x)dx=v_d\mathbb{E}(T^{\frac{d}{2}}),\end{array}$$

(7)

where $v_d$ is a volume constant $2{\pi }^{d/2}/\{{\tau }^{1/2}d\Gamma (d/2)\}$ because $\mathbb{E}(T)={\int }_0^{\infty }S(t)dt$ (Appendix A). This property is a key idea in the following discussion.

Lemma 1.

Assume that the survival function $S(t)$ in (1) is convex in t. We define a function G of $({\mu }^{*},\mu )$ as

$$\begin{array}{c}\hfill G({\mu }^{*},\mu )=\int S^{-1}({\mathcal{E}}_{\mu }(x))f(S^{-1}({\mathcal{E}}_{{\mu }^{*}}(x)))dx,\end{array}$$

(8)

Then for any μ and ${\mu }^{*}$

$$\begin{array}{c}\hfill G({\mu }^{*},\mu )\ge G({\mu }^{*},{\mu }^{*})\end{array}$$

(9)

with equality if and only if $\mu ={\mu }^{*}$.

Proof. 

We observe that the function $S^{-1}(t)$ is a decreasing function of t given as

$$\begin{array}{c}\hfill \frac{\partial S^{-1}(t)}{\partial t}=-\frac{1}{f(S^{-1}(t))}\end{array}$$

(10)

because $S(S^{-1}(t))=t$ and $(\partial /\partial t)S(t)=-f(t)$. Similarly,

$$\begin{array}{c}\hfill \frac{{\partial }^2{S}^{-1}(t)}{\partial t^2}=-\frac{f^{\prime }(S^{-1}(t))}{{\{f(S^{-1}(t))\}}^3},\end{array}$$

(11)

which is positive for all $t\ge 0$ because $({\partial }^2/\partial t^2)S(t)=-f^{\prime }(t)>0$ from the convexity assumption for $S(t)$. Therefore, $S^{-1}(t)$ is also convex in $t\in (0,1)$. This leads to

$$\begin{array}{ccc}\hfill G({\mu }^{*},\mu )-G({\mu }^{*},{\mu }^{*})& =& \int \{S^{-1}({\mathcal{E}}_{\mu }(x))-S^{-1}({\mathcal{E}}_{{\mu }^{*}})\}f(S^{-1}({\mathcal{E}}_{{\mu }^{*}}(x)))dx\hfill \end{array}$$

(12)

$$\begin{array}{ccc}& \ge & \int \{{\mathcal{E}}_{\mu }(x)-{\mathcal{E}}_{{\mu }^{*}}(x)\}\frac{\partial S^{-1}({\mathcal{E}}_{{\mu }^{*}}(x))}{\partial t}f(S^{-1}({\mathcal{E}}_{{\mu }^{*}}(x))dx\hfill \end{array}$$

(13)

$$\begin{array}{ccc}& =& -\int \{{\mathcal{E}}_{\mu }(x)-{\mathcal{E}}_{{\mu }^{*}}(x)\}dx\hfill \end{array}$$

(14)

$$\begin{array}{ccc}& =& 0.\hfill \end{array}$$

(15)

Here Equality in (13) holds if and only if $\mu ={\mu }^{*}$ from the convexity for $S^{-1}$. The Equality (14) is shown by

$$\begin{array}{c}\hfill -f(S^{-1}(t))\frac{\partial S^{-1}(t)}{\partial t}=1\end{array}$$

(16)

for any $t\ge 0$ as seen in (10). Equality (15) holds due to (7).    □

Theorem 1.

If the $p(x,{\mu }^{*})$ has a form such as

$$\begin{array}{c}\hfill p(x,{\mu }^{*})=Z({\mu }^{*})f(S^{-1}({\mathcal{E}}_{{\mu }^{*}}(x))),\end{array}$$

(17)

where $Z({\mu }^{*})>0$ is a normalizing constant. Then we have

$$\begin{array}{c}\hfill {\mathbb{L}}_S(\mu )\ge {\mathbb{L}}_S({\mu }^{*}).\end{array}$$

(18)

Proof. 

Note that

$$\begin{array}{ccc}\hfill {\mathbb{L}}_S(\mu )-{\mathbb{L}}_S({\mu }^{*})& =& \frac{Z({\mu }^{*})}{\tau }\{G({\mu }^{*},\mu )-G({\mu }^{*},{\mu }^{*})\},\hfill \end{array}$$

which concludes (18) from Lemma 1.    □

We note that $\widehat{\mu }$ is asymptotically consistent to true parameter ${\mu }^{*}$ if the probability density function has the form in (17).

2.1.1. Pareto Distribution

Let us consider a generalized Pareto distribution, where the survival function and its inverse function are defined by

$$\begin{array}{c}\hfill S(t)={(1+\beta t)}^{-\frac{1}{\beta }}\mathrm{and}{S}^{-1}(t)=\frac{t^{-\beta }-1}{\beta },\end{array}$$

where $\beta >0$ denotes the shape parameter. Then the generalized energy function is

$$\begin{array}{ccc}\hfill L_{\tau ,\beta }(\mu )& =& \frac{1}{\tau \beta }\sum _{i=1}^n\left[{\left\{\sum _{k=1}^K\frac{1}{K}\{1+\tau \beta \parallel x_i-{\mu }_k{{\parallel }^2\}}^{-\frac{1}{\beta }}\right\}}^{-\beta }-1\right].\hfill \end{array}$$

(19)

If we consider $\beta \to 0$, then

$$\begin{array}{c}\hfill \underset{\beta \to 0}{lim}{L}_{\tau ,\beta }(\mu )=-\frac{1}{\tau }\sum _{i=1}^{n}log\left\{\sum _{k=1}^K\frac{1}{K}exp\left(-\tau \parallel x_i-{\mu }_k{\parallel }^2\right)\right\},\end{array}$$

(20)

which is reduced to the energy function proposed by [10]. Hence, we can understand that Rose’s clustering (maximum-entropy clustering) is generated by a survival distribution function of an exponential distribution. Then we have

$$\begin{array}{cc}\hfill \underset{\tau \to \infty }{lim}{L}_{\tau ,\beta }(\mu )=& \underset{\tau \to \infty }{lim}\frac{1}{\tau \beta }\sum _{i=1}^n{\left\{\sum _{k=1}^K\frac{1}{K}\{1+\tau \beta \parallel x_i-{\mu }_k{{\parallel }^2\}}^{-\frac{1}{\beta }}\right\}}^{-\beta }\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill =& \sum _{i=1}^n{\left\{\sum _{k=1}^K\frac{1}{K}\{\parallel x_i-{\mu }_k{{\parallel }^2\}}^{-\frac{1}{\beta }}\right\}}^{-\beta }\hfill \end{array}$$

(22)

The gradient with respect to ${\mu }_k$ is given by

$$\begin{array}{c}\hfill \frac{2}{\beta }\sum _{i=1}^n\frac{1}{K}{\left[\frac{\{\parallel x_i-{\mu }_k{{\parallel }^2\}}^{-\frac{1}{\beta }}}{{\sum }_{\ell =1}^K{\pi }_{\ell }\{\parallel x_i-{\mu }_{\ell }{{\parallel }^2\}}^{-\frac{1}{\beta }}}\right]}^{1+\beta }(x_i-{\mu }_k),\end{array}$$

(23)

which exactly leads to the estimation equations of fuzzy c-means if we take $\beta =m-1$ [14]. Furthermore, we have

$$\begin{array}{ccc}\hfill \underset{\tau \to \infty ,\beta \to 0}{lim}{L}_{\tau ,\beta }(\mu )& =& \sum _{i=1}^n\underset{1\le k\le K}{min}{\parallel x_i-{\mu }_k\parallel }^2,\hfill \end{array}$$

(24)

which is the loss function of k-means. The corresponding survival function is ${lim}_{\beta \to 0}{(1+\beta t)}^{-\frac{1}{\beta }}=exp(-t)$. Note that the loss function is directly derived from (2) as

$$\underset{\tau \to \infty }{lim}\frac{1}{\tau }\sum _{i=1}^n{S}^{-1}\left(\frac{1}{K}\sum _{k=1}^{K}S(\tau \parallel x_i-{\mu }_k{\parallel }^2)\right)=\sum _{i=1}^n\underset{1\le k\le K}{min}{\parallel x_i-{\mu }_k\parallel }^2.$$

(25)

In addition, we have

$$\underset{\tau \to 0}{lim}\frac{1}{\tau }\sum _{i=1}^n{S}^{-1}\left(\frac{1}{K}\sum _{k=1}^{K}S(\tau \parallel x_i-{\mu }_k{\parallel }^2)\right)=\sum _{i=1}^n\frac{1}{K}\sum _{k=1}^K{\parallel x_i-{\mu }_k\parallel }^2$$

(26)

because $S(0)=1$.

2.1.2. Fréchet Distribution

Next, we consider Fréchet distribution with the survival function defined as

$$\begin{array}{c}\hfill S(t)=1-exp(-t^{\gamma }).\end{array}$$

(27)

where $\gamma <0$ is the shape parameter. The generalized energy function is given by

$$\begin{array}{ccc}\hfill L_{\gamma ,\tau }(\mu )& & =\sum _{i=1}^n{\left(-\frac{1}{{\tau }^{\gamma }}log\left[\frac{1}{K}\sum _{k=1}^{K}exp(-{\tau }^{\gamma }\parallel x_i-{\mu }_k{\parallel }^{2\gamma })\right]\right)}^{\frac{1}{\gamma }}.\hfill \end{array}$$

(28)

We find that

$$\begin{array}{ccc}\hfill \underset{\tau \to 0}{lim}{L}_{\gamma ,\tau }(\mu )& =& \sum _{i=1}^n\underset{\tau \to 0}{lim}{\left(-\frac{1}{{\tau }^{\gamma }}log\left[\frac{1}{K}\sum _{k=1}^{K}exp(-{\tau }^{\gamma }\parallel x_i-{\mu }_k{\parallel }^{2\gamma })\right]\right)}^{\frac{1}{\gamma }}\hfill \\ & =& \sum _{i=1}^n{\left(\underset{1\le k\le K}{min}{\parallel x_i-{\mu }_k\parallel }^{2\gamma }\right)}^{\frac{1}{\gamma }}\hfill \\ & =& \sum _{i=1}^n\underset{1\le k\le K}{min}{\parallel x_i-{\mu }_k\parallel }^2.\hfill \end{array}$$

(29)

Hence, this energy function is reduced to that of the K-means algorithm as shown in the Pareto distribution case. The estimating equation is given by

$$\begin{array}{ccc}\hfill \frac{\partial }{\partial {\mu }_k}{L}_{\gamma ,\tau }(\mu )& =& \sum _{i=1}^n{\omega }_k(x_i,\tau ,\gamma )({\mu }_k-x_i)=0,\hfill \end{array}$$

(30)

where

$$\begin{array}{ccc}\hfill {\omega }_k(x_i,\tau ,\gamma )& =& {\left(-\frac{1}{{\tau }^{\gamma }}log\left[\frac{1}{K}\sum _{\ell =1}^{K}exp(-{\tau }^{\gamma }\parallel x_i-{\mu }_{\ell }{\parallel }^{2\gamma })\right]\right)}^{\frac{1}{\gamma }-1}\hfill \\ & & \times \frac{exp(-{\tau }^{\gamma }\parallel x_i-{\mu }_k{\parallel }^{2\gamma })}{{\sum }_{\ell =1}^{K}exp(-{\tau }^{\gamma }\parallel x_i-{\mu }_{\ell }{\parallel }^{2\gamma })}{\parallel x_i-{\mu }_k\parallel }^{2\gamma -2}.\hfill \end{array}$$

(31)

When we assume the unbiasedness for the estimating function in (30), that is

$$\begin{array}{c}\hfill \mathbb{E}\{{\omega }_k(X,\tau ,\gamma )({\mu }_k-X)\}=0,\end{array}$$

(32)

the underlying distribution has a density function proportional to

$$\begin{array}{c}\hfill {\left(-\frac{1}{{\tau }^{\gamma }}logM(x,\mu )\right)}^{\frac{\gamma }{1-\gamma }}M(x,\mu ).\end{array}$$

(33)

where

$$\begin{array}{c}\hfill M(x,\mu )=\frac{1}{K}\sum _{\ell =1}^{K}exp(-{\tau }^{\gamma }\parallel x-{\mu }_{\ell }{\parallel }^{2\gamma }).\end{array}$$

(34)

We confirm that

$$\begin{array}{c}\hfill {\omega }_k(x_i,\tau ,\gamma )=\left\{\begin{array}{cc}1\hfill & \mathrm{if} \parallel x_i-{\mu }_k{\parallel }^2=\underset{1\le \ell \le K}{min}{\parallel x_i-{\mu }_{\ell }\parallel }^2\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(35)

as $\tau$ goes to 0. Then we consider the limit of $\tau$ to ∞, which provides

$$\begin{array}{c}\underset{\tau \to \infty }{lim}{L}_{\gamma ,\tau }(\mu )=\sum _{i=1}^n\underset{\tau \to \infty }{lim}{\left(-\frac{1}{{\tau }^{\gamma }}log\left[\frac{1}{K}\sum _{k=1}^{K}exp(-{\tau }^{\gamma }\parallel x_i-{\mu }_k{\parallel }^{2\gamma })\right]\right)}^{\frac{1}{\gamma }}\hfill \\ =\sum _{i=1}^n{\left(\frac{1}{K}\sum _{k=1}^K{\parallel x_i-{\mu }_k\parallel }^{2\gamma }\right)}^{\frac{1}{\gamma }}\hfill \end{array}$$

(36)

which is equal to (22). This also leads to Fuzzy c-means if we take as $\gamma =1/(1-m)$ [14].

2.2. Estimation of Variances and Mixing Proportions in Clusters

In stead of the Euclidean distance $\parallel x_i-{\mu }_k{\parallel }^2$, we consider $\parallel x_i-{\mu }_k{\parallel }_{{\sum }_k^{-1}}^2={(x_i-{\mu }_k)}^{\top }{\sum }_k^{-1}(x_i-{\mu }_k)$ to incorporate the variance structure around ${\mu }_k$. Bezdek et al. [14] considered a common variance structure ${\sum }_k={\sum }_{i=1}^n(x_i-\overline{x}){(x_i-\overline{x})}^{\top }$, where $\overline{x}=1/n{\sum }_{i=1}^n{x}_i$ for $k=1,\dots ,K$. On the other hand, we estimate distinct ${\sum }_k$ for each ${\mu }_k$.

For this purpose, we modify the generalized energy function in (2) to allow for a variances ${\sum }_1,\dots ,{\sum }_K$ and mixing proportions ${\pi }_1,\dots ,{\pi }_K$ (${\sum }_{k=1}^K{\pi }_k=1$ and ${\pi }_k\ge 0$ for $k=1,\dots ,K$) as

$$\begin{array}{ccc}\hfill L_S(\theta )& =& \frac{1}{\tau }\sum _{i=1}^n{S}^{-1}\left(\sum _{k=1}^K{\pi }_k|{\sum }_k{|}^{-\frac{1}{2}}S(\tau \parallel x_i-{\mu }_k{\parallel }_{{\sum }_k^{-1}}^2)\right),\hfill \end{array}$$

(37)

where $\theta ={({\mu }_k,{\sum }_k,{\pi }_k)}_{k=1}^K$. We assume that $S(t)$ is convex so that the domain of $S^{-1}(t)$ can be extended from $[0,1]$ to $[0,\infty )$ to allow for $|{\sum }_k{|}^{-\frac{1}{2}}$. The estimator of this modified generalized energy function is given as

$$\begin{array}{c}\hfill \widehat{\theta }=\underset{\theta }{\mathrm{argmin}}{L}_S(\theta ),\end{array}$$

(38)

The expected generalized energy function is given by

$$\begin{array}{c}\hfill {\mathbb{L}}_S(\theta )=\frac{1}{\tau }\int S^{-1}\left(\sum _{k=1}^K{\pi }_k|{\sum }_k{|}^{-\frac{1}{2}}S(\tau \parallel x-{\mu }_k{\parallel }_{{\sum }_k^{-1}}^2)\right)p(x,{\theta }^{*})dx,\end{array}$$

(39)

where $p(x,{\theta }^{*})$ is the underlying probability density function.

For a cumulative distribution function $F(t)=1-S(t)$ we have $L_S(\theta )=L_F(\theta )$ if and only if ${\sum }_{k=1}^K{\pi }_k{\left|{\sum }_k\right|}^{-1/2}=1$. On the other hand, it always holds that $L_S(\mu )=L_F(\mu )$ for the original generalized energy function in (2).

Similarly to (6), we define

$$\begin{array}{c}\hfill {\mathcal{E}}_{\theta }(x)=\sum _{k=1}^K{\pi }_k|{\sum }_k{|}^{-\frac{1}{2}}S(\tau \parallel x-{\mu }_k{\parallel }_{{\sum }_k^{-1}}^2),\end{array}$$

(40)

and we notice that $\int {\mathcal{E}}_{\theta }(x)dx$ is also independent of $\mu$ as

$$\int {\mathcal{E}}_{\theta }(x)dx=v_d\mathbb{E}(T^{\frac{d}{2}}).$$

(41)

Lemma 2.

Assume that the survival function $S(t)$ in (1) is convex in t. We define a function G of $({\mu }^{*},\mu )$ as

$$\begin{array}{c}\hfill G({\theta }^{*},\theta )=\int S^{-1}({\mathcal{E}}_{\theta }(x))f(S^{-1}({\mathcal{E}}_{{\theta }^{*}}(x)))dx.\end{array}$$

(42)

Then for any θ and ${\theta }^{*}$

$$\begin{array}{c}\hfill G({\theta }^{*},\theta )\ge G({\theta }^{*},{\theta }^{*})\end{array}$$

(43)

with equality if and only if $\theta ={\theta }^{*}$.

Proof. 

It is obvious from Lemma 1 and the fact that $\int {\mathcal{E}}_{\theta }(x)dx$ is independent of $\mu$.    □

From Lemma 2, we can easily show the following theorem regarding ${\mathbb{L}}_S(\theta )$.

Theorem 2.

If the $p(x,{\theta }^{*})$ has a form such as

$$\begin{array}{c}\hfill p(x,{\theta }^{*})=Z({\theta }^{*})f(S^{-1}({\mathcal{E}}_{{\theta }^{*}}(x))),\end{array}$$

(44)

where $Z({\theta }^{*})>0$ is a normalizing constant. Then we have

$$\begin{array}{c}\hfill {\mathbb{L}}_S(\theta )\ge {\mathbb{L}}_S({\theta }^{*}).\end{array}$$

(45)

For the Pareto distribution, we have from (37)

$$\begin{array}{ccc}\hfill L_{\tau ,\beta }(\theta )& =& \frac{1}{\tau \beta }\sum _{i=1}^n\left[{\left\{\sum _{k=1}^K{\pi }_k|{\sum }_k{|}^{-\frac{1}{2}}\{1+\tau \beta \parallel x_i-{\mu }_k{{\parallel }_{{\sum }_k^{-1}}^2\}}^{-\frac{1}{\beta }}\right\}}^{-\beta }-1\right]\hfill \end{array}$$

(46)

$$\begin{array}{ccc}& =& \frac{1}{\tau }\sum _{i=1}^n\varphi \left(\sum _{k=1}^K{\pi }_{k}w(x_i,{\mu }_k,{\sum }_k)\right),\hfill \end{array}$$

(47)

where

$$\begin{array}{ccc}\hfill w(x_i,{\mu }_k,{\sum }_k)& =& |{\sum }_k{|}^{-\frac{1}{2}}\{1+\tau \beta \parallel x_i-{\mu }_k{{\parallel }_{{\sum }_k^{-1}}^2\}}^{-\frac{1}{\beta }}\hfill \end{array}$$

(48)

$$\begin{array}{ccc}\hfill \varphi (t)& =& \frac{t^{-\beta }-1}{\beta }.\hfill \end{array}$$

(49)

From (44), the underlying probability density function is

$$p_{\tau ,\beta }({\theta }^{*})=Z_{\tau ,\beta }({\theta }^{*}){\left\{\sum _{k=1}^K{\pi }_k^{*}w(x_i,{\mu }_k^{*},{\sum }_k^{*})\right\}}^{1+\beta },$$

(50)

where $Z_{\tau ,\beta }({\theta }^{*})$ is a normalizing constant. When $\beta \to 0$, we have

$$\underset{\beta \to 0}{lim}{L}_{\tau ,\beta }(\theta )=-\frac{1}{\tau }\sum _{i=1}^{n}log\left\{\sum _{k=1}^K{\pi }_k|{\sum }_k{|}^{-\frac{1}{2}}exp(-\tau \parallel x_i-{\mu }_k{\parallel }_{{\sum }_k^{-1}}^2)\right\},$$

(51)

which is the negative log likelihood function of the normal mixture distributions apart from a constant term ${(2\pi )}^{-d/2}$ when $\tau =1/2$.

Similarly, we have the estimation equation of fuzzy c-means allowing for the Mahalanobis distance when $\tau \to \infty$. Moreover, we have k-means with the use of the Mahalanobis distance when $\tau \to \infty$ and $\beta \to 0$. For the other extreme cases, we observe that both ${lim}_{\beta \to \infty }{L}_{\tau ,\beta }(\theta )$ and ${lim}_{\tau \to 0}{L}_{\tau ,\beta }(\theta )$ diverge or converge to 0 depending on the values of ${\pi }_k$ and ${\sum }_k$ ($k=1,\dots ,K$). Hence we choose large values for $\tau$ and small values for $\beta$ in the subsequent data analysis.

2.3. Estimating Algorithm

The direct optimization of $L_{\tau ,\beta }(\theta )$ in (46) is difficult due to the mixture structure. Thus, we employ the idea of expectation and maximization (EM) algorithm [21] and the minorize-maximization (MM) algorithm [17] similar to [19]. Our proposed clustering method (Pareto clustering) is as follows in Algorithm 1.

Algorithm 1: Pareto clustering

1. Set initial values $({\mu }_k^{(0)},{\sum }_k^{(0)},{\pi }_k^{(0)})$ for $k=1,\dots ,K$. 2. Repeat the following steps for $t=0,\dots ,T-1$ and $k=1,\dots ,K$ until convergence. 3. $$\begin{array}{c}{q}_k^{(t)}(x_i)=\frac{{\pi }_k^{(t)}w(x_i,{\mu }_k^{(t)},{\sum }_k^{(t)})}{{\sum }_{\ell =1}^K{\pi }_k^{(t)}w(x_i,{\mu }_k^{(t)},{\sum }_k^{(t)})}\hfill \end{array}$$ (52) $$\begin{array}{c}{\mu }_k^{(t+1)}=\frac{{\sum }_{i=1}^n{\left\{q_k^{(t)}(x_i)\right\}}^{1+\beta }{x}_i}{{\sum }_{i=1}^n{\left\{q_k^{(t)}(x_i)\right\}}^{1+\beta }}\hfill \end{array}$$ (53) $${\sum }_k^{(t+1)}=\frac{\tau (2-d\beta ){\sum }_{i=1}^n{\left\{q_k^{(t)}(x_i)\right\}}^{1+\beta }(x_i-{\mu }_k^{(t+1)}){(x_i-{\mu }_k^{(t+1)})}^{\top }}{{\sum }_{i=1}^n{\left\{q_k^{(t)}(x_i)\right\}}^{1+\beta }}$$ (54) $${\pi }_k^{(t+1)}=\frac{{\left\{{\sum }_{i=1}^n{\left\{q_k^{(t)}(x_i)\right\}}^{1+\beta }{\left\{w(x_i,{\mu }_k^{(t+1)},{\sum }_k^{(t+1)})\right\}}^{-\beta }\right\}}^{\frac{1}{1+\beta }}}{{\sum }_{\ell =1}^K{\left\{{\sum }_{i=1}^n{\left\{q_{\ell }^{(t)}(x_i)\right\}}^{1+\beta }{\left\{w(x_i,{\mu }_{\ell }^{(t+1)},{\sum }_{\ell }^{(t+1)})\right\}}^{-\beta }\right\}}^{\frac{1}{1+\beta }}}$$ (55) 4. Output $({\widehat{\mu }}_k,{\widehat{\sum }}_k,{\widehat{\pi }}_k)=({\mu }_k^{(T)},{\sum }_k^{(T)},{\pi }_k^{(T)})$ for $k=1,\dots ,K$.

The initial values $({\mu }_k^{(0)},{\sum }_k^{(0)},{\pi }_k^{(0)})$ are determined by the hierarchical clustering in a similar way to the algorithm by [22]. The derivation of the estimating algorithm is as follows. First, we have

$$\begin{array}{ccc}\hfill L_{\tau ,\beta }(\theta )& =& \frac{1}{\tau }\sum _{i=1}^n\varphi \left(\sum _{k=1}^K{\pi }_{k}w(x_i,{\mu }_k,{\sum }_k)\right)\hfill \end{array}$$

(56)

$$\begin{array}{ccc}& =& \frac{1}{\tau }\sum _{i=1}^n\varphi \left(\sum _{k=1}^K{q}_k(x_i)\frac{{\pi }_{k}w(x_i,{\mu }_k,{\sum }_k)}{q_k(x_i)}\right)\hfill \end{array}$$

(57)

$$\begin{array}{ccc}& \le & \frac{1}{\tau }\sum _{i=1}^n\sum _{k=1}^K{q}_k(x_i)\varphi \left(\frac{{\pi }_{k}w(x_i,{\mu }_k,{\sum }_k)}{q_k(x_i)}\right)\hfill \end{array}$$

(58)

where $q_k(x_i)$ is a positive weight such as ${\sum }_{k=1}^K{q}_k(x_i)=1$ and $\varphi (t)$ is the convex function defined in (49). The equality holds if and only if

$$\begin{array}{c}\hfill \frac{{\pi }_{1}w(x_i,{\mu }_1,{\sum }_1)}{q_1(x_i)}=\cdots =\frac{{\pi }_{K}w(x_i,{\mu }_K,{\sum }_K)}{q_K(x_i)},\end{array}$$

(59)

which is equivalent to

$$\begin{array}{c}\hfill q_k(x_i)=\frac{{\pi }_{k}w(x_i,{\mu }_k,{\sum }_k)}{{\sum }_{k=1}^K{\pi }_{k}w(x_i,{\mu }_k,{\sum }_k)},(k=1,\dots ,K).\end{array}$$

(60)

Based on $q_k^{(t)}(x_i)$ in (52), we define

$$\begin{array}{cc}\hfill Q(\theta |{\theta }^{(t)})=& \frac{1}{\tau }\sum _{i=1}^n\sum _{k=1}^K{q}_k^{(t)}(x_i)\left\{{\left(\frac{{\pi }_{k}w(x_i,{\mu }_k,{\sum }_k)}{q_k^{(t)}(x_i)}\right)}^{-\beta }-1\right\}/\beta \hfill \end{array}$$

(61)

$$\begin{array}{cc}\hfill =& \frac{1}{\tau \beta }\sum _{i=1}^n\sum _{k=1}^K\left[{\{q_k^{(t)}(x_i)\}}^{1+\beta }{\pi }_k^{-\beta }{\left|{\sum }_k\right|}^{\frac{\beta }{2}}\{1+\tau \beta {(x_i-{\mu }_k)}^{\top }{\sum }_k^{-1}(x_i-{\mu }_k)\}-q_k^{(t)}(x_i)\right]\hfill \end{array}$$

(62)

$$\begin{array}{cc}\hfill =& \frac{1}{\tau \beta }\sum _{i=1}^n\sum _{k=1}^K\left[{\{q_k^{(t)}(x_i)\}}^{1+\beta }{\pi }_k^{-\beta }\{|V_k^{-1}{|}^{\frac{\beta }{d\beta -2}}+\tau \beta {(x_i-{\mu }_k)}^{\top }{V}_k^{-1}(x_i-{\mu }_k)\}-q_k^{(t)}(x_i)\right],\hfill \end{array}$$

(63)

where $V_k^{-1}={\left|{\sum }_k\right|}^{\beta /2}{\sum }_k^{-1}$. Then we have

$$\begin{array}{ccc}\hfill \frac{\partial }{\partial {\mu }_k}Q(\theta |{\theta }^{(t)})& =& -2{\pi }_k^{-\beta }{V}_k^{-1}\sum _{i=1}^n{\{q_k^{(t)}(x_i)\}}^{1+\beta }(x_i-{\mu }_k)\hfill \end{array}$$

(64)

$$\begin{array}{ccc}& =& 0\hfill \end{array}$$

(65)

which means that

$$\begin{array}{c}\hfill {\mu }_k^{(t+1)}=\frac{{\sum }_{i=1}^n{\{q_k^{(t)}(x_i)\}}^{1+\beta }{x}_i}{{\sum }_{i=1}^n{\{q_k^{(t)}(x_i)\}}^{1+\beta }}.\end{array}$$

(66)

Similarly, we have

$$\begin{array}{cc}\hfill \frac{\partial }{\partial V_k^{-1}}Q(\theta |{\theta }^{(t)}){|}_{{\mu }_k={\mu }_k^{(t+1)}}& =\frac{1}{\tau \beta }\sum _{i=1}^n{\{q_k^{(t)}(x_i)\}}^{1+\beta }{\pi }_k^{-\beta }\left\{\frac{\beta }{d\beta -2}{\left|V_k^{-1}\right|}^{\frac{\beta }{d\beta -2}}{V}_k+\tau \beta (x_i-{\mu }_k^{(t+1)}){(x_i-{\mu }_k^{(t+1)})}^{\top }\right\}\hfill \end{array}$$

(67)

$$\begin{array}{cc}\hfill & =\frac{1}{\tau \beta }\sum _{i=1}^n{\{q_k^{(t)}(x_i)\}}^{1+\beta }{\pi }_k^{-\beta }\left\{\frac{\beta }{d\beta -2}{\sum }_k+\tau \beta (x_i-{\mu }_k^{(t+1)}){(x_i-{\mu }_k^{(t+1)})}^{\top }\right\}\hfill \end{array}$$

(68)

$$\begin{array}{cc}\hfill & =0,\hfill \end{array}$$

(69)

which means that

$$\begin{array}{c}\hfill {\sum }_k^{(t+1)}=\frac{\tau (2-d\beta ){\sum }_{i=1}^n{\{q_k^{(t)}(x_i)\}}^{1+\beta }(x_i-{\mu }_k^{(t+1)}){(x_i-{\mu }_k^{(t+1)})}^{\top }}{{\sum }_{i=1}^n{\{q_k^{(t)}(x_i)\}}^{1+\beta }}.\end{array}$$

(70)

Next we consider

$$\begin{array}{c}\hfill R(\theta |{\theta }^{(t)})=Q(\theta |{\theta }^{(t)})+\lambda \left(\sum _{k=1}^K{\pi }_k-1\right),\end{array}$$

(71)

where $\lambda$ is a Lagrange multiplier. Then

$$\begin{array}{ccc}\hfill \frac{\partial }{\partial {\pi }_k}R(\theta |{\theta }^{(t)}){|}_{{\mu }_k={\mu }_k^{(t+1)},{\sum }_k={\sum }_k^{(t+1)}}& =& -\frac{1}{\tau }{\pi }_k^{-\beta -1}\sum _{i=1}^n{\{q_k^{(t)}(x_i)\}}^{1+\beta }w{(x_i,{\mu }_k^{(t+1)},{\sum }_k^{(t+1)})}^{-\beta }+\lambda \hfill \end{array}$$

(72)

$$\begin{array}{ccc}& =& 0,\hfill \end{array}$$

(73)

which means

$$\begin{array}{ccc}\hfill {\pi }_k^{(t+1)}& =& \frac{{\{{\sum }_{i=1}^n{\{q_k^{(t)}(x_i)\}}^{1+\beta }w{(x_i,{\mu }_k^{(t+1)},{\sum }_k^{(t+1)})}^{-\beta }\}}^{\frac{1}{1+\beta }}}{{\sum }_{\ell =1}^K{\{{\sum }_{i=1}^n{\{q_{\ell }^{(t)}(x_i)\}}^{1+\beta }w{(x_i,{\mu }_{\ell }^{(t+1)},{\sum }_{\ell }^{(t+1)})}^{-\beta }\}}^{\frac{1}{1+\beta }}}.\hfill \end{array}$$

(74)

Remark 2.

The generalized energy function in (46) is monotonically decreasing in the estimating algorithm. That is, we have

$$L_{\tau ,\beta }({\theta }^{(t+1)})\le Q({\theta }^{(t+1)}|{\theta }^{(t)})\le Q({\theta }^{(t)}|{\theta }^{(t)})=L_{\tau ,\beta }({\theta }^{(t)}).$$

(75)

See Appendix B for more details.

Remark 3.

The estimating algorithm of fuzzy c-means by [14] is given as

$$\begin{array}{ccc}\hfill u_{ik}^{(t)}& =& {\left[\sum _{\ell =1}^K{\left(\frac{\parallel x_i-{\mu }_k^{(t)}{\parallel }^2}{\parallel x_i-{\mu }_{\ell }^{(t)}{\parallel }^2}\right)}^{\frac{1}{m-1}}\right]}^{-m},\hfill \end{array}$$

(76)

$$\begin{array}{ccc}\hfill {\mu }_k^{(t)}& =& \frac{{\sum }_{i=1}^n{u}_{ik}^{(t)}{x}_i}{{\sum }_{i=1}^n{u}_{ik}^{(t)}},\hfill \end{array}$$

(77)

where ${\{u_{ik}^{(t)}\}}^{1/m}$ is called the membership function of $x_i$ in cluster k at the iteration step t. These are special cases of (52) and (53) with $\tau \to \infty$, ${\sum }_k=I$, ${\pi }_k=1/K$ and $\beta =m-1$. Hence we observe that the original algorithm of fuzzy c-means can be interpreted as the EM algorithm.

Remark 4.

In analogy with the membership function of fuzzy c-means by [14], we define $q_k^{(t)}(x_i)$ in (52) as a membership function of $x_i$ in cluster k at the iteration step t in Pareto clustering. Hence we estimate cluster $C_k$ as

$$C_k=\{x_i|q_k^{(T)}(x_i)\ge q_{\ell }^{(T)}(x_i),\ell =1,\dots ,K,i=1,\dots ,n\},$$

(78)

where ${\cup }_{k=1}^K{C}_k=\{x_1,\dots ,x_n\}$.

Remark 5.

In high-dimensional setting $p\gg 1$, we consider the ridge regularization for ${\sum }_k^{(t+1)}$ as in [23]

$${\sum }_k^{(t+1)}(\alpha )=\alpha {\sum }_k^{(t+1)}+(1-\alpha ){\widehat{\sigma }}_k^{2}I,$$

where $\alpha =0.95$ and ${\sigma }_k^2$ is the scalar variance estimated to be the maximum value of the diagonal elements of ${\sum }_k^{(t+1)}$. Moreover, we take $\beta \ll 1$ to make ${\sum }_k^{(t+1)}$ positive definite.

2.4. Evaluation of Clustering Methods

We compare the performances of k-means, fuzzy c-means, Gaussian mixture modeling (Gaussian), partitioning around medoids (PAM), and Pareto clustering. To implement these methods, we use the kmeans function in the stat package [24], the cmeans function in the e1071 package [14], the Mclust in the mclust package [22] and the pam function in the cluster package [25] in the statistical software R, where the default settings are used for each function. In Pareto clustering, $\tau =0.5$ and $\beta =1$ are used as the default settings. We assume that the number of clusters K is known and compare the performances as in [26].

2.4.1. Metrics

Cluster $C_k$ ($k=1,\dots ,K$) estimated by a clustering method is evaluated by a predefined reference class set $D_{\ell }$ ($\ell =1,\dots ,L$) such as

$$\begin{array}{ccc}\hfill \mathrm{Precision}(C_k,D_{\ell })& =& \frac{|C_k\cap D_{\ell }|}{|C_k|}\hfill \\ \hfill \mathrm{Recall}(C_k,D_{\ell })& =& \frac{|C_k\cap D_{\ell }|}{|D_{\ell }|},\hfill \end{array}$$

where $\mathrm{Recall}(C_k,D_{\ell })=\mathrm{Precision}(D_{\ell },C_k)$. $\mathrm{Precision}(C_k,D_{\ell })$ counts data points in cluster $C_k$ belonging to class ℓ. Hence ${max}_{\ell }\mathrm{Precision}(C_k,D_{\ell })$ represents the purity of cluster $C_k$ regarding the classes. By taking the weighted average, we have

$$\mathrm{Purity}=\sum _{k=1}^K\frac{|C_k|}{n}\underset{\ell }{max}\mathrm{Precision}(C_k,D_{\ell }),$$

where n is the sample size. $\mathrm{Recall}(C_k,D_{\ell })$ counts data points in a class set $D_{\ell }$ estimated to be in cluster $C_k$. Precision and recall correspond to the positive predictive value and sensitivity, respectively [27].

A metric combining precision and recall is proposed by [28] such as

$$\text{F-value}=\sum _{k=1}^K\frac{|D_k|}{n}\underset{\ell }{max}F(D_k,C_{\ell }),$$

where

$$F(D_k,C_{\ell })=\frac{2\times \mathrm{Recall}(D_k,C_{\ell })\times \mathrm{Precision}(D_k,C_{\ell })}{\mathrm{Recall}(D_k,C_{\ell })+\mathrm{Precision}(D_k,C_{\ell })},$$

which is the harmonic mean of $\mathrm{Precision}(D_k,C_{\ell })$ and $\mathrm{Recall}(D_k,C_{\ell })$, and is called the F-measure [29].

The cluster level similarity between the estimated center $\widehat{\mu }=({\widehat{\mu }}_1,\dots ,{\widehat{\mu }}_K)$ and the reference value (ground truth) of center ${\mu }^{*}=({\mu }_1^{*},\dots ,{\mu }_K^{*})$ is the centroid index (CI) proposed by [26] as

$$\begin{array}{c}\hfill \mathrm{CI}(\widehat{\mu },{\mu }^{*})=max({\mathrm{CI}}^{\prime }(\widehat{\mu },{\mu }^{*}),{\mathrm{CI}}^{\prime }({\mu }^{*},\widehat{\mu })),\end{array}$$

where

$$\begin{array}{ccc}\hfill {\mathrm{CI}}^{\prime }(\widehat{\mu },{\mu }^{*})& =& \sum _{k=1}^K\mathrm{orphan}({\mu }_k^{*})\hfill \\ \hfill \mathrm{orphan}({\mu }_k^{*})& =& \left\{\begin{array}{cc}1\hfill & q_{\ell }\ne k\forall \ell \hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\hfill \\ \hfill q_{\ell }& =& \underset{1\le k\le K}{\mathrm{argmin}}{\parallel {\widehat{\mu }}_{\ell }-{\mu }_k^{*}\parallel }^2.\hfill \end{array}$$

Here, $q_{\ell }$ indicates the index of the element of the reference center ${\mu }^{*}$ that is the nearest to ${\widehat{\mu }}_{\ell }$. The function $\mathrm{orphan}({\mu }_k^{*})$ indicates whether ${\mu }_k^{*}$ is an isolated element (orphan) or not, which is $not$ nearest to any elements of $\widehat{\mu }$. Hence ${\mathrm{CI}}^{\prime }(\widehat{\mu },{\mu }^{*})$ indicates the dissimilarity between $\widehat{\mu }$ and ${\mu }^{*}$. Due to the asymmetry of ${\mathrm{CI}}^{\prime }(\widehat{\mu },{\mu }^{*})$ with respect to $\widehat{\mu }$ and ${\mu }^{*}$, we take the maximum of ${\mathrm{CI}}^{\prime }(\widehat{\mu },{\mu }^{*})$ and ${\mathrm{CI}}^{\prime }({\mu }^{*},\widehat{\mu })$. Hence $\mathrm{CI}(\widehat{\mu },{\mu }^{*})$ measures how many clusters are differently located among $\widehat{\mu }$ and ${\mu }^{*}$.

Another metric to measure the similarity between $\widehat{\mu }$ and ${\mu }^{*}$ is defined as the mean squared error (MSE) over the number of clusters K, which is given as

$$\mathrm{MSE}=\frac{1}{K}\sum _{k=1}^K\left|\right|{\widehat{\mu }}_k-{\mu }_k^{*}{\left|\right|}^2.$$

Differently from Purity and F-value, MSE can be calculated based on only estimated and reference centers $\widehat{\mu }$ and ${\mu }^{*}$. This property is useful in a situation where the reference class sets $D_1,\dots ,D_K$ are difficult to determine but ${\mu }^{*}$ is easily identified. We use MSE in the simulation studies to evaluate the accuracy of $\widehat{\mu }$ and Purity and F-value in the analysis of benchmark datasets.

2.4.2. Simulation Studies

We generate samples according to the density function $p_{\tau ,\beta }({\theta }^{*})$ in (50) using the Metropolis-Hastings algorithm [30,31] as

$$x\sim Z_{\tau ,\beta }({\theta }^{*}){\left\{\sum _{k=1}^K{\pi }_k^{*}w(x_i,{\mu }_k^{*},{\sum }_k^{*})\right\}}^{1+\beta },$$

(79)

where ${\mu }_1^{*}={(0,0)}^{\top }$, ${\mu }_2^{*}=(5,5)$, ${\mu }_3^{*}={(-5,-5)}^{\top }$, ${\pi }_1^{*}=0.5$, ${\pi }_2^{*}=0.2$, ${\pi }_3^{*}=0.3$ and

$${\sum }_1^{*}=\left(\begin{array}{cc}2& -0.5\\ -0.5& 1\end{array}\right),{\sum }_2^{*}=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right),{\sum }_3^{*}=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right).$$

Figure 1 illustrates the perspective plots and contour plots for $(\tau ,\beta )=(0.5,1),(0.5,0),$ (10, 1). The shape of $p_{\tau ,\beta }({\theta }^{*})$ varies according to the values of $\tau$ and $\beta$. The Gaussian mixture distribution corresponds with $\tau =0.5$ and $\beta =0$ in panel (b). When $\beta =1$, the variance of each component increases and the contours connect with each other. On the other hand, for a large value of $\tau =10$, the distribution shows high peaks around the centers. This indicates that $p_{\tau ,\beta }({\theta }^{*})$, including fuzzy c-means when $\tau \to \infty$, has a quite different shape from he Gaussian mixture distribution. Other versions of the shapes are also illustrated in Appendix C. The performance of each method is evaluated by MSE based on 100 simulated samples with sample size $n=3000$.

2.4.3. Benchmark Data Analysis

The performance of our proposed method is evaluated using benchmark datasets prepared by [32]. It includes a variety of datasets with low and high cluster overlap, various sample sizes, low and high dimensionalities and unbalanced cluster sizes. Hence, these datasets are suitable for clarifying the statistical performance of the clustering methods. In this setting, we compare the performance of k-means, fuzzy c-means, Gaussian, PAM, and Pareto clustering as well as the variants of Pareto clustering with several values of $(\tau ,\beta )$ as explained in Table S1. The characteristics of the benchmark datasets such as the sample sizes, the number of clusters, and dimensionality are summarized in Table S2.

3. Results

Figure 2 illustrates the results of MSE in the simulation studies. Pareto clustering provides the best performance in panel (a), where the samples are generated by the underlying distribution $p_{0.5,1}({\theta }^{*})$ of Pareto clustering. The shape of the distribution is similar to Gaussian mixture; however, the variance of each component becomes larger and the contour lines are connected to each other as in panel (a) of Figure 1. On the other hand, in panel (c), the variance of each component becomes smaller and contour lines are completely separated. In the both cases, the performances of the Gaussian mixture are clearly degenerated. In the case of panel (b) in which the data are generated from the Gaussian mixture, the performances are comparable to each other, suggesting that k-means, fuzzy c-means, PAM, and Pareto clustering are robust to the underlying distributions to some extent.

In the benchmark data analysis, metrics of Purity, F-value, and CI are evaluated in Tables S3–S5, where variance ${\sum }_k$ and mixing proportion ${\pi }_k$$(k=1,\dots ,K)$ in Pareto clustering are estimated. For the two-dimensional shape datasets such as Flame, Compound, D31, Aggregation, Jain, Pathbased, and Spiral in the upper rows of Table S3, existing methods such as k-means, fuzzy c-means, PAM, and Gaussian mixture outperform our proposed methods. In high-dimensional data with $d=1024$ (Dim1024), k-means and a Gaussian mixture do not work well. Other methods achieved the maximum value (1) of Purity. For datasets with a large number of clusters, D31 ($K=31$) and A3 ($K=50$), PAM performs best. For datasets with large sample size of $n\ge 5000$ and a moderate number of clusters $K=8,15$, our proposed method performs best. As for the effect of $\tau$, it barely affects the performance of our proposed method. On the other hand, $\beta$ slightly affects the performance, resulting that the intermediates among Gaussian mixture, Pareto clustering, k-means, and fuzzy c-means, such as GP, GPKF${}_1$, GPKF${}_{10}$, and GPKF${}_{100}$, show relatively good performances as a whole. We observe similar tendencies regarding the F-value (Table S4).

As for the CI, the values are relatively small for all methods, suggesting that cluster locations are properly estimated. However, some methods do not work for some datasets: Gaussian mixture for A3 (CI = 18), Birch1 (CI = 34) and Birch2 (CI = 49); k-means for D31 (CI = 7), Dim1024 (CI = 4), A3 (CI = 6), Birch1 (CI = 12) and Birch2 (CI = 23); and fuzzy c-means for D31 (CI = 5), A3 (CI = 7), Birch1 (CI = 18) and Birch2 (CI = 25). On the other hand, PAM and our proposed methods show stable results. The results, where ${\sum }_k$ and ${\pi }_k$ are not estimated and fix ${\sum }_k=I$ and ${\pi }_k=1/K$ in Pareto clustering, are shown in Tables S6–S8.

4. Discussion

We propose a new clustering method based on the generalized energy function derived from the Kolmogorov–Nagumo average. The survival function used in the generalized energy function plays an important role to ensure the minimum consistency of the parameters, which is shown in Lemma 1 using the property of divergence $G({\mu }^{*},\mu )$. We consider two examples of the survival function based on the Pareto and Fréchet distributions and show a connection among k-means, fuzzy c-means, and Gaussian mixture, leading to new methods that are intermediates among them. For the underlying distribution of our method in (50), we observe that k-means and fuzzy c-means do not have probabilistic interpretations because the corresponding underlying distributions become singular. We also propose an estimating algorithm for cluster locations, variances, and mixing proportions using the MM algorithm.

Simulation studies and benchmark data analysis show that intermediates among k-means, fuzzy c-means, and the Gaussian mixture perform well. This observation suggests that our proposed method has a wide range of applications in which k-means, fuzzy c-means, and the Gaussian mixture are used. For example, simultaneous deep learning and clustering [33] in which a deep neural network and k-means are jointly used, image segmentation using fuzzy c-means in a deep neural network [34], an application of fuzzy c-means in classification problems [35] and a parallel computation for large datasets by fuzzy c-means [36] can be investigated in the framework of the generalized energy function by the Pareto distribution.

As for the tuning parameters $\tau$ and $\beta$, we consider an approach using the leave-one-out cross validation in the Supplementary Materials in order to improve the clustering performance. The objective function in the leave-one-out cross validation is derived from an anchor loss as in [37] to estimate the optimal values of $\tau$ and $\beta$ properly. The benchmark data analysis suggests that the performance is insensitive to the values of $\tau$ but is sensitive to the values of $\beta$. Hence, this approach should be useful to determine the optimal value of $\beta$.

Banerjee et al. [38] has proposed a clustering method based on Bregman divergences and clarified the relationship between the exponential families and the corresponding Bregman divergences. They separately consider hard and soft clustering; the former corresponds to k-means style clustering and the latter corresponds to mixture model clustering. In our proposed model, the tuning parameters $\tau$ and $\beta$ bridge the gap between them and the performances are investigated by simulation studies and benchmark datasets. The extension of our method by replacing the squared distance $\parallel x_i-{\mu }_k{\parallel }_{{\sum }_k^{-1}}^2$ with Bregman divergence should improve its practical flexibility and utility. When $\beta$ or $\gamma$ divergence is used, the clustering method should be robust to contamination in observations as suggested by [39,40].

It is well known that MM algorithm and EM algorithm converge to a local optimum and the resultant clusters are sensitive to initial values [41]. One way to circumvent this difficulty is to prepare several sets of initial values and select the best one among them such as the global k-means algorithm [42]. Another approach is to combine MM algorithm and genetic algorithm (GA) to expand thoroughly the search space for the optimal solution [41,43]. The both approach can be incorporated into the Pareto clustering to make it robust to the initial values and to escape from local optimal solutions.