Advancing Spectral Clustering for Categorical and Mixed-Type Data: Insights and Applications

Abstract

This study focuses on adapting spectral clustering, a numeric data-clustering technique, for categorical and mixed-type data. The method enhances spectral clustering for categorical and mixed-type data with novel kernel functions, showing improved accuracy in real-world applications. Despite achieving better clustering for datasets with mixed variables, challenges remain in identifying suitable kernel functions for categorical relationships.

1. Introduction

Clustering for categorical and mixed-type data has been a topic of increasing interest in recent years. Several research articles have explored the application of clustering methods to datasets that involve categorical and mixed-type variables. The researchers on this topic have contributed to the development of techniques and algorithms tailored to the challenges presented by such data. Clustering categorical and mixed-type data is a complex task, due to the discrete and non-ordinal nature of categorical variables, as categorical variables cannot be ordered. For this reason, research based on categorical similarity measures is an important theme in data mining. Categorical data clustering has proved to be a particularly complicated topic to be addressed. Some categorical clustering methods have been discussed in [1,2,3,4,5]. Specifically, in [1], the author compared model-based and distance-based clustering approaches for categorical data, assessing their performance through simulation studies. The work addressed the crucial question of whether the objectives of classifying similar objects and identifying observations from the same distribution yield consistent clustering results. In [2], introducing the multiple correspondence k-means methodology, this work simultaneously applied discrete and continuous factorial models to categorical data. Its primary contribution lay in achieving a dual objective: optimal partitioning of objects and identification of orthogonal linear combinations of factors, providing a valuable alternative to tandem analysis for categorical data. In [3], addressing the need for formalized cluster descriptions in categorical data, the authors proposed CACTUS, a summarization-based algorithm. Noteworthy for its speed and scalability, CACTUS outperformed previous algorithms and enabled subspace clustering, demonstrating its efficiency in discovering clusters in subsets of attributes. The work of [4] introduced a novel approach to clustering sets of categorical data, leveraging weighted iterative methods. The methodology assigned and propagated weights on categorical values, deriving a similarity measure from value co-occurrence. The study highlighted the quick convergence of the proposed iterative methods through analytical evaluation of non-linear dynamical systems. Finally, in [5], proposing the clustering-and-dimension-reduction model, this research addressed the challenge of handling mixed qualitative and quantitative variables in two-way two-mode data matrices. The model integrated various clustering-and-dimension-reduction techniques, showcasing versatility and performance through simulation studies and real-data analyses. These works have collectively contributed to advancing the methodology of clustering categorical data, offering insights into various approaches and techniques for uncovering meaningful patterns in diverse datasets.

In addressing the intricacies of clustering categorical and mixed-type data, spectral clustering has proved to be a promising methodology. It was initially designed for continuous numerical data. This adaptation responds to the growing demand for clustering in diverse fields, such as data mining and social-network analysis. The research effort is not limited to introducing a spectral-clustering method for categorical data; it also extends the application of these kernel functions to handling categorical and mixed datasets. This adaptation involves transforming the functional forms of the self-tuning kernel (5) [6] and the adaptive density-aware kernel (6) [7], to accommodate four distinct categorical similarities, following the approach introduced in [8].

Building on the methodology proposed in [8], which successfully constructed a kernel function suitable for textual data, this article replicates the idea for categorical and mixed-type data. The adaptation includes constructing kernel functions aligned with the functional forms of the original numerical kernels, thus demonstrating the versatility of the proposed spectral-clustering approach. In terms of the kernel functions chosen in this analysis, the introduced kernel functions, initially designed for numerical data in [6,7], exhibited significant improvements in clustering outcomes compared to other commonly used kernel functions in this context.

A notable feature of (5) was its introduction of the concept of scaling the distance between two objects with the distance of those objects to their h-th nearest neighbor. Furthermore, (6) extended the idea from [6] by considering the density of objects among themselves, incorporating an additional parameter $\tau$ for enhanced flexibility in capturing relationships between data points. Therefore, the results obtained on seven categorical datasets (including three mixed-type datasets) showcase the efficacy of spectral clustering in the categorical context, proving competitive with other methods.

Spectral-clustering methods for categorical data have been already analyzed in [9] and, more recently, in [10]; however, here, a different spectral-clustering method is introduced.

The rest of the work is organized as follows: in Section 2, some similarity measures for categorical data are presented; in Section 3, the standard spectral-clustering method is summarized; in Section 4, the categorical and mixed-type spectral-clustering approach is introduced; in Section 5, the results on categorical and mixed-type real data are presented; finally, in Section 6, some concluding remarks are discussed.

2. Measures for Categorical and Mixed-Type Data

In this section, exploration and comparison of similarity measures for categorical and mixed-type data are undertaken, with the specific goal of adapting them for spectral clustering.

Selecting suitable similarity measures is crucial when clustering categorical data. Unlike numerical data, categorical variables do not possess an inherent order, which necessitates the development of distinct similarity measures. In this section, we review several important similarity measures for categorical and mixed-type data.

In the pursuit of identifying the most appropriate categorical measure for spectral clustering, a comparative analysis of 15 measures utilizing the k-means algorithm on the similarity/dissimilarity matrix was conducted and is outlined in

Table 1. Accuracy of k-means algorithm computed on similarity matrix for categorical datasets.

	BAL	BC	CA	CV	DER	HEA	LYM
Gower	0.5712	0.7292	0.7917	0.8782	0.8547	0.7723	0.5252
Dice	0.6096	0.6101	0.5222	0.8644	0.8408	0.7954	0.7465
Jaccard	0.5824	0.5776	0.5237	0.8621	0.8827	0.7954	0.7253
Eskin	0.4992	0.7076	0.8239	0.8759	0.7206	0.7855	0.4507
Goodall 1	0.5184	0.4837	0.8254	0.8736	0.8128	0.8119	0.6549
Goodall 2	0.5392	0.4874	0.8239	0.8804	0.8659	0.8152	0.6479
Goodall 3	0.5408	0.4874	0.8239	0.8759	0.8659	0.8185	0.6479
Goodall 4	0.544	0.7292	0.4717	0.8782	0.676	0.6898	0.5915
IOF	0.5072	0.7292	0.8192	0.8736	0.6788	0.8152	0.5563
OF	0.5312	0.7148	0.7917	0.8782	0.6592	0.7987	0.662
LIN	0.52	0.74	0.8178	0.8759	0.8187	0.8218	0.669
LIN1	0.464	0.74	0.536	0.8667	0.933	0.8251	0.7253
SM	0.5392	0.7328	0.8269	0.8759	0.8715	0.8251	0.5493
VE	0.5392	0.74	0.83	0.8759	0.7598	0.8086	0.5422
VM	0.5216	0.74	0.83	0.8759	0.7934	0.8119	0.5493

. Further insights into the specifics of the measures referenced in Table 1 can be found in [11,12].

Table 1 presents accuracy measures across various datasets and the categorical similarity measures considered. Specifically, accuracy provides a straightforward assessment of a classification model’s performance by indicating the proportion of correctly classified instances out of the total predictions. A higher accuracy value signifies better predictive ability, while a lower accuracy value indicates a higher rate of misclassification. Please note that in Table 1, the rows represent the different categorical similarity measures considered, while the columns represent the different datasets taken into consideration in our study. Our empirical investigations were conducted across seven distinct datasets, with the summarized details available in

Table 2. The properties of the categorical datasets.

Dataset Name	n	m	Type	Classes	Class Distribution (in %)
Balance scale (BAL)	625	4	Nominal	3	7.84, 46.08, 46.08
Breast Cancer (BC)	277	9	Nominal	2	70.76, 29.24
Credit Approval (CA)	653	15	Mixed	2	54.67, 45.33
Congressional Voting Records (CV)	435	16	Nominal	2	61.38, 38.62
Dermatology (DER)	358	34	Mixed	6	31.0, 16.76, 19.83, 13.41, 13.41, 5.59
Heart (HEA)	303	16	Mixed	2	45.54, 54.46
Lymphography (LYM)	142	18	Nominal	2	57.04, 42.96

. Specifically, our experimentation encompassed three mixed-type datasets: Credit Approval data, Dermatology data, and Heart data. Notably, barring the Gower distance, which has a predefined definition, all other categorical similarity measures were exclusively applied to categorical variables, omitting numerical variables. Consequently, within the mixed-type datasets, the computation of the Gower coefficient was directly performed on the data. Conversely, for all other categorical similarity measures, a prior categorization of continuous variables was undertaken before computing the categorical similarity/dissimilarity measures, as detailed in Section 5.

Furthermore, for computing similarities utilizing the Dice and Jaccard coefficients, an initial binarization of the data was applied, followed by the computation of the respective coefficients. Specifically, for the Dice coefficient, the binarization typically includes converting the categorical data into binary vectors, where each element corresponds to the presence or absence of a specific category. The coefficient is then calculated, based on the count of common categories between two vectors, normalized by the total count of categories in each vector. Similarly, in the case of the Jaccard coefficient, the binarization process entails representing categorical data as binary sets. Each set contains elements corresponding to the categories present in the data. The Jaccard coefficient is subsequently computed by measuring the intersection of these sets divided by their union. It is crucial to note that the binarization process is fundamental in enabling the application of these similarity measures to categorical or mixed-type data.

The k-means algorithm was executed with 100 starting points (the algorithm is quite robust regarding the number of starting points), and Table 1 presents the maximum values attained with respect to accuracy. The computation of categorical similarity/dissimilarity measures featured in Table 1 was facilitated through the utilization of the nomclust R package and the proxy R package.

Observing Table 1, it becomes evident that no categorical similarity/dissimilarity measure distinctly outperformed the others. Consequently, in order to ascertain an appropriate weighting of relationships among categorical variables within the context of spectral clustering, the LIN1 and VE dissimilarity measures, representing the most promising options, were selected. Note that among all the measures analyzed in Table 1, we chose to consider, for adapting the data to categorical spectral clustering, the two similarity measures LIN1 and VE, as they were the measures that most frequently provided the best results (see bold values in Table 1). Moreover, note that LIN1 and VE are considered promising dissimilarity measures for spectral clustering, due to their ability to capture semantic relationships, assess commonality between categories, and quantify information differences. These characteristics make them well-suited for the challenges posed by clustering categorical and mixed-type data, which led to their selection in this context.

Below, we summarize four measures for categorical data that we will consider in our categorical spectral-clustering studies.

Let $\mathit{X}=\left\{{\mathit{x}}_1,\dots ,{\mathit{x}}_n\right\}$ be a categorical dataset with n objects and m variables. Let $k^l$ for $l=1,\dots ,m$ be the number of categories of the lth variable. Let $p\left(u\right)$ for $u=1,\dots ,k^l$ be the relative frequency of the uth category of the lth variable.

2.1. Hamming Distance

The Hamming distance is a fundamental measure for categorical data. It quantifies dissimilarity between two data points based on the proportion of categorical attributes for which they differ. It is defined as

$$\mathrm{\Delta }({\mathit{x}}_i,{\mathit{x}}_j)=\sum _{l=1}^m\frac{\delta ({\mathbf{x}}_i^l,{\mathbf{x}}_j^l)}{k^l},\mathrm{for}\mathrm{all}{\mathbf{x}}_i^l,{\mathbf{x}}_j^l\in {\mathit{X}}^l,$$

(1)

where ${\mathit{X}}^l$ denotes the l-th column of the data matrix, $k^l$ is the number of categories in the l-th variable, $\delta ({\mathbf{x}}_i^l,{\mathbf{x}}_j^l)$ represents the binary difference between two values, i.e.,

$$\delta ({\mathbf{x}}_i^l,{\mathbf{x}}_j^l)=\left\{\begin{array}{c}1{\mathbf{x}}_i^l={\mathbf{x}}_j^l\hfill \\ 0\mathrm{otherwise}\hfill \end{array}\right.,\mathrm{for}l=\dots ,m,$$

and ${\mathbf{x}}_i^l$,${\mathbf{x}}_j^l$ are the i-th and j-th object of the l-th variable.

2.2. Chi-Square Distance

The chi-square distance [13] measures dissimilarity based on the chi-square statistic, which is adapted to categorical data. It is particularly useful for mixed-type datasets, combining categorical and numerical attributes. The chi-square distance is defined as

$$d({\mathit{x}}_i,{\mathit{x}}_j)=\sqrt{\left(\sum _{h=1}^n\sum _{l=1}^m{\mathit{x}}_h^l\right)\sum _{l=1}^m\frac{1}{{\sum }_{h=1}^n{\mathit{x}}_h^l}{\left(\frac{{\mathit{x}}_i^l}{{\sum }_{l=1}^m{\mathit{x}}_i^l}-\frac{{\mathit{x}}_j^l}{{\sum }_{l=1}^m{\mathit{x}}_j^l}\right)}^2}.$$

(2)

2.3. Variable Entropy Similarity

The Variable Entropy (VE) similarity [14] is a measure that takes into account the entropy of categorical attributes. It assigns higher weights to rare categories. The VE similarity for variables is defined as

$$s_l({\mathit{x}}_i^l,{\mathit{x}}_j^l)=\left\{\begin{array}{cc}-\frac{1}{ln{k}^l}\sum _{u=1}^{k^l}p\left(u\right)lnp\left(u\right)\hfill & \mathrm{if}{\mathit{x}}_i^l={\mathit{x}}_j^l\hfill \\ 0\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(3)

where $p\left(u\right)$ denotes the relative frequency of the uth category. This represents the similarity computed for all variables and it ranges between 0 and 1. VE similarity has a higher value to a match of two categories with large variability in the variable. In order to compute the proximity matrix, the following steps must be computed:

-

the total similarity $s({\mathit{x}}_i,{\mathit{x}}_j)=\frac{1}{m}{\sum }_{l=1}^m{s}_l({\mathit{x}}_i^l,{\mathit{x}}_j^l)$;

-

the dissimilarity transformation as $d({\mathit{x}}_i,{\mathit{x}}_j)=1-s({\mathit{x}}_i,{\mathit{x}}_j)$.

2.4. LIN1 Similarity

The LIN1 similarity [11] is designed to consider the frequency of categories in variables. It assigns different weights to categories based on their prevalence. This measure is particularly effective when one of the mismatched values is very frequent. The LIN1 similarity uses a subset of the ${\mathit{x}}^l$ variable denoted by $Q\subseteq {\mathit{x}}^l$, such that for all $q\in Q$, $p\left({\mathit{x}}_i^l\right)\le p\left(q\right)\le p\left({\mathit{x}}_j^l\right)$. It is defined by the following expression:

$$s_l({\mathit{x}}_i^l,{\mathit{x}}_j^l)=\left\{\begin{array}{c}\sum _{q\in Q}lnp\left(q\right)\mathrm{if}{\mathit{x}}_i^l={\mathit{x}}_j^l\hfill \\ 2\sum _{q\in Q}lnp\left(q\right)\mathrm{otherwise},\hfill \end{array}\right.$$

(4)

where ${\mathit{x}}_i^l$,${\mathit{x}}_j^l$ are the i-th and j-th object of the l-th variable.

Also in this case, we convert this similarity in a dissimilarity, following the steps:

-

compute the total similarity by $s({\mathit{x}}_i,{\mathit{x}}_j)=\frac{{\sum }_{l=1}^m{s}_l({\mathit{x}}_i^l,{\mathit{x}}_j^l)}{{\sum }_{l=1}^m\left(ln p\left({\mathit{x}}_i^l\right)+ln p\left({\mathit{x}}_j^l\right)\right)}$;

-

as the LIN1 measure in (4) has values greater than 1, the following dissimilarity transformation is used:

$$d({\mathit{x}}_i,{\mathit{x}}_j)=\frac{1}{s({\mathit{x}}_i,{\mathit{x}}_j)}-1.$$

The LIN1 measure captures the dissimilarity or similarity between categorical variables by assessing how often the categories align or diverge across different instances. In essence, it quantifies the agreement or disagreement in the categorical attributes, providing insight into the degree of similarity or dissimilarity in the patterns exhibited by those variables.

The measures defined in (1)–(4) are valuable for describing a similarity measure between statistical objects when composed of categorical variables. In this study, these measures were employed to compute the kernel function of spectral clustering, i.e., to calculate the similarity matrix associated with pairs of objects in the data. Within the realm of graph theory, this matrix can also be construed as a weight matrix determining the edge weight between two vertices in a graph.

3. Spectral Clustering

In this section, we refer to the spectral-clustering method for numerical data; further details on this methodology can be found in [8,15,16].

Consider a set of objects $\mathit{X}=\{{\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_n\}$ within $𝒳\subseteq {\mathbb{R}}^m$; in the spectral-clustering framework they can be interpreted also as vertices in an undirected graph. The initial step in grouping the data $\mathit{X}$ into K clusters involves defining a symmetric and continuous function $\kappa :𝒳\times 𝒳\to [0,\infty )$, called the kernel function. Subsequently, a weighted matrix $\mathit{W}=\left(w_{ij}\right)$ can be established, where $w_{ij}=\kappa ({\mathit{x}}_i,{\mathit{x}}_j)\ge 0$ for ${\mathit{x}}_i,{\mathit{x}}_j\in 𝒳$. In terms of graph theory, each element $w_{ij}$ of the weighted matrix $\mathit{W}$ represents the weight of the edge associated to the vertices ${\mathit{x}}_i$ and ${\mathit{x}}_j$. Moreover, for definition the matrix $\mathit{W}$ can be considered as a similarity matrix. In spectral-clustering algorithms, the frequently adopted choice is the Gaussian kernel $\kappa ({\mathit{x}}_i,{\mathit{x}}_j)=exp(-\parallel {\mathit{x}}_i-{\mathit{x}}_j{\parallel }^2/2ϵ)$, where $ϵ>0$ denotes a fixed parameter.

The selection of the kernel function plays a pivotal role in spectral-clustering algorithms, as it profoundly influences the data structure within the graph and, consequently, impacts the structure of the Laplacian and its corresponding eigenvectors. An optimal kernel function should ideally result in a weighted matrix $\mathit{W}$ exhibiting distinct diagonal blocks, facilitating well-separated groups and aiding in determining the number of groups within the dataset by merely counting these blocks.

However, within the Gaussian kernel, the primary concern arises in choosing the scale parameter $ϵ$. Addressing this, [6] proposed computing a local scaling parameter ${ϵ}_i$ for each ${\mathit{x}}_i$, introducing the following kernel function:

$${\kappa }_{ST}({\mathit{x}}_i,{\mathit{x}}_j)=exp\left(-\frac{\parallel {\mathit{x}}_i-{\mathit{x}}_j{\parallel }^2}{{ϵ}_i{ϵ}_j}\right),$$

(5)

where ${ϵ}_i=|{\mathit{x}}_i-{\mathit{x}}_h|$ and ${\mathit{x}}_h$ represents the h-th neighbor of point ${\mathit{x}}_i$ (similarly for ${ϵ}_j$). This approach, termed self-tuning, generates a similarity matrix dependent on pairwise proximities among points. Yet, despite its nomenclature, the self-tuning approach necessitates selecting the number h of neighbors for point ${\mathit{x}}_i$. Although [6] suggested $h=7$, this choice cannot be universally adopted, as highlighted in [16].

In [7], in order to amplify the intra-cluster similarity, the local density between the common neighbors of the points ${\mathit{x}}_i$ and ${\mathit{x}}_j$ was considered. The adaptive density-aware kernel function is defined by

$${\kappa }_{ADA}({\mathit{x}}_i,{\mathit{x}}_j)=exp\left(-\frac{{∥{\mathit{x}}_i-{\mathit{x}}_j∥}^2}{{ϵ}_i{ϵ}_j\left(CNN\left({\mathit{x}}_i{\mathit{x}}_j\right)+1\right)}\right),$$

(6)

where ${ϵ}_i,{ϵ}_j$ are defined as in (5), and where $CNN\left({\mathit{x}}_i{\mathit{x}}_j\right)=|\mathcal{B}({\mathit{x}}_i,\tau )\cap \mathcal{B}({\mathit{x}}_j,\tau )|$ is the number of points in the common region between the spheres with radius $\tau$ centered in ${\mathit{x}}_i$ and ${\mathit{x}}_j$, respectively. The advantage of this kernel is to be able to capture group membership in cases where point density is very important—as, for example, in cases where the shape of the clusters consists of spirals or U-shapes. In order to select the parameters $\tau$ and h, this function was also discussed in [16].

Upon computing the similarity matrix $\mathit{W}$ using a kernel function, the introduction of the normalized graph Laplacian, denoted as ${\mathbf{L}}_{\mathrm{sym}}\in {\mathbb{R}}^{n\times n}$, occurs. This matrix is formulated as

$${\mathbf{L}}_{\mathrm{sym}}=\mathbf{I}-{\mathit{D}}^{-1/2}\mathit{W}{\mathit{D}}^{-1/2},$$

where $\mathit{D}=diag({\mathit{d}}_1,{\mathit{d}}_2,\dots ,{\mathit{d}}_n)$ represents the degree matrix and ${\mathit{d}}_i$ denotes the degree of vertex ${\mathbf{x}}_i$, defined as

$${\mathit{d}}_i=\sum _j{w}_{ij}.$$

Subsequently, the K smallest eigenvalues ${\lambda }_1,{\lambda }_2,{\lambda }_3,\dots ,{\lambda }_K$ and corresponding eigenvectors ${\mathit{v}}_1,{\mathit{v}}_2,\dots ,{\mathit{v}}_K$ of ${\mathbf{L}}_{\mathrm{sym}}$ are computed.

The normalized Laplacian embedding in the K principal subspace maps the data from the input space $𝒳$ to a feature space defined by the K principal subspace of ${\mathbf{L}}_{\mathrm{sym}}$.

The map is defined as ${\mathrm{\Phi }}_{\mathrm{\Gamma }}:\{{\mathit{x}}_1,\dots ,{\mathit{x}}_n\}\to {\mathbb{R}}^K$, given by

$${\mathrm{\Phi }}_{\mathrm{\Gamma }}\left({\mathit{x}}_i\right)=(v_{1i},\dots ,v_{Ki}),i=1,\dots ,n,$$

(7)

where $v_{1i},\dots ,v_{Ki}$ are the i-th components of the eigenvectors of ${\mathbf{L}}_{\mathrm{sym}}$, ${\mathit{v}}_1,\dots ,{\mathit{v}}_K$, respectively.

Subsequently, $n\times K$ matrix $\mathit{Y}=({\mathit{y}}_1^{\prime },\dots ,{\mathit{y}}_n^{\prime })$ of the embedded data in the feature space is computed, where

$${\mathit{y}}_i={\mathrm{\Phi }}_{\mathrm{\Gamma }}\left({\mathit{x}}_i\right)=(v_{1i},\dots ,v_{Ki}),i=1,\dots ,n.$$

Thus, the clustered embedded data $\mathit{Y}$ undergo a clustering procedure, typically employing the k-means algorithm.

The spectral-clustering algorithm is outlined in Algorithm 1. Regarding the clustering of the embedded data, typically Step 6 of Algorithm 1 utilizes the k-means algorithm. However, ref. [8] proposed Gaussian mixtures as an alternative. Notably, both the k-means and mixture models generate convex clusters, with the latter accommodating more flexible shapes. The selection between the k-means algorithm and Gaussian mixtures for clustering embedded data involves trade-offs. Known for its simplicity and speed, k-means is computationally efficient and robust against noise but assumes spherical clusters and equal-sized cluster assignments. It may struggle with more complex data structures and varying cluster densities. On the other hand, Gaussian mixtures provide greater flexibility in modeling clusters with different shapes and sizes, accommodating overlapping clusters and offering probabilistic cluster assignments. Further investigations in [15] suggest that Gaussian-mixture models offer a balanced trade-off between model simplicity and efficacy after rigorous trials with diverse non-Gaussian component densities.

Algorithm 1 Spectral-Clustering Algorithm [16]

Input: dataset $\mathit{X}$, number of clusters K, kernel function $\kappa$.

Compute the similarity matrix $\mathit{W}$. The first step involves creating an affinity matrix, capturing the pairwise similarities between data points. This matrix is crucial as it forms the basis for identifying relationships within the dataset. Compute the normalized graph Laplacian ${\mathbf{L}}_{\mathrm{sym}}$ based on $\mathit{W}$. Note that the degree matrix is computed to represent the importance of each data point in the dataset. It provides a measure of the connectivity of each point to the rest of the dataset, aiding in the identification of densely connected regions. Therefore, the Laplacian matrix is then normalized to ensure consistent scaling across different datasets. This step enhances the stability and robustness of the algorithm, making it applicable to diverse data scenarios. Eigendecomposition: compute the K smallest eigenvalues $0={\lambda }_1\le {\lambda }_2\le {\lambda }_3\le \dots \le {\lambda }_K$ and consider the eigenvectors ${\mathit{v}}_1,{\mathit{v}}_2,\dots ,{\mathit{v}}_K$ corresponding to the eigenvalues ${\lambda }_1\le {\lambda }_2\le \dots \le {\lambda }_K$ of ${\mathbf{L}}_{\mathrm{sym}}$. This eigendecomposition on the normalized Laplacian matrix yields eigenvalues and corresponding eigenvectors, which encapsulate essential information about the dataset’s structure and connectivity. Embed the data in the K-th principal subspace: compute ${\mathrm{\Phi }}_{\mathrm{\Gamma }}\left({\mathit{x}}_i\right)$ according to the normalized Laplacian embedding and build the matrix $\mathit{Y}$. A subset of eigenvectors is chosen based on eigenvalues, and they play a crucial role in defining the cluster structure within the data. Data Clustering: Run a clustering algorithm on the matrix $\mathit{X}$.

Output: the data clustering $C_1,\dots ,C_K$.

4. Categorical Spectral Clustering

Spectral clustering, a widely-used method for clustering numerical data, can be adapted to categorical and mixed-type data by employing appropriate kernel functions and similarity measures. This section delves into the adaptation of spectral clustering to cater to the unique characteristics of categorical data.

In spectral clustering, the central idea is to transform the data into a graph representation, where each data point becomes a node, and the edges between nodes represent the pairwise similarities or distances between data points.

Spectral clustering requires defining appropriate kernel functions, which transform data into a form suitable for spectral analysis. In the context of categorical data, we introduce novel kernel functions, specifically designed for categorical and mixed-type datasets. These kernel functions are extensions of the self-tuning kernel and the adaptive density-aware kernel. The main contribution of this work advocates employing the kernel functions expounded in (8) and (9), wherein instead of utilizing the Euclidean distance, we incorporate specific categorical measures.

The spectral-clustering algorithm for categorical data has already been previously considered in [9], where the weighted Hamming distance introduced in (1) was used as a similarity measure to build the kernel function. Nevertheless, upon applying the kernel function suggested in [9] to the categorical datasets delineated in Table 2, the outcomes exhibited inferior performance compared to those achieved by the k-means method, concerning accuracy. Consequently, the main contribution of this work was to consider the categorical versions of the self-tuning kernel function (5) and the adaptive density-aware kernel function (6).

In order to group the categorical data $\mathbf{X}$ in K clusters via the spectral-clustering approach, the following appropriate kernel functions ${\kappa }_{ST}$ and ${\kappa }_{ADA}$ are defined:

$${\kappa }_{ST}({\mathit{x}}_i,{\mathit{x}}_j)=exp\left(-\frac{dist({\mathit{x}}_i,{\mathit{x}}_j)}{{ϵ}_i{ϵ}_j}\right),$$

(8)

$${\kappa }_{ADA}({\mathit{x}}_i,{\mathit{x}}_j)=exp\left(-\frac{dist({\mathit{x}}_i,{\mathit{x}}_j)}{{ϵ}_i{ϵ}_j\left(CNN\left({\mathit{x}}_i{\mathit{x}}_j\right)+1\right)}\right),$$

(9)

where

• $dist({\mathit{x}}_i,{\mathit{x}}_j)$ is a distance or dissimilarity measure for the categorical data presented in Section 2, i.e., the Hamming distance, chi-square distance, variable entropy similarity, and LIN1 similarity;
• ${ϵ}_i=dist({\mathit{x}}_i,{\mathit{x}}_h)$, where ${\mathit{x}}_h$ is the h-th neighbor of point ${\mathit{x}}_i$ (similarly for ${ϵ}_j$);
• $CNN\left({\mathit{x}}_i{\mathit{x}}_j\right)=|\mathcal{B}({\mathit{x}}_i,\tau )\cap \mathcal{B}({\mathit{x}}_j,\tau )|$, where $\mathcal{B}({\mathit{x}}_i,\tau )$ is the sphere centered in ${\mathit{x}}_i$ with radius $\tau$ (analogously for $\mathcal{B}({\mathit{x}}_i,\tau )$).

Following the spectral-clustering algorithm (Algorithm 1), once the similarity matrix $\mathit{W}$ through to these kernel functions is built, one has to compute the symmetric Laplacian matrix ${\mathbf{L}}_{\mathrm{sym}}$, which is a representation of the graph’s structure and provides insights into the data’s connectedness. Afterwards, eigenvalue decomposition on the Laplacian matrix is performed, devising the symmetric Laplacian embedding ${\mathrm{\Phi }}_{\mathrm{\Gamma }}$. This step yields the eigenvectors and eigenvalues, which provide valuable information about the data’s intrinsic structure. Moreover, creating a low-dimensional embedding of the data using the eigenvectors associated with the smallest eigenvalues yields an embedded representation suitable for clustering. Finally, Gaussian-mixture models are employed to cluster the low-dimensional embedding.

Moreover, note that the specific contributions of this work lie in the adaptation of kernel functions: namely, the self-tuning kernel function and adaptive density-aware kernel, for spectral clustering on categorical and mixed-type data. The novelty stems from extending these kernel functions, originally designed for numerical data, to handle the nuances of categorical variables. These adaptations address a gap in the literature, where existing kernel functions are typically designed for numerical data, making them unsuitable for nominal data. The advantages of these new kernel functions include their effectiveness in improving clustering outcomes for categorical and mixed-type data, as demonstrated by the clustering results. They provide a more nuanced and data-adaptive approach, contributing to the advancement of spectral-clustering methodologies tailored to handling diverse data types.

Remark 1. 

The choice of parameters, such as the number of clusters and the parameter(s) in the kernel functions, is essential for effective spectral clustering. These parameters are typically determined based on the characteristics of the dataset. Here, the number of clusters and the best parameter(s) in the kernel functions are assumed known values; see Section 5 for details.

The proposed approach has demonstrated its effectiveness on various real datasets, yielding competitive results compared to traditional clustering methods. The choice of specific similarity measures and kernels played a key role in the outcomes. Visualizations of embedded data and eigengap analysis provided an intuitive understanding of clustering structures. Additionally, the methodology outperformed alternatives specifically designed for categorical data, showcasing the adaptability of spectral clustering in mixed-data contexts.

5. Results on Real Data

Spectral clustering has demonstrated its effectiveness in various applications, and its adaptation for categorical data extends its utility to domains characterized by categorical and mixed-type attributes. In this section, we provide examples of real-world applications and demonstrate the practical utility of categorical spectral clustering.

The experiments were conducted using seven public datasets from the UCI Machine Learning Repository https://archive.ics.uci.edu/ml/datasets.php (accessed on 30 December 2023) and Kaggle https://www.kaggle.com/nareshbhat/health-care-data-set-on-heart-attack-possibility (accessed on 30 December 2023). These datasets include a mix of purely categorical and mixed-type data. They are described in Table 2.

Processing Mixed-Type Data. Some of the selected datasets (the Credit Approval, Dermatology, and Heart datasets) involve mixed-type attributes, including both categorical and numerical variables. The categorical measures described in Section 2, for definition, had to be computed on nominal variables, not on numerical variables. To accommodate this mixed-type data in spectral clustering, a categorization technique based on the Calinski–Harabasz index was applied. This approach categorizes numerical variables to make them compatible with categorical attributes, a critical step in ensuring the effective application of spectral clustering. This idea was introduced in [9]. The Caliski–Harabasz index was used to measure the validity index for the clustering of the continuous variables. The Caliski-Harabasz index was defined by

$$S_{k,n}^l=\frac{(n-k)S_b^l\left(K\right)}{(k-1)S_w^l\left(k\right)},$$

(10)

where $S_b^l$ is the between-cluster sum of squares and $S_w^l$ is the within-cluster sum of squares. The optimal number of categories $k_{best}^l$ was obtained by a clustering method to the numerical variable ${\mathit{x}}^l$ while calculating the corresponding validity index $S_{k,n}^l$ for ${\mathit{x}}^l$, so $k_{best}^l$ was the smallest k for which the $S_{k,n}^l$ had a local maximum. In particular, each variable was clustered according to one-dimensional k-means on every single numerical variable.

Choice of Parameters. Selecting appropriate parameters plays a crucial role in the success of spectral clustering. The choice of the number of clusters K and other kernel function parameters can significantly impact the clustering results.

The approach outlined in [16] for determining the number of clusters in spectral clustering distinguishes itself from conventional methods through the incorporation of a collective graphical analysis encompassing the similarity matrix, eigengap values, and eigenvector configurations post-Laplacian embedding. Accentuating the interdependence of proximity parameters (h and/or $\tau$) in the kernel function with the number of clusters (K), it facilitates a simultaneous analysis. The proposed multi-step procedure entails investigating a subset of proximity parameters and evaluating the similarity matrix, eigengap values, and scatter plot of the embedded data for each parameter. The key steps involve: (1) Identifying a distinct diagonal block structure or aligned/star structure. (2) Scrutinizing the eigengap plot and determining K based on a unique maximum eigengap. (3) In the presence of multiple maxima, analyzing embedded data plots for different K values based on local maxima, ensuring K is not smaller than the number of spikes.

Therefore, while there is no one-size-fits-all solution for parameter selection, some of the experiments carried out on the considered datasets indicated that setting the number of nearest neighbors h at half of the data size and determining the radius $\tau$ as the 10th percentile of the distance distribution of $\mathit{W}$ often leads to promising results.

Selecting the optimal number of clusters K is a critical step in spectral clustering. The eigengap method is used to determine the number of clusters. This method examines the first eigengap, which is the difference between consecutive eigenvalues in the spectral decomposition. Figure 1 shows that the eigengap analysis effectively identified the correct number of clusters in the Heart and Dermatology datasets. Moreover, visualizing the results of spectral clustering in categorical datasets can provide insights into the cluster structures. Figure 2 and Figure 3 show the shape of the embedded data for two example datasets: Dermatology data and Congressional Voting data. These visual representations help in understanding the clusters formed by the spectral-clustering algorithm; see [16] for further details.

Results and discussion. The performance of spectral clustering using different combinations of categorical similarity measures (1)–(4) and kernel functions (5) and (6) was evaluated. These combinations encompassed the Hamming distance, chi-square distance, variable entropy, and LIN1 similarity defined in Section 2 and the self-tuning kernel (5) and adaptive density-aware kernel (6). For completeness, we compared our approach to alternative clustering techniques, including k-Modes, h-Prototypes, k-Means applied to similarity matrices, and the spectral-clustering method proposed in a previous work [9] specifically tailored for categorical data.

Table 3. Accuracy of the spectral-clustering algorithm for different combinations of the categorical measures introduced in Section 2 and different kernel functions. In order to compare the results obtained, also shown is the accuracy of the k-means, k-Modes, h-Prototypes, and SpectralCAT algorithms.

		BAL	BC	CA	CV	DER	HEA	LYM
SpectralCAT [9]		0.4112	0.5054	0.8269	0.8736	0.87	0.8053	0.4577
$\mathit{k}$-Modes		0.5312	0.5523	0.7932	0.8667	0.5726	0.7888	0.6268
$\mathit{h}$-Prototypes		-	-	0.7976	-	0.7737	0.5875	-
$\mathit{k}$   means	Hamming	0.512	0.7076	0.8116	0.8586	0.8519	0.7987	0.4577
	Chi-square	0.4688	0.6895	0.5237	0.8804	0.7905	0.8053	0.7042
	Lin1	0.464	0.7401	0.536	0.8667	0.933	0.8251	0.7253
	VE	0.5392	0.7401	0.8392	0.8759	0.7598	0.8086	0.5422
Spectral clustering with self-tuning kernel Equation (8)	Hamming	0.4928	0.6967	0.8239	0.8827	0.905	0.8053	0.5775
	Chi-square	0.6048	0.7765	0.5268	0.8873	0.9721	0.8086	0.7746
	Lin1	0.4688	0.7473	0.5528	0.8713	0.9134	0.835	0.7324
	VE	0.4928	0.7294	0.8407	0.8804	0.8296	0.7987	0.5704
Spectral clustering with adaptive density- aware kernel Equation (9)	Hamming	0.5248	0.7292	0.8331	0.8827	0.905	0.8053	0.6056
	Chi-square	0.6608	0.7765	0.6171	0.8919	0.9721	0.8086	0.7958
	Lin1	0.4688	0.7545	0.5804	0.8759	0.9469	0.835	0.831
	VE	0.5392	0.7509	0.8407	0.8804	0.9022	0.8152	0.7958

presents the accuracy of the spectral-clustering method employing the Gaussian-mixture model on embedded data while exploring varied combinations of categorical measures and distinct kernel functions. Specifically, categorical measures from Section 2 substituted the generic distance $dist({\mathit{x}}_i,{\mathit{x}}_j)$ within the kernel functions. Notably, while SM, LIN1, and VM measures performed well (as indicated in Table 1), their results across all datasets were inferior to those reported in Table 3; hence, their omission from this discussion.

The comparative analysis in Table 3 underscores spectral clustering’s enhanced classification performance over k-Means, k-Modes, h-Prototypes, and the SpectralCat algorithm by [9]. All the experiments detailed in Table 3 were executed using R software. Our spectral-clustering algorithms were self-implemented, employing the kmodes function from the klaR R package for the k-Modes method, and the kproto function from the clustMixType R package for the h-Prototypes method.

Regarding kernel functions, the adaptive density-aware kernel (9) consistently matched or outperformed the self-tuning kernel (8). However, no single categorical measure universally fitted all the datasets. Nonetheless, the chi-square distance demonstrated commendable performance across most cases, leading us to assert that, in combination with the adaptive density-aware kernel (9), it stood as the most effective categorical measure for spectral clustering (2).

For completeness, we provide visual representations in Figure 2 and Figure 3 showcasing the embedded data shapes of two categorical datasets: the Dermatology dataset and the Congressional Voting dataset, respectively. Furthermore, Figure 1 displays the first eigengaps for the Dermatology and Heart datasets. Notably, Figure 1a reveals a maximum eigengap corresponding to $K=6$, while Figure 1b indicates a maximum eigengap at $K=2$, representing the actual number of clusters in these datasets. Additionally, Figure 4 illustrates the similarity matrix derived from the self-tuning kernel (8) with the chi-square distance for the Heart dataset, confirming the existence of two diagonal blocks and validating the actual number of clusters within the Heart data.

In our evaluations, the choice of kernel function and categorical similarity measure played a significant role in achieving the best clustering results. Notably, the chi-square distance, in combination with the adaptive density-aware kernel, consistently delivered competitive performance across different datasets. Our method outperformed other clustering techniques, as demonstrated in the results presented in Table 3. Moreover, for the sake of clarity and completeness, from results obtained in Table 3, the best spectral-clustering algorithm for categorical data can be summarized as follows:

• Compute the chi-squared distance from (2).
• Compute the similarity matrix $\mathit{W}$ from the adaptive density-aware kernel (9), using chi-square distance with $h=n/2$ and $\tau$ as the 10th percentile of the chi-squared distance distribution.
• Calculate the normalized graph Laplacian ${\mathbf{L}}_{\mathrm{sym}}$.
• Compute the eigenvalues of ${\mathbf{L}}_{\mathrm{sym}}$ and the number of clusters $K^{*}$ thanks to the first eigengap method.
• Consider the eigenvectors of ${\mathbf{L}}_{\mathrm{sym}}$ and introduce the matrix $\mathit{Y}$ with columns equal to the eigenvectors ${\mathit{v}}_1,\dots ,{\mathit{v}}_{K^{*}}$ associated with the $K^{*}$ smallest eigenvalues of ${\mathbf{L}}_{\mathrm{sym}}$.
• Re-normalize the rows of $\mathit{Y}$ to have unit length yielding $\mathit{Z}\in {\mathbb{R}}^{n\times K^{*}}$.
• Perform Gaussian-mixture model on rows of the matrix $\mathit{Z}$.

The experiments presented in the previous section have provided valuable insights into the performance of spectral clustering in the context of categorical and mixed-type data. Our findings underscore the adaptability of spectral clustering to diverse data domains and the importance of selecting appropriate similarity measures and kernel functions for optimal results. Moreover, note that the computational complexity of the suggested spectral-clustering approach depends on various factors, including the size of the dataset, the number of clusters, and the chosen similarity measures and kernel functions. The overall computational complexity is often dominated by the eigenvalue-decomposition step, making it potentially high for large datasets.

One of the critical aspects in spectral clustering for categorical data is the choice of similarity measures. Categorical data involve variables that are not ordered, making it essential to rely on appropriate similarity measures. Our analysis of different similarity measures, including Hamming distance, chi-square distance, variable entropy, and LIN1 similarity, demonstrated that not all measures perform equally well across all datasets. In particular, the chi-square distance, which considers the frequency distributions of categorical variables, consistently delivered competitive results. This result supports the notion that similarity measures that capture the variability within categories can provide effective representations for spectral clustering. The variability-based approach enables the model to identify the importance of rare categories in forming clusters.

Our experiments introduced two categorical versions of kernel functions: the self-tuning kernel and the adaptive density-aware kernel. These kernels played a crucial role in shaping the similarity matrix, affecting the clustering outcomes. The adaptive density-aware kernel, in most cases, demonstrated superiority over the self-tuning kernel. The kernel-function selection has a direct impact on how the data are represented and separated in the spectral embedding space. Our results suggest that kernel functions that adapt to the data’s density provide a more flexible and effective approach for spectral clustering.

The performance of our categorical spectral-clustering approach was compared to other established clustering techniques. These alternatives included k-Modes, h-Prototypes, k-Means applied to similarity matrices, and the spectral-clustering method specifically designed for categorical data [9]. The results clearly indicated the superiority of our approach over these alternatives. This outcome underlines the potential of spectral clustering as a robust and versatile technique for clustering categorical and mixed-type data. Moreover, it highlights the importance of tailored methods for these types of data, as traditional techniques may not provide competitive results.

Visualizations of the embedded data for specific datasets, as presented in Figure 2 and Figure 3, offer an intuitive understanding of the cluster structures formed by spectral clustering. Visual representations provide a practical way to inspect the results and assess their quality. Furthermore, determining the optimal number of clusters is essential for spectral clustering. The eigengap analysis, as demonstrated in Figure 1, effectively identified the correct number of clusters for specific datasets. This highlights the value of the eigengap method as a useful tool for number-of-clusters selection in spectral clustering. Furthermore, it is observed that, as demonstrated in [16], an alternative method to determine the optimal number of clusters involves counting the visible diagonal blocks within the similarity matrix $\mathit{W}$, as exemplified in Figure 4.

Moreover, across the various scenarios considered in Table 3, it can be observed that the method consistently demonstrated superiority over the alternatives, as indicated by the reported accuracy values. This consistent out-performance suggests that the spectral-clustering approach is not overly sensitive to specific dataset characteristics and maintains its effectiveness across diverse scenarios. The spectral-clustering approach appears to be robust and capable of generalizing well across different types of datasets, showcasing consistent and competitive performance across various scenarios. Rather, its sensitivity may lie in the choice of the initial categorical similarity measure used to transform the categorical data into numerical form.

In conclusion, our experiments emphasized the effectiveness of spectral clustering in categorical and mixed-type data, providing competitive results compared to other traditional clustering methods. The choice of similarity measures, kernel functions, and the visual inspection of results play pivotal roles in the success of spectral clustering. These findings are particularly significant for practitioners dealing with categorical and mixed-type data, where tailored clustering approaches are essential. Finally, we note that the computational complexity of spectral clustering, primarily due to eigenvalue decomposition, can be addressed for large datasets through various strategies. However, it is important to underline that the investigation into the overall computational complexity of the algorithm is beyond the scope of this work.

In summary, from the results obtained, one can state that the choice of similarity measure showed that the chi-square distance often provided superior performance, emphasizing the importance of capturing variability in categorical variables. The analysis highlighted that density-adaptive kernels, especially the density-aware kernel, often led to better results than self-tuning kernels. Furthermore, the evaluation of the optimal number of clusters, through eigengap analysis, proved to be an effective strategy. The discussion emphasizes that despite the computational complexity associated with spectral clustering, the promising results justify the use of this methodology, especially when dealing with categorical and mixed data.

6. Conclusions

In this article, we explored the application of spectral clustering to the challenging domain of categorical and mixed-type data. Spectral clustering effectively revealed intricate cluster structures in such data, providing competitive results when compared to alternative clustering methods.

Role of Similarity Measures. Selecting appropriate similarity measures for categorical data is critical. Among the measures we evaluated, the chi-square distance consistently yielded competitive results. By considering the frequency distributions of categorical variables, this measure captured the nuances of categorical data and proved to be a suitable choice.

Kernel Functions. The choice of kernel functions played a crucial role in spectral clustering. The adaptive density-aware kernel outperformed the self-tuning kernel in most cases. These kernel functions had a significant impact on the data representation and clustering results.

Competitive Edge. Our categorical spectral-clustering approach outperformed other traditional clustering methods, demonstrating the potential of spectral clustering as a robust and versatile technique for clustering categorical and mixed-type data.

Visual Analysis. Visualizing the embedded data provided valuable insights into the cluster structures formed by spectral clustering. This visual inspection was essential for assessing the quality of the clustering results.

Optimal number of clusters. The eigengap analysis effectively determined the optimal number of clusters for specific datasets. This technique assisted in selecting the correct cluster count, a crucial step in spectral clustering.

In conclusion, spectral clustering for categorical and mixed-type data is a promising and effective approach. Its adaptability to various data domains, versatility in handling categorical variables, and competitive performance make it a valuable tool for data-clustering tasks. Understanding the nuances of categorical data, selecting appropriate similarity measures and kernel functions, and visually inspecting the results are essential steps in achieving successful spectral-clustering outcomes.

Applications in Economic and Financial Datasets. Spectral clustering holds particular importance in economic and financial datasets, where variables range from categorical attributes, like industry sectors, to numerical data, such as financial-performance metrics. Its versatility in accommodating this amalgamation of data types enables effective pattern recognition and segmentation, elucidating nuanced relationships and inherent groupings. Spectral clustering emerges as a potent tool capable of integrating categorical, numerical, and mixed data, facilitating insightful analyses in risk assessment, fraud detection, portfolio optimization, and market segmentation.

Future investigations should focus on automating parameter-selection methods for spectral clustering in categorical and mixed-type data. This entails developing approaches to determining optimal parameters, including the number of clusters and tuning parameters for similarity measures and kernel functions. Additionally, exploring innovative similarity measures specifically designed for categorical data is crucial to enhancing clustering efficacy. Overall, the future trajectory involves automating parameter selection, exploring novel similarity measures, and conducting thorough investigations into the algorithm’s robustness for improved practical applications. The potential of spectral clustering in categorical and mixed-type data calls for future research emphasis on automated parameter selection and the creation of specialized similarity measures. This will further enhance the adaptability and effectiveness of spectral clustering in diverse domains. Categorical spectral clustering is a promising approach, providing new perspectives and insights into complex data structures.