Investigating How Reproducibility and Geometrical Representation in UMAP Dimensionality Reduction Impact the Stratification of Breast Cancer Tumors

Abstract

Advances in next-generation sequencing have provided high-dimensional RNA-seq datasets, allowing the stratification of some tumor patients based on their transcriptomic profiles. Machine learning methods have been used to reduce and cluster high-dimensional data. Recently, uniform manifold approximation and projection (UMAP) was applied to project genomic datasets in low-dimensional Euclidean latent space. Here, we evaluated how different representations of the UMAP embedding can impact the analysis of breast cancer (BC) stratification. We projected BC RNA-seq data on Euclidean, spherical, and hyperbolic spaces, and stratified BC patients via clustering algorithms. We also proposed a pipeline to yield more reproducible clustering outputs. The results show how the selection of the latent space can affect downstream stratification results and suggest that the exploration of different geometrical representations is recommended to explore data structure and samples’ relationships.

1. Introduction

Due to heterogeneity in different cancer types, the stratification of cancer patients is a major clinical challenge. It is also a key goal of precision medicine because it would facilitate targeted therapies [1]. Breast cancer (BC) is the world’s most diagnosed female tumor, with a high incidence ($11.7\%$ of female cancer cases) and mortality ($15.5\%$ of total female deaths) worldwide in 2020 [2]. Moreover, incidence is estimated to increase to around 3,000,000 cases in 2040 [3]. Tumor stratification for breast carcinoma will therefore become increasingly important in clinical management, clinical trials, and epidemiological and functional studies [4].

Recent advances in next-generation sequencing (NGS) have created massive datasets for genomic big data [5]. NGS has enabled the fast, accurate, and cost-effective analysis of DNA and RNA samples, allowing patients to be stratified by their transcriptomic profiles [6,7,8].

As with all high-dimensional data, RNA-seq datasets suffer from the curse of dimensionality [9]. Thus, several machine learning (ML) methods have been proposed to extract relevant features and information [10,11].

Of these, uniform manifold approximation and projection (UMAP) [12] has been widely used in several tumor stratification and RNA-seq analyses [13,14,15,16].

Unlike other popular dimensionality reduction methods, such as principal component analysis (PCA) [17], UMAP is nonlinear and emphasizes local data structures well. It is therefore used in a preprocessing step to reduce the dimensionality of a genomic dataset, either before implementing a full ML model [18] or before visualizing complex expression profiling data [19]. A recent comparative study [20] showed how UMAP can also considerably improve the performances of clustering algorithms.

In general, users tune the number of neighbors, i.e., the main UMAP hyperparameters, to project local data structures in a low-dimensional Euclidean space. Nevertheless, several other tuning parameters can be specified. In particular, users can choose the embedding metric onto which UMAP projects the data. By default (https://umap-learn.readthedocs.io/en/latest/parameters.html, accessed on 20 April 2022), it is a Euclidean distance. Basically, the more the geometry of the latent space matches the structure of the input data, the more important the quality of the embedding representation [21]; therefore, different metric spaces (onto which to project the data) might reveal different sample relationships and structures. Recently, Ref. [22] proposed a deep generative model to embed cells into low-dimensional hyperspherical or hyperbolic spaces, providing excellent visualization for data exploration. In another study, Ref. [23] demonstrated how a hyperbolic embedding can reveal meaningful hierarchies among samples starting from pairwise similarity information.

Hence, when we cannot know the geometry of the latent space a priori, different types of metrics should be explored. To the best of our knowledge, the literature contains no reports on investigating different curvatures of the UMAP embedding space.

Given the above, here we evaluated how non-Euclidean UMAP embeddings can impact the results of a breast cancer stratification analysis. We used UMAP on RNA-seq data with three different output metrics: Euclidean, spherical, and hyperbolic. Then, we stratified the samples via clustering algorithms applied on each UMAP embedding. We then compared the results in terms of clustering accuracy.

For the present study, we used the RNA-seq breast cancer data downloaded from The Cancer Genome Atlas (TCGA) website. Details of the dataset are presented in the Methods section.

Furthermore, UMAP reproducibility is not exact (https://umap-learn.readthedocs.io/en/latest/reproducibility.html, accessed on 20 April 2022) because it is a stochastic algorithm. Thus, downstream results might differ between runs. To overcome this, we propose a pipeline to combine the clustering outputs obtained from different embeddings into one final robust result.

Our results highlight how a different latent space, onto which practitioners project high-dimensional data, can affect the final results, suggesting that the choice of the embedding geometry is meaningful for tumor stratification. Specifically, our simulations show similar performances between Euclidean and hyperbolic metrics, while the spherical one performs worse than other metrics on three out of four tested algorithms.

2. Materials and Methods

2.1. Data

Because BC is the most commonly diagnosed female tumor [2] and similarly to [24], we used the RNA-seq dataset and the clinical variables related to BC, profiled by The Cancer Genome Atlas (TCGA) [25] (for more details on BC RNA-seq data, see the Data Availability Statement section of the manuscript). We filtered out the male and metastatic samples, then merged the data with the BC subtypes that include 192 basal-like, 563 luminal-A, 207 luminal-B, 82 HER2-enriched, and 40 normal-like samples. The final dataset comprises 20,530 gene expressions per 1084 samples.

2.2. UMAP

To reduce the dimensionality of the RNA-seq data, we used UMAP (umap-learn 0.4.0 package in Python: https://umap-learn.readthedocs.io/en/latest/release_notes.html, accessed on 20 April 2022), an unsupervised algorithm for nonlinear neighbor graph-based dimensionality reduction [12], which is popular in various fields [16,26,27]. This was the first step in a cluster analysis to address the “curse of dimensionality” [9], a common problem for clustering algorithms.

Briefly, UMAP computes a high-dimensional weighted graph of the data, with edge strength quantifying how a vertex (sample) is connected to another. Then, UMAP embeds the data points into a low-dimensional space, minimizing the fuzzy-set cross-entropy between the high- and low-dimensional graphs.

Several parameters must be specified to apply UMAP. First, the number of nearest neighbors (NNs) is required to construct the initial graph. To select the best NN, we embedded the data several times with different NN values, ranging from 10 to 100 (in steps of five), and chose the one that improved the clustering performance in terms of adjusted rand index (ARI) [28] and homogeneity score [29]. Second, to better visualize the projected data, UMAP allows one to define a minimum distance (MD) between nearest neighbors in low-dimensional space. Since a low MD value is more appropriate [30] for downstream clustering analysis, we set MD equal to zero. Finally, two metrics can be defined: one metric to compute the distance between the input data points, and the other to compute distances in the final embedding, i.e., the output metric. Common choices fall within Euclidean distance for both metrics, but there is no reason to assume that the best topological representation of the input data lies in a low-dimensional Euclidean space. Correspondence between the structure of the input data and the geometry of latent space can improve embedding representation and downstream analysis such as clustering and classification [31]. Therefore, setting the input metric as Euclidean, we projected the 20,530 gene expressions not only onto 3-dimensional (we chose 3 dimensions for the Euclidean space as it is viewable and more informative than the 2-dimensional representation) Euclidean space but also onto 2-dimensional spherical and hyperbolic surfaces, in order to visualize and evaluate how a different topology of the latent space can affect the final results in terms of clustering performance.

2.3. Clustering Algorithms

Tumor stratification from transcriptomic data aims to identify groups of samples with common characteristics (i.e., tumor subtype). This task is commonly addressed with clustering techniques. In this study, we tested four clustering algorithms, comparing their outputs: hierarchical density-based spatial clustering of applications with noise (HDBSCAN), density-based spatial clustering of applications with noise (DBSCAN), ordering points to identify the clustering structure (OPTICS), and agglomerative clustering. We chose these algorithms as they do not rely upon centroid computation which is a non trivial operation for non-Euclidean metrics.

HDBSCAN is a density-based hierarchical method developed by Campello et al. [32]. By creating a series of nested sample groups, it allows one to perform hierarchical clustering and so to explore tumor stratification on RNA-seq data in greater detail.

The algorithm does not assign lower density points to any cluster, marking them as Noise. In this work, we referred to them as Not Clusterable (NC) points. Moreover, the algorithm requires two main hyperparameters to be defined. The first hyperparameter, Min Cluster Size (MCS), is the minimum number of elements required to build a group. The second hyperparameter, Min Samples (MS), is related to the extent of NC points. In our analysis, the MCS value was set to 30, while the MS was optimized based on the value of ARI, with testing values ranging from 1 to 100.

The DBSCAN algorithm, proposed in 1996 [33], is the precursor of HDBSCAN. Its main hyperparameters are the maximum distance between two samples (EPS) and MS. The first was optimized by testing values from 0.05 to 10, and the second one was optimized as for HDBSCAN.

Moreover, the OPTICS algorithm [34] is closely related to DBSCAN and its main hyperparameter is MS.

Finally, the agglomerative clustering [35] was the last one we tested. It recursively merges pairs of clusters of sample data and it requires the number of clusters to be defined; we fixed that value at 4.

The evaluation of clustering quality and the hyperparameter optimization were based on two scores. The first score is ARI, a popular score for evaluating the quality of clustering [36,37]. Ranging from $-1$ to 1, it computes the similarity between the partitions generated by the algorithm and the one we expect, according to a specific clinical variable (in our analysis, the BC subtype). A value of 1 indicates that the algorithm is able, in an unsupervised manner, to aggregate the samples as defined in the clinical variable. The second score is the homogeneity [29], ranging from 0 to 1, where 1 corresponds to a perfectly homogeneous partition. A clustering result is homogeneous if all of its clusters contain only data points that are members of a single class.

We considered it to be a good stratification result if we obtained simultaneously high scores for ARI and homogeneity. In particular, a high ARI score means that the algorithm found similar tumor subtypes to those we expected, while a high homogeneity score indicates that tumor clusters comprise samples of the same BC subtype.

2.4. Reproducibility

Due to use of stochasticity for optimization, the UMAP reproducibility is not exact (https://umap-learn.readthedocs.io/en/latest/reproducibility.html, accessed on 20 April 2022). This means that, for some datasets such as RNA-seq data, where the tumor subgroups are a complex function of the transcriptomic profiles, applying a clustering algorithm on two different runs of UMAP might return two different outputs. We showed that, although relatively stable, the results have a minimum of variability in terms of the number of estimated clusters, or samples belonging to the same or different group. When it is important to identify the correct number of clusters, as in tumor stratification, it is crucial to rely on algorithms that reproduce the same output over several runs, in order to have reliable results.

Therefore, for each NN, to obtain a robust clustering result and to quantify how frequently a sample was paired to another sample set, we propose a new pipeline of analysis that applies a clustering algorithm to T different UMAP embeddings, where T is the number of runs, and compacts all the HDBSCAN tumor groups into one final reproducible output. Specifically, we defined the binary proximity matrix $A_t$, which indicates if two samples belong to the same cluster. Then, after T iterations, we computed the affinity matrix P as:

$$P=\frac{1}{T}\sum _{t=1}^T{A}_t.$$

The square matrix P quantifies how frequently a pair of samples was clustered together, ranging from 0 (never) to 1 (always). Hence, from P we know the strength with which two individuals belong to the same group, but to obtain a final stratification we must apply a clustering algorithm to P. Spectral clustering (SC) is the most straightforward way to perform clustering, starting from the affinity matrix [38,39,40]. However, the SC algorithm requires the number of clusters k to be returned as a hyperparameter. We set k equal to the mode of the number of partitions observed during the T runs, since it is the favorite solution proposed over the T runs. In the Supplementary Material, we demonstrate via simulations how our method yields a more reproducible clustering output than the implementation of a clustering algorithm on a single UMAP run.

Nevertheless, result reproducibility came at the cost of losing latent space visualization, since the final clusters were no longer computed on one specific embedding, but rather were estimated from several embeddings. However, to visualize the UMAP data projection in a way that best represents the final clustering labels, from T embeddings we selected the one with the most similar clustering result in terms of ARI. This is just a visual approximation, merely useful for results presentation; nevertheless, we were interested here in the reproducibility of the tumor stratification results, that is, in having tumor subgroups that are well-identifiable even if the entire analysis is run many times. Finally, once we had reproducible cluster outputs coming from different geometrical representations, we chose the one with the highest ARI.

Our methodological workflow is depicted in Figure 1.

Hyperparameters of the pipeline (Algorithm 1) are: the output metric M, the NN, the number of runs T, and the clustering algorithm. The pipeline can be summarized with the following steps

Algorithm 1 Pseudocode of the pipeline

for$clust\_algo$ in {HDBSCAN, DBSCAN, OPTICS, agglomerative clustering} do     for M in {Euclidean, Spherical, Hyperbolic} do         for NN in $\{10,15,\dots ,100\}$ do            for t in $\{1,2,\dots ,T=100\}$ do                generate UMAP embedding E with output metric equal to M and                 number of near neighbors equal to NN;                apply $clust\_algo$ on E and save the proximity matrix $A_t$, where the                $a_{ij}$ element of matrix $A_t$ is equal to 1 if the i-th and j-th sample were                assigned to the same cluster, 0 otherwise, for $i,j=1,\dots ,n$, with n                the sample size;            end for            set $P=\frac{1}{T}{\sum }_{t=1}^T{A}_t$;            set k equal to the mode of the number of clusters identified over            the T clustering results;            implement spectral clustering with k groups and P as affinity matrix.         end for     end for end for among the M UMAP embeddings with different NN, select, within the various $clust\_algo$, the cluster output with the highest ARI.

The code was implemented in R 4.1.3 (https://www.r-project.org/, accessed on 20 April 2022) and Python 3.8.10 (https://www.python.org/, accessed on 20 April 2022). The figures were created using the Plotly $4.10.0$ R package and Matplotlib 3.3.4 Python package. The entire pipeline run on the high-performance computing (HPC) cluster at the data center of Engineering D.HUB in Pont-Saint-Martin, which is equipped with CPU and GPU computational nodes. The code to run the entire pipeline is available at: https://bitbucket.org/jordy_bollon/jordy-bollon/src/master/, accessed on 20 April 2022.

3. Results

3.1. UMAP

We reduced the high dimensionality of RNA-seq data from 20,530 gene expressions to two (spherical and hyperbolic surface) or three (Euclidean space) latent dimensions. For clarity and to give a visual example of the latent spaces, for each metric, we only visualized one UMAP embedding with an NN that returns the best ARI, after HDBSCAN implementation (see Figure 2).

3.2. Clustering

3.2.1. Irreproducibility Issues

As described in the Reproducibility section, for each output metric and NN, we implemented the UMAP + clustering algorithm pipeline $T=100$ times to highlight how independent runs of UMAP + clustering yield inconsistent outputs in terms of variability of the N_c and ARI. In Figure 3, for each clustering technique, we report the absolute frequency of the estimated N_c over the T iterations for each NN. The Euclidean metric shows the least variability in estimating N_c over the T runs for each NN: the mode of N_c is equal to 2 for HDBSCAN, 4 for DBSCAN, and 3 for OPTICS. The hyperbolic space is slightly more variable than the Euclidean one: the mode of N_c is equal to 2 for HDBSCAN, except for NN equal to 10 and 20, 4 for DBSCAN, and it varies between 3 and 4 for OPTICS. On the contrary, the spherical surface has the highest variability in estimating N_c: the mode of N_c varies between 3 and 4 for HDBSCAN, it ranges from 2 to 5 for DBSCAN, and it is equal to 3 for OPTICS.

The inconsistency in results arises even when considering the clustering quality, which is quantified by means of the ARI score, as described in the Methods section. To assess its irreproducibility, we evaluated the interquantile range of the ARI returned by each implementation of the clustering method applied on T embeddings. Looking at Figure 4, except for HDBSCAN and OPTICS applied on Euclidean and hyperbolic spaces, the interquartile range of the ARI is quite large for each NN. Despite this high variability, if we consider the median of the ARI value, the Euclidean and hyperbolic metrics have higher performances than the spherical one on three (DBSCAN, OPTICS, and agglomerative clustering) clustering algorithms out of four.

3.2.2. Reproducible Results

To compare the clustering performances with reproducible output, as described in the Methods section, from 100 clustering outputs, we computed a final and reproducible clustering result for each NN. Of these, we selected the one with an NN that maximizes the ARI. As shown in supplementary results (

Table A2. Comparison of clustering performances. N_c, ARI, and homogeneity score estimated for each NN and latent space (spherical, Euclidean, hyperbolic) for HDBSCAN and DBSCAN clustering methods.

HDBSCAN	Euclidean			Sphere			Hyperboloid
NN	Nclust	Ari	Homog	Nclust	Ari	Homog	Nclust	Ari	Homog
10	2	0.33	0.31	4	0.45	0.43	3	0.46	0.42
15	2	0.33	0.32	4	0.46	0.44	2	0.33	0.32
20	2	0.33	0.32	4	0.46	0.44	3	0.45	0.42
25	2	0.33	0.32	4	0.46	0.44	2	0.33	0.32
30	2	0.33	0.32	4	0.46	0.44	2	0.33	0.32
35	2	0.33	0.32	4	0.45	0.43	2	0.33	0.32
40	2	0.33	0.32	3	0.37	0.35	2	0.33	0.32
45	2	0.33	0.32	3	0.37	0.35	2	0.33	0.32
50	2	0.33	0.32	4	0.46	0.43	2	0.33	0.32
55	2	0.33	0.32	4	0.46	0.43	2	0.33	0.32
60	2	0.33	0.32	4	0.46	0.43	2	0.33	0.32
65	2	0.33	0.32	4	0.47	0.43	2	0.33	0.32
70	2	0.33	0.32	4	0.47	0.43	2	0.33	0.32
75	2	0.33	0.32	4	0.47	0.43	2	0.33	0.32
80	2	0.33	0.32	4	0.46	0.43	2	0.33	0.32
85	2	0.33	0.32	4	0.46	0.43	2	0.33	0.32
90	2	0.33	0.32	3	0.45	0.4	2	0.33	0.32
95	2	0.33	0.32	3	0.45	0.39	2	0.33	0.32
100	2	0.33	0.32	3	0.46	0.4	2	0.33	0.32
10	4	0.39	0.42	4	0.28	0.39	4	0.45	0.43
15	4	0.45	0.42	2	0.33	0.32	4	0.46	0.43
20	4	0.45	0.43	2	0.33	0.32	4	0.45	0.42
25	4	0.45	0.43	2	0.33	0.32	4	0.46	0.43
30	4	0.45	0.43	2	0.32	0.31	4	0.45	0.43
35	4	0.45	0.42	2	0.31	0.3	4	0.45	0.43
40	4	0.45	0.42	2	0.32	0.3	4	0.45	0.43
45	4	0.45	0.42	2	0.32	0.3	4	0.45	0.43
50	4	0.44	0.42	2	0.32	0.31	4	0.45	0.43
55	4	0.43	0.4	2	0.29	0.28	4	0.45	0.43
60	4	0.45	0.42	2	0.31	0.29	4	0.45	0.43
65	4	0.45	0.42	2	0.32	0.31	4	0.45	0.43
70	4	0.36	0.38	2	0.32	0.3	4	0.45	0.43
75	4	0.39	0.38	3	0.37	0.38	4	0.45	0.43
80	4	0.37	0.36	3	0.38	0.38	4	0.45	0.43
85	4	0.37	0.36	2	0.32	0.31	4	0.44	0.43
90	4	0.38	0.37	3	0.4	0.39	4	0.45	0.43
95	4	0.37	0.36	2	0.32	0.31	4	0.45	0.43
100	4	0.36	0.36	3	0.39	0.38	4	0.44	0.42

and

Table A3. Comparison of clustering performances. N_c, ARI, and homogeneity score estimated for each NN and latent space (spherical, Euclidean, hyperbolic) for OPTICS and agglomerative clustering (Agg.).

OPTICS	Euclidean			Sphere			Hyperboloid
NN	Nclust	Ari	Homog	Nclust	Ari	Homog	Nclust	Ari	Homog
10	3	0.42	0.39	3	0.4	0.38	3	0.43	0.4
15	3	0.43	0.4	3	0.41	0.38	3	0.44	0.41
20	3	0.44	0.41	3	0.41	0.39	4	0.36	0.41
25	3	0.44	0.41	3	0.41	0.39	3	0.44	0.41
30	3	0.44	0.41	3	0.41	0.39	3	0.44	0.41
35	3	0.43	0.41	3	0.4	0.38	3	0.43	0.41
40	3	0.44	0.41	3	0.4	0.38	3	0.44	0.41
45	3	0.43	0.41	3	0.4	0.38	3	0.44	0.41
50	3	0.44	0.41	3	0.4	0.38	3	0.44	0.41
55	3	0.44	0.41	3	0.4	0.38	3	0.44	0.41
60	3	0.44	0.41	3	0.4	0.38	3	0.44	0.41
65	3	0.43	0.41	3	0.4	0.38	3	0.43	0.41
70	3	0.43	0.41	3	0.4	0.38	3	0.42	0.4
75	3	0.43	0.41	3	0.4	0.38	3	0.43	0.4
80	3	0.43	0.41	3	0.4	0.38	3	0.43	0.41
85	3	0.43	0.4	3	0.41	0.39	3	0.43	0.4
90	3	0.43	0.41	3	0.4	0.38	3	0.43	0.4
95	3	0.43	0.4	3	0.4	0.38	3	0.42	0.4
100	3	0.43	0.41	3	0.39	0.37	3	0.42	0.4
10	4	0.37	0.48	4	0.28	0.42	4	0.38	0.49
15	4	0.37	0.48	4	0.28	0.42	4	0.38	0.49
20	4	0.36	0.48	4	0.24	0.4	4	0.36	0.49
25	4	0.36	0.49	4	0.24	0.4	4	0.37	0.47
30	4	0.37	0.48	4	0.24	0.4	4	0.36	0.47
35	4	0.37	0.48	4	0.24	0.4	4	0.36	0.47
40	4	0.37	0.49	4	0.24	0.41	4	0.36	0.47
45	4	0.38	0.49	4	0.24	0.4	4	0.36	0.47
50	4	0.38	0.49	4	0.24	0.4	4	0.35	0.47
55	4	0.38	0.49	4	0.24	0.4	4	0.35	0.47
60	4	0.37	0.49	4	0.24	0.41	4	0.35	0.47
65	4	0.37	0.48	4	0.24	0.41	4	0.35	0.47
70	4	0.38	0.49	4	0.24	0.4	4	0.36	0.48
75	4	0.37	0.49	4	0.24	0.4	4	0.36	0.48
80	4	0.37	0.48	4	0.24	0.4	4	0.35	0.48
85	4	0.37	0.49	4	0.24	0.4	4	0.36	0.48
90	4	0.38	0.49	4	0.24	0.4	4	0.35	0.48
95	4	0.37	0.48	4	0.24	0.39	4	0.34	0.47
100	4	0.38	0.49	4	0.24	0.41	4	0.34	0.47

), the best reproducible results are:

• N${}_c=4$ and ARI equal to 0.47 (spherical metric), N${}_c=2$ and ARI equal to 0.33 (Euclidean metric), and N${}_c=3$ and ARI equal to 0.46 (hyperbolic metric), for HDBSCAN;
• N${}_c=3$ and ARI equal to 0.39 (spherical metric), N${}_c=4$ and ARI equal to 0.45 (Euclidean metric), and N${}_c=4$ and ARI equal to 0.46 (hyperbolic metric), for DBSCAN;
• N${}_c=4$ and ARI equal to 0.28 (spherical metric), N${}_c=4$ and ARI equal to 0.38 (Euclidean metric), and N${}_c=4$ and ARI equal to 0.38 (hyperbolic metric), for agglomerative clustering;
• N${}_c=3$ and ARI equal to 0.41 (spherical metric), N${}_c=3$ and ARI equal to 0.44 (Euclidean metric), and N${}_c=3$ and ARI equal to 0.44 (hyperbolic metric), for OPTICS.

Even with our reproducible pipeline, in terms of ARI the Euclidean and hyperbolic metrics have identical performances, higher than the spherical one on three (DBSCAN, OPTICS, and agglomerative clustering) clustering algorithms out of four. Only with HDBSCAN does the Euclidean metric perform worse than spherical and hyperbolic metrics which have similar best ARI values: 0.47 and 0.46, respectively. Overall, we observed the best performance with spherical embedding coupled with HDBSCAN: an ARI index of 0.47, a homogeneity score of 0.43, and four clusters. However, as explained in the Reproducibility section, we cannot have a truthful latent space visualization. We therefore plotted our best final clustering labels on the more representative UMAP embedding (Figure 5b).

3.2.3. Biological Interpretation

At a glance, by comparing Figure 5a with Figure 5b, some of the identified clusters can be directly associated with the different subtypes in the BC dataset. In particular, as we expected (https://www.breastcancer.org/symptoms/types/molecular-subtypes, accessed on 20 April 2022), the basal samples are detached well from the others, and form a separate cluster (cluster 2 in Figure 5). However, as reported in the literature [41], the normal, luminal-A, and luminal-B samples are more similar to each other. Unsurprisingly, they are grouped together in cluster 1, while cluster 3 consists entirely of Her2 points.

This comparison between subtypes and cluster groups is quantified in Figure 6, which reports the relative and absolute frequency of BC subtypes for each estimated cluster. It is clear that clusters 1, 2, and 3 represent three BC subtypes: luminal, basal, and Her2, respectively. In contrast, the NC cluster is a mixture of all subtypes. However, we computed how frequently these samples were assigned to the NC group and we discovered that all 41 points were clustered as such in more than $55\%$ of the iterations performed.

Although Figure 5b is not the authentic representation of the cluster labels, we can see that the NC points (yellow) are those at the border of the luminal groups.

4. Discussion

Using the UMAP algorithm, here we projected 20,530 gene expression data onto three different latent metric spaces in order to evaluate how different curvatures of the UMAP embedding could affect breast cancer stratification. The evaluation focused on clustering performances in terms of ARI, homogeneity, and number of estimated clusters returned by HDBSCAN, DBSCAN, agglomerative clustering, and OPTICS. We also addressed UMAP reproducibility by proposing an iterative approach to yield more stable clustering outputs.

UMAP has been widely implemented for transcriptomic analyses [13,14,42,43]. In particular, starting from single-cell RNA-seq data, Bao et al. [42] and Landry et al. [43] visualized tumor cell heterogeneity in triple-negative breast cancer and glioblastoma, respectively. Yang et al. [13] and Lebedev et al. [14] performed transcriptomic analyses, applying the HDBSCAN algorithm to UMAP Euclidean embedding. Moreover, Yang et al. [13] also addressed how the choice of the input metric in UMAP (i.e., how we measure the distance between samples in high-dimensional input data) could influence the visualization of clustering structures.

Nevertheless, none of the above approaches raised concerns about the reproducibility and the geometry of the UMAP embedding and, to the best of our knowledge, no study has addressed these issues.

Our work focused on the choice of the output metric in UMAP, showing that the selection of the latent space can affect downstream clustering results. However, by exploring different metric spaces, we encountered UMAP reproducibility issues: some pairwise points were inconsistently projected onto latent spaces and so were not always clustered together over several runs, leading to unstable clustering results (see Figure 3 and Figure 4). Since this work was the first attempt to investigate different curvatures of UMAP embedding, this reproducibility issue spurred us to propose a reproducible pipeline that returns consistent results, regardless of the selected output metrics.

As reported in the Appendix (Table A2 and Table A3), clustering performances were improved by applying HDBSCAN to spherical and hyperbolic metric spaces. On non-Euclidean embedding, the ARI score increased by 14 percentage points (from $0.33$ to $0.47$) with respect to the Euclidean latent space. Hence, in this analysis the Euclidean output metric, when applied on HDBSCAN, performs poorly compared to the other metrics. On the contrary, Euclidean and hyperbolic latent spaces showed similar ARI scores that were higher than the spherical one for three clustering algorithms out of four.

The above results are dataset-dependent. Therefore, to validate the generalizability of our approach and the potentiality of our proposed pipeline, more datasets (single-cell RNA-seq, multi-omics, methylation data) should be tested.

5. Conclusions

We consider our results a warning for future UMAP implementations applied upstream of other analyses, such as clustering. For the case study of tumor stratification, we showed that, with HDBSCAN, keeping default UMAP parameters, i.e., Euclidean output metric, would have impoverished the downstream results. Furthermore, we observed high variability in estimating the number of clusters and in clustering performances. For datasets with well-separated groups, it might not be necessary to worry about the UMAP reproducibility issue and investigate various topological UMAP embeddings. However, for more complex data such as RNA-seq, different geometries of output spaces should be explored if one cannot be assumed a priori.

UMAP has mostly been implemented with a Euclidean metric. We hope our work will encourage further research on non-Euclidean embeddings for the analysis of clustering or tumor stratification, even with other metric spaces to investigate further samples’ relationships. Future works should validate different UMAP output metrics on several datasets, analogous to Yang et al. [13] who evaluated the importance of the UMAP input metric. Moreover, as suggested by [44], supervised or semi-supervised dimensionality reduction can provide more informative latent space representations. Therefore, future efforts can integrate our pipeline with a supervised version of UMAP. Finally, even multi-omics data should be exploited [45] within our approach to enhance medical and biological interpretation of the final clustering results.