A Geometric Perspective on Functional Outlier Detection

Abstract

We consider functional outlier detection from a geometric perspective, specifically: for functional datasets drawn from a functional manifold, which is defined by the data’s modes of variation in shape, translation, and phase. Based on this manifold, we developed a conceptualization of functional outlier detection that is more widely applicable and realistic than previously proposed taxonomies. Our theoretical and experimental analyses demonstrated several important advantages of this perspective: it considerably improves theoretical understanding and allows describing and analyzing complex functional outlier scenarios consistently and in full generality, by differentiating between structurally anomalous outlier data that are off-manifold and distributionally outlying data that are on-manifold, but at its margins. This improves the practical feasibility of functional outlier detection: we show that simple manifold-learning methods can be used to reliably infer and visualize the geometric structure of functional datasets. We also show that standard outlier-detection methods requiring tabular data inputs can be applied to functional data very successfully by simply using their vector-valued representations learned from manifold learning methods as the input features. Our experiments on synthetic and real datasets demonstrated that this approach leads to outlier detection performances at least on par with existing functional-data-specific methods in a large variety of settings, without the highly specialized, complex methodology and narrow domain of application these methods often entail.

1. Introduction

1.1. Problem Setting and Proposal

Outlier detection for functional data is a challenging problem due to the complex and information-rich units of observations, which can be “outlying” or unusual in many different ways. Functional outliers are often categorized into magnitude and shape outliers  [1,2], whereas Hubert et al. [3] differentiated between isolated and persistent outliers, the latter were further subdivided into shift, amplitude, and shape outliers. However, neither of these taxonomies yield precise, explicit, fully general definitions, which makes it difficult to theoretically describe, analyze, and compare functional outliers. Magnitude outliers, for example, have been defined as functional observations “outlying in some part or across the whole design domain” [1] (p. 1), or as “curves lying outside the range of the vast majority of the data” [2] (p. 2), whereas Hubert et al. [3] (p. 3) defined isolated outliers as observations that “exhibit outlying behavior during a very short time interval”, in contrast to persistent outliers, which “are outlying on a large part of the domain”.

To cut through the confusion, we propose a geometric perspective on functional outlier detection based on the well-known “manifold hypothesis” [4,5]. This refers to the assumption that ostensibly complex, high-dimensional data lie on a much simpler, lower-dimensional manifold embedded in the observation space and that this manifold’s structure can be learned and then represented in a low-dimensional space, often simply called embedding space. We argue that such a perspective both clarifies and generalizes the concept of functional outliers, without the need for any strong assumptions or prior knowledge about the underlying data-generating process or its outliers. In terms of theoretical development, the approach allows us to consistently formalize and systematically analyze functional outlier detection in full generality. We also demonstrate that procedures based on this perspective simplify and improve functional outlier detection in practice: this suggests a principled, yet flexible approach for applying well-established, highly performant standard outlier-detection methods such as local outlier factors (LOF) [6] to functional data, based on embedding coordinates obtained via manifold learning or dimension-reduction methods. Our experiments show that doing so performs at least on par with existing functional-data-specific outlier-detection methods, without the methodological complexity and limited applicability that methods specific to functional data often entail. Moreover, such lower-dimensional representations serve as an easily accessible visualization and exploration tool that helps uncover complex and subtle data structures that cannot be sufficiently reflected by one-dimensional outlier scores or labels, nor captured by many of the previously proposed 2D diagnostic visualizations for functional outliers.

1.2. Background and Related Work

Functional data analysis (FDA) [7] focuses on data where the units of observation are realizations of stochastic processes over compact domains. In many cases, the intrinsic dimensionality of functional data (FD) is much lower than the observed. First, while FD are infinite-dimensional in theory, they are high-dimensional in practice: functional observations are usually recorded on fine and dense grids of argument values. Second, the dominant drivers of the differences among functional observations are often comparatively low-dimensional, so that just a few modes of variation capture most of the structured variability in the data.

However, FD usually contain shape and translation, as well as phase variation, i.e., both “vertical” and “horizontal” variability. These different kinds of variability contribute to the difficulty of precisely defining and differentiating the various forms of functional outliers and developing methods that can “catch them all”, making outlier detection a highly investigated research topic in FDA. For example, Arribas-Gil and Romo [2] argued that the proposed outlier taxonomy of Hubert et al. [3] can be made more precise in terms of expectation functions $f\left(t\right)$ and $g\left(t\right)$, with $f\left(t\right)$ a “common” process; see Figure 1.

Despite these attempts, some fundamental issues remain unsolved. The proposed taxonomies do not provide precise definitions, and some of the definitions are contradictory to some extent. Finally, many outlier scenarios for realistic data-generating processes are not covered by the described taxonomies at all. As Arribas-Gil and Romo [2] themselves pointed out that settings with phase-varying data (i.e., “horizontal” variability through elastic deformations of the functions’ domains) are not sufficiently reflected, as functions deviating in terms of phase may be considered as shape outliers in cases where there are only a few such functions, but not in settings where all functions display such variation.

In addition, the taxonomy in Figure 1 provides a reasonable conceptual framework only if the nonoutlying data from the “common” data-generating process is characterized adequately just by its global mean function. This cannot be assumed for many real datasets, which often contain highly variable sets of functions, which display several modes of phase, shape, and/or amplitude variation simultaneously and/or come from multiple classes with class-specific means and higher moments (see Figure 5).

Published research focuses mostly on the development of outlier detection methods specifically for functional data, and a multitude of methods based on a variety of different concepts such as functional data depths [8,9], functional PCA [10], functional isolation forests [11], robust functional archetypoids [12], or functional outlier metrics such as directional outlyingness [13,14], often narrowly focused on detecting specific kinds of functional outliers, have been put forth. Dai et al. [1] proposed a transformation-based approach to functional outlier detection and claimed that sequentially transforming shape outliers, which “are much more challenging to handle”, into magnitude outliers makes them easier to detect with established methods [1] (p. 2). The approach allows defining functional outliers more precisely in terms of the transformations being used, such as normalizing or centering functions or taking their derivatives, but practitioners still need to be able to come up with appropriate transformations for the data at hand first.

Recently, Xie et al. [15] introduced a decomposition of functional observations into amplitude, phase, and shift components, based on which specific types of outliers can be identified in a more general geometric framework without necessarily requiring functional data to be of comparatively low rank. Similar in spirit to our proposal, Hyndman and Shang [16] used kernel density estimation and half-space depth contours of two-dimensional robustified FPCA scores to construct functional boxplot equivalents and detect outliers, and Ali et al. [17] used data representations in two dimensions obtained from manifold methods for outlier detection and clustering, but the focus of both was on practicalities without considering the theoretical implications and general applicability of embedding-based approaches, nor did they consider the necessity of higher-dimensional representations. While Hyndman and Shang’s HDR boxplots were based on a similar combination of methods as our approach, they did not consider their geometrical foundations and, thus, did not make use of their full potential, firstly by considering only the two largest PCs and secondly by dichotomizing observations into outliers and inliers instead of providing continuous scores of outlyingness. Yu et al. [18] developed a test statistic for outlier detection based on the observed maxima of scaled PC score vectors, i.e., outlyingness defined in terms of a single mode of variation. However, this NHST framework for outlier detection needs to assume both that the common data have a single consistent mean function and that all deviations from this mean function are i.i.d. realizations of a mean-zero Gaussian process. Both of these assumptions seem highly restrictive to us and are likely to be untenable in many real-world applications.

The remainder of the paper is structured as follows: We provide a theoretical formalization and discussion of our geometric approach in Section 2. Based on these theoretical considerations, Section 3 presents extensive experiments. Section 3.1 covers a detailed qualitative analysis of real-world data, while Section 3.2 provides quantitative experiments and systematic comparisons to previously proposed methods on complex synthetic outlier scenarios. We conclude with a discussion in Section 4.

2. Functional Outlier Detection as a Manifold-Learning Problem

In this section, we first define two forms of functional outliers from a geometric view point: off- and on-manifold outliers. We then illustrate how this perspective contains and extends existing outlier taxonomies and how it can be used to formalize a large variety of additional scenarios for functional data with outliers.

2.1. The Two Notions of Functional Outliers: Off- and On-Manifold

Our approach to functional outlier detection rests on the manifold assumption, i.e., the assumption that observed high-dimensional data are intrinsically low-dimensional. Specifically, we put forth that observed functional data $x\left(t\right)\in \mathcal{F}$, where $\mathcal{F}$ is a function space, arise as the result of a mapping $\varphi :\Theta \to \mathcal{F}$ from a (low-dimensional) parameter space $\Theta \subset {\mathbb{R}}^{d_2}$ to $\mathcal{F}$, i.e., $x\left(t\right)=\varphi \left(\theta \right)$. Conceptually, a $d_2$-dimensional parameter vector $\theta \in \Theta$ represents a specific combination of values for the modes of variation in the observed functional data, such as level or phase shifts, amplitude variability, class labels, and so on. These parameter vectors are drawn from a probability distribution P over ${\mathbb{R}}^{d_2}$: ${\theta }_i\sim P\forall {\theta }_i\in \Theta ,$ with $\Theta =\{\theta :f_P\left(\theta \right)>0\}$ and $f_P$ the density to P. Mapping this parameter space to the function space creates a functional manifold ${\mathcal{M}}_{\Theta ,\varphi }$ defined by $\varphi$ and $\Theta$: ${\mathcal{M}}_{\Theta ,\varphi }=\{x\left(t\right):x\left(t\right)=\varphi \left(\theta \right)\in \mathcal{F},\theta \in \Theta \}\subset \mathcal{F}$, and an example is depicted in Figure 2. For $\mathcal{F}=L^2$ with data from a single functional manifold that is isomorphic to some Euclidean subspace, Chen and Müller [19] developed the notions of a manifold mean and modes of variation. Similarly, Dimelgio et al. [20] developed a robust algorithm for template curve estimation for connected smooth submanifolds of ${\mathbb{R}}^d$.

Unlike these single-manifold settings, our conceptualization of outlier detection is based on two functional manifolds. That is, we assume a dataset $X=\{x_1\left(t\right),\cdots ,x_n\left(t\right)\}$ with n functional observations coming from two separate functional manifolds ${\mathcal{M}}_c={\mathcal{M}}_{{\Theta }_c,{\varphi }_c}$ and ${\mathcal{M}}_a={\mathcal{M}}_{{\Theta }_a,{\varphi }_a}$, with ${\mathcal{M}}_j\subset \mathcal{F}$, $j\in \{c,a\}$ and $X\subset \{{\mathcal{M}}_c\cup {\mathcal{M}}_a\}$, with ${\mathcal{M}}_c$ representing the “common” data-generating process and ${\mathcal{M}}_a$ containing anomalous data. Moreover, for the purpose of outlier detection and in contrast to the settings with a single manifold described in the referenced literature, we are less concerned with precisely approximating the intrinsic geometry of each manifold. Instead, it is crucial to consider the manifolds ${\mathcal{M}}_c$ and ${\mathcal{M}}_a$ as submanifolds of $\mathcal{F}$, since we require not just a notion of distance between objects on a single manifold, but also a notion of distance between objects on different manifolds using the metric in $\mathcal{F}$. Note that function spaces such as $\mathcal{C}$ or $L^2$, which are commonly assumed in FDA [22], are naturally endowed with such a metric structure. Both $\mathcal{C}\left(D\right)$ and all $L^p\left(D\right)$ spaces over compact domain D are Banach spaces for $p\ge 1$ and, thus, also metric spaces [23].

Finally, we assume that we can learn from the data an embedding function $e:\mathcal{F}\to \mathcal{Y}$ that maps observed functions to a $d_1$-dimensional vector representation $y\in \mathcal{Y}\subset {\mathbb{R}}^{d_1}$ with $e\left(x\right(t\left)\right)=y$, which preserves at least the topological structure of $\mathcal{F}$, i.e., if ${\mathcal{M}}_c$ and ${\mathcal{M}}_a$ are unconnected components of $\mathcal{F}$, their images under e are also unconnected in $\mathcal{Y}$ and ideally yield a close approximation of the ambient geometry of $\mathcal{F}$.

Definition 1.

Off- and on-manifold outliers in functional data.

Without loss of generality, let $r=\frac{\left|\right\{x_i\left(t\right):x_i\left(t\right)\in {\mathcal{M}}_a\left\}\right|}{\left|\right\{x_i\left(t\right):x_i\left(t\right)\in {\mathcal{M}}_c\left\}\right|}⋘1$ be the outlier ratio, i.e., most observations are assumed to stem from ${\mathcal{M}}_c$. Furthermore, let ${\Theta }_c$ and ${\Theta }_a$ follow the distributions $P_c$ and $P_a$, respectively. Let ${\Omega }_{\alpha ,P}^{*}$ be an $\alpha$-minimum volume set of P for some $\alpha \in (0,1)$, where ${\Omega }_{\alpha ,P}^{*}$ is defined as a set minimizing the quantile function $V\left(\alpha \right)={inf}_{C\in \mathcal{C}}\{\mathrm{Leb}\left(C\right):P\left(C\right)\ge \alpha \},0<\alpha <1$} for i.i.d. random variables in ${\mathbb{R}}^d$ with distribution P, $\mathcal{C}$ a class of measurable subsets in ${\mathbb{R}}^d$, and Lebesgue measure Leb [24], i.e., ${\Omega }_{\alpha ,P}^{*}$ is the smallest region containing a probability mass of at least $\alpha$.

A functional observation $x_i\left(t\right)\in X$ is then:

• An off-manifold outlier if $x_i\left(t\right)\in {\mathcal{M}}_a$ and $x_i\left(t\right)\notin {\mathcal{M}}_c$;
• An on-manifold outlier if $x_i\left(t\right)\in {\mathcal{M}}_c$ and ${\theta }_i\notin {\Omega }_{\alpha ,P_c}^{*}$.

To paraphrase, we assume that there is a single “common” process generating the bulk of observations on ${\mathcal{M}}_c$ and an “anomalous” process defining structurally different observations on ${\mathcal{M}}_a$. We follow the standard notion of outlier detection in this, which assumes that there are two data-generating processes [1,25,26]. Note that this does not necessarily imply that off-manifold outliers are similar to each other in any way: $P_a$ could be very widely dispersed and/or ${\mathcal{M}}_a$ could consist of multiple unconnected components representing different kinds of anomalous data. The essential assumption here is that the process from which most of the observations are generated yields structurally relatively similar data. This is reflected by the notion of the two manifolds ${\mathcal{M}}_c$ and ${\mathcal{M}}_a$ and the ratio r. We consider settings with $r\in [0,0.1]$ as suitable for outlier detection. By definition, the number of on-manifold outliers, i.e., distributional outliers on ${\mathcal{M}}_c$ as opposed to the structural outliers on ${\mathcal{M}}_a$, only depends on the $\alpha$-level for ${\Omega }_{\alpha ,P_c}^{*}$.

Note that outlyingness in functional data is often defined only in terms of shape or magnitude, but the concept ought to be conceived much more generally. The most important aspect from a practical perspective is that any kind of structural difference will be reliably reflected in low-dimensional representations that can be learned via manifold methods, as we show in Section 3. These methods yield embedding coordinates $y\in \mathcal{Y}$ that capture the structure of data and their outliers.

2.2. Methods

To illustrate some of the implications of our general perspective on functional outlier detection and showcase its practical utility, we mostly use metric multidimensional scaling (MDS) [27] for dimension reduction and local outlier factors (LOF) [6] for outlier scoring in the following. Note, however, that the proposed approach is not at all limited to these specific methods, and many other combinations of outlier detection methods applied to lower-dimensional embeddings from manifold-learning methods are possible. However, MDS and LOF have some important favorable properties: First of all, both methods are well understood and widely used and tend to work reliably without extensive tuning since they do not have many hyperparameters. Specifically, LOF only requires a single parameter minPts, which specifies the number of nearest neighbors used to define the local neighborhoods of the observations, and MDS only requires specification of the embedding dimension.

More importantly, our geometric approach rests on the assumption that functional outlier detection can be based on some notion of distance or dissimilarity between functional observations, i.e., that abnormal or outlying observations are separated from the bulk of the data in some ambient (function) space. As MDS optimizes for an embedding, which preserves all pairwise distances as closely as possible (i.e., tries to project the data isometrically), it also retains a notion of the distance between unconnected manifolds in the ambient space. This property of the embedding coordinates retaining the ambient space geometry as much as possible is crucial for outlier detection. This also suggests that manifold-learning methods such as ISOMAP [28], t-SNE [29], or UMAP [30], which do not optimize for the preservation of ambient space geometry via isometric embeddings by default, may require much more careful tuning in order to be used in this way. Our experiments support this theoretical consideration, as can be see in Figure 11. For LOF, this implies that larger values for minPts are to be preferred here, since such LOF scores take into account more of the global ambient space geometry of the data instead of only the local neighborhood structure. In Section 3, we show that minPts $=0.75n$, with n the number of functional observations in a dataset, seems to be a reliable and useful default for the range of datasets we consider.

Two additional aspects need to be pointed out here. First, throughout this paper, we compute most distances using the $L_2$ metric. This yields MDS coordinates that are equivalent to standard functional PCA scores (up to rotation). The proposed approach, however, is not restricted to $L_2$ distances. Combining MDS with distances other than $L_2$ yields embedding solutions that are no longer equivalent to PCA scores, and suitable alternative distance measures may yield better results in particular settings. We illustrate this aspect using the $L_{10}$ metric and two phase-specific distance measures in Section 3.3, which we apply to simulated data with isolated outliers and a real dataset of outlines of neolithic arrowheads, respectively. Similarly, using alternative manifold-learning methods could be beneficial in specific settings, as long as they are able to represent not just the local neighborhood structure or on-manifold geometry, but also the global ambient space geometry.

Second, even though the LOF could also be applied directly to the dissimilarity matrix of a functional dataset without an intermediate embedding step, most anomaly-scoring methods cannot be applied directly to such distance matrices and require tabular data inputs. By using embeddings that accurately reflect the (outlier) structure of a functional dataset, any anomaly-scoring method requiring tabular data inputs can be applied to functional data as well. In this work, we apply LOF on MDS coordinates to evaluate whether functional data embeddings can faithfully retain the outlier structure. Furthermore, embedding the data before running outlier-detection methods often provides large additional value in terms of visualization and exploration, as the ECG data analysis in Section 3.1 shows.

2.3. Examples of Functional Outlier Scenarios

We can now give precise formalizations of different functional outlier scenarios and investigate the corresponding low-dimensional representations. In this section, we first show that the geometrical approach is able to describe existing taxonomies (see Figure 1) more consistently and precisely. We then illustrate its ability to formalize a much broader general class of outlier detection scenarios and discuss the choice of the distance metric and the dimensionality of the embedding.

2.3.1. Outlier Scenarios Based on Existing Taxonomies

Structure induced by shape: In the taxonomy depicted in Figure 1, top, the common data-generating process is defined by the expectation function $f\left(t\right)$. This can be formalized in our geometrical terms as follows: the set of functions defined by the “common process” $f\left(t\right)$ defines a functional manifold (in terms of shape), i.e., the structural component is represented by the expectation function of the common process. That means we can define ${\mathcal{M}}_c=\{x\left(t\right):x\left(t\right)=\theta f\left(t\right)=\varphi (\theta ,t)\}$ or ${\mathcal{M}}_c=\{x\left(t\right):x\left(t\right)=f\left(t\right)+\theta =\varphi (\theta ,t),\theta \in \mathbb{R}\}$. More generally, we can also model this jointly with ${\mathcal{M}}_c=\{x\left(t\right):{\theta }_{1}f\left(t\right)+{\theta }_2=\varphi (\theta ,t),\theta ={({\theta }_1,{\theta }_2)}^{\prime }\in {\mathbb{R}}^2\}$. In each case, magnitude and (vertical) shift outliers as defined in the taxonomy correspond to on-manifold outliers in the geometrical approach, as such observations are elements of ${\mathcal{M}}_c$. Isolated and shape outliers, on the other hand, are by definition off-manifold outliers, as long as “g is not related to f” is specified as $g\ne \theta f\forall \theta \in \mathbb{R}$. For example, if we define ${\mathcal{M}}_a=\{x\left(t\right):x\left(t\right)=\theta g\left(t\right)\}$, it follows that ${\mathcal{M}}_c\cap {\mathcal{M}}_a=\varnothing$. The same applies to isolated outliers, because $g\left(t\right)=f\left(t\right)+I_U\left(t\right)h\left(t\right)\ne {\theta }_{1}f\left(t\right)+{\theta }_2$.

Figure 3 shows an example of such an outlier scenario taken from [8]. Following their notation, the two manifolds can be defined as ${\mathcal{M}}_c=\left\{x\left(t\right)\right|x\left(t\right)=b+0.05t+cos\left(20\pi t\right),b\in \mathbb{R}\}$ and ${\mathcal{M}}_a=\left\{x\left(t\right)\right|x\left(t\right)=a+0.05t+sin\left(\pi t^2\right),a\in \mathbb{R}\}$ with $t\in [0,1]$ and $a\sim N(\mu =5,\sigma =4)$, $b\sim N(\mu =5,\sigma =3)$. Note that the off-manifold outliers lie within the mass of data in the visual representation of the curves, whereas in the low-dimensional embedding, they are clearly separable.

However, we argue that the way shape outliers are defined in Figure 1 is too restrictive, as many isolated outliers clearly differ in shape from the main data, but are not captured by the given definition if the shape is considered in terms of “g not related to f”. In contrast, the geometrical perspective with its concepts of off- and on-manifold outliers reflects that consistently. Another issue with the considered taxonomy concerns horizontal shift outliers $f(t+\alpha )$ or $f\left(h\right(t\left)\right)$. Aribas-Gil and Romo [2] specifically tackled that aspect in their discussion. They distinguished between situations where “all the curves present horizontal variation” (Case I), which is the no-outlier scenario for them, and situations where only a few phase-varying observations are present (Case II), which constitutes an outlier scenario. Again, the geometric perspective allows reflecting that consistently. In Appendix A, we make these two notions explicit by defining manifolds accordingly.

2.3.2. General Functional Outlier Scenarios

As already noted, the concept of structural difference we propose is much more general. It is straightforward to conceptualize other outlier scenarios with an induced structure beyond shape. Consider the following theoretical example: take a parameter manifold $\Theta \subset [0,\infty ]\times [0,\infty ]\times [0,\infty ]\times [0,\infty ]$ and an induced functional manifold $\mathcal{M}=\{f\left(t\right);t\in [0,1]:f\left(t\right)={\theta }_1+{\theta }_2{t}^{{\theta }_3}+I(t\in [{\theta }_4\pm 0.1])\}$. Each dimension of the parameter space controls a different characteristic of the functional manifold: ${\theta }_1$ the level, ${\theta }_2$ the magnitude, ${\theta }_3$ the shape, and ${\theta }_4$ the presence of an isolated peak around $t={\theta }_4$. One can now define a “common” data-generating process, i.e., a manifold ${\mathcal{M}}_c$, by holding some of the dimensions of $\Theta$ fixed and only varying the rest, either independently or not. On the other hand, one can define an “anomalous” data-generating process, i.e., a structurally different manifold ${\mathcal{M}}_a$, by letting those fixed in ${\mathcal{M}}_c$ vary, or simply setting them to values unequal to those used for ${\mathcal{M}}_c$, or by using different dependencies between parameters than for ${\mathcal{M}}_c$, e.g., if ${\theta }_1={\theta }_2$ for ${\mathcal{M}}_c$, let ${\theta }_1=-{\theta }_2$ for ${\mathcal{M}}_a$. This implies that one can define data-generating processes so that any functional characteristic (level, magnitude, shape, “peaks”, and their combinations) can be on-manifold or off-manifold outliers, depending on how the “common” data manifold ${\mathcal{M}}_c$ is defined.

Figure 4 shows a setting in which ${\mathcal{M}}_c$ is defined purely in terms of complex shape variation, while ${\mathcal{M}}_a$ contains vertically shifted versions of elements in ${\mathcal{M}}_c$: Let ${\mathcal{M}}_c$ be the functional manifold of Beta densities $f_{\mathrm{B}}(t;{\theta }_1,{\theta }_2)$ with shape parameters ${\theta }_1,{\theta }_2\in [1,2]$, and let ${\mathcal{M}}_a$ be the functional manifold of Beta densities with shape parameters ${\theta }_1,{\theta }_2\in [1,2]$ shifted vertically by some scalar quantity ${\theta }_3\in [0,0.5]$, that is ${\mathcal{M}}_c=\{f\left(t\right);t\in [0,1]:f\left(t\right)=f_{\mathrm{B}}(t;{\theta }_1,{\theta }_2)\}$ with ${\Theta }_c={[1,2]}^2$ and ${\mathcal{M}}_a=\{f\left(t\right);t\in [0,1]:f\left(t\right)=f_{\mathrm{B}}(t;{\theta }_1,{\theta }_2)+{\theta }_3\}$ with ${\Theta }_a={\Theta }_c\times [0,0.5]$.

As can be seen in Figure 4, both manifolds contain substantial shape variation that is identically structured, but those from ${\mathcal{M}}_a$ are also shifted upwards by small amounts. Note that many shifted observations lie within the main bulk of the data on large parts of the domain. In the 2D embeddings based on unnormalized $L_1$-Wasserstein distances [31] (also know as the “Earth mover’s distance”, top right) and 3D embeddings based on standard $L_2$ distances (bottom right), we see that this structure is captured with high accuracy, even though it is hardly visible in the functional data, with most anomalous observations clearly separated from the common manifold data, whose embeddings are concentrated on a narrow subregion of the embedding space. An observation on ${\mathcal{M}}_a$ that is very close to ${\mathcal{M}}_c$, lying well within the main bulk of functional observations, also appears very close to ${\mathcal{M}}_c$ in both embeddings. This example shows that the two functional manifolds do not need to be completely disjoint, nor yield visually distinct observations for our approach to yield useful results. It also shows that the choice of an appropriate dissimilarity metric for the data can make a difference: a 2D embedding is sufficient for the more suitable Wasserstein distance, which is designed for (unnormalized) densities (top right panel), while a 3D embedding is necessary for representing the relevant aspects of the data geometry if the embedding is based on the standard $L_2$ metric (lower right panels). For a comparison with currently available outlier visualization methods for this example, see Figure A4 in Appendix D.

In summary, we propose that the manifold perspective allows defining and representing a very broad range of functional outlier scenarios and data-generating processes. We argue that these properties make the geometrical approach very compelling for functional data, because it is flexible, conceptualizes outliers on a much more general level (for example, structural differences not in terms of shape) than before, and allows theoretically assessing a given setting.

Beyond its theoretical utility of providing a general notion of functional outliers, it has crucial practical implications: outlier characteristics of functional data, in particular structural differences, can be represented and analyzed using low-dimensional representations provided by manifold-learning methods, regardless of which functional properties define the “common” data manifold and which properties are expressed in structurally different observations. From a practical perspective, on-manifold outliers will appear “connected”, whereas off-manifold outliers will appear “separated” in the embedding, and the clearer these structural differences are, the clearer the separation in the embedding will be. Note that this implies that shape outliers, which pose particular challenges to many previously proposed methods, will often be particularly easily detectable. Moreover, all methods for outlier detection that have been developed for tabular data inputs can be (indirectly) applied to functional data as well based on this framework, simply by using the embedding coordinates as feature inputs: The embedding space $\mathcal{Y}$ is typically a low-dimensional Euclidean space in which conventional outlier detection works well and the essential geometrical structure encoded in the pairwise functional distance matrix is conserved in these lower-dimensional embeddings. In the next section, we illustrate this practical utility in detail by extensive quantitative and qualitative analyses.

3. Experiments

To illustrate the practical relevance of the outlined geometrical approach, we first qualitatively investigate real datasets. In the second part of this section, we quantitatively investigate the anomaly detection performance of several detection methods based on synthetic data.

3.1. Qualitative Analysis of Real Data

We start with an in-depth analysis of the ECG200 data [32,33], a functional dataset with a complex structure: it seems to contain subgroups with phase and amplitude variation and different mean functions. As a result, the dataset appears visually complex (Figure 5, left). Without the color coding, it would be challenging to identify the three subgroups (as in the lower left plot in Figure 6). Moreover, there are five left-shifted observations (apparent at $t\in [10,25]$) and a single (partly) vertically shifted outlier (apparent at $t\in [50,75]$), clearly detectable by the naked eye.

Much of the general structure (and the anomaly structure in particular) becomes evident in a 5D MDS embedding. To begin with, in the first two embedding dimensions, depicted on the right-hand side of Figure 5, three subgroups are easily recognizable. The color coding in Figure 5 is based on this visualization. It makes apparent that the substructures correspond to two smaller, horizontally shifted subgroups of curves (orange: left-shifted, purple: right-shifted) and a central subgroup encompassing the majority of the observations (green). In addition, we computed LOF scores on the 5D embedding coordinates. The observations with LOF scores in the top decile are shown in black in Figure 5. This set contains all the clearly outlying observations.

More importantly, note that these observations are clearly separated from the rest in the 5D embedding shown in Figure 6: the five clearly left-shifted observations in the fourth embedding dimension and the single vertically shifted observation in the subspace spanned by the first and third embedding dimension. The figure shows a scatterplot matrix of all five embedding dimensions with observations color-coded according to the 5D-embedding LOF scores. The clearly left-shifted outliers obtain the highest LOF scores due their isolation in the subspaces including the fourth embedding dimension. Note, moreover, that other observations with higher LOF scores appear in peripheral regions of the different subspaces, but they are not as clearly separable as the six observations described before. Regarding Figure 7A, which shows the 20 most outlying curves according to LOF scores, this can be explained by the fact that these other observations stem from one of the two shifted subgroups and can thus be seen as on-manifold outliers, whereas the six other, visually clearly outlying observation, are clearly off-manifold outliers.

We contrast these findings with the results of directional outlyingness [14,34], which performs very well (see Section 3.2) on simple synthetic datasets. Figure 7 shows the ECG curves color-coded by the variation of directional outlyingness (B), the 20 most outlying curves by the variation of directional outlyingness (C), and the observations labeled as outliers by directional outlyingness respectively by the MS-plot (D). First of all, it can be seen that many observations yield a high variation of directional outlyingness, and observations in the right-shifted subgroup obtain most of the highest values. In fact, among the twenty observations with the highest variation of directional outlyingness, only one is from the left-shifted group, and thirteen are from the right-shifted group. Moreover, applying directional outlyingness to this dataset results in 72 observations being labeled as outliers, which is about 36% of all observations. We would argue that it is questionable whether 36% of all observations should be labeled as outliers.

In this regard, the ECG data serve as an example that illustrates the advantages of the geometric approach. First of all, it yields readily available visualizations, which reveal much more of the inherent structure of a dataset than just its anomaly structure. This is specifically important for data with a complex structure (i.e., subgroups or multiple modes and large variability). Moreover, it allows applying well-established and powerful outlier scoring methods such as LOF to functional data. This exemplifies that the approach not only improves theoretical understanding and consideration as outlined in the previous section, it also has large practical utility in complex real data settings in which previously proposed methods may not provide useful answers.

In the ECG example, we saw that a 5D embedding yielded reasonable results and sufficiently reflected many aspects of the data. In particular, the extremely left-shifted observations became clearly separable in the fourth embedding dimension. In Appendix E, we analyze a synthetic dataset in the same way as the ECG data, which yields similar findings. Moreover, note that the Spearman rank correlation between LOF scores computed on the 5D embedding and LOF scores computed directly on the ECG data distances is 0.99. This shows that the outlier structure retained in the 5D embedding is highly consistent with the outlier structure in the high-dimensional observation space, an important aspect with respect to anomaly-scoring methods requiring (low-dimensional) tabular inputs.

Finally, note that even fewer than five embedding dimensions may suffice to reflect much of the inherent structure. Consider the examples depicted in Figure 8, which shows the functional observations and the first two embedding dimensions of a corresponding 5D MDS embedding of another four real datasets. The Octane data consist of spectra from 60 gasoline samples [35,36], the Spanish weather data of annual temperature curves of 73 weather stations [37], the Tecator data of spectrometric curves of meat samples [37,38], and the Wine data of spectrometric curves of wine samples [32,39]. As before, the observations are colored according to LOF scores based on the 5D embedding. In addition, the 12 observations with highest LOF scores are depicted as triangles. These datasets are much simpler than the ECG data, and the first two embedding dimensions already reflect the (outlier) structure fairly accurately: observations with high LOF scores appear separated in the first two embedding dimensions, and more general substructures are revealed as well. The substructure of the weather data is rather obvious already regarding the functional observations, for example, the observations with less variability in terms of temperature, all of which obtained high LOF scores. The substructure of the wine data—for example, the small cluster in the lower part of the embedding—is much harder to detect based on visualizations of the curves alone.

Appendix B summarizes a more detailed analysis of the sensitivity of the approach to the choice of the dimensionality of the embedding. We conclude that sensitivity seems to be fairly low. For all five real datasets we considered, the rank order of LOF scores is very similar or even identical whether based on two-, five-, or even twenty-dimensional embeddings (cf.

Table A1. Spearman correlation between LOF scores based on embeddings of different dimensionality for the 5 considered real datasets and metrics $L_{0.5}$, $L_1$, …, $L_{10}$, and unnormalized $L_1$-Wasserstein. MDS embeddings with 5 dimensions are compared to embeddings with 2 (2 vs. 5) and 20 (5 vs. 20) dimensions.

	${\mathit{L}}_{0.5}$		${\mathit{L}}_1$		${\mathit{L}}_2$		${\mathit{L}}_3$		${\mathit{L}}_4$		${\mathit{L}}_5$
	2 vs. 5	5 vs. 20	2 vs. 5	5 vs. 20	2 vs. 5	5 vs. 20	2 vs. 5	5 vs. 20	2 vs. 5	5 vs. 20	2 vs. 5	5 vs. 20
ECG	0.96	0.97	0.98	0.97	0.97	0.99	0.94	0.99	0.94	0.98	0.90	0.97
Octane	0.94	0.99	0.96	0.98	0.97	0.99	0.98	0.99	0.98	0.99	0.96	0.98
Weather	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Tecator	0.97	0.99	0.96	0.99	0.99	1.00	0.99	1.00	0.99	1.00	1.00	1.00
Wine	0.98	0.99	0.99	1.00	1.00	1.00	1.00	1.00	0.99	1.00	0.99	1.00
	${\mathit{L}}_{\mathbf{6}}$		${\mathit{L}}_{\mathbf{7}}$		${\mathit{L}}_{\mathbf{8}}$		${\mathit{L}}_{\mathbf{9}}$		${\mathit{L}}_{\mathbf{10}}$		Wasserstein
	2 vs. 5	5 vs. 20	2 vs. 5	5 vs. 20	2 vs. 5	5 vs. 20	2 vs. 5	5 vs. 20	2 vs. 5	5 vs. 20	2 vs. 5	5 vs. 20
ECG	0.89	0.96	0.87	0.96	0.86	0.95	0.86	0.95	0.85	0.94	0.98	0.97
Octane	0.96	0.98	0.95	0.99	0.96	0.98	0.94	0.97	0.94	0.97	0.95	0.96
Weather	1.00	1.00	0.99	1.00	0.99	1.00	0.99	1.00	1.00	1.00	1.00	1.00
Tecator	1.00	1.00	1.00	1.00	0.99	1.00	0.99	1.00	0.99	1.00	0.96	0.99
Wine	0.99	1.00	0.98	1.00	0.98	1.00	0.98	0.99	0.98	0.99	0.99	1.00

).

Following Mead [40], we quantified the goodness of fit (GOF) for a $d_1$-dimensional MDS embedding as: $GOF\left(d_1\right)=\frac{{\sum }_{i=1}^{d_1}max(0,{\lambda }_i)}{{\sum }_{j=1}^{n}max(0,{\lambda }_j)},$ where ${\lambda }_k$ are the eigenvalues (sorted in decreasing order) of the kth eigenvectors of the centered distance matrix. For all of the considered real datasets, a 5D embedding achieved a goodness of fit over 0.8, the four less-complex examples even over 0.95 (see Figure A2). As a rule of thumb, the embedding dimension does not seem crucial as long as the goodness of fit (GOF) of the embedding is over 0.8 for $L_2$ distances. This rule of thumb also yielded compelling quantitative performance results, as shown in Section 3.2.

Figure 6 and Figure 8 show visualizations that combine MDS embeddings with LOF outlier scores. To put them into context, we compare them to existing visualization techniques in this section. For the sake of clarity, only the results are summarized here. The figures for the various alternative methods can be found in Appendix D. Figure A5 shows the results for the MBD-MEI “Outliergram” by Aribas-Gil and Romo [41] (implementation: [42]) for shape outlier detection and the magnitude–shape plot method of Dai and Genton [34]. Figure A6 and Figure A7 show the results for the translation–phase–amplitude boxplots by Xie et al. [15] and the elastic depth boxplot for shape outlier detection by Harris et al. [9]. Finally, Figure A8, Figure A9, Figure A10, Figure A11, Figure A12 and Figure A13 show the corresponding functional and bivariate HDR boxplots by Hyndman and Shang [16] (implementation: [43]). Considering the MBD-MEI outliergram and the magnitude–shape plots, both of these visualization methods mostly fail to identify shift outliers (by design, in the case of the outliergram). The outliergram tends to mislabel very central observations as outliers in datasets with little shape variability (e.g., the supposed “shape outliers” detected by MBD-MEI in the central region of the Tecator data) and fails to detect even egregious shape outliers in datasets with high variability (e.g., not a single MBD-MEI outlier in ECG 200), as well as shape outliers that are also outlying in their level (e.g., the three shape outliers identified by msplot in the upper region of the Tecator data). Note that some central functions of the Spanish weather data, which are labeled as outliers by the magnitude–shape plot (and partly by the outliergram) are also reflected in the 2D embedding in Figure 8.

They are fairly numerous relative to the overall sample size and are very similar to each other. As such, they form a clearly defined separate cluster within the data, which can be seen in the middle bottom part of the embedding. The translation–phase–amplitude boxplots mostly fail to detect outliers in data with high variability: no outliers at all are detected for the Spanish weather data despite their visually apparent anomalies, and only a single translation outlier is detected for the ECG data. Moreover, the implementation of the approach seems to break down for data with very little variation, and it was not possible to compute the phase boxplot for the Wine data, a dataset with almost no variability in terms of phase.

The results of the elastic depth boxplots do not seem to be consistent over all considered datasets. The results appear reasonable for the Octane, the Wine, and, in part, for the ECG data, where both amplitude and phase outliers are detected. However, in the ECG data, mostly observations from the left-shifted subgroup are detected as phase outliers and only two from the right-shifted subgroup. The results for the Spanish weather and the Tecator data are even less convincing. Among the Tecator data, the method labels 41 curves, i.e., 19% of all observations, as outliers, while it does not discover a single outlier in the Spanish weather data. Note, however, that the elastic depth boxplots are more robust than the translation–phase–amplitude boxplots. While the latter method only detected a single translation outlier and was not able to compute the phase boxplot for the Wine data at all, the elastic depth boxplots detect several amplitude outliers and simply do not yield phase outliers.

Finally, HDR boxplots based on PC projections of the data yield mostly similar results as the $L_2$-distance-based MDS embeddings. However, we would argue that dichotomizing the observations into inliers and outliers by a fixed outlier threshold makes the visualizations much less suited as an exploratory tool. Consider, for example, the Spanish weather data. The small cluster of observations with rather constant temperature (∼17–25${}^{\circ }$) does not fall into the outlier region according to the dichotomization threshold, and so, they are also not shown individually in the functional HDR boxplots. Whether they are considered to be outliers or rather a subgroup surely depends on the observer, but we would argue that an outlier visualization method should emphasize and not hide such structures. Our approach of colors according to continuous scores does that very well, reflecting at the same time both the general and the outlier structure. More importantly, the outlier structure of the ECG dataset is not captured in the embedding used by the HDR boxplots. As outlined, more than two embedding dimensions are necessary to fully reflect the outlier structure of this dataset, and the density estimators underlying the HDR boxplot will break down fairly rapidly as the number of embedding dimensions increases. As such, the available implementation is limited to only using the first two PC scores for the embedding, regardless of the actual rank of the underlying data.

3.2. Quantitative Analysis of Synthetic Data

In this section, we investigate the outlier detection performance quantitatively, based on synthetic datasets for which the true (outlier) structure is known.

3.2.1. Methods

In addition to applying LOF to 5D embeddings and directly to the functional data, we investigate the performance of four “functional data”-specific outlier-detection methods: directional outlyingness (DO) [14,34], total variational depth (TV) [44], elastic depth (ED_amp, ED_pha) [9], and the approach based on translation, phase, and amplitude boxplots (AP_BOX) presented by Xie et al. [15]. For the first two methods, we use implementations provided by the package fdaoutlier [45] and use the variation of directional outlyingness as returned by the function dir_out as outlier scores for DO and the total variation depths as returned by the function total_variation_depth for TV. For the latter two methods, we use implementations provided by Harris et al. [9]. Outlier scores for these methods are based on elastic depths as computed by the function depth.R1 from the package elasticdepth [9] and time-warped functions as computed by the function time_warping from the package fdasrvf [46]. Note that the elastic depth approach does not produce a single outlier score per observation, but scores amplitude and phase outliers separately. Both amplitude (ED_amp) and phase (ED_pha) scores are shown in Figure 9.

3.2.2. Data-Generating Processes

The methods are applied to data from four different data-generating processes (DGPs), the first two of which are based on the simulation models introduced by Ojo et al. [25] and provided in the corresponding R package fdaoutlier [45]. We also provide the results of additional experiments based on the original DGPs from the package fdaoutlier in Appendix C. However, we consider most of these DGPs as too simple for a realistic assessment, as most methods achieve almost perfect performance on them, and we use more complex DGPs here. In both DGPs 1 and 2, the inliers from simulation_model1 from the package fdaoutlier serve as ${\mathcal{M}}_c$, i.e., the common data-generating process. This results in simple functional observations with a positive linear trend. In addition, simulation_model1 generates simple shift outliers. Additionally, our DGP 1 also includes shape outliers stemming from simulation_model8, which serves as ${\mathcal{M}}_a$. In contrast, DGP 2 contains shape outliers from all of the other DGPs in fdaoutlier, which means ${\mathcal{M}}_a$ contains observations from several different data-generating processes.

For DGPs 3 and 4, we define ${\mathcal{M}}_c$ by generating a random, wiggly template function over $[0,1]$ for each dataset, generated from a B-spline basis with 15 or 25 basis functions, respectively, with i.i.d. $\mathcal{N}(0,1)$ spline coefficients. Functions in ${\mathcal{M}}_c$ are generated as elastically deformed versions of this template, with random warping functions drawn from the ECDFs of $\mathrm{Beta}(a,b)$ distributions with $a,b\sim U[4,6]$ (DGP 3) or $a,b\sim U[3,8]$ (DGP 4). Functions in ${\mathcal{M}}_a$ are also generated as elastically deformed versions of this template, with Beta ECDF random warping functions with $a,b\sim U[3,4]$ for DGP 3 and with 50:50 Beta mixture ECDF random warping functions with $a,b\sim 0.5U[3,8]:0.5U[0.1,3]$ (DGP 4). Finally, white noise with $\sigma =0.1,0.15$, respectively, for DGPs 3 and 4 is added to all resulting functions. Appendix F shows visualizations of example datasets drawn from these DGPs.

3.2.3. Performance Assessment

From these four DGPs, we sampled data $B=500$ times with three different outlier ratios $r\in \{0.1,0.05,0.01\}$. Based on the outlier scores, we computed the area under the ROC curve (AUC) and Mathew’s correlation coefficient (MCC) as the performance measures and report the results over all 500 replications. Note that, for $r\in \{0.1,0.05\}$, the number of sampled observations was $n=100$, whereas for $r=0.01$, we sampled $n=1000$ observations. Since computing the elastic depths and time-warped functions requires more than an hour for a single dataset with 1000 observations, we only included them for the settings with 100 observations.

3.2.4. Results

We note that LOF applied directly to functional data distances yielded very similar results as LOF applied to their 5D embeddings. This agrees with our findings in the qualitative analyses. In the following, we simply refer to the geometrical approach and do not distinguish between the LOF based on MDS embeddings and the LOF applied directly to the functional distance matrix. Figure 9 shows that the proposed geometrical approach is highly competitive with existing functional-data-specific outlier-detection methods. It yields better results than TV for all of the four DGPs and performs at least on par with DO. In comparison to DO, it performs better on DGP 1 and DGP 3, on par on DGP 4, and worse on DGP 2. Note that DO struggles to detect simple shift outliers: among these methods, it performs worst on the first DGP. Similar conclusions can be reported for other settings, where it performs even worse if there are only shift outliers (cf. Figure A3 and Figure A15). Moreover, while the approaches based on elastic depth proposed by Harris et al. (ED_amp and ED_pha) and the approach proposed by Xie et al. (AP_BOX) perform well on DGP 2, they are outperformed by DO in this setting, and on DGPs 1, 3, and 4, they clearly perform the worst. Thus, these two methods yield the worst performances overall.

Note that the insights we gain on synthetic data are confirmed by all of the real data applications we investigate in Section 3.1. In addition to the experiments conducted here, we applied the considered methods and their accompanying visualization approaches to these five real datasets. The results of the previously proposed visualizations are presented in detail in Appendix D, Figure A5, Figure A6 and Figure A7. In contrast to the proposed geometrical approach, none of them yields satisfactory results consistently for all of the considered datasets. For example, the outliergram, as well as the approach based on translation, phase and amplitude boxplots and the elastic depth approach fail to identify any outliers in some of these datasets, while the magnitude–shape plot, for example, labels an entire third of all observations in the ECG data as outliers (as already outlined in Section 3.1).

In summary, based on the conducted experiments, the proposed geometrical approach yields very compelling results: On synthetic data, it leads to outlier scoring performances at least on par with specialized functional-outlier-detection methods even in its simplest version (MDS with $L_2$ distances and LOF). Moreover, in contrast to the other methods, it yields consistently useful and sensible results on all of the considered real datasets, while providing more intuitive and more easily interpretable visualizations. Going further, our approach can be adapted to specific settings simply by choosing metrics other than $L_2$. As the next section shows, this can improve the outlier-detection performance considerably.

3.3. General Dissimilarity Measures and Manifold Methods

So far, we have computed MDS embeddings mostly based on $L_2$ distances. In the following, we show that the approach is more general. The geometric structure of a dataset is captured in the matrix of pairwise distances among observations. Different metrics emphasize different aspects of differences in the data and can thus lead to different geometries. MDS based on $L_2$ distances yielded compelling results in many of the examples considered above, but other distances are likely to lead to better performance in certain settings. To illustrate the effect, we consider two additional settings—one simulated and one on real data—in the following. The results are displayed in Figure 10.

The simulated setting is based on isolated outliers, i.e., observations that deviated from functions in ${\mathcal{M}}_c$ only on small parts of their domain. In such settings, higher-order $L_p$ metrics lead to better results, since such metrics amplify the contribution of small segments with large differences to the total distance. We use as an example data generated from simulation_model2 from the package fdaoutlier. Figure 10A shows the AUC values of LOF scores on MDS embeddings based on $L_2$ and $L_{10}$ distances. Again, 500 datasets were generated form the model over different outlier ratios. In contrast to $L_2$-based MDS, using $L_{10}$ distances yielded almost perfect detection. In embeddings based on $L_{10}$, isolated outliers are clearly separable in the first two or three embedding dimensions.

As a second example, we consider the ArrowHead dataset [47,48], which contains outlines of three different types of neolithic arrowheads (see Appendix G for visualizations of the dataset). Using the 78 structurally similar observations from class “Avonlea” as our data on ${\mathcal{M}}_c$ and sampling outliers from the 126 structurally similar observations from the other two classes, we can compute AUC values based on the given class labels. We generate 500 datasets for each outlier ratio $r\in \{0.05,0.1\}$. Since there are only 78 observations in the class “Avonlea”, we do not use $r=0.01$ for this example. Embeddings are computed using three different dissimilarity measures: the standard $L_2$ metric, the unnormalized $L_1$-Wasserstein metric [31], and the dynamic time warping (DTW) distance [49]. Note that the DTW distance does not define a proper metric [50].

Figure 10B shows that small performance improvements can be achieved in this case if one uses dissimilarity measures that are more appropriate for the comparison of shapes, but not as much as in the isolated outlier example. Note that even though the DTW distance is not a proper metric, it improves the outlier-scoring performance in this example. This indicates that, from a practical perspective, general dissimilarity measures can be sufficient for our approach to work. This opens up further possibilities, as there are many general dissimilarity measures for functional data, for example the semimetrics introduced by Fuchs et al. [51]. Overall, these examples illustrate the generality of the approach: using suitable dissimilarity measures can make the respective structural differences more easily distinguishable.

More complex embedding methods, on the other hand, do not necessarily lead to better or even comparable results as MDS. Figure 11 shows the distribution of the AUC for embedding methods ISOMAP and UMAP. Both methods require a parameter that controls the neighborhood size used to construct a nearest neighbor graph from which the manifold structure of the data is inferred. The larger this value, the more of the global structure is retained. For both methods, embeddings were computed for very small and very large neighborhood sizes of five and ninety.

The results show that neither method performs better than MDS; UMAP even performs considerably worse. Note that ISOMAP is equivalent to MDS based on the geodesic distances derived from the nearest neighbor graph, and the larger the neighborhood size the more similar to direct pairwise distances these geodesic distances become. This is also reflected in the results, as ISOMAP-90 performs better than ISOMAP-5 on average. For DGP-2, ISOMAP-90 slightly outperforms MDS, indicating that more complex manifold methods could improve the results somewhat in specific settings.

In general, however, these findings confirm the theoretical considerations sketched in Section 2.2. Embedding methods that preserve the geometry of the space $\mathcal{F}$ of which ${\mathcal{M}}_c$ and ${\mathcal{M}}_a$ are submanifolds, i.e., the ambient space geometry, are more suited for outlier detection than methods that focus on approximating the intrinsic geometry of the manifold(s). Thus, more sophisticated embedding methods, which often focus on approximating the intrinsic geometry, should not be applied lightly and certainly require careful parameter selection in order to be applicable for outlier detection. Since hyperparameter tuning for unsupervised methods remains an unsolved problem, this is unlikely to be achieved in real-world applications. In particular, consider that both UMAP and t-SNE [29] have been found to be—in general—oblivious to local density, which means that clusters of different density in the observation space tend to become clusters of more equal density in the embedding space [52]. Although there may exist a parameter setting where this effect is reduced (note that there are now density-preserving versions of t-SNE and and UMAP [52]), we are skeptical that outliers can be faithfully represented in such an embedding given the difficulties of hyperparameter tuning in unsupervised settings. Moreover, these methods are not designed to preserve important aspects of the outlier structure. For example, UMAP is subject to a local connectivity constraint, which ensures that every observation is at least connected to its nearest neighbor (in more technical terms: that a vertex in the fuzzy graph approximating the manifold is connected by at least one edge with an edge weight equal to one [30]), which makes it unlikely that UMAP can be tuned so that it is able to sensibly embed off-manifold outliers, which should, by definition, not be connected to the common data manifold. The poor performance of UMAP embeddings in our experiments confirms these concerns.

4. Discussion

Based on a geometrical perspective of functional outlier detection, we defined two general types of functional outliers: off- and on-manifold outliers. Our investigation showed that this perspective clarifies the theoretical concepts and improves practical results. From a theoretical perspective, it allows formalizing functional outlier scenarios in precise and consistent terms, beyond differences in terms of either shape, level, or magnitude. This simplifies reasoning about specific outlier settings and provides a fully general theoretical conceptualization of the problem.

From an applied perspective, we formulated two important consequences. First of all, as was demonstrated with a comprehensive analysis of a complex, real dataset of ECG curves, the geometrical approach allows for easily accessible and highly informative visualizations. These are obtained by means of low-dimensional embeddings reflecting the inherent structure of a functional dataset in much detail. Such visualizations provide more accurate and complete pictures of the (outlier) structure of functional data. In particular, off-manifold outliers reliably appear as clearly separated (groups of) points in the low-dimensional embeddings.

Second, the proposed approach makes it possible to apply highly developed and performant standard outlier-detection methods to functional data, since the geometric structure of the data is captured and reflected in their pairwise distance matrices. Outlier detection and scoring methods that can be applied to distance matrices can therefore directly be used for functional data as well. Furthermore, detection methods requiring tabular inputs can also be applied simply by using the embedding coordinates obtained with embedding methods as proxy data for the original functions. Our experiments using LOF scores showed that the two approaches yielded very similar results. This simultaneously simplifies and improves functional outlier detection: It simplifies since functional data analysis becomes more accessible to a broader audience with general outlier-detection methods that are widely used in other areas and that do not require an understanding of complex methodological details of functional data methods. It improves the state-of-the-art since many functional outlier methods can only detect specific kinds of functional outliers by design or fail in more complex realistic data that are widely dispersed or that contain multiple nonoutlying subgroups, such as the ECG data. Moreover, note that our proposal is not limited to univariate functional data. Extending it to multivariate functions is completely straightforward, as long as a suitable dissimilarity measure is available to compute pairwise distances.

In this paper, most embeddings were obtained using MDS based on $L_2$ distances. This implies a close similarity to functional bagplots and highest-density region (HDR) boxplots [16], which are based on the first two robust principal component scores. However, this similarity only applies if our geometrical approach is implemented with 2D MDS embeddings based on $L_2$ distances. As outlined, our proposal is neither limited to the $L_2$ metric as a distance measure nor to MDS as an embedding method or just two embedding dimensions. Other metrics and (higher-dimensional) embedding methods can be used as well, and our results indicate that an alternative distance measure can further improve the performance in specific settings, sometimes considerably. In particular, even nonmetric dissimilarity measures may be applicable as our results based on DTW distances indicate. On the other hand, the results also show that more sophisticated embedding methods such as ISOMAP and UMAP cannot be used as straightforwardly as MDS. Such methods, which do not take into account the ambient space geometry by default, at least require very careful parameter selection.

In terms of practical applicability, the $O\left(n^3\right)$ time complexity and $O\left(n^2\right)$ storage complexity of standard MDS may prove problematic for large data, but generalizations such as Landmark MDS [53], Pivot MDS [54], or multilevel MDS exploiting GPU performance [55] scale much better with the number of available observations.

Finally, we would argue that existing functional outlier detection approaches mostly lack the principled geometrical underpinning and conceptualization presented here. As outlined, we argue that such a conceptualization is necessary to make functional outlier detection tractable in full generality. Specifically, consider that existing methods typically limit themselves to creating a 1D or 2D representation of each curve (e.g., MBD-MEI, MO-VO, functional bagplots, HDR plots), often based on preconceived notions of the characteristics of functional outliers. Our investigations and experiments suggested that this is often not sufficient for real-world functional outlier detection: there is no valid reason to limit our representations to two dimensions with modern outlier-detection methods, and the geometrical perspective often strongly suggests otherwise in the case of complex functional data. Even more importantly, it is much more flexible to learn maximally informative low-dimensional representations directly from data instead of starting with rigid notions of which characteristics to look at and to ignore the rest. The latter is likely to lead to results not capturing the entire (outlier) structure of a given dataset, which is essential in real-world unsupervised settings and exploratory analyses.

Based on the theoretical considerations and the empirical results outlined above, we conclude that the proposed approach is well suited for both the theoretical conceptualization and the practical implementation of functional outlier detection. In particular, the choice of embedding method should consider whether it is able to preserve the extrinsic geometry of the function space, and simple MDS embeddings based on functional distances provide a very strong baseline for that. On the basis of this work, we intend to further investigate the implications of the geometrical perspective, such as the effects other dissimilarity measures, embedding, and outlier-detection methods, in future research. We are also investigating the use of mass volume curves [56] for hyperparameter tuning in functional outlier detection. Such a criterion will permit analysts to optimize the combination of the functional distance metric, embedding dimensionality, and outlier-scoring method parameters. In the absence of quantitative criteria for optimizing these settings, our recommendations are to (1) use the standard $L_2$ metric as the default, which proved to be a very strong baseline in our experiments for a wide variety of data settings and outlier types, (2) make use of substantive knowledge about the data at hand, either from an initial exploratory data analysis or expertise about the data-generating process, in order to choose metrics that are sensitive to the relevant kinds of structural deviations, and (3) supplement and verify the results with results based on alternative metrics, since our proposal has a low computational cost for typical functional dataset sizes.