On the nature and types of anomalies: a review of deviations in data

This section defines the employed concepts to ensure that the reader understands the terms as intended, regardless of his or her discipline (senior scholars may choose to only do a quick scan). An anomaly, in its broadest meaning, is something that is different or peculiar given what is usual or expected [88‐90]. In the philosophy of science, anomalies play a crucial role as observations or predictions that are inconsistent with the models in the prevailing academic paradigm [91‐94]. Such anomalies require an explanation and consequently initiate the advancement of knowledge by the refinement of current theories. Over time, anomalies that constitute fundamental novelties may accumulate and trigger an academic crisis in which the old paradigm is replaced by a wholly different one. Newtonian physics, for example, was succeeded by Einstein’s theory of general relativity, which was better capable of predicting and explaining a variety of observed astronomical phenomena, such as anomalies pertaining to the perihelion of Mercury. In statistics, data mining and AI an anomalous occurrence deviates from some notion of normality for the given data and setting. Deviants that can be detected in an unsupervised fashion, which are the focus of this study, can be defined more precisely. An anomaly in this context is a case, or a group of cases, that in some way is unusual and does not fit the general patterns exhibited by the majority of the data [3, 4, 8, 10, 11, 69, 325, 326]. The detection of anomalies is a highly relevant task, not only because they should be handled appropriately during inferential research, but also because the goal of analyses is often to discover interesting new phenomena [9, 37‐39, 95‐98]. The remainder of this section will focus on terms and concepts pertaining to anomalies in data.

The term cases refers to the individual instances in a dataset, also called data points, rows, records, or observations [57, 99, 323]. These cases are described by one or more attributes, also referred to as variables, columns, fields, dimensions or features. Some of these attributes will be required for data management and context, such as identification (ID) and time variables. In addition, the dataset will contain substantive attributes, i.e., the meaningful domain-specific variables of interest, such as income and temperature. Measuring and recording the actual attribute values is prone to errors, the discovery of which may indeed be one of the reasons to conduct anomaly detection. The term occurrence is used here in a broad fashion and may refer to an individual case or a group of cases, an object or an event, and anomalous or regular data.

The term dependency is used in the literature to refer to two aspects of relationships, both of which are relevant for this study. First, there can be a dependency between the attributes, meaning there is a relationship between the variables [59, 96, 99‐101, 182]. Income, for example, may be correlated with education and parental financial status. A second form of dependency, referred to as dependent data, deals with the relationship between the dataset’s individual cases or rows [7, 20, 57, 102, 323]. A set with such dependent cases contains an intrinsic relation between the observations. Examples are time series, spatial and graph data, and sets with hierarchical relationships. The dependencies in such datasets are typically captured by time, location, linking or grouping attributes. These inter-case relations are absent from independent data, such as in i.i.d. random samples for cross-sectional surveys, in which every row represents a stand-alone observation.

Describing and understanding the different types of anomalies in a concrete and data-centric manner is not feasible without referring to the functional data structures that host them. This section therefore shortly discusses several important formats for organizing and storing data [cf. 5, 57, 95, 106, 110‐115, 184]. Some analyses are conducted on unstructured and semi-structured text documents. However, most datasets have an explicitly structured format. Cross-sectional data consist of observations on unit instances—e.g., individual people, organizations or countries—at one point in time. The cases in such a set are generally considered to be unordered and otherwise independent, as opposed to the following structures with dependent data. Time series data consist of observations on one unit instance (e.g., one country) at different points in time. Time-oriented panel data, or longitudinal data, consist of a set of time series and are therefore comprised of observations on multiple individual entities at different points in time (e.g., income history for a sample of citizens followed over a five-year period). The general term sequence data will be used to refer to time series, time-oriented panel data, as well as to sets with an ordering not based on time. Sequence data have broad applications and, besides time-oriented phenomena, are able to capture genomic and other biological features, user actions, spectroscopy wavelengths, trajectories, audio, and even visual information such as the shape of physical objects and moving elements in a video [5, 10, 114, 116‐123, 278]. Each of the above data structures can be implemented with a single matrix or table, but when several inter-related entities need to be modeled a relational model is often used [95, 124, 125]. This allows many functional structures, including domain-specific designs and analytical star schemas [95, 103, 104]. A graph is a related data structure and typically consists of vertices (nodes), edges (connections or links), edge directions and edge weights [20, 57, 95, 106, 112, 113]. An attributed graph has, in addition to these structural properties, any number of substantive domain-specific variables. Such structures are highly relevant for modeling, e.g., social networks, chemical compounds, Internet data, and wireless sensor networks. Graphs can take many forms, including tree data structures. A tree consists of a root, a given amount of parent and child nodes, and does not feature any closed paths (so-called cycles). Storing graphs typically involves both a set of vertices (e.g., as a list of nodes and their properties) and a set of edges (e.g., as an adjacency matrix with relations, directions and weights). This is similar to spatial data, which usually consist of a set of coordinates and a set of substantive features [8, 66, 95, 126‐129, 277, 323]. The latter may pertain to, e.g., population density, settlement type or availability of utility infrastructure elements. The representations often constitute points (atomic positions such as addresses), lines, arcs (e.g., roads or rivers) and polygons (regions such as neighborhoods or states). However, a rasterization of continuous data can also be used, such as satellite imagery and brain scans represented as a grid of pixels. This format therefore also captures image material in general, with the data points being 2D pixels or 3D voxels. The coordinates represent a position on a canvas or frame, while the features store the visual information as gray intensities or color information (e.g., RGB, multi- or hyperspectral). Spatio-temporal data feature a sequence dimension in addition to the coordinates and may capture, e.g., video and historical geographical information [281, 283]. Even more detailed distinctions can be made, but the key formats described above suffice to properly discuss the wide variety of anomaly types.

The concept of an aggregate is often used in the context of dealing with noise or obtaining a more abstract representation at the level of interest. When aggregating the cases the analyst is able to treat multiple individual rows as a whole or a group and consequently obtain summary statistics—e.g., means and totals—or other derived properties of the collective [5, 57, 95, 103‐106, 282, 323]. This allows the data and perspective to be transformed, for example, from days to months or from individual people to households. In terms of data structures, typical examples of aggregates are subgraphs, subsequences, and regions in spatial data. Often, however, the AD analysis will simply focus on the individual cases, i.e., on the atomic level of the set’s microdata. One can also aggregate the attributes in order to reduce dimensionality or to obtain for individual or grouped cases meaningful and complex latent variables or manifolds [3, 96, 107‐109]. However, this may mainly be relevant in the context of high-level semantic anomalies, in which such aggregates are generally difficult to determine without any prior theory, rules or supervised training (see Sect. 3.2 for more information) [13, 310].

To conclude this section it is valuable to briefly discuss what constitutes a typology. To theoretically distinguish between concepts, scholars have various intellectual tools at their disposal, amongst which are taxonomies, classifications, dendrograms, and typologies [130]. These all make use of one or more classificatory principles (explicit dimensions) to differentiate between the relevant elements. A classification uses a single principle, whereas a typology uses two or more simultaneously. A typology is therefore well suited to theoretically distinguish between complex concept types—offering not only a fundamental and summarized description of a general concept, but also an exhaustive and mutually exclusive overview of its distinct but related types. The term classification will be used more loosely in this study and will also refer to conceptualizations with classes that are not based on clear principles and that are neither mutually exclusive nor jointly exhaustive.

The literature acknowledges various ways to distinguish between different manifestations of anomalies. Barnett and Lewis [2, cf. 31, 131] make a distinction between extreme but genuine members of the main population, i.e., random fluctuations at the tails of the focal distribution, and contaminants, which are observations from a different distribution.

Wainer [34] differentiates between distant outliers, which exhibit extreme values and are clearly in error, and fringeliers, which are unusual but with their position about three standard deviations from the majority of the data cannot be said to be extremely rare and unequivocally erroneous. Essentially the same distinction is made in [132] with white crows and in-disguise anomalies, respectively. Relatedly, in [5, 133] a distinction is made between a weak outlier (noise) and a strong outlier (a significant deviation from normal behavior). The latter category can be sub-divided in events, i.e., unusual changes in the real-world state, and measurement errors, such as a faulty sensor [134, 135]. An overall classification is presented in [96], with the classes of anomalies indicating the underlying reasons for their deviant nature: a procedural error (e.g., a coding mistake), an extraordinary event (such as a hurricane), an extraordinary observation (unexplained deviation), and a unique value combination (which has normal values for its individual attributes). Other sources refer to similar explanations in a more free-format fashion [39, 97, 136]. In [184] a distinction is made between 9 types of anomalies. Another broad classification is that of [7], which differentiates between three general categories. A point anomaly refers to one or several individual cases that are deviant with respect to the rest of the data. A contextual anomaly appears normal at first, but is deviant when an explicitly selected context is taken into account [cf. 137]. An example is a temperature value that is only remarkably low in the context of the summer season. Finally, a collective anomaly refers to a collection of data points that belong together and, as a group, deviate from the rest of the data.

Several specific and concrete classifications are also known, especially those dedicated to sequence and graph analysis. Many of their anomaly types will be described in detail in Sect. 3. In time series analysis several within-sequence types are acknowledged, such as the additive outlier, temporary change, level shift and innovational outlier [138‐141, 191]. The taxonomy presented in [142] focuses on between-sequence anomalies in panel data and makes a distinction between isolated outliers, shift outliers, amplitude outliers, and shape outliers. Another specific classification is known from regression analysis, in which it is common to distinguish between outliers, high-leverage points and influential points [3, 143‐145]. Two anomalies with regard to the trajectories of moving entities are presented in [118, cf. 146, 147], namely the positional outlier, which is positioned in a low-density area of the trajectory space, and the angular outlier, which has a direction different from regular trajectories. The subfield of graph mining has also acknowledged several specific classes of anomalies, with anomalous vertices, edges, and subgraphs being the basic forms [18, 20, 112, 113, 148, 149]. Table 1 summarizes the anomaly classes acknowledged in the extant literature. In Sect. 3 these anomalies, particularly those that allow a data-centric definition, will be discussed in more detail and positioned within this study’s typology.

The classifications in Table 1 are either too general and abstract to provide a clear and concrete understanding of anomaly types, or feature well-defined types that are only relevant for a specific purpose (such as time series analysis, graph mining or regression modeling). The fifth column also makes clear that extant overviews hardly offer clear principles to systematically partition the classificatory space to obtain meaningful categories of anomalies. They thus do not constitute a classification or typology as defined by [130]. To the best of my knowledge this study’s framework and its predecessors offer the first overall typology of anomalies that presents a comprehensive overview of concrete anomaly types.

Many of the existing overviews also do not offer a data-centric conceptualization. Classifications often involve algorithm- or formula-dependent definitions of anomalies [cf. 8, 11, 17, 86, 150, 184], choices made by the data analyst regarding the contextuality of attributes [e.g., 7, 137], or assumptions, oracle knowledge, and references to unknown populations, distributions, errors and phenomena [e.g., 1, 2, 39, 96, 131, 136]. This does not mean these conceptualizations are not valuable. On the contrary, they often provide important insights as to the underlying reasons why anomalies exist and the options that a data analyst can exploit. However, this study exclusively uses the intrinsic properties of the data to define and distinguish between the different sorts of anomalies, because this yields a typology that is generally and objectively applicable. Referencing external and unknown phenomena in this context would be problematic because the true underlying causes usually cannot be ascertained, which means distinguishing between, e.g., extreme genuine observations and contaminants is difficult at best and subjective judgments necessarily play a major role [2, 4, 5, 34, 314, 323]. A data-centric typology also allows for an integrative and all-encompassing framework, as all anomalies are ultimately represented as part of a data structure. This study’s principled and data-oriented typology therefore offers an overview of anomaly types that not only is general and comprehensive, but also comes with tangible, meaningful and practically useful descriptions.

To end this section it is good to note that many valuable classifications of anomaly detection techniques are available [5, 7, 13, 14, 55, 84, 135, 150‐152, 299‐301, 318‐320, 330]. Because the core focus of the current study is on anomalies, detection techniques are only discussed if valuable in the context of the typification of data deviations. A review of AD techniques is therefore out of scope, but note that the many references direct the reader to information on this topic.

This section presents the five fundamental data-oriented dimensions employed to describe the types and subtypes of anomalies: data type, cardinality of relationship, anomaly level, data structure, and data distribution. The typology’s framework, as depicted in Fig. 2, comprises three main dimensions, namely data type, cardinality of relationship and anomaly level, each of which represents a classificatory principle that describes a key characteristic of the nature of data [57, 96, 101, 106]. Together these dimensions distinguish between nine basic anomaly types. The first dimension represents the types of data involved in describing the behavior of the occurrences. This pertains to these data types of the attributes responsible for the deviant character of a given anomaly type [10, 57, 96, 97, 114, 161]:

The second dimension is the cardinality of relationship, which represents how the various attributes relate to each other when describing anomalous behavior. These attributes are individually or jointly responsible for the deviant character of the occurrences [39, 59, 96, 100, 105, 106, 136, 158, 285]:

The cardinality of relationship essentially refers to whether one attribute is sufficient to define and detect the anomaly type or that multiple attributes need to be taken into account simultaneously. Note that, owing to the massive data collection nowadays, a dataset is likely to contain many attributes beyond the hosting subspace (i.e., the subset of attributes required to describe and detect a given anomaly). As a matter of fact, an occurrence can be deviant in one subspace and normal in others [133, 162‐164, 180, 182, 303]. An occurrence could even be one type of anomaly in one subspace and another type in a second subspace.

The third dimension is the anomaly level, which represents the distinction between atomic anomalies (individual low-level cases or data points) versus aggregate anomalies (groups or collective structures). In theory this is also an independent dimension, but in practice univariate data only contain atomic anomalies. Multivariate data may also host aggregate anomalies, which alongside substantive attributes typically require data management attributes (e.g., time stamps or group designations) that allow the formation of collective structures. However, should future research introduce noteworthy examples of aggregate anomalies in univariate data, the framework can be easily extended to accommodate this.

The fifth dimension is the data distribution, which refers to the collection of attribute values and their pattern or dispersion throughout the data space [98, 165]. An anomaly, per definition, is defined by its difference with regard to the remainder of the data, which makes the distribution of the dataset an important factor to take into account. The distribution is strongly dependent on the classificatory factors mentioned above, but allows focusing on density and other dispersion-related aspects of the set. It therefore offers additional descriptive and delineating capabilities.¹ This dimension will not only be used to subdivide between anomaly subtypes within the typology’s nine cells, but occasionally also to illustrate how an altered distribution would result in a different manifestation of a given anomaly.

The five classificatory principles of the typology are not only fundamental in the sense that they describe theoretically crucial properties of data, but also because they deeply impact analysis and storage solutions. Some examples of this: jointly analyzing qualitative and quantitative data requires specialized multivariate techniques; analyzing dependent data usually needs to account for autocorrelation; and locating clusters and other patterns in multidimensional data implies discovering inter-variable relationships and dealing with exponential scaling issues as datasets increase in size.

The preliminary typology presented in [69, 70, cf. 6] is summarized in the first row of Table 1 (it essentially lacked the lowest layer for aggregate anomalies present in Fig. 2). Although this version was able to implicitly accommodate complex anomalies, several discussions at conferences pointed to the fact that the types in the multivariate row were rather broad and demanded further subdivision and clarification. Atomic and aggregate anomalies are therefore acknowledged explicitly in the framework now, yielding nine basic anomaly types. In addition, some of the terminology is updated. The new typology framework is depicted in Fig. 2. Detailed subtypes are included in Fig. 3 and illustrated in the data plots throughout this article. The nine main types of anomalies, which follow naturally and objectively from the classificatory principles, are described in Sect. 3.2. (Note on visualization: in many of the diagrams the data points’ colors and shapes represent different categorical values; the reader might want to zoom in on a digital screen to see colors, shapes and patterns in detail.)

This section presents the anomaly types and their concrete subtypes. The typology’s rows represent three broad groups of anomaly types. Atomic univariate anomalies are single cases with a deviant value for one or possibly multiple individual attributes. They are relatively easy to describe and detect because the individual values of these observations are unusual. Relationships between attributes or cases are not relevant for such occurrences. An example is an extremely high numerical value, such as a person reported to be 246 cm high. There may be several anomalous atomic occurrences (albeit apparently not in a way so as to form a ‘normal pattern’), but the essence is that each individual case is anomalous in its own right. Moreover, should an individual case have multiple unusual values, then each of them will be anomalous (e.g., not only 246 cm high, but also 117 years old). Atomic multivariate anomalies are single cases whose deviant nature lies in their relationships, with the individual values not being anomalous. In independent data this will manifest itself in the unusual combination of a case’s own attribute values, such as a 10-year-old person with a body length of 180 cm. However, the multivariate nature also allows defining and detecting deviations in dependent data, i.e., in the relation with the other cases to which the given case is linked. An example is a time series temperature measurement that is unusually high for winter, but that would have been normal in summer. Atomic multivariate anomalies hide in multidimensionality, as they cannot be described and detected by simply analyzing the individual variables separately. Finally, aggregate anomalies are groups of cases that deviate as a collective, of which the constituent cases usually are not individually anomalous. Relationships between attributes and between cases play a key role here, not only to position an occurrence in the set with dependent data, but often also to form a pre-defined or derived group. Owing to their complex and intricate nature, these anomalies are generally the most difficult to describe and detect. A deviant subsequence is one manifestation of an aggregate anomaly, such as a whole winter with many unusually high temperatures compared to other winters. Although the above may be abstract at first reading, the detailed discussion below will offer a concrete understanding.

Before presenting the individual types and subtypes it is worthwhile to make some assumptions explicit. It is assumed that an atomic case (individual row) with p attributes represents a single data point in p-dimensional space (not a set of p distinct data points). A time series consists of multiple atomic data points, each of which has p attributes. The typology also assumes a parsimonious data structure, without redundant information. For example, the degree of a vertex, i.e., the number of edges that connect it to other vertices in the graph, is a latent characteristic of which the value is seen as being derived runtime during the analysis and is therefore assumed not to be explicitly included in the original dataset. When relevant the set does explicitly include management and dependency attributes, such as ID, sequence, group and link information, as these represent crucial structural properties (note that in the special case of text or symbolic data the sequence may be present implicitly). The typology also assumes the unaltered and original dataset in which one aims to declare anomalies. The reason for this is that, e.g., normalization [16, 86], dimensionality reduction [166], log transformations [167] and data type conversions [70] have all been shown to have significant impact on the presence and detection of anomalies. To be sure, transformations are allowed, but the typology then either assumes the newly derived dataset as the starting point for typification or remains agnostic as to any transformations performed as part of the AD algorithm. Finally, if one needs to choose between potential anomaly types, then the norm is to opt for the simplest type that captures the deviant occurrence (see the Discussion for more on this).

The types and subtypes are visualized schematically in Fig. 3 and discussed in detail in the remainder of this section. Since even the subtypes can be quite broad when multivariate in nature, ample examples are also provided.

In order to sharply determine the nature of a given data deviation, one should opt for the simplest applicable (sub)type. For example, a case that is a Type II anomaly, per definition, will also have unique class value combinations in a larger subspace that includes additional categorical attributes. However, this does not imply one should see this instance as a Type V anomaly. After all, the deviation can be defined more parsimoniously. To be sure, this does not exclude the possibility that the case in question is also a Type V anomaly, but that should then pertain to a different subset of attributes.

There are several general principles to determine which (sub)types are simpler than others. The univariate types are simpler than the multivariate types. Anomalies that require only qualitative or quantitative attributes are simpler than those defined in terms of mixed data. Subtypes based on independent data are simpler than those based on dependent data, and atomic anomalies are simpler than aggregate types.

Simple subtypes such as the extreme tail value and unusual class may be part of a larger set with dependent data, e.g., one that constitutes a graph. The fact that the graph structure need not be referenced in the definition of these anomalies obviously does not prohibit the researcher to meaningfully discuss the deviation in the context of the given graph. Often it makes perfect sense to discuss a Type II occurrence as an ‘anomalous node’, as long as one acknowledges that the graph structure itself is not required to define and detect the anomaly from a data perspective. The same holds for other complex data structures, such as spatial data and time series. This insight is the reason that the typology does not feature an anomaly subtype for a globally extreme high or low time series value. This is simply a Type I anomaly and, unless the position in the sequence should be identified, the time-related dependency with other cases in the dataset is irrelevant for defining and detecting the deviation.

This section concludes with a brief tightening of the terminology used in this article. The term anomaly and deviant are synonyms and used as general terms. The term outlier refers to numerically isolated occurrences, such as the classical Type I and Type IVa-d cases. The term novelty can also be defined more strictly, as referring to occurrences that in some way represent new and hitherto unknown events or objects.

In addition to offering an understanding of the nature and types of anomalies, the typology is aimed at facilitating the evaluation of AD algorithms. This is a relevant contribution because most research publications do not make it very clear which types can be identified by the anomaly detection algorithms studied, nor do they position the targeted anomalies in a broader context. However, given the wide variety of anomalies, it is clear that individual algorithms will be incapable of identifying all types [6, 17, 60, 82, 84‐86, 184]. Researchers can thus employ the predefined typology to provide a clear and objective insight into the functional capabilities of their AD algorithms by explicitly stating which anomaly (sub)types can be detected. Using the typology in this way also gives due acknowledgment of the no free lunch theorem [80‐83] and demand for transparent and explainable analytics and AI [71‐75] (also see next section).

However, the typology offers more opportunities for algorithm evaluation than merely clarifying which types and subtypes can be detected, since the typology is ideally also used for creating test sets. AD studies often evaluate algorithms by using existing AD benchmark datasets that are flawed [321]. Moreover, the common practice of treating (a sample of) a minority class as anomalies [9, 17, 86, 133, 159, 163, 182, 186, 195, 316] also poses problems. The cases of such a class may be very similar because they are part of a true pattern, and may even be very similar to other normal classes in the dataset. This latter situation can indeed be observed for several classes in the above-mentioned real-world Polis administration. In addition, there is no guarantee that all relevant anomaly subtypes will be present in such a test set. A better approach for creating AD test sets is therefore to use the typology as a basis for systematically creating and inserting instances of each relevant anomaly subtype in a real-world or simulated dataset. Such an injection approach ensures that the different anomalies are present and a thorough evaluation of the algorithm can subsequently be conducted. Researchers should aim to include the subtypes that, given the domain and typology dimensions, are relevant for the problem being studied.

Table 2 illustrates how the typology can facilitate the evaluation of several unsupervised algorithms for detecting anomalies in independent data. The focus is mainly on the algorithms, which are presented in the rows of the table, with the columns representing evaluation characteristics such as the capability to detect the individual anomaly types, metrics such as the AUC (area under the curve), F1 and sensitivity, as well as any remarks that may be relevant. Such an evaluation is appropriate when various (versions of) algorithms are evaluated and evaluation characteristics need to be presented at the level of the algorithm.

Grubbs’ parametric test aims at verifying whether a vector contains one or two outliers [30, 264], while the related GESD procedure is intended for testing whether a vector contains multiple outliers [35]. SECODA is a nonparametric AD algorithm for mixed data [6, 70]. Distance-based algorithms use a nearest neighbor approach to detect anomalies [51, 52, 54]. One could also include several versions of the algorithms or pre-processing steps. For example, with distance-based AD techniques the numerical attributes could be normalized and the categorical attributes transformed using dummy variables or inverse document frequency IDF (inverse document frequency), and such techniques could be evaluated in several combinations. Also, evaluation metrics based on the ROC (receiver operating characteristic) and confusion matrices could be calculated and reported for each anomaly (sub)type separately.

Table 3 presents the anomaly types in the rows, with the columns providing more details on different evaluation characteristics. This format was used in [70] for studying the impact of discretization and is appropriate if the focus is mainly on the anomaly (sub)types and few algorithms are being compared.

The typology provides a generic framework for understanding anomalies through a data lens and is relevant for both research and practice. Typifying an individual deviant case as a specific anomaly subtype means it is described meaningfully in terms of five fundamental dimensions. This provides transparency and explains the nature of the deviation, i.e., makes clear how it differs from the normal cases in terms of key data characteristics. This adds value to an anomaly analysis because AD algorithms yield an anomaly score or label, but, although they may be able to detect multiple subtypes, typically do not provide information on how the individual anomalies differ from the rest of the data. The typology provides a structured way to analyze this and clarify the detection results.

For example, a general-purpose AD analysis on independent data, using, e.g., distance or density based algorithms, may detect 12 subtypes, namely ST-Ia-b, IIa-b, IIIa-b, IVa-d, Va and VIa. These obviously represent a wide variety of deviations that may be present in the dataset. If the detection algorithm merely informs the analyst on whether (or to what degree) a case is anomalous, the results therefore remain a black box. The typology can be used to bring about the necessary clarification, with its 12 clearly distinguishable subtypes and names, as well as five dimensions that explain how each subtype differs from normal data. Figure 4 illustrates different anomalies in a real-life dataset. Some of these occurrences differ from regular data because they have an extreme numerical value (ST-Ia) or have numerical values that, although individually normal, in combination position them outside the regular space (ST-IVa). Some cases are deviant because they feature a rare code (ST-IIa), while others have a code that, although normal, is unusual in their own numerical neighborhood (ST-VIa). By employing the typology these different subtypes can be clearly distinguished, labeled and understood. The plot also makes clear that visualization can be valuable during this explanation process and can be of help in typifying the detected anomalies.

Understanding the nature of the anomaly types present in a set is also relevant because of their different implications. In the context of data management AD may be conducted as an exploratory data quality analysis, which can be used to decide what kind of automated checks and corrections should be implemented. In real-time monitoring processes ST-Ia cases may then be handled with a simple threshold, ST-Ib cases with an interval, ST-IVa cases with a rule that combines thresholds or intervals, and perhaps it is decided that ST-IVc cases should be ignored because they merely represent subtle deviations.

The typology also yields tangible clues for further interpretation and sense-making. For example, a limited number of dispersed ST-Ia or b anomalies are likely to be non-informative random events or glitches, while the aggregate ST-VIIa, b, i and j occurrences may very well imply a more significant event or mechanism, and ST-VIIc and e even come with a drift in the data that represents a fundamental distributional change.

There are several perspectives on the concept of locality, all of which can be meaningfully described in terms of the dimensions of anomalies that have been put forward in this study. The first perspective focuses on data structures with independent data and uses the cardinality of relationship to define the difference between local and global. Univariate anomalies are simply seen as global because they are unconditionally deviant relative to the remainder of the rows in the dataset [69, 70]. A single variable describes the entire (univariate) data space, which renders an unusual case in this view per definition a global anomaly. Therefore, when taking all the set’s cases into account, Type I anomalies will always have an extremely low, high, or otherwise unusual numerical value for the given attribute, without any condition and regardless of the other attributes. A similar argument can be given for Type II and III anomalies. The multivariate anomaly types, on the other hand, are only deviant given the categorical condition or the specific numerical area the case in question is located in. This is due to the fact that a multivariate anomaly is only unusual as a combination of values from multiple attributes and is therefore normal across the entire one-dimensional space. This is clearly illustrated with the mixed data types of the ST-VIa cases in the right plot of Fig. 6. These anomalies are normal with regard to the values of each individual numerical and categorical attribute. However, while, e.g., the blue points are normal in the global space, they are deviant in their own local numerical neighborhood. A local anomaly thus exists in some area, subsegment or class of the data [278]. Other examples are provided by [57] in a discussion on global versus local anomalies. A male with a body length of 175 cm is normal in the general population, but is an anomaly within the local class of professional basketball players. The reverse is also true: someone with a body length of 195 cm may be unusually tall with respect to the population, but not when only considering the class of professional basketball players.

A second perspective on the concept of locality focuses on data structures with independent data as well, but uses the data distribution to make the distinction between local and global. Local anomalies are described in terms of neighborhood density or similar characteristics, a perspective that is particularly relevant if the set contains multiple clusters that differ in this regard [8, 17, 53, 55, 273]. The outlyingness of a single case can be seen as being dependent upon the degree of isolation relative to its local neighborhood (rather than to the global space). Techniques for this setting, such as LOF and LOCI, therefore determine the density of a case in relation to the density of its neighboring points.

A third perspective on locality focuses on data structures with dependent data. In such datasets the individual cases are by their very nature intrinsically related, allowing the data management or contextual attributes used for linking the data points to naturally define neighborhoods [8, 126, 128, 222, 223]. Local anomalies in this context are typically deviations from autocorrelation patterns. They can perhaps most clearly be illustrated with spatial data, in which latitude–longitude attributes or other sorts of coordinates explicitly define locations and neighborhoods. An occurrence is locally anomalous if the values of one or more substantive attributes are unusual in its own spatial neighborhood, but normal in other regions. This logic also pertains to images and videos, with data points being pixels or voxels with a fixed position in the canvas or frame. An example of a local anomaly in an aerial photograph is an isolated tree in a certain region of the picture, while a large forest is present in another region [67]. The small blue region in Fig. 9c is an example as well. A similar reasoning holds for time series data, in which the timestamp explicitly positions cases at time points and in periods, which are the temporal equivalents of spatial locations and neighborhoods. A local anomaly then has a value that is normal when taking into account the entire history, but unusual in the period in which the occurrence lies. The ST-IVe subtype is such an anomaly and is illustrated in Fig. 8a. Local spatio-temporal anomalies can be viewed in the same vein. Finally, graphs also feature explicit structural positions and neighborhoods, although their visualization generally allows more freedom in where to graphically depict the nodes or cases.

The typology presented here offers an all-encompassing framework to describe the types of anomalies acknowledged in the literature, on the condition that they can be defined in terms of their data properties. For example, the typology can accommodate the well-known time series anomalies, i.e., the additive, temporary change, level shift, and innovational outlier [138, 140, 141]. In fact, the typology makes a more detailed distinction, because the classic additive outlier type does not distinguish between outliers that are globally extreme and outliers that are only deviant from a local perspective, e.g., in their climatic season [141]. This study’s typology identifies them as extreme tail values (ST-Ia) and local additive anomalies (ST-IVe), respectively. Another notable classification is that of [184], as it is both broad and data-oriented. However, it distinguishes between 9 concrete anomaly classes instead of 63 and does not use any classificatory principles.

The anomalies acknowledged in classifications that are not data-centric are not explicitly present in the current typology. For example, [96] presents classes of outliers that are not defined in terms of observed data characteristics, but instead refer to external causal phenomena that are often beyond the knowledge of the data analyst. This particularly holds for the procedural error (e.g., a data entry mistake), the extraordinary event (e.g., a hurricane), and the extraordinary observation (a non-explained measurement). Other classifications seemingly refer to the data, but are ultimately grounded in phenomena external to the dataset. Causes of outliers presented in [136] are, e.g., measurement errors and data from other distributions. To ascertain whether this is the case, one often requires additional information and subjective interpretation [2, 4, 34, 323]. This is by no means to say that these conceptualizations are not valuable, because analysts should certainly possess knowledge of the potential causes of anomalies. However, this study defines anomalies in terms of observable data characteristics and five theoretical dimensions. It allows the objective and principled declaration of anomaly (sub)types in a tangible and explainable fashion, using the data at hand and leaving relatively little room for discussion and doubt.

The classification in [7] distinguishes between point, contextual and collective anomalies. This differs from the three broad groups of this study’s typology (i.e., atomic univariate anomalies, atomic multivariate anomalies, and aggregate anomalies), which are rooted in the requirements a typology brings with it. The former classification features classes of anomalies that are not mutually exclusive, because a collective anomaly can also be a contextual anomaly. This is an undesirable property for any well-formed typology or classification, as the aim is to offer clear distinctions between concepts [130]. Strong classificatory principles and mutual exclusiveness were therefore demanded for the study at hand. The classification in [7] is also very general in nature, yielding rather abstract anomaly types. This is made clear by the fact that Type I to VI occurrences can all manifest themselves as a point anomaly, and a similar argument holds for contextual anomalies. In order to provide a concrete understanding, the current study offers not only a high-level framework hosting 3 broad groups and 9 basic types, but also a full typology with 63 tangible and detailed subtypes.

The concept of the anomaly refers to occurrences being both rare and different [9, 17, 319]. This can be seen as ultimately referring to anomalies being non-concentrated and positioned in the lowest density areas [13, 66, 311], with them being more extreme if they have more attributes that deviate and with values that deviate in a more severe fashion [297, 298]. However, it is worth discussing some nuances, mainly in the context of the typology’s data distribution dimension. For example, the global density anomaly (ST-IVd) shows that for a numerical dataset in which random noise is the norm, a small cluster actually forms the anomaly [184, 307]. Similarly, high-density regions may constitute the anomalies in dependent data (ST-VIk-l), such as when detecting geographical hot-spots of traffic accidents or disease outbreaks [66, 277]. Relatedly, categorical data may consist of nearly-unique vales, meaning that values that do repeat form the exception (ST-IIb) [181].

Anomalies can also be deviant on only one or two attributes and show unremarkable behavior on the other variables [57, 162, 163, 298]. A more complex manifestation is the high-density anomaly (HDA), which is an occurrence that deviates from the norm but in some subspace is located in a high-density region, and as such is positioned amongst or is a member of the most normal cases [297]. In cross-sectional (independent) data, subtypes IIa-b, Va, and VIa may turn out to be a HDA in its most basic form, but in principle all 63 subtypes can have high-density counterparts when taking into account an additional subspace. HDAs can be interpreted as deviant occurrences that hide in normality, and are especially relevant in misbehavior detection. Identifying them implies solving a delicate balancing problem, taking both anomalousness and normalness into account. The centrally positioned ST-IIa case in the left panel of Fig. 6 is a HDA, while the top one is positioned in a moderate or low density area.

The AD field features many overviews of anomalies that are specific to a given domain. An example is the classification of [265, cf. 254], which describes a collection of anomaly types in maritime traffic data that may point to erroneous or falsified messages. Examples of types are ‘too fast for the given vessel,’ ‘vessel type incompatible with size,’ ‘non-declared flag change’ and ‘outside of usual roads.’ Examples of anomaly types observed on stock markets are ‘forward rate bias,’ ‘the new December effect,’ ‘short-term price drift’ and ‘the weekend effect’ [266]. Examples from the domain of computer network traffic are ‘port scans,’ ‘denial of service attacks,’ ‘alpha flows,’ and ‘outage events’ [267]. Other domain-specific classifications describe land parcel anomalies [268], data center anomalies [269], biological malformations [270], and wireless sensor network threats [152, 271]. Some of these domain types can be detected by unsupervised techniques (and can be typified using this study’s typology). However, many types represent deviations from specific patterns and are only relevant for a given domain or problem and thus require supervised or rule-based detection methods.