Bounding open space risk with decoupling autoencoders in open set recognition

Deep neural networks (DNNs) have achieved state-of-the-art classification results in diverse domains such as natural language understanding [73], computer vision [27], and speech recognition [39]. Despite their success, DNNs are generally not well equipped for real-world applications [3] and tend to fail when exposed to data from unseen distributions.

This issue often goes unnoticed, as state-of-the-art classification results are primarily obtained in extremely controlled benchmark environments with an inherent closed world assumption, raising the question of applicability to real-world scenarios [28, 29, 37, 53]. Specifically, DNNs tend to generalize only well within the concepts they were trained on but tend to provide incorrect predictions with exaggerated confidence when exposed to samples from unseen distributions [3, 21, 28, 29, 37, 51], jeopardizing model robustness. As visualized in Fig. 1b, the multi-layer perceptron (MLP) learns to separate the two half-moons and XOR circles but fails to generalize to the uniform noise. Note that the MLP and MiMo [24] models were trained via empirical risk minimization (ERM) to separate the two classes, so there is no preference to which class the outliers should be attributed. However, we would expect a well-generalizing model to be uncertain about these samples and not assign them to one of the classes with high confidence.

In this work, we consider open set recognition (OSR) as the generalized version of one-vs-rest (OVR) classification [5], in which the rest samples not only stem from known classes but also from different unknown sources of outlier generating processes [6, 62]. Therefore, the aforementioned deficiencies of DNNs also pose a severe threat to OSR. As suggested by Scheirer et al. [62], these deficiencies can be alleviated by framing the optimization problem in OSR as a combination of ERM and open space risk minimization [62]. The open space is defined as the set of points that have at least distance d to any of the inlier samples, i.e., the closed space. The open space risk is then defined as the relative measure of the open space volume of false inliers over the closed space volume classified as inliers (i.e., true inliers), as illustrated in Fig. 2. Consequently, minimizing the open space risk forces the model to learn a hull around the inliers leading to increased model robustness.

The difference between ERM and ERM in conjunction with open space risk minimization is illustrated in Fig. 1. Due to its bounded open space risk DAE, learns a hull around the inlier data, whereas the MLP assigns an infinite space to the inlier class resulting in infinite open space risk. In practice, this problem is often mitigated by the long-established background class setup [41], in which all rest classes are subsumed within a single background class [19, 57]. This approach can be an effective measure to learn a working OSR model, as illustrated in Fig. 1h. On the downside, this approach is also highly data-intensive, rendering it often infeasible and error-prone while still having an unbounded open space risk.

In OSR, we usually deal with significant class imbalance. When the focus lies on filtering a single, narrow class of interest (COI), similar to the needle in the haystack problem, then this inlier class tends to be underrepresented. When the focus shifts toward outlier detection, for instance, in the case of computer virus detection, inlier samples outnumber instances of the rest class. Further, only some classes of the problem domain are usually known for the set of rest classes (RCs), and outliers or dataset shifts are only witnessed at test time, making OSR a highly imbalanced semi-supervised setting. Throughout the paper, we refer to COI samples as inliers, RC samples as rest samples, and samples of unseen rest classes as outliers.

To measure model robustness in isolation, we subdivide the OSR task into three disjunct subtasks by gradually increasing the scope of RC: (1) OVR classification task \(T_c\): The model is evaluated on the COI and the RCs it was trained on. (2) Contextual outlier detection task \(T_o\): Comprises the evaluation on COI and conceptually related RCs of \(T_c\). In practice, the rest samples stem from the same dataset, but from RCs the model was not trained on. Consistent with the literature, we define samples originating from RC in this case as contextual outliers, since they are possibly generated from a completely different underlying mechanism [1, 2, 25]. 3) Dataset shift task \(T_d\): In this case, rest samples stem from a new dataset, equivalent to the evaluation approach in [29]. By extending the scope of RC in tasks \(T_o\) and \(T_d\) to rest classes unseen during training, aspects of outlier detection and robustness to dataset shift become predominant. In fact, tasks \(T_o\) and \(T_d\) can be seen as semi-supervised one-class classification (OCC): In this semi-supervised setting, the model is only exposed to COI samples during training time and supposed to learn to reject samples that deviate from the COI representation, i.e., outliers [48, 60]. With our three-task experiment design, we are therefore able to bridge the gap between OVR and OCC within OSR and precisely pinpoint the classification and robustness performance for each algorithm.

Due to the widespread application of OSR in safety-critical environments such as medical diagnosis [9, 64, 69], fraud detection [13, 55, 76] and intrusion detection [18, 33], the extension of deep learning methods toward generalization and robustness with open space risk regularization is of the essence.

To this end, we propose the decoupling autoencoder (DAE) method, a novel autoencoder-based architecture that learns a radial basis function (RBF) kernel mapping the reconstruction error to class probabilities. The reconstruction error as a measure of outlierness is learned by a novel adversarial loss function that separates inliers from rest samples in reconstruction error space and is based on gradient ascend as suggested by Lübbering et al. [45]. The inlier and outlier distributions are separated by a decision boundary that is optimized end-to-end to be as close as possible to the inlier distribution. Thus, observed and unobserved rest samples can be effectively rejected, resulting in an increased robustness. While the related ATA method employs gradient ascend to samples irrespective of their reconstruction error and estimates the decision boundary offline in a brute-force fashion, the DAE loss function scales down the reconstruction error with increasing distance from the decision boundary. Therefore, samples close to the decision boundary are heavily optimized for better separation; hence the term “decoupling” in DAE. Furthermore, ATA only provides binary predictions therefore lacking rudimentary means of interpretation. In contrast, DAE yields subjective probability scores on the inlierness of a sample, substantially improving the models interpretability especially when deployed in safety-critical environments.

Generally, our method combines the merits of MLPs succeeding on classification tasks and autoencoder methods with their strong robustness. Further, we prove the existence of an upper bound on the open space risk for DAE and leverage this insight to actively minimize open space risk within DAE’s loss function. Throughout a variety of experiments, we empirically show its benefits by outperforming the two most prominent OSR methods C2AE [52] and OpenMax [4], outlier detection methods ATA [45] and one-class autoencoder (OCA), ensemble method MiMo [24], and one-vs-rest MLP.

Since this is an extension paper to Lübbering et al. [43], previous contributions are summarized as follows:This paper comprises the following additional contributions: While the previous paper focused on OVR, we examine DAE in light of the OSR framework in this paper and prove its upper bound on the open space risk. The benefits of the bound and DAE’s ability to minimize open space risk are empirically verified on a range of text and image classification datasets with a varying number of inlier classes. These experiments reveal the resulting robustness gain and pinpoint DNNs’ deficiencies with their lack of a bounded open space risk. We additionally benchmark DAE against recent OSR methods C2AE and OpenMax and show its superiority from an intuitive and empirical point of view.

Further, we visually compare the reconstruction differences on two image datasets between DAE and the autoencoder-based baselines and explain the benefits of the adversarial loss function. Moreover, we show DAE’s robustness w.r.t. local stability in feature space and adversarial robustness using the FGSM method [21]. Finally, we provide an ablation study empirically verifying that only the combination of all loss terms yields the desired classification and robustness properties.

With its added theoretical foundation and empirical verification, DAE becomes a promising candidate for deployment in safety-critical systems.

The OVR classification strategy is often applied to binary models in order to extend them to multi-class classification [5]. Naturally, there is no necessity for OVR classification for DNNs, since they already support multi-class classification by design. However, there are common situations where OVR becomes relevant, e.g., if faced with only positively labeled samples and all remaining samples with potentially unknown sources are assigned to a single negative class [16, 31, 66] or if the goal, as in this paper, is to filter normal samples from abnormal ones [6]. As motivated previously, vanilla MLPs are unsuitable in this case due to their infinite open space risk and consequential robustness deficiencies toward outliers [6]. While in modern architectures, researchers often try to circumvent this problem by subsuming all rest classes within a single background class [19, 57], this effectively only reduces but does not solve the unbounded open space risk [6]. However, there have been few attempts to bound the open space risk in DNNs. For instance, Bendale and Boult [4] and Rudd et al. [59] leverage extreme value theory (EVT) to determine a compact abating probability model based on the deep features of the full network outputs. Noteworthy, both approaches are offline and are therefore not involved in training the network. The autoencoder-based approach C2AE proposed by Oza and Patel [52] also uses EVT to determine the decision boundary in reconstruction error space and requires at least two inlier classes. Other approaches try to bound open space risk by using tent activation functions [58]. They show that it increases robustness to adversarial attacks while potentially compromising classification performance. In conclusion, as stated in a recent survey, OSR is still largely unsolved within the deep learning domain [6].

In the related OCC setting with its focus on outlier detection, DNN-based approaches have been researched from three angles: (1) combining kernel methods [65] with DNN methods [14, 60, 71], (2) generative models (e.g., generative adversarial networks [22] or variational autoencoders [35])-based outlier detectors [50, 64, 72], and (3) based on (semi-) supervised autoencoders [8, 10, 26, 44, 45, 47]. Here, the key idea is to learn a representation of the inlier distribution and subsequently to estimate the outlierness of a sample via its reconstruction error.

Other contributions focus on the calibration, as DNNs tend to provide wrong predictions with overly high confidence estimates for out-of-distribution samples [23, 37]. Since OVR classification incorporates outliers when extended to OCC, this issue is also prominent in OVR classification. Several methods have been proposed: either by adding a calibration task to the model, which aligns it with the target probability distribution [12] or by incorporating diverse predictions from ensemble methods such as deep ensembles [37] and MiMo [24]. These methods combine multiple neural networks as weak classifiers, whose diverse outputs are aggregated to well-calibrated predictions, thus compensating for overly confident predictions. Conversely, as pointed out by Van Amersfoort et al. [67] and also supported by our findings, the diversity of the weak classifiers within the ensemble methods is not strong enough to generalize well to out-of-distribution samples. Instead, Amersfoort et al. propose the kernel-based method DUQ, which learns centroids of classes in a lower-dimensional space [67]. Uncertainty is measured as a distance from the class centroids to out-of-distribution points.

In contrast to Van Amersfoort et al. [67], our DAE method incorporates radial basis functions to estimate the outlierness of a sample via the distance to its reconstruction and therefore does not require any centroid updating routines. Further, ATA [45] optimizes the decision boundary w.r.t. F1 score in an offline brute-force line search in reconstruction error space, which does not actively minimize the open space risk. For DAE, we designed a customized loss function that allows learning the decision boundary end-to-end and minimizes open space risk. Furthermore, it forces the classes to be more separated in reconstruction error space than ATA’s adversarial loss function.

Similar to existing autoencoder-based approaches for outlier detection [8, 26, 45], the decoupling autoencoder (DAE) method learns the outlierness of a sample via its reconstruction error. Existing approaches estimate the decision boundary via brute-force algorithms [1, 45] or learn the decision boundary via a subsequent downstream layer [44]. In contrast, DAE learns the decision boundary end-to-end while optimizing for a pessimistic decision boundary that is most close to the inlier samples without compromising generalization performance. Thus, the decision boundary’s open space risk is actively minimized, a favorable setting in safety-critical systems.

From an architectural point of view, as displayed in Fig. 3, the network reconstructs a sample \({\mathbf {x}} \in {\mathbb {R}}^n\) using the autoencoder \(\varphi ({\mathbf {x}})=d(e({\mathbf {x}}))\) consisting of an encoder \(e(\mathbf {\cdot })\) and a decoder \(d(\mathbf {\cdot })\). The reconstruction error \(e_{\text {MSE}}\) between the original and reconstructed sample \({{\hat{\mathbf{x}}}} \in {\mathbb {R}}^n\) computes to \(e_{\text {MSE}}({\mathbf {x}}, {{\hat{\mathbf{x}}}}) = \frac{1}{n} \sum _i^n (x_i - {\hat{x}}_i)^2\). The reconstruction error is mapped to the inlier probability via Gaussian \(g: z \rightarrow e^{-\frac{z^2}{2\sigma _1^2}}\) with a mean of zero. Thus, the higher the reconstruction error, the smaller the inlier probability. More formally, the full network f is given byNote that the standard deviation \(\sigma _1\) is directly coupled with the decision boundary t via \(t=\sqrt{-2\sigma ^2_1\ln {\frac{1}{2}}}\), as the threshold is fixed at the 0.5 level of function g, also shown in Fig. 4.

During training, the network has three objectives: The overall loss function \(\hat{L}\) incorporates these three training objectives by combining the adversarial loss function \(L_R\), binary cross-entropy (BCE) classification loss \(L_{\text {BCE}}({\hat{y}}, y) = - [y \ln {\hat{y}} + (1-y) \ln (1- {\hat{y}})] \) and a regularizer term \( |t |\), as follows:where y and \({\hat{y}}\) denote the target label and the model’s predicted COI probability of sample \({\mathbf {x}}\), respectively. Factors \(\lambda _2\) and \(\lambda _3\) scale the classification loss term and \(|t |\) regularization.

The adversarial reconstruction loss \(L_R\) comprises the minimization and maximization of reconstruction errors w.r.t. inliers and rest samples, as defined by:where scaling factors \(\lambda _0\in {\mathbb {R}}^{+}\) and \(\lambda _1\in {\mathbb {R}}^{+}\) determine the minimization/maximization magnitude, respectively. Within \(L_R\), the mean squared error \(L_{\text {MSE}}({\mathbf {x}}, {{\hat{\mathbf{x}}}}) = \frac{1}{n} \sum _i^n (x_i - {\hat{x}}_i)^2\) is weighted by \(w_i: L_{\text {MSE}}\rightarrow {\mathbb {R}}\) for inliers and by \(w_o: L_{\text {MSE}} \rightarrow {\mathbb {R}}\) for rest samples. These two functions, given by Eqs. (6) and (7), push the reconstruction errors of inliers and rest samples away from the decision boundary t toward the origin and \(\infty \), respectively, thereby providing a clear class separation:Note that in both cases, the standard deviation \(\sigma _2\) of the Gaussian is a hyperparameter and determines how far the two classes are being separated. \(\sigma _2\) is not to be confused with \(\sigma _1\), which is coupled with the decision boundary t. The reconstruction error maximization of rest samples is achieved in Eq. (7) by the negation which is equal to flipping the loss gradients [45] and thus corresponds to gradient ascend. The reconstruction error objective \(L_R\) is conceptually visualized in Fig. 5.

Due to the spacious separation of the inlier set and rest sample set in reconstruction loss space, there is a wide range of possible decision boundaries t. We argue that the best t is as close as possible to the inlier set without compromising the classification performance. Thus, offering a reasonable trade-off between classification (\(T_c\)) and outlier detection (\(T_o\))/dataset shift (\(T_d\)), while reducing the open space risk. The trade-off is modeled via the second and third addends of \(\hat{L}\) in Eq. (2). The \(||t ||_1\) regularizer minimizes the decision boundary t toward 0, eventually leading to an impractical classifier always predicting RC independently of \({\mathbf {x}}\). This impractical solution is prevented by the classification loss term \(L_{\text {BCE}}\) acting as a stopping criterion, as visualized in Fig. 4. Note that the four scaling factors \(\lambda _0, \dots , \lambda _3\) trade off these three objectives of reconstruction minimization/maximization, classification performance and out-of-distribution robustness.

The combination of \(L_R\) and \(L_{\text {BCE}}\) also solves the vanishing gradient problem, which would occur for large \(e_{\text {MSE}}\), due to the Gaussian output activation function. When solely minimizing the classification loss \(L_{\text {BCE}}\) jointly with the regularizer term, i.e., \(\hat{L}^*= L_{\text {BCE}} + ||t ||_1\), we can show thatwhere \(\varvec{\Theta }\) are the network weights of autoencoder \(\varphi \). The total derivative of \(L^*_{\text {total}}\) computes toAs \({\mathbf {x}}\) tends to infinity, Gaussian g becomes a horizontal line, resulting in gradients equal to 0:Thus, the expression in Eq. (8) zeros out, which is why the gradient updates become ineffective for inliers with large \(e_{\text {MSE}}\). As \(L_R\) is independent of Gaussian g, it is not affected by this problem and enforces convergence by minimizing these inliers.

During training of DNNs, we minimize a surrogate loss function L, such as negative log-likelihood (NLL) instead of the non-differentiable 0-1 loss, over a given empirical data distribution \({\hat{p}}_{\text {data}}({\mathbf {x}}, y)\), as the true \(p_{\text {data}}({\mathbf {x}}, y)\) is unknown [20]. This corresponds to minimizing the empirical risk which is given by \({\mathbb {E}}_{{\mathbf {x}}, y \sim {\hat{p}}_{\text {data}} ({\mathbf {x}}, y)} [L(f({\mathbf {x}}, \varvec{\Theta }),y)] = \frac{1}{N}\sum _{i = 1}^{N} L(f({\mathbf {x}}^{(i)}, \varvec{\Theta }),y^{(i)}),\) where N is the training set size and f the model with parameters \(\varvec{\Theta }\). This procedure optimizes for a discriminating function which correctly separates the classes in the training set, i.e., optimization for classification performance. Under the assumption that the empirical data generating distribution \({\hat{p}}_{\text {data}}({\mathbf {x}}, y)\) is similar to the true data generating distribution \(p_{\text {data}}({\mathbf {x}}, y)\), then within the problem domain the discriminatory function will generalize to unseen data \({\mathbf {x}},y \sim p_{\text {data}}({\mathbf {x}}, y)\).

A problem arises when the model is exposed to samples that are highly unlikely according to \({\hat{p}}_{\text {data}}({\mathbf {x}}, y)\), i.e., outliers because the model was not optimized w.r.t. such samples. This issue is visualized in Fig. 1 for MLP and MiMo on the Half-Moon and XOR Circles datasets. Both methods learn to separate the observed data (i.e., \({\hat{p}}_{\text {data}}({\mathbf {x}}, y)\)). When we consider only the training data (red and blue samples), the learned model indeed minimizes the empirical risk. However, for out-of-distribution data (orange points), we would like to observe high uncertainty. Since this is not reflected in the training objective, the model often predicts one of the two classes with high confidence, even though the out-of-distribution data cannot be attributed to any of the two classes [28].

A simple solution would be to facilitate reconstructive representation learning with, e.g., autoencoders by forcing the reconstruction error to capture the outlierness of a sample. As shown in Fig. 1 for DAE and in Fig. 6 for ATA, each model has learned a representation of COI and rejects any major deviation from it as RC. While this indeed increases the robustness [44, 45], it can also harm the classification performance since a representation for all input features needs to be learned [67]. Uninformative, non-causal features can therefore also have a diminishing effect on classification performance. Models trained within the ERM framework do not suffer from this problem, as the feature extractor would neglect these features [67].

In conclusion, there is a trade-off between classification performance and robustness to outliers/dataset shift which DAE aims to alleviate within the OSR framework.

In recent years, novel deep learning algorithms have advanced state-of-the-art in many classification tasks. However, as noted in the previous section, it has also been shown that these algorithms, when solely optimized for empirical risk, often give wrong predictions with high confidence when exposed to dataset shift and outliers. In this section, we formalize this issue in line with Scheirer et al. [63] and prove that our approach has an upper bound on the open space risk, a primary criterion for robust OSR.

OSR was first defined by Scheirer et al. [63] and recently surveyed by Geng et al. [17] and Boult et al. [6] and is still a largely unsolved topic within the deep learning domain [4, 52, 63]. OSR formalizes the problem of distinguishing a class of interest (i.e., samples originating from an observed set of classes) from samples derived from, e.g., outlier generating processes, dataset shifts, or other possibly unobserved but related classes. As deep learning classifiers are generally trained based on ERM by leveraging a surrogate loss such as cross-entropy, they only learn to differentiate the observed classes. This can be viewed as closed set classification, which is illustrated in Fig. 1: the MLP successfully learns to distinguish the two half-moons resembling the closed set. A problem arises when we zoom out of the problem domain and consider samples from the open set (i.e., outside of the closed set), then these samples that are far away from the closed set are still assigned to one of the two half-moons.

As a solution, OSR proposes an indicator function f over input space \(\chi \), that maps inliers to 1 and rest samples to 0. Partially following the notation of Scheirer et al. [63], let \({\hat{V}}\) be the COI and \(S_{\hat{V}} = \{{\mathbf {x}} \in \chi \,\, |\,\, \min _{{\mathbf {s}} \in \hat{V}} |{\mathbf {x}} - {\mathbf {s}} |< d\}\) be the corresponding closed set, i.e., the set of all points within \(\chi \) that are in d proximity to at least one of the inlier samples \({\mathbf {s}} \in \hat{V}\). Let \(\mathcal {O}=\chi - S_{\hat{V}}\) be the open space, then the open space risk is defined aswhich yields the ratio of the false inlier area over the true inlier area. Thus, \(R_{\mathcal {O}}\) can be minimized by reducing the volume of the indicator function f. In Fig. 2, \(R_\mathcal {O}\) can be interpreted as the ratio of the blue area (positive region of f outside \(S_{{\hat{V}}}\), i.e., false positives) over the area of correctly predicted inliers in \(S_{\hat{V}}\) (i.e., true positives).

Note that a trivial solution for \(R_{\mathcal {O}}\) minimization could be achieved by predicting all samples (except for one inlier to prevent division by zero) as RC independently of their true class, i.e., resulting in a volume of 0 of the indicator function f over the open space. That is why it is crucial to counteract this solution by framing OSR as a two-fold problem with the two objectives: a) Minimization of \(R_{\mathcal {O}}(f)\) and b) ERM as regularization.

As shown by the decision boundary of the MLP in Fig. 1 and denoted in prior research, vanilla MLPs do not provide an upper bound on the open space risk and in practice are generally unbounded [4, 58, 63]. Therefore, we have to turn toward deep learning architectures, such as autoencoders, that are capable of learning manifolds on the input space and by design learn a hull around the inliers:

Lemmas 1 and 2 have multiple practical implications. Firstly, our autoencoder methods are capable of bounding the open space risk and are therefore by design superior to MLP-based architectures in the OSR setting. Secondly, approximating the open space risk enables us to filter models with higher robustness during the model selection process. Thirdly, given that the approximated bound is a hypercube with its center in the origin of the feature space, it is recommendable to perform feature transformation such that the inlier class is located close to the origin, thus allowing for a smaller hypercube. Furthermore, Lemma 1 depicts the dependency of the open space risk on the weights after the last layer with saturating activation functions. By regularizing the weights in conjunction with the centering of the inlier samples, it is possible to actively minimize the bound on the open space risk. While this idea is out-of-scope of this contribution, it is a promising future research direction. For instance, it is possible to handcraft the connections in the first layer of \(\varphi \) such that the weights perform translation and scaling of the feature space.

Adversarial perturbations offer an effective way of measuring a model’s robustness locally as well as globally. As initially described by Goodfellow et al. [21], the idea of gradient-based adversarial attacks is to confuse a neural network f with parameters \(\varvec{\Theta }\) by adding an imperceptible perturbation to the original sample \({\mathbf {x}}\) with target y. The perturbation itself is not arbitrary but maximizes the loss L of the network. A common methodology of calculating these adversarial examples is the fast gradient sign method (FGSM) [21], which determines the perturbation by taking the sign of the gradients w.r.t. the sample:where scaling factor \(\epsilon \) determines the volume of change. Note that since \(\epsilon \) is fixed, also the perturbation’s volume is fixed for all steps and across models.

While in practice, adversarial examples are often used to improve model robustness via adversarial training [21], in this work, we use the FGSM framework for robustness estimation of trained models. By perturbing a sample in a step-wise adversarial fashion, the model confidence development can be tracked which provides deep insights into local stability and with an increasing number of steps also into global robustness of the model. Technically, at a given step i, a sample \({\mathbf {x}}^{i-1}\) is perturbed according toand the difference in confidence \(\Delta c_i\) at step i w.r.t. the original sample is defined aswhere \({\mathbf {x}}^0\) denotes the original sample. By varying the step-size \(\epsilon \) and the number of steps, the aforementioned local stability and global robustness can be easily estimated. Further, there also exists another adversarial robustness metric [74], which is defined as:and computes the Kullback–Leibler divergence \(D_{KL}\) in the denominator as the relative entropy between the two confidence estimates. However, we did not consider this measure, as \(D_{KL}\) rapidly changes around (0, 0) and (1, 1) and small changes in this area therefore lead to overly pessimistic robustness scores.

Generally, we hypothesize that DAE, as a representative for an OSR architecture, is more robust than the MLP, as the latter one rather slices the spaces in discriminative hyperplanes w.r.t. the two classes. Therefore, we can expect a given dataset shift sample to be closer to a hyperplane than DAE’s hull which is learned directly around the inlier class. We would assume similar adversarial robustness between MLP and DAE for an inlier sample.

In this section, we discuss the experiment setup and compare the performance of DAE to the aforementioned baselines. The approaches are compared on an algorithmic level by applying nested cross-validation (CV) [32, 56, 68]. The best models are selected by the highest area under the precision-recall curve (AUPR) score and are reported along with area under the receiver operating characteristics (AUROC) and F1 score with respective confidence measurements.

Open set recognition (OSR) is a common setup in machine learning applications. Whenever the objective is to distinguish at least one class of interest (COI) from all remaining, possibly unknown classes (RC), e.g., ordinary internet traffic from novel intrusion attempts or general discussions from all types of hate speech, OSR methods seem natural. Our analysis revealed that when extending the scope of RC, OSR poses severe challenges of outlier detection and dataset shift to deep neural networks solely optimized via empirical risk minimization. We provide an effective solution to these deficiencies with our proposed decoupling autoencoder (DAE) architecture. We proved the existence of a bounded open space risk for DAE and supported its classification and (adversarial) robustness benefits across three different subtasks of OSR. Specifically, we benchmarked DAE against capable baselines from various domains (DNNs, ensemble methods, outlier detection, and OSR) w.r.t. the OSR subtasks of classification, outlier detection, and dataset shift. DAE showed superior robustness across all subtasks throughout all experiments compared to the baselines, which failed on at least one of the tasks apart from ATA. In comparison with ATA, DAE can actively minimize the open space risk and does not require an offline brute-force line search for decision boundary estimation.

For future work, it would be interesting to extend DAE toward multi-class classification with a bounded open space risk, which would allow for robust multi-class classification under extreme dataset shift conditions. Finally, a promising idea could be the development of feature extractors that prevent the model from learning representations of noisy or uninformative features, thereby further alleviating the trade-off between classification and robustness performance.