1 Introduction
-
WisCon approach: We propose a novel approach for contextual anomaly detection, namely Wisdom of the Contexts (WisCon), that automatically creates contexts, where true contextual and behavioral attributes are not known beforehand, and constructs an ensemble of multiple contexts with an active learning model,
-
Sampling strategy: We propose a new committee-based query strategy, low confidence anomaly (LCA) sampling, designed to select anomalous samples that cannot be detected under the majority of the contexts. This strategy allows us to actively query for the anomalies that provide more information about which contexts are “relevant” or “irrelevant” so that importance of contexts can be estimated within a small budget.
-
Context Ensembles: We design an ensemble over the weighted combination of different contexts, in which the results from different contexts are aggregated using their importance scores estimated with active learning and a pruning strategy that eliminates irrelevant contexts.
-
An empirical study: We conduct comprehensive experiments including statistical comparisons with baselines in different categories; performance comparisons and budget analysis among the different state of the art query strategies and our novel strategy; and a study on showing individual benefits of two core concepts, i.e., active learning and multi-context ensembles, in this problem.
2 Related work
3 Problem definition
Symbol | Definition |
---|---|
U | Unlabeled dataset |
L | Labeled dataset |
x | Data point, \(x \in U\) |
(x, y) | Data point, \((x,y)\in L\) |
F | Set of features, \(|F|=d\) |
C | Context, \(C \subset F\) |
B | Behavior, \(B \subset F\), \(B= F {\setminus } C\) |
\(C^\prime \) | True context |
\({\hat{C}}\) | Set of contexts extracted from F, \(C_i \in {\hat{C}}\) |
R(C, x) | Reference group of x w.r.t C |
\(s_{i,j}\) | Anomaly score of \(x_j \in U\) within \(C_i\in {\hat{C}}\), \(s_{i,j} \in [0,1]\) |
\(S_i\) | Set of anomaly scores for \(C_i\in {\hat{C}}\), \(s_{i,j}\in S_i\) |
\({\hat{S}}\) | Matrix of anomaly scores, \({\hat{S}} \in {\mathbb {R}}^{n \times m}\), \(n=|U|\), \(m=|{\hat{C}}|\), and \(S_i\) is the column of \({\hat{S}}\) |
\(p_{i,j}\) | Prediction of \(x_j \in U\) in \(C_i\in {\hat{C}}\), \(p_{i,j} \in \{0,1\}\) |
O | Oracle |
b | Budget |
\(\epsilon _i\) | Detection error in \(C_i\) |
Q() | Query strategy |
\(\theta _j\) | Sample weight of \(x_j \in U\) |
\(I_i\) | The importance score of \(C_i\) |
-
a dataset of points U described by real valued features F, in which the true context \(C^\prime \) is unknown apriori,
-
an oracle O,
-
and a budget b;
-
a contextual anomaly score for each point in U after obtaining b labels from O, such that the anomaly detection performance is maximized
4 Wisdom of the contexts (WisCon) framework
4.1 Framework overview
4.2 Base detector
4.3 Active learning
4.4 Importance measure
4.5 Ensemble pruning and final aggregation
5 Query strategies
5.1 Random sampling
5.2 Query by committee
5.3 Most-likely anomalous sampling
5.4 Low confidence anomaly sampling
6 Experiments
-
Q1. How does the performance of WisCon compare to the performances of the state-of-the-art competitors? (Sect. 6.4)
-
Q2. How do different query strategies affect the performance of WisCon under different budgets? (Sect. 6.5)
-
Q3. How beneficial is employing active learning or context ensembles under the WisCon approach for detecting contextual anomalies? (Sect. 6.6).
Dataset | Anomalies | #Pts n | % Anomalies | Dim. d |
---|---|---|---|---|
Synthetic 1 | Injected | 25250 | 250 (1%) | 10 |
Synthetic 2 | Injected | 5100 | 100 (2%) | 10 |
Synthetic 3 | Injected | 5100 | 50 (2%) | 50 |
Synthetic 4 | Injected | 5100 | 50 (2%) | 10 |
El Nino | Injected | 94784 | 1879 (2%) | 11 |
Houses | Injected | 20640 | 413 (2%) | 9 |
Abalone | Real | 1920 | 29 (1.5%) | 9 |
ANN-Thyroid | Real | 7200 | 534 (7.4%) | 6 |
Arrhythmia | Real | 452 | 66 (15%) | 274 |
Letter Recognition | Real | 1600 | 100 (6.25%) | 32 |
Mammography | Real | 11183 | 260 (2.32%) | 6 |
Optdigits | Real | 5216 | 150 (3%) | 64 |
Pendigits | Real | 6870 | 156 (2.27%) | 16 |
Satelite | Real | 6435 | 2036 (32%) | 36 |
Satimage | Real | 5803 | 71 (1.2%) | 36 |
Thyroid | Real | 3772 | 93 (2.5%) | 6 |
Vowels | Real | 1456 | 50 (3.4%) | 12 |
Yeast | Real | 1364 | 64 (4.7%) | 8 |
Baseline | Parameter | Value set |
---|---|---|
WisCon | Max. no. clusters (XMeans) | {5, 10, 20} |
No. estimators (iForest) | 100 | |
Max. samples (iForest) | 256 | |
\(\lambda \) (LCA Sampling) | 0.96 | |
AAD | \(\tau \) | 0.03 |
\(C_A\) | 100 | |
\(C_{\xi }\) | 0.001 | |
No. estimators (iForest) | 100 | |
Max. samples (iForest) | 256 | |
Active-RF | No. estimators | {10, 50, 100, 150, 200} |
Max. features | {auto, sqrt, log2} | |
Class weight | {None, balanced, balanced subsample} | |
Bootstrap | {True, false} | |
Active-KNN | No. neighbors (k) | {2, 4,..., 10} |
Active-SVM | \(\gamma \) | \(10^x\), \(x \in \{-4, -3,\ldots , 4\}\) |
C | \(10^x\), \(x \in \{-2,\ldots , 4\}\) | |
Kernel | {rbf, polynomial, sigmoid} | |
ConOut | \(\gamma \) | {0.0001,0.001,0.01,0.1,1,10,100,1000} |
ROCOD | No. neighbors (k) | {10, 20,..., 100} |
Max. depth | {5, 10, 15, 20} | |
Min. samples split | {10, 20,..., 100} | |
CAD | No. components | {5, 10, 20, 40} |
iForest-Con | Max no. clusters | {5, 10, 20} |
No. estimators | {100, 200,..., 500} | |
Max features | {0.1, 0.2,..., 1} | |
LOF-Con | Max no. clusters | {5, 10, 20} |
No. neighbors (k) | {10, 20,..., 200} | |
OCSVM-Con | Max no. clusters | {5, 10, 20} |
\(\gamma \) | \(10^x\), \(x \in \{-4, -3,\ldots , 4\}\) | |
\(\nu \) | \(\{0.01, 0.5, 0.99\}\) | |
Kernel | {rbf, polynomial, sigmoid} | |
iForest | No. estimators | {100, 200,..., 500} |
Max features | {0.1, 0.2,..., 1} | |
LOF | No. neighbors (k) | {10, 20,..., 200} |
OCSVM | \(\gamma \) | \(10^x\), \(x \in \{-4, -3,\ldots , 4\}\) |
\(\nu \) | \(\{0.01, 0.5, 0.99\}\) | |
Kernel | {rbf, polynomial, sigmoid} | |
LODA | No. bins | {2, 4,..., 100} |
No. random cuts | {40, 60,..., 500} | |
SOD | No. neighbors (k) | {10, 20,..., 200} |
No. reference set | {10, 20,..., 100} | |
\(\alpha \) | {0.1, 0.2,..., 1} | |
Feature Bagging | No. estimators | {100, 200,..., 500} |
Max features | {0.1, 0.2,..., 1} |
6.1 Datasets
6.2 Baselines
-
AAD: Active Anomaly Discovery (AAD) (Das et al. 2016, 2017) is one of the recent methods for incorporating expert feedback into an ensemble of anomaly detectors. AAD has been implemented with different ensemble detectors such as LODA (Das et al. 2016) and iForest (Das et al. 2016). In this work, we use iForest-AAD as it has been shown to outperform LODA-AAD in (Das et al. 2017).
-
Active-RF: Active Random Forest is the active version of Random Forest classifier. It uses pool-based setting with uncertainty sampling to query instances under budget b.
-
Active-KNN: Active KNN is the active version of the KNN classifier. It uses pool-based setting with uncertainty sampling to query instances under budget b.
-
Active-SVM: Active-SVM is the active version of SVM classifier. It uses pool-based active learning with uncertainty sampling.
-
ROCOD: Robust contextual outlier detection (Liang and Parthasarathy 2016) (ROCOD) simultaneously considers local and global effects in outlier detection. Specically, kNN regression is used to generate a local expectation for each sample, and a ridge regression (ROCOD.RIDGE) or tree regression (ROCOD.CART) is used to produce a global expectation for each sample. Then, these two estimations are combined to generate a total expectation for the behavioral attribute value. In this work, we use ROCOD.CART since it has been shown to perform better in (Liang and Parthasarathy 2016).
-
ConOut: ConOut is a contextual anomaly detection algorithm that combines multiple contexts identified automatically. It measures the pairwise dependencies between attributes in a feature set and combines attributes into contexts in which highly dependent features are not present together. Then, it quantifies the outlierness of a sample in each context with a custom anomaly detector called context-incorporated outlier detection, and takes the maximum of the scores across all contexts as final outlier scores.
-
CAD: Conditional Anomaly Detection (CAD) (Song et al. 2007) is a generative approach to model the relation between context and behavior. Both the context and the behavior are modeled separately as a mixture of multiple Gaussians. With a probabilistic mapping function, it captures how behavioral attributes are related to contextual attributes.
-
iForest-Con: Contextual Isolation Forest (IF-Con) is implemented following the similar technique explained in Sect. 4.2, in which an anomaly detector (i.e., iForest) is trained only using behavioral attributes for each reference group, while the reference groups are estimated by XMeans using contextual attributes.
-
LOF-Con: Contextual Local Outlier Factor (IF-Con) is implemented similar to iForest-Con. Instead of iForest, it uses LOF algorithm to detect anomalies in each reference group.
-
OCSVM-Con: Contextual One-Class SVM (IF-Con) is implemented similar to iForest-Con and LOF-Con. It uses OCSVM algorithm to detect anomalies in each reference group.
-
iForest: Isolation Forest (iForest) (Liu et al. 2008) is the isolation based tree ensemble detector. The algorithm randomly partitions the data on randomly selected features and stores this partitioning in a tree structure. The samples that travel shorter into the tree are assumed to more likely to be anomalies as they require less cuts to isolate them.
-
LOF: Local Outlier Factor (LOF) (Breunig et al. 2000) is a widely used anomaly detection algorithm that compares the local density of each point to that of its neighbors. Points with significantly lower density compared to their neighbors are regarded as outliers.
-
OC-SVM: One class SVM (Schölkopf et al. 1999) is another popular method based on the principles of support vectors. The method projects the data to a high-dimensional feature space and tries to find a hyperplane best separating the data from the origin.
-
LODA: Lightweight online detector of anomalies (LODA) (Pevnỳ 2016) is an ensemble method using 1-dimensional random projections in combination with histogram-based detectors to spot anomalies in high-dimensional data.
-
SOD: Subspace outlier detection (SOD) (Kriegel et al. 2009) aims at detecting outliers that are visible in different subspaces of a high dimensional feature space. It uses shared nearest neighbors as a reference set for each object and derives a subspace where the reference set exhibits low variance.
-
FB: Feature bagging (FB) (Lazarevic and Kumar 2005) is also an ensemble detector based on randomly sampling subspaces and combining anomaly scores from these subspaces measured by LOF algorithm.
6.3 Experimental setup
6.4 Results Q1: comparison of the WisCon over baselines
Dataset/method | WisCon | AAD | Active-RF | Active-KNN | Active-SVM | |
---|---|---|---|---|---|---|
b = 20 | Synthetic1 | \(\mathbf {0.82 \pm 0.02}\) | \(0.24 \pm 0.09\downarrow \) | \(0.19 \pm 0.10\downarrow \) | \(0.10 \pm 0.03\downarrow \) | \(0.06 \pm 0.06\downarrow \) |
Synthetic2 | \(\mathbf {0.69 \pm 0.08}\) | \(0.36 \pm 0.05\downarrow \) | \(0.27 \pm 0.13\downarrow \) | \(0.06 \pm 0.03\downarrow \) | \(0.01 \pm 0.0\downarrow \) | |
Synthetic3 | \(\mathbf {0.86 \pm 0.04}\) | \(0.12 \pm 0.05\downarrow \) | \(0.09 \pm 0.05\downarrow \) | \(0.02 \pm 0.01\downarrow \) | \(0.01 \pm 0.0\downarrow \) | |
Synthetic4 | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | \(0.32 \pm 0.12\downarrow \) | \(0.23 \pm 0.11\downarrow \) | \(0.58 \pm 0.27\downarrow \) | |
El Nino | \(\mathbf {0.62 \pm 0.04}\) | \(0.21 \pm 0.0\downarrow 1\) | \(0.08 \pm 0.05\downarrow \) | \(0.11 \pm 0.01\downarrow \) | \(0.10 \pm 0.03\downarrow \) | |
Houses | \(\mathbf {0.63 \pm 0.09}\) | \(0.08 \pm 0.02\downarrow \) | \(0.21 \pm 0.15\downarrow \) | \(0.11 \pm 0.06\downarrow \) | \(0.26 \pm 0.20\downarrow \) | |
Abalone | \(\mathbf {0.73 \pm 0.08}\) | \(0.66 \pm 0.08\downarrow \) | \(0.22 \pm 0.12\downarrow \) | \(0.06 \pm 0.04\downarrow \) | \(0.16 \pm 0.13\downarrow \) | |
Annthyroid | \(\mathbf {0.75 \pm 0.05}\) | \(0.32 \pm 0.06\downarrow \) | \(0.71 \pm 0.05\) | \(0.17 \pm 0.04\downarrow \) | \(0.46 \pm 0.16\downarrow \) | |
Arrhythmia | \(\mathbf {0.58 \pm 0.10}\) | \(0.50 \pm 0.06\downarrow \) | \(0.31 \pm 0.10\downarrow \) | \(0.21 \pm 0.05\downarrow \) | \(0.10 \pm 0.01\downarrow \) | |
Letters | \(\mathbf {0.36 \pm 0.07}\) | \(0.11 \pm 0.02\downarrow \) | \(0.13 \pm 0.05\downarrow \) | \(0.07 \pm 0.01\downarrow \) | \(0.13 \pm 0.11\downarrow \) | |
Mammography | \(\mathbf {0.48 \pm 0.07}\) | \(0.35 \pm 0.10\downarrow \) | \(0.24 \pm 0.13\downarrow \) | \(0.20 \pm 0.01\downarrow \) | \(0.40 \pm 0.08\downarrow \) | |
Optdigits | \(0.58 \pm 0.07\downarrow \) | \(0.13 \pm 0.19\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | \(0.31 \pm 0.11\downarrow \) | \(0.79 \pm 0.38\) | |
Pendigits | \(0.72 \pm 0.08\downarrow \) | \(0.48 \pm 0.11\downarrow \) | \(\mathbf {0.83 \pm 0.14}\) | \(0.66 \pm 0.13\downarrow \) | \(0.26 \pm 0.39\downarrow \) | |
Satellite | \(0.53 \pm 0.06\downarrow \) | \(\mathbf {0.65 \pm 0.02}\) | \(0.61 \pm 0.18\) | \(0.60 \pm 0.05\downarrow \) | \(0.40 \pm 0.22\) | |
SatImage | \(0.82 \pm 0.1\downarrow \) | \(\mathbf {0.94 \pm 0.03}\) | \(0.91 \pm 0.01\downarrow \) | \(0.89 \pm 0.01\downarrow \) | \(0.42 \pm 0.40\) | |
Thyroid | \(0.85 \pm 0.12\) | \(0.68 \pm 0.02\downarrow \) | \(\mathbf {0.92 \pm 0.03}\) | \(0.59 \pm 0.11\downarrow \) | \(0.30 \pm 0.36\downarrow \) | |
Vowels | \(\mathbf {0.63 \pm 0.02}\) | \(0.39 \pm 0.11\downarrow \) | \(0.57 \pm 0.04\downarrow \) | \(0.22 \pm 0.09\downarrow \) | \(0.19 \pm 0.15\downarrow \) | |
Yeast | \(\mathbf {0.41 \pm 0.10}\) | \(0.31 \pm 0.09\downarrow \) | \(0.08 \pm 0.06\downarrow \) | \(0.08 \pm 0.03\downarrow \) | \(0.07 \pm 0.04\downarrow \) | |
Avg Rank | \(\mathbf {1.58}\) \(\mathbf {(1)}\) | 2.69 (2) | 2.72 (3) | 4.13 (5) | 3.86 (4) | |
b = 60 | Synthetic1 | \(\mathbf {0.85 \pm 0.01}\) | \(0.41 \pm 0.06\downarrow \) | \(0.49 \pm 0.05\downarrow \) | \(0.35 \pm 0.08\downarrow \) | \(0.24 \pm 0.01\downarrow \) |
Synthetic2 | \(\mathbf {0.71 \pm 0.07}\) | \(0.48 \pm 0.06\downarrow \) | \(0.26 \pm 0.15\downarrow \) | \(0.23 \pm 0.14\downarrow \) | \(0.30 \pm 0.18\downarrow \) | |
Synthetic3 | \(\mathbf {0.85 \pm 0.05}\) | \(0.14 \pm 0.03\downarrow \) | \(0.07 \pm 0.05\downarrow \) | \(0.13 \pm 0.04\downarrow \) | \(0.07 \pm 0.17\downarrow \) | |
Synthetic4 | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(0.41 \pm 0.23\downarrow \) | \(0.53 \pm 0.11\downarrow \) | \(0.79 \pm 0.21\) | |
El Nino | \(\mathbf {0.68 \pm 0.01}\) | \(0.25 \pm 0.03\downarrow \) | \(0.31 \pm 0.03\downarrow \) | \(0.15 \pm 0.03\downarrow \) | \(0.23 \pm 0.06\downarrow \) | |
Houses | \(\mathbf {0.66 \pm 0.04}\) | \(0.35 \pm 0.06\downarrow \) | \(0.57 \pm 0.11\downarrow \) | \(0.24 \pm 0.10\downarrow \) | \(0.22 \pm 0.09\downarrow \) | |
Abalone | \(\mathbf {0.79 \pm 0.07}\) | \(0.64 \pm 0.12\downarrow \) | \(0.14 \pm 0.13\downarrow \) | \(0.09 \pm 0.13\downarrow \) | \(0.08 \pm 0.13\downarrow \) | |
Annthyroid | \(\mathbf {0.77 \pm 0.04}\) | \(0.35 \pm 0.06\) | \(0.76 \pm 0.04\downarrow \) | \(0.32 \pm 0.02\downarrow \) | \(0.69 \pm 0.05\downarrow \) | |
Arrhythmia | \(\mathbf {0.60 \pm 0.09}\) | \(0.30 \pm 0.07\downarrow \) | \(0.30 \pm 0.08\downarrow \) | \(0.15 \pm 0.04\downarrow \) | \(0.20 \pm 0.12\downarrow \) | |
Letters | \(\mathbf {0.35 \pm 0.08}\) | \(0.19 \pm 0.04\downarrow \) | \(0.16 \pm 0.07\downarrow \) | \(0.14 \pm 0.04\downarrow \) | \(0.18 \pm 0.10\downarrow \) | |
Mammography | \(\mathbf {0.49 \pm 0.06}\) | \(0.40 \pm 0.08\) | \(0.48 \pm 0.02\) | \(0.38 \pm 0.06\downarrow \) | \(0.48 \pm 0.05\) | |
Optdigits | \(0.60 \pm 0.07\downarrow \) | \(0.47 \pm 0.37\downarrow \) | \(\mathbf {1.0 \pm 0.0}\) | \(0.99 \pm 0.0\) | \(0.99 \pm 0.0\) | |
Pendigits | \(0.68 \pm 0.07\downarrow \) | \(0.67 \pm 0.09\downarrow \) | \(0.97 \pm 0.0\downarrow \) | \(\mathbf {0.98 \pm 0.0}1\) | \(\mathbf {0.98 \pm 0.0}\) | |
Satellite | \(0.57 \pm 0.04\downarrow \) | \(0.67 \pm 0.02\downarrow \) | \(\mathbf {0.81 \pm 0.14}\) | \(0.77 \pm 0.03\) | \(0.71 \pm 0.19\) | |
SatImage | \(0.84 \pm 0.09\downarrow \) | \(\mathbf {0.96 \pm 0.02}\) | \(0.91 \pm 0.01\) | \(0.91 \pm 0.02\) | \(0.91 \pm 0.02\) | |
Thyroid | \(0.86 \pm 0.07\downarrow \) | \(0.83 \pm 0.06\downarrow \) | \(\mathbf {0.97 \pm 0.01}\) | \(0.79 \pm 0.05\downarrow \) | \(0.28 \pm 0.35\downarrow \) | |
Vowels | \(0.65 \pm 0.08\downarrow \) | \(0.63 \pm 0.12\downarrow \) | \(\mathbf {0.98 \pm 0.01}\) | \(0.76 \pm 0.07\downarrow \) | \(0.69 \pm 0.34\downarrow \) | |
Yeast | \(\mathbf {0.41 \pm 0.11}\) | \(0.34 \pm 0.07\) | \(0.06 \pm 0.02\downarrow \) | \(0.03 \pm 0.0\downarrow \) | \(0.06 \pm 0.01\downarrow \) | |
Avg Rank | \(\mathbf {2.0}\) \(\mathbf {(1)}\) | 3.19 (3) | 2.55 (2) | 3.77 (5) | 3.47 (4) | |
b = 100 | Synthetic1 | \(\mathbf {0.88 \pm 0.01}\) | \(0.45 \pm 0.04\downarrow \) | \(0.61 \pm 0.10\downarrow \) | \(0.44 \pm 0.11\downarrow \) | \(0.36 \pm 0.06\downarrow \) |
Synthetic2 | \(\mathbf {0.72 \pm 0.06}\) | \(0.53 \pm 0.05\downarrow \) | \(0.20 \pm 0.12\downarrow \) | \(0.31 \pm 0.08\downarrow \) | \(0.33 \pm 0.16\downarrow \) | |
Synthetic3 | \(\mathbf {0.86 \pm 0.04}\) | \(0.16 \pm 0.03\downarrow \) | \(0.03 \pm 0.03\downarrow \) | \(0.25 \pm 0.12\downarrow \) | \(0.04 \pm 0.06\downarrow \) | |
Synthetic4 | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(0.54 \pm 0.24\downarrow \) | \(0.84 \pm 0.18\downarrow \) | \(0.79 \pm 0.20\downarrow \) | |
El Nino | \(\mathbf {0.76 \pm 0.01}\) | \(0.27 \pm 0.02\downarrow \) | \(0.51 \pm 0.05\downarrow \) | \(0.67 \pm 0.01\downarrow \) | \(0.29 \pm 0.13\downarrow \) | |
Houses | \(\mathbf {0.69 \pm 0.03}\) | \(0.35 \pm 0.06\downarrow \) | \(0.62 \pm 0.06\downarrow \) | \(0.28 \pm 0.10\downarrow \) | \(0.27 \pm 0.17\downarrow \) | |
Abalone | \(\mathbf {0.81 \pm 0.09}\) | \(0.62 \pm 0.17\downarrow \) | \(0.12 \pm 0.14\downarrow \) | \(0.07 \pm 0.09\downarrow \) | \(0.09 \pm 0.16\downarrow \) | |
Annthyroid | \(\mathbf {0.80 \pm 0.04}\) | \(0.44 \pm 0.08\downarrow \) | \(\mathbf {0.80 \pm 0.02}\) | \(0.35 \pm 0.01\downarrow \) | \(0.73 \pm 0.04\downarrow \) | |
Arrhythmia | \(\mathbf {0.60 \pm 0.09}\) | \(0.18 \pm 0.03\downarrow \) | \(0.25 \pm 0.07\downarrow \) | \(0.08 \pm 0.02\downarrow \) | \(0.31 \pm 0.12\downarrow \) | |
Letters | \(\mathbf {0.36 \pm 0.08}\) | \(0.22 \pm 0.06\downarrow \) | \(0.14 \pm 0.07\downarrow \) | \(0.14 \pm 0.04\downarrow \) | \(0.20 \pm 0.07\downarrow \) | |
Mammography | \(0.50 \pm 0.05\) | \(0.50 \pm 0.05\) | \(\mathbf {0.52 \pm 0.02}\) | \(0.41 \pm 0.04\downarrow \) | \(\mathbf {0.52 \pm 0.03}\) | |
Optdigits | \(0.62 \pm 0.08\downarrow \) | \(0.80 \pm 0.21\) | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | |
Pendigits | \(0.72 \pm 0.09\downarrow \) | \(0.93 \pm 0.03\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | \(0.98 \pm 0.0\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | |
Satellite | \(0.60 \pm 0.04\downarrow \) | \(0.69 \pm 0.02\downarrow \) | \(\mathbf {0.89 \pm 0.02}\) | \(0.83 \pm 0.01\downarrow \) | \(0.81 \pm 0.06\downarrow \) | |
SatImage | \(0.86 \pm 0.08\downarrow \) | \(\mathbf {0.98 \pm 0.0}\) | \(0.96 \pm 0.01\downarrow \) | \(\mathbf {0.98 \pm 0.01}\) | \(0.93 \pm 0.0\downarrow \) | |
Thyroid | \(0.87 \pm 0.05\downarrow \) | \(0.91 \pm 0.02\downarrow \) | \(\mathbf {0.97 \pm 0.01}\) | \(0.82 \pm 0.06\downarrow \) | \(0.35 \pm 0.28\downarrow \) | |
Vowels | \(0.65 \pm 0.08\downarrow \) | \(0.74 \pm 0.07\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | \(0.88 \pm 0.02\downarrow \) | \(0.94 \pm 0.01\downarrow \) | |
Yeast | \(\mathbf {0.41 \pm 0.10}\) | \(0.39 \pm 0.06\) | \(0.03 \pm 0.02\downarrow \) | \(0.03 \pm 0.01\downarrow \) | \(0.04 \pm 0.02\downarrow \) | |
Avg Rank | \(\mathbf {2.38}\) \(\mathbf {(1)}\) | 2.66 (2) | 2.86 (3) | 3.52 (4) | 3.55 (5) |
Dataset/method | WisCon | AAD | Active-RF | Active-KNN | Active-SVM | |
---|---|---|---|---|---|---|
b = 20 | Synthetic1 | \(\mathbf {0.98 \pm 0.0}\) | \(0.89 \pm 0.03\downarrow \) | \(0.60 \pm 0.14\downarrow \) | \(0.67 \pm 0.08\downarrow \) | \(0.40 \pm 0.23\downarrow \) |
Synthetic2 | \(\mathbf {0.95 \pm 0.02}\) | \(0.82 \pm 0.02\downarrow \) | \(0.55 \pm 0.06\downarrow \) | \(0.50 \pm 0.06\downarrow \) | \(0.36 \pm 0.06\downarrow \) | |
Synthetic3 | \(\mathbf {0.97 \pm 0.01}\) | \(0.77 \pm 0.04\downarrow \) | \(0.54 \pm 0.07\downarrow \) | \(0.53 \pm 0.05\downarrow \) | \(0.39 \pm 0.05\downarrow \) | |
Synthetic4 | \(0.99 \pm 0.0\) | \(\mathbf {1.0 \pm 0.0}\) | \(0.65 \pm 0.06\downarrow \) | \(0.60 \pm 0.05\downarrow \) | \(0.62 \pm 0.31\downarrow \) | |
El Nino | \(\mathbf {0.97 \pm 0.01}\) | \(0.87 \pm 0.01\downarrow \) | \(0.62 \pm 0.05\downarrow \) | \(0.58 \pm 0.07\downarrow \) | \(0.48 \pm 0.07\downarrow \) | |
Houses | \(\mathbf {0.97 \pm 0.0}\) | \(0.81 \pm 0.04\downarrow \) | \(0.81 \pm 0.02\downarrow \) | \(0.77 \pm 0.04\downarrow \) | \(0.56 \pm 0.18\downarrow \) | |
Abalone | \(\mathbf {0.94 \pm 0.02}\) | \(\mathbf {0.94 \pm 0.02}\) | \(0.56 \pm 0.09\downarrow \) | \(0.53 \pm 0.09\downarrow \) | \(0.56 \pm 0.11\downarrow \) | |
Annthyroid | \(\mathbf {0.97 \pm 0.0}\) | \(0.79 \pm 0.05\downarrow \) | \(0.96 \pm 0.02\) | \(0.67 \pm 0.05\downarrow \) | \(0.73 \pm 0.11\downarrow \) | |
Arrhythmia | \(\mathbf {0.83 \pm 0.04}\) | \(0.71 \pm 0.05\downarrow \) | \(0.71 \pm 0.08\downarrow \) | \(0.63 \pm 0.07\downarrow \) | \(0.24 \pm 0.03\downarrow \) | |
Letters | \(\mathbf {0.73 \pm 0.04}\) | \(0.61 \pm 0.07\downarrow \) | \(0.61 \pm 0.06\downarrow \) | \(0.54 \pm 0.05\downarrow \) | \(0.47 \pm 0.12\downarrow \) | |
Mammography | \(\mathbf {0.90 \pm 0.01}\) | \(0.79 \pm 0.03\downarrow \) | \(0.86 \pm 0.12\) | \(0.82 \pm 0.04\downarrow \) | \(0.47 \pm 0.37\downarrow \) | |
Optdigits | \(0.96 \pm 0.01\downarrow \) | \(0.69 \pm 0.14\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | \(0.96 \pm 0.02\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | |
Pendigits | \(0.98 \pm 0.0\downarrow \) | \(0.93 \pm 0.01\downarrow \) | \(0.98 \pm 0.01\) | \(0.95 \pm 0.01\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | |
Satellite | \(0.69 \pm 0.01\) | \(0.73 \pm 0.01\) | \(0.70 \pm 0.11\) | \(0.73 \pm 0.02\) | \(\mathbf {0.75 \pm 0.19}\) | |
SatImage | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | \(0.97 \pm 0.0\downarrow \) | \(0.96 \pm 0.02\downarrow \) | \(0.97 \pm 0.02\downarrow \) | |
Thyroid | \(\mathbf {0.99 \pm 0.0}\) | \(0.98 \pm 0.0\) | \(\mathbf {0.99 \pm 0.0}\) | \(0.95 \pm 0.02\downarrow \) | \(0.97 \pm 0.03\) | |
Vowels | \(\mathbf {0.94 \pm 0.01}\) | \(0.82 \pm 0.05\downarrow \) | \(0.89 \pm 0.06\downarrow \) | \(0.85 \pm 0.03\downarrow \) | \(0.92 \pm 0.07\) | |
Yeast | \(\mathbf {0.71 \pm 0.06}\) | \(0.68 \pm 0.04\) | \(0.57 \pm 0.10\downarrow \) | \(0.60 \pm 0.04\downarrow \) | \(0.54 \pm 0.09\downarrow \) | |
Avg Rank | \(\mathbf {1.58}\) \(\mathbf {(1)}\) | 2.73 (2) | 2.83 (3) | 4.0 (5) | 3.86 (4) | |
b = 60 | Synthetic1 | \(\mathbf {0.98 \pm 0.0}\) | \(0.92 \pm 0.01\downarrow \) | \(0.62 \pm 0.10\downarrow \) | \(0.81 \pm 0.04\downarrow \) | \(0.61 \pm 0.13\downarrow \) |
Synthetic2 | \(\mathbf {0.95 \pm 0.02}\) | \(0.86 \pm 0.03\downarrow \) | \(0.51 \pm 0.06\downarrow \) | \(0.60 \pm 0.05\downarrow \) | \(0.54 \pm 0.18\downarrow \) | |
Synthetic3 | \(\mathbf {0.97 \pm 0.01}\) | \(0.78 \pm 0.02\downarrow \) | \(0.45 \pm 0.07\downarrow \) | \(0.68 \pm 0.12\downarrow \) | \(0.40 \pm 0.04\downarrow \) | |
Synthetic4 | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(0.65 \pm 0.07\downarrow \) | \(0.76 \pm 0.09\downarrow \) | \(0.91 \pm 0.09\downarrow \) | |
El Nino | \(\mathbf {0.98 \pm 0.0}\) | \(0.85 \pm 0.01\downarrow \) | \(0.68 \pm 0.06\downarrow \) | \(0.68 \pm 0.02\downarrow \) | \(0.47 \pm 0.08\downarrow \) | |
Houses | \(\mathbf {0.98 \pm 0.0}\) | \(0.83 \pm 0.09\downarrow \) | \(0.72 \pm 0.16\downarrow \) | \(0.84 \pm 0.03\downarrow \) | \(0.61 \pm 0.24\downarrow \) | |
Abalone | \(\mathbf {0.95 \pm 0.02}\) | \(0.92 \pm 0.05\) | \(0.48 \pm 0.09\downarrow \) | \(0.59 \pm 0.06\downarrow \) | \(0.38 \pm 0.15\downarrow \) | |
Annthyroid | \(\mathbf {0.98 \pm 0.0}\) | \(0.76 \pm 0.05\downarrow \) | \(0.97 \pm 0.01\) | \(0.73 \pm 0.03\downarrow \) | \(0.83 \pm 0.01\downarrow \) | |
Arrhythmia | \(\mathbf {0.83 \pm 0.04}\) | \(0.56 \pm 0.07\downarrow \) | \(0.73 \pm 0.04\downarrow \) | \(0.67 \pm 0.05\downarrow \) | \(0.25 \pm 0.05\downarrow \) | |
Letters | \(\mathbf {0.75 \pm 0.04}\) | \(0.60 \pm 0.07\downarrow \) | \(0.64 \pm 0.08\downarrow \) | \(0.62 \pm 0.08\downarrow \) | \(0.61 \pm 0.11\downarrow \) | |
Mammography | \(\mathbf {0.90 \pm 0.01}\) | \(0.82 \pm 0.02\downarrow \) | \(\mathbf {0.90 \pm 0.01}\) | \(0.89 \pm 0.02\) | \(0.89 \pm 0.0\downarrow \) | |
Optdigits | \(0.97 \pm 0.0\downarrow \) | \(0.88 \pm 0.13\downarrow \) | \(\mathbf {1.0 \pm 0.0}\) | \(0.99 \pm 0.0\) | \(\mathbf {1.0 \pm 0.0}\) | |
Pendigits | \(0.98 \pm 0.04\) | \(0.98 \pm 0.01\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | |
Satellite | \(0.69 \pm 0.01\downarrow \) | \(0.74 \pm 0.02\downarrow \) | \(\mathbf {0.91 \pm 0.03}\) | \(0.84 \pm 0.04\downarrow \) | \(0.82 \pm 0.02\downarrow \) | |
SatImage | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | \(0.97 \pm 0.01\downarrow \) | \(0.96 \pm 0.01\downarrow \) | \(0.98 \pm 0.0\downarrow \) | |
Thyroid | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | \(0.97 \pm 0.01\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | |
Vowels | \(0.95 \pm 0.02\downarrow \) | \(0.92 \pm 0.04\downarrow \) | \(0.97 \pm 0.02\downarrow \) | \(0.96 \pm 0.02\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | |
Yeast | \(\mathbf {0.72 \pm 0.06}\) | \(0.71 \pm 0.06\) | \(0.55 \pm 0.06\) | \(0.55 \pm 0.08\) | \(0.55 \pm 0.06\) | |
Avg Rank | \(\mathbf {1.91}\) \(\mathbf {(1)}\) | 3.16 (3) | 3.0 (2) | 3.33 (4) | 3.58 (5) | |
b = 100 | Synthetic1 | \(\mathbf {0.99 \pm 0.0}\) | \(0.92 \pm 0.01\) | \(0.65 \pm 0.10\) | \(0.82 \pm 0.06\) | \(0.66 \pm 0.14\) |
Synthetic2 | \(\mathbf {0.96 \pm 0.02}\) | \(0.88 \pm 0.03\downarrow \) | \(0.50 \pm 0.08\downarrow \) | \(0.62 \pm 0.07\downarrow \) | \(0.74 \pm 0.15\downarrow \) | |
Synthetic3 | \(\mathbf {0.98 \pm 0.01}\) | \(0.80 \pm 0.04\downarrow \) | \(0.42 \pm 0.07\downarrow \) | \(0.76 \pm 0.08\downarrow \) | \(0.41 \pm 0.05\downarrow \) | |
Synthetic4 | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(0.69 \pm 0.07\downarrow \) | \(0.82 \pm 0.12\downarrow \) | \(0.89 \pm 0.10\downarrow \) | |
El Nino | \(\mathbf {0.98 \pm 0.0}\) | \(0.86 \pm 0.01\downarrow \) | \(0.71 \pm 0.10\downarrow \) | \(0.71 \pm 0.06\downarrow \) | \(0.50 \pm 0.07\downarrow \) | |
Houses | \(\mathbf {0.98 \pm 0.0}\) | \(0.88 \pm 0.02\downarrow \) | \(0.59 \pm 0.22\downarrow \) | \(0.86 \pm 0.03\downarrow \) | \(0.72 \pm 0.07\downarrow \) | |
Abalone | \(\mathbf {0.95 \pm 0.02}\) | \(0.93 \pm 0.05\) | \(0.58 \pm 0.09\downarrow \) | \(0.55 \pm 0.07\downarrow \) | \(0.41 \pm 0.07\downarrow \) | |
Annthyroid | \(\mathbf {0.98 \pm 0.0}\) | \(0.83 \pm 0.05\downarrow \) | \(\mathbf {0.98 \pm 0.02}\) | \(0.74 \pm 0.01\downarrow \) | \(0.86 \pm 0.11\downarrow \) | |
Arrhythmia | \(\mathbf {0.83 \pm 0.04}\) | \(0.43 \pm 0.05\downarrow \) | \(0.75 \pm 0.05\downarrow \) | \(0.59 \pm 0.08\downarrow \) | \(0.31 \pm 0.02\downarrow \) | |
Letters | \(\mathbf {0.76 \pm 0.04}\) | \(0.69 \pm 0.06\downarrow \) | \(0.74 \pm 0.04\) | \(0.67 \pm 0.05\downarrow \) | \(0.67 \pm 0.04\downarrow \) | |
Mammography | \(\mathbf {0.91 \pm 0.01}\) | \(0.85 \pm 0.03\downarrow \) | \(0.90 \pm 0.02\) | \(0.87 \pm 0.02\downarrow \) | \(0.89 \pm 0.05\) | |
Optdigits | \(0.97 \pm 0.0\downarrow \) | \(0.99 \pm 0.03\) | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | |
Pendigits | \(0.99 \pm 0.0\) | \(0.99 \pm 0.0\) | \(0.99 \pm 0.0\) | \(0.99 \pm 0.0\) | \(\mathbf {1.0 \pm 0.0}\) | |
Satellite | \(0.70 \pm 0.0\downarrow \) | \(0.75 \pm 0.01\downarrow \) | \(\mathbf {0.91 \pm 0.04}\) | \(\mathbf {0.91 \pm 0.04}\) | \(0.85 \pm 0.0\downarrow \) | |
SatImage | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | \(0.98 \pm 0.01\) | \(0.96 \pm 0.02\downarrow \) | \(0.98 \pm 0.01\) | |
Thyroid | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | \(0.98 \pm 0.0\) | \(\mathbf {0.99 \pm 0.0}\) | |
Vowels | \(0.95 \pm 0.01\downarrow \) | \(0.95 \pm 0.02\downarrow \) | \(0.99 \pm 0.0\) | \(0.96 \pm 0.06\) | \(\mathbf {1.0 \pm 0.0}\) | |
Yeast | \(0.72 \pm 0.06\) | \(\mathbf {0.75 \pm 0.06}\) | \(0.54 \pm 0.09\downarrow \) | \(0.59 \pm 0.04\downarrow \) | \(0.57 \pm 0.07\downarrow \) | |
Avg Rank | \(\mathbf {2.0}\) \(\mathbf {(1)}\) | 2.77 (2) | 3.22 (3) | 3.72 (5) | 3.27 (4) |
Dataset | WisCon | ConOut | ROCOD | CAD | IF-Con | LOF-Con | OCSVM-Con | IF | LOF | OCVM | LODA | SOD | FB |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Synthetic1 | \(\mathbf {0.88 \pm 0.01}\) | \(0.16 \pm 0.03\downarrow \) | \(0.76 \pm 0.01\downarrow \) | \(0.35 \pm 0.18\downarrow \) | \(0.81 \pm 0.0\downarrow \) | \(0.78 \pm 0.02\downarrow \) | \(0.67 \pm 0.05\downarrow \) | \(0.47 \pm 0.03\downarrow \) | \(0.77 \pm 0.0\downarrow \) | \(0.52 \pm 0.05\downarrow \) | \(0.12 \pm 0.02\downarrow \) | \(0.13 \pm 0.03\downarrow \) | \(0.20 \pm 0.01\downarrow \) |
Synthetic2 | \(\mathbf {0.72 \pm 0.06}\) | \(0.12 \pm 0.03\downarrow \) | \(0.61 \pm 0.08\downarrow \) | \(0.35 \pm 0.15\downarrow \) | \(0.50 \pm 0.07\downarrow \) | \(0.55 \pm 0.10\downarrow \) | \(0.50 \pm 0.12\downarrow \) | \(0.10 \pm 0.04\downarrow \) | \(0.61 \pm 0.08\downarrow \) | \(0.40 \pm 0.08\downarrow \) | \(0.06 \pm 0.02\downarrow \) | \(0.14 \pm 0.05\downarrow \) | \(0.42 \pm 0.06\downarrow \) |
Synthetic3 | \(0.86 \pm 0.04\downarrow \) | \(0.25 \pm 0.04\downarrow \) | \(0.81 \pm 0.04\downarrow \) | \(0.03 \pm 0.02\downarrow \) | \(0.67 \pm 0.05\downarrow \) | \(0.92 \pm 0.02\downarrow \) | \(\mathbf {0.94 \pm 0.03}\) | \(0.08 \pm 0.02\downarrow \) | \(0.64 \pm 0.08\downarrow \) | \(0.01 \pm 0.0\downarrow \) | \(0.06 \pm 0.04\downarrow \) | \(0.15 \pm 0.03\downarrow \) | \(0.46 \pm 0.05\downarrow \) |
Synthetic4 | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(0.76 \pm 0.27\downarrow \) | \(0.78 \pm 0.26\downarrow \) | \(0.96 \pm 0.06\) | \(0.75 \pm 0.16\downarrow \) | \(0.77 \pm 0.16\downarrow \) | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(0.99 \pm 0.01\) | \(0.87 \pm 0.07\downarrow \) |
El Nino | \(\mathbf {0.76 \pm 0.01}\) | – | \(0.63 \pm 0.01\downarrow \) | \(0.42 \pm 0.02\downarrow \) | \(0.62 \pm 0.05\downarrow \) | \(0.71 \pm 0.01\downarrow \) | \(0.74 \pm 0.0\downarrow \) | \(0.23 \pm 0.01\downarrow \) | \(0.61 \pm 0.01\downarrow \) | \(0.23 \pm 0.01\downarrow \) | \(0.17 \pm 0.03\downarrow \) | \(0.43 \pm 0.01\downarrow \) | \(0.58 \pm 0.02\downarrow \) |
Houses | \(\mathbf {0.69 \pm 0.03}\) | \(0.21 \pm 0.02\) | \(0.68 \pm 0.01\) | \(0.19 \pm 0.10\downarrow \) | \(0.63 \pm 0.06\downarrow \) | \(0.17 \pm 0.02\downarrow \) | \(0.55 \pm 0.04\downarrow \) | \(0.06 \pm 0.01\downarrow \) | \(0.13 \pm 0.01\downarrow \) | \(0.10 \pm 0.02\downarrow \) | \(0.05 \pm 0.01\downarrow \) | \(0.08 \pm 0.01\downarrow \) | \(0.08 \pm 0.11\downarrow \) |
Abalone | \(\mathbf {0.81 \pm 0.09}\) | \(0.52 \pm 0.12\downarrow \) | \(0.23 \pm 0.01\downarrow \) | \(0.09 \pm 0.11\downarrow \) | \(0.66 \pm 0.11\downarrow \) | \(0.57 \pm 0.15\downarrow \) | \(0.47 \pm 0.24\downarrow \) | \(0.54 \pm 0.17\downarrow \) | \(0.27 \pm 0.06\downarrow \) | \(0.49 \pm 0.13\downarrow \) | \(0.07 \pm 0.07\downarrow \) | \(0.59 \pm 0.12\downarrow \) | \(0.38 \pm 0.09\downarrow \) |
Annthyroid | \(0.80 \pm 0.04\) | \(0.32 \pm 0.02\downarrow \) | \(0.58 \pm 0.03\downarrow \) | \(0.53 \pm 0.06\downarrow \) | \(0.80 \pm 0.02\downarrow \) | \(0.72 \pm 0.03\downarrow \) | \(\mathbf {0.82 \pm 0.03}\) | \(0.34 \pm 0.04\downarrow \) | \(0.26 \pm 0.04\downarrow \) | \(0.18 \pm 0.01\downarrow \) | \(0.17 \pm 0.03\downarrow \) | \(0.41 \pm 0.02\downarrow \) | \(0.09 \pm 0.0\downarrow \) |
Arrhythmia | \(\mathbf {0.60 \pm 0.09}\) | – | \(0.45 \pm 0.13\downarrow \) | \(0.19 \pm 0.04\downarrow \) | \(0.54 \pm 0.08\) | \(0.57 \pm 0.07\) | \(0.47 \pm 0.06\downarrow \) | \(0.47 \pm 0.11\downarrow \) | \(0.47 \pm 0.08\downarrow \) | \(0.14 \pm 0.0\downarrow \) | \(0.47 \pm 0.10\downarrow \) | \(0.46 \pm 0.09\downarrow \) | \(0.49 \pm 0.06\downarrow \) |
Letters | \(0.36 \pm 0.08\downarrow \) | \(0.19 \pm 0.03\downarrow \) | \(0.22 \pm 0.04\downarrow \) | \(0.07 \pm 0.01\downarrow \) | \(0.33 \pm 0.08\downarrow \) | \(0.31 \pm 0.05\downarrow \) | \(0.34 \pm 0.07\downarrow \) | \(0.09 \pm 0.01\downarrow \) | \(0.33 \pm 0.04\downarrow \) | \(\mathbf {0.50 \pm 0.12}\) | \(0.07 \pm 0.01\downarrow \) | \(0.31 \pm 0.05\downarrow \) | \(0.34 \pm 0.14\downarrow \) |
Mammog | \(\mathbf {0.50 \pm 0.05}\) | \(0.21 \pm 0.06\downarrow \) | \(0.39 \pm 0.02\downarrow \) | \(0.14 \pm 0.12\downarrow \) | \(0.45 \pm 0.03\downarrow \) | \(0.02 \pm 0.0\downarrow \) | \(0.47 \pm 0.05\) | \(0.23 \pm 0.04\downarrow \) | \(0.18 \pm 0.03\downarrow \) | \(0.24 \pm 0.03\downarrow \) | \(0.18 \pm 0.08\downarrow \) | \(0.18 \pm 0.04\downarrow \) | \(0.04 \pm 0.0\downarrow \) |
Optdigits | \(0.62 \pm 0.08\downarrow \) | – | \(\mathbf {0.76 \pm 0.03}\) | \(0.33 \pm 0.27\downarrow \) | \(0.40 \pm 0.05\downarrow \) | \(0.30 \pm 0.02\downarrow \) | \(0.50 \pm 0.05\downarrow \) | \(0.05 \pm 0.01\downarrow \) | \(0.07 \pm 0.02\downarrow \) | \(0.02 \pm 0.0\downarrow \) | \(0.06 \pm 0.05\downarrow \) | \(0.05 \pm 0.0\downarrow \) | \(0.07 \pm 0.02\downarrow \) |
Pendigits | \(\mathbf {0.72 \pm 0.09}\) | \(0.34 \pm 0.06\) | \(0.52 \pm 0.05\downarrow \) | \(0.25 \pm 0.16\downarrow \) | \(0.41 \pm 0.07\downarrow \) | \(0.10 \pm 0.04\downarrow \) | \(0.52 \pm 0.18\downarrow \) | \(0.31 \pm 0.05\downarrow \) | \(0.03 \pm 0.0\downarrow \) | \(0.32 \pm 0.03\downarrow \) | \(0.26 \pm 0.09\downarrow \) | \(0.27 \pm 0.03\downarrow \) | \(0.04 \pm 0.01\downarrow \) |
Satellite | \(0.60 \pm 0.04\downarrow \) | \(\mathbf {0.68 \pm 0.01}\) | \(0.61 \pm 0.01\downarrow \) | \(0.51 \pm 0.05\downarrow \) | \(0.62 \pm 0.01\downarrow \) | \(0.56 \pm 0.01\downarrow \) | \(\mathbf {0.68 \pm 0.03}\) | \(0.63 \pm 0.01\downarrow \) | \(0.38 \pm 0.01\downarrow \) | \(0.51 \pm 0.01\downarrow \) | \(0.63 \pm 0.01\downarrow \) | \(0.54 \pm 0.02\downarrow \) | \(0.37 \pm 0.0\downarrow \) |
SatImage | \(0.86 \pm 0.08\downarrow \) | \(0.93 \pm 0.04\downarrow \) | \(0.96 \pm 0.02\downarrow \) | \(0.12 \pm 0.06\downarrow \) | \(0.90 \pm 0.05\downarrow \) | \(0.32 \pm 0.0\downarrow \) | \(0.96 \pm 0.03\) | \(0.92 \pm 0.03\downarrow \) | \(0.23 \pm 0.03\downarrow \) | \(\mathbf {0.97 \pm 0.02}\) | \(0.91 \pm 0.04\downarrow \) | \(0.90 \pm 0.02\downarrow \) | \(0.03 \pm 0.02\downarrow \) |
Thyroid | \(\mathbf {0.87 \pm 0.05}\) | \(0.63 \pm 0.05\downarrow \) | \(0.59 \pm 0.08\downarrow \) | \(0.11 \pm 0.02\downarrow \) | \(0.80 \pm 0.03\downarrow \) | \(0.69 \pm 0.08\downarrow \) | \(0.48 \pm 0.09\downarrow \) | \(0.66 \pm 0.09\downarrow \) | \(0.24 \pm 0.03\downarrow \) | \(0.15 \pm 0.03\downarrow \) | \(0.25 \pm 0.08\downarrow \) | \(0.55 \pm 0.13\downarrow \) | \(0.02 \pm 0.0\downarrow \) |
Vowels | \(\mathbf {0.65 \pm 0.08}\) | \(0.17 \pm 0.08\downarrow \) | \(0.42 \pm 0.04\downarrow \) | \(0.12 \pm 0.04\downarrow \) | \(0.48 \pm 0.07\downarrow \) | \(0.53 \pm 0.10\downarrow \) | \(0.43 \pm 0.09\downarrow \) | \(0.18 \pm 0.07\downarrow \) | \(0.41 \pm 0.10\downarrow \) | \(0.54 \pm 0.10\downarrow \) | \(0.06 \pm 0.03\downarrow \) | \(0.34 \pm 0.09\downarrow \) | \(0.35 \pm 0.12\downarrow \) |
Yeast | \(\mathbf {0.41 \pm 0.10}\) | \(0.28 \pm 0.10\downarrow \) | \(0.28 \pm 0.07\downarrow \) | \(0.23 \pm 0.06\downarrow \) | \(0.38 \pm 0.09\) | \(0.22 \pm 0.08\downarrow \) | \(0.28 \pm 0.09\downarrow \) | \(0.21 \pm 0.06\downarrow \) | \(0.26 \pm 0.12\downarrow \) | \(0.19 \pm 0.05\downarrow \) | \(0.16 \pm 0.10\downarrow \) | \(0.22 \pm 0.08\downarrow \) | \(0.16 \pm 0.06\downarrow \) |
Avg rank | \(\mathbf {2.22}\) \(\mathbf {(1)}\) | 7.77 (7) | 5.52 (4) | 9.83 (11) | 4.05 (2) | 6.22 (5) | 4.27 (3) | 7.88 (8) | 7.58 (6) | 7.88 (8) | 10.02 (12) | 8.36 (9) | 9.33 (10) |
Dataset | WisCon | ConOut | ROCOD | CAD | IF-Con | LOF-Con | OCSVM-Con | IF | LOF | OCVM | LODA | SOD | FB |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Synthetic1 | \(\mathbf {0.99 \pm 0.0}\) | \(0.85 \pm 0.01\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | \(0.77 \pm 0.20\downarrow \) | \(0.98 \pm 0.0\downarrow \) | \(0.92 \pm 0.0\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | \(0.89 \pm 0.02\downarrow \) | \(0.96 \pm 0.0\downarrow \) | \(0.95 \pm 0.0\downarrow \) | \(0.88 \pm 0.02\downarrow \) | \(0.72 \pm 0.02\downarrow \) | \(0.93 \pm 0.01\downarrow \) |
Synthetic2 | \(\mathbf {0.96 \pm 0.02}\) | \(0.79 \pm 0.02\downarrow \) | \(0.94 \pm 0.0\downarrow \) | \(0.81 \pm 0.12\downarrow \) | \(0.94 \pm 0.01\downarrow \) | \(0.93 \pm 0.02\downarrow \) | \(0.94 \pm 0.02\downarrow \) | \(0.82 \pm 0.03\downarrow \) | \(0.93 \pm 0.01\downarrow \) | \(0.93 \pm 0.01\downarrow \) | \(0.76 \pm 0.04\downarrow \) | \(0.78 \pm 0.04\downarrow \) | \(0.91 \pm 0.01\downarrow \) |
Synthetic3 | \(0.98 \pm 0.01\) | \(0.85 \pm 0.02\downarrow \) | \(0.98 \pm 0.0\downarrow \) | \(0.57 \pm 0.04\downarrow \) | \(0.97 \pm 0.01\downarrow \) | \(0.98 \pm 0.01\) | \(\mathbf {0.99 \pm 0.0}\) | \(0.79 \pm 0.04\downarrow \) | \(0.95 \pm 0.01\downarrow \) | \(0.50 \pm 0.0\downarrow \) | \(0.72 \pm 0.05\downarrow \) | \(0.65 \pm 0.04\downarrow \) | \(0.93 \pm 0.0\downarrow \) |
Synthetic4 | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(0.98 \pm 0.0\downarrow \) | \(0.70 \pm 0.18\downarrow \) | \(0.98 \pm 0.03\downarrow \) | \(0.83 \pm 0.14\downarrow \) | \(0.96 \pm 0.01\downarrow \) | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(\mathbf {1.0 \pm 0.0}\) | \(0.99 \pm 0.01\) | \(0.97 \pm 0.04\downarrow \) |
El Nino | \(\mathbf {0.98 \pm 0.0}\) | – | \(0.96 \pm 0.0\downarrow \) | \(0.78 \pm 0.5\downarrow \) | \(0.96 \pm 0.0\downarrow \) | \(0.91 \pm 0.0\downarrow \) | \(0.93 \pm 0.0\downarrow \) | \(0.83 \pm 0.01\downarrow \) | \(0.97 \pm 0.0\downarrow \) | \(0.96 \pm 0.01\downarrow \) | \(0.82 \pm 0.05\downarrow \) | \(0.92 \pm 0.01\downarrow \) | \(0.97 \pm 0.0\downarrow \) |
Houses | \(\mathbf {0.98 \pm 0.0}\) | \(0.83 \pm 0.01\downarrow \) | \(\mathbf {0.98 \pm 0.0}\) | \(0.87 \pm 0.03\downarrow \) | \(0.96 \pm 0.02\downarrow \) | \(0.93 \pm 0.01\downarrow \) | \(0.95 \pm 0.0\downarrow \) | \(0.75 \pm 0.02\downarrow \) | \(0.71 \pm 0.01\downarrow \) | \(0.93 \pm 0.0\downarrow \) | \(0.77 \pm 0.08\downarrow \) | \(0.70 \pm 0.03\downarrow \) | \(0.75 \pm 0.01\downarrow \) |
Abalone | \(\mathbf {0.95 \pm 0.02}\) | \(0.93 \pm 0.01\) | \(0.84 \pm 0.05\downarrow \) | \(0.53 \pm 0.14\downarrow \) | \(0.94 \pm 0.04\) | \(0.94 \pm 0.02\) | \(0.94 \pm 0.03\) | \(0.84 \pm 0.08\downarrow \) | \(0.89 \pm 0.03\downarrow \) | \(0.78 \pm 0.05\downarrow \) | \(0.79 \pm 0.06\downarrow \) | \(0.88 \pm 0.06\downarrow \) | \(0.90 \pm 0.05\downarrow \) |
Annthyroid | \(\mathbf {0.98 \pm 0.0}\) | \(0.81 \pm 0.01\downarrow \) | \(0.94 \pm 0.0\downarrow \) | \(0.96 \pm 0.01\downarrow \) | \(0.97 \pm 0.0\) | \(0.93 \pm 0.01\downarrow \) | \(0.97 \pm 0.0\) | \(0.82 \pm 0.03\downarrow \) | \(0.66 \pm 0.01\downarrow \) | \(0.61 \pm 0.01\downarrow \) | \(0.64 \pm 0.03\downarrow \) | \(0.73 \pm 0.03\downarrow \) | \(0.47 \pm 0.0\downarrow \) |
Arrhythmia | \(\mathbf {0.83 \pm 0.04}\) | – | \(0.63 \pm 0.07\downarrow \) | \(0.51 \pm 0.04\downarrow \) | \(0.80 \pm 0.02\downarrow \) | \(0.79 \pm 0.05\downarrow \) | \(0.76 \pm 0.04\downarrow \) | \(0.77 \pm 0.04\downarrow \) | \(0.77 \pm 0.04\downarrow \) | \(0.50 \pm 0.0\downarrow \) | \(0.76 \pm 0.04\downarrow \) | \(0.74 \pm 0.03\downarrow \) | \(0.78 \pm 0.03\downarrow \) |
Letters | \(0.76 \pm 0.04\downarrow \) | \(0.58 \pm 0.01\downarrow \) | \(0.70 \pm 0.03\downarrow \) | \(0.49 \pm 0.08\downarrow \) | \(0.76 \pm 0.04\downarrow \) | \(0.79 \pm 0.02\downarrow \) | \(0.81 \pm 0.03\downarrow \) | \(0.59 \pm 0.05\downarrow \) | \(\mathbf {0.87 \pm 0.02}\) | \(0.67 \pm 0.05\downarrow \) | \(0.50 \pm 0.03\downarrow \) | \(0.86 \pm 0.01\) | \(0.83 \pm 0.09\) |
Mammog | \(\mathbf {0.91 \pm 0.01}\) | \(0.84 \pm 0.02\downarrow \) | \(0.86 \pm 0.02\downarrow \) | \(0.82 \pm 0.03\downarrow \) | \(0.88 \pm 0.0\downarrow \) | \(0.79 \pm 0.03\downarrow \) | \(0.33 \pm 0.02\downarrow \) | \(0.85 \pm 0.01\downarrow \) | \(0.49 \pm 0.0\downarrow \) | \(0.56 \pm 0.01\downarrow \) | \(0.69 \pm 0.14\downarrow \) | \(0.84 \pm 0.03\downarrow \) | \(0.49 \pm 0.09\downarrow \) |
Optdigits | \(\mathbf {0.98 \pm 0.0}\) | – | \(0.97 \pm 0.01\) | \(0.83 \pm 0.23\downarrow \) | \(0.95 \pm 0.0\downarrow \) | \(0.95 \pm 0.0\downarrow \) | \(0.96 \pm 0.0\downarrow \) | \(0.67 \pm 0.02\downarrow \) | \(0.59 \pm 0.05\downarrow \) | \(0.40 \pm 0.01\downarrow \) | \(0.47 \pm 0.04\downarrow \) | \(0.56 \pm 0.09\downarrow \) | \(0.59 \pm 0.03\downarrow \) |
Pendigits | \(\mathbf {0.99 \pm 0.0}\) | \(0.96 \pm 0.0\downarrow \) | \(0.98 \pm 0.0\downarrow \) | \(0.89 \pm 0.04\downarrow \) | \(0.96 \pm 0.0\downarrow \) | \(0.77 \pm 0.06\downarrow \) | \(0.81 \pm 0.01\downarrow \) | \(0.95 \pm 0.0\downarrow \) | \(0.59 \pm 0.02\downarrow \) | \(0.96 \pm 0.0\downarrow \) | \(0.86 \pm 0.07\downarrow \) | \(0.84 \pm 0.02\downarrow \) | \(0.51 \pm 0.04\downarrow \) |
Satellite | \(0.70 \pm 0.0\downarrow \) | \(0.71 \pm 0.01\downarrow \) | \(0.74 \pm 0.01\downarrow \) | \(0.69 \pm 0.06\downarrow \) | \(0.73 \pm 0.01\downarrow \) | \(0.70 \pm 0.01\downarrow \) | \(\mathbf {0.79 \pm 0.0}\) | \(0.72 \pm 0.02\downarrow \) | \(0.59 \pm 0.01\downarrow \) | \(0.67 \pm 0.0\downarrow \) | \(0.74 \pm 0.01\downarrow \) | \(0.67 \pm 0.01\downarrow \) | \(0.56 \pm 0.01\downarrow \) |
SatImage | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | \(0.89 \pm 0.06\downarrow \) | \(0.89 \pm 0.02\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | \(0.47 \pm 0.01\downarrow \) | \(\mathbf {0.99 \pm 0.01}\) | \(0.98 \pm 0.01\) | \(0.97 \pm 0.0\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | \(\mathbf {0.99 \pm 0.0}\) | \(0.90 \pm 0.02\downarrow \) | \(0.53 \pm 0.04\downarrow \) |
Thyroid | \(\mathbf {0.99 \pm 0.0}\) | \(0.95 \pm 0.01\downarrow \) | \(0.95 \pm 0.01\downarrow \) | \(0.77 \pm 0.24\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | \(0.97 \pm 0.0\downarrow \) | \(0.95 \pm 0.03\downarrow \) | \(\mathbf {0.99 \pm 0.0}\) | \(0.95 \pm 0.01\downarrow \) | \(0.79 \pm 0.03\downarrow \) | \(0.95 \pm 0.01\downarrow \) | \(0.92 \pm 0.04\downarrow \) | \(0.47 \pm 0.0\downarrow \) |
Vowels | \(\mathbf {0.95 \pm 0.01}\) | \(0.77 \pm 0.03\downarrow \) | \(0.91 \pm 0.03\downarrow \) | \(0.86 \pm 0.08\downarrow \) | \(0.93 \pm 0.05\) | \(0.92 \pm 0.0\downarrow \) | \(0.92 \pm 0.02\) | \(0.67 \pm 0.07\downarrow \) | \(0.93 \pm 0.01\downarrow \) | \(0.92 \pm 0.03\downarrow \) | \(0.67 \pm 0.05\downarrow \) | \(0.88 \pm 0.02\downarrow \) | \(0.93 \pm 0.01\downarrow \) |
Yeast | \(\mathbf {0.72 \pm 0.06}\) | \(0.63 \pm 0.04\downarrow \) | \(0.63 \pm 0.06\downarrow \) | \(0.60 \pm 0.08\downarrow \) | \(0.70 \pm 0.05\) | \(0.60 \pm 0.04\downarrow \) | \(0.70 \pm 0.06\) | \(0.59 \pm 0.07\downarrow \) | \(0.68 \pm 0.05\) | \(0.71 \pm 0.05\) | \(\mathbf {0.72 \pm 0.07}\) | \(0.61 \pm 0.08\downarrow \) | \(0.56 \pm 0.01\downarrow \) |
Avg Rank | \(\mathbf {2.25}\) \(\mathbf {(1)}\) | 8.27 (9) | 5.41 (4) | 9.94 (13) | 3.88 (2) | 6.94 (6) | 5.16 (3) | 7.69 (7) | 7.02 (5) | 8.00 (8) | 8.50 (10) | 9.11 (12) | 8.77 (11) |
6.5 Results Q2: comparison of the query strategies
6.6 Results Q3: justification of ensembles and active learning
Dataset | WisCon | Single context | Unsupervised ensemble | ||
---|---|---|---|---|---|
WisCon-single | Wiscon-true | Averaging | Maximization | ||
Synthetic 1 | \(\mathbf {0.88 \pm 0.01}\) | \(0.81 \pm 0.0\) | \(0.81 \pm 0.0\) | \(0.78 \pm 0.01\) | \(0.39 \pm 0.0\) |
Synthetic 2 | \(\mathbf {0.72 \pm 0.06}\) | \(0.27 \pm 0.14\) | \(0.50 \pm 0.07\) | \(0.40 \pm 0.08\) | \(0.20 \pm 0.05\) |
Synthetic 3 | \(\mathbf {0.86 \pm 0.04}\) | \(0.30 \pm 0.14\) | \(0.67 \pm 0.05\) | \(0.53 \pm 0.04\) | \(0.33 \pm 0.06\) |
Synthetic 4 | \(\mathbf {1.0 \pm 0.0}\) | \(0.70 \pm 0.12\) | \(0.96 \pm 0.06\) | \(0.77 \pm 0.05\) | \(0.99 \pm 0.01\) |
El Nino | \(\mathbf {0.76 \pm 0.01}\) | \(0.71 \pm 0.02\) | \(0.62 \pm 0.05\) | \(0.42 \pm 0.01\) | \(0.35 \pm 0.03\) |
Houses | \(\mathbf {0.69 \pm 0.04}\) | \(0.39 \pm 0.08\) | \(0.63 \pm 0.06\) | \(0.12 \pm 0.18\) | \(0.08 \pm 0.21\) |
Abalone | \(\mathbf {0.81 \pm 0.09}\) | \(0.62 \pm 0.24\) | \(0.66 \pm 0.11\) | \(0.56 \pm 0.13\) | \(0.25 \pm 0.12\) |
Ann-Thyroid | \(\mathbf {0.80 \pm 0.04}\) | \(\mathbf {0.80 \pm 0.04}\) | \(\mathbf {0.80 \pm 0.04}\) | \(0.32 \pm 0.01\) | \(0.29 \pm 0.02\) |
Arrhythmia | \(\mathbf {0.60 \pm 0.09}\) | \(\mathbf {0.60 \pm 0.09}\) | \(0.54 \pm 0.08\) | \(0.44 \pm 0.08\) | \(0.48 \pm 0.12\) |
Letter | \(\mathbf {0.36 \pm 0.08}\) | \(0.28 \pm 0.09\) | \(0.33 \pm 0.08\) | \(0.17 \pm 0.03\) | \(0.23 \pm 0.04\) |
Mammography | \(\mathbf {0.50 \pm 0.05}\) | \(0.45 \pm 0.03\) | \(0.45 \pm 0.03\) | \(0.14 \pm 0.02\) | \(0.09 \pm 0.0\) |
Optdigits | \(\mathbf {0.62 \pm 0.08}\) | \(0.15 \pm 0.12\) | \(0.40 \pm 0.05\) | \(0.07 \pm 0.01\) | \(0.05 \pm 0.01\) |
Pendigits | \(\mathbf {0.72 \pm 0.09}\) | \(0.32 \pm 0.07\) | \(0.41 \pm 0.07\) | \(0.14 \pm 0.03\) | \(0.10 \pm 0.02\) |
Satellite | \(0.60 \pm 0.04\) | \(0.59 \pm 0.01\) | \(\mathbf {0.62 \pm 0.01}\) | \(0.46 \pm 0.13\) | \(0.50 \pm 0.01\) |
Satimage | \(0.86 \pm 0.08\) | \(0.29 \pm 0.28\) | \(\mathbf {0.90 \pm 0.05}\) | \(0.07 \pm 0.03\) | \(0.36 \pm 0.03\) |
Thyroid | \(\mathbf {0.89 \pm 0.05}\) | \(0.74 \pm 0.04\) | \(0.80 \pm 0.03\) | \(0.50 \pm 0.08\) | \(0.37 \pm 0.04\) |
Vowels | \(\mathbf {0.65 \pm 0.08}\) | \(0.42 \pm 0.04\) | \(0.48 \pm 0.06\) | \(0.38 \pm 0.06\) | \(0.17 \pm 0.03\) |
Yeast | \(\mathbf {0.41 \pm 0.10}\) | \(0.30 \pm 0.09\) | \(0.38 \pm 0.09\) | \(0.14 \pm 0.03\) | \(0.30 \pm 0.09\) |