1 Introduction
2 Related Work
3 Powershap
3.1 Powershap Algorithm
3.2 Automatic Mode
4 Experiments
4.1 Feature Selection Methods
4.2 Simulation Dataset
make_classification
function of sklearn. This function creates a classification dataset, however, exactly the same can be done for obtaining a regression dataset (by using make_regression
). The simulations are run using 20, 100, 250, and 500 total features to understand the performance on varying dimensions of feature sets. The ratio of informative features is varied as 10%, 33%, 50%, and 90% of the total feature set, allowing for assessing the quality of the selected features in terms of this ratio. The resulting simulation datasets each contain 5000 samples. Each simulation experiment was repeated five times with different random seeds. The number of redundant features, which are linear combinations of informative features, and the number of duplicate features were set to zero. Redundant features and duplicate features reduce the performance of models, but they cannot be discerned from true informative features as they are inherently informative. Therefore they are not included in the simulation dataset as the goal of powershap is to find informative features. The powershap method is compared to shapicant, chi\(^{2}\), borutashap, and the f-test for feature selection on this simulation dataset. Due to time complexity constraints, forward feature selection was not included in the simulation benchmarking.4.3 Benchmark Datasets
Dataset | Type | Source | # features | train size | test size |
---|---|---|---|---|---|
Madelon | Classification | OpenML | 500 | 1950 | 650 |
Gina priori | Classification | OpenML | 784 | 2601 | 867 |
Scene | Classification | OpenML | 294 | 1805 | 867 |
CT location | Regression | UCI | 384 | 41347 | 12153 |
Appliances | Regression | UCI | 30 | 14801 | 4934 |
5 Results
5.1 Simulation Dataset
5.2 Benchmark Datasets
Duration (s) | |||||||
---|---|---|---|---|---|---|---|
Dataset | powershap | borutashap | shapicant | forward | chi\(^{2}\) | f test | default |
Madelon | 132 s | 186 s | 632 s | 10483 s | \(<1s\) | \(<1s\) | N/A |
Gina priori | 184 s | 299 s | 812 s | 68845 s | \(<1s\) | \(<1s\) | N/A |
Scene | 115 s | 220 s | 749 s | 12496 s | \(<1s\) | \(<1s\) | N/A |
CT location | 459 s | 543 s | 1553 s | 56879 s | N/A | \(<1s\) | N/A |
Appliances | 34 s | 48 s | 134 s | 1913 s | N/A | \(<1s\) | N/A |
Selected features | |||||||
Madelon | 22 | 10 | 30 | 8 | 43 | 18 | 500 |
Gina priori | 105 | 37 | 106 | 26 | 328 | 405 | 784 |
Scene | 36 | 14 | 56 | 15 | 93 | 220 | 294 |
CT location | 123 | 162 | 74 | 75 | N/A | 350 | 384 |
Appliances | 24 | 24 | 10 | 13 | N/A | 20 | 30 |
6 Discussion
make_classification
function uses by default a hypercube to create its classification problem, resulting in a linear classification problem, which is inherently easier to classify [15]. The compared filter methods were chosen by their most common usage and availability, however, these are fast and simple methods and are of a much lower complexity than powershap. The same argument could be made for our choice of the forward feature selection method (as wrapper method) compared to other methods such as genetic algorithm based solutions. Furthermore, wrapper methods, and thus also powershap, are highly dependent on the used model, as the feature selection quality suffers from modeling issues such as for example overfitting and underfitting. Therefore, the true potential achievable performances on the benchmark datasets may differ since every use-case and dataset requires its own tuned model to achieve optimal performance. Additionally, the cut-off values and hyperparameters of none of the methods were optimized and are either set to the same value as in powershap or used with their default values. This might impact the performance and could have skewed the benchmark results in both directions. However, choosing the same model and the same values for hyperparameters (if possible) in all experiments, reduces potential performance differences and facilitates a fair enough comparison.