1 Introduction
- The proposition of a novel mutual diversity measure based on the non-pairwise and averaged pairwise diversity, which allows to evaluate the impact of a particular predictor on a given classifier ensemble diversity. Thus, it could be used as the criterion for ensemble pruning.
- The formalization of an algorithm that uses the proposed measure for ensemble pruning and multistage organization of majority voting.
- An extensive experimental analysis on a large number of benchmark datasets comparing the performance of proposed methods and the state-of-the-art ensemble methods which are backed up by the statistical tests.
2 Related works
- Ranking-based pruning chooses a fixed number of the best-ranked individual classifiers according to a given metric (as kappa statistics) [24].
- Clustering-based pruning looks for groups of base classifiers, where individuals in the same group behave similarly while different groups have large diversity. Then, from each cluster, the representative is selected, which is placed in the final ensemble.
2.1 Ensemble diversity
\(\varPsi _k\) correct (1) | \(\varPsi _k\) wrong (0) | |
---|---|---|
\(\varPsi _i\) correct (1) | \(N^{11}\) | \(N^{10}\) |
\(\varPsi _i\) wrong (0) | \(N^{01}\) | \(N^{00}\) |
3 Proposed methods
3.1 Clustering criterion
3.2 Diversity-based one-dimensional clustering space and cluster pruning
3.3 Two-step majority voting organization
4 Experimental study
Dataset | Instances | Features | Classes | Dataset | Instances | Features | Classes |
---|---|---|---|---|---|---|---|
Appendicitis | 106 | 7 | 2 | NewThyroid | 215 | 5 | 3 |
Australian | 690 | 14 | 2 | Pima | 768 | 8 | 2 |
Bands | 365 | 19 | 2 | Saheart | 462 | 9 | 2 |
Bupa | 345 | 6 | 2 | Segment | 2310 | 19 | 7 |
Cleveland | 303 | 13 | 5 | Sonar | 208 | 60 | 2 |
Contraceptive | 1473 | 9 | 3 | Spambase | 4596 | 57 | 2 |
Dermatology | 366 | 7 | 8 | Spectfheart | 267 | 44 | 2 |
Ecoli | 336 | 7 | 8 | Vehicle | 846 | 18 | 4 |
Glass | 214 | 9 | 7 | Vowel | 990 | 13 | 11 |
Heart | 270 | 13 | 2 | wdbc | 596 | 30 | 2 |
HouseVotes | 232 | 16 | 2 | Wine | 178 | 13 | 3 |
ILPD | 583 | 10 | 2 | WineRed | 1599 | 11 | 11 |
Ionosphere | 351 | 33 | 2 | Winconsin | 683 | 9 | 2 |
Libras | 360 | 90 | 15 | Yeast | 1484 | 8 | 10 |
MuskV1 | 476 | 166 | 2 | ZOO | 101 | 16 | 7 |
- Which set of parameters (approach, diversity measure, base learner type, number of clusters) yields the best results for the given dataset?
- How the number of clusters affects the performance of methods?
- Does the proposed ensemble pruning and multistage organization methods lead to improvements in accuracy over state-of-the-art methods?
4.1 Datasets
4.2 Setup
- Approach MV—majority voting, Aggr—aggregation of probabilities, Mo—multistage voting organization, MoR—multistage voting using sampling with replacement and Pr—clustering-based pruning,
- Classifier Mlp—Multilayer perceptron, Cart—classification and regression trees, Nb—Gaussian naïve Bayes and Knn—k-nearest neighbors classifier,
- DiversityMeasure E—the entropy measure, KW—Kohavi–Wolpert variance, K—measurement of interrater agreement, Q—the averaged Q statistics and Dis—the averaged disagreement measure.
4.3 Statistical evaluation
Dataset | BestMethod | 2C | 3C | 4C | 5C | 6C | 7C | 8C | 9C | 10C |
---|---|---|---|---|---|---|---|---|---|---|
Appendicitis | PrCartQ | 90.56 | 91.52 | 90.56 | 89.65 | 89.61 | 88.66 | 90.56 | 89.61 | 89.61 |
Australian | PrCartQ | 82.47 | 88.41 | 86.24 | 89.57 | 88.7 | 90.73 | 89.58 | 90.88 | 90.15 |
Bands | PrCartK | 69.86 | 76.71 | 76.99 | 82.19 | 80.27 | 81.37 | 82.47 | 84.11 | 82.74 |
Bupa | PrMlpE | 70.43 | 73.91 | 74.78 | 72.75 | 74.78 | 74.78 | 73.91 | 76.23 | 74.2 |
Cleveland | PrCartKw | 62.28 | 62.95 | 62.58 | 62.95 | 63.62 | 64.3 | 66.34 | 63.31 | 62.32 |
Contraceptive | PrMlpQ | 55.67 | 56.89 | 56.21 | 56.28 | 56.08 | 56.21 | 56.83 | 57.71 | 56.62 |
Dermatology | PrMlpKw | 96.4 | 98.33 | 97.52 | 99.72 | 98.87 | 99.72 | 100.0 | 99.17 | 99.44 |
Ecoli | PrMlpE | 82.5 | 84.31 | 84.82 | 86.35 | 86.06 | 86.37 | 86.96 | 87.81 | 87.84 |
Glass | PrCartK | 74.72 | 84.07 | 80.98 | 81.3 | 81.73 | 84.13 | 85.43 | 86.85 | 85.95 |
Heart | PrCartKw | 77.78 | 87.04 | 82.96 | 87.41 | 84.81 | 90.0 | 87.78 | 86.67 | 87.04 |
HouseVotes | PrMlpQ | 97.86 | 96.58 | 97.44 | 96.56 | 96.58 | 94.83 | 96.14 | 93.98 | 94.85 |
ILPD | PrCartK | 76.49 | 74.61 | 75.12 | 74.1 | 73.41 | 74.78 | 74.1 | 74.27 | 74.27 |
Ionosphere | PrCartDis | 93.16 | 95.73 | 97.44 | 98.01 | 98.01 | 97.15 | 98.29 | 97.73 | 98.58 |
Libras | PrCartK | 72.87 | 76.2 | 80.8 | 83.33 | 85.4 | 84.07 | 86.33 | 85.87 | 87.13 |
MuskV1 | PrMlpKw | 86.57 | 89.48 | 90.34 | 93.91 | 92.44 | 96.02 | 95.18 | 95.81 | 96.44 |
NewThyroid | PrNbKw | 94.42 | 97.21 | 94.42 | 96.28 | 95.81 | 96.74 | 94.88 | 96.74 | – |
Pima | PrNbK | 74.87 | 78.39 | 75.26 | 79.95 | 76.04 | 76.95 | 75.91 | 76.17 | 76.3 |
Saheart | PrNbE | 75.31 | 76.61 | 75.96 | 77.69 | 77.26 | 75.96 | 76.39 | 75.32 | 75.09 |
Segment | PrCartQ | 96.71 | 97.92 | 98.27 | 98.48 | 98.53 | 98.61 | 98.61 | 98.7 | 98.74 |
Sonar | PrCartQ | 85.59 | 88.48 | 88.45 | 92.33 | 90.83 | 95.7 | 92.77 | 94.71 | 93.75 |
Spambase | PrCartQ | 89.45 | 93.08 | 92.73 | 93.82 | 93.65 | 94.56 | 94.41 | 94.98 | 94.39 |
Spectfheart | PrCartQ | 81.62 | 86.49 | 87.99 | 87.99 | 88.37 | 89.13 | 89.5 | 88.36 | 87.99 |
Vehicle | PrCartK | 72.22 | 77.31 | 78.26 | 80.98 | 80.27 | 81.21 | 81.33 | 82.04 | 82.05 |
Vowel | PrKnnE | 87.58 | 91.92 | 94.14 | 94.65 | 95.35 | 95.66 | 95.35 | 95.86 | 96.57 |
wdbc | PrCartE | 95.43 | 98.07 | 96.84 | 97.37 | 97.72 | 97.9 | 97.72 | 98.07 | – |
Wine | PrCartKw | 96.11 | 96.08 | 99.46 | 98.87 | 98.32 | 99.43 | 100.0 | 99.43 | 100.0 |
WineRed | PrCartK | 66.36 | 67.1 | 69.41 | 68.98 | 69.48 | 69.61 | 70.79 | 70.23 | 71.11 |
Wisconsin | PrCartQ | 97.37 | 98.1 | 98.68 | 98.54 | 98.68 | 98.39 | 98.68 | 98.68 | 98.83 |
Yeast | PrMlpE | 48.57 | 55.98 | 51.68 | 56.66 | 53.56 | 57.47 | 56.12 | 57.4 | 55.52 |
ZOO | PrCartKw | 96.99 | 99.05 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 |