Introduction
Class imbalance background
The class imbalance problem
Performance metrics
Actual positive | Actual negative | |
---|---|---|
Predicted positive | True positive (TP) | False positive (FP) |
Predicted negative | False negative (FN) | True negative (TN) |
Machine learning techniques for class imbalanced data
Data-level methods
Algorithm-level methods
Actual positive | Actual negative | |
---|---|---|
Predicted positive |
\(C(1,1)= c_{11}\)
|
\(C(1,0)= c_{10}\)
|
Predicted negative |
\(C(0,1)= c_{01}\)
|
\(C(0,0)= c_{00}\)
|
Hybrid methods
Deep learning background
Introduction to deep learning
Representation learning
Deep learning milestones
Deep learning with big data
Deep learning methods for class imbalanced data
Data-level methods
Balancing training data with ROS
Total | Airplane | Automobile | Bird | Cat | Deer | Dog | Frog | Horse | Ship | Truck | |
---|---|---|---|---|---|---|---|---|---|---|---|
Dist. 1 | 0.73 | 0.78 | 0.84 | 0.62 | 0.57 | 0.70 | 0.62 | 0.80 | 0.76 | 0.84 | 0.80 |
Dist. 2 | 0.69 | 0.74 | 0.75 | 0.58 | 0.33 | 0.58 | 0.65 | 0.84 | 0.78 | 0.87 | 0.79 |
Dist. 3 | 0.66 | 0.71 | 0.75 | 0.59 | 0.30 | 0.52 | 0.61 | 0.79 | 0.77 | 0.85 | 0.73 |
Dist. 4 | 0.27 | 0.78 | 0.24 | 0.12 | 0.08 | 0.19 | 0.24 | 0.33 | 0.27 | 0.21 | 0.27 |
Dist. 5 | 0.10 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
Dist. 6 | 0.73 | 0.74 | 0.86 | 0.65 | 0.53 | 0.71 | 0.63 | 0.81 | 0.76 | 0.83 | 0.79 |
Dist. 7 | 0.73 | 0.75 | 0.86 | 0.66 | 0.52 | 0.71 | 0.63 | 0.80 | 0.78 | 0.84 | 0.79 |
Dist. 8 | 0.66 | 0.63 | 0.75 | 0.55 | 0.35 | 0.51 | 0.58 | 0.82 | 0.74 | 0.84 | 0.80 |
Dist. 9 | 0.10 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 |
Dist. 10 | 0.69 | 0.75 | 0.77 | 0.56 | 0.42 | 0.66 | 0.63 | 0.76 | 0.70 | 0.81 | 0.79 |
Dist. 11 | 0.69 | 0.74 | 0.82 | 0.58 | 0.44 | 0.59 | 0.64 | 0.80 | 0.69 | 0.83 | 0.81 |
Total | Airplane | Automobile | Bird | Cat | Deer | Dog | Frog | Horse | Ship | Truck | |
---|---|---|---|---|---|---|---|---|---|---|---|
Dist. 2 | 0.72 | 0.77 | 0.80 | 0.57 | 0.51 | 0.68 | 0.64 | 0.81 | 0.78 | 0.84 | 0.82 |
Dist. 3 | 0.73 | 0.73 | 0.80 | 0.59 | 0.53 | 0.63 | 0.65 | 0.82 | 0.81 | 0.87 | 0.83 |
Dist. 4 | 0.73 | 0.76 | 0.82 | 0.60 | 0.54 | 0.68 | 0.63 | 0.81 | 0.78 | 0.85 | 0.83 |
Dist. 5 | 0.73 | 0.80 | 0.84 | 0.61 | 0.55 | 0.68 | 0.63 | 0.82 | 0.79 | 0.82 | 0.81 |
Dist. 6 | 0.73 | 0.75 | 0.86 | 0.65 | 0.51 | 0.65 | 0.66 | 0.81 | 0.78 | 0.85 | 0.80 |
Dist. 7 | 0.73 | 0.73 | 0.85 | 0.62 | 0.51 | 0.71 | 0.66 | 0.81 | 0.79 | 0.86 | 0.80 |
Dist. 8 | 0.73 | 0.78 | 0.84 | 0.62 | 0.56 | 0.66 | 0.64 | 0.81 | 0.77 | 0.83 | 0.80 |
Dist. 9 | 0.72 | 0.73 | 0.84 | 0.58 | 0.55 | 0.68 | 0.59 | 0.79 | 0.78 | 0.83 | 0.82 |
Dist. 10 | 0.73 | 0.78 | 0.85 | 0.63 | 0.50 | 0.67 | 0.63 | 0.82 | 0.75 | 0.87 | 0.80 |
Dist. 11 | 0.72 | 0.82 | 0.87 | 0.58 | 0.64 | 0.64 | 0.49 | 0.78 | 0.69 | 0.84 | 0.80 |
Two-phase learning
Classifier | All classes | L5 | Rest |
---|---|---|---|
(A) Full | 0.1773 | 0.7773 | 0.1548 |
(B) Noise | 0.2465 | 0.5409 | 0.3599 |
(C) Aug | 0.2726 | 0.5776 | 0.3700 |
(D) Thresh | 0.3086 | 0.6510 | 0.4044 |
(E) Noise + full | 0.3038 | 0.7531 | 0.2971 |
(F) Aug + full | 0.3212 | 0.7668 | 0.3156 |
(G) Thresh + full | 0.3339 | 0.7791 | 0.3262 |
Dynamic sampling
Classifier | Acc. | Avg. F1 | WAvg. F1 |
---|---|---|---|
(A) Basic CNN | 0.649 | 0.254 | 0.630 |
(B) Deep CNN features + SVM | 0.746 | 0.528 | 0.747 |
(C) TL + no aug. | 0.765 | 0.432 | 0.755 |
(D) TL + basic aug. | 0.792 | 0.502 | 0.779 |
(E) TL + balanced aug. | 0.759 | 0.553 | 0.766 |
(F) Proposed model | 0.802 | 0.599 | 0.794 |
ROS, RUS, and two-phase learning
Summary of data-level methods
Algorithm-level methods
Mean false error (MFE) loss
Data set | Imbalance level (%) | F-measure | AUC | ||||
---|---|---|---|---|---|---|---|
MSE | MFE | MSFE | MSE | MFE | MSFE | ||
Household | 20 | 0.3913 |
0.4138
|
0.4271
| 0.7142 |
0.7397
|
0.7354
|
10 | 0.2778 |
0.2797
|
0.3151
| 0.7125 |
0.7179
|
0.7193
| |
5 | 0.1143 |
0.1905
|
0.2353
| 0.6714 |
0.6950
|
0.6970
| |
Tree 1 | 20 | 0.5500 | 0.5500 | 0.5366 | 0.8100 |
0.8140
|
0.8185
|
10 | 0.4211 | 0.4211 | 0.4211 | 0.7960 |
0.7990
|
0.7990
| |
5 | 0.1667 |
0.2353
|
0.2353
| 0.7920 |
0.8000
|
0.8000
| |
Tree 2 | 20 | 0.4348 | 0.4255 | 0.4255 | 0.8480 | 0.8450 | 0.8440 |
10 | 0.1818 |
0.2609
|
0.2500
| 0.8050 | 0.8050 |
0.8060
| |
5 | 0.0000 |
0.1071
|
0.1481
| 0.5480 |
0.6520
| 0.7000 |
Data set | Imbalance level (%) | F-measure | AUC | ||||
---|---|---|---|---|---|---|---|
MSE | MFE | MSFE | MSE | MFE | MSFE | ||
Doc. 1 | 20 | 0.2341 |
0.2574
|
0.2549
| 0.5948 |
0.5995
|
0.5987
|
10 | 0.1781 |
0.1854
|
0.1961
| 0.5349 |
0.5462
|
0.5469
| |
5 | 0.1356 |
0.1456
|
0.1456
| 0.5336 |
0.5436
|
0.5436
| |
Doc. 2 | 20 | 0.3408 | 0.3393 | 0.3393 | 0.6462 |
0.6464
|
0.6464
|
10 | 0.2094 | 0.2000 | 0.2000 | 0.6310 |
0.6319
|
0.6322
| |
5 | 0.1256 | 0.1171 |
0.1262
| 0.6273 |
0.6377
|
0.6431
| |
Doc. 3 | 20 | 0.2929 |
0.2957
|
0.2957
| 0.5862 |
0.5870
|
0.5870
|
10 | 0.1596 |
0.1627
|
0.1698
| 0.5577 |
0.5756
|
0.5865
| |
5 | 0.0941 |
0.1118
|
0.1084
| 0.5314 |
0.5399
|
0.5346
| |
Doc. 4 | 20 | 0.3723 |
0.3843
| 0.3668 | 0.6922 |
0.7031
|
0.7054
|
10 | 0.1159 |
0.2537
|
0.2574
| 0.5623 |
0.6802
|
0.6816
| |
5 | 0.1287 |
0.1720
|
0.1720
| 0.6041 |
0.6090
|
0.6090
| |
Doc. 5 | 20 | 0.3103 |
0.3222
|
0.3222
| 0.6011 | 0.5925 | 0.5925 |
10 | 0.1829 | 0.1808 |
0.1839
| 0.5777 |
0.5836
|
0.5837
| |
5 | 0.0946 |
0.1053
|
0.1053
| 0.5682 |
0.5730
|
0.5730
|
Focal loss
Backbone | AP |
\(AP_{50}\)
|
\(AP_{75}\)
|
\(AP_{S}\)
|
\(AP_{M}\)
|
\(AP_{L}\)
| |
---|---|---|---|---|---|---|---|
Two-stage methods | |||||||
Faster R-CNN+++ | ResNet-101-C4 | 34.9 | 55.7 | 37.4 | 15.6 | 38.7 | 50.9 |
Faster R-CNN w FPN | ResNet-101-FPN | 36.2 | 59.1 | 39.0 | 18.2 | 39.0 | 48.2 |
Faster R-CNN by G-RMI | Inception-ResNet-v2 | 34.7 | 55.5 | 36.7 | 13.5 | 38.1 | 52.0 |
Faster R-CNN w TDM | Inception-ResNet-v2-TDM | 36.8 | 57.7 | 39.2 | 16.2 | 39.8 |
52.1
|
One-stage methods | |||||||
YOLOv2 | DarkNet-19 | 21.6 | 44.0 | 19.2 | 5.0 | 22.4 | 35.5 |
SSD513 | ResNet-101-SSD | 31.2 | 50.4 | 33.3 | 10.2 | 34.5 | 49.8 |
DSSD513 | ResNet-101-DSSD | 33.2 | 53.3 | 35.2 | 13.0 | 35.4 | 51.1 |
RetinaNet | ResNet-101-FPN | 39.1 | 59.1 | 42.3 | 21.8 | 42.7 | 50.2 |
RetinaNet | ResNeXt-101-FPN |
40.8
|
61.1
|
44.1
|
24.1
|
44.2
| 51.2 |
Cost-sensitive deep neural network (CSDNN)
Learning cost matrices with cost-sensitive CNN (CoSen)
Dataset | Imbalance protocol | SMOTE (%) | RUS (%) | SMOTE RSB (%) | CoSen SVM (%) | CoSen RF (%) | SOSR CNN (%) | Baseline CNN (%) | CoSen CNN (%) |
---|---|---|---|---|---|---|---|---|---|
MNIST | 10% of odd classes | 94.5 | 92.1 | 96.0 | 96.8 | 96.3 | 97.8 | 97.6 |
98.6
|
CIFAR-100 | 10% of odd classes | 32.2 | 28.8 | 37.5 | 39.9 | 39.0 | 55.8 | 55.0 |
60.1
|
Caltech-101 | 10% of odd classes | 67.7 | 61.4 | 68.2 | 70.1 | 68.7 | 77.4 | 77.4 |
83.2
|
MIT-67 | 10% of odd classes | 33.9 | 28.4 | 34.0 | 35.5 | 35.2 | 49.8 | 50.4 |
56.9
|
DIL | Standard split | 50.3 | 46.7 | 52.6 | 55.3 | 54.7 | 68.9 | 69.5 |
72.6
|
MLC | Standard split | 38.9 | 31.4 | 43.0 | 47.7 | 46.5 | 65.7 | 66.1 |
68.6
|
Cost-sensitive DBN with differential evolution (CSDBN-DE)
Output thresholding
Category centers
Classifier | Dist A (\(\rho = 10\)) | Dist B (\(\rho = 20\)) | Dist C (\(\rho \in [1, 20]\)) |
---|---|---|---|
(A) Baseline CNN | 0.779 | 0.747 | 0.857 |
(B) CC last conv. layer | 0.824 | 0.775 | 0.859 |
(C) CC last FC layer | 0.826 | 0.772 | 0.865 |
(D) Baseline CNN + over-sampling | 0.787 | 0.770 | 0.850 |
(E) CC last conv. layer + over-sampling | 0.831 | 0.792 | 0.861 |
(F) CC last FC layer + over-sampling | 0.830 | 0.796 | 0.862 |
Very-deep neural networks
FAU class | Imbalance level | (A) Handcraft | (B) Plain34 | (C) Res10 | (D) Res18 | (E) Res34 | (F) Res50 |
---|---|---|---|---|---|---|---|
AU1 | 2.5% | 0.05 | 0.43 | 0.46 | 0.44 | 0.44 | 0.46 |
AU2 | 1.5% | 0.00 | 0.42 | 0.44 | 0.45 | 0.45 | 0.46 |
AU4 | 1.1% | 0.00 | 0.44 | 0.42 | 0.41 | 0.40 | 0.42 |
AU5 | 0.9% | 0.00 | 0.42 | 0.41 | 0.42 | 0.40 | 0.46 |
AU6 | 3.6% | 0.41 | 0.52 | 0.51 | 0.55 | 0.52 | 0.51 |
AU9 | 0.6% | 0.00 | 0.41 | 0.36 | 0.38 | 0.40 | 0.44 |
AU12 | 45.7% | 0.89 | 0.89 | 0.90 | 0.89 | 0.90 | 0.90 |
AU17 | 0.4% | 0.00 | 0.19 | 0.20 | 0.26 | 0.27 | 0.24 |
AU20 | 0.01% | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
AU25 | 29.9% | 0.79 | 0.85 | 0.86 | 0.86 | 0.86 | 0.85 |
AU26 | 25.6% | 0.64 | 0.73 | 0.74 | 0.73 | 0.74 | 0.73 |
Average | – | 0.29 | 0.48 | 0.48 | 0.49 | 0.49 | 0.50 |
Summary of algorithm-level methods
Hybrid-methods
Large Margin Local Embedding (LMLE)
Attractive | Mouth open | Smiling | Wear lipstick | High cheekbones | Male | Heavy makeup | Wavy hair | Oval face | Pointy nose | Arched eyebrows | Black hair | Big lips | Big nose | Young | Straight hair | Brown hair | Bags under eyes | Wear earings | No beard | Bangs | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Imbalance level | 1 | 2 | 2 | 3 | 5 | 8 | 11 | 18 | 22 | 22 | 23 | 26 | 26 | 27 | 28 | 29 | 30 | 30 | 31 | 33 | 35 |
Triplet-kNN | 83 | 92 | 92 | 91 | 86 | 91 | 88 | 77 | 61 | 61 | 73 | 82 | 55 | 68 | 75 | 63 | 76 | 63 | 69 | 82 | 81 |
PANDA | 85 | 93 | 98 | 97 | 89 | 99 | 95 | 78 | 66 | 67 | 77 | 84 | 56 | 72 | 78 | 66 | 85 | 67 | 77 | 87 | 92 |
ANet | 87 | 96 | 97 | 95 | 89 | 99 | 96 | 81 | 67 | 69 | 76 | 90 | 57 | 78 | 84 | 69 | 83 | 70 | 83 | 93 | 90 |
LMLE-kNN | 88 | 96 | 99 | 99 | 92 | 99 | 98 | 83 | 68 | 72 | 79 | 92 | 60 | 80 | 87 | 73 | 87 | 73 | 83 | 96 | 98 |
Blond hair | Bushy eyebrows | Wear necklace | Narrow eyes | 5 o’clock shadow | Receding hairline | Wear necktie | Eyeglasses | Rosy cheeks | Goatee | Chubby | Sideburns | Blurry | Wear hat | Double chin | Pale skin | Gray hair | Mustache | Bald | Average | ||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Imbalance level | 35 | 36 | 38 | 38 | 39 | 42 | 43 | 44 | 44 | 44 | 44 | 44 | 45 | 45 | 45 | 46 | 46 | 46 | 48 | ||
Triplet-kNN | 81 | 68 | 50 | 47 | 66 | 60 | 73 | 82 | 64 | 73 | 64 | 71 | 43 | 84 | 60 | 63 | 72 | 57 | 75 | 72 | |
PANDA | 91 | 74 | 51 | 51 | 76 | 67 | 85 | 88 | 68 | 84 | 65 | 81 | 50 | 90 | 64 | 69 | 79 | 63 | 74 | 77 | |
ANet | 90 | 82 | 59 | 57 | 81 | 70 | 79 | 95 | 76 | 86 | 70 | 79 | 56 | 90 | 68 | 77 | 85 | 61 | 73 | 80 | |
LMLE-kNN | 99 | 82 | 59 | 59 | 82 | 76 | 90 | 98 | 78 | 95 | 79 | 88 | 59 | 99 | 74 | 80 | 91 | 73 | 90 | 84 |
Deep over-sampling (DOS)
Model | Reduction rate | Class | MNIST | MNISTbr | SVHN | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Pr | Re | F1 | AUC | Pr | Re | F1 | AUC | Pr | Re | F1 | AUC | |||
CNN | 0.90 | mnr | 0.98 | 0.93 | 0.96 | 0.99 | 0.31 | 0.76 | 0.43 | 0.68 | 0.62 | 0.77 | 0.55 | 0.78 |
mjr | 0.96 | 0.99 | 0.97 | 1.00 | 0.77 | 0.56 | 0.65 | 0.77 | 0.80 | 0.76 | 0.75 | 0.89 | ||
0.95 | mnr | 0.99 | 0.89 | 0.94 | 0.99 | 0.27 | 0.76 | 0.23 | 0.57 | 0.13 | 0.86 | 0.21 | 0.61 | |
mjr | 0.93 | 0.98 | 0.95 | 0.99 | 0.52 | 0.69 | 0.58 | 0.67 | 0.89 | 0.61 | 0.71 | 0.87 | ||
0.99 | mnr | 0.65 | 0.98 | 0.77 | 0.96 | 0.31 | 0.71 | 0.43 | 0.68 | 0.50 | 0.72 | 0.42 | 0.74 | |
mjr | 0.99 | 0.82 | 0.89 | 0.99 | 0.78 | 0.56 | 0.65 | 0.77 | 0.73 | 0.67 | 0.60 | 0.81 | ||
CNN-CL | 0.90 | mnr | 0.99 | 0.90 | 0.94 | 1.00 | 0.22 | 0.77 | 0.31 | 0.69 | 0.59 | 0.60 | 0.42 | 0.78 |
mjr | 0.94 | 0.99 | 0.96 | 0.98 | 0.78 | 0.53 | 0.63 | 0.78 | 0.77 | 0.75 | 0.73 | 0.86 | ||
0.95 | mnr | 0.99 | 0.83 | 0.90 | 0.97 | 0.28 | 0.77 | 0.24 | 0.60 | 0.04 | 0.68 | 0.07 | 0.61 | |
mjr | 0.89 | 0.99 | 0.94 | 1.00 | 0.52 | 0.68 | 0.57 | 0.69 | 0.89 | 0.60 | 0.70 | 0.84 | ||
0.99 | mnr | 0.75 | 0.98 | 0.85 | 0.95 | 0.22 | 0.72 | 0.31 | 0.69 | 0.47 | 0.71 | 0.37 | 0.72 | |
mjr | 0.99 | 0.86 | 0.92 | 0.99 | 0.78 | 0.53 | 0.63 | 0.78 | 0.71 | 0.57 | 0.56 | 0.78 | ||
DOS (\(\hbox {k}=5\)) | 0.90 | mnr | 0.99 | 0.97 | 0.98 | 1.00 | 0.66 | 0.77 | 0.71 | 0.79 | 0.71 | 0.82 | 0.72 | 0.84 |
mjr | 0.98 | 0.99 | 0.98 | 1.00 | 0.75 | 0.68 | 0.71 | 0.81 | 0.85 | 0.79 | 0.81 | 0.92 | ||
0.95 | mnr | 0.98 | 0.96 | 0.97 | 1.00 | 0.56 | 0.75 | 0.63 | 0.72 | 0.40 | 0.89 | 0.55 | 0.73 | |
mjr | 0.97 | 0.99 | 0.98 | 1.00 | 0.64 | 0.74 | 0.69 | 0.78 | 0.90 | 0.69 | 0.78 | 0.91 | ||
0.99 | mnr | 0.91 | 0.99 | 0.95 | 0.99 | 0.61 | 0.73 | 0.66 | 0.75 | 0.51 | 0.91 | 0.64 | 0.80 | |
mjr | 0.98 | 0.94 | 0.96 | 1.00 | 0.77 | 0.70 | 0.73 | 0.82 | 0.89 | 0.68 | 0.77 | 0.90 |
Classifier | k | Class | MNIST | MNISTbr | SVHN | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Pr | Re | F1 | AUC | Pr | Re | F1 | AUC | Pr | Re | F1 | AUC | |||
CNN | mnr | 0.65 | 0.98 | 0.77 | 0.96 | 0.31 | 0.76 | 0.43 | 0.68 | 0.50 | 0.72 | 0.42 | 0.74 | |
mjr | 0.99 | 0.82 | 0.89 | 0.99 | 0.78 | 0.56 | 0.65 | 0.77 | 0.73 | 0.67 | 0.60 | 0.81 | ||
CNN-CL | mnr | 0.75 | 0.98 | 0.85 | 0.95 | 0.22 | 0.77 | 0.31 | 0.69 | 0.47 | 0.71 | 0.37 | 0.72 | |
mjr | 0.99 | 0.86 | 0.92 | 0.99 | 0.78 | 0.53 | 0.63 | 0.78 | 0.71 | 0.57 | 0.56 | 0.78 | ||
DOS | 3 | mnr | 0.91 | 0.98 | 0.95 | 0.99 | 0.65 | 0.77 | 0.70 | 0.78 | 0.67 | 0.77 | 0.66 | 0.83 |
mjr | 0.99 | 0.94 | 0.96 | 1.00 | 0.75 | 0.68 | 0.71 | 0.80 | 0.80 | 0.74 | 0.74 | 0.86 | ||
5 | mnr | 0.91 | 0.99 | 0.95 | 0.99 | 0.66 | 0.77 | 0.71 | 0.79 | 0.51 | 0.91 | 0.64 | 0.80 | |
mjr | 0.98 | 0.94 | 0.96 | 1.00 | 0.75 | 0.68 | 0.71 | 0.81 | 0.89 | 0.68 | 0.77 | 0.90 | ||
10 | mnr | 0.91 | 0.99 | 0.95 | 0.99 | 0.61 | 0.73 | 0.66 | 0.75 | 0.40 | 0.89 | 0.55 | 0.73 | |
mjr | 0.99 | 0.94 | 0.96 | 1.00 | 0.77 | 0.70 | 0.73 | 0.82 | 0.90 | 0.69 | 0.78 | 0.91 |
Class rectification loss (CRL) and hard sample mining
Attractive | Mouth open | Smiling | Wear lipstick | High cheekbones | Male | Heavy makeup | Wavy hair | Oval face | Pointy nose | Arched eyebrows | Black hair | Big lips | Big nose | Young | Straight hair | Brown hair | Bags under eyes | Wear earings | No beard | Bangs | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Imbalance (1:x) | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 3 | 3 | 3 | 3 | 3 | 3 | 4 | 4 | 4 | 4 | 4 | 5 | 6 |
Triplet-kNN | 83 | 92 | 92 | 91 | 86 | 91 | 88 | 77 | 61 | 61 | 73 | 82 | 55 | 68 | 75 | 63 | 76 | 63 | 69 | 82 | 81 |
PANDA | 85 | 93 |
98
|
97
|
89
|
99
| 95 | 78 | 66 | 67 | 77 | 84 | 56 | 72 | 78 | 66 |
85
| 67 | 77 | 87 | 92 |
ANet |
87
|
96
| 97 | 95 |
89
|
99
|
96
|
81
| 67 | 69 | 76 |
90
| 57 |
78
|
84
| 69 | 83 | 70 |
83
|
93
| 90 |
DeepID2 | 78 | 89 | 89 | 92 | 84 | 94 | 88 | 73 | 63 | 66 | 77 | 83 | 62 | 73 | 76 | 65 | 79 | 74 | 75 | 88 | 91 |
Over-sampling | 77 | 89 | 90 | 92 | 84 | 95 | 87 | 70 | 63 | 67 |
79
| 84 | 61 | 73 | 75 | 66 | 82 | 73 | 76 | 88 | 90 |
Down-sampling | 78 | 87 | 90 | 91 | 80 | 90 | 89 | 70 | 58 | 63 | 70 | 80 | 61 | 76 | 80 | 61 | 76 | 71 | 70 | 88 | 88 |
Cost-sensitive | 78 | 89 | 90 | 91 | 85 | 93 | 89 | 75 | 64 | 65 | 78 | 85 | 61 | 74 | 75 | 67 | 84 | 74 | 76 | 88 | 90 |
Threshold-Adj | 69 | 89 | 88 | 89 | 83 | 95 | 89 | 77 |
72
|
72
| 76 | 86 |
66
| 76 | 24 |
73
| 81 |
76
| 76 | 15 | 93 |
LMLE-kNN |
88
|
96
|
99
|
99
|
92
|
99
|
98
|
83
|
68
|
72
|
79
|
92
| 60 |
80
|
87
|
73
|
87
| 73 |
83
|
96
|
98
|
CRL | 81 |
94
| 92 | 95 | 87 |
98
| 90 | 79 | 66 |
71
|
80
| 88 |
67
| 77 | 83 | 72 | 84 |
79
|
84
|
93
|
95
|
Blond hair | Bushy eyebrows | Wear necklace | Narrow eyes | 5 o’clock shadow | Receding hairline | Wear necktie | Eyeglasses | Rosy cheeks | Goatee | Chubby | Sideburns | Blurry | Wear hat | Double chin | Pale skin | Gray hair | Mustache | Bald | Mean | ||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Imbalance (1:x) | 6 | 6 | 7 | 8 | 8 | 11 | 13 | 14 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 22 | 23 | 24 | 43 | ||
Triplet-kNN | 81 | 68 | 50 | 47 | 66 | 60 | 73 | 82 | 64 | 73 | 64 | 71 | 43 | 84 | 60 | 63 | 72 | 57 | 75 | 72 | |
PANDA | 91 | 74 | 51 | 51 | 76 | 67 | 85 | 88 | 68 | 84 | 65 | 81 | 50 | 90 | 64 | 69 | 79 | 63 | 74 | 77 | |
ANet | 90 |
82
| 59 | 57 | 81 | 70 | 79 | 95 | 76 | 86 | 70 | 79 | 56 | 90 | 68 | 77 | 85 | 61 | 73 | 80 | |
DeepID2 | 90 | 78 | 70 | 64 |
85
| 81 | 83 | 92 | 86 |
90
| 81 | 89 | 74 | 90 | 83 | 81 | 90 | 88 | 93 | 81 | |
Over-sampling | 90 | 80 |
71
| 65 |
85
| 82 | 79 | 91 |
90
| 89 |
83
|
90
| 76 | 89 |
84
| 82 | 90 |
90
| 92 | 82 | |
Down-sampling | 85 | 75 | 66 | 61 | 82 | 79 | 80 | 85 | 82 | 85 | 78 | 80 | 68 | 90 | 80 | 78 | 88 | 60 | 79 | 78 | |
Cost-sensitive | 89 | 79 |
71
| 65 | 84 | 81 | 82 | 91 |
92
| 86 | 82 |
90
| 76 | 90 |
84
| 80 | 90 | 88 |
93
| 82 | |
Threshold-Adj | 92 |
84
| 62 |
71
| 82 |
83
| 76 | 95 | 82 | 89 | 81 | 89 |
78
|
95
| 83 |
85
|
91
| 86 |
93
| 79 | |
LMLE-kNN |
99
|
82
| 59 | 59 | 82 | 76 |
90
|
98
| 78 |
95
| 79 | 88 | 59 |
99
| 74 | 80 |
91
| 73 | 90 |
84
| |
CRL |
95
|
84
|
73
|
73
|
89
|
88
|
87
|
99
|
90
|
95
|
87
|
95
|
86
|
99
|
89
|
92
|
96
|
93
|
99
|
87
|
Summary of hybrid methods
Discussion of surveyed works
Method | Network type | Method type | Description |
---|---|---|---|
CNN | Data | ROS of minority classes until class balance is achieved | |
RUS [23] | CNN | Data | RUS of majority classes until class balance is achieved |
CNN | Data | Pre-training with RUS or ROS, then fine-tuning with all data | |
Dynamic sampling [21] | CNN | Data | Sampling rates adjust throughout training based on previous iteration’s class-wise F1-scores |
MFE and MSFE loss [18] | MLP | Algorithm | New loss functions allow positive and negative classes to contribute to loss equally |
CNN | Algorithm | New loss function down-weights easy-to-classify samples, reducing their impact on total loss | |
CSDNN [89] | MLP | Algorithm | CE loss function modified to incorporate a pre-defined cost matrix |
CoSen CNN [19] | CNN | Algorithm | Cost matrix is learned through backpropagation and incorporated into output layer |
CSDBN-DE [90] | DBN | Algorithm | Cost matrix is learned through evolutionary algorithm and incorporated into output layer |
Threshold moving [23] | CNN | Algorithm | Decision threshold is adjusted by dividing output probabilities by prior class probabilities |
Category centers [91] | CNN | Algorithm | Class centroids are calculated in deep feature space and K-NN method discriminates |
Very-deep NNs [92] | CNN | Algorithm | CNN network depths of up to 50 layers are used to examine convergence rates |
LMLE [22] | CNN | Hybrid | Triple-header hinge loss and quintuplet sampling generate more discriminative features |
DOS [117] | CNN | Hybrid | Minority class over-sampled in deep feature space using K-NN and micro-cluster loss |
CRL loss [118] | CNN | Hybrid | Class Rectification loss and hard sample mining produce more discriminative features |
Paper | Data sets | Data type | Class count | Data set size | Min class size | Max class size | \(\rho\) (Eq. 1) |
---|---|---|---|---|---|---|---|
[79] | CIFAR-10 | Image | 10 | 60,000 | 2340 | 3900 | 2.3 |
[20] | WHOI-Plankton | Image | 103 | 3,400,000 | < 3500 | 2,300,000 | 657 |
[21] | Public cameras | Image | 19 | 10,000 | 14 | 6986 | 499 |
[18] | CIFAR-100 (1) | Image | 2 | 6000 | 150 | 3000 | 20 |
CIFAR-100 (2) | Image | 2 | 1200 | 30 | 600 | 20 | |
CIFAR-100 (3) | Image | 2 | 1200 | 30 | 600 | 20 | |
20 News Group (1) | Text | 2 | 1200 | 30 | 600 | 20 | |
20 News Group (2) | Text | 2 | 1200 | 30 | 600 | 20 | |
[88] | COCO | Image | 2 | 115,000 | 10 | 100,000 | 10,000 |
[103] | Building changes | Image | 6 | 203,358 | 222 | 200,000 | 900 |
[89] | GHW | Structured | 2 | 2565 | 406 | 2159 | 5.3 |
ORP | Structured | 2 | 700 | 124 | 576 | 4.6 | |
[19] | MNIST | Image | 10 | 70,000 | 600 | 6000 | 10 |
CIFAR-100 | Image | 100 | 60,000 | 60 | 600 | 10 | |
CALTECH-101 | Image | 102 | 9144 | 15 | 30 | 2 | |
MIT-67 | Image | 67 | 6700 | 10 | 100 | 10 | |
DIL | Image | 10 | 1300 | 24 | 331 | 13 | |
MLC | Image | 9 | 400,000 | 2600 | 196,900 | 76 | |
[90] | KEEL | Structured | 2 | 3339 | 26 | 3313 | 128 |
[91] | CIFAR-10 | Image | 10 | 60,000 | 250 | 5000 | 20 |
CIFAR-100 | Image | 100 | 60,000 | 25 | 500 | 20 | |
[22] | CelebA | Image | 2 | 160,000 | 3200 | 156,800 | 49 |
[117] | MNIST | Image | 10 | 60,000 | 50 | 5000 | 100 |
MNIST-back-rot | Image | 10 | 62,000 | 12 | 1200 | 100 | |
CIFAR-10 | Image | 10 | 60,000 | 5000 | 5000 | 1 | |
SVHN | Image | 10 | 99,000 | 73 | 7300 | 100 | |
STL-10 | Image | 10 | 13,000 | 500 | 500 | 1 | |
[118] | CelebA | Image | 2 | 160,000 | 3200 | 156,800 | 49 |
[92] | EmotioNet | Image | 2 | 450,000 | 45 | 449,955 | 10,000 |
[23] | MNIST | Image | 10 | 60,000 | 1 | 5000 | 5000 |
CIFAR-10 | Image | 10 | 60,000 | 100 | 5000 | 50 | |
ImageNet | Image | 1000 | 1,050,000 | 10 | 1000 | 100 |