Introduction
Types of feature selection
Discussion
Motivation and problem definition
Paper organization
Related work
DN | Dataset name | NoI | NoF | NoSF | SF (DRW) |
---|---|---|---|---|---|
1 | Accut1 | 120 | 7 | 5 | 5,6,4,1,3 |
2 | Accute2 | 120 | 5 | 5 | 1,4,7,5,2 |
3 | Abalone | 4177 | 7 | 4 | 3,5,6,1 |
4 | Glass | 214 | 9 | 5 | 3,8,7,5,6 |
5 | Wine | 178 | 13 | 5 | 7,10,1,2,4 |
6 | Iris | 150 | 4 | 3 | 3,1,2 |
7 | Monk1 | 432 | 6 | 4 | 5,2,6,4 |
8 | Monk2 | 432 | 6 | 3 | 5,2,1 |
9 | Monk3 | 432 | 6 | 4 | 5,4,6,3 |
10 | Tic-Tac Toe | 958 | 9 | 5 | 5,3,9,1,7 |
11 | Zoo | 101 | 16 | 5 | 13,4,8,3,1 |
12 | Diabetes | 768 | 9 | 4 | 2,8,5,4 |
13 | Car | 1728 | 7 | 5 | 6,4,1,2,5 |
14 | Heart-statlog | 270 | 14 | 5 | 13,12,8,11,10 |
15 | Breast-cancerW | 699 | 10 | 5 | 2,3,7,1,5 |
16 | Colon-cancer | 62 | 2000 | 5 | 765,1771,1972,1672,737 |
17 | Lung-cancer | 73 | 325 | 5 | 23,243,27,205,12 |
18 | Lymphoma | 45 | 4026 | 5 | 734,760,37,853,2659 |
19 | SRBCT | 83 | 2308 | 5 | 509,107,1105,1915,1916 |
20 | TUIDS | 20 | 5 | 17,16,19,18,6 | |
21 | NSL-KDD | 43 | 5 | 5,3,4,6,42 |
EFS-MI: the proposed MI based ensemble feature selection
X
|
p
|
t
|
k
|
a
|
i
|
u
|
---|---|---|---|---|---|---|
p(X) |
\(\frac{1}{8}\)
|
\(\frac{1}{4}\)
|
\(\frac{1}{8}\)
|
\(\frac{1}{4}\)
|
\(\frac{1}{8}\)
|
\(\frac{1}{8}\)
|
P(x,y) |
p
|
t
|
k
|
P(y) |
---|---|---|---|---|
a
|
\(\frac{1}{16}\)
|
\(\frac{3}{8}\)
|
\(\frac{1}{16}\)
|
\(\frac{1}{2}\)
|
i
|
\(\frac{1}{16}\)
|
\(\frac{3}{16}\)
| 0 |
\(\frac{1}{4}\)
|
u
| 0 |
\(\frac{3}{16}\)
|
\(\frac{1}{16}\)
|
\(\frac{1}{4}\)
|
P(x) |
\(\frac{1}{8}\)
|
\(\frac{3}{4}\)
|
\(\frac{1}{8}\)
|
D | Decision tree | Random forest | KNN | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Rank | SU | GR | IG | RF | CS | EFS | SU | GR | IG | RF | CS | EFS | SU | GR | IG | RF | CS | EFS | |
20 |
\(f=5\)
| 0.95 | 0.90 | 0.95 | 0.93 | 0.95 | 0.95 | 0.96 | 0.89 | 0.96 | 0.88 | 0.96 | 0.92 | 0.96 | 0.87 | 0.95 | 0.9 | 0.96 | 0.97 |
\(f=3\)
| 0.96 | 0.95 | 0.95 | 0.9 | 0.95 | 0.96 | 0.97 | 0.95 | 0.97 | 0.95 | 0.97 | 0.93 | 0.93 | 0.96 | 0.92 | 0.87 | 0.96 | 0.96 | |
\(f=4\)
| 0.96 | 0.97 | 0.95 | 0.94 | 0.96 | 0.96 | 0.98 | 0.97 | 0.98 | 0.97 | 0.97 | 0.96 | 0.92 | 0.96 | 0.93 | 0.88 | 0.93 | 0.96 | |
\(f=6\)
| 0.97 | 0.97 | 0.97 | 0.95 | 0.97 | 0.97 | 0.98 | 0.98 | 0.98 | 0.97 | 0.98 | 0.97 | 0.92 | 0.94 | 0.93 | 0.88 | 0.92 | 0.95 | |
\(f=42\)
| 0.97 | 0.97 | 0.97 | 0.95 | 0.97 | 0.97 | 0.98 | 0.98 | 0.98 | 0.97 | 0.98 | 0.98 | 0.92 | 0.93 | 0.93 | 0.88 | 0.93 | 0.94 | |
\(f=33\)
| 0.97 | 0.97 | 0.97 | 0.95 | 0.97 | 0.97 | 0.98 | 0.98 | 0.98 | 0.97 | 0.98 | 0.98 | 0.92 | 0.92 | 0.92 | 0.90 | 0.92 | 0.95 | |
\(f=23\)
| 0.97 | 0.97 | 0.97 | 0.96 | 0.97 | 0.97 | 0.98 | 0.98 | 0.98 | 0.97 | 0.98 | 0.98 | 0.92 | 0.92 | 0.92 | 0.91 | 0.92 | 0.96 | |
\(f=34\)
| 0.97 | 0.97 | 0.97 | 0.97 | 0.97 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.92 | 0.92 | 0.92 | 0.92 | 0.92 | 0.96 | |
ACA | 0.95 | 0.92 | 0.95 | 0.94 | 0.97 | 0.96 | 0.97 | 0.94 | 0.96 | 0.91 | 0.96 | 0.96 | 0.92 | 0.91 | 0.95 | 0.88 | 0.96 | 0.95 | |
21 |
\(f=1\)
| 0.79 | 0.98 | 0.98 | 0.98 | 0.98 | 0.99 | 0.79 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.79 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 |
\(f=17\)
| 0.99 | 0.99 | 0.98 | 0.98 | 0.98 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.98 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | |
\(f=16\)
| 0.99 | 0.98 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.99 | |
\(f=19\)
| 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.99 | |
\(f=18\)
| 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.99 | |
ACA | 0.95 | 0.98 | 0.98 | 0.98 | 0.98 | 0.99 | 0.95 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.94 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 |
EFS-MI: Algorithm
Function of a combiner
Complexity analysis
Experimental results
Result analysis
Analysis on network intrusion datasets
Analysis on gene expression datasets
D | Decision tree | Random forest | KNN | SVM | |||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Rank | SU | GR | IG | RF | CS | EFS | SU | GR | IG | RF | CS | EFS | SU | GR | IG | RF | CS | EFS | SU | GR | IG | RF | CS | EFS | |
15 |
\(f=2\)
| 0.94 | 0.93 | 0.94 | 0.94 | 0.93 | 0.95 | 0.97 | 0.96 | 0.96 | 0.97 | 0.96 | 0.97 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.97 |
\(f=3\)
| 0.94 | 0.93 | 0.94 | 0.94 | 0.94 | 0.94 | 0.97 | 0.96 | 0.97 | 0.96 | 0.97 | 0.98 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.97 | |
\(f=7\)
| 0.94 | 0.93 | 0.94 | 0.94 | 0.94 | 0.95 | 0.97 | 0.96 | 0.96 | 0.97 | 0.96 | 0.97 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.97 | |
\(f=1\)
| 0.94 | 0.94 | 0.94 | 0.94 | 0.94 | 0.94 | 0.96 | 0.97 | 0.96 | 0.97 | 0.97 | 0.97 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.97 | |
\(f=5\)
| 0.94 | 0.94 | 0.94 | 0.94 | 0.94 | 0.94 | 0.96 | 0.97 | 0.96 | 0.97 | 0.97 | 0.97 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.97 | |
ACA | 0.94 | 0.93 | 0.94 | 0.94 | 0.93 | 0.94 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.97 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.97 | |
16 |
\(f=1\)
| 0.84 | 0.83 | 0.84 | 0.83 | 0.84 | 0.84 | 0.84 | 0.84 | 0.83 | 0.83 | 0.83 | 0.84 | 0.76 | 0.66 | 0.64 | 0.64 | 0.68 | 0.77 | 0.65 | 0.64 | 0.65 | 0.64 | 0.73 | 0.83 |
\(f=11\)
| 0.79 | 0.79 | 0.76 | 0.75 | 0.76 | 0.78 | 0.82 | 0.81 | 0.82 | 0.83 | 0.84 | 0.85 | 0.79 | 0.80 | 0.75 | 0.77 | 0.79 | 0.80 | 0.85 | 0.84 | 0.87 | 0.87 | 0.84 | 0.85 | |
\(f=21\)
| 0.73 | 0.79 | 0.76 | 0.75 | 0.74 | 0.76 | 0.83 | 0.85 | 0.85 | 0.83 | 0.84 | 0.86 | 0.82 | 0.81 | 0.79 | 0.81 | 0.79 | 0.78 | 0.86 | 0.83 | 0.87 | 0.86 | 0.88 | 0.86 | |
\(f=51\)
| 0.74 | 0.76 | 0.79 | 0.80 | 0.80 | 0.77 | 0.84 | 0.83 | 0.85 | 0.86 | 0.86 | 0.86 | 0.84 | 0.85 | 0.82 | 0.83 | 0.80 | 0.85 | 0.84 | 0.83 | 0.86 | 0.87 | 0.85 | 0.86 | |
\(f=81\)
| 0.80 | 0.76 | 0.78 | 0.77 | 0.78 | 0.76 | 0.85 | 0.86 | 0.86 | 0.85 | 0.87 | 0.87 | 0.83 | 0.84 | 0.83 | 0.82 | 0.80 | 0.85 | 0.86 | 0.84 | 0.86 | 0.88 | 0.86 | 0.86 | |
ACA | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.83 | 0.83 | 0.84 | 0.84 | 0.84 | 0.85 | 0.80 | 0.79 | 0.76 | 0.77 | 0.77 | 0.81 | 0.81 | 0.79 | 0.82 | 0.82 | 0.83 | 0.85 | |
17 |
\(f=1\)
| 0.48 | 0.42 | 0.49 | 0.47 | 0.38 | 0.48 | 0.47 | 0.38 | 0.47 | 0.45 | 0.37 | 0.45 | 0.34 | 0.32 | 0.34 | 0.35 | 0.30 | 0.40 | 0.45 | 0.33 | 0.41 | 0.45 | 0.45 | 0.44 |
\(f=11\)
| 0.57 | 0.59 | 0.59 | 0.63 | 0.59 | 0.60 | 0.85 | 0.87 | 0.82 | 0.77 | 0.75 | 0.79 | 0.81 | 0.84 | 0.80 | 0.75 | 0.70 | 0.75 | 0.89 | 0.85 | 0.84 | 0.83 | 0.74 | 0.70 | |
\(f=31\)
| 0.57 | 0.54 | 0.58 | 0.57 | 0.57 | 0.60 | 0.84 | 0.85 | 0.84 | 0.86 | 0.87 | 0.87 | 0.81 | 0.82 | 0.82 | 0.81 | 0.83 | 0.87 | 0.87 | 0.86 | 0.88 | 0.87 | 0.86 | 0.73 | |
\(f=51\)
| 0.57 | 0.53 | 0.57 | 0.54 | 0.59 | 0.55 | 0.84 | 0.85 | 0.85 | 0.87 | 0.84 | 0.87 | 0.82 | 0.84 | 0.82 | 0.85 | 0.83 | 0.84 | 0.87 | 0.87 | 0.87 | 0.87 | 0.86 | 0.80 | |
\(f=91\)
| 0.60 | 0.57 | 0.55 | 0.60 | 0.58 | 0.58 | 0.86 | 0.87 | 0.85 | 0.86 | 0.85 | 0.87 | 0.84 | 0.84 | 0.86 | 0.87 | 0.84 | 0.86 | 0.87 | 0.88 | 0.86 | 0.85 | 0.86 | 0.86 | |
ACA | 0.55 | 0.53 | 0.55 | 0.56 | 0.54 | 0.56 | 0.77 | 0.76 | 0.76 | 0.76 | 0.73 | 0.77 | 0.72 | 0.73 | 0.72 | 0.72 | 0.7 | 0.74 | 0.79 | 0.75 | 0.77 | 0.77 | 0.75 | 0.70 | |
18 |
\(f=1\)
| 0.49 | 0.52 | 0.60 | 0.64 | 0.48 | 0.64 | 0.47 | 0.49 | 0.59 | 0.57 | 0.51 | 0.61 | 0.44 | 0.47 | 0.57 | 0.59 | 0.52 | 0.56 | 0.34 | 0.36 | 0.58 | 0.42 | 0.57 | 0.38 |
\(f=101\)
| 0.64 | 0.57 | 0.61 | 0.61 | 0.66 | 0.69 | 0.69 | 0.67 | 0.63 | 0.70 | 0.67 | 0.81 | 0.65 | 0.65 | 0.58 | 0.56 | 0.67 | 0.70 | 0.76 | 0.78 | 0.75 | 0.79 | 0.76 | 0.83 | |
\(f=301\)
| 0.64 | 0.57 | 0.64 | 0.83 | 0.70 | 0.75 | 0.76 | 0.71 | 0.71 | 0.76 | 0.71 | 0.87 | 0.69 | 0.63 | 0.63 | 0.64 | 0.63 | 0.80 | 0.76 | 0.78 | 0.78 | 0.79 | 0.84 | 0.90 | |
\(f=401\)
| 0.74 | 0.71 | 0.84 | 0.80 | 0.67 | 0.70 | 0.79 | 0.78 | 0.79 | 0.77 | 0.76 | 0.91 | 0.63 | 0.67 | 0.66 | 0.66 | 0.64 | 0.87 | 0.84 | 0.81 | 0.81 | 0.76 | 0.86 | 0.94 | |
\(f=1201\)
| 0.82 | 0.63 | 0.77 | 0.68 | 0.71 | 0.78 | 0.88 | 0.88 | 0.86 | 0.87 | 0.88 | 0.95 | 0.72 | 0.73 | 0.74 | 0.70 | 0.70 | 0.85 | 0.84 | 0.88 | 0.88 | 0.88 | 0.86 | 0.97 | |
ACA | 0.66 | 0.6 | 0.69 | 0.71 | 0.64 | 0.71 | 0.71 | 0.70 | 0.71 | 0.73 | 0.70 | 0.83 | 0.62 | 0.63 | 0.63 | 0.63 | 0.63 | 0.75 | 0.70 | 0.72 | 0.76 | 0.72 | 0.77 | 0.80 | |
19 |
\(f=1\)
| 0.55 | 0.45 | 0.48 | 0.55 | 0.63 | 0.58 | 0.52 | 0.45 | 0.51 | 0.51 | 0.60 | 0.62 | 0.49 | 0.43 | 0.52 | 0.51 | 0.61 | 0.52 | 0.51 | 0.44 | 0.47 | 0.49 | 0.68 | 0.62 |
\(f=21\)
| 0.85 | 0.83 | 0.84 | 0.89 | 0.85 | 0.84 | 0.99 | 0.99 | 0.98 | 0.97 | 0.98 | 0.98 | 0.98 | 0.99 | 0.97 | 0.98 | 0.96 | 0.98 | 0.99 | 0.96 | 0.97 | 0.97 | 0.98 | 0.98 | |
\(f=51\)
| 0.86 | 0.86 | 0.83 | 0.85 | 0.84 | 0.84 | 0.99 | 1.0 | 0.99 | 0.97 | 0.99 | 0.99 | 0.99 | 0.97 | 0.99 | 0.99 | 0.98 | 0.99 | 1.0 | 0.98 | 1.0 | 0.98 | 1.0 | 1.0 | |
\(f=71\)
| 0.86 | 0.84 | 0.84 | 0.84 | 0.83 | 0.84 | 1.0 | 0.99 | 0.99 | 0.99 | 0.99 | 1.0 | 0.99 | 0.98 | 0.99 | 0.99 | 1.0 | 1.0 | 1.0 | 0.98 | 1.0 | 0.99 | 1.0 | 1.0 | |
\(f=91\)
| 0.82 | 0.82 | 0.81 | 0.85 | 0.83 | 0.83 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 1.0 | 0.99 | 0.99 | 0.99 | 0.98 | 1.0 | 1.0 | 1.0 | 0.99 | 1.0 | 1.0 | 1.0 | 1.0 | |
ACA | 0.78 | 0.76 | 0.76 | 0.79 | 0.79 | 0.78 | 0.89 | 0.88 | 0.89 | 0.88 | 0.91 | 0.91 | 0.88 | 0.87 | 0.89 | 0.89 | 0.91 | 0.89 | 0.9 | 0.87 | 0.88 | 0.88 | 0.93 | 0.92 |
Analysis on UCI datasets
D | Decision tree | Random forest | KNN | SVM | |||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
R | SU | GR | IG | RF | CS | EFS | SU | GR | IG | RF | CS | EFS | SU | GR | IG | RF | CS | EFS | SU | GR | IG | RF | CS | EFS | |
1 |
\(f=5\)
| 0.82 | 0.82 | 0.82 | 0.82 | 0.82 | 0.82 | 0.82 | 0.82 | 0.82 | 0.82 | 0.82 | 0.82 | 0.69 | 0.63 | 0.81 | 0.71 | 0.73 | 0.73 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
\(f=6\)
| 0.91 | 0.91 | 0.91 | 0.78 | 0.91 | 0.91 | 0.91 | 0.91 | 0.91 | 0.80 | 0.91 | 0.91 | 0.87 | 0.87 | 0.86 | 0.83 | 0.89 | 0.89 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | |
\(f=4\)
| 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | |
\(f=1\)
| 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | |
\(f=3\)
| 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | |
ACA | 0.94 | 0.94 | 0.94 | 0.92 | 0.94 | 0.94 | 0.94 | 0.94 | 0.94 | 0.92 | 0.94 | 0.94 | 0.91 | 0.9 | 0.93 | 0.90 | 0.92 | 0.92 | 1 | 1 | 1 | 1 | 1 | 1 | |
2 |
\(f=1\)
| 0.91 | 0.91 | 0.91 | 0.83 | 0.91 | 0.91 | 0.91 | 0.91 | 0.91 | 0.83 | 0.91 | 0.91 | 0.82 | 0.86 | 0.87 | 0.74 | 0.86 | 0.86 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
\(f=4\)
| 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | |
\(f=7\)
| 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | |
\(f=5\)
| 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | |
\(f=2\)
| 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | |
ACA | 0.97 | 0.97 | 0.97 | 0.95 | 0.97 | 0.97 | 0.97 | 0.97 | 0.97 | 0.95 | 0.97 | 0.97 | 0.95 | 0.96 | 0.96 | 0.93 | 0.96 | 0.96 | 1 | 1 | 1 | 1 | 1 | 1 | |
3 |
\(f=3\)
| 0.21 | 0.21 | 0.21 | 0.21 | 0.21 | 0.24 | 0.25 | 0.25 | 0.25 | 0.24 | 0.24 | 0.25 | 0.20 | 0.20 | 0.20 | 0.20 | 0.20 | 0.13 | 0.14 | 0.14 | 0.14 | 0.14 | 0.14 | 0.15 |
\(f=5\)
| 0.21 | 0.21 | 0.21 | 0.21 | 0.21 | 0.20 | 0.24 | 0.24 | 0.25 | 0.25 | 0.24 | 0.24 | 0.20 | 0.20 | 0.20 | 0.20 | 0.20 | 0.19 | 0.14 | 0.14 | 0.15 | 0.15 | 0.14 | 0.15 | |
\(f=6\)
| 0.21 | 0.21 | 0.21 | 0.21 | 0.21 | 0.23 | 0.24 | 0.24 | 0.24 | 0.24 | 0.25 | 0.25 | 0.20 | 0.20 | 0.20 | 0.20 | 0.20 | 0.21 | 0.14 | 0.15 | 0.15 | 0.14 | 0.14 | 0.15 | |
\(f=1\)
| 0.21 | 0.21 | 0.21 | 0.21 | 0.21 | 0.23 | 0.24 | 0.24 | 0.24 | 0.24 | 0.25 | 0.25 | 0.20 | 0.20 | 0.20 | 0.20 | 0.20 | 0.20 | 0.15 | 0.15 | 0.15 | 0.14 | 0.15 | 0.15 | |
\(f=4\)
| 0.21 | 0.21 | 0.21 | 0.21 | 0.21 | 0.21 | 0.25 | 0.24 | 0.24 | 0.24 | 0.24 | 0.24 | 0.20 | 0.20 | 0.20 | 0.20 | 0.20 | 0.20 | 0.14 | 0.15 | 0.14 | 0.14 | 0.14 | 0.15 | |
ACA | 0.21 | 0.21 | 0.21 | 0.21 | 0.21 | 0.22 | 0.24 | 0.24 | 0.24 | 0.24 | 0.24 | 0.24 | 0.2 | 0.2 | 0.2 | 0.2 | 0.2 | 0.18 | 0.14 | 0.14 | 0.14 | 0.14 | 0.14 | 0.15 | |
4 |
\(f=3\)
| 0.42 | 0.42 | 0.55 | 0.42 | 0.68 | 0.43 | 0.42 | 0.38 | 0.51 | 0.42 | 0.78 | 0.41 | 0.36 | 0.43 | 0.57 | 0.37 | 0.72 | 0.34 | 0.33 | 0.15 | 0.35 | 0.34 | 0.37 | 0.24 |
\(f=8\)
| 0.64 | 0.46 | 0.62 | 0.63 | 0.67 | 0.45 | 0.65 | 0.49 | 0.65 | 0.64 | 0.78 | 0.49 | 0.65 | 0.46 | 0.65 | 0.65 | 0.72 | 0.42 | 0.46 | 0.35 | 0.45 | 0.45 | 0.36 | 0.23 | |
\(f=7\)
| 0.65 | 0.54 | 0.63 | 0.65 | 0.68 | 0.59 | 0.69 | 0.54 | 0.67 | 0.69 | 0.79 | 0.65 | 0.66 | 0.53 | 0.68 | 0.66 | 0.73 | 0.61 | 0.49 | 0.46 | 0.50 | 0.49 | 0.36 | 0.24 | |
\(f=5\)
| 0.65 | 0.63 | 0.62 | 0.64 | 0.67 | 0.63 | 0.70 | 0.71 | 0.71 | 0.68 | 0.79 | 0.67 | 0.69 | 0.67 | 0.68 | 0.64 | 0.72 | 0.65 | 0.51 | 0.54 | 0.45 | 0.44 | 0.38 | 0.23 | |
\(f=6\)
| 0.65 | 0.64 | 0.65 | 0.66 | 0.68 | 0.64 | 0.73 | 0.72 | 0.72 | 0.71 | 0.79 | 0.71 | 0.67 | 0.66 | 0.67 | 0.69 | 0.72 | 0.66 | 0.48 | 0.53 | 0.48 | 0.52 | 0.36 | 0.23 | |
ACA | 0.60 | 0.53 | 0.61 | 0.6 | 0.67 | 0.59 | 0.63 | 0.56 | 0.65 | 0.62 | 0.78 | 0.58 | 0.60 | 0.55 | 0.65 | 0.60 | 0.72 | 0.53 | 0.45 | 0.40 | 0.44 | 0.44 | 0.36 | 0.23 | |
5 |
\(f=7\)
| 0.74 | 0.61 | 0.72 | 0.62 | 0.90 | 0.72 | 0.71 | 0.60 | 0.72 | 0.59 | 0.98 | 0.71 | 0.75 | 0.66 | 0.76 | 0.67 | 0.74 | 0.69 | 0.79 | 0.63 | 0.79 | 0.63 | 0.97 | 0.91 |
\(f=10\)
| 0.77 | 0.78 | 0.86 | 0.77 | 0.90 | 0.89 | 0.81 | 0.80 | 0.88 | 0.80 | 0.98 | 0.94 | 0.82 | 0.82 | 0.69 | 0.82 | 0.76 | 0.92 | 0.83 | 0.83 | 0.89 | 0.83 | 0.97 | 0.90 | |
\(f=1\)
| 0.90 | 0.89 | 0.92 | 0.89 | 0.89 | 0.92 | 0.92 | 0.92 | 0.96 | 0.92 | 0.98 | 0.96 | 0.92 | 0.92 | 0.70 | 0.69 | 0.75 | 0.93 | 0.88 | 0.88 | 0.94 | 0.90 | 0.97 | 0.89 | |
\(f=2\)
| 0.92 | 0.92 | 0.92 | 0.89 | 0.89 | 0.92 | 0.96 | 0.96 | 0.96 | 0.92 | 0.97 | 0.92 | 0.71 | 0.71 | 0.71 | 0.69 | 0.74 | 0.92 | 0.94 | 0.94 | 0.94 | 0.93 | 0.97 | 0.89 | |
\(f=4\)
| 0.92 | 0.91 | 0.92 | 0.92 | 0.90 | 0.92 | 0.97 | 0.97 | 0.97 | 0.97 | 0.97 | 0.96 | 0.70 | 0.71 | 0.71 | 0.70 | 0.74 | 0.91 | 0.94 | 0.94 | 0.94 | 0.95 | 0.97 | 0.90 | |
\(f=5\)
| 0.91 | 0.91 | 0.91 | 0.91 | 0.90 | 0.92 | 0.97 | 0.97 | 0.97 | 0.97 | 0.98 | 0.97 | 0.71 | 0.71 | 0.71 | 0.71 | 0.74 | 0.82 | 0.95 | 0.94 | 0.95 | 0.94 | 0.97 | 0.90 | |
ACA | 0.86 | 0.83 | 0.87 | 0.83 | 0.89 | 0.88 | 0.89 | 0.87 | 0.91 | 0.86 | 0.97 | 0.91 | 0.76 | 0.75 | 0.71 | 0.71 | 0.74 | 0.86 | 0.88 | 0.86 | 0.90 | 0.86 | 0.97 | 0.89 | |
6 |
\(f=3\)
| 0.95 | 0.95 | 0.93 | 0.95 | 0.93 | 0.93 | 0.95 | 0.95 | 0.92 | 0.95 | 0.92 | 0.91 | 0.95 | 0.95 | 0.94 | 0.95 | 0.94 | 0.92 | 0.96 | 0.96 | 0.94 | 0.96 | 0.94 | 0.72 |
\(f=1\)
| 0.95 | 0.95 | 0.95 | 0.94 | 0.95 | 0.93 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.93 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.91 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.72 | |
\(f=2\)
| 0.94 | 0.94 | 0.93 | 0.94 | 0.93 | 0.94 | 0.95 | 0.95 | 0.95 | 0.94 | 0.95 | 0.92 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.93 | 0.95 | 0.95 | 0.96 | 0.95 | 0.96 | 0.72 | |
ACA | 0.94 | 0.94 | 0.93 | 0.94 | 0.93 | 0.93 | 0.93 | 0.95 | 0.94 | 0.95 | 0.94 | 0.92 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.92 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.72 | |
7 |
\(f=5\)
| 0.75 | 0.75 | 0.75 | 0.40 | 0.85 | 0.75 | 0.75 | 0.75 | 0.75 | 0.40 | 0.95 | 0.75 | 0.69 | 0.66 | 0.69 | 0.49 | 0.78 | 0.68 | 0.66 | 0.66 | 0.66 | 0.45 | 0.66 | 0.67 |
\(f=2\)
| 0.75 | 0.75 | 0.75 | 0.83 | 0.84 | 0.75 | 0.75 | 0.74 | 0.75 | 0.83 | 0.95 | 0.75 | 0.69 | 0.68 | 0.69 | 0.78 | 0.78 | 0.65 | 0.66 | 0.66 | 0.66 | 0.41 | 0.67 | 0.67 | |
\(f=6\)
| 1.0 | 1.0 | 0.75 | 1.0 | 0.84 | 0.75 | 0.98 | 0.98 | 0.75 | 0.98 | 0.96 | 0.75 | 1.0 | 1.0 | 0.67 | 1.0 | 0.78 | 0.69 | 0.66 | 0.66 | 0.66 | 0.66 | 0.67 | 0.67 | |
ACA | 0.83 | 0.83 | 0.75 | 0.74 | 0.84 | 0.75 | 0.82 | 0.82 | 0.75 | 0.73 | 0.95 | 0.75 | 0.79 | 0.78 | 0.68 | 0.75 | 0.78 | 0.67 | 0.66 | 0.66 | 0.66 | 0.50 | 0.66 | 0.67 | |
8 |
\(f=5\)
| 0.67 | 0.67 | 0.67 | 0.67 | 0.90 | 0.67 | 0.67 | 0.67 | 0.67 | 0.67 | 0.75 | 0.67 | 0.59 | 0.61 | 0.62 | 0.59 | 0.61 | 0.56 | 0.52 | 0.56 | 0.52 | 0.50 | 0.67 | 0.67 |
\(f=2\)
| 0.67 | 0.67 | 0.67 | 0.67 | 0.91 | 0.67 | 0.67 | 0.67 | 0.67 | 0.67 | 0.74 | 0.67 | 0.59 | 0.61 | 0.59 | 0.61 | 0.62 | 0.58 | 0.51 | 0.52 | 0.51 | 0.48 | 0.67 | 0.67 | |
\(f=1\)
| 0.65 | 0.63 | 0.64 | 0.66 | 0.91 | 0.65 | 0.67 | 0.67 | 0.67 | 0.67 | 0.75 | 0.67 | 0.56 | 0.58 | 0.56 | 0.61 | 0.62 | 0.54 | 0.52 | 0.52 | 0.52 | 0.48 | 0.67 | 0.67 | |
ACA | 0.66 | 0.65 | 0.66 | 0.66 | 0.90 | 0.66 | 0.67 | 0.67 | 0.67 | 0.67 | 0.74 | 0.67 | 0.58 | 0.6 | 0.59 | 0.60 | 0.6 | 0.56 | 0.51 | 0.53 | 0.51 | 0.48 | 0.67 | 0.67 | |
9 |
\(f=5\)
| 0.80 | 0.80 | 0.77 | 0.77 | 1.0 | 0.77 | 0.80 | 0.80 | 0.77 | 0.77 | 0.98 | 0.77 | 0.78 | 0.80 | 0.72 | 0.71 | 0.87 | 0.68 | 0.79 | 0.80 | 0.63 | 0.63 | 0.80 | 0.71 |
\(f=4\)
| 0.97 | 0.97 | 0.97 | 0.97 | 1.0 | 0.77 | 0.97 | 0.97 | 0.97 | 0.97 | 0.98 | 0.78 | 0.96 | 0.96 | 0.96 | 0.96 | 0.87 | 0.72 | 0.80 | 0.80 | 0.80 | 0.80 | 0.80 | 0.72 | |
\(f=6\)
| 1.0 | 1.0 | 1.0 | 0.97 | 1.0 | 0.78 | 0.97 | 0.97 | 0.97 | 0.97 | 0.98 | 0.78 | 1.0 | 1.0 | 1.0 | 0.96 | 0.87 | 0.68 | 0.80 | 0.80 | 0.80 | 0.80 | 0.80 | 0.73 | |
\(f=3\)
| 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.78 | 1.0 | 1.0 | 1.0 | 1.0 | 0.98 | 0.78 | 0.99 | 0.99 | 0.99 | 1.0 | 0.87 | 0.66 | 0.80 | 0.80 | 0.80 | 0.80 | 0.80 | 0.72 | |
ACA | 0.94 | 0.94 | 0.93 | 0.92 | 1 | 0.77 | 0.93 | 0.93 | 0.92 | 0.92 | 0.98 | 0.77 | 0.93 | 0.93 | 0.91 | 0.90 | 0.87 | 0.68 | 0.79 | 0.8 | 0.75 | 0.75 | 0.8 | 0.72 | |
10 |
\(f=5\)
| 0.85 | 0.85 | 0.84 | 0.85 | 0.85 | 0.70 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.70 | 0.74 | 0.75 | 0.75 | 0.75 | 0.75 | 0.53 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 |
\(f=3\)
| 0.84 | 0.85 | 0.85 | 0.85 | 0.85 | 0.68 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.71 | 0.74 | 0.75 | 0.76 | 0.75 | 0.75 | 0.56 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | |
\(f=9\)
| 0.85 | 0.85 | 0.85 | 0.84 | 0.85 | 0.72 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.72 | 0.75 | 0.76 | 0.75 | 0.75 | 0.76 | 0.64 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | |
\(f=1\)
| 0.85 | 0.84 | 0.85 | 0.84 | 0.84 | 0.79 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.76 | 0.74 | 0.74 | 0.75 | 0.75 | 0.74 | 0.72 | 0.65 | 0.55 | 0.64 | 0.55 | 0.55 | 0.64 | |
\(f=7\)
| 0.85 | 0.85 | 0.85 | 0.85 | 0.85 | 0.81 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.82 | 0.75 | 0.75 | 0.74 | 0.75 | 0.75 | 0.81 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | |
ACA | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.74 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.74 | 0.74 | 0.75 | 0.75 | 0.75 | 0.75 | 0.65 | 0.65 | 0.63 | 0.64 | 0.63 | 0.63 | 0.64 | |
11 |
\(f=13\)
| 0.87 | 0.88 | 0.87 | 0.88 | 0.88 | 0.73 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.73 | 0.96 | 0.96 | 0.97 | 0.97 | 0.97 | 0.52 | 0.95 | 0.94 | 0.94 | 0.95 | 0.94 | 0.96 |
\(f=4\)
| 0.87 | 0.87 | 0.88 | 0.87 | 0.88 | 0.87 | 0.95 | 0.94 | 0.95 | 0.95 | 0.95 | 0.82 | 0.97 | 0.97 | 0.97 | 0.97 | 0.96 | 0.82 | 0.94 | 0.94 | 0.94 | 0.94 | 0.93 | 0.96 | |
\(f=8\)
| 0.87 | 0.87 | 0.87 | 0.88 | 0.88 | 0.91 | 0.95 | 0.95 | 0.96 | 0.95 | 0.95 | 0.80 | 0.97 | 0.97 | 0.97 | 0.97 | 0.97 | 0.87 | 0.94 | 0.94 | 0.94 | 0.94 | 0.93 | 0.95 | |
\(f=3\)
| 0.88 | 0.88 | 0.87 | 0.88 | 0.87 | 0.91 | 0.95 | 0.95 | 0.96 | 0.95 | 0.95 | 0.85 | 0.97 | 0.96 | 0.97 | 0.97 | 0.96 | 0.87 | 0.93 | 0.94 | 0.95 | 0.94 | 0.94 | 0.95 | |
\(f=1\)
| 0.88 | 0.87 | 0.88 | 0.88 | 0.87 | 0.88 | 0.95 | 0.95 | 0.96 | 0.96 | 0.94 | 0.82 | 0.96 | 0.96 | 0.97 | 0.97 | 0.97 | 0.88 | 0.95 | 0.95 | 0.94 | 0.94 | 0.94 | 0.95 | |
ACA | 0.87 | 0.87 | 0.87 | 0.87 | 0.87 | 0.86 | 0.95 | 0.94 | 0.95 | 0.95 | 0.94 | 0.80 | 0.96 | 0.96 | 0.97 | 0.97 | 0.96 | 0.79 | 0.94 | 0.94 | 0.94 | 0.94 | 0.93 | 0.95 | |
12 |
\(f=2\)
| 0.70 | 0.70 | 0.70 | 0.70 | 0.70 | 0.71 | 0.76 | 0.76 | 0.76 | 0.76 | 0.76 | 0.71 | 0.67 | 0.67 | 0.67 | 0.67 | 0.67 | 0.64 | 0.77 | 0.77 | 0.77 | 0.77 | 0.77 | 0.75 |
\(f=8\)
| 0.71 | 0.70 | 0.71 | 0.70 | 0.71 | 0.71 | 0.76 | 0.76 | 0.76 | 0.76 | 0.76 | 0.72 | 0.67 | 0.67 | 0.67 | 0.67 | 0.67 | 0.68 | 0.76 | 0.77 | 0.76 | 0.76 | 0.76 | 0.74 | |
\(f=5\)
| 0.70 | 0.70 | 0.70 | 0.70 | 0.70 | 0.71 | 0.76 | 0.76 | 0.76 | 0.76 | 0.76 | 0.75 | 0.68 | 0.67 | 0.68 | 0.68 | 0.68 | 0.68 | 0.76 | 0.76 | 0.76 | 0.76 | 0.76 | 0.74 | |
\(f=4\)
| 0.70 | 0.70 | 0.70 | 0.70 | 0.70 | 0.71 | 0.76 | 0.76 | 0.76 | 0.76 | 0.76 | 0.73 | 0.67 | 0.67 | 0.67 | 0.67 | 0.67 | 0.69 | 0.76 | 0.77 | 0.76 | 0.77 | 0.76 | 0.74 | |
ACA | 0.70 | 0.7 | 0.70 | 0.7 | 0.70 | 0.71 | 0.76 | 0.76 | 0.76 | 0.76 | 0.76 | 0.72 | 0.67 | 0.67 | 0.67 | 0.67 | 0.67 | 0.67 | 0.76 | 0.76 | 0.76 | 0.76 | 0.76 | 0.74 | |
13 |
\(f=6\)
| 0.95 | 0.951 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.97 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.79 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | 0.70 |
\(f=4\)
| 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.77 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | 0.64 | |
\(f=1\)
| 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.79 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | |
\(f=2\)
| 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.88 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | |
\(f=5\)
| 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.90 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | 0.93 | |
ACA | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.82 | 0.65 | 0.65 | 0.65 | 0.65 | 0.65 | 0.71 | |
14 |
\(f=13\)
| 0.76 | 0.76 | 0.75 | 0.75 | 0.76 | 0.77 | 0.75 | 0.75 | 0.75 | 0.75 | 0.82 | 0.75 | 0.75 | 0.68 | 0.69 | 0.74 | 0.58 | 0.61 | 0.76 | 0.76 | 0.76 | 0.75 | 0.83 | 0.76 |
\(f=12\)
| 0.78 | 0.79 | 0.74 | 0.74 | 0.74 | 0.79 | 0.79 | 0.79 | 0.73 | 0.73 | 0.83 | 0.79 | 0.77 | 0.75 | 0.72 | 0.74 | 0.58 | 0.69 | 0.79 | 0.79 | 0.77 | 0.77 | 0.82 | 0.76 | |
\(f=8\)
| 0.85 | 0.77 | 0.85 | 0.75 | 0.76 | 0.77 | 0.85 | 0.80 | 0.85 | 0.75 | 0.82 | 0.72 | 0.84 | 0.77 | 0.83 | 0.73 | 0.57 | 0.69 | 0.80 | 0.80 | 0.80 | 0.77 | 0.83 | 0.76 | |
\(f=11\)
| 0.84 | 0.78 | 0.85 | 0.85 | 0.74 | 0.76 | 0.83 | 0.79 | 0.83 | 0.84 | 0.83 | 0.78 | 0.83 | 0.79 | 0.83 | 0.82 | 0.58 | 0.71 | 0.83 | 0.83 | 0.82 | 0.83 | 0.83 | 0.81 | |
\(f=10\)
| 0.80 | 0.73 | 0.80 | 0.83 | 0.76 | 0.77 | 0.81 | 0.79 | 0.81 | 0.82 | 0.83 | 0.81 | 0.79 | 0.75 | 0.77 | 0.80 | 0.58 | 0.72 | 0.84 | 0.83 | 0.84 | 0.84 | 0.83 | 0.81 | |
ACA | 0.80 | 0.76 | 0.79 | 0.78 | 0.75 | 0.77 | 0.80 | 0.78 | 0.79 | 0.77 | 0.82 | 0.77 | 0.79 | 0.74 | 0.76 | 0.76 | 0.57 | 0.68 | 0.80 | 0.80 | 0.79 | 0.79 | 0.82 | 0.78 |