1 Introduction
2 Literature review
2.1 Ensemble approaches for concept-drifting data streams
References | Task | Strategy |
---|---|---|
Wang et al. (2003) | Classification | Batch growing ensemble using each chunk of data to build a new model |
Fan (2004) | Classification | Batch growing ensemble using selected past data to build new models |
Kolter and Maloof (2005) | Classification | Ensemble based on incremental algorithms to adapt to every new instance. New models are added according to a threshold parameter and excluded based on age or accuracy. |
Gao et al. (2008) | Classification | Batch growing ensemble and sampling mechanism to deal with unbalanced datasets |
Masud et al. (2011) | Classification | Batch growing ensemble designed to deal with limited labelled data and novel class detection |
Bifet et al. (2009) | Classification | Fixed ensemble that uses drift detector and restarting trees to update the model. |
Elwell and Polikar (2011) | Classification | Batch growing ensemble that updates using a dynamically weighted majority voting scheme |
Kadlec and Gabrys (2011) | Regression | Fixed ensemble based on PLS with local and global updating. |
Farid et al. (2013) | Classification | Fixed ensemble that trains new models based on optimised data selection and detects new classes based on clustering |
Ikonomovska et al. (2015) | Regression | Incremental Hoeffding-based regression trees built based on bagging and low-performing models are excluded |
Soares and Araújo (2015a) | Regression | PLS models are updated at every new instance. Each model is weighted according to its accuracy on a sliding window |
Soares and Araújo (2015b) | Regression | The models (ELM variant) are updated at every instance, and the weights are updated based on accuracy on a sliding window |
Yin et al. (2015) | Classification | Combination of ensembles that builds a new ensemble at each new chunk of data |
Ding et al. (2017) | Regression | NNRW models trained using decorrelation learning that can be updated at each instance or by chunk |
Ren et al. (2018) | Classification | Bach growing ensemble that incorporates drift detection mechanisms and applies online and chunk-based updating mechanisms to cope with various types of drift |
Iwashita et al. (2019) | Classification | Bach growing ensemble using OPF-based classifiers that consider approaches to train the new models (full memory, no memory, and window of fixed size) |
2.2 Base models
3 Methodology
3.1 Hyperparameter optimisation
3.2 Neural networks with random weights
3.3 Proposed bagging of NNRW approach
3.4 Online DNNE
4 Experiments
4.1 Datasets
Name | Dataset | # samples | # attributes |
---|---|---|---|
3-D Mex. Hat (Mex) | Synthetic | 5000 | 2 |
Friedman #1 (Fried1) | Synthetic | 5000 | 5 |
Friedman # 3 (Fried3) | Synthetic | 5000 | 4 |
Multi (Multi) | Synthetic | 5000 | 5 |
California housinga (Housing) | Benchmark | 20,640 | 8 |
Wine qualityb (Quality) | Benchmark | 4898 | 11 |
Condition-based maintenanceb (Maintenance) | Benchmark | 11,934 | 14 |
Appliances energy predictionb (Energy) | Benchmark | 20,640 | 26 |
4.2 Hyperparameter adjustment
- Number of models (M) Number of base models that compose the ensemble;
- Number of nodes (N) Number of hidden nodes for each base model;
- Regularisation factor (R) In the case of B-NNRW, the regularisation factor is responsible for penalising large weights in the optimisation process. For DNNE, it acts to control the decorrelation term in the optimisation function;
- Random weights range (W) This hyperparameter determines the interval in which the initial random weights are uniformly distributed. Although the authors (Alhamdoosh and Wang 2014; Ding et al. 2017) suggest the weights of DNNE to be set in the interval [− 1, 1], this hyperparameter plays a vital role in the accuracy of the models. The effect of initial weights in RVFL was investigated by Zhang and Sugantham (2016);
- Number of attributes (A) This hyperparameter is exclusive of B-NNRW. It determines the fraction of total inputs that are randomly selected to train each NNRW base model.
Factors | Algorithm | Levels | ||||
---|---|---|---|---|---|---|
M | B-NNRW | 40 | 60 | 80 | 100 | 120 |
DNNE | 3 | 6 | 9 | 12 | 15 | |
N | B-NNRW | 8x | 10x | 12x | 14x | 16x |
DNNE | 60 | 80 | 100 | 120 | 140 | |
R | B-NNRW | 0.0001 | 0.0010 | 0.0050 | 0.0100 | 0.0500 |
DNNE | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | |
W | B-NNRW | [− 0.50, 0.50] | [− 0.75, 0.75] | [− 1.00, 1.00] | [− 1.25, 1.25] | [− 1.50, 1.50] |
DNNE | [− 0.005, 0.005] | [− 0.020, 0.020] | [− 0.035, 0.035] | [− 0.050, 0.050] | [− 0.065, 0.065] | |
A | B-NNRW | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
DNNE | – | – | – | – | – |
4.3 Hyperparameter analysis
p value | F0 | % | Cum% | p value | F0 | % | Cum% | ||
---|---|---|---|---|---|---|---|---|---|
A) Mex | B) Fried1 | ||||||||
P × M | 0.06 | 1.6 | 19.0 | 19.0 | W | ≪ 0.01 | 28,287.6 | 93.7 | 93.7 |
M × W | 0.17 | 1.3 | 15.6 | 34.6 | M | ≪ 0.01 | 957.8 | 3.2 | 96.9 |
N | 0.36 | 1.1 | 12.8 | 47.4 | M × W | ≪ 0.01 | 547.8 | 1.8 | 98.7 |
C) Fried3 | D) Multi | ||||||||
W | ≪ 0.01 | 4719.8 | 88.1 | 88.1 | W | ≪ 0.01 | 5361.3 | 93.1 | 93.1 |
M | ≪ 0.01 | 335.7 | 6.3 | 94.4 | M | ≪ 0.01 | 160.9 | 2.8 | 95.9 |
M × W | ≪ 0.01 | 149.3 | 2.8 | 97.2 | M × W | ≪ 0.01 | 96.6 | 1.7 | 97.6 |
E) Housing | F) Quality | ||||||||
W | ≪ 0.01 | 122.5 | 69.7 | 69.7 | W | ≪ 0.01 | 124.3 | 40.2 | 40.2 |
M | ≪ 0.01 | 19.5 | 11.1 | 80.8 | N | ≪ 0.01 | 104.9 | 33.9 | 74.0 |
N | ≪ 0.01 | 7.0 | 4.0 | 84.7 | R | ≪ 0.01 | 35.4 | 11.4 | 85.5 |
G) Maintenance | H) Energy | ||||||||
W | ≪ 0.01 | 39,694.5 | 98.7 | 98.7 | W | ≪ 0.01 | 286.8 | 46.5 | 46.5 |
M | ≪ 0.01 | 210.8 | 0.5 | 99.2 | N | ≪ 0.01 | 233.9 | 37.9 | 84.3 |
M × W | ≪ 0.01 | 126.5 | 0.3 | 99.5 | M | ≪ 0.01 | 36.2 | 5.9 | 90.2 |
p value | F0 | % | Cum% | p value | F0 | % | Cum% | ||
---|---|---|---|---|---|---|---|---|---|
A) Mex | B) Fried1 | ||||||||
A | ≪ 0.01 | 53.4 | 55.2 | 55.2 | A | ≪ 0.01 | 183,045.5 | 91.9 | 91.9 |
W | ≪ 0.01 | 18.5 | 19.1 | 74.3 | P | ≪ 0.01 | 7160.7 | 3.6 | 95.5 |
P | ≪ 0.01 | 6.9 | 7.2 | 81.5 | N | ≪ 0.01 | 5398.2 | 2.7 | 98.3 |
C) Fried3 | D) Multi | ||||||||
A | ≪ 0.01 | 33,207.2 | 93.2 | 93.2 | A | ≪ 0.01 | 146,166.6 | 99.8 | 99.8 |
P | ≪ 0.01 | 1485.6 | 4.2 | 97.3 | W | ≪ 0.01 | 146.2 | 0.1 | 99.9 |
N | ≪ 0.01 | 477.6 | 1.3 | 98.7 | N | ≪ 0.01 | 41.4 | 0.0 | 99.9 |
E) Housing | F) Quality | ||||||||
W | ≪ 0.01 | 334.9 | 45.6 | 45.6 | N | ≪ 0.01 | 39.2 | 28.7 | 28.7 |
R | ≪ 0.01 | 121.5 | 16.6 | 62.2 | A | ≪ 0.01 | 24.3 | 17.8 | 46.6 |
A | ≪ 0.01 | 121.4 | 16.5 | 78.7 | W | ≪ 0.01 | 15.4 | 11.3 | 57.9 |
G) Maintenance | G) Energy | ||||||||
R | ≪ 0.01 | 235,822.8 | 64.2 | 64.2 | N | ≪ 0.01 | 173.9 | 41.0 | 41.0 |
W | ≪ 0.01 | 58,369.6 | 15.9 | 80.1 | M | ≪ 0.01 | 45.2 | 10.7 | 51.7 |
A | ≪ 0.01 | 40,241.0 | 11.0 | 91.0 | A | ≪ 0.01 | 41.5 | 9.8 | 61.5 |
Mex | Fried1 | Fried3 | Multi | Housing | Quality | Maintenance | Energy | |
---|---|---|---|---|---|---|---|---|
B-NNRW | ||||||||
P | 0.05 | 0.0001 | 0.0001 | 0.0001 | 0.0010 | 0.0050 | 0.0001 | 0.0050 |
W | ± 1.5 | ± 0.5 | ± 0.75 | ± 0.5 | ± 1.50 | ± 1.00 | ± 1.50 | ± 1.00 |
N | 10 × | 16 × | 14 × | 16 × | 14 × | 14 × | 16 × | 14 × |
M | 40 | 100 | 100 | 80 | 80 | 60 | 40 | 100 |
A | 0.9 | 0.9 | 0.9 | 0.9 | 0.7 | 0.7 | 1.0 | 0.8 |
DNNE | ||||||||
P | 0.5 | 0.4 | 0.2 | 0.1 | 0.1 | 0.1 | 0.3 | 0.5 |
W | ± 0.05 | ± 0.065 | ± 0.065 | ± 0.05 | ± 0.005 | ± 0.005 | ± 0.065 | ± 0.035 |
N | 120 | 100 | 60 | 140 | 60 | 60 | 60 | 120 |
M | 9 | 12 | 9 | 9 | 12 | 6 | 3 | 12 |
4.4 Data stream evaluation set-up
4.5 Results and discussion
Mex | Fried1 | Fried3 | Multi | |||||
---|---|---|---|---|---|---|---|---|
MSE | SD | MSE | SD | MSE | SD | MSE | SD | |
O-DNNE | 2.718E−02 | 0.000 | 5.514 | 0.002 | 0.872E−02 | 0.000 | 0.268 | 0.000 |
B-NNRW | 2.697E−02 | 0.000 | 10.943 | 0.033 | 1.803E−02 | 0.000 | 0.516 | 0.001 |
BP-NNRW | ||||||||
0.1 | 2.696E−02 | 0.000 | 10.884 | 0.045 | 1.773E−02 | 0.000 | 0.515 | 0.001 |
0.3 | 2.695E−02 | 0.000 | 10.793 | 0.037 | 1.661E−02 | 0.000 | 0.513 | 0.001 |
0.5 | 2.695E−02 | 0.000 | 10.731 | 0.044 | 1.599E−02 | 0.001 | 0.512 | 0.001 |
0.7 | 2.694E−02 | 0.000 | 10.613 | 0.031 | 1.498E−02 | 0.000 | 0.511 | 0.001 |
0.9 | 2.695E−02 | 0.000 | 10.457 | 0.051 | 1.402E−02 | 0.001 | 0.509 | 0.001 |
Average | 2.695E−02 | 10.696 | 1.587E−02 | 0.512 | ||||
BR-NNRW | ||||||||
0.1 | 2.683E−02 | 0.000 | 3.374 | 0.054 | 1.601E−02 | 0.000 | 0.065 | 0.002 |
0.2 | 2.680E−02 | 0.000 | 2.547 | 0.046 | 1.548E−02 | 0.000 | 0.046 | 0.001 |
0.3 | 2.680E−02 | 0.000 | 2.294 | 0.032 | 1.499E−02 | 0.000 | 0.039 | 0.001 |
0.4 | 2.683E−02 | 0.000 | 2.176 | 0.018 | 1.492E−02 | 0.000 | 0.037 | 0.000 |
0.5 | 2.686E−02 | 0.000 | 2.106 | 0.028 | 1.497E−02 | 0.000 | 0.035 | 0.001 |
Average | 2.682E−02 | 2.499 | 1.527E−02 | 0.044 |
0–1000 | 1001–2000 | 2001–3000 | 3001–4000 | Updating time (s) | |
---|---|---|---|---|---|
Mex | |||||
O-DNNE | 3.614E−02 | 3.026E−02 | 2.448E−02 | 1.786E−02 | 2.393 |
BR-NNRW | 3.577E−02 | 3.004E−02 | 2.405E−02 | 1.735E−02 | 0.003 |
Fried1 | |||||
O-DNNE | 0.107 | 2.625 | 5.737 | 13.586 | 3.152 |
BR-NNRW | 0.410 | 1.616 | 2.245 | 4.153 | 0.015 |
Fried3 | |||||
O-DNNE | 0.680E−02 | 0.770E−02 | 1.073E−02 | 0.964E−02 | 0.539 |
BR-NNRW | 1.495E−02 | 1.552E−02 | 1.552E−02 | 1.397E−02 | 0.009 |
Multi | |||||
O-DNNE | 0.112E−02 | 8.803E−02 | 34.332E−02 | 63.884E−02 | 3.462 |
BR-NNRW | 0.335E−02 | 3.437E−02 | 4.811E−02 | 5.454E−02 | 0.012 |
Housing | ||||||||
---|---|---|---|---|---|---|---|---|
Updating interval | 250 | 500 | 1000 | 1500 | ||||
MSE | SD | MSE | SD | MSE | SD | MSE | SD | |
O-DNNE | 1.08E+10 | 1.53E+07 | 9.85E+09 | 1.51E+07 | 9.13E+09 | 2.06E+07 | 8.70E+09 | 2.98E+07 |
B-NNRW | 1.14E+10 | 1.93E+08 | 1.04E+10 | 1.48E+08 | 9.83E+09 | 1.96E+08 | 9.18E+09 | 1.26E+08 |
BP-NNRW | ||||||||
0.1 | 1.12E+10 | 1.50E+08 | 1.02E+10 | 1.82E+08 | 9.47E+09 | 2.05E+08 | 9.09E+09 | 1.88E+08 |
0.3 | 1.09E+10 | 1.13E+08 | 9.92E+09 | 1.97E+08 | 9.31E+09 | 1.56E+08 | 8.84E+09 | 2.13E+08 |
0.5 | 1.08E+10 | 1.11E+08 | 9.85E+09 | 1.59E+08 | 9.19E+09 | 1.46E+08 | 8.67E+09 | 1.11E+08 |
0.7 | 1.07E+10 | 1.09E+08 | 9.84E+09 | 1.78E+08 | 9.17E+09 | 1.48E+08 | 8.59E+09 | 1.57E+08 |
0.9 | 1.09E+10 | 8.89E+07 | 1.01E+10 | 2.34E+08 | 9.35E+09 | 2.33E+08 | 8.88E+09 | 1.80E+08 |
Average | 1.09E+10 | 9.99E+09 | 9.30E+09 | 8.81E+09 | ||||
BR-NNRW | ||||||||
0.1 | 9.83E+09 | 9.96E+07 | 1.02E+10 | 6.66E+08 | 9.47E+09 | 1.70E+08 | 8.41E+09 | 1.63E+08 |
0.2 | 1.07E+10 | 1.02E+08 | 1.21E+10 | 1.21E+09 | 9.99E+09 | 1.42E+08 | 8.70E+09 | 1.46E+08 |
0.3 | 1.11E+10 | 1.36E+08 | 1.30E+10 | 1.12E+09 | 1.05E+10 | 1.34E+08 | 9.20E+09 | 1.06E+08 |
0.4 | 1.14E+10 | 1.17E+08 | 1.30E+10 | 1.02E+09 | 1.12E+10 | 1.37E+08 | 9.70E+09 | 1.33E+08 |
0.5 | 1.17E+10 | 1.04E+08 | 1.34E+10 | 8.99E+08 | 1.18E+10 | 1.63E+08 | 1.02E+10 | 2.26E+08 |
Average | 1.10E+10 | 1.09E+10 | 9.82E+09 | 9.00E+09 |
Quality | ||||||||
---|---|---|---|---|---|---|---|---|
Updating interval | 250 | 500 | 1000 | 1500 | ||||
MSE | SD | MSE | SD | MSE | SD | MSE | SD | |
O-DNNE | 0.549 | 0.006 | 0.557 | 0.008 | 0.546 | 0.005 | 0.575 | 0.009 |
B-NNRW | 0.584 | 0.004 | 0.580 | 0.003 | 0.565 | 0.005 | 0.571 | 0.003 |
BP-NNRW | ||||||||
0.1 | 0.577 | 0.005 | 0.574 | 0.006 | 0.559 | 0.005 | 0.566 | 0.004 |
0.3 | 0.575 | 0.006 | 0.571 | 0.005 | 0.557 | 0.007 | 0.564 | 0.005 |
0.5 | 0.575 | 0.007 | 0.567 | 0.005 | 0.553 | 0.007 | 0.567 | 0.006 |
0.7 | 0.575 | 0.006 | 0.569 | 0.006 | 0.557 | 0.006 | 0.570 | 0.007 |
0.9 | 0.605 | 0.008 | 0.599 | 0.010 | 0.585 | 0.013 | 0.601 | 0.009 |
Average | 0.581 | 0.576 | 0.562 | 0.574 | ||||
BR-NNRW | ||||||||
0.1 | 0.562 | 0.003 | 0.546 | 0.005 | 0.532 | 0.004 | 0.550 | 0.004 |
0.2 | 0.550 | 0.005 | 0.536 | 0.003 | 0.519 | 0.003 | 0.538 | 0.003 |
0.3 | 0.546 | 0.004 | 0.533 | 0.004 | 0.513 | 0.003 | 0.532 | 0.004 |
0.4 | 0.539 | 0.004 | 0.535 | 0.004 | 0.510 | 0.003 | 0.528 | 0.003 |
0.5 | 0.537 | 0.004 | 0.535 | 0.004 | 0.509 | 0.004 | 0.526 | 0.003 |
Average | 0.547 | 0.537 | 0.516 | 0.535 |
Maintenance | ||||||||
---|---|---|---|---|---|---|---|---|
Updating interval | 250 | 500 | 1000 | 1500 | ||||
MSE | SD | MSE | SD | MSE | SD | MSE | SD | |
O-DNNE | 5.95E−07 | 3.56E−08 | 5.27E−07 | 4.49E−08 | 5.13E−07 | 6.62E−08 | 8.73E−07 | 1.77E−07 |
B-NNRW | 3.84E−06 | 3.06E−07 | 3.69E−06 | 3.93E−07 | 3.52E−06 | 4.25E−07 | 3.86E−06 | 4.05E−07 |
BP-NNRW | ||||||||
0.1 | 3.72E−06 | 3.53E−07 | 3.39E−06 | 2.483E−07 | 3.33E−06 | 4.49E−07 | 3.45E−06 | 4.06E−07 |
0.3 | 3.56E−06 | 3.03E−07 | 3.06E−06 | 4.642E−07 | 3.03E−06 | 2.83E−07 | 3.16E−06 | 2.71E−07 |
0.5 | 3.07E−06 | 3.08E−07 | 2.92E−06 | 3.773E−07 | 2.78E−06 | 3.80E−07 | 2.90E−06 | 2.90E−07 |
0.7 | 2.79E−06 | 2.48E−07 | 2.70E−06 | 2.982E−07 | 2.45E−06 | 3.03E−07 | 2.70E−06 | 3.79E−07 |
0.9 | 2.51E−06 | 2.60E−07 | 2.36E−06 | 3.903E−07 | 2.11E−06 | 3.35E−07 | 2.37E−06 | 4.47E−07 |
Average | 3.13E−06 | 2.89E−06 | 2.74E−06 | 2.92E−06 | ||||
BR-NNRW | ||||||||
0.1 | 1.04E−06 | 2.12E−08 | 9.22E−07 | 2.40E−08 | 5.61E−07 | 5.51E−08 | 9.11E−07 | 1.37E−07 |
0.2 | 1.08E−06 | 1.53E−08 | 9.47E−07 | 1.99E−08 | 5.15E−07 | 3.79E−08 | 8.35E−07 | 8.70E−08 |
0.3 | 1.10E−06 | 1.58E−08 | 9.56E−07 | 1.88E−08 | 5.03E−07 | 4.89E−08 | 8.45E−07 | 8.01E−08 |
0.4 | 1.12E−06 | 1.81E−08 | 9.64E−07 | 1.53E−08 | 4.98E−07 | 3.38E−08 | 8.19E−07 | 7.95E−08 |
0.5 | 1.13E−06 | 9.95E−09 | 9.73E−07 | 1.42E−08 | 5.04E−07 | 2.44E−08 | 8.58E−07 | 8.14E−08 |
Average | 1.09E−06 | 9.53E−07 | 5.16E−07 | 8.53E−07 |
Energy | ||||||||
---|---|---|---|---|---|---|---|---|
Updating interval | 250 | 500 | 1000 | 1500 | ||||
MSE | SD | MSE | SD | MSE | SD | MSE | SD | |
O-DNNE | 1.72E+04 | 8.45E+02 | 2.48E+04 | 2.67E+03 | 2.60E+04 | 2.34E+03 | 2.54E+04 | 2.16E+03 |
B-NNRW | 3.50E+04 | 2.79E+03 | 3.31E+04 | 2.24E+03 | 3.48E+04 | 2.04E+03 | 3.52E+04 | 3.32E+03 |
BP-NNRW | ||||||||
0.1 | 3.54E+04 | 2.30E+03 | 3.34E+04 | 2.10E+03 | 3.48E+04 | 1.81E+03 | 3.38E+04 | 2.94E+03 |
0.3 | 3.45E+04 | 2.05E+03 | 3.30E+04 | 2.02E+03 | 3.26E+04 | 2.46E+03 | 3.19E+04 | 2.09E+03 |
0.5 | 3.46E+04 | 1.32E+03 | 3.30E+04 | 2.28E+03 | 3.36E+04 | 2.07E+03 | 3.13E+04 | 2.78E+03 |
0.7 | 3.81E+04 | 2.15E+03 | 3.45E+04 | 2.01E+03 | 3.56E+04 | 2.34E+03 | 3.35E+04 | 2.59E+03 |
0.9 | 5.32E+04 | 3.55E+03 | 4.93E+04 | 2.78E+03 | 4.71E+04 | 2.84E+03 | 4.38E+04 | 4.10E+03 |
Average | 3.92E+04 | 3.66E+04 | 3.67E+04 | 3.49E+04 | ||||
BR-NNRW | ||||||||
0.1 | 1.39E+04 | 3.51E+02 | 1.63E+04 | 4.27E+02 | 1.74E+04 | 4.44E+02 | 1.74E+04 | 7.55E+02 |
0.2 | 1.45E+04 | 3.62E+02 | 1.63E+04 | 5.93E+02 | 1.63E+04 | 6.21E+02 | 1.65E+04 | 5.55E+02 |
0.3 | 1.48E+04 | 3.85E+02 | 1.71E+04 | 7.09E+02 | 1.62E+04 | 4.76E+02 | 1.61E+04 | 3.72E+02 |
0.4 | 1.51E+04 | 3.95E+02 | 1.82E+04 | 5.77E+02 | 1.63E+04 | 3.86E+02 | 1.62E+04 | 5.44E+02 |
0.5 | 1.54E+04 | 3.71E+02 | 1.87E+04 | 6.04E+02 | 1.62E+04 | 4.11E+02 | 1.66E+04 | 4.91E+02 |
Average | 1.47E+04 | 1.73E+04 | 1.65E+04 | 1.66E+04 |
Dataset | Housing | Energy | Maintenance | Quality | ||||
---|---|---|---|---|---|---|---|---|
Interval/rep. rate | 1500/0.1 | 250/0.1 | 1000/0.4 | 1000/0.5 | ||||
Avg | SD | Avg | SD | Avg | SD | Avg | SD | |
MSE | ||||||||
BR-NNRW | 8.41E+09 | 1.63E+08 | 1.39E+04 | 3.51E+02 | 4.98E−07 | 3.38E−08 | 0.509 | 0.004 |
O-DNNE | 8.70E+09 | 2.98E+07 | 1.72E+04 | 8.45E+02 | 5.13E−07 | 6.62E−08 | 0.546 | 0.005 |
% decrease in MSE | 3.4% | 18.9% | 2.9% | 6.8% | ||||
Updating time (s) | ||||||||
BR-NNRW | 0.023 | 0.002 | 0.044 | 0.004 | 0.135 | 0.006 | 0.088 | 0.005 |
O-DNNE | 1.599 | 0.029 | 5.037 | 0.083 | 0.080 | 0.004 | 0.322 | 0.012 |
Times faster | 69.7 | 113.6 | 0.6 | 3.7 |