1 Introduction
2 Background
2.1 Rerunning-Based Flaky Test Detection
2.1.1 Rerun
2.1.2 iDFlakies
2.1.3 Pairwise
2.2 The Flake16 Feature Set
3 The CANNIER Approach
3.1 Motivating Example
3.2 Single-Model CANNIER
3.3 Multi-Model CANNIER
4 Tooling
4.1 pytest-CANNIER
# | Feature | Description |
---|---|---|
1 | Read Count | Number of times the filesystem had to perform input (I/O statistics fields 2022). |
2 | Write Count | Number of times the filesystem had to perform output (I/O statistics fields 2022). |
3 | Run Time | Elapsed wall-clock time of the whole test case execution. |
4 | Wait Time | Elapsed wall-clock time spent waiting for input/output operations to complete. |
5 | Context Switches | Number of voluntary context switches. |
6 | Covered Lines | Number of lines covered. |
7 | Source Covered Lines | Number of lines covered that are not part of test cases. |
8 | Covered Changes | Total number of times each covered line has been modified in the last 75 commits. |
9 | Max. Threads | Peak number of concurrently running threads. |
10 | Max. Children | Peak number of concurrently running child processes. |
11 | Max. Memory | Peak memory usage. |
12 | AST Depth | Maximum depth of nested program statements in the test case code. |
13 | Assertions | Number of assertion statements in the test case code. |
14 | External Modules | Number of non-standard modules (i.e., libraries) used by the test case. |
15 | Halstead Volume | |
16 | Cyclomatic Complexity | |
17 | Test Lines of Code | |
18 | Maintainability |
4.2 CANNIER-Framework
4.2.1 Model Training and Evaluation Data
ϕ | Name | Domain (\(\mathcal {T}_{\phi }\)) | Positive Class (\(\mathcal {T}^{+}_{\phi }\)) |
---|---|---|---|
1 | NOD-vs-Rest | \(\mathcal {T}\) | \(\{t | t \in \mathcal {T}_{\phi }, 0 < B_{t} < N_{B}\}\) |
2 | NOD-vs-Victim | \(\{t | t \in \mathcal {T}, B_{t} < N_{B} \wedge S_{t} > 0\}\) | \(\{t | t \in \mathcal {T}_{\phi }, 0 < B_{t} < N_{B}\}\) |
3 | Victim-vs-Rest | \(\mathcal {T}\) | \(\{ t | t \in \mathcal {T}_{\phi }, (B_{t} = 0 \vee B_{t} = N_{B}) \wedge B_{t} \neq S_{t} \}\) |
4 | Polluter-vs-Rest | \(\mathcal {T}\) | \(\{ p | p \in \mathcal {T}_{\phi }, \exists v \in \mathcal {T}_{\phi } - \{p\} (o_{p, v} \neq o_{v}) \}\) |
4.2.2 Model Training and Evaluation Procedure
4.2.3 Technique Evaluation Procedure
5 Empirical Evaluation
5.1 Subject Set
GitHub repository | Tests | NOD | Victims | Polluters | Pairs | Cost (s) |
---|---|---|---|---|---|---|
apache/airflow | 3251 | 66 | 279 | 3241 | 45819 | 7.77 × 102 |
celery/celery | 2332 | − | 15 | 17 | 24 | 1.31 × 102 |
quantumlib/Cirq | 12048 | − | 17 | 2 | 32 | 8.67 × 102 |
conan-io/conan | 3687 | − | 13 | 13 | 18 | 1.48 × 103 |
dask/dask | 8015 | 1 | 1 | 37 | 37 | 1.34 × 103 |
encode/django-rest-framework | 1402 | − | 1 | 3 | 3 | 2.63 × 103 |
spesmilo/electrum | 542 | 1 | 1 | 2 | 2 | 5.99 × 101 |
Flexget/Flexget | 1330 | 1 | 4 | 3 | 4 | 1.73 × 103 |
fonttools/fonttools | 3448 | 1 | 42 | − | − | 1.19 × 102 |
graphql-python/graphene | 346 | − | 1 | 1 | 1 | 1.73 × 101 |
facebookresearch/hydra | 1538 | − | 19 | 348 | 952 | 1.77 × 102 |
HypothesisWorks/hypothesis | 4348 | 5 | 6 | 3699 | 7401 | 3.92 × 103 |
ipython/ipython | 807 | 6 | 297 | 796 | 118869 | 1.10 × 102 |
celery/kombu | 1024 | 2 | 23 | 20 | 63 | 3.62 × 101 |
apache/libcloud | 9809 | 3 | 133 | 471 | 1686 | 2.66 × 102 |
Delgan/loguru | 1255 | 4 | 21 | 6 | 26 | 6.23 × 101 |
mitmproxy/mitmproxy | 1232 | − | 18 | 338 | 735 | 3.12 × 101 |
python-pillow/Pillow | 2567 | − | 26 | 2 | 26 | 9.38 × 101 |
PrefectHQ/prefect | 7035 | 25 | 20 | 227 | 230 | 1.56 × 103 |
PyGithub/PyGithub | 711 | − | 4 | 678 | 2712 | 5.55 × 101 |
Pylons/pyramid | 2633 | − | 4 | 252 | 383 | 5.98 × 101 |
psf/requests | 535 | 5 | − | − | − | 1.40 × 102 |
saltstack/salt | 2672 | 12 | 4 | 65 | 65 | 2.52 × 102 |
scikit-image/scikit-image | 6275 | − | 12 | 5882 | 5890 | 2.54 × 103 |
mwaskom/seaborn | 1020 | − | 8 | 1 | 7 | 5.01 × 102 |
pypa/setuptools | 694 | 1 | 23 | 4 | 4 | 2.08 × 102 |
sunpy/sunpy | 1857 | − | 2 | 9 | 9 | 4.31 × 102 |
tornadoweb/tornado | 1159 | 1 | 1 | − | − | 4.03 × 101 |
urllib3/urllib3 | 1320 | 15 | 1 | − | − | 8.57 × 101 |
xonsh/xonsh | 4776 | 9 | 19 | 3114 | 9459 | 1.81 × 102 |
Overall | 89668 | 158 | 1015 | 19231 | 194457 | 1.99 × 104 |
Topic classifier | Frequency |
---|---|
Communications | 1 |
Education | 1 |
Education :: Testing | 1 |
Internet | 5 |
Internet :: Proxy Servers | 1 |
Internet :: WWW/HTTP | 5 |
Internet :: WWW/HTTP :: WSGI | 1 |
Multimedia | 3 |
Multimedia :: Graphics | 3 |
Multimedia :: Graphics :: Capture | 1 |
Multimedia :: Graphics :: Capture :: Digital Camera | 1 |
Multimedia :: Graphics :: Capture :: Screen Capture | 1 |
Multimedia :: Graphics :: Graphics Conversion | 2 |
Multimedia :: Graphics :: Viewers | 1 |
Scientific/Engineering | 4 |
Scientific/Engineering :: Physics | 1 |
Scientific/Engineering :: Visualization | 1 |
Security | 1 |
Software Development | 12 |
Software Development :: Build Tools | 1 |
Software Development :: Libraries | 7 |
Software Development :: Libraries :: Python Modules | 3 |
Software Development :: Object Brokering | 1 |
Software Development :: Testing | 2 |
System | 9 |
System :: Archiving | 1 |
System :: Archiving :: Packaging | 1 |
System :: Clustering | 1 |
System :: Distributed Computing | 4 |
System :: Logging | 1 |
System :: Monitoring | 1 |
System :: Networking | 2 |
System :: Networking :: Monitoring | 1 |
System :: Shells | 1 |
Text Processing | 1 |
Text Processing :: Fonts | 1 |
Utilities | 1 |
5.2 Methodology
5.2.1 RQ1. How Effective is Machine Learning-Based Flaky Test Detection?
5.2.2 RQ2. What Impact do Mean Feature Vectors have on the Performance of Machine Learning-Based Flaky Test Detection?
5.2.3 RQ3. What Contribution do Individual Features have on the Output Values of Machine Learning Models for Detecting Flaky Tests?
5.2.4 RQ4. What Impact Does CANNIER have on the Performance and Time Cost of Rerunning-Based Flaky Test Detection?
5.3 Threats to Validity
6 Results
6.1 RQ1. How Effective is Machine Learning-Based Flaky Test Detection?
Model | Trees | Balancing | MCC |
---|---|---|---|
(a) NOD-vs-Rest | |||
ExtraTrees | 100 | SMOTE | 0.532 |
ExtraTrees | 100 | +Tomek | 0.529 |
ExtraTrees | 75 | SMOTE | 0.527 |
ExtraTrees | 50 | SMOTE | 0.526 |
ExtraTrees | 25 | SMOTE | 0.521 |
ExtraTrees | 50 | +Tomek | 0.519 |
ExtraTrees | 75 | +Tomek | 0.519 |
ExtraTrees | 25 | +Tomek | 0.507 |
RandomForest | 100 | SMOTE | 0.488 |
RandomForest | 75 | SMOTE | 0.479 |
RandomForest | 50 | SMOTE | 0.479 |
RandomForest | 25 | SMOTE | 0.477 |
(b) NOD-vs-Victim | |||
RandomForest | 75 | SMOTE | 0.693 |
RandomForest | 100 | SMOTE | 0.690 |
ExtraTrees | 100 | SMOTE | 0.689 |
ExtraTrees | 50 | SMOTE | 0.686 |
ExtraTrees | 75 | SMOTE | 0.686 |
RandomForest | 50 | SMOTE | 0.685 |
RandomForest | 75 | +Tomek | 0.684 |
RandomForest | 25 | SMOTE | 0.679 |
RandomForest | 100 | +Tomek | 0.675 |
ExtraTrees | 100 | +Tomek | 0.673 |
ExtraTrees | 75 | +Tomek | 0.669 |
RandomForest | 25 | +Tomek | 0.668 |
(c) Victim-vs-Rest | |||
ExtraTrees | 75 | SMOTE | 0.520 |
ExtraTrees | 100 | SMOTE | 0.519 |
ExtraTrees | 50 | SMOTE | 0.518 |
ExtraTrees | 100 | +Tomek | 0.515 |
ExtraTrees | 75 | +Tomek | 0.513 |
ExtraTrees | 50 | +Tomek | 0.511 |
ExtraTrees | 25 | SMOTE | 0.510 |
ExtraTrees | 25 | +Tomek | 0.502 |
RandomForest | 50 | SMOTE | 0.501 |
RandomForest | 75 | SMOTE | 0.498 |
RandomForest | 100 | SMOTE | 0.498 |
RandomForest | 25 | SMOTE | 0.490 |
(d) Polluter-vs-Rest | |||
RandomForest | 100 | SMOTE | 0.946 |
RandomForest | 75 | SMOTE | 0.945 |
RandomForest | 50 | SMOTE | 0.944 |
RandomForest | 25 | SMOTE | 0.943 |
RandomForest | 100 | +Tomek | 0.941 |
ExtraTrees | 100 | SMOTE | 0.941 |
RandomForest | 75 | +Tomek | 0.940 |
ExtraTrees | 75 | SMOTE | 0.940 |
RandomForest | 50 | +Tomek | 0.940 |
ExtraTrees | 50 | SMOTE | 0.939 |
RandomForest | 25 | +Tomek | 0.937 |
ExtraTrees | 100 | +Tomek | 0.937 |
Project | TN | FN | FP | TP | MCC |
---|---|---|---|---|---|
(a) NOD-vs-Rest (μ = 0.52,σ = 0.29) | |||||
airflow | 3159 | 26 | 26 | 40 | 0.59 |
celery | 2332 | − | − | − | ⊥ |
Cirq | 12048 | − | − | − | ⊥ |
conan | 3687 | − | 0 | − | ⊥ |
dask | 8014 | 1 | 0 | − | ⊥ |
django-rest-... | 1402 | − | − | − | ⊥ |
electrum | 540 | 1 | 1 | − | ⊥ |
Flexget | 1329 | 1 | − | − | ⊥ |
fonttools | 3447 | 1 | − | − | ⊥ |
graphene | 346 | − | − | − | ⊥ |
hydra | 1538 | − | − | − | ⊥ |
hypothesis | 4341 | 5 | 2 | − | 0.00 |
ipython | 800 | 2 | 0 | 4 | 0.76 |
kombu | 1022 | 2 | − | − | ⊥ |
libcloud | 9806 | 3 | − | − | ⊥ |
loguru | 1250 | 2 | 1 | 2 | ⊥ |
mitmproxy | 1232 | − | − | − | ⊥ |
Pillow | 2567 | − | 0 | − | ⊥ |
prefect | 7005 | 13 | 5 | 12 | 0.58 |
PyGithub | 711 | − | − | − | ⊥ |
pyramid | 2633 | − | − | − | ⊥ |
requests | 530 | 2 | − | 3 | 0.82 |
salt | 2660 | 4 | 0 | 8 | 0.82 |
scikit-image | 6275 | − | 0 | − | ⊥ |
seaborn | 1020 | − | − | − | ⊥ |
setuptools | 693 | 1 | − | − | ⊥ |
sunpy | 1857 | − | − | − | ⊥ |
tornado | 1157 | 1 | 1 | − | ⊥ |
urllib3 | 1295 | 13 | 10 | 2 | 0.16 |
xonsh | 4763 | 5 | 4 | 4 | 0.45 |
Overall | 89460 | 83 | 50 | 75 | 0.53 |
(b) NOD-vs-Victim (μ = 0.55,σ = 0.22) | |||||
airflow | 250 | 11 | 25 | 45 | 0.66 |
celery | 14 | − | 1 | − | ⊥ |
Cirq | 17 | − | − | − | ⊥ |
conan | 13 | − | 0 | − | ⊥ |
dask | 1 | − | − | − | ⊥ |
django-rest-... | 0 | − | 1 | − | ⊥ |
electrum | 0 | 1 | 1 | 0 | ⊥ |
Flexget | 3 | 1 | 1 | − | ⊥ |
fonttools | 42 | − | − | − | ⊥ |
graphene | 1 | − | − | − | ⊥ |
hydra | 19 | − | − | − | ⊥ |
hypothesis | 6 | 3 | 0 | 0 | ⊥ |
ipython | 296 | 2 | 1 | 4 | 0.71 |
kombu | 23 | 1 | − | − | ⊥ |
libcloud | 133 | 3 | 0 | − | ⊥ |
loguru | 20 | 1 | 1 | 2 | 0.55 |
mitmproxy | 6 | − | 0 | − | ⊥ |
Pillow | 26 | − | 0 | − | ⊥ |
prefect | 17 | 1 | 3 | 16 | 0.79 |
PyGithub | 4 | − | − | − | ⊥ |
pyramid | 4 | − | − | − | ⊥ |
requests | − | 0 | − | 4 | ⊥ |
salt | 3 | 1 | 1 | 11 | 0.69 |
scikit-image | 12 | − | − | − | ⊥ |
seaborn | 8 | − | 0 | − | ⊥ |
setuptools | 23 | 0 | 0 | 1 | ⊥ |
sunpy | 2 | − | − | − | ⊥ |
tornado | 0 | − | 1 | − | ⊥ |
urllib3 | − | 3 | 1 | 12 | 0.12 |
xonsh | 14 | 3 | 5 | 6 | 0.33 |
Overall | 957 | 32 | 42 | 99 | 0.69 |
Project | TN | FN | FP | TP | MCC |
---|---|---|---|---|---|
(a) Victim-vs-Rest (μ = 0.51,σ = 0.24) | |||||
airflow | 2880 | 81 | 92 | 198 | 0.67 |
celery | 2316 | 9 | 1 | 6 | 0.59 |
Cirq | 12030 | 3 | 1 | 14 | 0.88 |
conan | 3666 | 8 | 8 | 5 | 0.38 |
dask | 8013 | 1 | 1 | − | ⊥ |
django-rest-... | 1400 | 1 | 1 | − | ⊥ |
electrum | 539 | 1 | 2 | − | ⊥ |
Flexget | 1325 | 3 | 1 | 1 | ⊥ |
fonttools | 3395 | 6 | 11 | 36 | 0.82 |
graphene | 345 | 1 | 0 | − | ⊥ |
hydra | 1513 | 13 | 6 | 6 | 0.37 |
hypothesis | 4341 | 3 | 1 | 3 | 0.57 |
ipython | 377 | 206 | 133 | 91 | 0.05 |
kombu | 1000 | 12 | 1 | 11 | 0.68 |
libcloud | 9612 | 86 | 64 | 47 | 0.38 |
loguru | 1225 | 5 | 9 | 16 | 0.69 |
mitmproxy | 1213 | 13 | 1 | 5 | 0.50 |
Pillow | 2530 | 18 | 11 | 8 | 0.37 |
prefect | 7014 | 16 | 1 | 4 | 0.40 |
PyGithub | 707 | 1 | 0 | 3 | ⊥ |
pyramid | 2629 | 3 | 0 | 1 | ⊥ |
requests | 535 | − | − | − | ⊥ |
salt | 2668 | 4 | 0 | − | ⊥ |
scikit-image | 6261 | 5 | 2 | 7 | 0.69 |
seaborn | 1009 | 8 | 3 | 0 | ⊥ |
setuptools | 668 | 5 | 3 | 18 | 0.81 |
sunpy | 1855 | 2 | − | − | ⊥ |
tornado | 1156 | 1 | 2 | − | 0.00 |
urllib3 | 1318 | 1 | 1 | − | ⊥ |
xonsh | 4753 | 14 | 4 | 5 | 0.36 |
Overall | 88292 | 529 | 361 | 486 | 0.52 |
(b) Polluter-vs-Rest (μ = 0.46,σ = 0.34) | |||||
airflow | 0 | 5 | 10 | 3236 | 0.01 |
celery | 2311 | 15 | 4 | 2 | 0.20 |
Cirq | 12032 | 1 | 14 | 1 | 0.12 |
conan | 3621 | 6 | 53 | 7 | 0.25 |
dask | 7947 | 0 | 31 | 37 | 0.73 |
django-rest-... | 1397 | 3 | 2 | − | ⊥ |
electrum | 525 | 2 | 15 | − | 0.01 |
Flexget | 1327 | 3 | 0 | − | ⊥ |
fonttools | 3443 | − | 5 | − | ⊥ |
graphene | 344 | 1 | 1 | − | ⊥ |
hydra | 1186 | 13 | 4 | 335 | 0.97 |
hypothesis | 563 | 11 | 86 | 3688 | 0.91 |
ipython | 7 | 165 | 4 | 631 | 0.13 |
kombu | 1002 | 15 | 2 | 5 | 0.42 |
libcloud | 9315 | 195 | 23 | 276 | 0.73 |
loguru | 1249 | 6 | 0 | 0 | ⊥ |
mitmproxy | 880 | 71 | 14 | 267 | 0.82 |
Pillow | 2534 | 2 | 31 | 0 | 0.02 |
prefect | 6781 | 184 | 27 | 43 | 0.33 |
PyGithub | 17 | 8 | 16 | 670 | 0.58 |
pyramid | 2355 | 52 | 26 | 200 | 0.82 |
requests | 530 | − | 4 | − | ⊥ |
salt | 2605 | 16 | 2 | 49 | 0.85 |
scikit-image | 331 | 191 | 62 | 5691 | 0.71 |
seaborn | 990 | 1 | 29 | 0 | 0.01 |
setuptools | 686 | 2 | 4 | 2 | 0.39 |
sunpy | 1835 | 6 | 13 | 2 | 0.21 |
tornado | 1156 | − | 3 | − | ⊥ |
urllib3 | 1310 | − | 10 | − | ⊥ |
xonsh | 1608 | 102 | 54 | 3012 | 0.93 |
Overall | 69889 | 1077 | 548 | 18154 | 0.95 |
6.2 RQ2. What Impact do Mean Feature Vectors have on the Performance of Machine Learning-based Flaky Test Detection?
6.3 RQ3. What Contribution do Individual Features have on the Output Values of Machine Learning Models for Detecting Flaky Tests?
6.4 RQ4. What Impact Does CANNIER have on the Performance and Time Cost of Rerunning-Based Flaky Test Detection?
Time cost (seconds) | |||||||
---|---|---|---|---|---|---|---|
Project | TN | FN | FP | TP | MCC | CANNIER+ | Original |
airflow | 3185 | 3 | − | 63 | 0.98 | 7.71 × 105 | 1.69 × 106 |
celery | 2332 | − | − | − | ⊥ | 8.53 × 104 | 3.28 × 105 |
Cirq | 12048 | − | − | − | ⊥ | 2.32 × 105 | 2.17 × 106 |
conan | 3687 | − | − | − | ⊥ | 2.42 × 105 | 3.70 × 106 |
dask | 8014 | − | − | 1 | 1.00 | 3.01 × 105 | 3.34 × 106 |
django-rest-... | 1402 | − | − | − | ⊥ | 8.85 × 104 | 6.57 × 106 |
electrum | 541 | 1 | − | − | ⊥ | 5.12 × 104 | 1.39 × 105 |
Flexget | 1329 | 1 | − | − | ⊥ | 7.25 × 104 | 4.32 × 106 |
fonttools | 3447 | 1 | − | − | ⊥ | 1.09 × 104 | 2.97 × 105 |
graphene | 346 | − | − | − | ⊥ | 2.55 × 103 | 4.31 × 104 |
hydra | 1538 | − | − | − | ⊥ | 2.34 × 104 | 4.42 × 105 |
hypothesis | 4343 | 4 | − | 1 | ⊥ | 1.68 × 106 | 9.57 × 106 |
ipython | 801 | 0 | − | 6 | 0.98 | 5.04 × 104 | 2.63 × 105 |
kombu | 1022 | 1 | − | 1 | ⊥ | 1.27 × 104 | 9.05 × 104 |
libcloud | 9806 | 1 | − | 2 | ⊥ | 5.00 × 103 | 6.66 × 105 |
loguru | 1251 | 2 | − | 2 | ⊥ | 4.73 × 104 | 1.48 × 105 |
mitmproxy | 1232 | − | − | − | ⊥ | 2.11 × 103 | 7.79 × 104 |
Pillow | 2567 | − | − | − | ⊥ | 2.89 × 104 | 2.35 × 105 |
prefect | 7010 | 2 | − | 23 | 0.96 | 4.05 × 105 | 3.80 × 106 |
PyGithub | 711 | − | − | − | ⊥ | 1.41 × 104 | 1.39 × 105 |
pyramid | 2633 | − | − | − | ⊥ | 1.10 × 103 | 1.49 × 105 |
requests | 530 | 0 | − | 5 | 1.00 | 8.48 × 104 | 3.46 × 105 |
salt | 2660 | 1 | − | 11 | 0.96 | 8.75 × 104 | 6.27 × 105 |
scikit-image | 6275 | − | − | − | ⊥ | 5.73 × 105 | 6.36 × 106 |
seaborn | 1020 | − | − | − | ⊥ | 6.41 × 104 | 1.25 × 106 |
setuptools | 693 | 1 | − | − | ⊥ | 1.40 × 105 | 4.75 × 105 |
sunpy | 1857 | − | − | − | ⊥ | 3.89 × 105 | 1.08 × 106 |
tornado | 1158 | 1 | − | − | ⊥ | 6.44 × 103 | 1.01 × 105 |
urllib3 | 1305 | 3 | − | 12 | 0.89 | 3.53 × 104 | 2.13 × 105 |
xonsh | 4767 | 1 | − | 8 | 0.94 | 2.94 × 104 | 4.50 × 105 |
Overall | 89510 | 24 | − | 134 | 0.92 | 5.54 × 106 | 4.91 × 107 |
Time cost (seconds) | |||||||
---|---|---|---|---|---|---|---|
Project | TN | FN | FP | TP | MCC | CANNIER+ | Original |
airflow | 275 | 3 | − | 53 | 0.97 | 9.22 × 106 | 7.24 × 107 |
celery | 15 | − | − | − | ⊥ | 6.95 × 104 | 2.06 × 105 |
Cirq | 17 | − | − | − | ⊥ | 3.23 × 104 | 4.47 × 106 |
conan | 13 | − | − | − | ⊥ | 2.47 × 104 | 4.84 × 105 |
dask | 1 | − | − | − | ⊥ | 1.87 × 104 | 6.60 × 105 |
django-rest-... | 1 | − | − | − | ⊥ | 1.00 × 106 | 9.66 × 105 |
electrum | 1 | 1 | − | 0 | ⊥ | 1.11 × 103 | 3.95 × 102 |
Flexget | 4 | 1 | − | 0 | ⊥ | 1.44 × 106 | 1.75 × 106 |
fonttools | 42 | − | − | − | ⊥ | 2.91 × 104 | 2.48 × 106 |
graphene | 1 | − | − | − | ⊥ | 2.42 × 102 | 4.27 × 103 |
hydra | 19 | − | − | − | ⊥ | 2.93 × 104 | 1.31 × 106 |
hypothesis | 6 | 1 | − | 2 | 0.85 | 6.32 × 105 | 9.55 × 105 |
ipython | 297 | 0 | − | 5 | 0.99 | 1.80 × 104 | 7.84 × 105 |
kombu | 23 | 1 | − | − | ⊥ | 4.63 × 103 | 1.38 × 105 |
libcloud | 133 | 1 | − | 2 | ⊥ | 8.73 × 104 | 8.74 × 106 |
loguru | 21 | 0 | − | 3 | 0.98 | 4.15 × 104 | 2.35 × 105 |
mitmproxy | 6 | − | − | − | ⊥ | 2.61 × 103 | 3.70 × 104 |
Pillow | 26 | − | − | − | ⊥ | 2.55 × 103 | 1.51 × 104 |
prefect | 20 | − | − | 17 | 1.00 | 7.84 × 105 | 1.04 × 106 |
PyGithub | 4 | − | − | − | ⊥ | 7.82 × 102 | 3.11 × 102 |
pyramid | 4 | − | − | − | ⊥ | 8.37 × 102 | 3.54 × 104 |
requests | − | − | − | 4 | ⊥ | 1.81 × 104 | 1.61 × 104 |
salt | 4 | − | − | 12 | 1.00 | 1.26 × 106 | 1.33 × 106 |
scikit-image | 12 | − | − | − | ⊥ | 4.89 × 105 | 6.98 × 105 |
seaborn | 8 | − | − | − | ⊥ | 7.64 × 104 | 1.29 × 105 |
setuptools | 23 | − | − | 1 | 1.00 | 7.53 × 104 | 1.95 × 105 |
sunpy | 2 | − | − | − | ⊥ | 6.20 × 103 | 1.94 × 105 |
tornado | 1 | − | − | − | ⊥ | 6.05 × 102 | 4.03 × 101 |
urllib3 | 1 | 0 | − | 15 | 0.99 | 1.25 × 104 | 1.13 × 104 |
xonsh | 19 | 1 | − | 8 | 0.95 | 5.18 × 105 | 5.75 × 105 |
Overall | 999 | 8 | − | 123 | 0.97 | 1.59 × 107 | 9.98 × 107 |
Time cost (seconds) | |||||
---|---|---|---|---|---|
Project | TP | P | TPR | CANNIER+ | Original |
airflow | 45490 | 45819 | 0.99 | 2.60 × 106 | 5.05 × 106 |
celery | 7 | 24 | 0.31 | 2.95 × 104 | 6.12 × 105 |
Cirq | 30 | 32 | 0.94 | 1.15 × 105 | 2.09 × 107 |
conan | − | 18 | − | 2.13 × 105 | 1.09 × 107 |
dask | 1 | 37 | 0.03 | 2.39 × 105 | 2.15 × 107 |
django-rest-... | 0 | 3 | 0.07 | 4.92 × 104 | 7.37 × 106 |
electrum | − | 2 | − | 7.57 × 103 | 6.49 × 104 |
Flexget | 2 | 4 | 0.43 | 1.70 × 104 | 4.60 × 106 |
fonttools | − | − | ⊥ | 1.93 × 104 | 8.19 × 105 |
graphene | − | 1 | − | 3.66 × 102 | 1.19 × 104 |
hydra | 839 | 952 | 0.88 | 3.48 × 104 | 5.44 × 105 |
hypothesis | 4071 | 7401 | 0.55 | 2.07 × 106 | 3.41 × 107 |
ipython | 112497 | 118869 | 0.95 | 1.56 × 105 | 1.78 × 105 |
kombu | 44 | 63 | 0.70 | 8.20 × 103 | 7.42 × 104 |
libcloud | 984 | 1686 | 0.58 | 1.42 × 105 | 5.23 × 106 |
loguru | 3 | 26 | 0.13 | 3.77 × 103 | 1.56 × 105 |
mitmproxy | 90 | 735 | 0.12 | 1.13 × 104 | 7.68 × 104 |
Pillow | 23 | 26 | 0.88 | 4.57 × 104 | 4.82 × 105 |
prefect | 103 | 230 | 0.45 | 5.29 × 105 | 2.19 × 107 |
PyGithub | 2703 | 2712 | 1.00 | 1.45 × 104 | 7.89 × 104 |
pyramid | 262 | 383 | 0.68 | 4.34 × 103 | 3.15 × 105 |
requests | − | − | ⊥ | 6.63 × 103 | 1.50 × 105 |
salt | 50 | 65 | 0.78 | 2.96 × 104 | 1.34 × 106 |
scikit-image | 5887 | 5890 | 1.00 | 6.30 × 106 | 3.19 × 107 |
seaborn | 5 | 7 | 0.72 | 8.31 × 104 | 1.02 × 106 |
setuptools | 4 | 4 | 1.00 | 1.76 × 104 | 2.89 × 105 |
sunpy | − | 9 | − | 1.48 × 105 | 1.60 × 106 |
tornado | − | − | ⊥ | 3.05 × 103 | 9.35 × 104 |
urllib3 | − | − | ⊥ | 7.90 × 103 | 2.26 × 105 |
xonsh | 9442 | 9459 | 1.00 | 1.15 × 105 | 1.73 × 106 |
Overall | 182538 | 194457 | 0.94 | 1.30 × 107 | 1.73 × 108 |