nach oben

Empirical Software Engineering

Erschienen in:

01.03.2021

An exploratory study on the introduction and removal of different types of technical debt in deep learning frameworks

verfasst von: Jiakun Liu, Qiao Huang, Xin Xia, Emad Shihab, David Lo, Shanping Li

Erschienen in: Empirical Software Engineering | Ausgabe 2/2021

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

To complete tasks faster, developers often have to sacrifice the quality of the software. Such compromised practice results in the increasing burden to developers in future development. The metaphor, technical debt, describes such practice. Prior research has illustrated the negative impact of technical debt, and many researchers investigated how developers deal with a certain type of technical debt. However, few studies focused on the removal of different types of technical debt in practice. To fill this gap, we use the introduction and removal of different types of self-admitted technical debt (i.e., SATD) in 7 deep learning frameworks as an example. This is because deep learning frameworks are some of the most important software systems today due to their prevalent use in life-impacting deep learning applications. Moreover, the field of the development of different deep learning frameworks is the same, which enables us to find common behaviors on the removal of different types of technical debt across projects. By mining the file history of these frameworks, we find that design debt is introduced the most along the development process. As for the removal of technical debt, we find that requirement debt is removed the most, and design debt is removed the fastest. Most of test debt, design debt, and requirement debt are removed by the developers who introduced them. Based on the introduction and removal of different types of technical debt, we discuss the evolution of the frequencies of different types of technical debt to depict the unresolved sub-optimal trade-offs or decisions that are confronted by developers along the development process. We also discuss the removal patterns of different types of technical debt, highlight future research directions, and provide recommendations for practitioners.

Vorheriger Artikel On the Removal of Feature Toggles

Nächster Artikel Automated end-to-end management of the modeling lifecycle in deep learning

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

https://www.tensorflow.org/api_docs/python/tf/contrib/gan

https://www.tensorflow.org/api_docs/python/tf/contrib/tpu

https://mxnet.incubator.apache.org/api/python/symbol/symbo-l.html#mxnet.symbol.BatchNorm

https://blog.github.com/2017-01-31-introducing-topics/

https://github.com/tensorflow/tensorflow

https://github.com/keras-team/keras

https://github.com/deeplearning4j/deeplearning4j

https://github.com/BVLC/caffe

https://github.com/pytorch/pytorch

https://github.com/apache/incubator-mxnet

https://github.com/Microsoft/CNTK

https://dwheeler.com/sloccount/

https://www.srcml.org/

https://docs.python.org/3/library/tokenize.html

tensorflow/tensorflow/core/kernels/cwise_ops.h

tensorflow/tensorflow/python/feature_column/feature_column_test.py

tensorflow/tensorflow/core/kernels/hinge-loss.h

deeplearning4j/deeplearning4j-nn/src/main/java/org/deeplearning4j/nn/params/Batch-NormalizationParamInitializer.java

tensorflow/tensorflow/python/keras/backend_test.py

caffe/include/caffe/common.hpp

caffe/src/caffe/layers/pooling_layer.cpp

https://www.qasymphony.com/software-testing-tools/qtest-manager/test-case-management/

https://github.com/facebookincubator/gloo

https://github.com/facebookincubator/gloo/commits/1d9e62aff9d7143129a69c8eb23e8351-e686ff3a/gloo/scatter.cc

Al-Qizwini M, Barjasteh I, Al-Qassab H, Radha H (2017) Deep learning algorithm for autonomous driving using googlenet. In: 2017 IEEE intelligent vehicles symposium (IV). IEEE, pp 89–96

Alves NS, Ribeiro LF, Caires V, Mendes TS, Spínola RO (2014) Towards an ontology of terms on technical debt. In: Sixth international workshop on managing technical debt (MTD), 2014. IEEE, pp 1–7

Bavota G, Russo B (2016) A large-scale empirical study on self-admitted technical debt. In: IEEE/ACM 13th working conference on mining software repositories (MSR), 2016. IEEE, pp 315–326

Brown N, Cai Y, Guo Y, Kazman R, Kim M, Kruchten P, Lim E, MacCormack A, Nord R, Ozkaya I, et al. (2010) Managing technical debt in software-reliant systems. In: Proceedings of the FSE/SDP workshop on future of software engineering research. ACM, pp 47–52

Carpenter M (1997) Survival analysis: a self-learning text

Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494CrossRef

Cox DR, Oakes D (1984) Analysis of survival data, vol 21. CRC Press

Cunningham W (1993) The wycash portfolio management system. ACM SIGPLAN OOPS Messenger 4(2):29–30CrossRef

de Almeida RR, Kulesza U, Treude C, Lima AHG, et al. (2018) Aligning technical debt prioritization with business objectives: a multiple-case study. In: 2018 IEEE international conference on software maintenance and evolution (ICSME), IEEE, pp 655–664

Dunn OJ (1961) Multiple comparisons among means. J Am Stat Assoc 56(293):52–64MathSciNetCrossRef

Ernst NA (2012) On the role of requirements in understanding and managing technical debt. In: Proceedings of the third international workshop on managing technical debt, IEEE Press, pp 61–64

Ernst NA, Bellomo S, Ozkaya I, Nord RL, Gorton I (2015) Measure it? Manage it? Ignore it? Software practitioners and technical debt. In: Proceedings of the 2015 10th joint meeting on foundations of software engineering, ACM, pp 50–60

Fontana FA, Ferme V, Spinelli S (2012) Investigating the impact of code smells debt on quality code evaluation. In: Proceedings of the third international workshop on managing technical debt, IEEE Press, pp 15–22

Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. arXiv:1406.2661 [cs, stat]

Huang Q, Shihab E, Xia X, Lo D, Li S (2018) Identifying self-admitted technical debt in open source projects using text mining. Empir Softw Eng 23(1):418–451CrossRef

Huval B, Wang T, Tandon S, Kiske J, Song W, Pazhayampallil J, Andriluka M, Rajpurkar P, Migimatsu T, Cheng-Yue R, et al. (2015) An empirical evaluation of deep learning on highway driving. arXiv:150401716

Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167 [cs]

Islam MJ, Nguyen G, Pan R, Rajan H (2019) A comprehensive study on deep learning bug characteristics. In: ESEC/FSE’19: the ACM joint european software engineering conference and symposium on the foundations of software engineering (ESEC/FSE), ESEC/FSE 2019

Jouppi NP, Young C, Patil N, Patterson D, Agrawal G, Bajwa R, Bates S, Bhatia S, Boden N, Borchers A, et al. (2017) In-datacenter performance analysis of a tensor processing unit. In: 2017 ACM/IEEE 44th annual international symposium on computer architecture (ISCA), IEEE, pp 1–12

Kaplan EL, Meier P (1958) Nonparametric estimation from incomplete observations. J Am Stat Assoc 53(282):457–481MathSciNetCrossRef

Klinger T, Tarr P, Wagstrom P, Williams C (2011) An enterprise perspective on technical debt. In: Proceedings of the 2nd Workshop on managing technical debt, ACM, pp 35–38

Kruchten P, Nord RL, Ozkaya I, Falessi D (2013) Technical debt: towards a crisper definition report on the 4th international workshop on managing technical debt. ACM SIGSOFT Software Engineering Notes 38(5):51–54CrossRef

Kruskal WH, Wallis WA (1952) Use of ranks in one-criterion variance analysis. J Am Stat Assoc 47(260):583–621CrossRef

Li Z, Avgeriou P, Liang P (2015) A systematic mapping study on technical debt and its management. J Syst Softw 101:193–220CrossRef

Lim E, Taksande N, Seaman C (2012) A balancing act: what software practitioners have to say about technical debt. IEEE Softw 29(6):22–27CrossRef

Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, Van Der Laak JA, Van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88CrossRef

Liu J, Huang Q, Xia X, Shihab E, Lo D, Li S (2020) Is using deep learning frameworks free? Characterizing technical debt in deep learning frameworks. In: Proceedings of the 42nd ACM/IEEE international conference on software engineering - SE in society (ICSE’20 SEIS), ACM/IEEE

Liu Z, Huang Q, Xia X, Shihab E, Lo D, Li S (2018) Satd detector: a text-mining-based self-admitted technical debt detection tool. In: Proceedings of the 40th international conference on software engineering: companion proceedings, ACM, pp 9–12

Ma L, Juefei-Xu F, Zhang F, Sun J, Xue M, Li B, Chen C, Su T, Li L, Liu Y, et al. (2018) Deepgauge: multi-granularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, pp 120–131

Maldonado E, Shihab E (2015) Detecting and quantifying different types of self-admitted technical debt. In: Proceedings of the 7th IEEE international workshop on managing technical debt (MTD’15), pp 9–15

Maldonado E, Abdalkareem R, Shihab E, Serebrenik A (2017) An empirical study on the removal of self-admitted technical debt. In: Proceedings of the 33rd international conference on software maintenance and evolution (ICSME’17), IEEE

Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat: 18(1):50–60MathSciNetCrossRef

Mantel N (1966) Evaluation of survival data and two new rank order statistics arising in its consideration. Cancer Chemother Rep 50:163–170

McElreath R (2020) Statistical rethinking: A Bayesian course with examples in R and Stan. CRC Press

McHugh ML (2012) Interrater reliability: the kappa statistic. Biochemia Medica: Biochemia Medica 22(3):276–282MathSciNetCrossRef

Miller RG Jr (2011) Survival analysis, vol 66. Wiley

Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: automated whitebox testing of deep learning systems. In: Proceedings of the 26th symposium on operating systems principles, ACM, pp 1–18

Potdar A, Shihab E (2014) An exploratory study on self-admitted technical debt. In: Proceedings of the 30th IEEE international conference on software maintenance and evolution (ICSME’14), pp 91–100

Quesenberry CP Jr, Fireman B, Hiatt RA, Selby JV (1989) A survival analysis of hospitalization among patients with acquired immunodeficiency syndrome. Am J Public Health 79(12):1643–1647CrossRef

Sallab AE, Abdou M, Perot E, Yogamani S (2017) Deep reinforcement learning framework for autonomous driving. Electronic Imaging 2017(19):70–76CrossRef

Sculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, Chaudhary V, Young M, Crespo JF, Dennison D (2015) Hidden technical debt in machine learning systems. In: Advances in neural information processing systems, pp 2503–2511

Seaman C, Guo Y (2011) Measuring and monitoring technical debt. In: Advances in Computers, Elsevier, vol 82, pp 25–46

Shalev-Shwartz S, Shammah S, Shashua A (2016) Safe, multi-agent, reinforcement learning for autonomous driving. arXiv:161003295

Shull F (2011) Perfectionists in a world of finite resources. IEEE Softw 28(2):4–6CrossRef

Spencer D (2009) Card sorting: designing usable categories. Rosenfeld Media

Spínola RO, Vetrò A, Zazworka N, Seaman C, Shull F (2013) Investigating technical debt folklore: shedding some light on technical debt opinion. In: 2013 4th international workshop on managing technical debt (MTD), IEEE, pp 1–7

Sun Y, Huang X, Kroening D (2018a) Testing deep neural networks. arXiv:180304792

Sun Y, Wu M, Ruan W, Huang X, Kwiatkowska M, Kroening D (2018b) Concolic testing for deep neural networks. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, pp 109–119

Tian Y, Pei K, Jana S, Ray B (2018) Deeptest: automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th international conference on software engineering, ACM, pp 303–314

Wehaibi S, Shihab E, Guerrouj L (2016) Examining the impact of self-admitted technical debt on software quality. In: IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER), 2016, IEEE, vol 1, pp 179–188

Wei L, Lachin J (1984) Two-sample asymptotically distribution-free tests for incomplete multivariate observations. J Am Stat Assoc 79(387):653–661MathSciNetCrossRef

Zampetti F, Serebrenik A, Di Penta M (2018) Was self-admitted technical debt removal a real removal? An in-depth perspective. In: 2018 IEEE/ACM 15th international conference on mining software repositories (MSR), IEEE, pp 526–536

Zampetti F, Serebrenik A, Di Penta M (2020) Automatically learning patterns for self-admitted technical debt removal. In: 2020 IEEE 27th international conference on software analysis, evolution and reengineering (SANER), IEEE, pp 355–366

Zazworka N, Shaw MA, Shull F, Seaman C (2011) Investigating the impact of design debt on software quality. In: Proceedings of the 2nd workshop on managing technical debt, ACM, New York, NY, USA, MTD ’11, pp 17–23. https://doi.org/10.1145/1985362.1985366

Zazworka N, Spínola RO, Vetro A, Shull F, Seaman C (2013) A case study on effectively identifying technical debt. In: Proceedings of the 17th international conference on evaluation and assessment in software engineering, ACM, New York, NY, USA, EASE ’13. https://doi.org/10.1145/2460999.2461005, pp 42–47

Zhang M, Zhang Y, Zhang L, Liu C, Khurshid S (2018a) Deeproad: gan-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, pp 132–142

Zhang Y, Chen Y, Cheung SC, Xiong Y, Zhang L (2018b) An empirical study on tensorflow program bugs. International Symposium on Software Testing and Analysis

Titel: An exploratory study on the introduction and removal of different types of technical debt in deep learning frameworks
verfasst von: Jiakun Liu
Qiao Huang
Xin Xia
Emad Shihab
David Lo
Shanping Li
Publikationsdatum: 01.03.2021
Verlag: Springer US
Erschienen in: Empirical Software Engineering / Ausgabe 2/2021
Print ISSN: 1382-3256
Elektronische ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-020-09917-5

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Weitere Artikel der Ausgabe 2/2021

Learning lenient parsing & typing via indirect supervision

Wikifying software artifacts

Comparing the results of replications in software engineering

A multi-dimensional analysis of technical lag in Debian-based Docker images

Automatically recommending components for issue reports using deep learning

variED: an editor for collaborative, real-time feature modeling

Premium Partner