skip to main content
research-article

How Far Have We Progressed in Identifying Self-admitted Technical Debts? A Comprehensive Empirical Study

Published:23 July 2021Publication History
Skip Abstract Section

Abstract

Background. Self-admitted technical debt (SATD) is a special kind of technical debt that is intentionally introduced and remarked by code comments. Those technical debts reduce the quality of software and increase the cost of subsequent software maintenance. Therefore, it is necessary to find out and resolve these debts in time. Recently, many automatic approaches have been proposed to identify SATD. Problem. Popular IDEs support a number of predefined task annotation tags for indicating SATD in comments, which have been used in many projects. However, such clear prior knowledge is neglected by existing SATD identification approaches when identifying SATD. Objective. We aim to investigate how far we have really progressed in the field of SATD identification by comparing existing approaches with a simple approach that leverages the predefined task tags to identify SATD. Method. We first propose a simple heuristic approach that fuzzily Matches task Annotation Tags (MAT) in comments to identify SATD. In nature, MAT is an unsupervised approach, which does not need any data to train a prediction model and has a good understandability. Then, we examine the real progress in SATD identification by comparing MAT against existing approaches. Result. The experimental results reveal that: (1) MAT has a similar or even superior performance for SATD identification compared with existing approaches, regardless of whether non-effort-aware or effort-aware evaluation indicators are considered; (2) the SATDs (or non-SATDs) correctly identified by existing approaches are highly overlapped with those identified by MAT; and (3) supervised approaches misclassify many SATDs marked with task tags as non-SATDs, which can be easily corrected by their combinations with MAT. Conclusion. It appears that the problem of SATD identification has been (unintentionally) complicated by our community, i.e., the real progress in SATD comments identification is not being achieved as it might have been envisaged. We hence suggest that, when many task tags are used in the comments of a target project, future SATD identification studies should use MAT as an easy-to-implement baseline to demonstrate the usefulness of any newly proposed approach.

References

  1. Python Developer's Guide. 2020. Retrieved from https://www.python.org/dev/peps/pep-0350/.Google ScholarGoogle Scholar
  2. Tips about Eclipse. 2020. Retrieved from http://www.javaperspective.com/how-to-use-todo-and-fixme-task-tags-in-eclipse.html.++6.Google ScholarGoogle Scholar
  3. Tasklist of NetBeans. 2020. Retrieved from https://ui.netbeans.org/docs/hi/promoB/tasklist.html.Google ScholarGoogle Scholar
  4. To-Do List plugin of CodeBlocks. 2020. Retrieved from http://wiki.codeblocks.org/index.php/To-Do_List_plugin.Google ScholarGoogle Scholar
  5. Task Tags Preferences of Eclipse. 2020. Retrieved from https://www.eclipse.org/pdt/help/html/task_tags.htm.Google ScholarGoogle Scholar
  6. TODO comments of IntelliJ IDEA. 2020. Retrieved from https://www.jetbrains.com/help/idea/using-todo.html.Google ScholarGoogle Scholar
  7. Task List of Visual Studio. 2020. Retrieved from https://docs.microsoft.com/zh-cn/visualstudio/ide/using-the-task-list?view=vs-2015.Google ScholarGoogle Scholar
  8. Code Climate. 2020. Retrieved from https://codeclimate.com/.Google ScholarGoogle Scholar
  9. Android Studio. 2020. Retrieved from https://developer.android.com/.Google ScholarGoogle Scholar
  10. Qiao Huang, Emad Shihab, Xin Xia, David Lo, and Shanping Li. 2018. Identifying self-admitted technical debt in open source projects using text mining. Empir. Softw. Eng. 23, 1 (2018), 418–451. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Ward Cunningham. 1993. The WyCash portfolio management system. ACM SIGPLAN OOPS Mess. 4, 2 (1993), 29–30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Zhongxin Liu, Qiao Huang, Xin Xia, Emad Shihab, David Lo, and Shanping Li. 2018. SATD detector: A text-mining-based self-admitted technical debt detection tool. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). ACM, 9–12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Meng Yan, Xin Xia, Emad Shihab, David Lo, Jianwei Yin, and Xiaohu Yang. 2018. Automating change-level self-admitted technical debt determination. IEEE Trans. Softw. Eng. 45, 12 (2018), 1211–1229.Google ScholarGoogle ScholarCross RefCross Ref
  14. Aniket Potdar and Emad Shihab. 2014. An exploratory study on self-admitted technical debt. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’14). IEEE, 91–100. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Sultan Wehaibi, Emad Shihab, and Latifa Guerrouj. 2016. Examining the impact of self-admitted technical debt on software quality. In Proceedings of the IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER’16). IEEE, 179–188.Google ScholarGoogle ScholarCross RefCross Ref
  16. Gabriele Bavota and Barbara Russo. 2016. A large-scale empirical study on self-admitted technical debt. In Proceedings of the IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR’16). IEEE, 315–326. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Everton da Silva Maldonado, Emad Shihab, and Nikolaos Tsantalis. 2017. Using natural language processing to automatically detect self-admitted technical debt. IEEE Trans. Softw. Eng. 43, 11 (2017), 1044–1062.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Everton da S. Maldonado and Emad Shihab. 2015. Detecting and quantifying different types of self-admitted technical debt. In Proceedings of the IEEE 7th International Workshop on Managing Technical Debt (MTD’15). IEEE, 9–15.Google ScholarGoogle Scholar
  19. Nico Zazworka, Antonio Vetrò, Clemente Izurieta, Sunny Wong, Yuanfang Cai, Carolyn B. Seaman, and Forrest Shull. 2014. Comparing four approaches for technical debt identification. Softw. Qual. J. 22, 3 (2014), 403–426. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Zengyang Li, Paris Avgeriou, and Peng Liang. 2015. A systematic mapping study on technical debt and its management. J. Syst. Softw. 101 (2015), 193–220. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Jiawei Han and Micheline Kamber. 2006. Data Mining: Concepts and Techniques, Second Edition (The Morgan Kaufmann Series in Data Management Systems). Elsevier. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Margaret-Anne D. Storey, Jody Ryall, R. Ian Bull, Del Myers, and Janice Singer. 2008. TODO or to bug: Exploring how task annotations play a role in the work practices of software developers. In Proceedings of the 30th International Conference on Software Engineering (ICSE’08). 251–260. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Nico Zazworka, Rodrigo O. Spínola, Antonio Vetrò, Forrest Shull, and Carolyn B. Seaman. 2013. A case study on effectively identifying technical debt. In Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering (EASE’13). 42–47. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Nico Zazworka, Michele A. Shaw, Forrest Shull, and Carolyn B. Seaman. 2011. Investigating the impact of design debt on software quality. In Proceedings of the IEEE 2nd International Workshop on Managing Technical Debt (MTD’11). 17–23. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Clemente Izurieta, Antonio Vetrò, Nico Zazworka, Yuanfang Cai, Carolyn B. Seaman, and Forrest Shull. 2012. Organizing the technical debt landscape. In Proceedings of the IEEE 3rd International Workshop on Managing Technical Debt (MTD’12). 23–26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Nicolli S. R. Alves, Leilane Ferreira Ribeiro, Vivyane Caires, Thiago Souto Mendes, and Rodrigo O. Spínola. 2014. Towards an ontology of terms on technical debt. In Proceedings of the 6th International Workshop on Managing Technical Debt (MTD’14). 1–7. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Beat Fluri, Michael Würsch, and Harald C. Gall. 2007. Do code and comments coevolve? On the relation between source code and comment changes. In Proceedings of 14th Working Conference on Reverse Engineering (WCRE’07). 70–79. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Haroon Malik, Istehad Chowdhury, Hsiao-Ming Tsou, Zhen Ming Jiang, and Ahmed E. Hassan. 2008. Understanding the rationale for updating a function's comment. In Proceedings of the IEEE International Conference on Software Maintenance (ICSM’08). 167–176.Google ScholarGoogle Scholar
  29. Nanette Brown, Yuanfang Cai, Yuepu Guo, Rick Kazman, Miryung Kim, Philippe Kruchten, Erin Lim, Alan MacCormack, Robert L. Nord, Ipek Ozkaya, Raghvinder S. Sangwan, Carolyn B. Seaman, Kevin J. Sullivan, and Nico Zazworka. 2010. Managing technical debt in software-reliant systems. In Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research (FoSER’10). 47–52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Frank Buschmann. 2011. To pay or not to pay technical debt. IEEE Softw. 28, 6 (2011), 29–31. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Erin Lim, Nitin Taksande, and Carolyn B. Seaman. 2012. A balancing act: What software practitioners have to say about technical debt. IEEE Softw. 29, 6 (2012). 22–27. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Philippe Kruchten, Robert L. Nord, Ipek Ozkaya, and Davide Falessi. 2013. Technical debt: Towards a crisper definition report on the 4th International Workshop on Managing Technical Debt. ACM SIGSOFT Softw. Eng. Notes 38, 5 (2013), 51–54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Gerard Salton, A. Wong, and Chung-Shu Yang. 1975. A vector space model for automatic indexing. Commun. ACM 18, 11 (1975), 613–620. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens. 2012. @tComment: Testing Javadoc comments to detect comment-code inconsistencies. In Proceedings of the IEEE 5th International Conference on Software Testing, Verification and Validation (ICST’12). IEEE, 260–269. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Ninus Khamis, René Witte, and Juergen Rilling. 2010. Automatic quality assessment of source code comments: The JavadocMiner. In Proceedings of the International Conference on Application of Natural Language to Information Systems (NLDB’10). 68–79. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Matthew J. Howard, Samir Gupta, Lori L. Pollock, and K. Vijay-Shanker. 2013. Automatically mining software-based, semantically-similar words from comment-code mappings. In Proceedings of the 10th Working Conference on Mining Software Repositories (MSR’13). 377–386. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Daniela Steidl, Benjamin Hummel, and Elmar Jürgens. 2013. Quality analysis of source code comments. In Proceedings of the 21st International Conference on Program Comprehension (ICPC’13). IEEE, 83–92.Google ScholarGoogle ScholarCross RefCross Ref
  38. Bradley L. Vinz and Letha H. Etzkorn. 2008. Improving program comprehension by combining code understanding with comment understanding. Knowl.-based Syst. 21, 8 (2008), 813–825. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Hirohisa Aman, and Hirokazu Okazaki. 2008. Impact of comment statement on code stability in open source development. In Proceedings of the 8th Joint Conference on Knowledge-based Software Engineering (JCKBSE’08). 415–419. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Armstrong A. Takang, Penny A. Grubb, and Robert D. Macredie. 1996. The effects of comments and identifier names on program comprehensibility: An experimental investigation. J. Prog. Lang. 4, 3 (1996), 143–167.Google ScholarGoogle Scholar
  41. Xiaobing Sun, Qiang Geng, David Lo, Yucong Duan, Xiangyue Liu, and Bin Li. 2016. Code comment quality analysis and improvement recommendation: An automated approach. Int. J. Softw. Eng. Knowl. Eng. 26, 6 (2016), 981–1000.Google ScholarGoogle ScholarCross RefCross Ref
  42. Paul W. Mcburney and Collin Mcmillan. 2016. An empirical study of the textual similarity between source code and source code summaries. Empir. Softw. Eng. 21, 1 (2016), 17–42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Fabrizio Sebastiani. 2002. Machine learning in automated text categorization. ACM Comput. Surv. 34, 1 (2002), 1–47. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Haibo He and Edwardo A. Garcia. 2009. Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 21, 9 (2009), 1263–1284. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Georgios Digkas, Mircea Lungu, Alexander Chatzigeorgiou, and Paris Avgeriou. 2017. The evolution of technical debt in the apache ecosystem. In Proceedings of the European Conference on Software Architecture (ECSA’17). 51–66.Google ScholarGoogle ScholarCross RefCross Ref
  46. Chao Liu, Cuiyun Gao, Xin Xia, David Lo, John C. Grundy, and Xiaohu Yang. 2020. On the replicability and reproducibility of deep learning in software engineering. CoRR abs/2006.14244 (2020).Google ScholarGoogle Scholar
  47. Yoav Benjamini and Daniel Yekutieli. 2001. The control of false discovery rate in multiple testing under dependency. Ann. Statist. 29, 4 (2001), 1165–1188.Google ScholarGoogle ScholarCross RefCross Ref
  48. Nikolaos Tsantalis, Theodoros Chaikalis, and Alexander Chatzigeorgiou. 2008. Jdeodorant: Identification and removal of type-checking bad smells. In Proceedings of the 12th European Conference on Software Maintenance and Reengineering (CSMR’08). 329–331. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Xiaoxue Ren, Zhenchang Xing, Xin Xia, David Lo, Xinyu Wang, and John Grundy. 2019. Neural network based detection of self-admitted technical debt: From performance to explainability. ACM Trans. Softw. Eng. Methodol. 28, 3 (2019). Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS’13). 3111–3119. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for Twitter sentiment classification. In Proceedings of the 52nd Meeting of the Association for Computational Linguistics (ACL’14). 1555–1565.Google ScholarGoogle ScholarCross RefCross Ref
  52. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS’12). 1106–1114. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Yibiao Yang, Yuming Zhou, Jinping Liu, Yangyang Zhao, Hongmin Lu, Lei Xu, Baowen Xu, and Hareton Leung. 2016. Effort-aware just-in-time defect prediction: Simple unsupervised models could be better than supervised models. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE’16). 157–168. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Wei Fu and Tim Menzies. 2017. Easy over hard: A case study on deep learning. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (FSE’17). 49–60. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Qiao Huang, Xin Xia, and David Lo. 2019. Revisiting supervised and unsupervised models for effort-aware just-in-time defect prediction. Empir. Softw. Eng. 24, 5 (2019), 2823–2862.Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Qiao Huang, Xin Xia, and David Lo. 2017. Supervised vs. unsupervised models: A holistic look at effort-aware just-in-time defect prediction. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’17). IEEE, 159–170.Google ScholarGoogle ScholarCross RefCross Ref
  57. Bowen Xu, Amirreza Shirani, David Lo, and Mohammad Amin Alipour. 2018. Prediction of relatedness in stack overflow: Deep learning vs. SVM: A reproducibility study. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM’18). 1–10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Margaret-Anne D. Storey, Jody Ryall, Janice Singer, Del Myers, Li-Te Cheng, and Michael J. Muller. 2009. How software developers use tagging to support reminding and refinding. IEEE Trans. Softw. Eng. 35, 4 (2009), 470–483. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Annie T. T. Ying, James L. Wright, and Steven Abrams. 2005. Source code that talks: an exploration of eclipse task comments and their implication to repository mining. In Proceedings of the International Workshop on Mining Software Repositories (MSR’05). 1–5. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Christopher D. Manning, and Dan Klein. 2003. Optimization, maxent models, and conditional estimation without magic. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL’03). 8–8. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Rahul, Krishna and Tim Menzies. 2017. Bellwethers: A baseline method for transfer learning. IEEE Trans. Softw. Eng. 45, 11 (2017), 1081–1105.Google ScholarGoogle Scholar
  62. Jianfeng Chen, Vivek Nair, Rahul Krishna, and Tim Menzies. 2019. “Sampling” as a baseline optimizer for search-based software engineering. IEEE Trans. Softw. Eng. 45, 6 (2019), 597–614.Google ScholarGoogle ScholarCross RefCross Ref
  63. Peter A. Whigham, Caitlin A. Owen, and Stephen G. MacDonell. 2015. A baseline model for software effort estimation. ACM Trans. Softw. Eng. Methodol. 24, 3 (2015). Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Yuming Zhou, Yibiao Yang, Hongmin Lu, Lin Chen, Yanhui Li, Yangyang Zhao, Junyan Qian, and Baowen Xu. 2018. How far we have progressed in the journey? An examination of cross-project defect prediction. ACM Trans. Softw. Eng. Methodol. 27, 1 (2018). Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Roberto Souto Maior de Barros and Silas Garrido Teixeira de Carvalho Santos. 2018. A large-scale comparison of concept drift detectors. Inf. Sci. 451-452, (2018), 348–370.Google ScholarGoogle Scholar
  66. Tsung-Han Chan, Kui Jia, Shenghua Gao, Jiwen Lu, Zinan Zeng, and Yi Ma. 2015. PCANet: A simple deep learning baseline for image classification. IEEE Trans. Image Proc. 24, 12 (2015), 5017–5032.Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. Zhiguang Wang, Weizhong Yan, and Tim Oates. 2017. Time series classification from scratch with deep neural networks: A strong baseline. In Proceedings of the International Joint Conference on Neural Networks (IJCNN’17). 1578–1585.Google ScholarGoogle ScholarCross RefCross Ref
  68. Hongjian Wang, Xianfeng Tang, Yu-Hsuan Kuo, Daniel Kifer, and Zhenhui Li. 2019. A simple baseline for travel time estimation using large-scale trip data. ACM Trans. Intell. Syst. Technol. 10, 2 (2019). Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. Bin Xiao, Haiping Wu, and Yichen Wei. 2018. Simple baselines for human pose estimation and tracking. In Proceedings of the European Conference on Computer Vision (ECCV’18). 472–487.Google ScholarGoogle ScholarCross RefCross Ref
  70. Kawin Ethayarajh. 2018. Unsupervised random walk sentence embeddings: A strong but simple baseline. In Proceedings of the 3rd Workshop on Representation Learning for NLP (Rep4NLP@ACL’18). 91–100.Google ScholarGoogle ScholarCross RefCross Ref
  71. Zheng Xu, Xitong Yang, Xue Li, and Xiaoshuai Sun. 2018. The effectiveness of instance normalization: A strong baseline for single image dehazing. CoRR abs/1805.03305 (2018).Google ScholarGoogle Scholar
  72. Yaming Tang, Fei Zhao, Yibiao Yang, Hongmin Lu, Yuming Zhou, and Baowen Xu. 2015. Predicting vulnerable components via text mining or software metrics? An effort-aware perspective. In Proceedings of the International Conference on Software Quality, Reliability and Security (QRS’15). 27–36. Google ScholarGoogle ScholarDigital LibraryDigital Library
  73. Roberto Souto Maior de Barros, Juan Isidro González Hidalgo, and Danilo Rafael de Lima Cabral. 2018. Wilcoxon rank sum test drift detector. Neurocomputing 275 (2018), 1954–1963. Google ScholarGoogle ScholarDigital LibraryDigital Library
  74. Annibale Panichella, Rocco Oliveto, and Andrea De Lucia. 2014. Cross-project defect prediction models: L'Union fait la force. In Proceedings of the IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE’14). 164–173.Google ScholarGoogle ScholarCross RefCross Ref
  75. Zhongxin Liu, Xin Xia, Ahmed E. Hassan, David Lo, Zhenchang Xing, and Xinyu Wang. 2018. Neural-machine-translation-based commit message generation: How far are we? In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). 373–384. Google ScholarGoogle ScholarDigital LibraryDigital Library
  76. Yuming Zhou, Baowen Xu, Hareton Leung, and Lin Chen. 2014. An in-depth study of the potentially confounding effect of class size in fault prediction. ACM Trans. Softw. Eng. Methodol. 23, 1 (2014). Google ScholarGoogle ScholarDigital LibraryDigital Library
  77. Suvodeep Majumder, Nikhila Balaji, Katie Brey, Wei Fu, and Tim Menzies. 2018. 500+ times faster than deep learning: A case study exploring faster methods for text mining stackoverflow. In Proceedings of the 15th International Conference on Mining Software Repositories (MSR’18). 554-563. Google ScholarGoogle ScholarDigital LibraryDigital Library
  78. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. XX, 1 (1960), 37–46.Google ScholarGoogle ScholarCross RefCross Ref
  79. Joseph L. Fleiss. 1981. The measurement of interrater agreement. In Statistical Methods Rates Proportions. John Wiley, NewYork.Google ScholarGoogle Scholar
  80. Retrieved from https://github.com/Naplues/MAT.Google ScholarGoogle Scholar
  81. Giancarlo Sierra, Emad Shihab, and Yasutaka Kamei. 2019. A survey of self-admitted technical debt. J. Syst. Softw. 152, (2019), 70–82.Google ScholarGoogle Scholar
  82. Supatsara Wattanakriengkrai, Rungroj Maipradit, Hideaki Hata, Morakot Choetkiertikul, Thanwadee Sunetnanta, and Kenichi Matsumoto. 2018. Identifying design and requirement self-admitted technical debt using n-gram IDF. In Proceedings of the International Workshop on Empirical Software Engineering in Practice (IWESEP’18). 7–12.Google ScholarGoogle ScholarCross RefCross Ref
  83. Maleknaz Nayebi, Yuanfang Cai, Rick Kazman, Guenther Ruhe, Qiong Feng, Chris Carlson, and Francis Chew. 2018. A longitudinal study of identifying and paying down architectural debt. CoRR abs/1811.12904 (2018).Google ScholarGoogle Scholar
  84. Nathalie Japkowicz and Mohak Shah. 2011. Evaluating Learning Algorithms: A Classification Perspective. Cambridge University Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  85. N. E. Breslow and N. E. Day. 1980. Statistical methods in cancer research: The analysis of case-control studies. IARC Sci. Public. 1, 32 (1980), 5–338.Google ScholarGoogle Scholar
  86. Matthew Hutson. 2018. Artificial intelligence faces reproducibility crisis. Comput. Sci. 359, 6377 (2018), 725–726.Google ScholarGoogle Scholar
  87. Andrew Lane Beam, Arjun K. Manrai, and Marzyeh Ghassemi. 2020. Challenges to the reproducibility of machine learning models in health care. J. Amer. Med. Assoc. 323, 4 (2020).Google ScholarGoogle ScholarCross RefCross Ref
  88. Feng Zhang, Iman Keivanloo, and Ying Zou. 2017. Data transformation in cross-project defect prediction. Empir. Softw. Eng. 22, 6 (2017), 3186–3218. Google ScholarGoogle ScholarDigital LibraryDigital Library
  89. Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016. Deep API learning. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE’16). 631–642. Google ScholarGoogle ScholarDigital LibraryDigital Library
  90. Chengnian Sun, David Lo, Xiaoyin Wang, Jing Jiang, and Siau-Cheng Khoo. 2010. A discriminative model approach for accurate duplicate bug report retrieval. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering (ICSE’10). 45–54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  91. Alex Graves, M. Abdel-rahman, and Geoffrey E. Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’13). 6645–6649.Google ScholarGoogle Scholar
  92. Bowen Xu, Deheng Ye, Zhenchang Xing, Xin Xia, Guibin Chen, and Shanping Li. 2016. Predicting semantically linkable knowledge in developer online forums via convolutional neural network. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). 51–62. Google ScholarGoogle ScholarDigital LibraryDigital Library
  93. Lin Ma, Zhengdong Lu, and Hang Li. 2016. Learning to answer questions from image using convolutional neural network. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI’16). 3567–3573. Google ScholarGoogle ScholarDigital LibraryDigital Library
  94. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’14). Association for Computational Linguistics, 1746–1751.Google ScholarGoogle ScholarCross RefCross Ref
  95. Zhe Yu, Fahmid Morshed Fahid, Huy Tu, and Tim Menzies. 2020. Identifying self-admitted technical debts with Jitterbug: A two-step approach. IEEE Trans. Softw. Eng. (2020).Google ScholarGoogle Scholar
  96. Fei Zhao, Yaming Tang, Yibiao Yang, Hongmin Lu, Yuming Zhou and Baowen Xu. 2015. Is learning-to-rank cost-effective in recommending relevant files for bug localization? In Proceedings of the International Conference on Software Quality, Reliability and Security (QRS’15). 298–303. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. How Far Have We Progressed in Identifying Self-admitted Technical Debts? A Comprehensive Empirical Study

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Software Engineering and Methodology
      ACM Transactions on Software Engineering and Methodology  Volume 30, Issue 4
      Continuous Special Section: AI and SE
      October 2021
      613 pages
      ISSN:1049-331X
      EISSN:1557-7392
      DOI:10.1145/3461694
      • Editor:
      • Mauro Pezzè
      Issue’s Table of Contents

      Copyright © 2021 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 23 July 2021
      • Accepted: 1 January 2021
      • Revised: 1 December 2020
      • Received: 1 June 2020
      Published in tosem Volume 30, Issue 4

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Research
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format