nach oben

Empirical Software Engineering

Erschienen in:

01.09.2021

Assessing exception handling testing practices in open-source libraries

verfasst von: Luan P. Lima, Lincoln S. Rocha, Carla I. M. Bezerra, Matheus Paixao

Erschienen in: Empirical Software Engineering | Ausgabe 5/2021

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Modern programming languages (e.g., Java and C#) provide features to separate error-handling code from regular code, seeking to enhance software comprehensibility and maintainability. Nevertheless, the way exception handling (EH) code is structured in such languages may lead to multiple, different, and complex control flows, which may affect the software testability. Previous studies have reported that EH code is typically neglected, not well tested, and its misuse can lead to reliability degradation and catastrophic failures. However, little is known about the relationship between testing practices and EH testing effectiveness. In this exploratory study, we (i) measured the adequacy degree of EH testing concerning code coverage (instruction, branch, and method) criteria; and (ii) evaluated the effectiveness of the EH testing by measuring its capability to detect artificially injected faults (i.e., mutants) using 7 EH mutation operators. Our study was performed using test suites of 27 long-lived Java libraries from open-source ecosystems. Our results show that instructions and branches within catch blocks and throw instructions are less covered, with statistical significance, than the overall instructions and branches. Nevertheless, most of the studied libraries presented test suites capable of detecting more than 70% of the injected faults. From a total of 12,331 mutants created in this study, the test suites were able to detect 68% of them.

Vorheriger Artikel To what extent do DNN-based image classification models make unreliable inferences?

Nächster Artikel Weighted software metrics aggregation and its application to defect prediction

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

http://wiki.c2.com/?ExceptionPatterns

https://apache.org/index.html#projects-list

https://commons.apache.org/

http://cobertura.github.io/cobertura/

https://www.jacoco.org/jacoco/

http://openclover.org/

https://www.jacoco.org/jacoco/trunk/coverage/report.dtd

https://www.jacoco.org/jacoco/trunk/coverage/jacoco.xml

https://www.eclipse.org/jgit/

https://maven.apache.org/shared/maven-invoker/

https://docs.gradle.org/current/userguide/embedding.html

https://spoon.gforge.inria.fr/

https://commons.apache.org/proper/commons-io/project-reports.html

https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.friedmanchisquare.html

Ahmed I, Gopinath R, Brindescu C, Groce A, Jensen C (2016) Can testedness be effectively measured?. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, ACM, New York, NY, USA, FSE 2016, pp 547–558. https://doi.org/10.1145/2950290.2950324

Ammann P, Offutt J (2016) Introduction to software testing, 2nd edn. Cambridge University Press, CambridgeCrossRef

Antinyan V, Derehag J, Sandberg A, Staron M (2018) Mythical unit test coverage. IEEE Softw 35(3):73–79. https://doi.org/10.1109/MS.2017.3281318CrossRef

Barbosa EA, Garcia A, Barbos SDJ (2014) Categorizing faults in exception handling: A study of open source projects. In: 2014 Brazilian Symposium on Software Engineering (SBES), pp 11–20

Bavota G, De Lucia A, Di Penta M, Oliveto R, Palomba F (2015) An experimental investigation on the innate relationship between quality and refactoring. J Syst Softw 107:1–14CrossRef

Benjamini Y, Yekutieli D (2001) The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics 29(4):1165–1188. http://www.jstor.org/stable/2674075MathSciNetCrossRef

Bloch J (2008) Effective java (The Java Series), 2nd edn. Prentice Hall PTR, Upper Saddle River

Cacho N, Barbosa EA, Araujo J, Pranto F, Garcia A, Cesar T, Soares E, Cassio A, Filipe T, Garcia I (2014a) How does exception handling behavior evolve? an exploratory study in java and c# applications. In: 2014 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 31–40

Cacho N, César T, Filipe T, Soares E, Cassio A, Souza R, Garcia I, Barbosa EA, Garcia A (2014b) Trading robustness for maintainability: An empirical study of evolving c# programs. In: Proceedings of the 36th international conference on software engineering, ACM, New York, NY, USA, ICSE 2014, pp 584–595. https://doi.org/10.1145/2568225.2568308

Chang BM, Choi K (2016) A review on exception analysis. Inf Softw Technol 77(C):1–16CrossRef

Chekam TT, Papadakis M, Le Traon Y, Harman M (2017) An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pp 597–608. https://doi.org/10.1109/ICSE.2017.61

Chen H, Dou W, Jiang Y, Qin F (2019) Understanding exception-related bugs in large-scale cloud systems. In: Proceedings of the 34rd ACM/IEEE International Conference on Automated Software Engineering, ACM, New York, NY, USA, ASE

Cliff N (1993) Dominance statistics: Ordinal analyses to answer ordinal questions. Psychol Bull 114(4):494–509CrossRef

Coelho R, Almeida L, Gousios G, Deursen A V, Treude C (2017) Exception handling bug hazards in android. Empirical Softw Engg 22(3):1264–1304CrossRef

Dalton F, Ribeiro M, Pinto G, Fernandes L, Gheyi R, Fonseca B (2020) Is exceptional behavior testing an exception? an empirical assessment using java automated tests. In: Proceedings of the Evaluation and Assessment in Software Engineering, Association for Computing Machinery, New York, NY, USA, EASE ’20, pp 170–179. https://doi.org/10.1145/3383219.3383237

Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30. http://dl.acm.org/citation.cfm?id=1248547.1248548MathSciNetMATH

Digkas G, Lungu M, Avgeriou P, Chatzigeorgiou A, Ampatzoglou A (2018) How do developers fix issues and pay back technical debt in the apache ecosystem?. In: 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp 153–163. https://doi.org/10.1109/SANER.2018.8330205

Ebert F, Castor F, Serebrenik A (2015) An exploratory study on exception handling bugs in java programs. J Syst Softw 106(C):82–101CrossRef

Eck M, Palomba F, Castelluccio M, Bacchelli A (2019) Understanding flaky tests: The developer’s perspective. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, New York, NY, USA, ESEC/FSE, pp 830–840. https://doi.org/10.1145/3338906.3338945

Filho JLM, Rocha L, Andrade R, Britto R (2017) Preventing erosion in exception handling design using static-architecture conformance checking. In: Proceedings of the 11th European Conference on Software Architecture, Springer International Publishing, Cham, ECSA ’17, pp 67–83. https://doi.org/10.1007/978-3-319-65831-5_5

Friedman M (1940) A comparison of alternative tests of significance for the problem of m rankings. The Annals of Mathematical Statistics 11(1):86–92MathSciNetCrossRef

Gallardo R, Hommel S, Kannan S, Gordon J, Zakhour S B (2014) The Java tutorial: a short course on the basics. Java Series, 6th edn. Addison-Wesley Professional, Boston

Garcia A F, Rubira C M, Romanovsky A, Xu J (2001) A comparative study of exception handling mechanisms for building dependable object-oriented software. J Syst Softw 59(2):197–222CrossRef

Gligoric M, Groce A, Zhang C, Sharma R, Alipour MA, Marinov D (2013) Comparing non-adequate test suites using coverage criteria. In: Proceedings of the 2013 International symposium on software testing and analysis, association for computing machinery, New York, NY, USA, ISSTA 2013, pp 302–313. https://doi.org/10.1145/2483760.2483769

Goffi A, Gorla A, Ernst MD, Pezzè M (2016) Automatic generation of oracles for exceptional behaviors. In: Proceedings of the 25th International symposium on software testing and analysis, association for computing machinery, New York, NY, USA, ISSTA 2016, pp 213–224. https://doi.org/10.1145/2931037.2931061

Goodenough J B, Gerhart S L (1975) Toward a theory of test data selection. IEEE Trans Softw Eng (2)156–173

Gopinath R, Jensen C, Groce A (2014) Code coverage for suite evaluation by developers. In: Proceedings of the 36th International conference on software engineering, association for computing machinery, New York, NY, USA, ICSE 2014, pp 72–82. https://doi.org/10.1145/2568225.2568278

Gulati S, Sharma R (2017) Java unit testing with JUnit 5: Test Driven Development with JUnit 5, 1st edn. Apress, USACrossRef

Hilton M, Bell J, Marinov D (2018) A large-scale study of test coverage evolution. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, New York, NY, USA, ASE 2018, pp 53–63. https://doi.org/10.1145/3238147.3238183

Hunt A, Thomas D (2003) Pragmatic Unit Testing in Java with JUnit. The pragmatic programmers

Inozemtseva L, Holmes R (2014a) Coverage is not strongly correlated with test suite effectiveness. In: Proceedings of the 36th International conference on software engineering, association for computing machinery, New York, NY, USA, ICSE 2014, pp 435–445. https://doi.org/10.1145/2568225.2568271

Inozemtseva L, Holmes R (2014b) Coverage is not strongly correlated with test suite effectiveness. In: Proceedings of the 36th International conference on software engineering, ACM, New York, NY, USA, ICSE 2014, pp 435–445. https://doi.org/10.1145/2568225.2568271

Ivanković M, Petrović G, Just R, Fraser G (2019) Code coverage at google. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, ACM, New York, NY, USA, ESEC/FSE 2019, pp 955–963. https://doi.org/10.1145/3338906.3340459

Jenkov J (2013) Java Exception Handling. Jenkov Aps

Ji C, Chen Z, Xu B, Wang Z (2009) A new mutation analysis method for testing java exception handling. In: Computer software and applications conference, 2009. COMPSAC’09. 33rd Annual IEEE International, vol 2. IEEE, pp 556–561

Just R, Jalali D, Inozemtseva L, Ernst M D, Holmes R, Fraser G (2014) Are mutants a valid substitute for real faults in software testing?. In: Proceedings of the 22Nd ACM SIGSOFT international symposium on foundations of software engineering, ACM, New York, NY, USA, FSE 2014, pp 654–665. https://doi.org/10.1145/2635868.2635929

Kechagia M, Spinellis D (2014) Undocumented and unchecked: Exceptions that spell trouble. In: Proceedings of the 11th Working Conference on Mining Software Repositories, ACM, New York, NY, USA, MSR 2014, pp 312–315

Koch S (2007) Software evolution in open source projects—a large-scale investigation. J Softw Maint Evol 19(6):361–382CrossRef

Kochhar PS, Thung F, Lo D (2015) Code coverage and test suite effectiveness: Empirical study with real bugs in large systems. In: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp 560–564. https://doi.org/10.1109/SANER.2015.7081877

Kochhar P S, Lo D, Lawall J, Nagappan N (2017) Code coverage and postrelease defects: A large-scale study on open source projects. IEEE Trans Reliab 66(4):1213–1228. https://doi.org/10.1109/TR.2017.2727062CrossRef

Kumar K, Gupta P, Parjapat R (2011) New mutants generation for testing java programs. In: International conference on advances in communication, network, and computing. Springer, pp 290–294

Lima L, Rocha L, Bezerra C, Paixao M (2021) Replication package for the paper: “assessing exception handling testing practices in open-source software systems. https://zenodo.org/record/4732368, Accessed: May 01, 2021

Luo Q, Hariri F, Eloussi L, Marinov D (2014) An empirical analysis of flaky tests. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pp 643–653

Marinescu C (2011) Are the classes that use exceptions defect prone?. In: Proceedings of the 12th International workshop on principles of software evolution and the 7th annual ERCIM workshop on software evolution. ACM, pp 56–60

Marinescu C (2013) Should we beware the exceptions? an empirical study on the eclipse project. In: 2013 15th International symposium on symbolic and numeric algorithms for scientific computing (SYNASC). IEEE, pp 250–257

Martins AL, Hanazumi S, d Melo ACV (2014) Testing java exceptions: An instrumentation technique. In: 2014 IEEE 38th international computer software and applications conference workshops, pp 626–631. https://doi.org/10.1109/COMPSACW.2014.105

McMinn P (2004) Search-based software test data generation: a survey. Software Testing, Verification and reliability 14(2):105–156CrossRef

Nagappan M, Zimmermann T, Bird C (2013) Diversity in software engineering research. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering - ESEC/FSE 2013. ACM Press, New York, p 466

Oliveira J, Borges D, Silva T, Cacho N, Castor F (2018) Do android developers neglect error handling? a maintenance-centric study on the relationship between android abstractions and uncaught exceptions. J Syst Softw 136(C):1–18. https://doi.org/10.1016/j.jss.2017.10.032CrossRef

Osman H, Chiş A, Corrodi C, Ghafari M, Nierstrasz O (2017) Exception evolution in long-lived java systems. In: Proceedings of the 14th International Conference on Mining Software Repositories, IEEE Press, Piscataway, NJ, USA, MSR ’17, pp 302–311

de Pádua GB, Shang W (2017a) Revisiting exception handling practices with exception flow analysis. In: 2017 IEEE 17th International working conference on source code analysis and manipulation (SCAM), pp 11–20. https://doi.org/10.1109/SCAM.2017.16

de Pádua GB, Shang W (2017b) Studying the prevalence of exception handling anti-patterns. In: Proceedings of the 25th International conference on program comprehension. IEEE Press, Piscataway, NJ, USA, ICPC’17, pp 328–331

de Pádua GB, Shang W (2018) Studying the relationship between exception handling practices and post-release defects. In: Proceedings of the 15th International conference on mining software repositories. ACM, New York, NY, USA, MSR ’18, pp 564–575. https://doi.org/10.1145/3196398.3196435

Paixao M, Krinke J, Han D, Ragkhitwetsagul C, Harman M (2017) Are developers aware of the architectural impact of their changes?. In: ASE 2017 - Proceedings of the 32nd IEEE/ACM International conference on automated software engineering

Papadakis M, Shin D, Yoo S, Bae DH (2018) Are mutation scores correlated with real fault detection?: A large scale empirical study on the relationship between mutants and real faults. In: Proceedings of the 40th International conference on software engineering. ACM, New York, NY, USA, ICSE ’18, pp 537–548. https://doi.org/10.1145/3180155.3180183

Papadakis M, Kintis M, Zhang J, Jia Y, Traon Y L, Harman M (2019) Chapter six - mutation testing advances: An analysis and survey. Advances in Computers. Elsevier 112:275–378

Pawlak R, Monperrus M, Petitprez N, Noguera C, Seinturier L (2016) Spoon: A library for implementing analyses and transformations of java source code. Softw Pract Exper 46(9):1155–1179 . https://doi.org/10.1002/spe.2346CrossRef

Rashkovits R, Lavy I (2012) Students’ misconceptions of java exceptions. In: Rahman H, Mesquita A, Ramos I, Pernici B (eds) Proceedings of the 7th Mediterranean conference on information systems. Springer, Berlin, MCIS ’12, pp 1–21. https://doi.org/10.1007/978-3-642-33244-9_1

Reales P, Polo M, Fernández-Alemán JL, Toval A, Piattini M (2014) Mutation testing. IEEE Software 31(3):30–35. https://doi.org/10.1109/MS.2014.68CrossRef

Romano D, Di Penta M, Antoniol G (2011) An approach for search based testing of null pointer exceptions. In: 2011 Fourth IEEE international conference on software testing, verification and validation. IEEE, pp 160–169

Saha RK, Lyu Y, Lam W, Yoshida H, Prasad MR (2018) Bugs.jar: A large-scale, diverse dataset of real-world java bugs. In: Proceedings of the 15th international conference on mining software repositories, association for computing machinery, New York, NY, USA, MSR ’18, pp 10–13. https://doi.org/10.1145/3196398.3196473

Sawadpong P, Allen EB (2016) Software defect prediction using exception handling call graphs: A case study. In: 2016 IEEE 17th International symposium on high assurance systems engineering (HASE). IEEE, pp 55–62

Schwartz A, Puckett D, Meng Y, Gay G (2018) Investigating faults missed by test suites achieving high code coverage. J Syst Softw 144:106–120. https://doi.org/10.1016/j.jss.2018.06.024CrossRef

Shah H, Harrold MJ (2009) Exception handling negligence due to intra-individual goal conflicts. In: Proceedings of the 2009 ICSE workshop on cooperative and human aspects on software engineering. IEEE Computer Society, Washington, DC, USA, CHASE ’09, pp 80–83. https://doi.org/10.1109/CHASE.2009.5071417

Shah H, Görg C, Harrold MJ (2008) Why do developers neglect exception handling?. In: Proceedings of the 4th International Workshop on Exception Handling, ACM, New York, NY, USA, WEH ’08, pp 62–68. https://doi.org/10.1145/1454268.1454277

Shah H, Gorg C, Harrold M J (2010) Understanding exception handling: Viewpoints of novices and experts. IEEE Trans Softw Eng 36(2):150–161CrossRef

Shahrokni A, Feldt R (2013) A systematic review of software robustness. Inf Softw Technol 55(1):1–17CrossRef

Shi L, Zhong H, Xie T, Li M (2011) An empirical study on evolution of api documentation. In: Proceedings of the 14th International conference on fundamental approaches to software engineering: part of the joint european conferences on theory and practice of software. Springer, Berlin, Heidelberg, FASE’11/ETAPS’11, pp 416–431

Sinha S, Harrold MJ (2000) Analysis and testing of programs with exception handling constructs. IEEE Trans Softw Eng 26(9):849–871. https://doi.org/10.1109/32.877846CrossRef

Turner AJ, White DR, Drake JH (2016) Multi-objective regression test suite minimisation for mockito. In: Sarro F, Deb K (eds) Search based software engineering. Springer International Publishing, Cham, pp 244–249

Vieira R, da Silva A, Rocha L, Gomes JaP (2019) From reports to bug-fix commits: A 10 years dataset of bug-fixing activity from 55 apache’s open source projects. In: Proceedings of the fifteenth international conference on predictive models and data analytics in software engineering, Association for Computing Machinery, New York, NY, USA, PROMISE’19, pp 80–89. https://doi.org/10.1145/3345629.3345639

Wirfs-Brock R J (2006) Toward exception-handling best practices and patterns. IEEE Softw 23(5):11–13. https://doi.org/10.1109/MS.2006.144CrossRef

Wohlin C, Runeson P, Hst M, Ohlsson M C, Regnell B, Wessln A (2012) Experimentation in software engineering. Springer Publishing Company Incorporated

Yang Y, Zhou Y, Sun H, Su Z, Zuo Z, Xu L, Xu B (2019) Hunting for bugs in code coverage tools via randomized differential testing. In: Proceedings of the 41st international conference on software engineering. IEEE Press, Piscataway, NJ, USA, ICSE ’19, pp 488–499. https://doi.org/10.1109/ICSE.2019.00061

Zhai H, Casalnuovo C, Devanbu P (2019) Test coverage in python programs. In: Proceedings of the 16th international conference on mining software repositories. IEEE Press, Piscataway, NJ, USA, MSR ’19, pp 116–120. https://doi.org/10.1109/MSR.2019.00027

Zhang P, Elbaum S (2014) Amplifying tests to validate exception handling code: An extended study in the mobile application domain. ACM Trans Softw Eng Methodol 23(4):32:1–32:28. https://doi.org/10.1145/2652483CrossRef

Zhong H, Mei H (2019) An empirical study on api usages. IEEE Trans Softw Eng 45(4):319–334. https://doi.org/10.1109/TSE.2017.2782280CrossRef

Zhu H, Hall PAV, May JHR (1997) Software unit test coverage and adequacy. ACM Comput Surv 29(4):366–427. https://doi.org/10.1145/267580.267590CrossRef

Titel: Assessing exception handling testing practices in open-source libraries
verfasst von: Luan P. Lima
Lincoln S. Rocha
Carla I. M. Bezerra
Matheus Paixao
Publikationsdatum: 01.09.2021
Verlag: Springer US
Erschienen in: Empirical Software Engineering / Ausgabe 5/2021
Print ISSN: 1382-3256
Elektronische ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-021-09983-3

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Weitere Artikel der Ausgabe 5/2021

Does class size matter? An in-depth assessment of the effect of class size in software defect prediction

Perceived diversity in software engineering: a systematic literature review

FeatCompare: Feature comparison for competing mobile apps leveraging user reviews

Understanding shared links and their intentions to meet information needs in modern code review:

Industry practices and challenges for the evolvability assurance of microservices

Understanding peer review of software engineering papers

Premium Partner