Arianna Blasi
Michael D. Ernst
Mauro Pezzè
ABSTRACT
Procedure specifications are useful in many software development tasks. As one example, in automatic test case generation they can guide testing, act as test oracles able to reveal bugs, and identify illegal inputs. Whereas formal specifications are seldom available in practice, it is standard practice for developers to document their code with semi-structured comments. These comments express the procedure specification with a mix of predefined tags and natural language. This paper presents Jdoctor, an approach that combines pattern, lexical, and semantic matching to translate Javadoc comments into executable procedure specifications written as Java expressions. In an empirical evaluation, Jdoctor achieved precision of 92% and recall of 83% in translating Javadoc into procedure specifications. We also supplied the Jdoctor-derived specifications to an automated test case generation tool, Randoop. The specifications enabled Randoop to generate test cases of higher quality.
- Gabor Angeli, Melvin Johnson Premkumar, and Christopher D Manning. 2015. Leveraging Linguistic Structure for Open Domain Information Extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL ’15). Association for Computational Linguistics.Google Scholar
Cross Ref
- Sergio Antoy and Dick Hamlet. 2000. Automatically Checking an Implementation against Its Formal Specification. IEEE Transactions on Software Engineering 26, 1 (2000), 55–69. Google ScholarDigital Library
- Wladimir Araujo, Lionel C. Briand, and Yvan Labiche. 2011. Enabling the Runtime Assertion Checking of Concurrent Contracts for the Java Modeling Language. In Proceedings of the International Conference on Software Engineering (ICSE ’11). 786–795. Google ScholarDigital Library
- Marc J. Balcer, William M. Hasling, and Thomas J. Ostrand. 1989. Automatic Generation of Test Scripts from Formal Test Specifications. In Proceedings of the Symposium on Software Testing, Analysis, and Verification (TAV3 ’89). ACM, 210–218. Google ScholarDigital Library
- Boris Beizer. 1990. Software Testing Techniques (2 ed.). Van Nostrand Reinhold Co., New York, NY, USA. Google ScholarDigital Library
- Arianna Blasi and Alessandra Gorla. 2018. RepliComment: Identifying Clones in Code Comments. In Proceedings of the International Conference on Program Comprehension (ICPC’18). ACM. Google ScholarDigital Library
- Antonio Carzaniga, Alberto Goffi, Alessandra Gorla, Andrea Mattavelli, and Mauro Pezzè. 2014. Cross-checking Oracles from Intrinsic Software Redundancy. In Proceedings of the International Conference on Software Engineering (ICSE ’14). ACM, 931–942. Google ScholarDigital Library
- Tsong Y. Chen, F.-C. Kuo, T. H. Tse, and Zhi Quan Zhou. 2003. Metamorphic Testing and Beyond. In International Workshop on Software Technology and Engineering Practice (STEP ’03). IEEE Computer Society, 94–100. Google ScholarDigital Library
- Yoonsik Cheon. 2007. Abstraction in Assertion-Based Test Oracles. In Proceedings of the International Conference on Quality Software (QSIC ’07). 410–414. Google ScholarDigital Library
- Yoonsik Cheon and Gary T. Leavens. 2002. A Simple and Practical Approach to Unit Testing: The JML and JUnit Way. In Proceedings of the European Conference on Object-Oriented Programming (ECOOP ’02). 231–255. Google ScholarDigital Library
- Christoph Csallner and Yannis Smaragdakis. 2004. JCrasher: an automatic robustness tester for Java. Software: Practice and Experience 34, 11 (September 2004), 1025–1050. Google ScholarDigital Library
- Christoph Csallner and Yannis Smaragdakis. 2005. Check ’n’ Crash: Combining static checking and testing. In ICSE 2005, Proceedings of the 27th International Conference on Software Engineering. St. Louis, MO, USA, 422–431. Google ScholarDigital Library
- J. D. Day and J. D. Gannon. 1985. A Test Oracle Based on Formal Specifications. In Proceedings of the Conference on Software Development Tools, Techniques, and Alternatives (SOFTAIR ’85). 126–130. Google ScholarDigital Library
- Luciano Del Corro and Rainer Gemulla. 2013. ClausIE: Clause-based Open Information Extraction. In Proceedings of the International Conference on World Wide Web (WWW ’13). ACM, 355–366. Google ScholarDigital Library
- Nelly Delgado, Ann Quiroz Gates, and Steve Roach. 2004. A taxonomy and catalog of runtime software-fault monitoring tools. IEEE Transactions on Software Engineering 30, 12 (Dec. 2004), 859–872. Google ScholarDigital Library
- Giovanni Denaro, Alessandra Gorla, and Mauro Pezzè. 2008. Contextual Integration Testing of Classes. In Proceedings of the 11th International Conference on Fundamental Approaches to Software Engineering (FASE ’08). Springer, 246–260. Google ScholarDigital Library
- Roong-Ko Doong and Phyllis G. Frankl. 1994. The ASTOOT approach to testing object-oriented programs. ACM Transactions on Software Engineering and Methodology 3, 2 (1994), 101–130. Google ScholarDigital Library
- Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying Relations for Open Information Extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP ’11). Association for Computational Linguistics, 1535–1545. Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. In Proceedings of the European Software Engineering Conference held jointly with the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE ’11). ACM, 416–419. Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2013. Whole Test Suite Generation. IEEE Transactions on Software Engineering 39, 2 (2013), 276–291. Google ScholarDigital Library
- Susumu Fujiwara, Gregor von Bochmann, Ferhat Khendek, Mokhtar Amalou, and Abderrazak Ghedamsi. 1991. Test Selection Based on Finite State Models. IEEE Transactions on Software Engineering 17, 6 (1991), 591–603. Google ScholarDigital Library
- John Gannon, Paul McMullin, and Richard Hamlet. 1981. Data Abstraction, Implementation, Specification, and Testing. ACM Transactions on Programming Languages and Systems 3, 3 (1981), 211–223. Google ScholarDigital Library
- Alejandra Garrido and Jose Meseguer. 2006. Formal Specification and Verification of Java Refactorings. In Proceedings of the IEEE International Workshop on Source Code Analysis and Manipulation (SCAM ’06). 165–174. Google ScholarDigital Library
- Alberto Goffi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezzè. 2016. Automatic generation of oracles for exceptional behaviors. In ISSTA 2016, Proceedings of the 2016 International Symposium on Software Testing and Analysis. Saarbrücken, Genmany, 213–224. Google ScholarDigital Library
- Arnaud Gotlieb. 2003. Exploiting Symmetries to Test Programs. In Proceedings of the International Symposium on Software Reliability Engineering (ISSRE ’03). IEEE Computer Society, 365–374. Google ScholarDigital Library
- Florian Gross, Gordon Fraser, and Andreas Zeller. 2012. Search-based system testing: high coverage, no false alarms. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA ’12). ACM, 67–77. Google ScholarDigital Library
- A. Hartman. 2002. Is ISSTA research relevant to industry?. In ISSTA 2002, Proceedings of the 2002 International Symposium on Software Testing and Analysis. Rome, Italy, 205–206. Google ScholarDigital Library
- Susmit Jha, Sumit Gulwani, Sanjit A. Seshia, and Ashish Tiwari. 2010. Oracleguided Component-based Program Synthesis. In Proceedings of the International Conference on Software Engineering (ICSE ’10). ACM, 215–224. Google ScholarDigital Library
- James A. Jones, Mary Jean Harrold, and John Stasko. 2002. Visualization of test information to assist fault localization. In Proceedings of the International Conference on Software Engineering (ICSE ’02). ACM, 467–477. Google ScholarDigital Library
- Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From Word Embeddings to Document Distances. In Proceedings of the International Conference on International Conference on Machine Learning (ICML ’15). JMLR.org, 957–966. Google ScholarDigital Library
- Andreas Leitner, Ilinca Ciupa, Manuel Oriol, Bertrand Meyer, and Arno Fiva. 2007. Contract Driven Development = Test Driven Development - Writing Test Cases. In Proceedings of the European Software Engineering Conference held jointly with the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE ’07). ACM, 425–434. Google ScholarDigital Library
- Lei Ma, Cyrille Artho, Cheng Zhang, Hiroyuki Sato, Johannes Gmeiner, and Rudolf Ramler. 2015. GRT: Program-Analysis-Guided Random Testing. In Proceedings of the International Conference on Automated Software Engineering (ASE ’15). ACM, 212–223.Google ScholarDigital Library
- Leonardo Mariani, Mauro Pezzè, Oliviero Riganelli, and Mauro Santoro. 2012. AutoBlackTest: Automatic Black-Box Testing of Interactive Applications. In Proceedings of the International Conference on Software Testing, Verification and Validation (ICST ’12). IEEE Computer Society, 81–90. Google ScholarDigital Library
- Marie-Catherine Marneffe, Bill MacCartney, and Christopher Manning. 2006. Generating Typed Dependency Parses from Phrase Structure Parses. In Proceedings of the International Conference on Language Resources and Evaluation (LREC ’06). European Language Resources Association (ELRA), 449–454.Google Scholar
- Jason Mcdonald. 1998. Translating Object-Z Specifications to Passive Test Oracles. In Proceedings of the International Conference on Formal Engineering Methods (ICFEM ’98). 165–174. Google ScholarDigital Library
- Bertrand Meyer. 1988. Object-Oriented Software Construction (1st ed.). Prentice Hall. Google ScholarDigital Library
- Bertrand Meyer, Ilinca Ciupa, Andreas Leitner, and Lisa Ling Liu. 2007. Automatic Testing of Object-Oriented Software. In Proceedings of the 33rd Conference on Current Trends in Theory and Practice of Computer Science (SOFSEM ’07). Springer, 114–129. Google ScholarDigital Library
- Erich Mikk. 1995. Compilation of Z Specifications into C for Automatic Test Result Evaluation. In Proceedings of the 9th International Conference of Z Users (ZUM ’95). 167–180. Google ScholarDigital Library
- Carlos Pacheco and Michael D. Ernst. 2005. Eclat: Automatic generation and classification of test inputs. In ECOOP 2005 — Object-Oriented Programming, 19th European Conference. Glasgow, Scotland, 504–527. Google ScholarDigital Library
- Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In ICSE 2007, Proceedings of the 29th International Conference on Software Engineering. Minneapolis, MN, USA, 75–84. Google ScholarDigital Library
- Rahul Pandita, Xusheng Xiao, Hao Zhong, Tao Xie, Stephen Oney, and Amit Paradkar. 2012. Inferring method specifications from natural language API descriptions. In ICSE 2011, Proceedings of the 34th International Conference on Software Engineering. Zürich, Switzerland, 815–825. Google ScholarDigital Library
- Rahul Pandita, Xusheng Xiao, Hao Zhong, Tao Xie, Stephen Oney, and Amit Paradkar. 2012. Inferring Method Specifications from Natural Language API Descriptions. In Proceedings of the International Conference on Software Engineering (ICSE ’12). IEEE Computer Society, 815–825. Google ScholarDigital Library
- Hung Phan, Hoan Anh Nguyen, Tien N. Nguyen, and Hridesh Rajan. 2017. Statistical Learning for Inference Between Implementations and Documentation. In Proceedings of the 39th International Conference on Software Engineering: New Ideas and Emerging Results Track (ICSE-NIER ’17). IEEE Press, Piscataway, NJ, USA, 27–30. Google ScholarDigital Library
- Research Triangle Institute. 2002. The Economic Impacts of Inadequate Infrastructure for Software Testing. NIST Planning Report 02-3. National Institute of Standards and Technology.Google Scholar
- David S. Rosenblum. 1995. A Practical Approach to Programming With Assertions. IEEE Transactions on Software Engineering 21, 1 (1995), 19–31. Google ScholarDigital Library
- Michael Schmitz, Robert Bart, Stephen Soderland, Oren Etzioni, et al. 2012. Open language learning for information extraction. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL ’12). Association for Computational Linguistics, 523–534. ISSTA’18, July 16–21, 2018, Amsterdam, Netherlands A. Blasi, A. Goffi, K. Kuznetsov, A. Gorla, M. D. Ernst, M. Pezzè, S. D. Castellanos Google ScholarDigital Library
- Saurabh Srivastava, Sumit Gulwani, and Jeffrey S. Foster. 2010. From Program Verification to Program Synthesis. In Proceedings of the Annual ACM SIGPLANSIGACT Symposium on Principles of Programming Languages (POPL ’10). ACM, 313–326. Google ScholarDigital Library
- Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. /*iComment: Bugs or Bad Comments?*/. In SOSP 2007, Proceedings of the 21st ACM Symposium on Operating Systems Principles. Stevenson, WA, USA, 145–158. Google ScholarDigital Library
- Lin Tan, Yuanyuan Zhou, and Yoann Padioleau. 2011. aComment: Mining Annotations from Comments and Code to Detect Interrupt Related Concurrency Bugs. In Proceedings of the International Conference on Software Engineering (ICSE ’11). 11–20. Google ScholarDigital Library
- Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens. 2012. @tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies. In Proceedings of the International Conference on Software Testing, Verification and Validation (ICST ’12). IEEE Computer Society, 260–269. Google ScholarDigital Library
- Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens. 2012. @tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies. In ICST 2012: Fifth International Conference on Software Testing, Verification and Validation (ICST). Montreal, Canada, 260–269. Google ScholarDigital Library
- Richard N. Taylor. 1983. An Integrated Verification and Testing Environment. Software: Practice and Experience 13, 8 (1983), 697–713.Google ScholarCross Ref
- Willem Visser, Klaus Havelund, Guillaume Brat, SeungJoon Park, and Flavio Lerda. 2003. Model Checking Programs. Automated Software Engineering 10, 2 (2003), 203–232. Google ScholarDigital Library
- Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and isolating failureinducing input. IEEE Transactions on Software Engineering 28, 3 (February 2002), 183–200. Google ScholarDigital Library
Index Terms
- Translating code comments to procedure specifications
Recommendations
Call Me Maybe: Using NLP to Automatically Generate Unit Test Cases Respecting Temporal Constraints
ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software EngineeringA class may need to obey temporal constraints in order to function correctly. For example, the correct usage protocol for an iterator is to always check whether there is a next element before asking for it; iterating over a collection when there are no ...
C2S: translating natural language comments to formal program specifications
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringFormal program specifications are essential for various software engineering tasks, such as program verification, program synthesis, code debugging and software testing. However, manually inferring formal program specifications is not only time-...
A Decompositional Approach to Automatic Test Case Generation Based on Formal Specifications
SSIRI '10: Proceedings of the 2010 Fourth International Conference on Secure Software Integration and Reliability ImprovementAutomatic functional testing is a potentially effective technique for software reliability, but it has been a challenge due to the difficulties in automatically deriving adequate test cases and test oracle from informal or semi-formal specifications that ...
Comments