Marat Boshernitsan
Roongko Doong
ABSTRACT
Developer testing is of one of the most effective strategies for improving the quality of software, reducing its cost, and accelerating its development. Despite its widely recognized benefits, developer testing is practiced by only a minority of developers. The slow adoption of developer testing is primarily due to the lack of tools that automate some of the more tedious and time-consuming aspects of this practice. Motivated by the need for a solution, and helped and inspired by the research in software test automation, we created a developer testing tool based on software agitation. Software agitation is a testing technique that combines the results of research in test-input generation and dynamic invariant detection. We implemented software agitation in a commercial testing tool called Agitator. This paper gives a high-level overview of software agitation and its implementation in Agitator, focusing on the lessons and challenges of leveraging and applying the results of research to the implementation of a commercial product.
- R. S. Arnold. Sotware restructuring. In Proceedings of the IEEE, volume 77, pages 607--617, 1989.Google Scholar
Cross Ref
- K. Beck and E. Gamma. Test infected: Programmers love writing tests. In Java Report, volume 3, pages 37--50, 1998.Google Scholar
- R. E. Brooks. Commercial reality. Psychology of Programming Interest Group Mailing List, March 2005. http://www.mail-archive.com/[email protected]/msg00958.html.Google Scholar
- J. deRaeve and S. P. McCarron. Automated test generation technology. Technical report, X/Open Company Ltd., 1997. http://adl.opengroup.org/documents/Archive/adl10rep.pdf.Google Scholar
- Eclipse.org. Eclipse platform: technical overview, 2003. http://eclipse.org/white-papers/ eclipse-overview.pdf.Google Scholar
- M. D. Ernst. Dynamically discovering likely program invariants. PhD thesis, University of Washington, 2000. Google ScholarDigital Library
- M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin. Dynamically discovering likely program invariants to support program evolution. In ICSE '99: 21st International Conference on Software Engineering, pages 213--224, 1999. Google ScholarDigital Library
- M. Fowler. Refactoring: improving the design of existing code. Object Technology Series. Addison-Wesley, 1999. Google ScholarDigital Library
- P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. In PLDI '05: 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 213--223, 2005. Google ScholarDigital Library
- W. G. Griswold and D. Notkin. Automated assistance for program restructuring. ACM Transactions on Software Engineering and Methodology, 2(3):228--269, July 1993. Google ScholarDigital Library
- M. Harder, J. Mellen, and M. D. Ernst. Improving test suites via operational abstraction. In ICSE '03: 27th International Conference on Software Engineering, pages 60--73, 2003. Google ScholarDigital Library
- J. Henkel and A. Diwan. Discovering algebraic specifications from Java classes. In ECOOP '03: European Conference on Object-Oriented Programming, pages 431--456, 2003.Google ScholarCross Ref
- Hybernate.org. Relational persistence for Java and .NET. http://www.hibernate.org/.Google Scholar
- D. Ingalls. Fabrik: A visual programming environment. In OOPSLA '88: International Conference on Object-Oriented Programming, Systems, Languages, and Applications, volume 23, pages 176--190, Nov. 1988. Google ScholarDigital Library
- JUnit. http://www.junit.org.Google Scholar
- G. J. Myers. The Art of Software Testing. Wiley - Interscience, New York, 1979. Google ScholarDigital Library
- NUnit. http://www.nunit.org.Google Scholar
- W. F. Opdyke. Refactoring object-oriented frameworks. PhD thesis, University of Illinois at Urbana-Champaign, 1992. Google ScholarDigital Library
- C. Pacheco and M. D. Ernst. Eclat: Automatic generation and classification of test inputs. In ECOOP '05: European Conference on object-Oriented Programming, pages 504--527, 2005. Google ScholarDigital Library
- S. G. Parker, D. M. Weinstein, and C. R. Johnson. The SCIRun computational steering software system. In E. Arge, A. M. Bruaset, and H. P. Langtangen, editors, Modern Software Tools in Scientific Computing. Birkhauser Press, 1997. Google ScholarDigital Library
- J. H. Perkins and M. D. Ernst. Efficient incremental algorithms for dynamic detection of likely invariants. In SIGSOFT '04/FSE-12: Proceedings of the 12th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 23--32, New York, NY, USA, 2004. ACM Press. Google ScholarDigital Library
- N. H. Petschenik. Building awareness of system testing issues. In ICSE '85: 8th International Conference on Software Engineering, pages 182--188, Los Alamitos, CA, USA, 1985. IEEE Computer Society Press. Google ScholarDigital Library
- Robby, M. B. Dwyer, and J. Hatcliff. Bogor: an extensible and highly-modular software model checking framework. In ESEC/FSE-11: Proceedings of the 9th European Software Engineering Conference/11th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 267--276, New York, NY, USA, 2003. ACM Press. Google ScholarDigital Library
- T. Robschink and G. Snelting. Efficient path conditions in dependence graphs. In ICSE '02: Proceedings of the 24th International Conference on Software Engineering, pages 478--488, New York, NY, USA, 2002. ACM Press. Google ScholarDigital Library
- D. Saff and M. D. Ernst. An experimental evaluation of continuous testing during development. In ISSTA '04: Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 76--85, New York, NY, USA, 2004. ACM Press. Google ScholarDigital Library
- S. Sankar and R. Hayes. Specifying and testing software components using ADL. Technical Report SMLI TR-94-23, Sun Microsystems Laboratories, April 1994. http://research.sun.com/techrep/ 1994/smli tr-94-23.pdf. Google ScholarDigital Library
- K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for C. In ESEC/FSE-13: 10th European Software Engineering Conference/13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 263--272, 2005. Google ScholarDigital Library
- SpringFramework.org. Spring application framework. http://www.springframework.org/.Google Scholar
- Sun Microsystems. Java platform, Enterprise Edition. http://java.sun.com/javaee/.Google Scholar
- The Apache Software Foundation. Apache Struts Project. http://struts.apache.org/.Google Scholar
- The Jakarta Project. Commons collections. http://jakarta.apache.org/commons/collections/.Google Scholar
- D. Thomas and A. Hunte. Mock objects. IEEE Software, 19(3):22--24, 2002. Google ScholarDigital Library
- N. Tillmann and W. Schulte. Parameterized unit tests. In ESEC/FSE-13: 10th European Software Engineering Conference/13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 253--262, 2005. Google ScholarDigital Library
- R. Vanmegen and D. B. Meyerhoff. Costs and benefits of early defect detection-experiences from developing client-server and host applications. Software Quality Journal, 4(4):247--256, 1995.Google ScholarCross Ref
- W. Visser, C. S. Pasareanu, and S. Khurshid. Test input generation with Java PathFinder. In ISSTA '04: Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 97--107, New York, NY, USA, 2004. ACM Press. Google ScholarDigital Library
- T. Xie, D. Marinov, W. Schulte, and D. Notkin. Symstra: A framework for generating object-oriented unit tests using symbolic execution. In TACAS '05: 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, volume 3440 of LNCS, pages 365--381. Springer-Verlag, Apr. 2005. Google ScholarDigital Library
- T. Xie and D. Notkin. Tool-assisted unit test selection based on operational violations. In ASE '03: International Conference on Automated Software Engineering, pages 40--48, 2003.Google Scholar
- T. Xie and D. Notkin. Automatically identifying special and common unit tests for object-oriented programs. In ISSRE '05: International Symposium on Software Reliability Engineering, pages 277--287, 2005. Google ScholarDigital Library
Index Terms
- From daikon to agitator: lessons and challenges in building a commercial tool for developer testing
Recommendations
Smart Unit Testing Framework
ISSREW '12: Proceedings of the 2012 IEEE 23rd International Symposium on Software Reliability Engineering WorkshopsUnit testing(UT) is an important step in ensuring the quality of software. Considerable effort is spent in unit testing. There are several frameworks to help with UT. Some common frameworks are Cunit, Junit, Nunit etc. All of these have very similar ...
Future of developer testing: building quality in code
FoSER '10: Proceedings of the FSE/SDP workshop on Future of software engineering researchAlthough much progress has been made in software verification, software testing remains by far the most widely used technique for improving software reliability. Among various types of testing, developer testing is a type of testing where developers ...
When, how, and why developers (do not) test in their IDEs
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software EngineeringThe research community in Software Engineering and Software Testing in particular builds many of its contributions on a set of mutually shared expectations. Despite the fact that they form the basis of many publications as well as open-source and ...
Comments