skip to main content
10.1145/1291535.1291536acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
Article

Achieving both model and code coverage with automated gray-box testing

Published:09 July 2007Publication History

ABSTRACT

We have devised a novel technique to automatically generate test cases for a software system, combining black-box model-based testing with white-box parameterized unit testing. The former provides general guidance for the structure of the tests in the form of test sequences, as well as the oracle to check for conformance of an application under test with respect to a behavioral model. The latter finds a set of concrete parameter values that maximize code coverage using symbolic analysis. By applying these techniques together, we can produce test definitions (expressed as code to be run in a test management framework) that exercise all selected paths in the model, while also covering code branches specific to the implementation. These results cannot be obtained from any of the individual approaches alone, as the model cannot predict what values are significant to a particular implementation, while parameterized unit testing requires manually written test sequences and correctness validations. We provide tool support, integrated into our model-based testing tool.

References

  1. R. Alur, T. A. Henzinger, O. Kupferman, and M. Vardi. Alternating refinement relations. In Proceedings of the 9th International Conference on Concurrency Theory (CONCUR '98), volume 1466 of LNCS, pages 163--178, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. C. Artho, H. Barringer, A. Goldberg, K. Havelund, S. Khurshid, M. Lowry, C. Pasareanu, G. Rosu, K. Sen, W. Visser, and R. Washington. Combining test case generation and runtime verification. Theor. Comput. Sci., 336(2--3):209--234, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. M. Barnett, W. Grieskamp, L. Nachmanson, W. Schulte, N. Tillmann, and M. Veanes. Towards a tool environment for model-based testing with AsmL. In Petrenko and Ulrich, editors, Formal Approaches to Software Testing, FATES 2003, volume 2931 of LNCS, pages 264--280. Springer, 2003.Google ScholarGoogle Scholar
  4. K. Beck. Extreme Programming Explained: Embrace Change. Addison-Wesley, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. K. Beck. Test-Driven Development: By Example. Addison-Wesley, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. C. Campbell, W. Grieskamp, L. Nachmanson, W. Schulte, N. Tillmann, and M. Veanes. Testing concurrent object-oriented systems with spec explorer. In J. Fitzgerald, I. J. Hayes, and A. Tarlecki, editors, FM 2005: Formal Methods, International Symposium of Formal Methods Europe, volume 3582 of Lecture Notes in Computer Science, pages 542--547. Springer, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. L. de Alfaro and T. A. Henzinger. Interface automata. In Proceedings of the 8th European Software Engineering Conference and the 9th ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 109--120. ACM, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. E. Gamma and K. Beck. JUnit: A regression testing framework, 2001. http://www.junit.org.Google ScholarGoogle Scholar
  9. P. Godefroid, N. Klarlund, and K. Sen. DART: Directed automated random testing. In ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation (PLDI'05), pages 213--223, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. W. Grieskamp. Multi-paradigmatic model-based testing. In K. Havelund, M. Núñez, G. Rosu, and B. Wolff, editors, FATES/RV 2006: Formal Approaches to Testing and Runtime Verification, volume 4262 of Lecture Notes in Computer Science, pages 1--19. Springer, 2006. invited contribution. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. W. Grieskamp, Y. Gurevich, W. Schulte, and M. Veanes. Generating finite state machines from abstract state machines. In ISSTA'02, volume 27 of Software Engineering Notes, pages 112--122. ACM, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. W. Grieskamp and N. Kicillof. A schema language for coordinating construction and composition of partial behavior descriptions. In J. Whittle, L. Geiger, and M. Meisinger, editors, SCESM, pages 59--66. ACM, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. W. Grieskamp, N. Kicillof, and N. Tillmann. Action machines: a framework for encoding and composing partial behaviors. International Journal of Software Engineering and Knowledge Engineering, 16(5):705--726, 2006.Google ScholarGoogle ScholarCross RefCross Ref
  14. R. Grosu and S. A. Smolka. Monte carlo model checking. In N. Halbwachs and L. D. Zuck, editors, TACAS, volume 3440 of Lecture Notes in Computer Science, pages 271--286. Springer, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. J. C. King. Symbolic execution and program testing. Commun. ACM, 19(7):385--394, 1976. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Z. Li, L. Tan, X. Wang, S. Lu, Y. Zhou, and C. Zhai. Have things changed now? -- An empirical study of bug characteristics in modern open source software. In ICSE: Proceedings 29th International Conference on Software Engineering, 2007.Google ScholarGoogle Scholar
  17. H. Robinson. Finite state model-based testing on a shoestring. In Proceedings of the International Conference on Software Testing Analysis and Review (STARWEST 1999), Software Quality Engineering, San Jose, CA, USA, October 1999.Google ScholarGoogle Scholar
  18. K. Sen and G. Agha. Cute and jCUTE: Concolic unit testing and explicit path model-checking tools. In 18th International Conference on Computer Aided Verification (CAV'06), volume 4144 of Lecture Notes in Computer Science, pages 419--423. Springer, 2006. (Tool Paper). Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. K. Stobie. Model based testing in practice at microsoft. In Proceedings of the Workshop on Model Based Testing (MBT 2004), volume 111 of Electronic Notes in Theoretical Computer Science. Elsevier, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. N. Tillmann and W. Schulte. Unit tests reloaded: Parameterized unit testing with symbolic execution. IEEE software, 23:38--47, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. O. Tkachuk, M. B. Dwyer, and C. S. Pasareanu. Automated environment generation for software model checking. In ASE, pages 116--129. IEEE Computer Society, 2003.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. J. Tretmans and A. Belinfante. Automatic testing with formal methods. In EuroSTAR'99: 7th European Int. Conference on Software Testing, Analysis & Review, Barcelona, Spain, November 8--12, 1999. EuroStar Conferences, Galway, Ireland.Google ScholarGoogle Scholar
  23. M. Utting and B. Legeard. Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. G. Venolia, R. DeLine, and T. LaToza. Software development at microsoft observed. Technical Report MSR-TR-2005-140, Microsoft Research, October 2005.Google ScholarGoogle Scholar
  25. W. Visser, C. S. Pasareanu, and R. Pelánek. Test input generation for red-black trees using abstraction. In D. F. Redmiles, T. Ellman, and A. Zisman, editors, ASE, pages 414--417. ACM, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Achieving both model and code coverage with automated gray-box testing

              Recommendations

              Comments

              Login options

              Check if you have access through your login credentials or your institution to get full access on this article.

              Sign in
              • Published in

                cover image ACM Conferences
                A-MOST '07: Proceedings of the 3rd international workshop on Advances in model-based testing
                July 2007
                127 pages
                ISBN:9781595938503
                DOI:10.1145/1291535

                Copyright © 2007 ACM

                Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

                Publisher

                Association for Computing Machinery

                New York, NY, United States

                Publication History

                • Published: 9 July 2007

                Permissions

                Request permissions about this article.

                Request Permissions

                Check for updates

                Qualifiers

                • Article

                Upcoming Conference

                ICSE 2025

              PDF Format

              View or Download as a PDF file.

              PDF

              eReader

              View online with eReader.

              eReader