Andreas Holzer
Christian Schallhart
ABSTRACT
Although testing is central to debugging and software certification, there is no adequate language to specify test suites over source code. Such a language should be simple and concise in daily use, feature a precise semantics, and of course, it has to facilitate suitable engines to compute test suites and assess the coverage achieved by a test suite.
This paper introduces the language FQL designed to fit these purposes. We achieve the necessary expressive power by a natural extension of regular expressions which matches test suites rather than individual executions. To evaluate the language, we show for a list of informal requirements how to express them in FQL. Moreover, we present a test case generation engine for C programs and perform practical experiments with the sample specifications.
- }}P. Ammann, J. Offutt, and W. Xu. Coverage criteria for state based specifications. In FORTEST, pages 118--156, 2008. Google Scholar
Digital Library
- }}T. Ball. A theory of predicate-complete test coverage and generation. In FMCO, pages 1--22, 2004. Google ScholarDigital Library
- }}D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majumdar. Generating Tests from Counterexamples. In ICSE, pages 326--335, 2004. Google ScholarDigital Library
- }}D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majumdar. The Blast Query Language for Software Verification. In SAS, pages 2--18, 2004. Google ScholarDigital Library
- }}D. L. Bird and C. U. Munoz. Automatic generation of random self-checking test cases. IBM Systems Journal, 22(3):229--245, 1983. Google ScholarDigital Library
- }}J. Blom, A. Hessel, B. Jonsson, and P. Pettersson. Specifying and generating test cases using observer automata. In FATES, pages 125--139, 2004. Google ScholarDigital Library
- }}L. B. Briones, E. Brinksma, and M. Stoelinga. A semantic framework for test coverage. In ATVA, pages 399--414, 2006. Google ScholarDigital Library
- }}BullseyeCoverage 7.11.15. http://www.bullseye.com/.Google Scholar
- }}C. Cadar, D. Dunbar, and D. R. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, pages 209--224, 2008. Google ScholarDigital Library
- }}E. M. Clarke, D. Kroening, and F. Lerda. A Tool for Checking ANSI-C Programs. In TACAS, pages 168--176, 2004.Google ScholarCross Ref
- }}CoverageMeter 5.0.3. http://www.coveragemeter.com/.Google Scholar
- }}CTC++ 6.5.3. http://www.verifysoft.com/en.html.Google Scholar
- }}G. Din. TTCN-3. In Model-Based Testing of Reactive Systems, pages 465--496, 2004.Google Scholar
- }}Software Considerations in Airborne Systems and Equipment Certification (DO-178B). RTCA, 1992.Google Scholar
- }}M. Friske, H. Schlingloff, and S. Weißleder. Composition of model-based test coverage criteria. In MBEES, 2008.Google Scholar
- }}A. Galloway, G. Lüttgen, J. T. Mühlberg, and R. Siminiceanu. Model-checking the linux virtual file system. In VMCAI, pages 74--88, 2009. Google ScholarDigital Library
- }}P. Godefroid. Compositional dynamic test generation. In POPL, pages 47--54, 2007. Google ScholarDigital Library
- }}P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. In PLDI, pages 213--223, 2005. Google ScholarDigital Library
- }}B. S. Gulavani, T. A. Henzinger, Y. Kannan, A. V. Nori, and S. K. Rajamani. SYNERGY: a new algorithm for property checking. In SIGSOFT FSE, pages 117--127, 2006. Google ScholarDigital Library
- }}T. A. Henzinger, R. Jhala, R. Majumdar, and G. Sutre. Lazy abstraction. In POPL, pages 58--70, 2002. Google ScholarDigital Library
- }}A. Hessel, K. G. Larsen, M. Mikucionis, B. Nielsen, P. Pettersson, and A. Skou. Testing real-time systems using UPPAAL. In FORTEST, pages 77--117, 2008. Google ScholarDigital Library
- }}A. Holzer, V. Januzaj, S. Kugele, C. Schallhart, M. Tautschnig, H. Veith, and B. Langer. Slope testing for activity diagrams and safety critical software. Technical Report TUD-CS-2009-0184, TU Darmstadt, 2009.Google Scholar
- }}A. Holzer, C. Schallhart, M. Tautschnig, and H. Veith. FShell: Systematic Test Case Generation for Dynamic Analysis and Measurement. In CAV, pages 209--213, 2008. Google ScholarDigital Library
- }}A. Holzer, C. Schallhart, M. Tautschnig, and H. Veith. Query-Driven Program Testing. In VMCAI, pages 151--166, 2009. Google ScholarDigital Library
- }}H. S. Hong, I. Lee, O. Sokolsky, and H. Ural. A temporal logic based theory of test coverage and generation. In TACAS, pages 327--341, 2002. Google ScholarDigital Library
- }}J. C. Huang. An approach to program testing. ACM Comput. Surv., 7(3):113--128, 1975. Google ScholarDigital Library
- }}G. Myers. The Art of Software Testing. Wiley, 2004. Google ScholarDigital Library
- }}S. C. Ntafos. A comparison of some structural testing strategies. IEEE Trans. Software Eng., 14(6):868--874, 1988. Google ScholarDigital Library
- }}Rational Test RealTime 7.5. http://www.ibm.com/software/awdtools/test/realtime/.Google Scholar
- }}I. Schieferdecker, Z. R. Dai, J. Grabowski, and A. Rennoch. The UML 2.0 testing profile and its relation to TTCN-3. In TestCom, pages 79--94, 2003. Google ScholarDigital Library
- }}K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for C. In ESEC/SIGSOFT FSE, pages 263--272, 2005. Google ScholarDigital Library
- }}S. A. Vilkomir and J. P. Bowen. From MC/DC to RC/DC: Formalization and analysis of control-flow testing criteria. In FORTEST, pages 240--270, 2008. Google ScholarDigital Library
- }}M. Zolda, S. Bünte, and R. Kirner. Towards Adaptable Control Flow Segmentation for Measurement-Based Execution Time Analysis. In RTNS, 2009.Google Scholar
Index Terms
- How did you specify your test suite
Recommendations
Coverage is not strongly correlated with test suite effectiveness
ICSE 2014: Proceedings of the 36th International Conference on Software EngineeringThe coverage of a test suite is often used as a proxy for its ability to detect faults. However, previous studies that investigated the correlation between code coverage and test suite effectiveness have failed to reach a consensus about the nature and ...
Assertions are strongly correlated with test suite effectiveness
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software EngineeringCode coverage is a popular test adequacy criterion in practice. Code coverage, however, remains controversial as there is a lack of coherent empirical evidence for its relation with test suite effectiveness. More recently, test suite size has been ...
Achieving scalable mutation-based generation of whole test suites
Without complete formal specification, automatically generated software tests need to be manually checked in order to detect faults. This makes it desirable to produce the strongest possible test set while keeping the number of tests as small as ...
Comments