Lennart C.L. Kats
Eelco Visser
ABSTRACT
The reliability of compilers, interpreters, and development environments for programming languages is essential for effective software development and maintenance. They are often tested only as an afterthought. Languages with a smaller scope, such as domain-specific languages, often remain untested. General-purpose testing techniques and test case generation methods fall short in providing a low-threshold solution for test-driven language development. In this paper we introduce the notion of a language-parametric testing language (LPTL) that provides a reusable, generic basis for declaratively specifying language definition tests. We integrate the syntax, semantics, and editor services of a language under test into the LPTL for writing test inputs. This paper describes the design of an LPTL and the tool support provided for it, shows use cases using examples, and describes our implementation in the form of the Spoofax testing language.
- K. Beck. Test-driven development: by example. Addison-Wesley Professional, 2003. Google Scholar
Digital Library
- B. Beizer. Software testing techniques. Dreamtech Press, 2002.Google Scholar
- A. Boujarwah and K. Saleh. Compiler test case generation methods: a survey and assessment. Information and Software Technology, 39(9):617--625, 1997.Google ScholarCross Ref
- J. Bovet and T. Parr. ANTLRWorks: an ANTLR grammar development environment. Software: Practice and Experience, 38(12):1305--1332, 2008. Google ScholarDigital Library
- M. Bravenboer, K. T. Kalleberg, R. Vermaas, and E. Visser. Stratego/XT 0.17. A language and toolset for program transformation. Science of Computer Programming, 72(1-2):52-- 70, 2008. Google ScholarDigital Library
- M. Bravenboer and E. Visser. Concrete syntax for objects: domain-specific language embedding and assimilation without restrictions. In J. M. Vlissides and D. C. Schmidt, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2004, pages 365--383. ACM, 2004. Google ScholarDigital Library
- P. Charles, R. M. Fuhrer, S. M. Sutton, Jr., E. Duesterwald, and J. Vinju. Accelerating the creation of customized, language-specific IDEs in Eclipse. In Proceeding of the 24th ACM SIGPLAN conference on Object oriented programming systems languages and applications, OOPSLA '09, pages 191--206. ACM, 2009. Google ScholarDigital Library
- B. Daniel, D. Dig, K. Garcia, and D. Marinov. Automated testing of refactoring engines. In I. Crnkovic and A. Bertolino, editors, 6th joint meeting of the European Software Engineering Conference and the Int. Symposium on Foundations of Software Engineering, pages 185--194. ACM, 2007. Google ScholarDigital Library
- M. de Jonge, E. Nilsson-Nyman, L. C. L. Kats, and E. Visser. Natural and flexible error recovery for generated parsers. In M. van den Brand, D. Gasevic, and J. Gray, editors, Software Language Engineering, SLE 2009, volume 5969 of LNCS, pages 204--223. Springer, 2009. Google ScholarDigital Library
- P. Degano and C. Priami. Comparison of syntactic error handling in LR parsers. Software -- Practice and Experience, 25(6):657--679, 1995. Google ScholarDigital Library
- S. Efftinge and M. Völter. oAW xText - a framework for textual DSLs. In Modeling Symposium, Eclipse Summit, 2006.Google Scholar
- M. Fowler. Refactoring: Improving the design of existing code. In D. Wells and L. A. Williams, editors, Extreme Programming and Agile Methods - XP/Agile Universe 2002, Second XP Universe and First Agile Universe Conference, volume 2418 of LNCS, page 256. Springer, 2002. Google ScholarDigital Library
- M. Fowler. Language workbenches: The killer-app for domain specific languages? http://martinfowler.com/articles/languageWorkbench.html, 2005.Google Scholar
- J. D. Frens and A. Meneely. Fifteen compilers in fifteen days. In D. Baldwin, P. T. Tymann, S. M. Haller, and I. Russell, editors, 39th Technical Symposium on Computer Science Education, 2006, pages 92--96. ACM, 2006. Google ScholarDigital Library
- E. Gamma and K. Beck. Test infected: Programmers love writing tests. Java Report, 3(7):37--50, 1998.Google Scholar
- J. B. Goodenough. The Ada compiler validation capability. In Proceedings of the ACM-SIGPLAN symposium on Ada programming language, SIGPLAN '80, pages 1--8. ACM, 1980. Google ScholarDigital Library
- gUnit - grammar unit testing. http://www.antlr.org/wiki/display/ANTLR3/gUnit+-+Grammar+ Unit+Testing.Google Scholar
- R. Gómez, J. C. Augusto, and A. Galton. Testing an event specification language. In Software Engineering & Knowledge Engineering, SEKE 2001, pages 341--345, 2001.Google Scholar
- P. Hamill. Unit Test Frameworks, chapter. Chapter 3: The xUnit Family of Unit Test Frameworks. O'Reilly, 2004.Google Scholar
- Z. Hemel and E. Visser. Declaratively programming the mobile web with mobl. In Proceedings of the 26th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2011, Portland, Oregon, USA, 2011. ACM. Google ScholarDigital Library
- P. Hudak. Modular domain specific languages and tools. In Software Reuse, ICSR '98. Computer Society, jun 1998. Google ScholarDigital Library
- Jacks (Jacks is an Automated Compiler Killing Suite). http://sources.redhat.com/mauve/jacks. html.Google Scholar
- L. C. L. Kats, M. de Jonge, E. Nilsson-Nyman, and E. Visser. Providing rapid feedback in generated modular language environments: adding error recovery to scannerless generalized- LR parsing. In S. Arora and G. T. Leavens, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2009, pages 445--464. ACM, 2009. Google ScholarDigital Library
- L. C. L. Kats, K. T. Kalleberg, and E. Visser. Interactive disambiguation of meta programs with concrete object syntax. In M. van den Brand, B. Malloy, and S. Staab, editors, Software Language Engineering, SLE 2010, LNCS. Springer, 2011. Google ScholarDigital Library
- L. C. L. Kats and E. Visser. The Spoofax language workbench: rules for declarative specification of languages and IDEs. In W. R. Cook, S. Clarke, and M. C. Rinard, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2010, pages 444--463. ACM, 2010. Google ScholarDigital Library
- L. C. L. Kats, E. Visser, and G. Wachsmuth. Pure and declarative syntax definition: paradise lost and regained. In W. R. Cook, S. Clarke, and M. C. Rinard, editors, Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2010, pages 918--932. ACM, 2010. Google ScholarDigital Library
- A. Kossatchev and M. Posypkin. Survey of compiler testing methods. Programming and Computer Software, 31(1):10-- 19, 2005. Google ScholarDigital Library
- H. Krahn, B. Rumpe, and S. Völkel. Monticore: Modular development of textual domain specific languages. In R. F. Paige and B. Meyer, editors, Objects, Components, Models and Patterns, TOOLS EUROPE 2008, volume 11 of Lecture Notes in Business Information Processing, pages 297--315. Springer, 2008.Google Scholar
- R. Lämmel. Grammar testing. In H. Hußmann, editor, Fundamental Approaches to Software Engineering, FASE 2001, volume 2029 of LNCS, pages 201--216. Springer, 2001. Google ScholarDigital Library
- B. A. Malloy, J. F. Power, and J. T. Waldron. Applying software engineering techniques to parser design: the development of a C# parser. In SAICSIT '02: 2002 research conference of the South African institute of computer scientists and information technologists on Enablement through technology. South African Institute for Computer Scientists and Information Technologists, 2002. Google ScholarDigital Library
- G. Myers. The art of software testing, 2nd edition. WileyIndia, 2008. Google ScholarDigital Library
- R. W. Selby, editor. Software Engineering: Barry W. Boehm's Lifetime Contributions to Software Development, Management, and Research. Wiley-Computer Society Press, 2007. Google ScholarDigital Library
- M. Strembeck and U. Zdun. An approach for the systematic development of domain-specific languages. Software: Practice and Experience, 39(15):1253--1292, 2009. Google ScholarDigital Library
- L. Tratt. Domain specific language implementation via compile-time meta-programming. Trans. Program. Lang. Syst., 30(6), 2008. Google ScholarDigital Library
- E. Visser. Syntax Definition for Language Prototyping. PhD thesis, University of Amsterdam, September 1997.Google Scholar
- E. Visser. WebDSL: A case study in domain-specific language engineering. In R. Lämmel, J. Visser, and J. Saraiva, editors, Generative and Transformational Techniques in Software Engineering II, Int. Summer School, GTTSE 2007, volume 5235 of LNCS, pages 291--373. Springer, 2007. Google ScholarDigital Library
- M. Voelter and K. Solomatov. Language modularization and composition with projectional language workbenches illustrated with MPS. In M. van den Brand, B. Malloy, and S. Staab, editors, Software Language Engineering, SLE 2010, LNCS. Springer, 2010.Google Scholar
- B. Wichmann and Z. Ciechanowicz. Pascal compiler validation. John Wiley & Sons, Inc. New York, NY, USA, 1983. Google ScholarDigital Library
- H. Wu, J. Gray, and M. Mernik. Grammar-driven generation of domain-specific language debuggers. Software: Practice and Experience, 38(10):1073--1103, 2008. Google ScholarDigital Library
- X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in C compilers. In Conference on Programming Language Design and Implementation, PLDI 2011. Press, June 2011. Google ScholarDigital Library
Index Terms
- Integrated language definition testing: enabling test-driven language development
Recommendations
Integrated language definition testing: enabling test-driven language development
OOPSLA '11The reliability of compilers, interpreters, and development environments for programming languages is essential for effective software development and maintenance. They are often tested only as an afterthought. Languages with a smaller scope, such as ...
The spoofax language workbench: rules for declarative specification of languages and IDEs
OOPSLA '10: Proceedings of the ACM international conference on Object oriented programming systems languages and applicationsSpoofax is a language workbench for efficient, agile development of textual domain-specific languages with state-of-the-art IDE support. Spoofax integrates language processing techniques for parser generation, meta-programming, and IDE development into ...
Testing domain-specific languages
OOPSLA '11: Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companionThe Spoofax testing language provides a new approach to testing domain-specific languages as they are developed. It allows test cases to be written using fragments of the language under test, providing full IDE support for writing test cases and ...
Comments