Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Software and Systems Modeling
Published in: Software and Systems Modeling 3/2021

17-10-2020 | Regular Paper

Wodel-Test: a model-based framework for language-independent mutation testing

Authors: Pablo Gómez-Abajo, Esther Guerra, Juan de Lara, Mercedes G. Merayo

Published in: Software and Systems Modeling | Issue 3/2021

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Mutation testing (MT) targets the assessment of test cases by measuring their efficiency to detect faults. This technique involves modifying the program under test to emulate programming faults, and assessing whether the existing test cases detect such mutations. MT has been extensively studied since the 70’s, and many tools have been proposed for widely used languages like C, Java, Fortran, Ada and SQL; and for notations like Petri-nets. However, building MT tools is costly and error-prone, which may prevent their development for new programming and domain-specific (modelling) languages. In this paper, we propose a framework called Wodel-Test to reduce the effort to create MT tools. For this purpose, it follows a model-driven approach by which MT tools are synthesized from a high-level description. This description makes use of the domain-specific language Wodel to define and execute model mutations. Wodel is language-independent, as it allows the creation of mutation operators for any language defined by a meta-model. Starting from the definition of the mutation operators, Wodel-Test generates a MT environment which parses the program under test into a model, applies the mutation operators, and evaluates the test-suite against the generated mutants, offering a rich collection of MT metrics. We report on an evaluation of the approach based on the creation of MT tools for Java and the Atlas transformation language.
previous article Implementing QVT-R via semantic interpretation in UML-RSDS
next article Live modeling in the context of state machine models and code generation

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now
Appendix
Available only for authorised users
Footnotes
1
Wodel is built atop EMF. In EMF, it is customary to place all objects (except one, acting as root) inside some composition reference, called its container reference.
 
3
A mutation operator may not get applied if its application conditions do not occur in the source program, or if it always produces incorrect programs.
 
6
In refining mode, the input model of a transformation is changed in-place and produced as output.
 
Literature
1.
2.
Agrawal, H., DeMillo, R.A., Hathaway, B., Hsu, W., Hsu, W., Krauser, E., Martin, R., Mathur, A.P., Spafford, E.: Design of mutant operators for the C programming language. Technical report, Purdue University (1989)
3.
Aichernig, B.K., Brandl, H., Jöbstl, E., Krenn, W., Schlick, R., Tiran, S.: Killing strategies for model-based mutation testing. Softw. Test. Verif. Reliab. 25(8), 716–748 (2015)CrossRef
4.
Alhwikem, F., Paige, R.F., Rose, L., Alexander, R.: A systematic approach for designing mutation operators for MDE languages. In: Workshop on Model-Driven Engineering, Verification and Validation (MoDeVVa), pp. 54–59 (2016)
5.
Alsmadi, I.M.: Using mutation to enhance GUI testing coverage. IEEE Softw. 30(1), 67–73 (2013)CrossRef
6.
Andrews, J.H., Briand, L.C., Labiche, Y.: Is mutation an appropriate tool for testing experiments? In: International Conference on Software Engineering (ICSE). ACM, pp. 402–411 (2005)
7.
Aranega, V., Mottu, J.-M., Etien, A., Degueule, T., Baudry, B., Dekeyser, J.-L.: Towards an automation of the mutation analysis dedicated to model transformation. Softw. Test. Verif. Reliab. 25(5–7), 653–683 (2014)
8.
Ariel González, G.B., Carlos, L.: Mutation testing for Java based on model-driven development. In: Simposio Latinoamericano de Ingeniería de Software (CLEI-SLISW) (in Spanish) (2018)
9.
Aziz, B.: Towards a mutation analysis of IoT protocols. Inf. Softw. Technol. 100, 183–184 (2018)CrossRef
10.
Baker, R., Habli, I.: An empirical evaluation of mutation testing for improving the test quality of safety-critical software. IEEE Trans. Softw. Eng. 39(6), 787–805 (2013)CrossRef
11.
Baudry, B., Ghosh, S., Fleurey, F., France, R.B., Traon, Y.L., Mottu, J.: Barriers to systematic model transformation testing. Commun. ACM 53(6), 139–143 (2010)CrossRef
12.
Bradbury, J.S., Cordy, J.R., Dingel, J.: Mutation operators for concurrent Java (J2SE 5.0). In: Workshop on Mutation Analysis (Mutation), pp. 83–92 (2006)
13.
Brambilla, M., Cabot, J., Wimmer, M.: Model-Driven Software Engineering in Practice, Second Edition. Synthesis Lectures on Software Engineering. Morgan & Claypool Publishers (2017)
14.
Bruel, J., Combemale, B., Guerra, E., Jézéquel, J., Kienzle, J., de Lara, J., Mussbacher, G., Syriani, E., Vangheluwe, H.: Comparing and classifying model transformation reuse approaches across metamodels. Softw. Syst. Model. 19(2), 441–465 (2020)CrossRef
15.
Brunelière, H., Cabot, J., Dupé, G., Madiot, F.: MoDisco: a model driven reverse engineering framework. Inf. Softw. Technol. 56(8), 1012–1032 (2014)CrossRef
16.
Cañizares, P.C., Núñez, A., Merayo, M.G.: Mutomvo: mutation testing framework for simulated cloud and HPC environments. J. Syst. Softw. 143, 187–207 (2018)CrossRef
17.
18.
Delamaro, M.E., Maldonado.: Proteum—a tool for the assessment of test adequacy for C programs. In: Conference on Performability in Computing Systems, pp. 79–95 (1996)
19.
Delgado-Pérez, P., Medina-Bulo, I., Palomo-Lozano, F., García-Domínguez, A., Domínguez-Jiménez, J.J.: Assessment of class mutation operators for C++ with the MuCPP mutation system. Inf. Softw. Technol. 81, 169–184 (2017)CrossRef
20.
DeMillo, R.A., Guindi, D.S., McCracken, W.M., Offutt, A.J., King, K.N.: An extended overview of the Mothra software testing environment. In: Workshop on Software Testing, Verification, and Analysis, pp. 142–151 (1988)
21.
DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. IEEE Comput. 11(4), 34–41 (1978)CrossRef
22.
Deng, L., Offutt, A.J., Ammann, P., Mirzaei, N.: Mutation operators for testing android apps. Inf. Softw. Technol. 81, 154–168 (2017)CrossRef
23.
Denisov, A., Pankevich, S.: Mull it over: mutation testing based on LLVM. In: IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 25–31 (2018)
24.
Devroey, X., Perrouin, G., Papadakis, M., Legay, A., Schobbens, P., Heymans, P.: Model-based mutant equivalence detection using automata language equivalence and simulations. J. Syst. Softw. 141, 1–15 (2018)CrossRef
25.
Estero-Botaro, A., Palomo-Lozano, F., Medina-Bulo, I.: Mutation operators for WS-BPEL 2.0. In: International Conference on Software & Systems Engineering and their Applications (ICSSEA) (2008)
26.
Fabbri, S.C.P.F., Delamaro, M.E., Maldonado, J.C., Masiero, P.C..: Mutation analysis testing for finite state machines. In: International Symposium on Software Reliability Engineering (ISSRE), pp. 220–229 (1994)
27.
Fabbri, S.C.P.F., Maldonado, J.C.,P.C., Masiero, M.E,. Delamaro, Wong, W.E.: Mutation testing applied to validate specifications based on Petri nets. In: International Conference on Formal Description Techniques, volume 43 of IFIP Conference Proceedings. Chapman & Hall, pp. 329–337 (1995)
28.
Fabbri, S.C.P.F., Maldonado, J.C., Sugeta, T., Masiero, P.C.: Mutation testing applied to validate specifications based on statecharts. In: International Symposium on Software Reliability Engineering (ISSRE), p. 210 (1999)
29.
Ferenc, R., Beszedes, A., Tarkiainen, M., Gyimothy, T.: Columbus—reverse engineering tool and schema for C++. In: International Conference on Software Maintenance, pp. 172–181 (2002)
30.
Fraser, G., Zeller, A.: Mutation-driven generation of unit tests and oracles. IEEE Trans. Softw. Eng. 38(2), 278–292 (2012)CrossRef
31.
Gligoric, M., Badame, S., Johnson, R.: SMutant: a tool for type-sensitive mutation testing in a dynamic language. In: ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE) and European Software Engineering Conference (ESEC), pp. 424–427 (2011)
32.
33.
Gómez-Abajo, P., Guerra, E., de Lara, J.: A domain-specific language for model mutation and its application to the automated generation of exercises. Comput. Lang. Syst. Struct. 49, 152–173 (2017)
34.
Gómez-Abajo, P., Guerra, E., de Lara, J., Merayo, M.G.: A tool for domain-independent model mutation. Sci. Comput. Program. 163, 85–92 (2018)CrossRef
35.
Gómez-Abajo, P., Guerra, E., de Lara, J., Merayo, M.G.: Towards a model-driven engineering solution for language independent mutation testing. In: Jornadas de Ingeniería del Software y Bases de Datos (JISBD), p. 4. Biblioteca digital SISTEDES (2018)
36.
Gómez-Abajo, P., Guerra, E., de Lara, J., Merayo, M.G.: Mutation testing for DSLs (tool demo). In: ACM SIGPLAN International Workshop on Domain-Specific Modeling (DSM). ACM, pp. 60–62 (2019)
37.
Gong, D., Zhang, G., Yao, X., Meng, F.: Mutant reduction based on dominance relation for weak mutation testing. Inf. Softw. Technol. 81, 82–96 (2017)CrossRef
38.
González-Jiménez, M., de Lara, J.: Datalyzer: streaming data applications made easy. In: International Conference on Web Engineering (ICWE), volume 10845 of LNCS. Springer, pp. 420–429 (2018)
39.
Gopinath, R., Alipour, A., Ahmed, I., Jensen, C., Groce, A.: How hard does mutation analysis have to be, anyway? In: 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), pp. 216–227 (2015)
40.
Granda, M.F., Condori-Fernández, N., Vos, T.E.J., Pastor, O.: Mutation operators for UML class diagrams. In: International Conference on Advanced Information Systems Engineering (CAiSE), volume 9694 of LNCS. Springer, pp. 325–341 (2016)
41.
Groce, A., Holmes, J., Marinov, D., Shi, A., Zhang, L.: An extensible, regular-expression-based tool for multi-language mutant generation. In: International Conference on Software Engineering (ICSE). ACM, pp. 25–28 (2018)
42.
Guerra, E., Cuadrado, J.S., de Lara, J.: Towards effective mutation testing for ATL. In: ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS). IEEE, pp. 78–88 (2019)
43.
44.
Heidenreich, F., Johannes, J., Karol, S., Seifert, M., Wende, C.: Derivation and refinement of textual syntax for models. In: Proceedings of ECMDA-FA, volume 5562 of Lecture Notes in Computer Science. Springer, pp. 114–129 (2009)
45.
Heidenreich, F., Johannes, J., Reimann, J., Seifert, M.., Wende, C., Werner, C., Wilke, C., Assmann, U.: Model-driven modernisation of java programs with jamopp. In: Proceedings of Model-Driven Software Migration, volume 708 CEUR Workshop Proceedings, pp. 1–4 (2011)
46.
Hierons, R.M., Merayo, M.G.: Mutation testing from probabilistic and stochastic finite state machines. J. Syst. Softw. 82(11), 1804–1818 (2009)CrossRef
47.
Jia, Y., Harman, M.: MILU: a customizable, runtime-optimized higher order mutation testing tool for the full C language. In: Testing: Academic Industrial Conference—Practice and Research Techniques (TAICPART), pp. 94–98 (2008)
48.
Jia, Y., Harman, M.: Higher order mutation testing. Inf. Softw. Technol. 51(10), 1379–1393 (2009)CrossRef
49.
Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011)CrossRef
50.
Jouault, F., Allilaire, F., Bézivin, J., Kurtev, I.: ATL: a model transformation tool. Sci. Comput. Program. 72(1–2), 31–39 (2008)MathSciNetCrossRef
51.
Just, R.: The Major mutation framework: efficient and scalable mutation analysis for Java. In: International Symposium on Software Testing and Analysis (ISSTA). ACM, pp. 433–436 (2014). http://​mutation-testing.​org/​
52.
Kelly, S., Tolvanen, J.: Domain-Specific Modeling-Enabling Full Code Generation. Wiley, Hoboken (2008)CrossRef
53.
Khan, Y., Hassine, J.: Mutation operators for the Atlas Transformation Language. In: International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 43–52 (2013)
54.
Kim, S., Clark, J.A., McDermid, J.A.: Investigating the effectiveness of object-oriented testing strategies using the mutation method. Softw. Test. Verif. Reliab. 11(3), 207–225 (2001)CrossRef
55.
King, K.N., Offutt, A.J.: A Fortran language system for mutation-based software testing. Softw. Pract. Exp. 21(7), 685–718 (1991)CrossRef
56.
Kintis, M., Papadakis, M., Jia, Y., Malevris, N., Traon, Y.L., Harman, M.: Detecting trivial mutant equivalences via compiler optimisations. IEEE Trans. Softw. Eng. 44(4), 308–333 (2018)CrossRef
57.
Kintis, M., Papadakis, M., Papadopoulos, A., Valvis, E., Malevris, N., Traon, Y.L.: How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults. Empir. Softw. Eng. 23(4), 2426–2463 (2018)CrossRef
58.
Krenn, W., Schlick, R., Tiran, S., Aichernig, B., Jobstl, E., Brandl, H.: MoMut: UML model-based mutation testing for UML. In: IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 1–8 (2015)
59.
Kusano, M., Wang, C.: CCmutator: mutation generator for concurrency constructs in multithreaded C/C++ applications. In: IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Press, pp. 722–725 (2013)
60.
Laurent, T., Papadakis, M., Kintis, M., Henard, C., Traon, Y.L., Ventresque, A.: Assessing and improving the mutation testing practice of PIT. In: International Conference on Software Testing, Verification and Validation (ICST), pp. 430–435 (2017)
61.
Le, D., Alipour, M.A., Gopinath, R., Groce, A.: Mucheck: an extensible tool for mutation testing of haskell programs. In: International Symposium on Software Testing and Analysis (ISSTA). ACM, pp. 429–432 (2014)
62.
Le Goues, C., Nguyen, T., Forrest, S., Weimer, W.: Genprog: a generic method for automatic software repair. IEEE Trans. Softw. Eng. 38(1), 54–72 (2012)CrossRef
63.
Lee, S.C., Offutt, A.J.: Generating test cases for XML-based web component interactions using mutation analysis. In: International Symposium on Software Reliability Engineering (ISSRE), pp. 200–209 (2001)
64.
López, S., Alfonzo, G.A., Perez, O., Gonzalez, S., Montes, R.: A metamodel to carry out reverse engineering of C++ code into UML sequence diagrams. In: Electronics, Robotics and Automotive Mechanics Conference (CERMA), vol. 2, pp. 331–336 (2006)
65.
Ma, Y.S., Kwon, Y.R., Offutt, A.J.: Inter-class mutation operators for Java. In: International Symposium on Software Reliability Engineering (ISSRE), pp. 352–366 (2002)
66.
Ma, Y.-S., Offutt, J., Kwon, Y.R.: MuJava: a mutation system for Java. In: International Conference on Software Engineering (ICSE), pp. 827–830 (2006)
67.
Ma, Y.-S., Offutt, J., Kwon, Y.R.: Mujava: an automated class mutation system. Softw. Test. Verif. Reliab. 15(2), 97–133 (2005)CrossRef
68.
Mariya, F., Barkhas, D.: A comparative analysis of mutation testing tools for Java. In: IEEE East-West Design Test Symposium (EWDTS), pp. 1–3 (2016)
69.
Mirshokraie, S., Mesbah, A., Pattabiraman, K.: Guided mutation testing for JavaScript web applications. IEEE Trans. Softw. Eng. 41(5), 429–444 (2015)CrossRef
70.
71.
Mottu, J., Baudry, B., Traon, Y.L.: Mutation analysis testing for model transformations. In: European Conference on Model Driven Architecture—Foundations and Applications (ECMDA-FA), pp. 376–390 (2006)
72.
Mottu, J., Sen, S., Tisi, M., Cabot, J.: Static analysis of model transformations for effective test generation. In: International Symposium on Software Reliability Engineering (ISSRE), pp. 291–300 (2012)
73.
74.
75.
Offutt, A.J., Lee, A., Rothermel, G., Untch, R.H., Zapf, C.: An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Methodol. 5(2), 99–118 (1996)CrossRef
76.
Offutt, A.J., Voas, J., Payne, J.: Mutation operators for ADA. Technical report, Information and Software Systems Engineering, George Mason University (1996)
77.
Papadakis, M., Henard, C., Harman, M., Jia, Y., Le Traon, Y.: Threats to the validity of mutation-based test assessment. In: Proceedings of the 25th International Symposium on Software Testing and Analysis, pp. 354–365 (2016)
78.
Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y.L., Harman, M.: Chapter six - mutation testing advances: an analysis and survey, volume 112 of Advances in Computers. Elsevier, pp. 275–378 (2019)
79.
80.
Sánchez Cuadrado, J., Guerra, E., de Lara, J.: Static analysis of model transformations. IEEE Trans. Softw. Eng. 43(9), 868–897 (2017)CrossRef
81.
82.
Sen, S., Baudry, B.: Mutation-based model synthesis in model driven engineering. In: Workshop on Mutation Analysis (Mutation) (2006)
83.
Sen, S., Baudry, B., Mottu, J.-M.: Automatic model generation strategies for model transformation testing. In: Theory and Practice of Model Transformations. Springer, Berlin, pp. 148–164 (2009)
84.
Shin, D., Jee, E., Bae, D.: Comprehensive analysis of FBD test coverage criteria using mutants. Softw. Syst. Model. 15(3), 631–645 (2016)CrossRef
85.
86.
Srivatanakul, T., Clark, J.A., Stepney, S., Polack, F.: Challenging formal specifications by mutation: a CSP security example. In: Asia-Pacific Software Engineering Conference (APSEC), pp. 340–350 (2003)
87.
Steinberg, D., Budinsky, F., Paternostro, M., Merks, E.: EMF: Eclipse Modeling Framework, 2nd Edition. Addison-Wesley Professional (2008)
88.
Trakhtenbrot, M.: New mutations for evaluation of specification and implementation levels of adequacy in testing of statecharts models. In: Workshop on Mutation Analysis (Mutation), pp. 151–160 (2007)
89.
Troya, J., Bergmayr, A., Burgueño, L., Wimmer, M.: Towards systematic mutations for and with ATL model transformations. In: International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–10 (2015)
90.
Tuya, J., Cabal, M.J.S., de la Riva, C.: SQLMutation: a tool to generate mutants of SQL database queries. In: Workshop on Mutation Analysis (Mutation), p. 1 (2006)
91.
Tuya, J., Cabal, M.J.S., de la Riva, C.: Mutating database queries. Inf. Softw. Technol. 49(4), 398–417 (2007)CrossRef
92.
Voelter, M.: DSL Engineering—Designing, Implementing and Using Domain-Specific Languages. dslbook.org (2013)
93.
Wagelaar, D., Tisi, M., Cabot, J., Jouault, F.: Towards a general composition semantics for rule-based model transformation. In: ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS), volume 6981 of LNCS. Springer, pp. 623–637 (2011)
94.
Wu, F., Nanavati, J., Harman, M., Jia, Y., Krinke, J.: Memory mutation testing. Inf. Softw. Technol. 81, 97–111 (2017)CrossRef
95.
Yazdani, S.A., Amiri, M.J., Parsa, S., Koupaee, M.: Automatic test cases generation from business process models. Requir. Eng. 24(8), 119–132 (2018)
96.
Zhang, L., Gligoric, M., Marinov, D., Khurshid, S.: Operator-based and random mutant selection: better together. In: Proceedings of ASE. IEEE, pp. 92–102 (2013)
Metadata
Title
Wodel-Test: a model-based framework for language-independent mutation testing
Authors
Pablo Gómez-Abajo
Esther Guerra
Juan de Lara
Mercedes G. Merayo
Publication date
17-10-2020
Publisher
Springer Berlin Heidelberg
Published in
Software and Systems Modeling / Issue 3/2021
Print ISSN: 1619-1366
Electronic ISSN: 1619-1374
DOI
https://doi.org/10.1007/s10270-020-00827-0

Other articles of this Issue 3/2021

Software and Systems Modeling 3/2021 Go to the issue

Editorial

Modeling in the large: model libraries

Guest Editorial

Guest editorial to the theme section on Multi-Paradigm Modeling for Cyber-Physical Systems

Theme Section Paper

Predictions-on-chip: model-based training and automated deployment of machine learning models at runtime

Regular Paper

Implementing QVT-R via semantic interpretation in UML-RSDS

Theme Section Paper

Handling causality and schedulability when designing and prototyping cyber-physical systems

Regular Paper

Pragmatic reuse for DSML development

Premium Partner

    Image Credits