Xavier Devroey
Pierre-Yves Schobbens
ABSTRACT
Model-based mutation analysis is a powerful but expensive testing technique. We tackle its high computation cost by proposing an optimization technique that drastically speeds up the mutant execution process. Central to this approach is the Featured Mutant Model, a modelling framework for mutation analysis inspired by the software product line paradigm. It uses behavioural variability models, viz., Featured Transition Systems, which enable the optimized generation, configuration and execution of mutants. We provide results, based on models with thousands of transitions, suggesting that our technique is fast and scalable. We found that it outperforms previous approaches by several orders of magnitude and that it makes higher-order mutation practically applicable.
- B. K. Aichernig, J. Auer, E. Jöbstl, R. Korosec, W. Krenn, R. Schlick, and B. V. Schmidt. Model-based mutation testing of an industrial measurement device. In Tests and Proofs, volume 8570 of LNCS, pages 1--19. Springer, 2014.Google Scholar
Cross Ref
- B. K. Aichernig, H. Brandl, E. Jöbstl, W. Krenn, R. Schlick, and S. Tiran. Killing strategies for model-based mutation testing. Software Testing, Verification and Reliability, 2014. Google ScholarDigital Library
- B. K. Aichernig, E. Jöbstl, and S. Tiran. Model-based mutation testing via symbolic refinement checking. Science of Computer Programming, 97:383--404, Jan. 2015.Google ScholarDigital Library
- P. E. Ammann, P. E. Black, and W. Majurski. Using model checking to generate tests from specifications. In ICFEM, pages 46--54. IEEE, 1998. Google ScholarDigital Library
- J. H. Andrews, L. C. Briand, Y. Labiche, and A. S. Namin. Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria. Software Engineering, IEEE Transactions on, 32(8):608--624, 2006. Google ScholarDigital Library
- C. Baier and J. Katoen. Principles of model checking. MIT Press, 2008. Google ScholarDigital Library
- R. Baker and I. Habli. An empirical evaluation of mutation testing for improving the test quality of safety-critical software. IEEE Transactions on Software Engineering, 39(6):787--805, 2013. Google ScholarDigital Library
- F. Belli and M. Beyazit. Event-Based Mutation Testing vs. State-Based Mutation Testing - An Experimental Comparison. In COMPSAC, pages 650--655. IEEE, July 2011. Google ScholarDigital Library
- F. Belli, C. J. Budnik, and W. E. Wong. Basic Operations for Generating Behavioral Mutants. In Proceedings of the Second Workshop on Mutation Analysis. IEEE, Nov. 2006. Google ScholarDigital Library
- P. E. Black, V. Okun, and Y. Yesha. Mutation operators for specifications. In ASE. IEEE, 2000. Google ScholarDigital Library
- L. C. Briand, M. D. Penta, and Y. Labiche. Assessing and improving state-based class testing: A series of experiments. IEEE Transactions on Software Engineering, 30(11):770--793, 2004. Google ScholarDigital Library
- T. A. Budd. Mutation Analysis of Program Test Data. PhD thesis, Yale University, New Haven, CT, USA, 1980. AAI8025191. Google ScholarDigital Library
- T. A. Budd and A. S. Gopal. Program testing by specification mutation. Computer Languages, 10(1):63--73, Jan. 1985. Google ScholarDigital Library
- A. Classen. Modelling with FTS: a Collection of Illustrative Examples. Technical Report P-CS-TR SPLMC-00000001, PReCISE Research Center, University of Namur, Namur, Belgium, 2010.Google Scholar
- A. Classen, M. Cordy, P.-Y. Schobbens, P. Heymans, A. Legay, and J.-F. Raskin. Featured Transition Systems: Foundations for Verifying Variability-Intensive Systems and Their Application to LTL Model Checking. IEEE Transactions on Software Engineering, 39(8):1069--1089, Aug. 2013. Google ScholarDigital Library
- M. E. Delamaro and J. C. Maldonado. Proteum/im 2.0: An integrated mutation testing environment. In W. E. Wong, editor, Mutation Testing for the New Century, pages 91--101. Kluwer Academic Publishers, Norwell, MA, USA, 2001. Google ScholarDigital Library
- X. Devroey and G. Perrouin. Variability Intensive system Behavioural teSting (VIBeS) v. 1.0.1. https://projects.info.unamur.be/vibes/, 2015.Google Scholar
- X. Devroey, G. Perrouin, M. Cordy, M. Papadakis, A. Legay, and P.-Y. Schobbens. A Variability Perspective of Mutation Analysis. In FSE, pages 841--844. ACM, 2014. Google ScholarDigital Library
- X. Devroey, G. Perrouin, M. Cordy, H. Samih, A. Legay, P.-Y. Schobbens, and P. Heymans. Statistical prioritization for software product line testing: an experience report. Software & Systems Modeling, pages 1--19, 2015.Google Scholar
- X. Devroey, G. Perrouin, A. Legay, M. Cordy, P.-Y. Schobbens, and P. Heymans. Coverage Criteria for Behavioural Testing of Software Product Lines. In ISoLA, volume 8802 of LNCS, pages 336--350. Springer, 2014. Google ScholarDigital Library
- S. Fabbri, J. C. Maldonado, and M. E. Delamaro. Proteum/FSM: a tool to support finite state machine validation based on mutation testing. In SCCC, pages 96--104. IEEE, 1999. Google ScholarDigital Library
- S. Fabbri, J. C. Maldonado, T. Sugeta, and P. C. Masiero. Mutation testing applied to validate specifications based on statecharts. In ISSRE, pages 210--219. IEEE, 1999. Google ScholarDigital Library
- G. Fraser and A. Arcuri. Achieving scalable mutation-based generation of whole test suites. Empirical Software Engineering, pages 1--30, 2014. Google ScholarDigital Library
- M. Gligoric, A. Groce, C. Zhang, R. Sharma, M. A. Alipour, and D. Marinov. Comparing non-adequate test suites using coverage criteria. In ISSTA, pages 302--313. ACM, 2013. Google ScholarDigital Library
- M. Harman, Y. Jia, P. Reales Mateo, and M. Polo. Angels and monsters: An empirical investigation of potential test effectiveness and efficiency improvement from strongly subsuming higher order mutation. In ASE, pages 397--408. ACM, 2014. Google ScholarDigital Library
- C. Henard, M. Papadakis, G. Perrouin, J. Klein, and Y. Le Traon. Assessing software product line testing via model-based mutation: An application to similarity testing. In ICST, pages 188--197. IEEE, 2013. Google ScholarDigital Library
- R. M. Hierons and M. G. Merayo. Mutation testing from probabilistic and stochastic finite state machines. Journal of Systems and Software, 82(11):1804--1818, 2009. Google ScholarDigital Library
- W. E. Howden. Reliability of the path analysis testing strategy. IEEE Transactions on Software Engineering, 2(3):208--215, 1976. Google ScholarDigital Library
- D. Jackson and M. R. Woodward. Parallel firm mutation of java programs. In W. E. Wong, editor, Mutation Testing for the New Century, pages 55--61. Kluwer Academic Publishers, Norwell, MA, USA, 2001. Google ScholarDigital Library
- Y. Jia. Higher order mutation testing. PhD thesis, University College London (University of London), 2013.Google Scholar
- Y. Jia and M. Harman. An Analysis and Survey of the Development of Mutation Testing. Software Engineering, IEEE Transactions on, 37(5):649--678, Sept. 2011. Google ScholarDigital Library
- R. Just, M. D. Ernst, and G. Fraser. Efficient mutation analysis by propagating and partitioning infected execution states. In ISSTA, pages 315--326. ACM, 2014. Google ScholarDigital Library
- R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Are Mutants a Valid Substitute for Real Faults in Software Testing? In FSE, pages 654--665. ACM, 2014. Google ScholarDigital Library
- K. Kang, S. Cohen, J. Hess, W. Novak, and A. S. Peterson. Feature-Oriented Domain Analysis (FODA) Feasibility Study. Technical report, Carnegie-Mellon University, Software Engineering Institute, 1990.Google ScholarCross Ref
- K. Kapoor and J. P. Bowen. Ordering mutants to minimise test effort in mutation testing. In FATES 2004, volume 3395 of LNCS, pages 195--209. Springer, 2004. Google ScholarDigital Library
- C. H. P. Kim, S. Khurshid, and D. S. Batory. Shared execution for efficiently testing product lines. In ISSRE, pages 221--230. IEEE, 2012. Google ScholarDigital Library
- C. H. P. Kim, D. Marinov, S. Khurshid, D. S. Batory, S. Souto, P. Barros, and M. d'Amorim. Splat: lightweight dynamic analysis for reducing combinatorics in testing configurable systems. In ESEC/FSE, 2013, pages 257--267. ACM, 2013. Google ScholarDigital Library
- W. Krenn and R. Schlick. Mutation-driven Test Case Generation Using Short-lived Concurrent Mutants -- First Results. Technical report, jan 2016.Google Scholar
- W. Krenn, R. Schlick, S. Tiran, B. Aichernig, E. Jobstl, and H. Brandl. MoMut::UML Model-Based Mutation Testing for UML. In ICST, pages 1--8. IEEE, April 2015.Google ScholarCross Ref
- H. Lackner and M. Schmidt. Towards the assessment of software product line tests: A mutation system for variable systems. In SPLC Workshops, SPLC '14, pages 62--69. ACM, 2014. Google ScholarDigital Library
- H. Lackner, M. Thomas, F. Wartenberg, and S. Weißleder. Model-based test design of product lines: Raising test design to the product line level. In ICST, pages 51--60, Cleveland, Ohio, USA, 2014. IEEE. Google ScholarDigital Library
- W. B. Langdon and M. Harman. Optimising existing software with genetic programming. IEEE Transactions on Evolutionary Computation, 19(1):118--135, Feb 2015.Google ScholarDigital Library
- C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer. GenProg: A generic method for automatic software repair. IEEE Transactions on Software Engineering, 38(1):54--72, Jan.-Feb. 2012. Google ScholarDigital Library
- N. Li and J. Offutt. A test automation language framework for behavioral models. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), number 1, pages 1--10. IEEE, apr 2015.Google ScholarCross Ref
- Y. Ma, J. Offutt, and Y. R. Kwon. Mujava: an automated class mutation system. Software Testing, Verification and Reliability, 15(2):97--133, 2005. Google ScholarDigital Library
- P. R. Mateo and M. P. Usaola. Reducing mutation costs through uncovered mutants. Software Testing, Verification and Reliability, 25(5-7):464--489, 2014. Google ScholarDigital Library
- J. Mottu, B. Baudry, and Y. Le Traon. Mutation analysis testing for model transformations. In ECMDA-FA, volume 4066 of LNCS, pages 376--390. Springer, 2006. Google ScholarDigital Library
- H. V. Nguyen, C. Kästner, and T. N. Nguyen. Exploring variability-aware execution for testing plugin-based web applications. In ICSE, pages 907--918. ACM, 2014. Google ScholarDigital Library
- A. J. Offutt, S. Liu, A. Abdurazik, and P. Ammann. Generating test data from state-based specifications. Softwre Testing, Verification and Reliability, 13(1):25--53, 2003.Google ScholarCross Ref
- J. Offutt. A mutation carol: Past, present and future. Information and Software Technology, 53(10):1098--1107, Oct. 2011.Google ScholarCross Ref
- M. Papadakis, C. Henard, and Y. Le Traon. Sampling program inputs with mutation analysis: Going beyond combinatorial interaction testing. In ICST, pages 1--10. IEEE, 2014. Google ScholarDigital Library
- M. Papadakis, Y. Jia, M. Harman, and Y. Le Traon. Trivial compiler equivalence: A large scale empirical study of a simple fast and effective equivalent mutant detection technique. In ICSE, pages 936--946. IEEE, 2015. Google ScholarDigital Library
- M. Papadakis and Y. Le Traon. Metallaxis-FL: mutation-based fault localization. Software Testing, Verification and Reliability, 2013. Google ScholarDigital Library
- M. Papadakis and N. Malevris. Automatic mutation test case generation via dynamic symbolic execution. In ISSRE, pages 121--130. IEEE, 2010. Google ScholarDigital Library
- M. Papadakis and N. Malevris. Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing. Software Quality Journal, 19(4):691--723, 2011. Google ScholarDigital Library
- R. Pelánek. Typical Structural Properties of State Spaces. In S. Graf and L. Mounier, editors, Model Checking Software, volume 2989 of LNCS, pages 5--22. Springer, 2004.Google Scholar
- R. Pelánek. Properties of state spaces and their applications. International Journal on Software Tools for Technology Transfer, 10(5):443--454, 2008.Google ScholarDigital Library
- P.-Y. Schobbens, P. Heymans, J.-C. Trigaux, and Y. Bontemps. Generic semantics of feature diagrams. Computer Networks, 51(2):456--479, 2007. Google ScholarDigital Library
- S. E. Sprenkle, L. L. Pollock, and L. M. Simko. Configuring effective navigation models and abstract test cases for web applications by analysing user behaviour. Software Testing, Verification and Reliability, 23(6):439--464, 2013.Google ScholarCross Ref
- R. H. Untch, A. J. Offutt, and M. J. Harrold. Mutation analysis using mutant schemata. In ISSTA, pages 139--148, 1993. Google ScholarDigital Library
- M. Utting and B. Legeard. Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann, 2007. Google ScholarDigital Library
- J. M. Voas and G. McGraw. Software Fault Injection: Inoculating Programs Against Errors. John Wiley & Sons, Inc., 1997. Google ScholarDigital Library
- M. R. Woodward. Errors in algebraic specifications and an experimental mutation testing tool. Software Engineering Journal, 8(4):221--224, July 1993.Google ScholarCross Ref
Index Terms
- Featured model-based mutation analysis
Recommendations
A variability perspective of mutation analysis
FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software EngineeringMutation testing is an effective technique for either improving or generating fault-finding test suites. It creates defective or incorrect program artifacts of the program under test and evaluates the ability of test suites to reveal them. Despite ...
On guiding the augmentation of an automated test suite via mutation analysis
Mutation testing has traditionally been used as a defect injection technique to assess the effectiveness of a test suite as represented by a "mutation score." Recently, mutation testing tools have become more efficient, and industrial usage of mutation ...
Should software testers use mutation analysis to augment a test set?
Mutation testing has historically been used to assess the fault-finding effectiveness of a test suite or other verification technique. Mutation analysis, rather, entails augmenting a test suite to detect all killable mutants. Concerns about the time ...
Comments