Mustafa Al-Hajjaji
Thomas Thüm
ABSTRACT
Mutation testing has emerged as one of the most promising techniques to increase the quality of software-intensive systems. In mutation testing, random faults based on a predefined set of mutation operators are automatically injected into a program to evaluate test suites. The effectiveness of mutation testing strongly depends on the representativeness of the mutation operators. Existing operators are not sufficient to represent typical faults caused by variability. Thus, we propose a set of mutation operators for software with preprocessor-based variability. We derive the operators systematically based on a taxonomy of variability-related faults and evaluate them by investigating their applicability to real-world faults that have previously been identified in research on configurable software systems. Our goal is to leverage mutation testing to highly-variable software for its practical application and to enable empirical evaluation of testing techniques.
- I. Abal, C. Brabrand, and A. Wasowski. 42 Variability Bugs in the Linux Kernel: A Qualitative Analysis. In Proc. Int'l Conf. Automated Software Engineering (ASE), pages 421--432. ACM, 2014. Google Scholar
Digital Library
- M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake. Similarity-Based Prioritization in Software Product-Line Testing. In Proc. Int'l Software Product Line Conf. (SPLC), pages 197--206. ACM, 2014. Google ScholarDigital Library
- P. Anbalagan and T. Xie. Automated Generation of Pointcut Mutants for Testing Pointcuts in AspectJ Programs. In Proc. Int'l Symposium Software Reliability Engineering (ISSRE), pages 239--248, 2008. Google ScholarDigital Library
- S. Apel, D. Batory, C. Kästner, and G. Saake. Feature-Oriented Software Product Lines: Concepts and Implementation. Springer, 2013. Google ScholarCross Ref
- P. Arcaini, A. Gargantini, and P. Vavassori. Generating Tests for Detecting Faults in Feature Models. In Proc. Int'l Conf. Software Testing, Verification and Validation (ICST), pages 1--10, 2015.Google ScholarCross Ref
- S. S. Batth, E. R. Vieira, A. Cavalli, and M. U. Uyar. Specification of Timed EFSM Fault Models in SDL. In Proc. IFIP Int'l Conf. Formal Methods for Open Object-Based Distributed Systems (FMOODS), pages 50--65. Springer, 2007. Google ScholarDigital Library
- P. Borba, L. Teixeira, and R. Gheyi. A Theory of Software Product Line Refinement. Theoretical Computer Science, 455(0):2--30, 2012. Google ScholarDigital Library
- P. Chevalley. Applying mutation analysis for object-oriented programs using a reflective approach. In Proc. Asia-Pacific Software Engineering Conference (APSEC), pages 267--270, 2001. Google ScholarDigital Library
- T. R. Devine, K. Goseva-Popstajanova, S. Krishnan, R. R. Lutz, and J. J. Li. An Empirical Study of Pre-Release Software Faults in an Industrial Product Line. In Proc. Int'l Conf. Software Testing, Verification and Validation (ICST), pages 181--190, 2012. Google ScholarDigital Library
- F. Ensan, E. Bagheri, and D. Gasevic. Evolutionary Search-Based Test Generation for Software Product Line Feature Models. In Proc. Int'l Conf. Advanced Information Systems Engineering (CAiSE), volume 7328, pages 613--628. Springer, 2012. Google ScholarDigital Library
- G. Fraser and F. Wotawa. Mutant Minimization for Model-Checker Based Test-Case Generation. In in Proc. of Workshop on Mutation Analysis, Published with Proc. of Testing: Academic and Industrial Conference Practice and Research Techniques, pages 161--168, 2007. Google ScholarDigital Library
- B. Garvin and M. Cohen. Feature Interaction Faults Revisited: An Exploratory Study. In Proc. Int'l Symposium Software Reliability Engineering (ISSRE), pages 90--99, 2011. Google ScholarDigital Library
- C. Henard, M. Papadakis, and Y. Le Traon. Mutation-Based Generation of Software Product Line Test Configurations. In C. Le Goues and S. Yoo, editors, Search-Based Software Engineering, volume 8636 of Lecture Notes in Computer Science, pages 92--106. Springer International Publishing, 2014.Google ScholarCross Ref
- Y. Jia and M. Harman. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Software Engineering (TSE), 37(5):649--678, 2011. Google ScholarDigital Library
- K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak, and A. S. Peterson. Feature-Oriented Domain Analysis (FODA) Feasibility Study. Technical Report CMU/SEI-90-TR-21, Software Engineering Institute, 1990.Google Scholar
- C. Kästner, P. G. Giarrusso, T. Rendel, S. Erdweg, K. Ostermann, and T. Berger. Variability-Aware Parsing in the Presence of Lexical Macros and Conditional Compilation. In Proc. Conf. Object-Oriented Programming, Systems, Languages and Applications (OOPSLA), pages 805--824. ACM, 2011. Google ScholarDigital Library
- S. Krishnan, R. R. Lutz, and K. Goševa-Popstojanova. Empirical Evaluation of Reliability Improvement in an Evolving Software Product Line. In Proc. of Working Conf. on Mining Software Repositories (MSR), pages 103--112. ACM, 2011. Google ScholarDigital Library
- S. Krishnan, C. Strasburg, R. R. Lutz, and K. Goševa-Popstojanova. Are Change Metrics Good Predictors for an Evolving Software Product Line? In Proc. of Int'l Conf. on Predictive Models in Software Engineering (Promise), pages 1--10. ACM, 2011. Google ScholarDigital Library
- H. Lackner and M. Schmidt. Towards the Assessment of Software Product Line Tests: A Mutation System for Variable Systems. In Proc. Workshop Software Product Line Analysis Tools (SPLat), pages 62--69. ACM, 2014. Google ScholarDigital Library
- M. Papadakis, C. Henard, and Y. Le Traon. Sampling Program Inputs with Mutation Analysis: Going Beyond Combinatorial Interaction Testing. In Proc. Int'l Conf. Software Testing, Verification and Validation (ICST), pages 1--10, 2014. Google ScholarDigital Library
- D. Reuling, J. Bürdek, S. Rotärmel, M. Lochau, and U. Kelter. Fault-Based Product-Line Testing: Effective Sample Generation Based on Feature-Diagram Mutation. In Proc. Int'l Software Product Line Conf. (SPLC), pages 131--140. ACM, 2015. Google ScholarDigital Library
- H. A. Richard, R. A. Demillo, B. Hathaway, W. Hsu, W. Hsu, E. Krauser, R. Martin, A. P. Mathur, and E. H. Spafford. Design Of Mutant Operators For The C Programming Language. Technical Report SERC-TR-41-P, Purdue University, West Lafayette, Indiana, 1989.Google Scholar
- A. Sánchez, S. Segura, J. Parejo, and A. Ruiz-Cortés. Variability Testing in the Wild: The Drupal Case Study. Software and System Modeling, pages 1--22, 2015.Google Scholar
- A. B. Sánchez, S. Segura, and A. Ruiz-Cortés. A Comparison of Test Case Prioritization Criteria for Software Product Lines. In Proc. Int'l Conf. Software Testing, Verification and Validation (ICST), pages 41--50. IEEE, 2014. Google ScholarDigital Library
- T. Thüm, C. Kästner, S. Erdweg, and N. Siegmund. Abstract Features in Feature Modeling. In Proc. Int'l Software Product Line Conf. (SPLC), pages 191--200. IEEE, 2011. Google ScholarDigital Library
- T. Thüm, J. Meinicke, F. Benduhn, M. Hentschel, A. von Rhein, and G. Saake. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines. In Proc. Int'l Software Product Line Conf. (SPLC), pages 177--186. ACM, 2014. Google ScholarDigital Library
Index Terms
- Mutation Operators for Preprocessor-Based Variability
Recommendations
Mutation Operators for Actor Systems
ICSTW '10: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation WorkshopsMutation testing is a well known technique for estimating and improving the quality of test suites. Given a test suite T for a system S, mutation testing systematically creates mutants of S and executes T to measure how many mutants T detects. If T does ...
Mutation Operators for Large Scale Data Processing Programs in Spark
Advanced Information Systems EngineeringAbstractThis paper proposes a mutation testing approach for big data processing programs that follow a data flow model, such as those implemented on top of Apache Spark. Mutation testing is a fault-based technique that relies on fault simulation by ...
Mutation Testing for Aspect-Oriented Programs
ICST '08: Proceedings of the 2008 International Conference on Software Testing, Verification, and ValidationMutation testing has been shown to be one of the strongest testing criteria for the evaluation of both programs and test suites. Comprehensive sets of mutants require strong test sets to achieve acceptable testing coverage. Moreover, mutation operators ...
Comments