Xavier Devroey
Pierre-Yves Schobbens
ABSTRACT
Dissimilar test cases have been proven to be effective to reveal faults in software systems. In the Software Product Line (SPL) context, this criterion has been applied successfully to mimic combinatorial interaction testing in an efficient and scalable manner by selecting and prioritising most dissimilar configurations of feature models using evolutionary algorithms. In this paper, we extend dissimilarity to behavioural SPL models (FTS) in a search-based approach, and evaluate its effectiveness in terms of product and fault coverage. We investigate different distances as well as as single-objective algorithms, (dissimilarity on actions, random, all-actions). Our results on four case studies show the relevance of dissimilarity-based test generation for behavioural SPL models, especially on the largest case-study where no other approach can match it.
- M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake. Similarity-based Prioritization in Software Product-line Testing. In SPLC - Vol. 1, pages 197--206, 2014. Google Scholar
Digital Library
- J. H. Andrews, L. C. Briand, Y. Labiche, and A. S. Namin. Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria. IEEE TSE, 32:608--624, 2006. Google ScholarDigital Library
- H. Beohar and M. R. Mousavi. Input-output Conformance Testing Based on Featured Transition Systems. In SAC, pages 1272--1278, 2014. Google ScholarDigital Library
- D. Brockhoff, T. Friedrich, and F. Neumann. Analyzing hypervolume indicator based algorithms. In Parallel Problem Solving from Nature -- PPSN X, pages 651--660. 2008. Google ScholarDigital Library
- E. G. Cartaxo, P. D. L. Machado, and F. G. O. Neto. On the use of a similarity function for test case selection in the context of model-based testing. STVR, 21(2):75--100, 2011. Google ScholarDigital Library
- T. Y. Chen, F.-C. Kuo, R. G. Merkel, and T. Tse. Adaptive Random Testing: The ART of test case diversity. JSS, 83:60--66, 2010. Google ScholarDigital Library
- A. Classen. Modelling with FTS: a Collection of Illustrative Examples. Technical Report P-CS-TR SPLMC-00000001, PReCISE Research Center, University of Namur, 2010.Google Scholar
- A. Classen, M. Cordy, P.-Y. Schobbens, P. Heymans, A. Legay, and J.-F. J.-F. Raskin. Featured Transition Systems: Foundations for Verifying Variability-Intensive Systems and Their Application to LTL Model Checking. IEEE TSE, 39:1069--1089, 2013. Google ScholarDigital Library
- M. Cordy, A. Classen, P. Heymans, P.-Y. Schobbens, and A. Legay. ProVeLines: A Product Line of Verifiers for Software Product Lines. In SPLC - Vol. 2, pages 141--146, 2013. Google ScholarDigital Library
- P. A. da Mota Silveira Neto, I. do Carmo Machado, J. D. McGregor, E. S. de Almeida, and S. R. de Lemos Meira. A systematic mapping study of software product lines testing. IST, 53(5):407--423, 2011. Google ScholarDigital Library
- X. Devroey and G. Perrouin. Variability Intensive system Behavioural teSting framework (VIBeS), 2014. version 1.1.2.Google Scholar
- X. Devroey, G. Perrouin, M. Cordy, M. Papadakis, A. Legay, and P.-Y. Schobbens. A Variability Perspective of Mutation Analysis. In FSE, pages 841--844, 2014. Google ScholarDigital Library
- X. Devroey, G. Perrouin, M. Cordy, P.-y. Schobbens, A. Legay, and P. Heymans. Towards Statistical Prioritization for Software Product Lines Testing. In VAMOS, 2014. Google ScholarDigital Library
- X. Devroey, G. Perrouin, A. Legay, M. Cordy, P.-Y. Schobbens, and P. Heymans. Coverage Criteria for Behavioural Testing of Software Product Lines. In ISoLA, pages 336--350, 2014. Google ScholarDigital Library
- X. Devroey, G. Perrouin, and P.-Y. Schobbens. Abstract Test Case Generation for Behavioural Testing of Software Product Lines. In SPLC - Vol. 2, pages 86--93, 2014. Google ScholarDigital Library
- S. Droste, T. Jansen, and I. Wegener. On the analysis of the (1+1) evolutionary algorithm. Theoretical Computer Science, 276(1--2):51--81, apr 2002. Google ScholarDigital Library
- E. Engström and P. Runeson. Software product line testing -- A systematic mapping study. IST, 53:2--13, 2011. Google ScholarDigital Library
- D. Gusfield. Algorithms on strings, trees and sequences: computer science and computational biology. 1997. Google ScholarCross Ref
- H. Hemmati, A. Arcuri, and L. Briand. Achieving scalable model-based testing through test case diversity. ACM TOSEM, 22:1--42, 2013. Google ScholarDigital Library
- H. Hemmati and L. Briand. An Industrial Investigation of Similarity Measures for Model-Based Test Case Selection. In ISSRE, pages 141--150, 2010. Google ScholarDigital Library
- C. Henard, M. Papadakis, M. Harman, and Y. L. Traon. Combining multi-objective search and constraint solving for configuring large software product lines. In ICSE, pages 517--528, 2015. Google ScholarDigital Library
- C. Henard, M. Papadakis, G. Perrouin, J. Klein, P. Heymans, and Y. Le Traon. Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines. IEEE TSE, 40:650--670, 2014. Google ScholarDigital Library
- C. Henard, M. Papadakis, G. Perrouin, J. Klein, Y. Le Traon, and Y. L. Traon. Assessing Software Product Line Testing Via Model-Based Mutation: An Application to Similarity Testing. ICSTW, pages 188--197, 2013. Google ScholarDigital Library
- P. Jaccard. Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bulletin del la Société Vaudoise des Sciences Naturelles, 37:547--579, 1901.Google Scholar
- M. F. Johansen, b. Haugen, and F. Fleurey. An Algorithm for Generating T-wise Covering Arrays from Large Feature Models. In SPLC - Vol. 1, pages 46--55, 2012. Google ScholarDigital Library
- K. Kang, S. Cohen, J. Hess, W. Novak, and A. S. Peterson. Feature-Oriented Domain Analysis (FODA) Feasibility Study. Technical report, Carnegie-Mellon University, Software Engineering Institute, 1990.Google ScholarCross Ref
- H. Lackner and M. Schmidt. Towards the Assessment of Software Product Line Tests: A Mutation System for Variable Systems. In SPLC - Vol. 2, pages 62--69, 2014. Google ScholarDigital Library
- N. Li and J. Offutt. A test automation language framework for behavioral models. In ICSTW, pages 1--10, 2015.Google ScholarCross Ref
- M. Lochau, S. Oster, U. Goltz, and A. Schürr. Model-based pairwise testing for feature interaction coverage in software product line engineering. SQJ, 20:567--604, 2011. Google ScholarDigital Library
- M. Lochau, I. Schaefer, J. Kamischke, and S. Lity. Incremental Model-Based Testing of Delta-Oriented Software Product Lines. In TAP, pages 67--82, 2012. Google ScholarDigital Library
- A. P. Mathur. Foundations of software testing. 2008. Google ScholarDigital Library
- D. Mondal, H. Hemmati, and S. Durocher. Exploring Test Suite Diversification and Code Coverage in Multi-Objective Test Case Selection. In ICST, pages 1--10, 2015.Google ScholarCross Ref
- M. Papadakis, C. Henard, and Y. le Traon. Sampling Program Inputs with Mutation Analysis: Going Beyond Combinatorial Interaction Testing. In ICST, pages 1--10, 2014. Google ScholarDigital Library
- G. Perrouin, S. Oster, S. Sen, J. Klein, B. Baudry, and Y. le Traon. Pairwise testing for software product lines: Comparison of two approaches. SQJ, 20:605--643, 2011. Google ScholarDigital Library
- S. C. Pinto Ferraz Fabbri, M. E. Delamaro, J. C. Maldonado, and P. C. Masiero. Mutation analysis testing for finite state machines. In ISSRE, pages 220--229, 1994.Google ScholarCross Ref
- M. Steffens, S. Oster, M. Lochau, and T. Fogdal. Industrial Evaluation of Pairwise SPL Testing with MoSo-PoLiTe. In VAMOS, pages 55--62, 2012. Google ScholarDigital Library
- J. Tretmans. Model based testing with labelled transition systems. Formal methods and testing, pages 1--38, 2008. Google ScholarDigital Library
Recommendations
Covering SPL Behaviour with Sampled Configurations: An Initial Assessment
VaMoS '15: Proceedings of the 9th International Workshop on Variability Modelling of Software-Intensive SystemsStructural approaches to Software Product Lines (SPL) testing (such as pairwise testing) have gained momentum as they are able to scale to larger SPLs described as feature diagrams (FD). However, these methods are agnostic with respect to behaviour: the ...
A survey on software product line testing
SPLC '12: Proceedings of the 16th International Software Product Line Conference - Volume 1Software product line (SPL) testing consists of two separate but closely related test engineering activities: domain testing and application testing. Various software product line testing approaches have been developed over the last decade, and surveys ...
Statistical prioritization for software product line testing: an experience report
Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature ...
Comments