Sandro Fouché
ABSTRACT
The increasing complexity of configurable software systems has created a need for more intelligent sampling mechanisms to detect and characterize failure-inducing dependencies between configurations. Prior work - in idealized environments - has shown that test schedules based on a mathematical object, called a covering array, in combination with classification techniques, can meet this need. Applying this approach in practice, however, is tricky because testing time and resource availability are unpredictable, and because failure characteristics can change from release to release. With current approaches developers must set a key covering array parameter (its strength) based on estimated release times and failure characterizations. This will influence the outcome of their results.
In this paper we propose a new approach that incrementally builds covering array schedules. This approach begins at a low strength, and then iteratively increases strength as resources allow. At each stage previously tested configurations are reused, thus avoiding duplication of work. With the incremental approach developers need never commit to a specific covering array strength. Instead, by using progressively stronger covering array schedules, failures due to few configuration dependencies can be found and classified as soon and as cheaply as possibly. Additionally, it eliminates the risks of committing to overly strong test schedules.
We evaluate this new approach through a case study on three consecutive releases of MySQL, an open source database. Our results suggest that our approach is as good or better than previous approaches, costing less in most cases, and allowing greater flexibility in environments with unpredictable development constraints.
- D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton. The AETG system: an approach to testing based on combinatorial design. IEEE Trans. on SW Eng., 23(7):437--44, 1997. Google Scholar
Digital Library
- M. B. Cohen, C. J. Colbourn, P. B. Gibbons, and W. B. Mugridge. Constructing test suites for interaction testing. In Proc. of the Int'l Conf. on SW Eng., (ICSE), pages 38--44, 2003. Google ScholarDigital Library
- M. B. Cohen, C. J. Colbourn, and A. C. H. Ling. Augmenting simulated annealing to build interaction test suites. In Proc. of Int'l Symp. on SW Reliability Eng. (ISSRE), pages 394--405, November 2003. Google ScholarDigital Library
- M. B. Cohen, M. B. Dwyer, and J. Shi. Interaction testing of highly-configurable systems in the presence of constraints. In Int'l Symp. on SW Testing and Analysis, (ISSTA), pages 129--139, July 2007. Google ScholarDigital Library
- M. B. Cohen, J. Snyder, and G. Rothermel. Testing across configurations: Implications for combinatorial testing. In Workshop on Advances in Model-based SW Testing, pages 1--9, November 2006.Google ScholarDigital Library
- S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, C. M. Lott, G. C. Patton, and B. M. Horowitz Model-based testing in practice. In Proc. of the Int'l Conf. on SW Eng., (ICSE), pages 285--294, 1999. Google ScholarDigital Library
- I. S. Dunietz, W. K. Ehrlich, B. D. Szablak, C. L. Mallows, and A. Iannino. Applying design of experiments to software testing. In Proc. of the Int'l Conf. on SW Eng., (ICSE), pages 205--215, 1997. Google ScholarDigital Library
- S. Fouche, M. B. Cohen, and A. Porter. Towards incremental adaptive covering arrays. In Proc. of the 14th ACM SIGSOFT Symp. on Foundations of SW Eng., pages 557--560, 2007. Google ScholarDigital Library
- J. -M. Kim and A. Porter. A history-based test prioritization technique for regression testing in resource constrained environments. In Proc. of the 24th Int'l Conf. on SW Eng., (ICSE), pages 119--129, New York, NY, USA, 2002. ACM Press. Google ScholarDigital Library
- D. Kuhn, D. R. Wallace, and A. M. Gallo. Sw fault interactions and implications for software testing IEEE Trans. on SW Eng., 30(6):418--421, 2004. Google ScholarDigital Library
- B. Liblit, A. Aiken, Z. Zheng, and M. Jordan. Bug isolation via remote program sampling. In Conf. on Programming Language Design and Implementation, pages 141--154. ACM, June 2003. Google ScholarDigital Library
- A. Memon, A. Porter, C. Yilmaz, A. Nagarajan, D. C. Schmidt, and B. N. rajan. Skoll: Distributed continuous quality assurance. In Proc. of the Int'l Conf. on SW Eng., (ICSE ), pages 459--468, 2004. Google ScholarDigital Library
- MySQL, 2006. http://www.mysql.com.Google Scholar
- K. C. Tai and L. Yu. A test generation strategy for pairwise testing. IEEE Trans. on SW Eng., 28(1):109--111, 2002. Google ScholarDigital Library
- I. H. Witten and E. Frank. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, 1999. Google ScholarDigital Library
- C. Yilmaz, M. B. Cohen, and A. Porter. Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans. on SW Eng., 31(1):20--34, Jan 2006. Google ScholarDigital Library
- A. Zeller and R.Hildebrandt. Simplifying and isolating failure-inducing input. IEEE Trans. on SW Eng., 28(2):183--200, 2002. Google ScholarDigital Library
Index Terms
- Incremental covering array failure characterization in large configuration spaces
Recommendations
An algorithm for generating t-wise covering arrays from large feature models
SPLC '12: Proceedings of the 16th International Software Product Line Conference - Volume 1A scalable approach for software product line testing is required due to the size and complexity of industrial product lines. In this paper, we present a specialized algorithm (called ICPL) for generating covering arrays from feature models. ICPL makes ...
Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces
Many modern software systems are designed to be highly configurable so they can run on and be optimized for a wide variety of platforms and usage scenarios. Testing such systems is difficult because, in effect, you are testing a multitude of systems, ...
Covering arrays for efficient fault characterization in complex configuration spaces
ISSTA '04: Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysisTesting systems with large configurations spaces that change often is a challenging problem. The cost and complexity of QA explodes because often there isn't just one system, but a multitude of related systems. Bugs may appear in certain configurations, ...
Comments