Myra B. Cohen
Abstract
User configurable software systems allow users to customize functionality at run time. In essence, each such system consists of a family of potentially thousands or millions of program instantiations. Testing methods cannot test all of these configurations, therefore some sampling mechanism must be applied. A common approach to providing such a mechanism has been to use combinatorial interaction testing. To date, however, little work has been done to quantify the effects of different configurations on a test suites' operation and effectiveness. In this paper we present a case study that investigates the effects of changing configurations on two types of test suites. Our results show that test coverage and fault detection effectiveness do not vary much across configurations for entire test suites; however, for individual test cases and certain types of faults, configurations matter.
- R. Brownlie, J. Prowse, and M. S. Phadke. Robust testing of AT&T PMX/StarMAIL using OATS. AT&T Technical Journal, 71(3):41--47, 1992.Google Scholar
Cross Ref
- K. Burr and W. Young. Combinatorial test techniques: Table-based automation, test generation and code coverage. In Proceedings of the International Conference on Software Testing Analysis & Review, 1998.Google Scholar
- K. Burroughs, A. Jain, and R. L. Erickson. Improved quality of protocol testing through techniques of experimental design. In Supercomm/IC, IEEE International Conference on Communications, pages 745--752, 1994.Google Scholar
- D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton. The AETG system: an approach to testing based on combinatorial design. IEEE Transactions on Software Engineering, 23(7):437--444, 1997. Google ScholarDigital Library
- D. M. Cohen, S. R. Dalal, J. Parelius, and G. C. Patton. The combinatorial design approach to automatic test generation. IEEE Software, 13(5):83--88, 1996. Google ScholarDigital Library
- S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, and C. M. Lott. Model-based testing of a highly programmable system. In Proceedings of the International Symposium on Software Reliability Engineering, pages 174--179, 1998. Google ScholarDigital Library
- S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, C. M. Lott, G. C. Patton, and B. M. Horowitz. Model-based testing in practice. In Proceedings of the International Conference on Software Engineering, pages 285--294, 1999. Google ScholarDigital Library
- S. R. Dalal, A. Jain, G. Patton, M. Rathi, and P. Seymour. AETGSM web: a Web based service for automatic efficient test generation from functional requirements. In Proceedings of the IEEE Workshop on Industrial Strength Formal Specification Techniques, pages 84--85, Oct 1998. Google ScholarDigital Library
- I. S. Dunietz, W. K. Ehrlich, B. D. Szablak, C. L. Mallows, and A. Iannino. Applying design of experiments to software testing. In Proceedings of the International Conference on Software Engineering, (ICSE), pages 205--215, 1997. Google ScholarDigital Library
- S. Elbaum, D. Gable, and G. Rothermel. The impact of software evolution on code coverage information. In International Conference on Software Maintenance, pages 169--179, 2001. Google ScholarDigital Library
- J. E. Forrester and B. P. Miller. An empirical study of the robustness of Windows NT applications using random testing. In USENIX Windows System Symposium, pages 1--10, July 2000. Google ScholarDigital Library
- M. Grindal, J. Offutt, and S. T. Andler. Combination testing strategies: A survey. Software Testing, Verification and Reliability, 15(3):167--199, 2005.Google ScholarCross Ref
- M. Hardojo, S. Elbaum, and Z. Wang. The effect of field data integrity on failure reproduction and fault isolation. In Workshop on Remote Analysis and Monitoring of Software Systems, pages 41--44, May 2004.Google ScholarCross Ref
- D. Kuhn and M. Reilly. An investigation of the applicability of design of experiments to software testing. In Proceedings of the NASA/IEEE Software Engineering Workshop, pages 91--95, 2002. Google ScholarDigital Library
- D. Kuhn, D. R. Wallace, and A. M. Gallo. Software fault interactions and implications for software testing. IEEE Transactions on Software Engineering, 30(6):418--421, 2004. Google ScholarDigital Library
- R. Mandl. Orthogonal Latin squares: An application of experiment design to compiler testing. Communications of the ACM, 28(10): 1054--1058, 1985. Google ScholarDigital Library
- A. Memon, A. Porter, C. Yilmaz, A. Nagarajan, D. Schmidt, and B. Natarajan. Skoll: Distributed continuous quality assurance. In Proceedings of the International Conference on Software Engineering, pages 459--468, May 2004. Google ScholarDigital Library
- B. Miller, G. Cooksey, and F. Moore. An empirical study of the robustness of MacOS applications using random testing. In First International Workshop on Random Testing, pages 46--54, July 2006. Google ScholarDigital Library
- T. J. Ostrand and M. J. Balcer. The category-partition method for specifying and generating functional tests. Communications of the ACM, 31:678--686, 1988. Google ScholarDigital Library
- G. Rothermel and M. J. Harrold. Analyzing regression test selection techniques. IEEE Transactions on Software Engineering, 22(8):529--551, Aug. 1996. Google ScholarDigital Library
- E. J. Weyuker. Testing component-based software: A cautionary tale. IEEE Software, 15(5):54--59, 1998. Google ScholarDigital Library
- C. Yilmaz, M. B. Cohen, and A. Porter. Covering arrays for efficient fault characterization in complex configuration spaces. In International Symposium on Software Testing and Analysis, pages 45--54, July 2004. Google ScholarDigital Library
- C. Yilmaz, M. B. Cohen, and A. Porter. Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Transactions on Software Engineering, 31(1):20--34, Jan 2006. Google ScholarDigital Library
- C. You. My IE3.1. http://www.myie2.com/html_en/home.htm.Google Scholar
Index Terms
- Testing across configurations: implications for combinatorial testing
Recommendations
Configuration-aware regression testing: an empirical study of sampling and prioritization
ISSTA '08: Proceedings of the 2008 international symposium on Software testing and analysisConfigurable software lets users customize applications in many ways, and is becoming increasingly prevalent. Researchers have created techniques for testing configurable software, but to date, only a little research has addressed the problems of ...
Prioritizing Variable-Strength Covering Array
COMPSAC '13: Proceedings of the 2013 IEEE 37th Annual Computer Software and Applications ConferenceCombinatorial interaction testing is a well-studied testing strategy, and has been widely applied in practice. Combinatorial interaction test suite, such as fixed-strength and variable-strength interaction test suite, is widely used for combinatorial ...
Measuring and Mitigating Gaps in Structural Testing
ICSE '23: Proceedings of the 45th International Conference on Software EngineeringStructural code coverage is a popular test adequacy metric that measures the percentage of program structure (e.g., statement, branch, decision) executed by a test suite. While structural coverage has several benefits, previous studies suggested that ...
Comments