Yanbing Yu
ABSTRACT
Fault-localization techniques that utilize information about all test cases in a test suite have been presented. These techniques use various approaches to identify the likely faulty part(s) of a program, based on information about the execution of the program with the test suite. Researchers have begun to investigate the impact that the composition of the test suite has on the effectiveness of these fault-localization techniques. In this paper, we present the first experiment on one aspect of test-suite composition--test-suite reduction. Our experiment studies the impact of the test-suite reduction on the effectiveness of fault-localization techniques. In our experiment, we apply 10 test-suite reduction strategies to test suites for eight subject programs. We then measure the differences between the effectiveness of four existing fault-localization techniques on the unreduced and reduced test suites. We also measure the reduction in test-suite size of the 10 test-suite reduction strategies. Our experiment shows that fault-localization effectiveness varies depending on the test-suite reduction strategy used, and it demonstrates the trade-offs between test-suite reduction and fault-localization effectiveness.
- R. Abreu, P. Zoeteweij, and A. J. C. van Gemund. On the accuracy of spectrum-based fault localization. In Testing: Academic and Industrial Conference, Practice and Research Techniques, Windsor, UK, September 2007. Google Scholar
Digital Library
- H. Agrawal, J. Horgan, S. London, and W. Wong. Fault localization using execution slices and dataflow tests. In Proceedings of IEEE Software Reliability Engineering, pages 143--151, 1995.Google ScholarCross Ref
- Aristotle Research Group. Aristotle analysis system, 2007. http://www.cc.gatech.edu/aristotle/.Google Scholar
- B. Baudrey, F. Fleurey, and Y. L. Traon. Improving test suites for efficient fault localization. In International Conference on Software Engineering, pages 82--91, Shanghai, China, May 2006. Google ScholarDigital Library
- T. Y. Chen and M. F. Lau. Dividing strategies for the optimization of a test suite. Information Processing Letters, 60(3):135--141, Mar. 1996. Google ScholarDigital Library
- H. Cleve and A. Zeller. Locating causes of program failures. In Proceedings of the International Conference on Software Engineering, pages 342--351, St. Louis, Missouri, May 2005. Google ScholarDigital Library
- D. Hao, Y. Pan, L. Zhang, W. Zhao, H. Mei, and J. Sun. A similarity-aware approach to testing based fault localization. In Proceedings of the Conference on Automated Software Engineering, pages 291--294, November 2005. Google ScholarDigital Library
- M. J. Harrold, R. Gupta, and M. L. Soffa. A methodology for controlling the size of a test suite. ACM Transactions on Software Engineering and Methodology, 2(3):270--285, July 1993. Google ScholarDigital Library
- J. R. Horgan and S. A. London. ATAC: A data flow coverage testing tool for C. In Proceedings of the Symp. on Assessment of Quality Software Development Tools, pages 2--10, May 1992.Google Scholar
- M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments on the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the International Conference on Software Engineering, pages 191--200, May 1994. Google ScholarDigital Library
- J. Jones and M. J. Harrold. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the International Conference on Automated Software Engineering, pages 273--282, November 2005. Google ScholarDigital Library
- J. Jones, M. J. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In Proceedings of the International Conference on Software Engineering, pages 467--477, Orlando, Florida, May 2002. Google ScholarDigital Library
- B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan. Scalable statistical bug isolation. In Proceedings of the conference on Programming language design and implementation, pages 15--26, 2005. Google ScholarDigital Library
- C. Liu, X. Yan, L. Fei, J. Han, and S. P. Midkiff. SOBER: statistical model-based bug localization. In Proceedings of European Software Engineering Conference and Foundations on Software Engineering, pages 286--295, September 2005. Google ScholarDigital Library
- J. Offutt, J. Pan, and J. M. Voas. Procedures for reducing the size of coverage-based test sets. In Proceedings of the International Conference on Testing Comp. Software, pages 111--123, June 1995.Google Scholar
- M. Renieris and S. Reiss. Fault localization with nearest neighbor queries. In Proceedings of the International Conference on Automated Software Engineering, pages 30--39, Montreal, Quebec, October 2003.Google ScholarCross Ref
- F. Vokolos and P. Frankl. Empirical evaluation of the textual differencing regression testing techniques. In Proceedings of the International Conference on Software Maintenance, November 1998. Google ScholarDigital Library
Index Terms
- An empirical study of the effects of test-suite reduction on fault localization
Recommendations
A test-suite reduction approach to improving fault-localization effectiveness
In order to improve the effectiveness of fault localization, researchers are interested in test-suite reduction to provide suitable test-suite inputs. Different test-suite reduction approaches have been proposed. However, the results are usually not ...
Balancing trade-offs in test-suite reduction
FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software EngineeringRegression testing is an important activity but can get expensive for large test suites. Test-suite reduction speeds up regression testing by identifying and removing redundant tests based on a given set of requirements. Traditional research on test-...
Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage
ICSM '01: Proceedings of the IEEE International Conference on Software Maintenance (ICSM'01)Software testing is particularly expensive for developers of high-assurance software, such as software that is produced for commercial airborne systems. One reason for this expense is the Federal Aviation Administration's requirement that test suites be ...
Comments