Akbar Siami Namin
ABSTRACT
We study the relationship between three properties of test suites: size, structural coverage, and fault-finding effectiveness. In particular, we study the question of whether achieving high coverage leads directly to greater effectiveness, or only indirectly through forcing a test suite to be larger. Our experiments indicate that coverage is sometimes correlated with effectiveness when size is controlled for, and that using both size and coverage yields a more accurate prediction of effectiveness than size alone. This in turn suggests that both size and coverage are important to test suite effectiveness. Our experiments also indicate that no linear relationship exists among the three variables of size, coverage and effectiveness, but that a nonlinear relationship does exist.
- J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In Proceedings of the 27th International Conference on Software Engineering (ICSE 2005), St. Louis, Missouri, May 2005. 402--411. Google Scholar
Digital Library
- J. H. Andrews, L. C. Briand, Y. Labiche, and A. Siami Namin. Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Transactions on Software Engineering, 32(8):608--624, August 2006. Google ScholarDigital Library
- Bullseye Technologies. BullseyeCoverage website. www.bullseye.com, accessed January 2009, 2009.Google Scholar
- M. E. Delamaro and J. C. Maldonado. Proteum -- a tool for the assessment of test adequacy for C programs. In Proceedings of the Conference on Performability in Computing Systems (PCS 96), pages 79--95, New Brunswick, NJ, July 1996.Google Scholar
- P. G. Frankl and O. Iakounenko. Further empirical studies of test effectiveness. In Proceedings of the 6th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 153--162, Lake Buena Vista, FL, USA, 1998. Google ScholarDigital Library
- P. G. Frankl and S. N. Weiss. An experimental comparison of the effectiveness of branch testing and data flow testing. IEEE Transactions on Software Engineering, 19(8):774--787, August 1993. Google ScholarDigital Library
- J. P. Guilford. Fundamental Statistics in Psychology and Education. McGraw-Hill, New York, 1956.Google Scholar
- J. Horgan and S. London. ATAC: A data flow coverage testing tool for C. In Proceedings of the Symposium on Assessment of Quality Software Development Tools, pages 2--10, May 1992.Google ScholarCross Ref
- M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the 16th International Conference on Software Engineering (ICSE 1994), pages 191--200, Sorrento, Italy, May 1994. Google ScholarDigital Library
- B. Marick. How to misuse code coverage. URL www.exampler.com/testing-com/writings/coverage.pdf. 13pp., 1997.Google Scholar
- A. J. Offutt and R. Untch. Mutation 2000: Uniting the orthogonal. In Mutation 2000: Mutation Testing in the Twentieth and the Twenty First Centuries, pages 45--55, San Jose, CA, October 2000.Google Scholar
- G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong. An empirical study of the effects of minimization on the fault detection capabilities of test suites. In Proceedings of the International Conference on Software Maintenance (ICSM '98), pages 34--43, Washington, DC, USA, November 1998. Google ScholarDigital Library
- G. Rothermel, M. J. Harrold, J. von Ronne, and C. Hong. Empirical studies of test-suite reduction. Software Testing, Verification and Reliability, 12:219--249, 2002.Google ScholarCross Ref
- A. Siami Namin, J. H. Andrews, and D. J. Murdoch. Sufficient mutation operators for measuring test effectiveness. In Proceedings of the 30th International Conference on Software Engineering (ICSE 2008), pages 351--360, Leipzig, Germany, May 2008. Google ScholarDigital Library
- W. N. Venables, D. M. Smith, and The R Development Core Team. An introduction to R. Technical report, R Development Core Team, June 2006.Google Scholar
Index Terms
- The influence of size and coverage on test suite effectiveness
Recommendations
Code coverage for suite evaluation by developers
ICSE 2014: Proceedings of the 36th International Conference on Software EngineeringOne of the key challenges of developers testing code is determining a test suite's quality -- its ability to find faults. The most common approach is to use code coverage as a measure for test suite quality, and diminishing returns in coverage or high ...
Coverage is not strongly correlated with test suite effectiveness
ICSE 2014: Proceedings of the 36th International Conference on Software EngineeringThe coverage of a test suite is often used as a proxy for its ability to detect faults. However, previous studies that investigated the correlation between code coverage and test suite effectiveness have failed to reach a consensus about the nature and ...
Assertions are strongly correlated with test suite effectiveness
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software EngineeringCode coverage is a popular test adequacy criterion in practice. Code coverage, however, remains controversial as there is a lack of coherent empirical evidence for its relation with test suite effectiveness. More recently, test suite size has been ...
Comments