William Dickinson
ABSTRACT
Observation-based testing calls for analyzing profiles of executions induced by potential test cases, in order to select a subset of executions to be checked for conformance to requirements. A family of techniques for selecting such a subset is evaluated experimentally. These techniques employ automatic cluster analysis to partition executions, and they use various sampling techniques to select executions from clusters. The experimental results support the hypothesis that with appropriate profiling, failures often have unusual profiles that are revealed by cluster analysis. The results also suggest that failures often form small clusters or chains in sparsely-populated areas of the profile space. A form of adaptive sampling called failure-pursuit sampling is proposed for revealing failures in such regions, and this sampling method is evaluated experimentally. The results suggest that failure-pursuit sampling is effective.
- 1.Borg, I. and Groenen, P. Modern Multidimensional Scaling: Theory and Applications, Springer, 1997.Google Scholar
Digital Library
- 2.Covey, J. Musical scores at Jeff Covey's homepage. http://www4.smart.net/jcovey/scores (accessed September 2000).Google Scholar
- 3.Dickinson, W., Leon, D., and Podgurski, A. Finding failures by cluster analysis of execution profiles. Proceedings of the 2001 International Conference on Software Engineering (Toronto, May 2001). Google ScholarDigital Library
- 4.Elbaum, S., Malishevsky, A.G., and Rothermel, G. Prioritizing test cases for regression testing. Proceedings of the 2000 International Symposium on Software Testing and Analysis (Portland, Oregon, August 2000), 102-112. Google ScholarDigital Library
- 5.Free Software Foundation. The GCC home page. http://www.gnu.org/software/gcc/gcc.html, 2000.Google Scholar
- 6.Greenacre, M.J. Theory and Applications of Correspondence Analysis, Academic Press, 1984.Google Scholar
- 7.Han, J. and Kamber, M. Data Mining: Concepts and Techniques, Morgan Kaufmann, 2000. Google ScholarDigital Library
- 8.Harrold, M.J., Rothermel, G., Wu, R., and Yi, L. An empirical investigation of program spectra. ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (Montreal, June 1998), 83- 90. Google ScholarDigital Library
- 9.Jain, A.K. and Dubes, R.C. Algorithms for Clustering Data, Prentice Hall, 1988. Google ScholarDigital Library
- 10.Latterman, D. Musical scores for GNU LilyPond. http://www.alqualonde.de/lilypond.html (accessed September 2000).Google Scholar
- 11.Leon, D., Podgurski, A., and White, L.J. Multivariate visualization in observation-based testing. Proceedings of the 2000 International Conference on Software Engineering (Limerick, Ireland, June 2000), ACM Press, 116-125. Google ScholarDigital Library
- 12.The Mutopia Project. Mutopia. http://www.mutopiaproject.org, (accessed September 2000).Google Scholar
- 13.Nienhuys, H. and Nieuwenhuizen, J. GNU LilyPond. http://www.cs.uu.nl/hanwen/lilypond, 2000.Google Scholar
- 14.Pavlopoulou, C. and Young, M. Residual test coverage monitoring. Proceedings of the 21st International Conference on Software Engineering (Los Angeles, May 1999), ACM Press, 277-284. Google ScholarDigital Library
- 15.Podgurski, A., Masri, W., McCleese, Y., Wolff, F.G., and Yang, C. Estimation of software reliability by stratified sampling. ACM Transactions on Software Engineering and Methodology 8, 9 (July, 1999), 263-283. Google ScholarDigital Library
- 16.The Pub Scouts. The Pub Scouts' Tune List. http://rigel.csuchico.edu/~pubscout/songs.html (accessed September 2000).Google Scholar
- 17.Reps, T., Ball, T., Das, M., and Larus, J. The use of program profiling for software maintenance with applications to the Year 2000 Problem. Proceedings of the 6th European Software Engineering Conference and 5th ACM SIGSOFT Symposium on the Foundations of Software Engineering (Zurich, September 1997), ACM Press, 432-449. Google ScholarDigital Library
- 18.Robbins, J. Debugging Applications, Microsoft Press, 2000. Google ScholarDigital Library
- 19.Robinson, R. Richard Robinson's tune book. http://www.leeds.ac.uk/music/Info/RRTuneBk/tunebook.htm l (accessed September 2000).Google Scholar
- 20.Rothermel, G., Untch, R., Chu, C., and Harrold, M.J. Testcase prioritization: an empirical study. Proceedings of the International Conference on Software Maintenance (August, 1999), 179-188. Google ScholarDigital Library
- 21.Shlien, S. Runabc.tcl. http://ifdo.pubmarks.com/seymor/runabc/top.html (accessed September 2000).Google Scholar
- 22.Steven, J., Chandra, P., Fleck, B., and Podgurski, A. jRapture: a capture/replay tool for observation-based testing. Proceedings of the 2000 International Symposium on Software Testing & Analysis (Portland, Oregon, August 2000), ACM Press, 158-167. Google ScholarDigital Library
- 23.Thompson, S. K. and Seber, G.A. Adaptive Sampling, Wiley, 1996.Google Scholar
- 24.Walshaw, C., The ABC musical notation language. http://www.gre.ac.uk/c.walshaw/abc, 2000.Google Scholar
Index Terms
- Pursuing failure: the distribution of program failures in a profile space
Recommendations
Finding failures by cluster analysis of execution profiles
ICSE '01: Proceedings of the 23rd International Conference on Software EngineeringWe experimentally evaluate the effectiveness of using cluster analysis of execution profiles to find failures among the executions induced by a set of potential test cases. We compare several filtering procedures for selecting executions to evaluate for ...
Estimation of software reliability by stratified sampling
A new approach to software reliability estimation is presented that combines operational testing with stratified sampling in order to reduce the number of program executions that must be checked manually for conformance to requirements. Automatic ...
Pursuing failure: the distribution of program failures in a profile space
Observation-based testing calls for analyzing profiles of executions induced by potential test cases, in order to select a subset of executions to be checked for conformance to requirements. A family of techniques for selecting such a subset is ...
Comments