Gregg Rothermel
Abstract
Regression testing is an expensive but necessary maintenance activity performed on modified software to provide confidence that changes are correct and do not adversely affect other portions of the softwore. A regression test selection technique choses, from an existing test set, thests that are deemed necessary to validate modified software. We present a new technique for regression test selection. Our algorithms construct control flow graphs for a precedure or program and its modified version and use these graphs to select tests that execute changed code from the original test suite. We prove that, under certain conditions, the set of tests our technique selects includes every test from the original test suite that con expose faults in the modified procedfdure or program. Under these conditions our algorithms are safe. Moreover, although our algorithms may select some tests that cannot expose faults, they are at lease as precise as other safe regression test selection algorithms. Unlike many other regression test selection algorithms, our algorithms handle all language constructs and all types of program modifications. We have implemented our algorithms; initial empirical studies indicate that our technique can significantly reduce the cost of regression testing modified software.
- AGRAWAL, H., HORGAN, J., KRAUSER, E., AND LONDON, S. 1993. Incremental regression testing. In Proceedings of the Conference on Software Maintenance--1993. IEEE, New York, 348-357. Google Scholar
- AHO, A.V., SETHI, R., AND ULLMAN, J.D. 1986. Compilers: Principles, Techniques, and Tools. Addison-Wesley, Reading, Mass. Google Scholar
- BALCER, M., HASLING, W., AND OSTRAND, T. 1989. Automatic generation of test scripts from formal test specifications. In Proceedings of the 3rd Symposium on Software Testing, Analysis, and Verification. ACM, New York, 210-218. Google Scholar
- BATES, S. AND HORWITZ, S. 1993. Incremental program testing using program dependence graphs. In Proceedings of the 20th ACM Symposium on Principles of Programming Languages. ACM, New York. Google Scholar
- BEIZER, B. 1990. Software Testing Techniques. Van Nostrand Reinhold, New York. Google Scholar
- BENEDUSI, P., CIMITILE, A., AND DE CARLINI, U. 1988. Post-maintenance testing based on path change analysis. In Proceedings of the Conference on Software Maintenance--1988. IEEE, New York, 352-361.Google Scholar
- BINKLEY, D. 1992. Using semantic differencing to reduce the cost of regression testing. In Proceedings of the Conference on Software Maintenance--1992. IEEE, New York, 41-50.Google Scholar
- BINKLEY, D. 1995. Reducing the cost of regression testing by semantics guided test case selection. In Proceedings of the Conference on Software Maintenance--1995. IEEE, New York. Google Scholar
- BOEHM, B.W. 1976. Software engineering. IEEE Trans. Comput. C-25, 12 (Dec.), 1226- 1241.Google Scholar
- BROWN, P.A. AND HOFFMAN, D. 1990. The application of module regression testing at TRIUMF. Nucl. Instr. Meth. Phys. Res. A293, 1-2 (Aug.), 377-381.Google Scholar
- CHEN, Y.F., ROSENBLUM, D.S., AND Vo, K.P. 1994. TestTube: A system for selective regression testing. In Proceedings of the 16th International Conference on Software Engineering. IEEE, New York, 211-222. Google Scholar
- DOGSA, T. AND ROZMAN, I. 1988. CAMOTE--Computer aided module testing and design environment. In Proceedings of the Conference on Software Maintenance--1988. IEEE, New York, 404-408.Google Scholar
- FISCHER, K.F. 1977. A test case selection method for the validation of software maintenance modifications. In Proceedings of COMPSAC '77. IEEE, New York, 421-426.Google Scholar
- FISCHER, K.F., RAJI, F., AND CHRUSCICKI, A. 1981. A methodology for retesting modified software. In Proceedings of the National Telecommunications Conference B-6-3. IEEE, New York, 1-6.Google Scholar
- GUPTA, R., HARROLD, M. g., AND SOFFA, M.L. 1992. An approach to regression testing using slicing. In Proceedings of the Conference on Software Maintenance--1992. IEEE, New York, 299 -308.Google Scholar
- HARROLD, M.g. AND SOFFA, M.L. 1988. An incremental approach to unit testing during maintenance. In Proceedings of the Conference on Software Maintenance--1988. IEEE, New York, 362-367.Google Scholar
- HARROLD, M. g. AND SOFFA, M.L. 1989a. An incremental data flow testing tool. In Proceedings of the 6th International Conference on Testing Computer Software. Frontier Technologies, Annapolis, Md.Google Scholar
- HARROLD, M. g. AND SOFFA, M.L. 1989b. Interprocedural data flow testing. In Proceedings of the 3rd Symposium on Software Testing, Analysis, and Verification. ACM, New York, 158-167. Google Scholar
- HARROLD, M. J., GUPTA, R., AND SOFFA, M.L. 1993. A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Methodol. 2, 3 (July), 270-285. Google Scholar
- HARROLD, M. J., LARSEN, L., LLOYD, J., NEDVED, D., PAGE, M., ROTHERMEL, G., SINGH, M., AND SMITH, M. 1995. Aristotle: A system for the development of program-analysis-based tools. In Proceedings of the 33rd Annual Southeast Conference. ACM, New York, 110-119. Google Scholar
- HARTMANN, J. AND ROBSON, D.J. 1989. Revalidation during the software maintenance phase. In Proceedings of the Conference on Software Maintenance--1989. IEEE, New York, 70-79.Google Scholar
- HARTMANN, J. AND ROBSON, D.J. 1990a. RETEST--Development of a selective revalidation prototype environment for use in software maintenance. In Proceedings of the 23rd Hawaii International Conference on System Sciences. IEEE, New York, 92-101.Google Scholar
- HARTMANN, J. AND ROBSON, D.J. 1990b. Techniques for selective revalidation. IEEE Softw. 16, 1 (Jan.), 31-38. Google Scholar
- HOFFMAN, D. 1989. A CASE study in module testing. In Proceedings of the Conference on Software Maintenance--1989. IEEE, New York, 100-105.Google Scholar
- HOFFMAN, D. AND BREALEY, C. 1989. Module test case generation. In Proceedings of the 3rd Symposium on Software Testing, Analysis, and Verification. ACM, New York, 97-102. Google Scholar
- HUTCHINS, M., FOSTER, H., GORADIA, T., AND OSTRAND, T. 1994. Experiments on the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the 16th International Conference on Software Engineering. IEEE, New York, 191-200. Google Scholar
- LASKI, J. AND SZERMER, W. 1992. Identification of program modifications and its applications in software maintenance. In Proceedings of the Conference on Software Maintenance-- 1992. IEEE, New York, 282-290.Google Scholar
- LEE, J. A. N. AND HE, X. 1990. A methodology for test selection. J. Syst. Softw. 13, 1 (Sept.), 177-185. Google Scholar
- LEUNG, H. K. N. AND WHITE, L. 1989. Insights into regression testing. In Proceedings of the Conference on Software Maintenance--1989. IEEE, New York, 60-69.Google Scholar
- LEUNG, H. K. N. AND WHITE, L. 1990a. Insights into testing and regression testing global variables. J. Softw. Maint. Res. Pract. 2 (Dec.), 209-222. Google Scholar
- LEUNG, H. K. N. AND WHITE, L.g. 1990b. A study of integration testing and software regression at the integration level. In Proceedings of the Conference on Software Maintenance--1990. IEEE, New York, 290-300.Google Scholar
- LEUNG, H. K. N. AND WHITE, L.g. 1991. A cost model to compare regression test strategies. In Proceedings of the Conference on Software Maintenance--1991. IEEE, New York, 201- 208.Google Scholar
- LEWIS, R., BECK, D.W., AND HARTMANN, J. 1989. Assay--A tool to support regression testing. In ESEC '89. 2nd European Software Engineering Conference Proceedings. Springer- Verlag, Berlin, 487-496. Google Scholar
- LIENTZ, B. P. AND SWANSON, E.B. 1980. Software Maintenance Management: A Study of the Maintenance of Computer Applications Software in 487 Data Processing Organizations. Addison-Wesley, Reading, Mass. Google Scholar
- LIENTZ, B.P., SWANSON, E.B., AND TOMPKINS, G.E. 1978. Characteristics of application software maintenance. Commun. ACM 21, 6 (June), 466-471. Google Scholar
- MARTINIG, F. 1996. Software testing: Poor consideration. Testing Tech. Newslett. (Oct.).Google Scholar
- NOSEK, g. T. AND PALVIA, P. 1990. Software maintenance management: Changes in the last decade. J. Softw. Maint. Res. Pract. 2, 157-174. Google Scholar
- OSTRAND, T. J. AND BALCER, M.J. 1988. The category-partition method for specifying and generating functional tests. Commun. ACM 31, 6 (June). Google Scholar
- OSTRAND, T. J. AND WEYUKER, E.J. 1988. Using dataflow analysis for regression testing. In the 6th Annual Pacific Northwest Software Quality Conference. Lawrence and Craig, Portland, Oreg., 233-247.Google Scholar
- PRESSMAN, R. 1987. Software Engineering: A Practitioner's Approach. McGraw-Hill, New York. Google Scholar
- ROSENBLUM, D. S. AND WEYUKER, E.J. 1996. Predicting the cost-effectiveness of regression testing strategies. In Proceedings of the ACM SIGSOFT '96 4th Symposium on the Foundations of Software Engineering. ACM, New York. Google Scholar
- ROTHERMEL, G. 1996. Efficient, effective regression testing using safe test selection techniques. Tech. Rep. 96-101, Clemson Univ., Clemson, S. Carol.Google Scholar
- ROTHERMEL, G. AND HARROLD, M.g. 1993. A safe, efficient algorithm for regression test selection. In Proceedings of the Conference on Software Maintenance--1993. IEEE, New York, 358-367. Google Scholar
- ROTHERMEL, G. AND HARROLD, M.J. 1994a. Selecting regression tests for object-oriented software. In Proceedings of the Conference on Software Maintenance--1994. IEEE, New York, 14-25. Google Scholar
- ROTHERMEL, G. AND HARROLD, M.J. 1994b. Selecting tests and identifying test coverage requirements for modified software. In Proceedings of the 1994 International Symposium on Software Testing and Analysis (ISSTA 94). ACM, New York. Google Scholar
- ROTHERMEL, G. AND HARROLD, M.g. 1996. Analyzing regression test selection techniques. IEEE Trans. Softw. Eng. 22, 8 (Aug.), 529-551. Google Scholar
- SCHACH, S. 1992. Software Engineering. Aksen Assoc., Boston, Mass. Google Scholar
- SHARON, D. 1996. Meeting the challenge of software maintenance. IEEE Softw. 13, 1 (Jan.), 122-125. Google Scholar
- SHERLUND, B. AND KOREL, B. 1991. Modification oriented software testing. In Conference Proceedings: Quality Week 1991. Software Research Inc., San Francisco, Calif., 1-17.Google Scholar
- SHERLUND, B. AND KOREL, B. 1995. Logical modification oriented software testing. In Proceedings: 12th International Conference on Testing Computer Software. Frontier Technologies, Annapolis, Md.Google Scholar
- TAHA, A. B., THEBAUT, S. M., AND LIU, S.S. 1989. An approach to software fault localization and revalidation based on incremental data flow analysis. In Proceedings of the 13th Annual International Computer Software and Applications Conference. IEEE, New York, 527-534.Google Scholar
- VON MAYRHAUSER, A., MRAZ, R. T., AND WALLS, J. 1994. Domain based regression testing. In Proceedings of the Conference on Software Maintenance--1994. IEEE, New York, 26-35. Google Scholar
- WHITE, L. J. AND LEUNG, H. K.N. 1992. A firewall concept for both control-flow and dataflow in regression integration testing. In Proceedings of the Conference on Software Maintenance--1992. IEEE, New York, 262-270.Google Scholar
- WHITE, L. J., NARAYANSWAMY, V., FRIEDMAN, T., KIRSCHENBAUM, M., PIWOWARSKI, P., AND OHA, M. 1993. Test Manager: A regression testing tool. In Proceedings of the Conference on Software Maintenance--1993. IEEE, New York, 338-347. Google Scholar
- TONG, W.E., HORGAN, J.R., LONDON, S., AND MATHUR, A.P. 1995. Effect of test set minimization on fault detection effectiveness. In the 17th International Conference on Software Engineering. IEEE, New York, 41-50. Google Scholar
- YAU, S. S. AND KISHIMOTO, Z. 1987. A method for revalidating modified programs in the maintenance phase. In COMPSAC '87: The 11th Annual International Computer Software and Applications Conference. IEEE, New York, 272-277.Google Scholar
- ZIEGLER, J., GRASSO, J. M., AND BURGERMEISTER, L.G. 1989. An Ada based real-time closedloop integration and regression test tool. In Proceedings of the Conference on Software Maintenance--1989. IEEE, New York, 81-90.Google Scholar
Index Terms
- A safe, efficient regression test selection technique
Recommendations
An empirical study of regression test selection techniques
Regression testing is the process of validating modified software to detect whether new errors have been introduced into previously tested code and to provide confidence that modifications are correct. Since regression testing is an expensive process, ...
Empirical Studies of a Safe Regression Test Selection Technique
Regression testing is an expensive testing procedure utilized to validate modified software. Regression test selection techniques attempt to reduce the cost of regression testing by selecting a subset of a program's existing test suite. Safe regression ...
Regression test selection on system requirements
ISEC '08: Proceedings of the 1st India software engineering conferenceRegression testing, which is performed after changes are made to a software system, can be used before release of new versions of the system. However, practitioners often have little time to perform this regression testing because of the quick-release ...
Reviews
Andrew Donald Booth
In the context of this paper, regression means the evaluation of any effects generated in unmodified parts of a program by modifications in other parts. This is particularly important when complex and bloated software is upgraded. Most industrial testing is done via test suites, that is, sets of procedures designed to exercise all parts of a program and to show up computational or operational errors. While the original suite could be used to test the upgrade, this might be time-consuming. The object of the work described in this paper is to derive an algorithm that will conduct only those tests which are really needed. The proposed algorithm is briefly described in pseudocode, and the results of applying it to two example programs are given. In the first example, a saving of about 10 percent results, and in the second the saving is an impressive 95 percent. This is a useful paper.
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments