Barton P. Miller
Abstract
The following section describes the tools we built to test the utilities. These tools include the fuzz (random character) generator, ptyjig (to test interactive utilities), and scripts to automate the testing process. Next, we will describe the tests we performed, giving the types of input we presented to the utilities. Results from the tests will follow along with an analysis of the results, including identification and classification of the program bugs that caused the crashes. The final section presents concluding remarks, including suggestions for avoiding the types of problems detected by our study and some commentary on the bugs we found. We include an Appendix with the user manual pages for fuzz and ptyjig.
- 1 Dijkstra, E. W. GOTO Statement Considered Harmful. Commun. ACM 11, 3 (March 1968), 147-8. Google Scholar
Digital Library
- 2 Rochlis, J. A., and Eichin, M. W. With microscope and tweezers: The Worm from MIT's perspective. Commun. ACM 32, 6 (June 1989), 689- 698. Google ScholarDigital Library
- 3 Spafford, E. H. The internet Worm: Crisis and aftermath. Commun. ACM 32, 6 (June 1989), 678-687. Google ScholarDigital Library
- 4 Wood, D. A., Gibson, G. A., and Katz, R. H. Verifying a multiprocessor cache controller using random case generation. Computer Science Tech. Rep. UCB/CSD 89/490, University of California, Berkeley (January 1989). Google Scholar
Index Terms
- An empirical study of the reliability of UNIX utilities
Recommendations
An Empirical Study of JUnit Test-Suite Reduction
ISSRE '11: Proceedings of the 2011 IEEE 22nd International Symposium on Software Reliability EngineeringAs test suites grow larger during software evolution, regression testing becomes expensive. To reduce the cost of regression testing, test-suite reduction aims to select a minimal subset of the original test suite that can still satisfy all the test ...
To Kill a Mutant: An Empirical Study of Mutation Testing Kills
ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and AnalysisMutation testing has been used and studied for over four decades as a method to assess the strength of a test suite. This technique adds an artificial bug (i.e., a mutation) to a program to produce a mutant, and the test suite is run to determine if ...
Assessing model-based testing: an empirical study conducted in industry
ICSE Companion 2014: Companion Proceedings of the 36th International Conference on Software EngineeringWe compare manual testing without any automation performed by a tester at a software company with model-based testing (MBT) performed by a tester at a research center.
The system under test (SUT), of which two different versions were tested by each ...
Reviews
Brett D. Fleisch
This study examined the reliability of UNIX utilities using empirical testing procedures. The authors tested roughly 90 different utility programs on seven versions of UNIX, including a commercial version. The paper presents the tools used to test the UNIX programs, the tests, the results, and an analysis of the results. The authors, using the tools and testing procedures described, were able to crash more than 24 percent of the UNIX utility programs. The authors point out that the study is important because (1) it uncovers a large set of bugs that could be corrected, (2) it provides a means of identifying programming practices that could cause bugs or security holes, and (3) it simulates real crashes that could occur because of careless typing, noisy input, or noisy phone lines. A good portion of the paper discusses common mistakes made by programmers that caused the utilities to fail. This paper is a good example of how relatively simple techniques can uncover significant results. Because of the pervasive use of UNIX and the diverse set of environments examined in this study, many readers can benefit by the analysis of UNIX utility reliability. Tables included in the paper give specific names of utilities that failed; the analysis suggests reasons for the failures.
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments