Software unit test coverage and adequacy

Reviewer: Curtis Roger Cook

The authors survey research in test data adequacy criteria, or criteria that test sets should satisfy. Since finding adequacy criteria that are both reliable and valid is impractical, research has shifted to finding practical approximations for stopping rules or measures of the quality of the testing. Classifying test adequacy criteria by the underlying testing approach provides an excellent organizational framework for this paper. The three basic testing approaches, which correspond to the main sections of the paper, are structural testing, fault-based testing, and error-based testing. Adequacy criteria for structural testing include control flow and data flow for both programs and specifications. Fault-based criteria, which include error seeding, mutation testing, and perturbation testing, measure the fault-detecting ability of the criteria. Error-based criteria measure how well the test set checks error-prone points. The authors make the important point that research into test adequacy criteria has been done mostly by academics, and industry has been slow to accept test adequacy measurement. The paper contains a section comparing the various adequacy criteria according to their fault-detecting ability, software reliability, and test cost. It indicates that much more work needs to be done if test adequacy criteria are to be more widely adopted by industry. The final section of the paper is an axiomatic study of the properties of adequacy criteria. Although the title “Software Unit Test Adequacy” would better reflect the contents of this paper, it is a readable, well-organized survey of test adequacy criteria.