Nachiappan Nagappan
Thomas Ball
ABSTRACT
During software development it is helpful to obtain early estimates of the defect density of software components. Such estimates identify fault-prone areas of code requiring further testing. We present an empirical approach for the early prediction of pre-release defect density based on the defects found using static analysis tools. The defects identified by two different static analysis tools are used to fit and predict the actual pre-release defect density for Windows Server 2003. We show that there exists a strong positive correlation between the static analysis defect density and the pre-release defect density determined by testing. Further, the predicted pre-release defect density and the actual pre-release defect density are strongly correlated at a high degree of statistical significance. Discriminant analysis shows that the results of static analysis tools can be used to separate high and low quality components with an overall classification rate of 82.91%.
- Basili, V., Briand, L., Melo, W., "A Validation of Object Oriented Design Metrics as Quality Indicators," IEEE Transactions on Software Engineering, Vol. 22, No. 10, pp. 751--761, 1996. Google Scholar
Digital Library
- Brace, N., Kemp, R., Snelgar, R., SPSS for Psychologists: Palgrave Macmillan, 2003.Google Scholar
- Briand, L. C., Thomas, W.M., Hetmanski, C.J., "Modeling and Managing Risk Early in Software Development," Proceedings of International Conference on Software Engineering, 1993, pp. 55--65. Google ScholarDigital Library
- Briand, L. C., Wuest, J., Daly, J.W., Porter, D.V., "Exploring the Relationship between Design Measures and Software Quality in Object Oriented Systems," Journal of Systems and Software, Vol. 51, No. 3, pp. 245--273, 2000. Google ScholarDigital Library
- Briand, L. C., Wuest, J., Ikonomovski, S., Lounis, H., "Investigating Quality Factors in Object-Oriented Designs : An Industrial Case Study," Proceedings of International Conference on Software Engineering, 1999, pp. 345--354. Google ScholarDigital Library
- Bush, W. R., Pincus, J.D., Sielaff, D.J., "A Static Analyzer for Finding Dynamic Programming Errors," Software-Practice and Experience, Vol. 20, No. 7, pp. 775--802, 2000. Google ScholarDigital Library
- Chidamber, S. R. and C. F. Kemerer, "A Metrics Suite for Object Oriented Design," IEEE Transactions on Software Engineering, Vol. 20, No. 6, 1994. Google ScholarDigital Library
- Denaro, G., Morasca, S., Pezze, M., "Deriving Models of Software Fault-Proneness," Proceedings of Software Engineering Knowledge Engineering, 2002, pp. 361--368. Google ScholarDigital Library
- Denaro, G., Pezze, M., "An Empirical Evaluation of Fault-Proneness Models," Proceedings of International Conference on Software Engineering, 2002, pp. 241--251. Google ScholarDigital Library
- Engler, D., Chelf, B., Chou, A., Hallem, S., "Checking System Rules Using System-Specific, Programmer-Written Compiler Extensions," Proceedings of OSDI 2000, 2000. Google ScholarDigital Library
- Evans, D., Guttag, J., Horning, J., Tan, Y., M., "LCLint: A Tool for Using Cpecifications to Check Code," Proceedings of ACM-SIGSOFT Foundations in Software Engineering, 1994, pp. 87--96. Google ScholarDigital Library
- Fenton, N. E., Pfleeger, S.L., Software Metrics. Boston, MA: International Thompson Publishing, 1997.Fenton, N. E., Pfleeger, S.L., Software Metrics. Boston, MA: International Thompson Publishing, 1997.Google Scholar
- Khoshgoftaar, T. M., Allen, E.B., Deng,J., "Using Regression Trees to Classify Fault-Prone Software Modules," IEEE Transactions on Reliability, Vol. 51, No. 4, pp. 455--462, 2002.Google ScholarCross Ref
- Khoshgoftaar, T. M., Allen, E.B., Goel, N., Nandi, A., McMullan, J., "Detection of Software Modules with high Debug Code Churn in a very large Legacy System," Proceedings of International Symposium on Software Reliability Engineering, 1996, pp. 364--371. Google ScholarDigital Library
- Khoshgoftaar, T. M., Allen, E.B., Hudepohl, J.P., Aud, S.J., "Application of neural networks to software quality modeling of a very large telecommunications system," IEEE Transactions on Neural Networks, Vol. 8, No. 4, pp. 902--909, 1997. Google ScholarDigital Library
- Khoshgoftaar, T. M., Allen, E.B., Jones,W.D., Hudepohl, J.P., "Classification-Tree Models of Software Quality Over Multiple Releases," IEEE Transactions on Reliability, Vol. 49, No. 1, pp. 4--11, 2000.Google ScholarCross Ref
- Khoshgoftaar, T. M., Allen, E.B., Kalaichelvan, K.S., Goel, N., Hudepohl, J.P., Mayrand, J., "Detection of fault-prone program modules in a very large telecommunications system," Proceedings of International Symposium Software Reliability Engineering, 1995, pp. 24--33.Google ScholarCross Ref
- Khoshgoftaar, T. M., Munson, J.C., Lanning, D.L., "A Comparative Study of Predictive Models for Program Changes During System Testing and Maintenance," Proceedings of International Conference on Software Maintenance, 1993, pp. 72--79. Google ScholarDigital Library
- Khoshgoftaar, T. M., Seliya, N., "Fault Prediction Modeling for Software Quality Estimation: Comparing Commonly Used Techniques," Empirical Software Engineering, Vol. 8, No. 3, pp. 255--283, 2003. Google ScholarDigital Library
- Larus, J. R., Ball, T., Das, M., DeLine, R., Fahndrich, M., Pincus, J., Rajamani, S.K., Venkatapathy, R., "Righting Software," in IEEE Software, vol. 21, 2004, pp. 92--100. Google ScholarDigital Library
- Munson, J. C., Khoshgoftaar, T.M., "The Detection of Fault-Prone Programs," IEEE Transactions on Software Engineering, Vol. 18, No. 5, pp. 423--433, 1992. Google ScholarDigital Library
- Munson, J. C., Khoshgoftaar,T.M., "Regression Modeling of Software quality : Empirical Investigation," Information and Software Technology, Vol. 32, No. 2, pp. 106--114, 1990. Google ScholarDigital Library
- Nagappan, N., Williams, L., Hudepohl, J., Snipes, W., Vouk, M., "Preliminary Results On Using Static Analysis Tools For Software Inspection," Proceedings of Fifteenth IEEE International Symposium on Software Reliability Engineering, St. Malo, France, 2004, pp. 429--439. Google ScholarDigital Library
- Schneidewind, N. F., "Methodology for Validating Software Metrics," IEEE Transactions on Software Engineering, Vol. 18, No., pp. 410--422, 1992. Google ScholarDigital Library
- Subramanyam, R., Krishnan, M.S., "Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects," IEEE Transactions on Software Engineering, Vol. 29, No. 4, pp. 297--310, 2003. Google ScholarDigital Library
- Tang, M.-H., Kao, M-H., Chen, M-H., "An empirical study on object-oriented metrics," Proceedings of Sixth International Software Metrics Symposium, 1999, pp. 242--249. Google ScholarDigital Library
Index Terms
- Static analysis tools as early indicators of pre-release defect density
Recommendations
Use of relative code churn measures to predict system defect density
ICSE '05: Proceedings of the 27th international conference on Software engineeringSoftware systems evolve over time due to changes in requirements, optimization of code, fixes for security and reliability bugs etc. Code churn, which measures the changes made to a component over a period of time, quantifies the extent of this change. ...
Are Slice-Based Cohesion Metrics Actually Useful in Effort-Aware Post-Release Fault-Proneness Prediction? An Empirical Study
Background. Slice-based cohesion metrics leverage program slices with respect to the output variables of a module to quantify the strength of functional relatedness of the elements within the module. Although slice-based cohesion metrics have been ...
A Comparison of Different Defect Measures to Identify Defect-Prone Components
IWSM-MENSURA '13: Proceedings of the 2013 Joint Conference of the 23nd International Workshop on Software Measurement (IWSM) and the 8th International Conference on Software Process and Product Measurement(Background) Defect distribution in software systems has been shown to follow the Pareto rule of 20-80. This motivates the prioritization of components with the majority of defects for testing activities. (Research goal) Are there significant variations ...
Reviews
Andrew Brooks
Static analysis tools have been used to detect pre-release defects at Microsoft for six years. More than 12 percent of the pre-release defects fixed in Windows Server 2003 were found with the PREfix and PREfast static analysis tools. This paper uses historical data to determine how well statically found defects can predict the pre-release defect density, as measured by defects found by all other pre-release methods. Data was analyzed at the component level for over 199 components of Windows Server 2003 (22 million lines of code). Employing the technique of data splitting, random samples of 132 components were used to build regression models whose predictive ability was assessed on the remaining 67 components. Figure 3 shows how the estimated defect density tracks the actual defect density for three random samples. A discriminant analysis is said to correctly identify 165 of the 199 components (82.91 percent) as fault, or not fault, prone. An omission is the failure to report the false positive rates for PREFix and PREfast. An indication is given that some false positives might have been entered into the defect database. While Figure 3 demonstrates prediction tracking in general, at least three components with much higher defect densities appear that are not tracked by regression modeling. Why were these particular components so much worse__?__ Could other techniques, for example, software metric approaches, have predicted that these components were very fault prone__?__ We do not know. This paper is recommended to those working in software quality assurance. Online Computing Reviews Service
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments