Foyzur Rahman
Premkumar Devanbu
ABSTRACT
The all-important goal of delivering better software at lower cost has led to a vital, enduring quest for ways to find and remove defects efficiently and accurately. To this end, two parallel lines of research have emerged over the last years. Static analysis seeks to find defects using algorithms that process well-defined semantic abstractions of code. Statistical defect prediction uses historical data to estimate parameters of statistical formulae modeling the phenomena thought to govern defect occurrence and predict where defects are likely to occur. These two approaches have emerged from distinct intellectual traditions and have largely evolved independently, in “splendid isolation”. In this paper, we evaluate these two (largely) disparate approaches on a similar footing. We use historical defect data to apprise the two approaches, compare them, and seek synergies. We find that under some accounting principles, they provide comparable benefits; we also find that in some settings, the performance of certain static bug-finders can be enhanced using information provided by statistical defect prediction.
- E. Arisholm, L. C. Briand, and M. Fuglerud. Data mining techniques for building fault-proneness models in telecom java software. In ISSRE, pages 215–224. IEEE Computer Society, 2007. Google Scholar
Digital Library
- E. Arisholm, L. C. Briand, and E. B. Johannessen. A systematic and comprehensive investigation of methods to build and evaluate fault prediction models. JSS, 83(1):2–17, 2010. Google ScholarDigital Library
- N. Ayewah, D. Hovemeyer, J. D. Morgenthaler, J. Penix, and W. Pugh. Using static analysis to find bugs. IEEE Software, 25(5):22–29, 2008. Google ScholarDigital Library
- A. Bessey, K. Block, B. Chelf, A. Chou, B. Fulton, S. Hallem, C. Henri-Gros, A. Kamsky, S. McPeak, and D. Engler. A few billion lines of code later: using static analysis to find bugs in the real world. Communications of the ACM, 53(2):66–75, 2010. Google ScholarDigital Library
- M. D’Ambros, M. Lanza, and R. Robbes. Evaluating defect prediction approaches: a benchmark and an extensive comparison. Empirical Software Engineering, 17(4-5):531–577, 2012. Google ScholarDigital Library
- K. El Emam, S. Benlarbi, N. Goel, and S. Rai. The confounding effect of class size on the validity of objectoriented metrics. TSE, 27(7):630–650, 2001. Google ScholarDigital Library
- D. Engler and K. Ashcraft. Racerx: effective, static detection of race conditions and deadlocks. In SOSP, volume 37, pages 237–252. ACM, 2003. Google ScholarDigital Library
- C. Flanagan, K. R. M. Leino, M. Lillibridge, G. Nelson, J. B. Saxe, and R. Stata. Extended static checking for java. In ACM Sigplan Notices, volume 37, pages 234– 245. ACM, 2002. Google ScholarDigital Library
- N. Jovanovic, C. Kruegel, and E. Kirda. Pixy: A static analysis tool for detecting web application vulnerabilities. In SP, pages 6–pp. IEEE, 2006. Google ScholarDigital Library
- S. Kim and M. D. Ernst. Which warnings should i fix first? In FSE, pages 45–54. ACM, 2007. Google ScholarDigital Library
- D. Larochelle and D. Evans. Statically detecting likely buffer overflow vulnerabilities. In USENIX Security Symposium, pages 177–190. Washington DC, 2001. Google ScholarDigital Library
- S. Lessmann, B. Baesens, C. Mues, and S. Pietsch. Benchmarking classification models for software defect prediction: A proposed framework and novel findings. TSE, 34(4):485–496, July 2008. Google ScholarDigital Library
- C. Lewis, Z. Lin, C. Sadowski, X. Zhu, R. Ou, and E. J. Whitehead Jr. Does bug prediction support human developers? findings from a google case study. In ICSE, pages 372–381. IEEE Press, 2013. Google ScholarDigital Library
- A. Marchenko and P. Abrahamsson. Predicting software defect density: a case study on automated static code analysis. In Agile Processes in Software Engineering and Extreme Programming, pages 137–140. Springer, 2007. Google ScholarDigital Library
- A. Meneely and L. A. Williams. Secure open source collaboration: an empirical study of linus’ law. In E. Al-Shaer, S. Jha, and A. D. Keromytis, editors, CCS, pages 453–462. ACM, 2009. Google ScholarDigital Library
- N. Nagappan and T. Ball. Static analysis tools as early indicators of pre-release defect density. In ICSE, pages 580–586. ACM, 2005. Google ScholarDigital Library
- N. Nagappan, B. Murphy, and V. Basili. The influence of organizational structure on software quality: an empirical case study. In ICSE, pages 521–530. ACM, 2008. Google ScholarDigital Library
- M. G. Nanda, M. Gupta, S. Sinha, S. Chandra, D. Schmidt, and P. Balachandran. Making defectfinding tools work for you. In ICSE, pages 99–108. ACM, 2010. Google ScholarDigital Library
- F. Rahman and P. Devanbu. How, and why, process metrics are better. In ICSE, pages 432–441. IEEE Press, 2013. Google ScholarDigital Library
- F. Rahman, D. Posnett, and P. Devanbu. Recalling the “imprecision” of cross-project defect prediction. In FSE. ACM, 2012. Google ScholarDigital Library
- F. Rahman, D. Posnett, I. Herraiz, and P. Devanbu. Sample size vs. bias in defect prediction. In FSE, 2013. Google ScholarDigital Library
- N. Ramasubbu, M. Cataldo, R. K. Balan, and J. D. Herbsleb. Configuring global software teams: a multicompany analysis of project productivity, quality, and profits. In ICSE, pages 261–270. ACM, 2011. Google ScholarDigital Library
- N. Rutar, C. B. Almazan, and J. S. Foster. A comparison of bug finding tools for java. In ISSRE, pages 245–256. IEEE, 2004. Google ScholarDigital Library
- A. September. IEEE standard glossary of software engineering terminology, 1990.Google Scholar
- J. Śliwerski, T. Zimmermann, and A. Zeller. When do changes induce fixes? In ACM sigsoft software engineering notes, volume 30, pages 1–5. ACM, 2005. Google ScholarDigital Library
- F. Thung, D. Lo, L. Jiang, F. Rahman, P. T. Devanbu, et al. To what extent could we detect field defects? an empirical study of false negatives in static bug finding tools. In ASE, pages 50–59. ACM, 2012. Google ScholarDigital Library
- S. Wagner, J. Jürjens, C. Koller, and P. Trischberger. Comparing bug finding tools with reviews and tests. In Testing of Communicating Systems, pages 40–55. Springer, 2005. Google ScholarDigital Library
- G. Wassermann and Z. Su. Sound and precise analysis of web applications for injection vulnerabilities. In ACM Sigplan Notices, volume 42, pages 32–41. ACM, 2007. Google ScholarDigital Library
- H. Zhang and S. Cheung. A cost-effectiveness criterion for applying software defect prediction models. In FSE, pages 643–646. ACM, 2013. Google ScholarDigital Library
- J. Zheng, L. Williams, N. Nagappan, W. Snipes, J. P. Hudepohl, and M. A. Vouk. On the value of static analysis for fault detection in software. TSE, 32(4):240– 253, 2006. Google ScholarDigital Library
Index Terms
- Comparing static bug finders and statistical prediction
Recommendations
Recalling the "imprecision" of cross-project defect prediction
FSE '12: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software EngineeringThere has been a great deal of interest in defect prediction: using prediction models trained on historical data to help focus quality-control resources in ongoing development. Since most new projects don't have historical data, there is interest in ...
Continuous Software Bug Prediction
ESEM '21: Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)Background: Many software bug prediction models have been proposed and evaluated on a set of well-known benchmark datasets. We conducted pilot studies on the widely used benchmark datasets and observed common issues among them. Specifically, most of ...
Cross-project smell-based defect prediction
AbstractDefect prediction is a technique introduced to optimize the testing phase of the software development pipeline by predicting which components in the software may contain defects. Its methodology trains a classifier with data regarding a set of ...
Comments