Nathaniel Ayewah
William Pugh
ABSTRACT
Static analysis tools find silly mistakes, confusing code, bad practices and property violations. But software developers and organizations may or may not care about all these warnings, depending on how they impact code behavior and other factors. In the past, we have tried to identify important warnings by asking users to rate them as severe, low impact or not a bug. In this paper, we observe that the user's rating may be more complicated depending on whether the warning is feasible, changes code behavior, occurs in deployed code and other factors. To better model this, we ask users to review warnings using a checklist which enables more detailed reviews. We find that reviews are consistent across users and across checklist questions, though some users may disagree about whether to fix or filter out certain bug classes.
- N. Ayewah and W. Pugh. A report on a survey and study of static analysis users. In DEFECTS '08: Proceedings of the 2008 workshop on Defects in large software systems, pages 1--5, New York, NY, USA, 2008. ACM. Google Scholar
Digital Library
- N. Ayewah, W. Pugh, J. D. Morgenthaler, J. Penix, and Y. Zhou. Evaluating static analysis defect warnings on production software. In PASTE '07: Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, pages 1--8, New York, USA, 2007. ACM. Google ScholarDigital Library
- D. Hovemeyer and W. Pugh. Finding bugs is easy. In OOPSLA '04: Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications, pages 132--136, New York, NY, USA, 2004. ACM. Google ScholarDigital Library
- D. Hovemeyer and W. Pugh. Finding more null pointer bugs, but not too many. In PASTE '07: Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, pages 9--14, New York, USA, 2007. ACM. Google ScholarDigital Library
- Y. P. Khoo, J. S. Foster, M. Hicks, and V. Sazawal. Path projection for user-centered static analysis tools. In PASTE '08: Proceedings of the 8th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, pages 57--63, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- S. Kim and M. D. Ernst. Which warnings should i fix first? In ESEC-FSE '07: Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, pages 45--54, New York, NY, USA, 2007. ACM. Google ScholarDigital Library
- N. Nagappan and T. Ball. Static analysis tools as early indicators of pre-release defect density. In ICSE '05: Proceedings of the 27th international conference on Software engineering, pages 580--586, New York, NY, USA, 2005. ACM. Google ScholarDigital Library
Index Terms
- Using checklists to review static analysis warnings
Recommendations
A report on a survey and study of static analysis users
DEFECTS '08: Proceedings of the 2008 workshop on Defects in large software systemsAs static analysis tools mature and attract more users, vendors and researchers have an increased interest in understanding how users interact with them, and how they impact the software development process. The FindBugs project has conducted a number ...
Evaluating static analysis defect warnings on production software
PASTE '07: Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineeringStatic analysis tools for software defect detection are becoming widely used in practice. However, there is little public information regarding the experimental evaluation of the accuracy and value of the warnings these tools report. In this paper, we ...
Finding bugs in eclipse
OOPSLA '07: Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companionThis will be a live demonstration of FindBugs, a static analysis bug finding tool, on the current development version of Eclipse 3.4. FindBugs reports issues such as null pointer dereferences, comparing incompatible types with equals, invalid method ...
Reviews
Serge Berger
Static analysis is important for detecting code defects. Most automatic tools based on static analysis have reached a certain level of maturity, making them trusted partners in code quality assurance. However, the tools are often limited in terms of adjusting to the right level of warning for code violations. This paper creates a checklist to correlate the intent of the program with the severity of the issue reported by a static analysis tool-the tool is FindBugs. Statistically, false positives are the most costly within the model. Ayewah and Pugh exploit the Pearson correlation coefficient to check the consistency of checklist responses, and use the chi-square test to trace the trends. The results are summarized in a few tables. Visualization-for example, receiver operating characteristic (ROC) curves-of the results over the false positives ratio would have better demonstrated the points and helped analyze statistically significant dependencies. The authors intend to continue researching the subject, with a larger number of developers involved in the checklist activities. Therefore, statistical characteristics of the model should improve. Online Computing Reviews Service
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments