Andrew Meneely
Alberto C. Rodriguez Tejeda
ABSTRACT
One of the guiding principles of open source software development is to use crowds of developers to keep a watchful eye on source code. Eric Raymond declared Linus'' Law as "many eyes make all bugs shallow", with the socio-technical argument that high quality open source software emerges when developers combine together their collective experience and expertise to review code collaboratively. Vulnerabilities are a particularly nasty set of bugs that can be rare, difficult to reproduce, and require specialized skills to recognize. Does Linus' Law apply to vulnerabilities empirically? In this study, we analyzed 159,254 code reviews, 185,948 Git commits, and 667 post-release vulnerabilities in the Chromium browser project. We formulated, collected, and analyzed various metrics related to Linus' Law to explore the connection between collaborative reviews and vulnerabilities that were missed by the review process. Our statistical association results showed that source code files reviewed by more developers are, counter-intuitively, more likely to be vulnerable (even after accounting for file size). However, files are less likely to be vulnerable if they were reviewed by developers who had experience participating on prior vulnerability-fixing reviews. The results indicate that lack of security experience and lack of collaborator familiarity are key risk factors in considering Linus’ Law with vulnerabilities.
- E. S. Raymond, The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary, 1st ed. O’Reilly Media, 2010. Google Scholar
Digital Library
- A. Meneely and O. Williams, “Interactive Churn: Socio-Technical Variants on Code Churn Metrics,” in Int’l Workshop on Software Quality, 2012, pp. 1–10.Google ScholarDigital Library
- Y. Shin, A. Meneely, L. Williams, and J. A. Osborne, “Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities,” IEEE Trans. Softw. Eng., vol. 37, no. 6, pp. 772–787, 2011. Google ScholarDigital Library
- A. Meneely and L. Williams, “Strengthening the Empirical Analysis of the Relationship Between Linus’ Law and Software Security,” in Empirical Software Engineering and Measurement, Bolzano-Bozen, Italy, 2010, pp. 1–10. Google ScholarDigital Library
- A. Meneely and L. Williams, “Secure Open Source Collaboration: an Empirical Study of Linus’ Law,” in Int’l Conference on Computer and Communications Security (CCS), Chicago, Illinois, USA, 2009, pp. 453–462. Google ScholarDigital Library
- A. Meneely and L. Williams, “Socio-Technical Developer Networks: Should We Trust Our Measurements?,” presented at the International Conference on Software Engineering, Waikiki, Hawaii, USA, 2011, p. to appear. Google ScholarDigital Library
- A. Meneely, L. Williams, W. Snipes, and J. Osborne, “Predicting Failures with Developer Networks and Social Network Analysis,” in 16th ACM SIGSOFT International Symposium on Foundations of software engineering, Atlanta, Georgia, 2008, pp. 13–23. Google ScholarDigital Library
- E. L. Trist and K. W. Bamforth, “Some social and psychological consequences of the longwall method of coalgetting,” Technol. Organ. Innov. Early Debates, p. 79, 2000.Google Scholar
- I. V. Krsual, “Software Vulnerability Analysis.” PhD Dissertation, Purdue University, 1998. Google ScholarDigital Library
- S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan, “The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects,” in 11th Working Conference on Mining Software Repositories, New York, NY, USA, 2014, pp. 192–201. Google ScholarDigital Library
- A. Bosu, “Characteristics of the Vulnerable Code Changes Identified Through Peer Code Review,” in 36th International Conference on Software Engineering, New York, NY, USA, 2014, pp. 736–738. Google ScholarDigital Library
- A. Meneely, H. Srinivasan, A. Musa, A. R. Tejeda, M. Mokary, and B. Spates, “When a patch goes bad: Exploring the properties of vulnerability-contributing commits,” Proc. 2013 ACM-IEEE Int. Symp. Empir. Softw. Eng. Meas., p. to appear, 2013.Google Scholar
- C. F. Kemerer and M. C. Paulk, “The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data,” IEEE Trans Softw Eng, vol. 35, no. 4, pp. 534–550, Jul. 2009. Google ScholarDigital Library
- N. Nagappan and T. Ball, “Use of Relative Code Churn Measures to Predict System Defect Density,” in 27th international Conference on Software Engineering (ICSE), St. Louis, MO, USA, 2005, pp. 284–292. Google ScholarDigital Library
- J. C. Munson and S. G. Elbaum, “Code churn: a measure for estimating the impact of code change,” in Software Maintenance, 1998. Proceedings. International Conference on, 1998, pp. 24 –31. Google ScholarDigital Library
- S. M. Garcia, K. Weaver, G. B. Moskowitz, and J. M. Darley, “Crowded minds: The implicit bystander effect,” J. Pers. Soc. Psychol., vol. 83, no. 4, pp. 843–853, 2002.Google ScholarCross Ref
- E. Coakes, “Socio-technical thinking: an holistic viewpoint,” in Socio-technical and human cognition elements of information systems, IGI Publishing, 2003, pp. 1–4. Google ScholarDigital Library
- G. McGraw, Software Security: Building Security In. Addison-Wesley Professional, 2006. Google ScholarDigital Library
- N. F. Schneidewind, “Methodology for Validating Software Metrics,” IEEE Trans. Softw. Eng. TSE, vol. 18, no. 5, pp. 410–422, 1992. Google ScholarDigital Library
- P. V. Rao, Statistical Research Methods in the Life Sciences, 1st ed. Duxbury Press, 1997.Google Scholar
Index Terms
- An empirical investigation of socio-technical code review metrics and security vulnerabilities
Recommendations
Code review quality: how developers see it
ICSE '16: Proceedings of the 38th International Conference on Software EngineeringIn a large, long-lived project, an effective code review process is key to ensuring the long-term quality of the code base. In this work, we study code review practices of a large, open source project, and we investigate how the developers themselves ...
Do bugs foreshadow vulnerabilities? An in-depth study of the chromium project
As developers face an ever-increasing pressure to engineer secure software, researchers are building an understanding of security-sensitive bugs (i.e. vulnerabilities). Research into mining software repositories has greatly increased our understanding ...
Identifying the characteristics of vulnerable code changes: an empirical study
FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software EngineeringTo focus the efforts of security experts, the goals of this empirical study are to analyze which security vulnerabilities can be discovered by code review, identify characteristics of vulnerable code changes, and identify characteristics of developers ...
Comments