Frank Li
Vern Paxson
ABSTRACT
Given how the "patching treadmill" plays a central role for enabling sites to counter emergent security concerns, it behooves the security community to understand the patch development process and characteristics of the resulting fixes. Illumination of the nature of security patch development can inform us of shortcomings in existing remediation processes and provide insights for improving current practices. In this work we conduct a large-scale empirical study of security patches, investigating more than 4,000 bug fixes for over 3,000 vulnerabilities that affected a diverse set of 682 open-source software projects. For our analysis we draw upon the National Vulnerability Database, information scraped from relevant external references, affected software repositories, and their associated security fixes. Leveraging this diverse set of information, we conduct an analysis of various aspects of the patch development life cycle, including investigation into the duration of impact a vulnerability has on a code base, the timeliness of patch development, and the degree to which developers produce safe and reliable fixes. We then characterize the nature of security fixes in comparison to other non-security bug fixes, exploring the complexity of different types of patches and their impact on code bases.
Among our findings we identify that: security patches have a lower footprint in code bases than non-security bug patches; a third of all security issues were introduced more than 3 years prior to remediation; attackers who monitor open-source repositories can often get a jump of weeks to months on targeting not-yet-patched systems prior to any public disclosure and patch distribution; nearly 5% of security fixes negatively impacted the associated software; and 7% failed to completely remedy the security hole they targeted.
Supplemental Material
- American Fuzzy Lop. http://lcamtuf.coredump.cx/afl/.Google Scholar
- cgit. https://git.zx2c4.com/cgit/about/.Google Scholar
- Core Infrastructure Initiative. https://www.coreinfrastructure.org.Google Scholar
- Exuberant Ctags. http://ctags.sourceforge.net/.Google Scholar
- GitLab. https://about.gitlab.com/.Google Scholar
- GitWeb. https://git-scm.com/book/en/v2/Git-on-the-Server-GitWeb.Google Scholar
- ISC Software Defect and Security Vulnerability Disclosure Policy. https://kb.isc.org/article/AA-00861/164/ISC-Software-Defect-and-Security- Vulnerability-Disclosure-Policy.html.Google Scholar
- Open Crypto Audit Project. https://opencryptoaudit.org.Google Scholar
- Undefined Behavior Sanitizer. https://clang.llvm.org/docs/UndefinedBehavior Sanitizer.html.Google Scholar
- Steve Christey and Brian Martin. Buying Into the Bias: Why Vulnerability Statistics Suck. In BlackHat, 2013.Google Scholar
- Zakir Durumeric, Frank Li, James Kasten, Nicholas Weaver, Johanna Amann, Jethro Beekman, Mathias Payer, David Adrian, Vern Paxson, Michael Bailey, and J. Alex Halderman. The Matter of Heartbleed. In ACM Internet Measurement Conference (IMC), 2014. Google ScholarDigital Library
- Forum of Incident Response and Security Teams. Common Vulnerability Scoring System v3.0: Specification Document. https://www.first.org/cvss/specification-document.Google Scholar
- Stefan Frei. End-Point Security Failures: Insights gained from Secunia PSI Scans. In USENIX Predict Workshop, 2011.Google Scholar
- Stefan Frei, Martin May, Ulrich Fiedler, and Bernhard Plattner. Large-Scale Vulnerability Analysis. In SIGCOMM Workshops, 2006. Google ScholarDigital Library
- Google. Sanitizers. https://github.com/google/sanitizers.Google Scholar
- Google Open Source Blog. Announcing OSS-Fuzz: Continuous Fuzzing for Open Source Software. https://opensource.googleblog.com/2016/12/announcing-oss-fuzz-continuous-fuzzing.html.Google Scholar
- Zhongxian Gu, Earl Barr, David Hamilton, and Zhendong Su. Has the Bug Really Been Fixed? In International Conference on Software Engineering (ICSE), 2010. Google ScholarDigital Library
- Zhen Huang, Mariana D'Angelo, Dhaval Miyani, and David Lie. Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Response. In IEEE Security and Privacy (S&P), 2016.Google Scholar
- Jonathan Corbet. Kernel Vulnerabilities: Old or New?, October 2010. https://lwn.net/Articles/410606/.Google Scholar
- Kees Cook. Security Bug Lifetime, October 2016. https://outflux.net/blog/archives/2016/10/18/security-bug-lifetime.Google Scholar
- Frank Li, Zakir Durumeric, Jakub Czyz, Mohammad Karami, Michael Bailey, Damon McCoy, Stefan Savage, and Vern Paxson. You've Got Vulnerability: Exploring Effective Vulnerability Notifications. In USENIX Security Symposium, 2016.Google Scholar
- T. J. McCabe. A Complexity Measure. In IEEE Transaction on Software Engineering, 1976. Google ScholarDigital Library
- MITRE Corporation. Common Vulnerabilities and Exposures. https://cve.mitre.org/.Google Scholar
- MITRE Corporation. CWE: Common Weakness Enumeration. https://cwe.mitre.org/.Google Scholar
- Nuthan Munaiah and Andrew Meneely. Vulnerability Severity Scoring and Bounties: Why the Disconnect? In International Workshop on Software Analytics (SWAN), 2016. Google ScholarDigital Library
- Emerson Murphy-Hill, Thomas Zimmermann, Christian Bird, and Nachiappan Nagappan. The Design of Bug Fixes. In International Conference on Software Engineering (ICSE), 2013. Google ScholarCross Ref
- Antonio Nappa, Richard Johnson, Leyla Bilge, Juan Caballero, and Tudor Dumitras. The Attack of the Clones: A Study of the Impact of Shared Code on Vulnerability Patching. In IEEE Security and Privacy (S&P), 2015. Google ScholarDigital Library
- Andy Ozment and Stuart E. Schechter. Milk or Wine: Does Software Security Improve with Age? In USENIX Security Symposium, 2006.Google ScholarDigital Library
- Jihun Park, Miryung Kim, Baishkhi Ray, and Doo-Hwan Bae. An Empirical Study on Supplementary Bug Fixes. In Mining Software Repositories (MSR), 2012.Google Scholar
- Henning Perl, Sergej Dechand, Matthew Smith, Daniel Arp, Fabian Yamaguchi, Konrad Rieck, Sascha Fahl, and Yasemin Acar. VCCFinder: Finding Potential Vulnerabilities in Open-Source Projects to Assist Code Audits. In ACM Conference on Computer and Communications Security (CCS), 2015. Google ScholarDigital Library
- RhodeCode. Version Control Systems Popularity in 2016. https://rhodecode.com/insights/version-control-systems-2016.Google Scholar
- Muhammad Shahzad, M. Zubair Shafiq, and Alex X. Liu. A Large Scale Exploratory Analysis of Software Vulnerability Life Cycles. In International Conference on Software Engineering (ICSE), 2012.Google Scholar
- Jacek Sliwerski, Thomas Zimmermann, and Andreas Zeller. When Do Changes Induce Fixes. In Mining Software Repositories (MSR), 2005. Google ScholarDigital Library
- Mauricio Soto, Ferdian Thung, Chu-Pan Wong, Claire Le Goues, and David Lo. A Deeper Look into Bug Fixes: Patterns, Replacements, Deletions, and Additions. In Mining Software Repositories (MSR), 2016.Google ScholarDigital Library
- U.S. National Institute of Standards and Technology. CVSS Information. https://nvd.nist.gov/cvss.cfm.Google Scholar
- U.S. National Institute of Standards and Technology. National Checklist Program Glossary. https://web.nvd.nist.gov/view/ncp/repository/glossary.Google Scholar
- U.S. National Institute of Standards and Technology. National Vulnerability Database. https://nvd.nist.gov/home.cfm.Google Scholar
- U.S. National Institute of Standards and Technology. NVD Data Feed. https://nvd.nist.gov/download.cfm.Google Scholar
- Zhengzi Xu, Bihuan Chen, Mahinthan Chandramohan, Yang Liu, and Fu Song. SPAIN: Security Patch Analysis for Binaries Towards Understanding the Pain and Pills. In International Conference on Software Engineering (ICSE), 2017. Google ScholarDigital Library
- Zuoning Yin, Ding Yuan, Yuanyuan Zhou, Shankar Pasupathy, and Lakshmi Bairavasundaram. How do Fixes become Bugs? In ACM European Conference on Foundations of Software Engineering (ESEC/FSE), 2011. Google ScholarDigital Library
- Shahed Zaman, Bram Adams, and Ahmed E. Hassan. Security Versus Performance Bugs: A Case Study on Firefox. In Mining Software Repositories (MSR), 2011. Google ScholarDigital Library
- Hao Zhong and Zhendong Su. An Empirical Study on Real Bug Fixes. In International Conference on Software Engineering (ICSE), 2015. Google ScholarCross Ref
Index Terms
- A Large-Scale Empirical Study of Security Patches
Recommendations
Locating the Security Patches for Disclosed OSS Vulnerabilities with Vulnerability-Commit Correlation Ranking
CCS '21: Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications SecuritySecurity patches play an important role in defending against the security threats brought by the increasing OSS vulnerabilities. However, the collection of security patches still remains a challenging problem. Existing works mainly adopt a matching-...
An empirical study of supplementary patches in open source projects
Developers occasionally make more than one patch to fix a bug. The related patches sometimes are intentionally separated, but unintended omission errors require supplementary patches. Several change recommendation systems have been suggested based on ...
Writing Acceptable Patches: An Empirical Study of Open Source Project Patches
ICSME '14: Proceedings of the 2014 IEEE International Conference on Software Maintenance and EvolutionSoftware developers submit patches to handle tens or even hundreds of bugs reported daily. However, not all submitted patches can be directly integrated into the code base, since they might not pass patch review that is adopted in most software ...
Comments