Akond Rahman
Amritanshu Agrawal
Rahul Krishna
ABSTRACT
Continuous integration (CI) tools integrate code changes by automatically compiling, building, and executing test cases upon submission of code changes. Use of CI tools is getting increasingly popular, yet how proprietary projects reap the benefits of CI remains unknown. To investigate the influence of CI on software development, we analyze 150 open source software (OSS) projects, and 123 proprietary projects. For OSS projects, we observe the expected benefits after CI adoption, e.g., improvements in bug and issue resolution. However, for the proprietary projects, we cannot make similar observations. Our findings indicate that only adoption of CI might not be enough to the improve software development process. CI can be effective for software development if practitioners use CI's feedback mechanism efficiently, by applying the practice of making frequent commits. For our set of proprietary projects we observe practitioners commit less frequently, and hence not use CI effectively for obtaining feedback on the submitted code changes. Based on our findings we recommend industry practitioners to adopt the best practices of CI to reap the benefits of CI tools for example, making frequent commits.
- Amritanshu Agrawal, Akond Rahman, Rahul Krishna, Alexander Sobran, and Tim Menzies. 2018. We Don’T Need Another Hero?: The Impact of "Heroes" on Software Development. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP ’18). ACM, New York, NY, USA, 245–253. Google Scholar
Digital Library
- Kent Beck. 2000. Extreme Programming Explained: Embrace Change. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA. Google ScholarDigital Library
- Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub. In Proceedings of the 14th International Conference on Mining Software Repositories (MSR ’17). IEEE Press, Piscataway, NJ, USA, 356–367. Google ScholarDigital Library
- Pamela Bhattacharya, Marios Iliofotou, Iulian Neamtiu, and Michalis Faloutsos. 2012. Graph-based Analysis and Prediction for Software Evolution. In Proceedings of the 34th International Conference on Software Engineering (ICSE ’12). IEEE Press, Piscataway, NJ, USA, 419–429. http://dl.acm.org/citation.cfm?id=2337223. Google ScholarDigital Library
- 2337273Google Scholar
- Christian Bird, Peter C Rigby, Earl T Barr, David J Hamilton, Daniel M German, and Prem Devanbu. 2009. The promises and perils of mining git. In Mining Software Repositories, 2009. MSR’09. 6th IEEE International Working Conference on. IEEE, 1–10. Google ScholarDigital Library
- Travis CI. 2017. Travis CI. https://travis-ci.org/. (2017).Google Scholar
- {Online; accessed 15-October-2017}.Google Scholar
- Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin 114, 3 (Nov. 1993), 494–509.Google ScholarCross Ref
- Amit Deshpande and Dirk Riehle. 2008. Continuous Integration in Open Source Software Development. Springer US, Boston, MA, 273–280. 1007/978-0-387-09684-1_23Google Scholar
- Paul Duvall, Stephen M. Matyas, and Andrew Glover. 2007. Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series). Addison-Wesley Professional. Google ScholarDigital Library
- Github. 2017. Github Showcases. https://github.com/showcases. (2017). {Online; accessed 13-October-2017}.Google Scholar
- Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017.Google Scholar
- Trade-offs in Continuous Integration: Assurance, Security, and Flexibility. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017). ACM, New York, NY, USA, 197–207. Google ScholarDigital Library
- M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). 426–437. Google ScholarDigital Library
- Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M German, and Daniela Damian. 2014. The promises and perils of mining github. In Proceedings of the 11th working conference on mining software repositories. ACM, 92–101. Google ScholarDigital Library
- Rahul Krishna, Amritanshu Agrawal, Akond Rahman, Alexander Sobran, and Tim Menzies. 2018. What is the Connection Between Issues, Bugs, and Enhancements?: Lessons Learned from 800+ Software Projects. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP ’18). ACM, New York, NY, USA, 306–315. Google ScholarDigital Library
- H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics 18, 1 (1947), 50–60. http://www.jstor.org/stable/2236101Google ScholarCross Ref
- Helena Holmstrom Olsson, Hiva Alahyari, and Jan Bosch. 2012. Climbing the "Stairway to Heaven" – A Mulitiple-Case Study Exploring Barriers in the Transition from Agile Development Towards Continuous Deployment of Software. In Proceedings of the 2012 38th Euromicro Conference on Software Engineering and Advanced Applications (SEAA ’12). IEEE Computer Society, Washington, DC, USA, 392–399. Google ScholarDigital Library
- James W. Paulson, Giancarlo Succi, and Armin Eberlein. 2004. An Empirical Study of Open-Source and Closed-Source Software Products. IEEE Trans. Softw. Eng. 30, 4 (April 2004), 246–256. Google ScholarDigital Library
- Akond Rahman, Asif Partho, David Meder, and Laurie Williams. 2017. Which Factors Influence Practitioners’ Usage of Build Automation Tools?. In Proceedings of the 3rd International Workshop on Rapid Continuous Software Engineering (RCoSE ’17). IEEE Press, Piscataway, NJ, USA, 20–26. RCoSE.2017..8 Google ScholarDigital Library
- Akond Ashfaque Ur Rahman, Eric Helms, Laurie Williams, and Chris Parnin. 2015. Synthesizing Continuous Deployment Practices Used in Software Development. In Proceedings of the 2015 Agile Conference (AGILE ’15). IEEE Computer Society, Washington, DC, USA, 1–10. Google ScholarDigital Library
- Brian Robinson and Patrick Francis. 2010. Improving Industrial Adoption of Software Engineering Research: A Comparison of Open and Closed Source Software. In Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ’10). ACM, New York, NY, USA, Article 21, 10 pages. Google ScholarDigital Library
- Everett M Rogers. 2010. Diffusion of innovations. Simon and Schuster.Google Scholar
- J. Romano, J.D. Kromrey, J. Coraggio, and J. Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’sd for evaluating group differences on the NSSE and other surveys?. In annual meeting of the Florida Association of Institutional Research. 1–3.Google Scholar
- Helen Sharp and Hugh Robinson. 2008. Collaboration and Co-ordination in Mature eXtreme Programming Teams. Int. J. Hum.-Comput. Stud. 66, 7 (July 2008), 506–518. Google ScholarDigital Library
- Daniel Stahl, Torvald Martensson, and Jan Bosch. 2017. The continuity of continuous integration: Correlations and consequences. Journal of Systems and Software 127, Supplement C (2017), 150 – 167. Google ScholarDigital Library
- Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and productivity outcomes relating to continuous integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 805–816. Google ScholarDigital Library
- Fiorella Zampetti, Simone Scalabrino, Rocco Oliveto, Gerardo Canfora, and Massimiliano Di Penta. 2017. How Open Source Projects Use Static Code Analysis Tools in Continuous Integration Pipelines. In Proceedings of the 14th International Conference on Mining Software Repositories (MSR ’17). IEEE Press, Piscataway, NJ, USA, 334–344. Google ScholarDigital Library
- Yangyang Zhao, Alexander Serebrenik, Yuming Zhou, Vladimir Filkov, and Bogdan Vasilescu. 2017. The Impact of Continuous Integration on Other Software Development Practices: A Large-Scale Empirical Study. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2017). ACM, New York, NY, USA. Google ScholarDigital Library
Index Terms
- Characterizing the influence of continuous integration: empirical results from 250+ open source and proprietary projects
Recommendations
Usage, costs, and benefits of continuous integration in open-source projects
ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software EngineeringContinuous integration (CI) systems automate the compilation, building, and testing of software. Despite CI rising as a big success story in automated software engineering, it has received almost no attention from the research community. For example, ...
Comparing practices for reuse in integration-oriented software product lines and large open source software projects
This article compares the organization and practices for software reuse in integration-oriented software product lines (SPLs) and open source software projects. The main observation is that both approaches are successful regarding large variability and ...
Enterprise-driven open source software: a case study on security automation
ICSE-SEIP '21: Proceedings of the 43rd International Conference on Software Engineering: Software Engineering in PracticeAgile and DevOps are widely adopted by the industry. Hence, integrating security activities with industrial practices, such as continuous integration (CI) pipelines, is necessary to detect security flaws and adhere to regulators' demands early. In this ...
Comments