Top

Empirical Software Engineering

Published in:

01-07-2023

The impact of a continuous integration service on the delivery time of merged pull requests

Authors: João Helis Bernardo, Daniel Alencar da Costa, Uirá Kulesza, Christoph Treude

Published in: Empirical Software Engineering | Issue 4/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Continuous Integration (CI) is a software development practice that builds and tests software frequently (e.g., at every push). One main motivator to adopt CI is the potential to deliver software functionalities more quickly than not using CI. However, there is little empirical evidence to support that CI helps projects deliver software functionalities more quickly. Through the analysis of 162,653 pull requests (PRs) of 87 GitHub projects, we empirically study whether adopting a CI service (TravisCI) can quicken the time to deliver merged PRs. We complement our quantitative study by analyzing 450 survey responses from participants of 73 software projects. Our results reveal that adopting a CI service may not necessarily quicken the delivery of merge PRs. Instead, the pivotal benefit of a CI service is to improve the decision making on PR submissions, without compromising the quality or overloading the project’s reviewers and maintainers. The automation provided by CI and the boost in developers’ confidence are key advantages of adopting a CI service. Furthermore, open-source projects planning to attract and retain developers should consider the use of a CI service in their project, since CI is perceived to lower the contribution barrier while making contributors feel more confident and engaged in the project.

previous article How do programmers fix bugs as workarounds? An empirical study on Apache projects

next article Much more than a prediction: Expert-based software effort estimation as a behavioral act

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

https://prdeliverydelay.github.io/#studied-projects

https://travis-ci.org/

http://github.com/haraka/haraka

https://prdeliverydelay.github.io/#rq3-r-squared-and-optimism

https://prdeliverydelay.github.io/#rq3-variables-explanatory-power

https://prdeliverydelay.github.io/#datasets

Bacchelli A, Bird C (2013) Expectations, outcomes, and challenges of modern code review. In: 2013 35th International Conference on Software Engineering (ICSE), IEEE, pp 712–721

Bavota G, Russo B (2015) Four eyes are better than two: On the impact of code reviews on software quality. In: 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, pp 81–90

Beck K (2000) Extreme Programming Explained Embrace Change. Addison-Wesley Professional

Bernardo JH, da Costa DA, Kulesza U (2018) Studying the impact of adopting continuous integration on the delivery time of pull requests. In: 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), IEEE, pp 131–141

Best D, Roberts D (1975) Algorithm as 89: the upper tail probabilities of spearman’s rho. Journal of the Royal Statistical Society Series C (Applied Statistics) 24(3):377–379

Braun V, Clarke V (2006) Using thematic analysis in psychology. Qualitative research in psychology 3(2):77–101CrossRef

Cassee N, Vasilescu B, Serebrenik A (2020) The silent helper: the impact of continuous integration on code reviews. 2020 IEEE 27th International Conference on Software Analysis. Evolution and Reengineering (SANER), IEEE, pp 423–434

Cliff N (1993) Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin 114(3):494CrossRef

Coelho J, Valente MT (2017) Why modern open source projects fail. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pp 186–196

da Costa DA, Abebe SL, McIntosh S, Kulesza U, Hassan AE (2014) An empirical study of delays in the integration of addressed issues. In: Software Maintenance and Evolution (ICSME), 2014 IEEE International Conference on, IEEE, pp 281–290

da Costa DA, McIntosh S, Kulesza U, Hassan AE (2016) The impact of switching to a rapid release cycle on the integration delay of addressed issues An empirical study of the mozilla firefox project. In: Proceedings of the 13th International Conference on Mining Software Repositories, ACM, New York, NY, USA, MSR ’16, pp 374–385

da Costa DA, McIntosh S, Treude C, Kulesza U, Hassan AE (2018) The impact of rapid release cycles on the integration delay of fixed issues. Empirical Software Engineering 23(2):835–904CrossRef

Debbiche A, Dienér M, Svensson RB (2014) Challenges when adopting continuous integration: A case study. In: International Conference on Product-Focused Software Process Improvement, Springer, pp 17–32

Duvall P, Matyas SM, Glover A (2007) Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series). Addison-Wesley Professional

Felidré W, Furtado L, da Costa DA, Cartaxo B, Pinto G (2019) Continuous integration theater. In: 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, pp 1–10

Fitzgerald B, Stol KJ (2017) Continuous software engineering: A roadmap and agenda. J Syst Softw 123:176–189CrossRef

Fowler M, Foemmel M (2006) Continuous integration. Thought-Works) http://www.thoughtworks.com /Continuous Integration pdf p 122

Gallaba K, McIntosh S (2018) Use and misuse of continuous integration features: An empirical study of projects that (mis) use travis ci. IEEE Trans Softw Eng 46(1):33–50CrossRef

Giger E, Pinzger M, Gall H (2010) Predicting the fix time of bugs. In: Proceedings of the 2nd International Workshop on Recommendation Systems for Software Engineering, ACM, pp 52–56

Goodman D, Elbaz M (2008) " it’s not the pants, it’s the people in the pants" learnings from the gap agile transformation what worked, how we did it, and what still puzzles us. In: Agile 2008 Conference, IEEE, pp 112–115

Gousios G, Zaidman A, Storey MA, Van Deursen A (2015) Work practices and challenges in pull-based development: the integrator’s perspective. In: Proceedings of the 37th International Conference on Software Engineering-Volume 1, IEEE Press, pp 358–368

Guo Y, Leitner P (2019) Studying the impact of ci on pull request delivery time in open source projects-a conceptual replication. PeerJ Comput Sci 5

Harrell F (2015) Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. Springer

Hars A, Shaosong O (2002) Working for free? motivations for participating in open-source projects. Int J Electron Commer 6(3):25–39CrossRef

Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering ASE 2016

Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation. Pearson Education

Jiang Y, Adams B, German DM (2013) Will my patch make it? and how fast? case study on the linux kernel. In: Mining Software Repositories (MSR), 2013 10th IEEE Working Conference on, IEEE, pp 101–110

Karvonen T, Behutiye W, Oivo M, Kuvaja P (2017) Systematic literature review on the impacts of agile release engineering practices. Inf Softw Technol 86:87–100CrossRef

Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. biometrics pp 159–174

Laukkanen E, Paasivaara M, Arvonen T (2015) Stakeholder perceptions of the adoption of continuous integration – a case study. In: Proceedings of the 2015 Agile Conference, IEEE Computer Society, AGILE ’15, pp 11–20

Mann HB, Whitney DR (1947) On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. Ann Math Stat 18(1):50–60. https://doi.org/10.1214/aoms/1177730491MathSciNetCrossRef

Michlmayr M, Fitzgerald B, Stol KJ (2015) Why and how should open source projects adopt time-based releases? IEEE Software 32(2):55–63CrossRef

Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density. In: Software Engineering, 2005. ICSE 2005. Proceedings. 27th International Conference on, IEEE, pp 284–292

Neely S, Stolt S (2013) Continuous delivery? easy! just change everything (well, maybe it is not that easy). In: 2013 Agile Conference, IEEE, pp 21–128

Nery GS, da Costa DA, Kulesza U (2019) An empirical study of the relationship between continuous integration and test code evolution. In: 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, pp 426–436

Nowell LS, Norris JM, White DE, Moules NJ (2017) Thematic analysis: Striving to meet the trustworthiness criteria. Int J Qual Methods 16(1):1609406917733847CrossRef

Perry DE, Porter AA, Votta LG (2000) Empirical studies of software engineering: A roadmap. In: Proceedings of the Conference on The Future of Software Engineering, ACM, ICSE ’00, pp 345–355

Rahman AAU, Helms E, Williams L, Parnin C (2015) Synthesizing continuous deployment practices used in software development. In: 2015 Agile Conference, IEEE, pp 1–10

Rahman MM, Roy CK (2017) Impact of continuous integration on code reviews. In: 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), IEEE, pp 499–502

Romano J, Kromrey J, Coraggio J, Skowronek J (2006) Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’sd for evaluating group differences on the NSSE and other surveys? In: annual meeting of the Florida Association of Institutional Research, pp 1–3

Santos J, Alencar da Costa D, Kulesza U (2022) Investigating the impact of continuous integration practices on the productivity and quality of open-source projects. In: Proceedings of the 16th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp 137–147

Schroter A, Schröter A, Bettenburg N, Premraj R (2010) Do stack traces help developers fix bugs? In: Mining Software Repositories (MSR), 2010 7th IEEE Working Conference on, IEEE, pp 118–121

Shahin M, Babar MA, Zhu L (2017) Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE Access 5:3909–3943CrossRef

Sharma P, Savarimuthu T, Stanger N (2021) Influence of roles in decision-making during oss development-a study of python. In: Evaluation and Assessment in Software Engineering, pp 50–59

Shihab E, Ihara A, Kamei Y, Ibrahim WM, Ohira M, Adams B, Hassan AE, Matsumoto Ki (2010) Predicting re-opened bugs: A case study on the eclipse project. In: Reverse Engineering (WCRE), 2010 17th Working Conference on, IEEE, pp 249–258

da Silva ACBG, de Figueiredo Carneiro G, de Paula ACM, Monteiro MP, e Abreu FB (2016) Agility and quality attributes in open source software projects release practices. In: 2016 10th International Conference on the Quality of Information and Communications Technology (QUATIC), IEEE, pp 107–112

Soares DM, de Lima Júnior ML, Plastino A, Murta L (2018) What factors influence the reviewer assignment to pull requests? Inf Softw Technol 98:32–43CrossRef

Soares E, Sizilio G, Santos J, da Costa DA, Kulesza U (2022) The effects of continuous integration on software development: a systematic literature review. Empir Softw Eng 27(3):1–61CrossRef

Ståhl D, Bosch J (2014) Modeling continuous integration practice differences in industry software development. J Syst Softw 87:48–59CrossRef

Vasilescu B, Van Schuylenburg S, Wulms J, Serebrenik A, van den Brand MG (2014) Continuous integration in a social-coding world: Empirical evidence from github. In: Software Maintenance and Evolution (ICSME), 2014 IEEE International Conference on, IEEE, pp 401–405

Vasilescu B, Yu Y, Wang H, Devanbu P, Filkov V (2015) Quality and productivity outcomes relating to continuous integration in GitHub. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering ESEC/FSE 2015

Weißgerber P, Neu D, Diehl S (2008) Small patches get in! In: Proceedings of the 2008 international working conference on Mining software repositories, pp 67–76

Wilks DS (2011) Statistical methods in the atmospheric sciences, vol 100. Academic press

Williamson DF, Parker RA, Kendrick JS (1989) The box plot: A simple visual method to interpret data. Ann Intern Med 110:916–921CrossRef

Yu Y, Yin G, Wang T, Yang C, Wang H (2016) Determinants of pull-based development in the context of continuous integration. Sci China Inf Sci 59(8)

Zampetti F, Bavota G, Canfora G, Di Penta M (2019) A study on the interplay between pull request review and continuous integration builds. 2019 IEEE 26th International Conference on Software Analysis. Evolution and Reengineering (SANER), IEEE, pp 38–48

Zhang X, Yu Y, Georgios G, Rastogi A (2022) Pull request decisions explained: An empirical overview. IEEE Trans Softw Eng 49:849–871CrossRef

Zhang X, Yu Y, Wang T, Rastogi A, Wang H (2022) Pull request latency explained: An empirical overview. Empir Softw Eng 27(6):1–38CrossRef

Zhao Y, Serebrenik A, Zhou Y, Filkov V, Vasilescu B (2017) The impact of continuous integration on other software development practices: a large-scale empirical study. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, IEEE Press, pp 60–71

Title: The impact of a continuous integration service on the delivery time of merged pull requests
Authors: João Helis Bernardo
Daniel Alencar da Costa
Uirá Kulesza
Christoph Treude
Publication date: 01-07-2023
Publisher: Springer US
Published in: Empirical Software Engineering / Issue 4/2023
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-023-10327-6

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 4/2023

Program transformation landscapes for automated program modification using Gin

Towards a taxonomy of Roxygen documentation in R packages

A new, evidence-based, theory for knowledge reuse in security risk analysis

Much more than a prediction: Expert-based software effort estimation as a behavioral act

Does agile methodology fit all characteristics of software projects? Review and analysis

BTLink : automatic link recovery between issues and commits based on pre-trained BERT model

Premium Partner