Top

Empirical Software Engineering

Published in:

01-03-2023

On the usage, co-usage and migration of CI/CD tools: A qualitative analysis

Authors: Pooya Rostami Mazrae, Tom Mens, Mehdi Golzadeh, Alexandre Decan

Published in: Empirical Software Engineering | Issue 2/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Continuous integration, delivery and deployment (CI/CD) is used to support the collaborative software development process. CI/CD tools automate a wide range of activities in the development workflow such as testing, linting, updating dependencies, creating and deploying releases, and so on. Previous quantitative studies have revealed important changes in the landscape of CI/CD usage, with the increasing popularity of cloud-based services, and many software projects migrating to other CI/CD tools. In order to understand the reasons behind these changes in CI/CD usage, this paper presents a qualitative study based on in-depth interviews with 22 experienced software practitioners reporting on their usage, co-usage and migration of 31 different CI/CD tools. Following an inductive and deductive coding process, we analyse the interviews and found a high amount of competition between CI/CD tools. We observe multiple reasons for co-using different CI/CD tools within the same project, and we identify the main reasons and detractors for migrating to different alternatives. Among all reported migrations, we observe a clear trend of migrations away from Travis and migrations towards GitHub Actions and we identify the main reasons behind them.

previous article Software selection in large-scale software engineering: A model and criteria based on interactive rapid reviews

next article Satisfaction and performance of software developers during enforced work from home in the COVID-19 pandemic

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

See https://docs.microsoft.com/en-us/azure/devops/boards/queries/history-and-auditing and https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-history

https://docs.microsoft.com/en-us/azure/devops/pipelines/release/?view=azure-devops

https://circleci.com/docs/2.0/workspaces

Azure’s Deployment History feature enables to go back in time to allow to redeploy a release that was for example available one year ago, just by selecting that deployment in the history. See https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-history?tabs=azure-portal.

Whenever 3+ CI tools are used together, each pair of CI tools is reported individually.

https://saucelabs.com/blog/announcing-travis-ci-for-mac-and-ios-powered-by-sauce-labs

https://blog.travis-ci.com/2018-10-11-windows-early-release

https://about.gitlab.com/blog/2020/01/21/windows-shared-runner-beta

https://about.gitlab.com/blog/2021/08/23/build-cloud-for-macos-beta

https://nvd.nist.gov/vuln/detail/CVE-2021-41077

https://dev.to/lyncozy/crypto-miners-are-killing-free-ci-4pii

¹²https://visualstudio.microsoft.com/services/github-codespaces/

Abdalkareem R, Mujahid S, Shihab E (2020) A machine learning approach to improve the detection of CI skip commits. IEEE Trans Softw Eng

Abdalkareem R, Mujahid S, Shihab E, Rilling J (2019) Which commits can be CI skipped? IEEE Trans Softw Eng 47(3):448–463CrossRef

Ampatzoglou A, Bibi S, Avgeriou P, Verbeek M, Chatzigeorgiou A (2019) Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Inf Softw Technol 106:201–230CrossRef

Beck K (2000) Extreme programming explained: embrace change. Addison-Wesley Professional

Beller M, Gousios G, Zaidman A (2017) Oops, my tests broke the build an explorative analysis of Travis CI with GitHub. In: International conference on mining software repositories (MSR), pp 356–367

Bernardo JH, da Costa DA, Kulesza U (2018) Studying the impact of adopting continuous integration on the delivery time of pull requests. In: International conference on mining software repositories (MSR), pp 131–141. IEEE

Betz RM, Walker RC (2013) Implementing continuous integration software in an established computational chemistry software package. In: International workshop on software engineering for computational science and engineering (SE-CSE), pp 68–74. IEEE

Chen L (2015) Continuous delivery: Huge benefits, but challengs too. IEEE Software - Special Issue on Release Engineering 32(2):50–54

Chen L (2017) Continuous delivery: overcoming adoption challenges. J Syst Softw 128:72–86CrossRef

Chen T, Zhang Y, Chen S, Wang T (2021) Let’s supercharge the workflows: an empirical study of GitHub actions. In: International conference on software quality, reliability and security companion (QRS-C), pp 01–10. IEEE

Decan A, Mens T, Mazrae PR, Golzadeh M (2022) On the use of GitHub actions in software development repositories. In: International conference on software maintenance and evolution (ICSME)

Duvall PM, Matyas S, Glover A (2007) Continuous integration: improving software quality and reducing risk. Addison-Wesley Professional

Elazhary O, Werner C, Li ZS, Lowlind D, Ernst NA, Storey M-A (2022) Uncovering the benefits and challenges of continuous integration practices. IEEE Trans Softw Eng 48(7):2570–2583CrossRef

Esfahani H, Fietz J, Ke Q, Kolomiets A, Lan E, Mavrinac E, Schulte W, Sanches N, Kandula S (2016) CloudBuild: Microsoft’s distributed and caching build service. In: International conference on software engineering (ICSE), pp 11–20

Foundjem A, Constantinou E, Mens T, Adams B (2022) A mixed-methods analysis of micro-collaborative coding practices in OpenStack. Empir Softw Eng 27(5):120CrossRef

Fowler M, Foemmel M (2006) Continuous integration

Fusch PI, Ness LR (2015) Are we there yet? Data saturation in qualitative research. The Qualitative Report, vol 20(9)

Gallaba K, Junqueira Y, Ewart J, Mcintosh S (2022) Accelerating continuous integration by caching environments and inferring dependencies. IEEE Trans Softw Eng 48(6):2040–2052CrossRef

Ghaleb TA, Da Costa DA, Zou Y (2019) An empirical study of the long duration of continuous integration builds. Empir Softw Eng 24(4):2102–2139CrossRef

Gmeiner J, Ramler R, Haslinger J (2015) Automated testing in the continuous delivery pipeline: a case study of an online company. In: International conference on software testing, verification and validation workshops (ICSTW), pp 1–6. IEEE

Golzadeh M, Decan A, Mens T (2022) On the rise and fall of CI services in GitHub. In: International conference on software analysis, evolution and Reengineering (SANER)

Guest G, Bunce A, Johnson L (2006) How many interviews are enough? An experiment with data saturation and variability. Field Methods 18 (1):59–82CrossRef

Gupta Y, Khan Y, Gallaba K, McIntosh S (2017) The impact of the adoption of continuous integration on developer attraction and retention. In: International Conference on Mining Software Repositories (MSR), pages 491–494. IEEE

Hilton M, Nelson N, Tunnell T, Marinov D, Dig D (2017) Trade-offs in continuous integration: assurance, security, and flexibility. In: Joint meeting on foundations of software engineering (FSE), pp 197–207

Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: International conference on automated software engineering (ASE), pp 426–437. IEEE

Holmstrom H, Conchúir EÓ, Agerfalk J, Fitzgerald B (2006) Global software development challenges: A case study on temporal, geographical and socio-cultural distance. In: International conference on global software engineering (ICGSE), pp 3–11. IEEE

Jin X, Servant F (2020) A cost-efficient approach to building in continuous integration. In: International conference on software engineering (ICSE), pp 13–25. IEEE

Jin X, Servant F (2021) What helped, and what did not? An evaluation of the strategies to improve continuous integration. In: International conference on software engineering (ICSE), pp 213–225. IEEE

Jin X, Servant F (2022) Which builds are really safe to skip? Maximizing failure observation for build selection in continuous integration. J Syst Softw 188:111292CrossRef

Kim M, Zimmermann T, DeLine R, Begel A (2016) The emerging role of data scientists on software development teams. In: International conference on software engineering (ICSE), pp 96–107. IEEE

Kinsman T, Wessel M, Gerosa MA, Treude C (2021) How do software developers use GitHub actions to automate their workflows?. In: International conference on mining software repositories (MSR)

Kulas M, Borelli JL, Gässler W, Peter D, Rabien S, de Xivry GO, Busoni L, Bonaglia M, Mazzoni T, Rahmer G (2014) Practical experience with test-driven development during commissioning of the multi-star AO system ARGOS. In: Software and Cyberinfrastructure for Astronomy III, vol 9152, pp 110–119. SPIE

Leppänen M, Mäkinen S, Pagels M, Eloranta V-P, Itkonen J, Mäntylä MV, Männistö T (2015) The highways and country roads to continuous deployment. IEEE Softw 32(2):64–72CrossRef

Li Z, Liu W, Chen H, Wang X, Liao X, Xing L, Zha M, Jin H, Zou D (2022) Robbery on: DevOps understanding and mitigating illicit cryptomining on continuous integration service platforms. In: IEEE symposium on security and privacy (SP), pp 2397–2412. IEEE

Lu J, Yang Z, Qian J (2014) Implementation of continuous integration and automated testing in software development of smart grid scheduling support system. In: International conference on power system technology, pp 2441–2446. IEEE

Machiraju V (2021) Change in Azure pipelines grant for public projects. https://devblogs.microsoft.com/devops/change-in-azure-pipelines-grant-for-public-projects/. Accessed Oct 14 2022

Maxwell J (1992) Understanding and validity in qualitative research. Harvard educational review 62(3):279–301CrossRef

Mendy M, Rios N, Rybinski M (2020) The new pricing model for travis-ci.com. https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing. Accessed Oct 14 2022

Meyer AN, Barr ET, Bird C, Zimmermann T (2019) Today was a good day: the daily life of software developers. IEEE Trans Softw Eng 47(5):863–880CrossRef

Rahman A, Agrawal A, Krishna R, Sobran A (2018) Characterizing the influence of continuous integration: empirical results from 250+ open source and proprietary projects. In: ACM SIGSOFT international workshop on software Analytics, pp 8–14

Ralph P, Tempero E (2018) Construct validity in software engineering research and software metrics. In: International conference on evaluation and assessment in software engineering, pp 13–23

Rausch T, Hummer W, Leitner P, Schulte S (2017) An empirical analysis of build failures in the continuous integration workflows of Java-based open-source software. In: International conference on mining software repositories (MSR), pp 345–355. IEEE

Russel Bernard H, Wutich A, Gery WR (2016) Analyzing qualitative data: systematic approaches. SAGE Publications 2nd edn.

Savor T, Douglas M, Gentili M, Williams L, Beck K, Stumm M (2016) Continuous deployment at Facebook and OANDA. In: International conference on software engineering (ICSE), pp 21–30. IEEE

Shahin A, Babar MA, Zhu L (2017) Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE Access 5:3909–3943CrossRef

Soares E, Sizilio G, Santos J, da Costa DA, Kulesza U (2022) The effects of continuous integration on software development: a systematic literature review. Empir Softw Eng 27(3):1–61CrossRef

Ståhl D, Bosch J (2013) Experienced benefits of continuous integration in industry software product development: a case study. In: IASTED international conference on software engineering, pp 736–743

Valenzuela-Toledo P, Bergel A (2022) Evolution of GitHub Action workflows. In: International conference on software analysis, evolution and Reengineering (SANER). IEEE

Vasilescu B, Yu Y, Wang H, Devanbu P, Filkov V (2015) Quality and productivity outcomes relating to continuous integration in GitHub. In: Joint meeting on foundations of software engineering (FSE), pp 805–816

Vassallo C, Palomba F (2018) Continuous refactoring in CI: a preliminary study on the perceived advantages and barriers. In: International conference on software maintenance and evolution (ICSME), pp 564–568. IEEE

Vassallo C, Proksch S, Gall HC, Di Penta M (2019) Automated reporting of anti-patterns and decay in continuous integration. In: International conference on software engineering (ICSE), pp 105–115. IEEE

Widder DG, Hilton M, Kästner C, Vasilescu B (2019) A conceptual replication of continuous integration pain points in the context of Travis CI. In: Joint meeting on european software engineering conference and symposium on the foundations of software engineering (ESEC/FSE), pp 647–658

Widder D, Vasilescu B, Hilton Ml, Kästner C (2018) I’m leaving you, Travis: a continuous integration breakup story. In: International conference on mining software repositories (MSR), pp 165–169. IEEE

Zampetti F, Scalabrino S, Oliveto R, Canfora G, Di Penta M (2017) How open source projects use static code analysis tools in continuous integration pipelines. In: International conference on mining software repositories (MSR), pp 334–344. IEEE

Title: On the usage, co-usage and migration of CI/CD tools: A qualitative analysis
Authors: Pooya Rostami Mazrae
Tom Mens
Mehdi Golzadeh
Alexandre Decan
Publication date: 01-03-2023
Publisher: Springer US
Published in: Empirical Software Engineering / Issue 2/2023
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-022-10285-5

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 2/2023

An empirical study of text-based machine learning models for vulnerability detection

What is asked about technical debt (TD) on Stack Exchange question-and-answer (Q&A) websites? An observational study

What makes Ethereum blockchain transactions be processed fast or slow? An empirical study

Calling relationship investigation and application on Ethereum Blockchain System

Correction to: Advantages and disadvantages of (dedicated) model transformation languages

Semantically-enhanced topic recommendation systems for software projects

Premium Partner