Skip to main content
Top
Published in: Empirical Software Engineering 2/2018

04-11-2017

The impact of rapid release cycles on the integration delay of fixed issues

Authors: Daniel Alencar da Costa, Shane McIntosh, Christoph Treude, Uirá Kulesza, Ahmed E. Hassan

Published in: Empirical Software Engineering | Issue 2/2018

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

The release frequency of software projects has increased in recent years. Adopters of so-called rapid releases—short release cycles, often on the order of weeks, days, or even hours—claim that they can deliver fixed issues (i.e., implemented bug fixes and new features) to users more quickly. However, there is little empirical evidence to support these claims. In fact, our prior work shows that code integration phases may introduce delays for rapidly releasing projects—98% of the fixed issues in the rapidly releasing Firefox project had their integration delayed by at least one release. To better understand the impact that rapid release cycles have on the integration delay of fixed issues, we perform a comparative study of traditional and rapid release cycles. Our comparative study has two parts: (i) a quantitative empirical analysis of 72,114 issue reports from the Firefox project, and a (ii) qualitative study involving 37 participants, who are contributors of the Firefox, Eclipse, and ArgoUML projects. Our study is divided into quantitative and qualitative analyses. Quantitative analyses reveal that, surprisingly, fixed issues take a median of 54% (57 days) longer to be integrated in rapid Firefox releases than the traditional ones. To investigate the factors that are related to integration delay in traditional and rapid release cycles, we train regression models that model whether a fixed issue will have its integration delayed or not. Our explanatory models achieve good discrimination (ROC areas of 0.80–0.84) and calibration scores (Brier scores of 0.05–0.16) for rapid and traditional releases. Our explanatory models indicate that (i) traditional releases prioritize the integration of backlog issues, while (ii) rapid releases prioritize issues that were fixed in the current release cycle. Complementary qualitative analyses reveal that participants’ perception about integration delay is tightly related to activities that involve decision making, risk management, and team collaboration. Moreover, the allure of shipping fixed issues faster is a main motivator for adopting rapid release cycles among participants (although this motivation is not supported by our quantitative analysis). Furthermore, to explain why traditional releases deliver fixed issues more quickly, our participants point out the rush for integration in traditional releases and the increased time that is invested on polishing issues in rapid releases. Our results suggest that rapid release cycles may not be a silver bullet for the rapid delivery of new content to users. Instead, our results suggest that the benefits of rapid releases are increased software stability and user feedback.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Appendix
Available only for authorised users
Footnotes
22
From 2013 to 2016 by the time of this study.
 
24
We did not observe a statistically significant difference in integration delays between issues that are fixed by the reporters themselves and issues that are fixed by a different team member.
 
Literature
go back to reference Adams B, McIntosh S (2016) Modern release engineering in a nutshell: why researchers should care. In: Proceedings of the 23rd international conference on software analysis, evolution, and reengineering (SANER), pp 78–90 Adams B, McIntosh S (2016) Modern release engineering in a nutshell: why researchers should care. In: Proceedings of the 23rd international conference on software analysis, evolution, and reengineering (SANER), pp 78–90
go back to reference AlGhamdi HM, Syer MD, Shang W, Hassan AE (2016) An automated approach for recommending when to stop performance tests. In: Proceedings of the international conference on software maintenance and evolution. IEEE, Piscataway, pp 279–289 AlGhamdi HM, Syer MD, Shang W, Hassan AE (2016) An automated approach for recommending when to stop performance tests. In: Proceedings of the international conference on software maintenance and evolution. IEEE, Piscataway, pp 279–289
go back to reference Antoniol G, Ayari K, Penta MD, Khomh F, Guéhéneuc Y (2008) Is it a bug or an enhancement?: a text-based approach to classify change requests. In: Proceedings of the 2008 conference of the centre for advanced studies on collaborative research (CASCON), pp 23–37 Antoniol G, Ayari K, Penta MD, Khomh F, Guéhéneuc Y (2008) Is it a bug or an enhancement?: a text-based approach to classify change requests. In: Proceedings of the 2008 conference of the centre for advanced studies on collaborative research (CASCON), pp 23–37
go back to reference Baskerville R, Pries-Heje J (2004) Short cycle time systems development. Inf Syst J 14:237–264CrossRef Baskerville R, Pries-Heje J (2004) Short cycle time systems development. Inf Syst J 14:237–264CrossRef
go back to reference Baysal O, Davis I, Godfrey MW (2011) A tale of two browsers. In: Proceedings of the 8th working conference on mining software repositories (MSR). ACM, New York, pp 238–241 Baysal O, Davis I, Godfrey MW (2011) A tale of two browsers. In: Proceedings of the 8th working conference on mining software repositories (MSR). ACM, New York, pp 238–241
go back to reference Beck K (2000) Extreme programming explained: embrace change. Addison-Wesley Professional, Reading Beck K (2000) Extreme programming explained: embrace change. Addison-Wesley Professional, Reading
go back to reference Beller M, Gousios G, Zaidman A (2015) How (much) do developers test?. In: 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 2. IEEE, Piscataway, pp 559–562 Beller M, Gousios G, Zaidman A (2015) How (much) do developers test?. In: 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 2. IEEE, Piscataway, pp 559–562
go back to reference Boehm BW (1988) A spiral model of software development and enhancement. Computer 21(5):61–72CrossRef Boehm BW (1988) A spiral model of software development and enhancement. Computer 21(5):61–72CrossRef
go back to reference Charmaz K (2014) Constructing grounded theory. SAGE, Newbury Park Charmaz K (2014) Constructing grounded theory. SAGE, Newbury Park
go back to reference Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114:494–509CrossRef Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114:494–509CrossRef
go back to reference da Costa DA, Abebe SL, McIntosh S, Kulesza U, Hassan AE (2014) An empirical study of delays in the integration of addressed issues. In: Proceedings of the 30th international conference on software maintenance and evolution (ICSME), pp 281–290 da Costa DA, Abebe SL, McIntosh S, Kulesza U, Hassan AE (2014) An empirical study of delays in the integration of addressed issues. In: Proceedings of the 30th international conference on software maintenance and evolution (ICSME), pp 281–290
go back to reference da Costa DA, McIntosh S, Kulesza U, Hassan AE (2016) The impact of switching to a rapid release cycle on the integration delay of addressed issues: an empirical study of the mozilla firefox project. In: Proceedings of the 13th international workshop on mining software repositories. ACM, New York, pp 374–385 da Costa DA, McIntosh S, Kulesza U, Hassan AE (2016) The impact of switching to a rapid release cycle on the integration delay of addressed issues: an empirical study of the mozilla firefox project. In: Proceedings of the 13th international workshop on mining software repositories. ACM, New York, pp 374–385
go back to reference Dunn OJ (1964) Multiple comparisons using rank sums. Technometrics 6 (3):241–252CrossRef Dunn OJ (1964) Multiple comparisons using rank sums. Technometrics 6 (3):241–252CrossRef
go back to reference Efron B (1986) How biased is the apparent error rate of a prediction rule?. In: Journal of the american statistical association, Taylor & Francis, Milton Park, vol 81, pp 461–470 Efron B (1986) How biased is the apparent error rate of a prediction rule?. In: Journal of the american statistical association, Taylor & Francis, Milton Park, vol 81, pp 461–470
go back to reference Fisher RA (1925) Statistical methods for research workers. Genesis Publishing Pvt Ltd, New DelhiMATH Fisher RA (1925) Statistical methods for research workers. Genesis Publishing Pvt Ltd, New DelhiMATH
go back to reference Giger E, Pinzger M, Gall H (2010) Predicting the fix time of bugs. In: Proceedings of the 2nd international workshop on recommendation systems for software engineering (RSSE). ACM, New York, pp 52–56 Giger E, Pinzger M, Gall H (2010) Predicting the fix time of bugs. In: Proceedings of the 2nd international workshop on recommendation systems for software engineering (RSSE). ACM, New York, pp 52–56
go back to reference Greer D, Ruhe G (2004) Software release planning: an evolutionary and iterative approach. Inf Softw Technol 46:243–253CrossRef Greer D, Ruhe G (2004) Software release planning: an evolutionary and iterative approach. Inf Softw Technol 46:243–253CrossRef
go back to reference Harrell FE (2001) Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. Springer, New YorkCrossRefMATH Harrell FE (2001) Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. Springer, New YorkCrossRefMATH
go back to reference Herraiz I, German DM, Gonzalez-Barahona JM, Robles G (2008) Towards a simplification of the bug report form in eclipse. In: Proceedings of the 2008 international working conference on mining software repositories (MSR), pp 145–148 Herraiz I, German DM, Gonzalez-Barahona JM, Robles G (2008) Towards a simplification of the bug report form in eclipse. In: Proceedings of the 2008 international working conference on mining software repositories (MSR), pp 145–148
go back to reference Howell DC (2005) Median absolute deviation. In: Encyclopedia of statistics in behavioral science. Wiley Online Library, Hoboken Howell DC (2005) Median absolute deviation. In: Encyclopedia of statistics in behavioral science. Wiley Online Library, Hoboken
go back to reference Iasonos A, Schrag D, Raj GV, Panageas KS (2008) How to build and interpret a nomogram for cancer prognosis. In: Journal of clinical oncology, american society of clinical oncology, vol 26, pp 1364–1370 Iasonos A, Schrag D, Raj GV, Panageas KS (2008) How to build and interpret a nomogram for cancer prognosis. In: Journal of clinical oncology, american society of clinical oncology, vol 26, pp 1364–1370
go back to reference Jeong G, Kim S, Zimmermann T (2009) Improving bug triage with bug tossing graphs. In: Proceedings of the 7th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE). ACM, New York, pp 111–120 Jeong G, Kim S, Zimmermann T (2009) Improving bug triage with bug tossing graphs. In: Proceedings of the 7th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE). ACM, New York, pp 111–120
go back to reference Jiang Y, Adams B (2014) How much does integrating this commit cost? - a position paper. In: 2nd International Workshop on Release Engineering (RELENG) Jiang Y, Adams B (2014) How much does integrating this commit cost? - a position paper. In: 2nd International Workshop on Release Engineering (RELENG)
go back to reference Jiang Y, Adams B, German DM (2013) Will my patch make it? and how fast?: case study on the linux kernel. In: Proceedings of the 10th working conference on mining software repositories (MSR), pp 101–110 Jiang Y, Adams B, German DM (2013) Will my patch make it? and how fast?: case study on the linux kernel. In: Proceedings of the 10th working conference on mining software repositories (MSR), pp 101–110
go back to reference Kampstra P et al. (2008) Beanplot: a boxplot alternative for visual comparison of distributions. J Stat Softw 28:1–9CrossRef Kampstra P et al. (2008) Beanplot: a boxplot alternative for visual comparison of distributions. J Stat Softw 28:1–9CrossRef
go back to reference Karlsson J, Ryan K (1997) A cost-value approach for prioritizing requirements. IEEE Softw 14:67–74CrossRef Karlsson J, Ryan K (1997) A cost-value approach for prioritizing requirements. IEEE Softw 14:67–74CrossRef
go back to reference Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? an empirical case study of mozilla firefox. In: Proceedings of the 9th IEEE working conference on mining software repositories (MSR). IEEE, Piscataway, pp 179–188 Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? an empirical case study of mozilla firefox. In: Proceedings of the 9th IEEE working conference on mining software repositories (MSR). IEEE, Piscataway, pp 179–188
go back to reference Kruskal WH, Wallis WA (1952) Use of ranks in one-criterion variance analysis. J Am Stat Assoc 47(260):583–621CrossRefMATH Kruskal WH, Wallis WA (1952) Use of ranks in one-criterion variance analysis. J Am Stat Assoc 47(260):583–621CrossRefMATH
go back to reference Leys C, Ley C, Klein O, Bernard P, Licata L (2013) Detecting outliers: do not use standard deviation around the mean, use absolute deviation around the median. J Exp Soc Psychol 49:764–766CrossRef Leys C, Ley C, Klein O, Bernard P, Licata L (2013) Detecting outliers: do not use standard deviation around the mean, use absolute deviation around the median. J Exp Soc Psychol 49:764–766CrossRef
go back to reference Mäntylä MV, Adams B, Khomh F, Engström E, Petersen K (2014) On rapid releases and software testing: a case study and a semi-systematic literature review. In: Journal of Empirical Software Engineering. Springer, Berlin, pp 1–42 Mäntylä MV, Adams B, Khomh F, Engström E, Petersen K (2014) On rapid releases and software testing: a case study and a semi-systematic literature review. In: Journal of Empirical Software Engineering. Springer, Berlin, pp 1–42
go back to reference McIntosh S, Kamei Y, Adams B, Hassan AE (2016) An empirical study of the impact of modern code review practices on software quality. Empir Softw Eng 21 (5):2146–2189CrossRef McIntosh S, Kamei Y, Adams B, Hassan AE (2016) An empirical study of the impact of modern code review practices on software quality. Empir Softw Eng 21 (5):2146–2189CrossRef
go back to reference Morakot C, Hoa Khanh D, Truyen T, Aditya G (2015a) Characterization and prediction of issue-related risks in software projects. In: 12th international conference on mining software repositories (MSR), pp 280–291 Morakot C, Hoa Khanh D, Truyen T, Aditya G (2015a) Characterization and prediction of issue-related risks in software projects. In: 12th international conference on mining software repositories (MSR), pp 280–291
go back to reference Morakot C, Hoa Khanh D, Truyen T, Aditya G (2015b) Predicting delays in software projects using networked classification. In: 30th international conference on automated software engineering (ASE) Morakot C, Hoa Khanh D, Truyen T, Aditya G (2015b) Predicting delays in software projects using networked classification. In: 30th international conference on automated software engineering (ASE)
go back to reference Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density. In: Proceedings of the 27th international conference on software engineering (ICSE). IEEE, Piscataway, pp 284–292 Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density. In: Proceedings of the 27th international conference on software engineering (ICSE). IEEE, Piscataway, pp 284–292
go back to reference Paetsch F, Eberlein A, Maurer F (2003) Requirements engineering and agile software development. In: Twelfth IEEE international workshops on enabling technologies: infrastructure for collaborative enterprises, pp 308–313 Paetsch F, Eberlein A, Maurer F (2003) Requirements engineering and agile software development. In: Twelfth IEEE international workshops on enabling technologies: infrastructure for collaborative enterprises, pp 308–313
go back to reference Panjer LD (2007) Predicting eclipse bug lifetimes. In: Proceedings of the 4th international workshop on mining software repositories (MSR), p 29 Panjer LD (2007) Predicting eclipse bug lifetimes. In: Proceedings of the 4th international workshop on mining software repositories (MSR), p 29
go back to reference Rahman MT, Rigby PC (2015) Release stabilization on linux and chrome. In: IEEE software journal. vol 2. IEEE, Piscataway, pp 81–88 Rahman MT, Rigby PC (2015) Release stabilization on linux and chrome. In: IEEE software journal. vol 2. IEEE, Piscataway, pp 81–88
go back to reference Regnell B, Brinkkemper S (2005) Market-driven requirements engineering for software products. In: Engineering and managing software requirements, pp 287–308 Regnell B, Brinkkemper S (2005) Market-driven requirements engineering for software products. In: Engineering and managing software requirements, pp 287–308
go back to reference Sandelowski M (1995) Sample size in qualitative research. Res Nurs Health 18 (2):179–183CrossRef Sandelowski M (1995) Sample size in qualitative research. Res Nurs Health 18 (2):179–183CrossRef
go back to reference Schroter A, Bettenburg N, Premraj R (2010) Do stack traces help developers fix bugs?. In: 2010 7th IEEE working conference on mining software repositories (MSR). IEEE, Piscataway, pp 118–121 Schroter A, Bettenburg N, Premraj R (2010) Do stack traces help developers fix bugs?. In: 2010 7th IEEE working conference on mining software repositories (MSR). IEEE, Piscataway, pp 118–121
go back to reference Schwaber K (1997) Scrum development process. In: Business object design and implementation. Springer, Berlin, pp 117–134 Schwaber K (1997) Scrum development process. In: Business object design and implementation. Springer, Berlin, pp 117–134
go back to reference Shihab E, Ihara A, Kamei Y, Ibrahim WM, Ohira M, Adams B, Hassan AE, Matsumoto K (2010) Predicting re-opened bugs: a case study on the eclipse project. In: Proceedings of 17th working conference on reverse engineering (WCRE). IEEE, Piscataway, pp 249–258 Shihab E, Ihara A, Kamei Y, Ibrahim WM, Ohira M, Adams B, Hassan AE, Matsumoto K (2010) Predicting re-opened bugs: a case study on the eclipse project. In: Proceedings of 17th working conference on reverse engineering (WCRE). IEEE, Piscataway, pp 249–258
go back to reference Shimagaki J, Kamei Y, McIntosh S, Pursehouse D, Ubayashi N (2016) Why are commits being reverted?: a comparative study of industrial and open source projects. In: 2016 IEEE international conference on software maintenance and evolution (ICSME). IEEE, Piscataway, pp 301–311 Shimagaki J, Kamei Y, McIntosh S, Pursehouse D, Ubayashi N (2016) Why are commits being reverted?: a comparative study of industrial and open source projects. In: 2016 IEEE international conference on software maintenance and evolution (ICSME). IEEE, Piscataway, pp 301–311
go back to reference Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: 6th international workshop on cooperative and human aspects of software engineering (CHASE), pp 89–92 Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: 6th international workshop on cooperative and human aspects of software engineering (CHASE), pp 89–92
go back to reference Souza R, Chavez C, Bittencourt RA (2014) Do rapid releases affect bug reopening? a case study of firefox. In: Proceedings of the brazilian symposium on software engineering (SBES). IEEE, Piscataway, pp 31–40 Souza R, Chavez C, Bittencourt RA (2014) Do rapid releases affect bug reopening? a case study of firefox. In: Proceedings of the brazilian symposium on software engineering (SBES). IEEE, Piscataway, pp 31–40
go back to reference Souza R, Chavez C, Bittencourt R (2015) Rapid releases and patch backouts: a software analytics approach. In: IEEE software journal, vol 32. IEEE, Piscataway, pp 89–96 Souza R, Chavez C, Bittencourt R (2015) Rapid releases and patch backouts: a software analytics approach. In: IEEE software journal, vol 32. IEEE, Piscataway, pp 89–96
go back to reference Spearman C (1904) The proof and measurement of association between two things. Am J Psychol 15(1):72–101CrossRef Spearman C (1904) The proof and measurement of association between two things. Am J Psychol 15(1):72–101CrossRef
go back to reference Subramaniam C, Sen R, Nelson ML (2009) Determinants of open source software project success: a longitudinal study. In: Journal of decision support systems, vol 46. Elsevier, Amsterdam, pp 576–585 Subramaniam C, Sen R, Nelson ML (2009) Determinants of open source software project success: a longitudinal study. In: Journal of decision support systems, vol 46. Elsevier, Amsterdam, pp 576–585
go back to reference Tian Y, Ali N, Lo D, Hassan AE (2015) On the unreliability of bug severity data. Empir Softw Eng 21:1–26 Tian Y, Ali N, Lo D, Hassan AE (2015) On the unreliability of bug severity data. Empir Softw Eng 21:1–26
go back to reference Wilks DS (2011) Statistical methods in the atmospheric sciences, vol 100. Academic Press, Cambridge Wilks DS (2011) Statistical methods in the atmospheric sciences, vol 100. Academic Press, Cambridge
go back to reference Zaman S, Adams B, Hassan AE (2011) Security versus performance bugs: a case study on firefox. In: Proceedings of the 8th working conference on mining software repositories. ACM, New York, pp 93–102 Zaman S, Adams B, Hassan AE (2011) Security versus performance bugs: a case study on firefox. In: Proceedings of the 8th working conference on mining software repositories. ACM, New York, pp 93–102
Metadata
Title
The impact of rapid release cycles on the integration delay of fixed issues
Authors
Daniel Alencar da Costa
Shane McIntosh
Christoph Treude
Uirá Kulesza
Ahmed E. Hassan
Publication date
04-11-2017
Publisher
Springer US
Published in
Empirical Software Engineering / Issue 2/2018
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI
https://doi.org/10.1007/s10664-017-9548-7

Other articles of this Issue 2/2018

Empirical Software Engineering 2/2018 Go to the issue

Premium Partner