Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Bücher
Zeitschriften
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
SLX-Digitalkonferenz: Zukunftswerkstatt 2024

Mehr Umsatz mit Top-Kontakten

Am 7. Mai erfahren Sie, wie Vertriebsteams Leadgenerierung, Leadnurturing und Leadstrategien effizient steuern können.
Erweiterte Suche
Anmelden
nach oben
Empirical Software Engineering
Erschienen in: Empirical Software Engineering 5/2015

01.10.2015

On rapid releases and software testing: a case study and a semi-systematic literature review

verfasst von: Mika V. Mäntylä, Bram Adams, Foutse Khomh, Emelie Engström, Kai Petersen

Erschienen in: Empirical Software Engineering | Ausgabe 5/2015

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Large open and closed source organizations like Google, Facebook and Mozilla are migrating their products towards rapid releases. While this allows faster time-to-market and user feedback, it also implies less time for testing and bug fixing. Since initial research results indeed show that rapid releases fix proportionally less reported bugs than traditional releases, this paper investigates the changes in software testing effort after moving to rapid releases in the context of a case study on Mozilla Firefox, and performs a semi-systematic literature review. The case study analyzes the results of 312,502 execution runs of the 1,547 mostly manual system-level test cases of Mozilla Firefox from 2006 to 2012 (5 major traditional and 9 major rapid releases), and triangulates our findings with a Mozilla QA engineer. We find that rapid releases have a narrower test scope that enables a deeper investigation of the features and regressions with the highest risk. Furthermore, rapid releases make testing more continuous and have proportionally smaller spikes before the main release. However, rapid releases make it more difficult to build a large testing community , and they decrease test suite diversity and make testing more deadline oriented. In addition, our semi-systematic literature review presents the benefits, problems and enablers of rapid releases from 24 papers found using systematic search queries and a similar amount of papers found through other means. The literature review shows that rapid releases are a prevalent industrial practice that are utilized even in some highly critical domains of software engineering, and that rapid releases originated from several software development methodologies such as agile, open source, lean and internet-speed software development. However, empirical studies proving evidence of the claimed advantages and disadvantages of rapid releases are scarce.
Vorheriger Artikel Automated prediction of bug report priority using multi-factor analysis

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren
Literatur
Agarwal P (2011) Continuous scrum: Agile management of saas products, pp 51–60
Azham Z, Ghani I, Ithnin N (2011) Security backlog in scrum security practices, pp 414–417
Baskerville R, Ramesh B, Levine L, Pries-Heje J (2006) High-speed software development practices: What works, what doesn’t. IT Professional 8:29–36CrossRef
Baysal O, Davis I, Godfrey MW (2011) A tale of two browsers. In: Proceedings of the 8th Working Conf. on Mining Software Repositories (MSR), pp 238–241
Baysal O, Kononenko O, Holmes R, Godfrey M (2012) The secret life of patches: a firefox case study, pp 447–455
Beck K, Andres C (2004) Extreme Programming explained: embrace change, 2nd edn. Addison-Wesley
Begel A, Nagappan N (2007) Usage and perceptions of agile software development in an industrial context: an exploratory study, pp 255–264
Bell R, Ostrand T, Weyuker E (2006) Looking for bugs in all the right places, vol. 2006, pp 61–71
Berner S, Weber R, Keller RK (2005) Observations and lessons learned from automated testing. In: Proceedings of the 27th international conference on Software engineering, ACM, pp 571–579
Boshuizen C, Marshall W, Bridges C, Kenyon S, Klupar P (2011) Learning to follow: embracing commercial technologies and open source for space missions 10:8097–8101
Brown AW (2011) A case study in agile-at-scale delivery. In: Proceeding of the 12th Intl. Conf. on Agile Processes in Software Engineering and Extreme Programming (XP), vol 77, pp 266–281
Byrt T, Bishop J, Carlin JB (1993) Bias, prevalence and kappa. J Clin Epidemiol 46:423–429CrossRef
Carney PA, Sickles EA, Monsees BS, Bassett LW, Brenner RJ, Feig SA, Smith RA, Rosenberg RD, Bogart TA, Browning S et al (2010) Identifying minimally acceptable interpretive performance criteria for screening mammography 1. Radiology 255:354–361CrossRef
Cicchetti DV, Feinstein AR (1990) High agreement but low kappa: Ii. resolving the paradoxes. J Clin Epidemiol 43:551–558CrossRef
Codabux Z, Williams B (2013) Managing technical debt: an industrial case study, pp 8–15
Cohan S (2007) Successful integration of agile development techniques within disa, pp 255–260
Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Academic Press
Cusumano MA, Yoffie DB (1999) Competing on internet time: Lessons from netscape and its battle with microsoft. Simon and Schuster
Gamma E (2005) Agile, open source, distributed, and on-time – inside the eclipse development process. Keynote at the 27th Intl. Conf. on Software Engineering (ICSE)
Greiler M, van Deursen A, Zaidman A (2012) Measuring test case similarity to support test suite understanding. In: Objects, models components patterns. Springer, pp 91–107
Gundebahar M, Kus Khalilov M (2013) A hybrid deployment model for financial systems with service oriented architecture (soa): Running from client via branch server, pp 365–370
Heikkilä V, Rautiainen K, Jansen S (2010) A revelatory case study on scaling agile release planning, pp 289–296
Hemmati H, Arcuri A, Briand L (2013) Achieving scalable model-based testing through test case diversity. ACM Trans Softw Eng Methodol (TOSEM) 22:6CrossRef
Hodgetts P, Phillips D (2002) 30. In: eXtreme Adoption eXperiences of a B2B Start Up. Addison-Wesley Longman Publishing Co., Inc. Extreme Programming Perspectives
Hollander M, Wolfe DA (1999) Nonparametric statistical methods, 2nd edn. John Wiley and Sons inc
HP (2012) Shorten release cycles aby bringing developers to application lifecycle management. HP applications handbook, Retrieved on February 08, 2012
Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation, 1st edn. Addison-Wesley Professional
InvestmentWatch (2011) Mozilla puts out firefox 5.0 web browser which carries over 1,000 improvements in just about 3 months of development. http://​bit.​ly/​aecRrL
Iwai A, Aoyama M (2011) Automotive cloud service systems based on service-oriented architecture and its evaluation, pp 638–645
Jansen S, Brinkkemper S (2006) Ten misconceptions about product software release management explained using update cost/value functions. In: Proceedings of the International Workshop on Software Product Management, pp 44–50
Jenkins J (2011) Velocity culture (the unmet challenge in ops). Presentation at O’Reilly Velocity Conference
Jones G, Leung V (2005) Visual surveillance: a systems engineering approach for rapid development. Number 2005-11033, pp 161–166
Kaply M (2012) Why do companies stay on old technology? Retrieved on January 12, 2012
Kettunen V, Kasurinen J, Taipale O, Smolander K (2010) A study on agility and testing processes in software organizations. In: Proc. of the 19th Intl. Symp. on Software Testing and Analysis (ISSTA), pp 231–240
Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? An empirical case study of mozilla firefox. In: MSR, pp 179–188
Kitchenham B (2004) Procedures for performing systematic reviews, vol. 33:2004
Kong S, Kendall JE, Kendall KE (2009) The challenge of improving software quality: Developers’ beliefs about the contribution of agile practices. In: Proc. of the Americas Conf. on Information Systems (AMCIS), p 12
Kühner G, Bluhm T, Heimann P, Hennig C, Kroiss H, Krom J, Laqua H, Lewerentz M, Maier J, Schacht J, Spring A, Werner A, Zilker M (2012) Progress on standardization and automation in software development on w7x. Fusion Eng Des 87:2232–2237CrossRef
Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics159–174
Lavoie T, Merlo E (2013) How much really changes? a case study of firefox version evolution using a clone detector, pp 83–89
Li J, Moe NB, Dybå T (2010) Transition from a plan-driven process to scrum: a longitudinal case study on software quality. In: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp 13:1–13:10
Li X, Li Y, Xie M, Ng S (2011) Reliability analysis and optimal version-updating for open source software. Inf Softw Technol 53:929–936CrossRef
Maalej W (2009) Task-first or context-first? tool integration revisited, pp 344–355
Mäntylä MV, Itkonen J (2013) More testers âĂŞ the effect of crowd size and time restriction in software testing. Inf Softw Technol 55:986–1003CrossRef
Mantyla M, Khomh F, Adams B, Engstrom E, Petersen K (2013) On rapid releases and software testing. In: Proceedings of the 29th IEEE International Conference on Software Maintenance (ICSM), pp 20–29
Marschall M (2007) Transforming a six month release cycle to continuous flow. In: Proc. of the conf. on AGILE, pp 395–400
Martin D, Rooksby J, Rouncefield M, Sommerville I (2007) ’good’organisational reasons for’bad’software testing: An ethnographic study of testing in a small software company. In: Software Engineering, 2007. ICSE 2007. 29th International Conference on, IEEE, pp 602–611
McDaniel LS (1990) The effects of time pressure and audit program structure on audit performance. J Account Res 28:267–285CrossRef
Middleton P (2001) Lean software development two case studies. Software Qual J 9:241–252CrossRef
Middleton P, Joyce D (2012) Lean software management: Bbc worldwide case study. IEEE Trans Eng Manag 59:20–32CrossRef
Mozilla (2013) Mozilla source code mercurial repositories
Olsson H, Alahyari H, Bosch J (2012) Climbing the “stairway to heaven” - a mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software, pp 392–399
Olsson H, Bosch J, Alahyari H (2013) Towards r&d as innovation experiment systems: a framework for moving beyond agile software development, pp 798–805
Otte T, Moreton R, Knoell HD (2008) Applied quality assurance methods under the open source development model. In: Proceeding of the 32nd Annual IEEE Intl. Computer Software and Applications Conf. (COMPSAC), pp 1247–1252
Oza N, Ebert C, Abrahamsson P (2012) Lean software development. IEEE Softw 29:0022–25
Paasivaara M, Lassenius C (2003) Collaboration practices in global inter-organizational software development projects. Softw Process Improv Pract 8:183–199CrossRef
Patel C, Lycett M, Macredie R, De Cesare S (2006) Perceptions of agility and collaboration in software development practice, vol. 1, p 10c
Pérez-Castillo R, Sánchez-González L, Piattini M, García F, Garcia-Rodriguez de Guzman I (2011) Obtaining thresholds for the effectiveness of business process mining. In: Empirical software engineering and measurement (ESEM), 2011 International Symposium on IEEE, pp 453–462
Petersen K, Wohlin C (2009) A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case. J Syst Softw 82:1479–1490CrossRef
Petersen K, Wohlin C (2010) The effect of moving from a plan-driven to an incremental software development approach with agile practices. Empir Softw Eng 15:654–693CrossRef
Poppendieck M (2007) Lean software development. In: Companion to the proceedings of the 29th International Conference on Software Engineering, IEEE Computer Society, pp 165–166
Porter A, Yilmaz C, Memon AM, Krishna AS, Schmidt DC, Gokhale A (2006) Techniques and processes for improving the quality and performance of open-source software. Softw Process Improv Pract 11:163–176CrossRef
Rafi DM, Moses KRK, Petersen K, Mäntylä MV (2012) Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. In: Automation of Software Test (AST), 2012 7th International Workshop on, IEEE, pp 36–42
Raymond ES (1999) The Cathedral and the Bazaar, 1st edn. O’Reilly & Associates, Inc., Sebastopol
Reinertsen DG (2009) The principles of product development flow: second generation lean product development. Celeritas Redondo Beach, Canada
Ruhe G, Saliu MO (2005) The art and science of software release planning. IEEE Softw 22:47–53CrossRef
Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14:131–164CrossRef
Schwaber K (1995) Scrum development process. In: Proceedings of the 10th Annual ACM Conference on Object Oriented Programming Systems, Languages, and Applications (OOPSLA), pp 117–134
Sonnentag S, Brodbeck FC, Heinbokel T, Stolte W (1994) Stressor-burnout relationship in software development teams. J Occup Organ Psychol 67:327–341CrossRef
Sundmark D, Petersen K, Larsson S (2011) An exploratory case study of testing in an automotive electrical system release process, pp 166–175
Szőke Á (2011) Conceptual scheduling model and optimized release scheduling for agile environments. Inf Softw Technol 53:574–591CrossRef
Torkar R, Minoves P, Garrigós J (2011) Adopting free/libre/open source software practices, techniques and methods for industrial use. J Assoc Inf Syst 12:88–122
Metadaten
Titel
On rapid releases and software testing: a case study and a semi-systematic literature review
verfasst von
Mika V. Mäntylä
Bram Adams
Foutse Khomh
Emelie Engström
Kai Petersen
Publikationsdatum
01.10.2015
Verlag
Springer US
Erschienen in
Empirical Software Engineering / Ausgabe 5/2015
Print ISSN: 1382-3256
Elektronische ISSN: 1573-7616
DOI
https://doi.org/10.1007/s10664-014-9338-4

Weitere Artikel der Ausgabe 5/2015

Empirical Software Engineering 5/2015 Zur Ausgabe

Editorial

Introduction to the special issue on software maintenance and evolution research

OriginalPaper

Automated prediction of bug report priority using multi-factor analysis

OriginalPaper

Developer initiation and social interactions in OSS: A case study of the Apache Software Foundation

OriginalPaper

How the Apache community upgrades dependencies: an evolutionary study

OriginalPaper

Should I follow this fault localization tool’s output?

OriginalPaper

Supporting and accelerating reproducible empirical research in software evolution and maintenance using TraceLab Component Library