Systematic literature review on the impacts of agile release engineering practices
Introduction
In software development, fast, incremental deliveries involve lightweight, efficient practices for continuous release planning [1] and release engineering [2], [3]. This paper focuses on engineering by synthesizing empirical studies for agile release engineering (ARE) practices. To the best of our knowledge, the concept of ARE has not been used before in other scientific papers. We use it in this paper to map the research topic and to incorporate the investigated release engineering practices that are involved in a “release engineering pipeline” [3]. Hence, by ARE practices, we mean contemporary software integration, testing, deployment and release practices that are often applied in modern release engineering [3]. Many of these practices originate from agile software development methodologies such as extreme programming (XP) [4]. From the research point of view, ARE practices derive from the theories of agile and lean software development [5], release engineering [2], [3], [6] and continuous software engineering [7], [8] research disciplines. According to Adams et al. [6], release engineering “deals with all activities in between regular development and delivery of a software product to the end user, i.e., integration, build, test execution, packaging and delivery of software.” Continuous software engineering is an emerging subtopic in software engineering (SE) that is focused on continuous experimentation, innovation and the elimination of discontinuities within and between the developmental, operational and business strategy functions. Fitzgerald and Stol [8] associate concepts in continuous software engineering with concepts of classic “lean thinking” [9] such as “value and waste,” “flow and batch size,” “autonomation and building-in quality” and “Kaizen and continuous improvement.”
Modern ARE practices are supposed to aid in delivering software faster and cheaper to end users; hence, claims of such impacts should be validated by rigorous and relevant empirical studies. In this systematic review, the objective is to understand the direct and indirect impacts of ARE practices. By investigating the direct and indirect impacts we emphasize the notion that “impacts may be desired already according to the explicit method rationale(s), or they may be unexpected, sometimes even unwanted[10].” In addition, we sought to evaluate primary studies to understand how the impacts of ARE practices have been investigated in empirical studies. The research questions are:
RQ1: What are the direct and indirect impacts of ARE practices?
We break down the main question into three sub-questions as follows:
RQ1a: What are the impacts associated with adoption of ARE practices?
RQ1b: What is the prevalence of ARE practices?
RQ1c: What are the impacts of ARE practices on the success of SW development?
In addition, we define a second main research question to understand how these impacts have been investigated:
RQ2: How have ARE practices been investigated in empirical studies?
In this study, we systematically searched and analyzed empirical studies investigating 1) continuous integration (CI) [11], [12], 2) continuous delivery (CD) [12], 3) rapid release (RR) [13] and 4) continuous deployment (CD2) [11], [12]. We analyzed and clustered studies by topic and research approach, and outlined a checklist for analyzing software development capabilities for CD2 in the context of software-intensive products. We applied a systematic literature review (SLR) [14] method that allowed us to critically compare, evaluate and synthesize the primary studies. Our main selection criterion for the primary studies was that they were conducted in real software development contexts. Literature reviews, mapping studies, opinion papers and small-scale experiments with students are not included in our analysis. To the best of our knowledge, a systematic synthesis focusing on the impacts of ARE practices has not previously been undertaken, although some of the practices have been synthesized either separately or from a different research question point of view, as we explain in more detail in the following section. With this systematic review, we aim to provide a reliable overview of the current state of existing empirical studies for ARE practices that may help in terms of scoping and planning future studies. Our study also helps practitioners to better understand the impacts and capabilities associated with ARE practices. Finally, this paper aims to contribute to the theorizing on software development practices [10] for ARE. The concepts used in this paper (i.e., learning, practice, development context, rationale, impact and theory) conform to definitions used for the Coat Hanger model [10].
Section snippets
Background
ARE practices aim to support the agile principle of “early and continuous delivery of valuable software [15].” Early and continuous deliveries allow mechanisms for fast feedback and transparency of the development process, allowing stakeholders to continuously review and evaluate the state of the system under development and, if needed, to make adjustments to the priority and content requirements accordingly. CI practice originated from the agile XP methodology. Beck and Andres [4] summarizes
Research design and implementation
We applied an SLR [14], [31] method in our study. The research was conducted in seven key phases: 1) Research planning; 2) Piloting of the search; 3) Searching articles; 4) Inclusion and exclusion of articles; 5) Quality assessment of primary studies; 6) Analysis and clustering of primary studies; and 7) Reporting the results of the SLR. The first version of the research plan and protocol was written and reviewed in a research group comprising all the authors. Later, the plan and protocol were
Clustering and assessment of the primary studies
This section presents the results of the assessment on the rigor of the primary study and its relevance to the industry. In addition, we cluster articles based on their research approach and main topics in terms of providing an answer to RQ2 (How have ARE practices been investigated in empirical studies?).
After undertaking the inclusion steps presented in the previous section, the articles were read multiple times and analyzed using the NVivo tool [32] to extract data for the synthesis and
Analysis of the impacts of ARE practices
This section elaborates the key findings from the primary studies for providing the answer to RQ1 (What are the direct and indirect impacts of ARE practices?). To structure the results we have specified three sub-questions: RQ1a (What are the impacts associated with adoption of ARE practices?), RQ1b (What is the prevalence of ARE practices?) and RQ1c (What are the impacts of ARE practices on the success of SW development?).
In Section 5.1, we focus on adoption of the practice. This viewpoint
Discussion on research implications
This section continues with the results’ interpretation and further discusses possible implications of the study. In the discussion, we also elaborated on existing research gaps and offered approaches for conducting future empirical studies on ARE practices.
Limitations of the study and future research ideas
In literature reviews, the main validity threats and limitations typically lie in the search, selection and analysis of the primary studies. As explained earlier, we performed search, selection and quality evaluations with two authors and cross-checked each decision to avoid the subjective selection of primary studies to be used in the analysis. Furthermore, the analysis and findings of the paper were reviewed by all of the authors. Our search was carried out using the specific concepts of
Conclusion
This paper presented a synthesis of the impacts of ARE practices by analyzing empirical studies that had investigated continuous integration, continuous delivery, continuous deployment, and rapid release. One of our main research objectives was to understand how ARE practices have been investigated in empirical studies (RQ2). Our results indicated that 33 out of 71 primary studies were experience reports that had neither an explicit research method nor a data collection approach specified.
Acknowledgements
This work was supported by TEKES as part of the Need for Speed Project (http://www.n4s.fi/) of DIMECC (Digital, Internet, Materials & Engineering Co-Creation).
References (99)
- et al.
Theorizing about software development practices
Sci. Comput. Program.
(2015) - et al.
Modeling continuous integration practice differences in industry software development
J. Syst. Softw.
(2014) - et al.
A systematic review of systematic review process research in software engineering
Inf. Softw. Technol.
(2013) - et al.
On the journey to continuous deployment: technical and social challenges along the way
Inf. Softw. Technol.
(2015) Continuous delivery: huge benefits, but challenges too
IEEE Softw.
(2015)Product Release Planning Methods, Tools and Applications
(2010)- et al.
The practice and future of release engineering: a roundtable with three release engineers
IEEE Softw.
(2015) - et al.
Modern release engineering in a nutshell: why researchers should care
- et al.
Extreme Programming Explained: Embrace Change
(2004) - et al.
Lean software development: a tutorial
IEEE Softw.
(2012)
1st International workshop on release engineering (RELENG 2013)
Continuous software engineering: an introduction
Contin. Softw. Eng.
Continuous software engineering: a roadmap and agenda
J. Syst. Softw.
Lean Thinking: Banish Waste and Create Wealth in Your Corporation
Continuous Integration: Improving Software Quality and Reducing Risk
Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
On rapid releases and software testing: a case study and a semi-systematic literature review
Empir. Softw. Eng.
Fit for continuous integration: how organizations assimilate an agile practice
Continuous deployment of software intensive products and services: a systematic mapping study
J. Syst. Softw.
Climbing the “Stairway to heaven” - A mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software
Building Products as Innovation Experiment Systems
Hitting the target: practices for moving toward innovation experiment systems
Climbing the stairway to heaven: evolving from agile development to continuous deployment of software
Towards agile and beyond: An empirical account on the challenges involved when advancing software development practices
Towards R&D as innovation experiment systems: a framework for moving beyond agile software development
Software Development as an Experiment System: A Qualitative Survey on the State of the Practice
Do faster releases improve software quality? An empirical case study of Mozilla Firefox
On rapid releases and software testing
NVivo
A method for evaluating rigor and industrial relevance of technology evaluations
Empir. Softw. Eng.
Emergence of agile methods: perceptions from software practitioners in Malaysia
Investigating the extreme programming system - An empirical study
Empir. Softw. Eng.
Extreme programming in action: a longitudinal case study
Influences on agile practice tailoring in enterprise software development
Requirements to Pervasive System Continuous Deployment
Agile processes
Software Engineering, and Extreme Programming
The highways and country roads to continuous deployment
IEEE Softw.
Continuous Integration and Automation for DevOps
DevOps patterns to scale web applications using cloud services
DevOps: A Definition and Perceived Adoption Impediments
Dimensions of DevOps.
Moving targets: security and rapid-release in Firefox
Factors impacting rapid releases: an industrial case study
Understanding the impact of rapid releases on software quality: the case of Firefox
Empir. Softw. Eng.
Cited by (35)
Test automation maturity improves product quality—Quantitative study of open source projects using continuous integration
2022, Journal of Systems and SoftwareCitation Excerpt :Changing customer requirements and emerging technologies are driving the frequent changes to software products (Fitzgerald and Stol, 2017; Karvonen et al., 2017).
On the effects of continuous delivery on code quality: A case study in industry
2022, Computer Standards and InterfacesCitation Excerpt :Karvonen et al. [26] reviewed literature about the implications of continuous delivery practices on software projects. For this, Karvonen [26] analyzed 71 primary studies which he selected systematically from 619 articles. The analyzed articles were published between 2005 and 2015.
Archetypes of delay: An analysis of online developer conversations on delayed work items in IBM Jazz
2021, Information and Software TechnologyCitation Excerpt :Such tools can help speed up coding processes that use a single pieces of text as an objects for topic identification [36,37]. This is the case for a priori-based coding where categories are available or can be easily deducted, as can be the case for, e.g., survey or interview transcript data [38]. In our analysis, we applied a conventional content analysis technique [39] to examine, review the conversations, and identify the main discussion topics as codes.
Management of quality requirements in agile and rapid software development: A systematic mapping study
2020, Information and Software TechnologyCitation Excerpt :However, Alsaqaf et al. [10] did so only in the context of large-scale ASD, whereas Villamizar et al. [22] examined only how security is handled in ASD. Existing secondary studies of RSD [8,23–25] also do not specifically focus on QRs. This makes it difficult for researchers and practitioners to obtain a clear understanding of QRs and their management in ASD and RSD.
Test Case Prioritization in Continuous Integration environments: A systematic mapping study
2020, Information and Software TechnologyCitation Excerpt :3 In the literature we can find some mappings, surveys and systematic reviews on continuous integration practices and environments [5,12], on regression testing [2], test case prioritization techniques in general [13–16], and on software testing in agile and continuous integration context [17–20]. Such works do not specifically address the subject test case prioritization in continuous integration environments.
The impact of a continuous integration service on the delivery time of merged pull requests
2023, Empirical Software Engineering