1 Introduction
2 Prior related work
-
Governance. There is a sharp contrast in terms of inclusiveness and control across these two settings. While a single firm releasing proprietary software has more control over its development processes, the network of firms needs to accommodate multiple and often conflicting agendas. Release management in open source software ecosystems needs to accommodate that participants can have different and competing business models and thereby different motivations to engage and contribute. Furthermore, release management in open source projects is more exposed to different mental models, different corporate and individual cultures in a setting or irrevocable openness [26]. In addition, open source ecosystems also tend to be more inclusive to third-party contributors (e.g., students, academics or users among others that do not need a certain organizational affiliation or license to contribute). The power and the influence on deciding what is and what is not released are shared across multiple and heterogeneous participants [16].
-
Transparency. While it is common for the single firm releasing proprietary software to hide information about the software being released (e.g., the source code, the bug tracking information), the same is not customary in open source software ecosystems where access and transparency are required for the community to function. It is also worth remarking that while most open source software is released on the Internet (i.e. to a repository or a website where it can be consumed by others), much proprietary software is only considered released once deployed at the customers’ production environment [24].
-
Market. While a single firm releasing proprietary software has more control over the commercialization of the software and its complements, in the case of open source software ecosystems, the value exploration around a component can not be fully controlled by a single firm. Multiple firms can compete for the value around the ’common’ software being released. In the case of OpenStack, when deciding whether a certain open source component is released or not, a number of issues can be raised, such as ’Will the component compete with some of our proprietary offerings (e.g., plug-ins)?’, ’Will the component allow us to win some consulting and deployment contracts?’, ’Will the component increase the demand for our hardware?’or ’Will the component increase the demand for our hosting infrastructure?’ among other market-related issues that can impact release management.
-
Intellectual property rights. Releasing software can create prior art in terms of intellectual property rights. Single firms releasing proprietary software tend to deal more with intellectual property protection issues. In open software ecosystems, there might be intellectual property issues as well, but they often need to be resolved across multiple participants. Furthermore, by releasing under an open source software license, the developers or firms are giving up rights that are automatically granted by law. Note that it is not uncommon for firms to submit related patents prior to releasing a certain software component. This aspect is more relevant in markets where software can be patentable (USA, Japan, and South Korea) [27]. From the point of view of brands and trademarks, there is also a sharp contrast. While a single firm often has control over the trademark of a given software product, in the case of open source software such control is often multi-lateral and negotiated across the different individuals and organizations that contribute to the joint development of open source software. After all, the quality of a given release can affect the value of the brand and the trademark associated with the software being released.
-
feature freeze no new functionality can be added, the focus should be on removing defects;
-
string freeze no messages displayed by the program, such as error messages, can be changed. This allows translating as many messages as possible before the release;4
-
code freeze a permission is required to make any changes, even to fix bugs.
3 Empirical background
4 Methodological design
5 Results: a processual view
5.1 Early days at NASA
5.2 Bootstrap as an open source project
’The most obvious value of release cycles is to help producing a release. It allows us to shift focus from feature development to release-critical bugfixing, which results in better quality in the end release. But to me, the most important value of release cycles is that they create a common rhythm of contribution, a common pulse, which is essential for our virtual and global community to feel part of the same project’ — Thierry Carrez, 1 July 201311.
’You may remember that we used to have 3-month cycles at the beginning of OpenStack. Currently, it takes us about 4 weeks after we stop adding features to come up with a valid release candidate. In a 6-month release cycle, it’s acceptable to be feature-frozen for one month. In a 3-month release cycle, less so. Releasing every 3 months also means maintaining twice as many stable branches. So if more people addressed critical bugs during the rest of the cycle (when we are not feature-frozen) and more people helped with stable branch maintenance and security updates, we could definitely consider going for 3-month cycles. I like to have a Design Summit at the beginning of each cycle (I think it helps us deliver better results), so we’d probably also have to convince the Foundation to pay for twice as many developer events’ — Thierry Carrez, 1 July 201313.
5.3 Reaching maturity as complex open source project
“This release marks the first six month release cycle of OpenStack. The next release, Essex, will also be a six month release cycle and development is now officially underway. While Diablo includes over 70 new features, the theme is scalability, availability, and stability.” — Devin Carlen, 29 September 201115.
“We started this five-year mission with two projects: Nova (Compute) and Shift (Object Store) and over time, the number of projects in OpenStack grew. Some of this where parts of the existing projects that split out to have their own separate teams and become little more modular. Other things were good new ideas that people had that fit within the realm of OpenStack. Like interesting things that you would want to do in or with a cloud. Over time, we built a process around that to deal with the fact that there were so many of these projects coming in.” — Sean Dague, 15 May 201518
Freeze | Description |
---|---|
FeatureFreeze
| Project teams are requested to stop merging code adding new features, new dependencies, new configuration options, database schema changes, changes in strings... all things that make the work of packagers, documenters or testers more difficult. |
SoftStringFreeze
| After the FeatureFreeze, translators start to translate the strings. To aid their work, any changes to existing strings is avoided, as this will invalidate some of their translation work. New strings are allowed for things like new log messages, as in many cases leaving those strings untranslated is better than not having any message at all. |
HardStringFreezee
| 10 days after the SoftStringFreeze, any string change after RC1 should be discussed with the translation team. |
stable/*
branch from the current state of the master branch and uses Access Control List (ACL) mechanisms to introduce any new release-critical fixes discovered by the release day. In other words, further changes at this stage require permission from the release team. In the words of OpenStack, they will be treated as feature Feature Freeze Exceptions (FFEs). Between the RC1 and the final release, OpenStack looks for regression and integration issues. RC1 may be used as is for the final release unless new release-critical issues are found that warrant an RC respinning. If this happens, a new milestone will be open (RC2), with bugs attached to it. Those RC bug fixes need to be merged in the master branch before they are allowed to land in the stable/*
branch. Once all release-critical bugs are fixed, the new RC is published. This process is repeated as many times as necessary before the final release. As the final release date gets closer, to avoid introducing last-minute regressions, the release team limits the number of changes and their impact: only extremely critical and non-invasive bug fixes can get merged. All the other bugs are documented as known issues in the Release Notes instead.-
Common cycle with development milestones The official and default time-based model followed by most teams. It results in a single release at the end of the development cycle and includes three development milestones (as in Fig. 1).
-
Common cycle with intermediary releases For project teams wanting to do a formal release more often, but still want to coordinate a release at the end of the cycle from which to maintain a stable branch. Recommended for libraries, and for more stable components, which add a limited set of new features and do not plan to go through large architectural changes.
-
Trailing the common cycle For project teams that rely on the completeness of other components (e.g., packaging, translation, and UI testing) and may not publish their final release at the same time the other projects. For example, teams packaging and deploying OpenStack components need the final releases of many other components to be available before they can run their own final tests. Cycle-trailing project teams are given an extra two weeks after the official release date to request the publication of their own releases. They may otherwise use intermediary releases or development milestones.
-
Independent release model For project teams that do not benefit from a coordinated release or from stable branches. They may opt to follow a completely independent release model. Suitable for example for the OpenStack’s own infrastructural systems (e.g., the ones supporting upstream testing and integration) as well for components with little dependence on the overall Openstack core architecture.
“We still have a coordinated release at the end of the six months for projects that are willing to adhere to those deadlines and milestones, but the main change is that we will move from managing most of them to refine processes and tools for each project to be able to produce those releases more easily. The development cycle will still be using a six months development cycle, even if some projects might do intermediary releases where it makes sense, but will still organize almost everything under a six months development cycle between design summits.” — Thierry Carrez, 15 May 201527.
6 Results: infrastructural tools
6.1 Discussion, planning and specifying
6.2 Orchestrating distributed work, tagging and version control
6.3 Reviewing
6.4 Testing, gating and continuously integrating
“OpenStack projects have robust automated testing. In general, we believe something not tested is broken. OpenStack is an extremely complex suite of software largely designed to interact with other software that can be operated in a variety of configurations. Manual localized testing is unlikely to be sufficient in this situation.” — OpenStack Project Team Guide, as last edited on 20 Sep 201740.
“OpenStack projects do not permit anyone to directly merge code to a source code repository. Instead, any member of a core reviewer team may approve a change for inclusion, but the actual process of merging is completely automated. After approval, a change is automatically run through tests again, and only if the change passes all of the tests, is it merged.This process ensures that the main branch of development is always working (at least as well as can be determined by its testing infrastructure). This means that a developer can, at any point, check out a copy of the repository and begin work on it without worrying about whether it actually functions.This is also an important part of the egalitarian structure of our community. With no member of the project able to override the results of the automated test system, no one is tempted to merge a change on their own authority under the perhaps mistaken impression that they know better than the test system whether a change functions correctly.” — OpenStack Project Team Guide, as last edited on 20 Sep 201740.
6.5 Cross-team release management
Tool | Tool description | Roles |
---|---|---|
Launchpad | Collaboration and hosting platform | Bug tracking and discussion of blueprints. |
Story board | Task tracker | Task tracking across multiple teams, repositories, and branches. |
Git | Distributed code repository system | Hosting, version and revision control. |
GnuPG | Hybrid-encryption software | For annotating and signing Git tags. |
Gerrit | Code review system | Orchestrates peer review of proposed code changes over Git. |
Jenkins | Automation server | Runs jobs of continuous iteration testing. |
Zuul | Pipeline oriented gating and automation | Acts as a configurable gateway between Gerrit and Jenkins. |
Logstash | Pipeline and oriented analysis of jobs | Analysis of logs. |
Etherpad | Collaborative online editor (real-time) | For each stable branch, it aggregates information on current bugs, failures of the continuous integration jobs, known problems/issues, and on recently closed problems/issues as well. |
Reno | Release notes manager. | Manages release notes in a standardized format. |
Sphinx | Documentation and specification generator. | Integrates with Reno to generate reports containing release notes for specific branches and versions. |
Mailman | Software for managing electronic mail discussion and e-newsletter lists | Official channel for discussion among developers. Results from code reviews, continuous integration testing, and code-merging are delivered by software bots via e-mail. |
Freenode | IRC | Textual discussions of developer across different channels (achievable material). |
MediaWiki | Wiki software | Collaborative documentation. |
7 Discussion
-
Release notes are automatically associated with the release version based on the repository tags applied to the repository. It is not necessary to track changes manually outside the repository (e.g., in a bug tracker, a spreadsheet or other tool).
-
Release notes are encoded within the source code repository at the side of correspondent features source code. This means that release notes can be written when the code changes within the same development environment.
-
Release notes go through the same review process used for managing code and other documentation changes.
-
Release notes are encoded in a standardized format. Notes are organized into logical groups based on whether they describe new features, bug fixes, known issues, or other topics of interest to deployers, users, and developers.
-
Prior to delivering new features to the release management team, release notes can be automatically aggregated and documented from the source code repository with Reno. Developers only need to run a script that invokes Reno.
-
Release notes can be easily located by project, release series, branch, earliest revision, and date, among other parameters. Developers can search for specific sets of release notes and sections.
-
Liberal as the official releases dates, the milestones, and the freezes are not strict but negotiated and applied by the release management team in cooperation with the different project teams. In an analogy with trains, a given train might be scheduled to depart at a given time, nevertheless, they might come a bit before or a bit later due to a number of organizational or technical issues. Still, the train schedules remain a useful artifact for planning purposes.
-
Liberal as different teams (e.g., testing, drivers, documentation, and translation) are granted with extra days, or even a few weeks, to deliver their own releases. Releasing depends on the context, nature and interdependencies of the artifacts being developed. Examples include testing how a certain hardware driver complies with new developments, developing a new library that relies on external APIs and translating recent documentation to another language.
-
Liberal as feature freeze exceptions are granted by the release management team and the technical committee in exceptional cases (e.g., for landing a release critical fix). These feature freeze exceptions need to be properly discussed and documented before being granted, controlled and monitored.
-
Liberal, as well, as OpenStack started more recently recommending four different release management models. Even if most of the models trail the common six-month release ’train’, the OpenStack governance recognized over time the need for certain teams and individuals to manage their own release strategy independently.