Exploring software development at the very large-scale: a revelatory case study and research agenda for agile method adaptation

Nerur et al. (2005) described the fundamental assumption behind traditional methods to be that information systems are fully specifiable and built through meticulous and extensive planning. Agile methods, on the other hand, assume that information systems can be built through continuous design, improvement, and testing based on rapid feedback and change. Learning and adaptation should be embraced (Conboy 2009). Adapting the method to the context will involve balance in a number of areas (Vinekar et al. 2006), as illustrated in Table 1. Very large-scale projects involve great risk and management attention. Boehm and Turner (2003) argued that traditional and agile methods should be balanced when facing risk such as increases in programme size.

The transition from traditional plan-driven development to a more agile method with iterations and development conducted in small teams was the focus of a study by Petersen and Wohlin (2010). They examined the development of three large subsystem components in an Ericsson product involving 117 people and found that many of the issues raised in traditional development were not raised after the transition to agile development. This suggests that agile methods can also work well in large-scale product development. Another study from Ericsson finds that agile principles in large scale contributed to knowledge-sharing, and led to increased project visibility and effectiveness in coordination (Lagerberg et al. 2013).

One of the very few studies on large projects combining traditional and agile methods examined a project to develop a web-based customer booking engine for an American cruise company (Batra et al. 2010). The study describes a $15 million project that lasted for 28 months. The project was distributed and combined Scrum with the Project Management Body of Knowledge³ framework. Customers were available but did not work together with developers on a daily basis. The iteration length was two weeks. Some of the challenges identified in the study were due to the size and involvement of a high number of internal business sponsors, users, project managers, analysts, and external developers from the UK and India; Customer involvement. The communications were mainly formal, and formal documents were needed for changes. However, the project was considered a success, and the study describes the balance between traditional and agile methods as essential in achieving both project control and agility.

A model organizing development as chains of Scrum teams (the output of one team is the input of the next) is described in a study of three cases of organizations with 150, 34, and 5 Scrum teams (Vlietland and van Vliet 2015). The study investigated strategy, structure, collaboration, coordination, communication, mind-set, and competence of people. The dependence of teams on the results from other teams introduces a number of challenges. The three cases illustrate challenges that include a lack of coordination in the chain, mismatch of backlog priority between teams, challenges in alignment between teams, and unpredictability in delivering on commitments, this suggests challenges with inter-team coordination and main design decisions as expressed in the software architecture.

Agile methods are people-centric and recognize the value that competent people and their relationships bring to software development (Nerur and Balijepally 2007). A key pillar in any agile method is the close and continual collaboration between clients and developers (Maiden and Jones 2010). Therefore, agile methods are highly dependent on the on-site customer identifying and prioritizing features, providing feedback, and guiding change in the course of the development (Vinekar et al. 2006). In their study on a large-scale agile project, Bjarnason et al. (2012) found that low customer involvement in near-development roles in combination with weak awareness of overall goals may result in an unrealistically large project scope. Further, overscoping can lead to a number of negative effects, including quality issues, delays, and failure to meet customer expectations. A study on large-scale and distributed projects found that understanding requirement dependencies is of paramount importance in such projects (Daneva et al. 2013).

Active participation and constant involvement of the customer in systems development yields greater benefits, but this reliance on the customer can fail if the on-site customer goals are misaligned with the goals of other stakeholders. Having multiple on-site customers in a large-scale project increases the risk of failure because of the challenge of establishing a common understanding among all customer representatives. A fragmented view of the system that each customer may have is likely to have a negative impact on the project (Ramesh et al. 2010). Further, when different stakeholder groups have different priorities, there is a need for open and transparent dialogue and cross stakeholder group communication in large-scale agile projects (Barney and Wohlin 2009).

The software architecture is the fundamental technical organization of a system. How and when to make architectural decisions have been the subject of major debate in the software engineering field (Abrahamsson et al. 2010). In traditional development, the architecture is defined prior to implementation and testing, whereas the architectural design emerges as a result of on-going clarification in ‘purist’ agile development. Traditionally, software architecture is associated with up-front designs and stable structures that accommodate pre-defined, non-functional requirements. The assumption is that the value of good architectural decisions will surface at a later point in time in forms such as more easily maintainable code and scalability (Faber 2010). Also, a sound architecture is largely invisible and provides an effective structure for subsequent functionality. It is interesting to note that software architecture has meaning and significance for a wide variety of stakeholders. Different stakeholders are also likely to associate different meanings to software architecture. Smolander (2002) suggested that the four metaphors blueprints, literature, language, and decisions capture the meaning of software architecture for different actors involved in software development. This underlines the pervasive role of software architecture.

Several approaches to architecture work have been taken in large agile projects. Some start with the architecture (‘big up-front design‘) and then use agile methods. Others spend the first iteration focusing on architecture. Others again start directly on development and let the architecture emerge. To construct a large software system developed by a number of teams, it is vitally important for the architecture to be agreed upon and communicated without introducing the bureaucracy and overhead associated with traditional methods. In a study on software product companies, Unphon and Dittrich (2010) found that architectural knowledge was transferred by face-to-face communication with chief architects taking the role of a “walking architecture”.

Awareness and social protocols are important perspectives on how architecture is communicated. Unphon and Dittrich do not discuss the increased challenges of architectural work at a large scale. In their study of a large-scale agile approach at Ericsson, Petersen and Wohlin (2010) found a need for a high-level architectural design to facilitate planning. Nord et al. (2014) argued that for large-scale agile projects, agility is enabled by architecture, and architecture is enabled by agility. They suggested several tactics for handling architecture in large-scale projects, including making use of a matrix structure and focusing on the production infrastructure.

Coordination can be defined as “the managing of dependencies” (Malone and Crowston 1994), where dependencies can be related to tasks, knowledge, resources, or technology. The central challenge in coordination is identifying the right form or artefacts, arenas, and degree of formalization in large projects with high uncertainty. In small agile projects, the development team coordinates work through frequent informal interaction among themselves and with customers, as in the customer-on-site practice in eXtreme Programming. Scrum has dedicated meetings for planning, review, and retrospectives. Many teams use visual boards, like in Kanban, to show who is working on what and the status of work tasks. Strode et al. (2012) explain coordination at the team level in agile teams and propose a model for coordination strategy and coordination effectiveness.

For large-scale projects, there is less support. Scrum prescribes regular meetings between Scrum teams (“Scrum of Scrums”) in order to manage the interfaces between teams. Eckstein shows techniques that are applicable to large projects in order to facilitate planning, status information, integration, and retrospectives in her book with recommendations to practitioners (Eckstein 2004). Some large-scale agile frameworks have been suggested by practitioners (Larman and Vodde 2013; Larman and Vodde 2017; Leffingwell et al. 2017) that describe roles and arenas for inter-team coordination. Visual boards was the primary form of inter-team coordination in a project in Sweden described in an experience report by Kniberg (2011).

There is a small body of studies on inter-team coordination. Vlietland and van Vliet (2015) propose that embedded coordination practices within and between Scrum teams positively impact delivery predictability in large projects. A study of “Scrum of Scrums” (Paasivaara et al. 2012) suggests that this forum did not lead to satisfactory coordination: feature-specific or site-specific fora were better, but coordination at the project level was still a challenge. Researchers working closely with SAP (Scheerer et al. 2014; Scheerer and Kude 2014) have developed models of coordination called “coordination configurations” and are exploring how coordination configuration influences coordination effectiveness. Paasivaara and Lassenius (2014) describe a very large-scale development initiative at Ericsson with 40 teams where four types of communities of practice (Wenger 1998) are used to coordinate teams. A survey on coordination in large-scale software teams found that respondents wished more effective and efficient communication, as well as the importance of good personal relationships for coordination (Begel et al. 2009).

A management science study (Ingvaldsen and Rolfsen 2012) suggests that inter-group coordination is a major challenge when groups are self-managing. Self-management involves giving teams authority to decide how to 1) execute tasks, and 2) monitor and manage their work process (Hackman 1986). Moe et al. (2009) describe challenges to self-management at the team level and highlight challenges at the organizational level, such as shared resources, organizational control, and specialist culture.

We chose to use two types of data for this revelatory study: Group interviews and documents: First, for each topic in the embedded case study design (customer involvement, software architecture, inter-team coordination), we organized three group interviews (Myers and Newman 2007), one for each organization. The reason for three interviews on each topic was that we knew from introductory meetings that there would be slightly different approaches in the three development subprojects, and having meetings in each organization might have led to more openness (and trust as discussed by Myers and Newman (2007)) than in joint group meetings. Due to the explorative character of the study we primarily wanted groups in order to facilitate a broader discussion.

We asked the three organizations to invite the most relevant people to attend each group interview. This involved people with responsibility for the topic, for example from team level, subproject level and programme level. At Accenture, the key persons working on architecture were no longer working at the company. Table 2 gives an overview of the interview groups and lists the number of participants for each group. In total, 24 people participated. At Steria and the Pension Fund, some participated in several groups, making a total of 19 individuals. These people had a number of roles in Perform, from project management, subproject management, technical architecture, functional architecture, testing, development leadership, GUI, information security, Scrum masters, and developers (See Table 2). Note that there is a bias in participation towards employees in management positions (discussed in the limitations section). However, many of the participants had started in the programme as developers and assumed other roles during the programme. The participants selected where the ones who had responsibility for the topics we discuss, but we do not claim to represent all views of stakeholders on the topics. At the time of the data collection, all participants were seniors, with at least four years of experience in development, architecture or management positions, and at least four years of experience from their own organization. Interviewees were informed that the study is registered by and performed according to specifications by the Norwegian Data Protection Official for Research, which regulates collection and use of personal data material.

Two researchers participated in each group interview, and there was one interview guide per topic (see Appendix A). We developed a timeline for the project, which was shown to the participants so that they could remember key events in the project. The researchers had roles of leading and moderating the discussion and taking notes. All interviews would start with participants explaining their role in the project. One meeting at Accenture included only one person and was conducted as a semi-structured interview (Myers and Newman 2007). One group interview at Steria included six persons, and we used brainstorming techniques to ensure contribution from everyone invited. Participants brainstormed on key events, on what they thought was conducted well in the programme within the topic of discussion, and on what they thought could be improved. All nine meetings lasted two hours and were recorded and transcribed. We took pictures of drawings on whiteboards. This produced a total of 247 pages of transcribed material. The transcripts were sent to participants for information and for comments.

Second were three documents: the official report after project completion, the internal experience report (both from 2012), and the quality assurance report for the project, which was written by an external consulting company in 2010. We chose to include these documents as they provide overall descriptions of the programme such as planned organization, processes and roles. The internal experience report provides main lessons learned by the programme, the quality assurance report provides an assessment of main risks. In total, these reports contained 277 pages of text.

We imported all interview transcripts and documents into a tool for qualitative analysis and did a descriptive and holistic coding (Saldaña 2009) of the topic inter-team coordination. Three researchers read the relevant documents and did individual coding, and then we agreed on coding in joint meetings. Units of text ranged from sentences to whole pages and were coded into topics such as “Scrum of Scrums” and “Metascrum”. The results of this pilot was presented and discussed with the rest of the research team, and we set up two research teams with two persons each to code the material on customer involvement and software architecture. Given the various topics and backgrounds of researchers, the level of detail in the coding varied between the research teams.

As an example, the statement, "On an overall level, the teams worked quite similarly after exchanging experience. So the Scrum boards were used quite similarly across the teams. There were differences in colours, but the function was quite similar", was coded as “board discussions” and grouped with 37 other concepts under “inter-team coordination”. The following passage was coded as “responsibility”, "it was important that when teams got user stories and were to detail test conditions … that everyone contributed to detailing, then it was sent to project business for approval, and when you had the stamp from the business resource, you had agreed on scope". This was coded with six other concepts under the category “participation”, which was one of seven concepts under “practice”, which again was one of four main categories under “customer involvement”. In total, there were 29 concepts related to this topic and 17 concepts related to software architecture.

After coding the material, we discussed what the most interesting findings were in the material. This was then made into a slide presentation, and feedback meetings were conducted with all three organizations Accenture, Steria, and the Pension Fund for member checking. These meetings were recorded and partially transcribed to add further data material, which led to small revisions of the first findings. Reactions to findings included surprise with the number of coordination arenas identified in the study, and comments on the thorough description of the architectural work in the programme.

Perform was initiated to enable the department to provide accurate and timely services to customers and ensure a cost-effective implementation of the reform. Because of the reform, the programme had a strict deadline. It is one of the largest IT programmes in Norway, with a final budget of about EUR 140 million. The programme started in January 2008 and lasted until March 2012. The project involved 175 people, of whom 100 were external consultants from five companies.⁶ The programme used both time and material and target price contracts for subcontractors. About 800,000 person-hours were used to develop around 300 epics, with a total of about 2500 user stories. These epics were divided into 12 releases as in Fig. 1.

As an example, release R8 contained coupling of workflow in the office automation system to the archive solution, a self-service solution for new legislation, simulation of services towards external public departments, and first data warehouse reports on new data warehouse architecture. Most user stories were identified prior to the first release but were supplemented and reprioritised for every release.

The existing office automation system was client/server-based and written in C. The new system is a service-oriented system written in Java and provides a richer interface using Flex.⁷ The database from the old system was kept, but the data model was changed. The regulations and legislations are implemented in the new system as rules using JRules. The system is integrated with a new document archive and with systems from another public department. Figure 2 gives an overall description of the existing solution and the new solution.

The programme was managed by a programme director who mainly focused on external relations, a programme manager focusing on the operations, and a controller and four project managers responsible for the business, construction, architecture, and test projects (see Fig. 3):

There were also projects for communication and adoption to prepare users for the new systems, in total six projects.

As shown in Fig. 3, the programme used a matrix structure where the business and development projects took part in the architecture and test projects. This matrix structure meant that a feature team would then mainly take part in project development, while also devoting resources to projects architecture (through the technical architect), business (through the functional architect), and test (through the test responsible) as shown in Table 3.

The programme started working at the main office of the public department, but in 2009, it was moved to a separate office building located in the same city. Here, all teams were organized around tables as shown in Fig. 4.

Initially, the development process included the four phases described in Fig. 5:

To keep the schedule of the project, solution descriptions needed to be ready in time for the feature teams. This meant that releases were constantly being planned, constructed, and tested. Thus, given the roles in Table 3, a feature team would constantly be engaged in construction for release “n”, approving delivered functionality in release “n-1”, and analysing needs for the next release (“n + 1), as shown in Fig. 5. After approval from the programme, new releases were acceptance tested, set in production, and underwent an approval phase before being accepted by the operational IT section of the department.

The main arena for customer involvement in Perform was the interface between business and architecture subprojects and the analysis of needs and solution description phases. We found three key characteristics of this work: first, that the solution description was developed using teamwork; second, that it was conducted continuously and iteratively; and third, that the boundaries between analysis of needs, solution description, and construction were blurred. We describe each of these characteristics in the following.

Two topics emerged as primary characteristics related to managing work on software architecture: the tension between up-front and emergent architecture, and the demanding role for architects in large-scale agile projects.

From our data material on coordination and knowledge-sharing between feature teams in the development programme, we identify three main findings. First, there were a number of arenas involved in order to achieve coordination and knowledge sharing. Second, the use of these arenas changed over time, and third, many emphasize the importance of the informal coordination arenas, which were enabled by the open work area. We describe results from each of these three areas in the following.

Agile methods describe the on-site customer to identify and prioritize features, provide feedback, and guide change through the course of the development (Vinekar et al. 2006). Customer involvement is a challenge in large-scale development efforts (Batra et al. 2010). Many representatives are needed, the representatives need to be aligned, and they must be available to all the teams. In large-scale projects the alignment challenge is considerable because different stakeholder groups have been found to have different priorities (Barney and Wohlin 2009). Further, a fragmented view of the system that each customer representative may have is likely to impact the project negatively (Ramesh et al. 2010), and the dependence on the on-site customer may cause project failure if the on-site customer is misaligned with the stakeholder goals (Vinekar et al. 2006). While each project in a large-scale effort may have success at the project level, there could still be overall failure at the program level if the projects are not aligned. The Perform program consisted of six subprojects and 175 people, which made alignment a challenge.

The issue of scale and availability was addressed by including 30 product owners and customer representatives from the Pension Fund that worked full time in the program. They were all in the “business” project and responsible for analysing needs through defining and prioritizing epics and user stories in a product backlog. Locating them in the same office space as the construction teams ensured their availability. Further, alignment among the on-site customers and between on-site customers and construction were supported throughout the whole program by 1) a number of arenas, 2) a continuous and iterative solution description process, and 3) the blurred boundaries between the phases of analysis and needs, solution description, and construction. Our findings are consistent with Barney and Wohlin (2009) who found that open and transparent dialogue and cross group communication as essential for aligning the different stakeholder groups.

Alignment within and across the six projects grew from the formal and informal arenas, continuous cooperation, and the constant opportunities to ask anyone about clarifications. Both customer representatives and developers could exchange and diffuse shared insights and knowledge through their network; that is, they could use their social capital (Wohlin et al. 2015) when aligning on all levels. Involving the teams in the solution description was important for the program success because it ensured alignment between teams and gave them end-to-end responsibility for features by involving them in the formulation of its requirement until the release. The importance of end-to-end responsibility in large-scale agile development is also consistent with Olsson et al. (2014) in their study of 30 teams in a large-scale program at Ericsson. Because there was always solution description work going on in the Perform program, the process can be understood as a continuous planning activity involving construction and business. Such continuous planning ensures alignment between the needs of the business and software development (Fitzgerald and Stol 2017).

While the programme under study enabled the concept of the on-site customer in practice, further research is needed to better understand the challenge of alignment between the on-site customer representatives and between on-site customer and construction in large-scale projects. Especially when co-location is not possible or feasible, if there is a high degree of uncertainty of what the customer wants, or there is a need for continuous deployment. We therefore propose the following research question for further investigation:

How can alignment be ensured among customer representatives and between customer representatives and developers in very large-scale programmes?

The system architecture and the development process are two vital areas of software development that could undergo changes during a development programme. For architects, it is difficult to balance concerns with delivering early customer value and taking care of overarching quality requirements (Hopkins and Harcombe 2014). The Pension Fund had to strike a balance and support both change and stability simultaneously (Vinekar et al. 2006). A system architecture defined up-front gave individual teams a stable working environment with room for teams to design their own local sub-architectures that probably contributed to team efficiency. Still, there were tensions from challenges that occurred because there had been few proof-of-concepts with core technology. With respect to the development process, many aspects remained unaltered. Team roles and iteration lengths are examples of this. Also, we found few changes in the larger-scale structures in Perform. The masterplan (Table 4)—the general matrix structure of the programme as well as the overall system architecture—remained largely unchanged during the programme period (Table 5).

Perform was a learning-centric programme, yet its overall architecture and main development process remained unchanged for the bulk of the programme’s duration. It is unclear how a more technically complex software system would impact the use of agile methods at such a large scale. An interesting research question is:

How can large-scale agile programmes enable learning and change in architecture and processes at inter-team and programme levels?

Self-managing teams offer potential advantages over traditionally managed teams because they bring decision-making authority to the level of operational problems and increase the speed and accuracy of problem solving (Moe et al. 2009). In large projects, there are a number of problems that span development teams as well as decisions that affect a number of teams. Perform found a balance between self-management and a centralized structure. In the results section, we described how the matrix structure in the programme enabled coordination of development teams through roles in the teams and special arenas for the project’s business, architecture, and test. This enabled direct communication between business and development, which we called blurred boundaries between phases.

A number of arenas in the early phase of the programme established communication and made it easy for team members to mutually adjust to activities in other teams. The open work area enabled quick oral coordination, but the extra roles (such as functional architect) and arenas (such as the Open Space) led to less autonomy for the team. The introduction of extra roles is a threat to self-management as the teams are instructed to establish certain roles, which Hackman (1986) refers to as managing their own work process. Further, teams needed to provide resources for projects other than development and had to engage in and accept decisions made in the projects that affected all team members. An example is the architectural guidelines and the overall architecture, which we have described as mainly being decided up-front. The standardization of roles and arenas across the project was primarily done to ensure needs at the programme level.

Contrary to findings reported by Paasivaara et al. (2012), we found that the Scrum of Scrums was perceived as a well-functioning coordination mechanism, but we could speculate that this could be because of the size or the focus on a large number of total arenas. We did not find use of Communities of Practice, which is described as a successful characteristic of an even larger development effort at Ericsson (Paasivaara and Lassenius 2014). We do not have insight on the total effectiveness of the coordination arenas in the Perform programme beyond knowing that the programme was recognized as a success. We could speculate that the dynamic nature of coordination with a large number of arenas made coordination efficient (Jarzabkowski et al. 2012).

It seems the programme found a good balance between inter-team coordination and self-management. However, this is a topic that deserves further research in describing other approaches and also examining the matrix structure applied in the Perform case in further detail. We pose the following research question for further investigation:

How do multi-team development programmes balance inter-team coordination and self-management?

We use the most relevant principles for interpretative field research from Klein and Myers (1999) in discussing limitations of this study. Given the revelatory nature of this study, we have not put emphasis on limitations related to theory.

A main challenge in studying very large-scale programmes is the size of the case, and thus a challenge to understand both whole and parts (principle of hermeneutic circle). We have sought to address this by building an understanding of the whole through a thick description of method adaptation, and building understanding of three critical themes in an embedded case study design. Given the choice of three themes, there are a number of other areas that we have left unexplored, such as how quality was handled at the very large scale.

We have sought to satisfy the principle of contextualization by providing a thick description of the adapted development method in the programme, describe the development of key ideas on agile development in small teams in the theory and show how these ideas are present in the very large programme in presenting the empirical results.

Another limitation is related to the interaction between researchers and subject. We relied on group interviews and documents and as two data sources. For the group interviews, there is a potential elite bias (Myers and Newman 2007) in our data collection as the people involved in the programme at the end had technical or leader roles. We have avoided asking questions to evaluate the programme outcome but asked participants to reflect critically on practices and concrete events. We also had fewer participants from Accenture than from the other companies, which could have given us less variety of opinions. Further, the internal authors of documents could have an interest in portraying the programme as successful.

With the focus on three themes, we have aimed at identifying multiple interpretations by the different groups who participated in the programme, for example how architects and business project participants had different opinions on what was important at stages in the programme.

To discover biases and distortions in our findings (principle of suspicion), we have separated group interviews and presentation of findings to Accenture, the Pension Fund and Steria. Further, we have conducted workshops between researchers where all have had particular insight in one of the three main topics, thus sought to identify misconceptions from several angles.