1 Introduction
To answer this research question, we adopt a qualitative research approach and conduct an embedded single-case study (Yin 2014) at a large German car manufacturing company that transitioned its whole IT department to agile methodologies. The case study helps us gain in-depth insights into the challenges and approaches of practitioners in an organizational environment. We focus on understanding the challenges that practitioners are facing with regard to reporting in their large-scale agile programs. Further, in our case study, we discuss and document recommendations on how large organizations can balance agility and reporting.What are the challenges of reporting in large-scale agile organizations that should be considered in the development of a reporting approach for large-scale agile organizations?
2 Background
2.1 Large-scale agile methodologies
2.2 Reporting in agile settings
3 Research approach
3.1 First phase of the study
No. | Alias | Role | Scaling level | Large-scale agile development experience | Softw. dev. experience | Duration (h:m) |
---|---|---|---|---|---|---|
1 | AM1 | Agile master | Program | 3–5 years | 3–5 years | 1:42 |
2 | PO1 | Program owner | Program | 3–5 years | 11–15 years | 1:10 |
3 | BE1 | Business expert | Portfolio | 6–10 years | 6–10 years | 1:29 |
4 | AM2 | Agile master | Portfolio | 3–5 years | 6–10 years | 1:21 |
5 | PO2 | Program owner | Program | 3–5 years | 11–15 years | 1:00 |
6 | PO3 | Program owner | Program | 3–5 years | 11–15 years | 0:52 |
7 | AM3 | Agile master | Program | 3–5 years | 11–15 years | 1:11 |
8 | STE1 | Solution train engineer | Portfolio | 3–5 years | 6–10 years | ca. 2:00 (not recorded) |
9 | LM1 | Line manager | Team | 6–10 years | 6–10 years | 1:15 |
10 | AM4 | Agile master | Program | 3–5 years | >20 years | 1:31 |
11 | AM5 | Agile master | Program | 11–15 years | 11–15 years | 1:24 |
12 | DEV1 | Developer, software architect | Team | 1–2 years | 6–10 years | 1:23 |
3.2 Second phase of the study
No. | Alias | Role | Scaling level | Large-scale agile development experience | Softw. dev. experience | Duration (h:m) |
---|---|---|---|---|---|---|
1 | PO1 | Program owner | Program | 3–5 years | 11–15 years | 0:25 |
2 | BE1 | Business expert | Portfolio | 6 - 10 years | 6–10 years | 0:38 |
3 | AM2 | Agile master | Portfolio | 3 - 5 years | 6–10 years | 0:52 |
4 |
PO4
| Product owner | Team | 3–5 years | 6–10 years | 0:41 |
5 |
PO5
| Product owner | Team | 6–10 years | 11–15 years | 0:37 |
6 |
AM6
| Agile master | Program | 6–10 years | 11–15 years | 0:37 |
7 |
PO6
| Portfolio owner | Portfolio | 3–5 years | >20 years | 0:38 |
8 | LM1 | Line manager | Team | 6–10 years | 6–10 years | 0:38 |
9 |
AM7
| Agile master | Program | 6–10 years | >20 years | 0:38 |
10 | DEV1 | Developer, software architect | Team | 1–2 years | 6–10 years | 0:27 |
11 | PO3 | Program owner | Program | 3–5 years | 11–15 years | 0:25 |
4 Case description
4.1 Agile masters and process-oriented reporting
At the program and portfolio level, the Agile Masters are focusing on trend analysis of several metrics over individual measurements at one point in time in their reporting. For this purpose, Value Stream Dashboards are used in several programs, which are based on data-warehousing approaches for Jira. This allows for sophisticated analysis of trends and variances in Backlogs over time. Further, at the program and portfolio level, Agile Masters are continuously employing automated quality checks on the Backlog Items. They generate statistics on whether Backlog Items contain proper linkage to parent items, a textual description, estimation in Story Points, definition of acceptance criteria, linkage to addressed defects, and more. Those checks are monitored for Backlog Items in both portfolio and program cycles.So, in terms of evaluation of the team, it is much more important to focus on the process side of things rather than the output side of things. Because the output is always a result of how well the team operates, and you can’t just force the team to produce more. There is always an underlying reason as to why they are potentially producing less. So, that’s what we focus on in terms of reporting on the team level [...].
4.2 Product owners and product-oriented reporting
4.3 Agile teams and development-oriented reporting
5 Findings
ID | Name | Description | Interviewees |
---|---|---|---|
Challenges of the creator of a report | |||
1 | Agile teams do not understand the purpose behind reports they are required to do | We observed that teams often are not fully aware of a clear purpose behind a certain report. In these cases, teams perceive reports as reporting for the pure sake of management control | DEV1 |
2 | Agile teams lack context information to clearly report how their individual team work contributes to overall product and portfolio goals | We observed that reports of agile teams at the case organization often lack expressive power because the reporting agile team is not able to contextualize the work they are reporting on in the larger program or portfolio environment. Reports assembled by individual agile teams do not clearly highlight how the team’s work contributes to the higher-level goals of the program or portfolio at the case organization | PO1, STE1 |
3 | Reporting demands are limiting the autonomy of agile teams | While teams in agile settings are granted extensive autonomy in planning and executing their work, the large-scale organizational structures at the case organization demand a certain level of reporting to be able to coordinate work and keep track of overall progress. Such reporting demands are "imposed" onto teams at the case organization, which limits teams in their autonomy because they feel pressured to plan and execute their work according to the goals they are required to report towards | AM4 |
4 | Increased frequency of inspect and adapt cycles increases the efforts of reporting and automating reports | Caused by the increased frequency of inspect and adapt cycles that is central to agile methodologies, the usual efforts of gathering the data necessary for reporting now have to be undertaken several times more often in a certain period of time than before. At the case organization, a typical reporting cycle on the program level that used to last a quarter of a year now contains four program cycles of three weeks each—each of which requires a program-level report at the end. Further, due to the increased frequency of inspect and adapt cycles and the resulting goal adaptations, existing reports also need to be adapted more often than before the introduction of agile methodologies. However, these frequent changes also inhibit the automation of reports | BE1, PO2, LM1, DEV1, AM4 |
Challenges of the recipient of a report | |||
5 | Recipients of reports on higher levels do not gain meaningful insights from the reports they receive | We observed missing expressiveness of aggregated reports on higher levels in the organization, and interviewees explained to us that important topics from the lower organizational levels often are not represented correctly or not at all by reports on the higher levels. Reports on the higher organizational levels are either too detailed and thus too extensive to grasp, or only represent the issues and prioritization from a subset of all involved teams. Thus, they fail to direct management’s focus on the few most central topics | AM1, STE1, LM1 |
6 | Increasingly large scale of agile organization causes delays in the reporting chain | The large size of the case organization often causes reports from lower levels to reach higher levels with a certain delay, because the reported information is reused in another reporting cycle on the next higher level. Such delays are causing the agile inspect and adapt cycles to lag behind the current state of work at the case organization. Ultimately, this might limit the scope of action compared to a more timely report and reaction | PO2 |
ID | Name | Aspects of the agility of the organization that determine the challenge | Aspects of the large scale of the organization that determine the challenge | Impacts of the challenge on the organization |
---|---|---|---|---|
1 | Agile teams do not understand the purpose behind reports they are required to do | No apparent connection of reports to the continuous improvement process of agile teams | Communication gap between the program / portfolio level and the team level | Team-level: Agile teams perceive reporting as a pure control mechanism |
2 | Agile teams lack context information to clearly report how their individual team work contributes to overall product and portfolio goals | Agile teams autonomously set their goals | Communication gap between the program / portfolio level and the team level | Team-level: Agile teams struggle to report their contribution to overall program or portfolio goals |
3 | Reporting demands are limiting the autonomy of agile teams | Agile teams want to maximize value delivery for their local customer | Program and portfolio need to coordinate their teams and ensure a common direction | Team-level: Agile teams are limited in their full autonomy on setting their own goals |
4 | Increased frequency of inspect and adapt cycles increases the efforts of reporting and automating reports | Frequent inspect and adapt cycles are central to agility and require frequent reporting to gather feedback | – | Program- and portfolio-level: The program and portfolio are required to adjust their reports more frequently to changing goals compared to non-agile settings. To allow for some automation, the program and portfolio also have to ensure a certain consistency of the used tools across their agile teams |
5 | Recipients of reports on higher levels do not gain meaningful insights from the reports they receive | Agile teams autonomously set their goals | Program and portfolio have to keep an aggregated overview across the work of their teams | Team-level: Aspects or issues important to the individual agile teams are under- or misrepresented in higher-level reports. Program- and portfolio-level: Higher-level reports are either too broad, i.e., too extensive to grasp, or too narrow, i.e., not covering all high-priority aspects |
6 | Increasingly large scale of agile organization causes delays in the reporting chain | – | Large-scale: Information takes time to travel across the organizational hierarchies | Team-level: Delayed reports and feedback inhibit inspect and adapt cycles and continuous improvement, and thus the agility, of the agile teams |
5.1 Challenges of the creator of a report
5.1.1 Agile teams do not understand the purpose behind reports they are required to do (C1)
While transparency is a central value to many agile methodologies (e.g., SAFe states transparency as a core value (Leffingwell 2018; Edison et al. 2022)), agile teams at the case organization seem to be uncomfortable with their stakeholders’ ability to look at their work and documentation tools at any time.I mean, also there [...] the Domain [i.e., portfolio] starts to do some basic checks in all the Jiras of the Products. But this is sometimes really seen as a kind of control effort by management and is not really seen very well by the teams, that management wants to look into their Jiras, if their Jiras have good quality.
5.1.2 Agile teams lack context information to clearly report how their individual team work contributes to the overall product and portfolio goals (C2)
While lower levels are frequently and systematically reporting to their next higher levels at the case organization, higher-level stakeholders often fail to pass down context information of overall strategy and goals in a similarly frequent and systematic manner.What I am missing, is the connection. You’re part of [program name], what’s the next step? What’s important — I keep asking this question, but I don’t always get the link — what corporate goal am I actually contributing to and how? [...] On the one hand, the link to the corporate strategy itself. And the other is that reporting should not always be in one direction, but also in the other. This is something that is often lacking.
5.1.3 Reporting demands are limiting the autonomy of agile teams (C3)
To summarize, the challenge is determined by a combination of aspects that relate to both the agility and the large scale of the organization. On the one hand, it is determined by the agility of the organization because agile teams want to maximize the value delivery to their customer. On the other hand, the challenge is also determined by the large scale of the organization, because the program and portfolio have to ensure that the work of the individual agile teams is coordinated towards a common direction. This is realized by posing reporting demands to the teams. Through the combination of these aspects, the impact can be seen at the agile teams, who, as a result, are limited in their goal-setting autonomy.I think one big thing that’s going on here is just the kind of mismatch again between old management structures and an interest of management to control, and then autonomy on the team and the product level. And there is quite a wish on the product level and team level to be autonomous and it’s a quite common statement by Product Owners to say ’hey, look at what I am delivering every quarter, all of it is fine, why do you want so much reporting? If I wouldn’t have to do that reporting, I would be able to implement even more within my product. But there is so much effort on reporting, that hinders me in achieving more goals’. — Interviewee AM4
5.1.4 Increased frequency of inspect and adapt cycles increases the efforts of reporting and automating reports (C4)
The impacts of this challenge, while present for agile teams also in small-scale organizations, can mostly be seen on the program and portfolio level. The introduction of a higher frequency of reporting cycles, tied to the inspect and adapt cycles of the agile methodology, also increases the efforts of reporting on the higher levels in the organizations compared to non-agile times.It’s really crucial that these artifacts are kept up to date. That’s also quite a challenge because it’s also some work to do in order to keep the stuff up to date. We have more than 80 to 100 Sagas within each Domain [i.e., portfolio] cycle. Then, within each Saga we derive, let’s say, two or three Epics. And within each Epic we have 15 to 20 Stories. So, each of these artifacts needs to be kept up to date. [...] And there’s of course the whole review meetings, the preparation of the review meetings, the preparation of the slides, and all this stuff, this takes a lot of time and is not easy sometimes to do. All the preparation of all that stuff.
On the other hand, however, a development-oriented report that focuses on the newly developed features of the product will most likely differ a lot between two iterations, given the use of frequent inspect and adapt cycles is tied to development iterations at the case organization. This makes it hard to automate. Automating the generation of such a report, therefore, is always a trade-off between the stability of the goals reported against and the effort of automation (LM1, DEV1). Interviewee LM1 explained:That’s... in parts it’s very disappointing, there is not as much automation there as possible. [...] Whenever you want to do automated reporting, you need similar structures over the different products. And that’s not existing everywhere. As I said, my product works with Sagas and Epics, most of the other products only work with Epics. And so, it’s hard to do this kind of automated reporting there. — Interviewee AM4
To summarize, the challenge is determined by aspects that relate to the agility of the organization, because frequent inspect and adapt cycles also require frequent reporting to gather feedback. The impacts of the challenge, in contrast, are related to the large scale of the organization. While the effort for agile teams remains largely the same in large-scale settings compared to small agile organizations, the effort increases mostly at the program and portfolio level at the case organization. The program and portfolio are required to adjust their reports more frequently to changing goals. To allow for some automation, the program and portfolio further have to ensure a certain consistency of the used tools across their agile teams.It’s always a question of how much effort you put in the automation and how stable the reportings are. If they are very stable, then it’s worth putting a lot of effort into the automation. If it changes from quarter to quarter, then you obviously won’t spend that much time, money, and effort in order to automate it. But as a general rule, sure, every reporting should be automated as much as possible.
5.2 Challenges of the recipient of a report
5.2.1 Recipients of reports on higher levels do not gain meaningful insights from the reports they receive (5)
The autonomy of agile teams to set their team-level goals on their own makes it increasingly hard for reports on the higher levels to capture and aggregate the critical issues from all agile teams. A further complication of this aspect is the large size of the organization (STE1) and the resulting high number of teams with autonomously defined team goals.Getting the real stuff there. [...] So, I would say some of the most important impediments are not represented. [...] So, I would say this is one of the biggest challenges. Having or bringing the real topics up there, even if they appeared there already one thousand times.
In several programs, goals on the higher levels are simply treated as overall objectives without any relative prioritization among each other. Because reporting is linked to prioritization (LM1), the lack thereof makes it hard for reports to focus on high-priority topics and provide focused insights on these topics.[...] the problem actually is the same on all levels, but it starts on the highest level of our hierarchy. The higher you get in a company like ours, the less people actually prioritize. They decide ’do it’ or ’don’t do it’, but they don’t prioritize. [...] I would say that’s the biggest challenge. Actually prioritizing on the highest level of our organization. And the reporting is directly linked to this prioritization.
5.2.2 Increasingly large scale of agile organization causes delays in the reporting chain (C6)
To summarize, the challenge is determined by aspects that relate to the large scale of the organization, while it impacts the agility on the team level. The aspect that information takes time to travel across the organizational hierarchy and reporting cycles relates to the large scale of the organization. The agile teams are impacted in their process of continuous improvement, ultimately negatively impacting a central concept of agility.I mean, as I already said, we are more than 100 people working on different topics. And sometimes the whole [reporting] chain, when something isn’t delivered or when there is some issue, sometimes it happens that this information comes at a quite late stage. [...] So, this whole chain from developer, feature team, product, portfolio... sometimes it’s not that easy to have a fluent communication going on.