Game-based Sprint retrospectives: multiple action research

OKE Poland was established in association with international partners to create user-friendly IT solutions. Currently, the company is mainly focused on improvement and development in the hybrid TV platforms area.

Dynatrace builds intelligent software designed to help companies manage their application performance and understand how the application performance impacts the users. The delivered tools allow DevOps and admins to leverage full stack monitoring by collecting data from across all components of a modern application and infrastructure stack.

SentiOne builds a Conversational AI Platform – product providing state-of-the-art customer service automation based on social listening and data analytics.

The responses for each question are presented with diverging stacked bar charts (Fig. 5) as proposed by Robbins and Heiberger (2011). From the charts, we can see that Dyna_B was the most enthusiastic about the idea of retrospective games, while Dyna_A and OKE_A, despite being the biggest skeptics in general, still found games that improved their retrospectives (e.g. Sailboat for OKE_A). We can also observe that opinions usually vary significantly within each team. Moreover, it is apparent that some teams did not like some games and therefore graded them low in almost all aspects even though individual team members had different perceptions. Specifically, OKE_A disfavored 5Ls and Mood++, OKE_B disfavored Sailboat, while both Dynatrace teams disfavored 5Ls.

Although overall, the retrospective games positively affected all evaluated aspects of the retrospective meetings (Q1, Q3–Q6), the improvements might seem to be rather slight at first sight. Nevertheless, even if a game positively affects the performance of just a few participants, the whole team usually benefits from the ideas put forward by those participants. The aspect that improved the most was “communication” (Q5), while the games that were favored in all aspects by most of the teams were Sailboat and 360 Degrees Appreciation.

Surprisingly, some games received single “strongly disagree” responses even for “communication” (Q5) or “easiness to play” (Q7). Accordingly, we decided to look into the individual questionnaire sheets and analyze them jointly with our hand-written notes taken during the participant observations to find an explanation for the low scores. In OKE_A, the participant who understated the results was a programmer who was probably not aware of the added value of retrospectives due to his low commitment to the project (he was engaged in another project at the same time). As for OKE_B, the tester was an older man who did not speak English. Usually, he had difficulty in comprehending the names of the categories, so he did not like the idea of playing the games and he gave low scores. In both SentiOne teams, there were individuals who did not like some games and rated them low in most aspects. In Dyna_A, two experienced developers evinced a negative attitude toward retrospective games from the very beginning and their attitude was reflected in the scores. They said that code writing is their job rather than debating over and over again. They believed that anything other than coding is just a waste of time. On the other hand, in Dyna_B, one team member, who was very knowledgeable in Scrum, was also very keen on running perfect retrospective meetings with the help of retrospective games. However, if he did not like a game, he graded it low, especially in comparison with his teammates. Accordingly, a “disagree” response does not necessarily mean that the game had a negative impact. It may also mean that the respondent disliked the game or did not find a positive impact regarding the evaluated aspect.

When it comes to Q2, the question turned out to be confusing and some team members interpreted it in a different way than we expected. What we meant by “permanently adopt the game” was to include the game in a toolbox of possible retrospective activities. As for Q6, most of our team members had already been willing to attend retrospectives, so this is probably the reason why “Neither Agree nor Disagree” was the most common answer regardless of the game.

A further examination was made from the perspective of the games by using spider charts (Fig. 6). For each game and statement, we calculated the fraction of participants who “somewhat agree” or “strongly agree” as well as those who “somewhat disagree” or “strongly disagree,” hereafter jointly referred to as “agree” and “disagree,” respectively. Since all games were considered easy to understand and play by almost all respondents (Q7), we did not visualize this category. Likewise, we did not visualize Q2, because it was understood differently by different team members.

Figure 6 shows that 360 Degrees Appreciation performed the best overall. Nevertheless, since this game is not a self-contained retrospective activity, it has (except 5Ls) the worst ratio of positive to negative responses for Q1, which states that the game produces better results than the standard approach. Among the other games, Sailboat, Starfish, and Mad/Sad/Glad still performed quite well. Sailboat has the best ratio of positive to negative responses overall, Starfish has the best ratio for Q1, while Mad/Sad/Glad has the fewest negative answers. Furthermore, 5Ls is dominated by Sailboat, because for each aspect, the latter has a better ratio of positive to negative responses. Finally, all of the games performed well as for “communication” (Q5).

To generate more understanding on the outcomes and gain deeper insights into the experiences and opinions of the participants, each internal researcher conducted a focus group (Kontio et al. 2008) with each team to which he/she was assigned. The meetings took place at the office of the participating companies, lasted about 30 min, and all team members were present. Each session started with an introduction from the facilitator about the purpose and process of the session; guidelines on how the participants should discuss; and how the results would be used. Special emphasis was given to assure participants of the anonymity of the discussions (no one will be identified with any comments or findings that are reported) and to make them aware about the importance that their opinions represent the real situation. Six questions were prepared (see Appendix 2) to ensure that the appropriate topics were covered and the discussion stayed focused. Audio recordings were made of all of the focus group sessions for transcription.

An initial analysis of the transcribed material revealed that for every session, the answers to each question were scattered through the whole discussion. The material was then processed using a deductive content analysis (Elo and Kyngäs 2008; Mayring 2014). We started with a set of predefined themes. Each theme corresponded to one of the six questions that guided our focus groups. Furthermore, we defined three categories of codes: team codes to refer to the team that generated the statement; object codes to point out the object concerned by the statement; and description codes that characterize the object. Each category contained a preliminary set of codes, which was extended during the analysis. The final sets of codes were (underlines denote codes that were developed during the analysis):

We analyzed the transcription to identify units of meaning and coded each unit with at least one code from each category. To facilitate coding, we used Microsoft Excel. Each unit was placed in a single row in the first column. We used a separate column for each of the three categories of codes (i.e. team codes, object codes, description codes). Table 8 provides some illustrative examples of coding of the discussion in Senti_B when answering the second question. Next, we developed a central descriptive narrative of the phenomenon under study for each focus group question. Ultimately, the outcome of our content analysis was enriched with examples coming from the participant observation in order to back up the findings. The final result is presented in the subsequent discussion. Note that during the focus groups, when someone expressed an opinion, some other participants either agreed with it, possibly adding something from themselves, or voiced different opinions. In either case, usually there were a few participants who needed encouragement to voice their opinions; so, we asked the question – “What is the opinion of the rest of the team members?” If the great majority of team members agreed on a particular view, it is presented as a team-level statement in the discussion below.

All of the teams except for OKE_B enjoyed game-based retrospectives and considered the new approach valuable even though there were single opposing voices. OKE_A agreed that the games allowed them to improve their work efficiency as well as the Scrum process. Among other things, they started to run the Daily Scrum; they decreased the pull request evaluation latency by granting the approval permission to another programmer; the availability of the Product Owner for the development team was increased; the Product Owner became aware of many problems that the development team faced; the need to hire a graphic designer was identified (this need was met shortly after the research process was completed); MVC controllers were refactored to REST API with Angular. Finally, playing the games also created a sense of community in the team.

OKE_B was the only team in which the new approach did not bring much value since most of the game sessions ended with a similar conclusion, i.e., the main problems were due to a communication gap between the Product Owner and the Development Team. Taking into account that the team worked under time pressure, playing games was considered too time-consuming. Furthermore, the Scrum Master viewed most of the games as too childish. Nevertheless, the team agreed that retrospective games should be played at times, for instance every third sprint, because the games still helped them to generate novel insights, e.g. how to customize JIRA (shortly before our research started, the team underwent a migration from TFS to JIRA).

As for Dyna_A, two team members would prefer not to use retrospective games in the future. They believed that similar conclusions could be made in the course of standard retrospectives, so there was no point in learning retrospective games. However, the rest of the team members were in favor of the new approach. The team also suggested exploring new retrospective games, such as Three little pigs and SWOT.¹ They said that playing the same game over and over brings similar comments, but when a new game with different categories is introduced, it makes the participants think more deeply and results in novel outcomes. As for Dyna_B, all team members were uniformly positive about the overall experience.

In both SentiOne teams, only one team member did not like game-based retrospectives. Senti_A said that the new way of running retrospective meetings gets participants engaged and stimulates the imagination, which in turn generates more ideas to discuss. Thanks to retrospective games, they began to specify backlog items more accurately and made developers be more involved in testing. Senti_B agreed that each game improved the meetings as well as the teamwork regardless of whether it was liked or disliked by someone, because it allowed them to elicit insights in a systematic way. They also concluded that they should conduct retrospective meetings, but not necessarily after each sprint.

Both OKE teams, Dyna_A and Senti_A said that different games should be played interchangeably, because playing the same game over and over would be boring, while playing a variety of games increases the chances of identifying a variety of issues. In contrast, Dyna_B and Senti_B concluded that they would prefer to play the same game for several sprints to fully comprehend it. Only after this learning period, they would like to switch to another game. Senti_B also noticed that using the games alternately is desirable, because team members have different preferences, so it would be hard to choose one game that satisfies everyone.

OKE_A said that a game board in the form of a table is not attractive, so Mood++ and 5Ls received low scores. OKE_B claimed that in their project, the goal, threats and problems usually remain the same. Accordingly, playing Sailboat more than once would generate very similar results, thus they graded the game low. Both OKE teams agreed that 360 Degrees Appreciation produces worse results than the standard approach, because it forecloses a full retrospective. Consequently, they graded the game negatively regarding Q1 and Q2, even though they appreciated the positive influence of the game on the other aspects. In contrast, Dyna_B gave 360 Degrees Appreciation high scores because it met their expectations – they always left the meeting in a good mood.

Team members of Dyna_A and Senti_A were surprised that Sailboat was assessed moderately, while, in fact, they liked the game as well as drawing its game board. For Dyna_A, action items generated during the game also helped the team to elaborate a systematic approach to analyzing and prioritizing bugs reported by customers. When it comes to Dyna_B, they liked Sailboat so much that they were surprised that someone graded it negatively with regard to Q1 and Q2.

Some Dyna_A members were also surprised that 5Ls received low scores even for communication (Q5), since the game not only helped them to identify problems in communication within the development team as well as between the team and the Product Owner, but also to solve these problems. However, other Dyna_A members as well as Dyna_B were not surprised and stated that this was due to the fact that 5Ls has weird and confusing categories.

Senti_B said that the scores would have been better if they had played the same game continuously for several sprints and only then moved on to the next one.

Finally, some Dyna_A members admitted that they had a negative attitude toward the research, because they knew that they were going to be moved to other teams. Accordingly, they had no interest in improving the teamwork, so they did not put much effort into the assessment.

OKE_A said that 360 Degrees Appreciation is a great way to acknowledge programmers’ efforts and boost the motivation, while OKE_B and Dyna_B suggested that the game seems to be suitable during the forming stage of team building, when members are discreet with their behavior. Besides this, Dyna_B switched to a standard approach after the game finished in order to continue the retrospective and to develop action items. In turn, most of the Dyna_A members had been reluctant to play this game. However, after the first session, they admitted that the game resulted in a better atmosphere. Moreover, they said that the game is particularly useful when there are misunderstandings or arguments between team members. During one of their implementations of that game, they found out that some tensions between them were due to different approaches to developing a particular feature. Thanks to the game, they also agreed on a solution that combined both approaches. Nevertheless, all teams agreed that 360 Degrees Appreciation should be used less often than other games, for instance, every two months.

Both Dynatrace teams also suggested that Mad/Sad/Glad and Sailboat are suitable to start the adventure with retrospective games, since they are easy to learn and play. In Dyna_A, playing Mad/Sad/Glad revealed that communication between them and another team that was supposed to help with some part of the project was inadequate because of the other team. As an action item, the team leader started to talk to that team, which resulted in better communication and a faster development process. Moreover, Dyna_B recommended using Starfish when the team wants to discuss a specific issue in a systematic way. This game helped them to realize that they had wasted too much time on up-front analysis and planning instead of implementing a feature and then discussing it with the users. Thanks to Starfish, they shifted into more agile ways of developing software. Subsequently, they figured out that some features that they had considered as worthless, in fact, had only been used in a wrong way.

When it comes to Senti_A, they stated that it is difficult to choose one game over another because all of them except 360 Degrees Appreciation are of the same type with just slightly different formats.

Answers to this question were given throughout the whole meeting. Recurring opinions are summarized in Table 9. There were also some individual opinions, e.g. “Sailboat prevents team members from expressing their personal feelings on what they liked or disliked about the Sprint,” but we discarded them. Interestingly, the Dynatrace teams suggested extensions to the games they used during the research. They proposed converting 360 Degrees Appreciation into 360 Degrees Feedback by allowing for negative comments. In addition, Dyna_B proposed extending Sailboat and Starfish with an Appreciation category.

At this point, we asked each team to look back at the retrospective problems they had encountered before our project started (Table 7) and consider whether retrospective games helped them to mitigate the problems. All of the teams indicated that the introduction of retrospective games only partially solved the “little added value” problem. Actually, we expected such answers. As a side effect of our project, we found out that the main causes of reducing the frequency of retrospective meetings were not the retrospective itself, but organizational dysfunctions. Issues such as difficulties in communication between developers and the Product Owner or between different teams, forcing teams to overload the Sprint Backlog, and assigning people to multiple Scrum teams were raised during retrospectives before our project started, but nothing was done since the solution would either violate the corporate governance or required corrective actions at the organizational level. This impossibility caused frustration, which in turn caused teams to skip retrospective meetings. We also did not try to deal with these issues, since it was out of the scope of our research and the agreement with the host organizations on carrying out the project did not authorize such intervention.

On the other hand, we expected our intervention to solve other retrospective problems faced by the teams. However, only one problem, namely “unequal participation,” was fully solved in all of the teams. As for the other problems, they were mitigated to some degree. Dyna_B and Senti_A benefited the most from the new approach to the Sprint Retrospective since 3 out of 5 identified problems were fully solved. Note that although Senti_A originally believed that their participation in the discussion had been even, they acknowledged that retrospective games improved the situation. Besides this, Dyna_A claimed that the introduction of games did not reduce the number of complaints, but this was not a problem because their complaints usually revealed actual problems, while our approach allowed them to see the importance of the problems (e.g. when all team members wrote almost the same comment). Consequently, it was easier to decide which problems should be tackled first.

By changing the way retrospectives are run, not only retrospective games break the habitual routine, but they also encourage teams to look back at the Sprint from varying viewpoints. In our experience, the former prevents anyone to be half-asleep (all participating teams occasionally experienced such situations during traditional retrospectives), while the latter allows teams to generate new insights, uncover new opportunities, and identify novel solutions. The overall result is that more ideas are generated. Nevertheless, according to the Cognitive Network Model (Santanen et al. 2000), people always like to use familiar solutions to resolve the problems they encounter. Our experience confirms that introducing a new game at every meeting may be overwhelming for some teams, hence we advise Scrum Masters to ask the team whether they are ready to learn a new game or would prefer to continue practicing the previous one(s).

In our view, although retrospective games have not proved to be enjoyable enough to trigger a flow state, they still keep participants more engaged and focused as they require participants to perform a specific task at each stage of the meeting. Moreover, according to our observations retrospective games elicit fun (participants smiled more often) and positive emotions (there was no mutual hostility even if difficult issues were discussed), which improves work satisfaction, breaks down barriers, and drives participants into a collaborative mindset. Consequently, more ideas are brought out. Furthermore, the closer and more intensive interaction between team members is a good way to socialize, and improves the team spirit (it was much easier to find volunteers to pursue action items). Unfortunately, even though playfulness and interaction are central in stimulating creativity, our findings suggest that the role of retrospective games in discovering speculative breakthrough opportunities is exaggerated.

As retrospective games require each participant to write down his/her ideas on post-it notes, it is our experience that they not only challenge less vocal people who do not feel comfortable raising issues, but also some participants with passive attitude to take a more active role. As for the former, they appreciated that finally they had an opportunity to voice out. In turn, when it comes to participants with passive attitude, the majority were activated by games, but two individuals with a negative attitude toward retrospective meetings still remained withdrawn. Besides, by encouraging equal participation, retrospective games seem to contribute to the divergence of views, which in turn creates the potential for constructive controversy. Overall, as a result of game-based retrospectives, more ideas are contributed and more opinions are voiced out.

Retrospective games provide structure for retrospective meetings and focus the discussion. As a consequence, we observed a paradigm shift from complaint-orientation to constructive solution-oriented discussions. Before implementing retrospective games in Dynatrace as well as SentiOne teams, barren discussions and complaints sometimes went on so long that there was not enough time to discuss all essential problems, thoroughly identify the sources of the problems or propose action items. Thereby, there was often an impression of unfinished discussion. Thanks to retrospective games, it was easier to identify essential problems (usually those that were raised by a number of post-it notes) and discuss them in the context of their potential resolutions, which enhanced the meeting productivity. On the other hand, in both OKE teams, game-based retrospectives usually lasted longer than their traditional counterparts as the discussions were much more comprehensive. Nevertheless, this may be considered as a disadvantage by some traditional managers especially if they control the team with the help of metrics.

According to our observations, retrospective games also make it easier to facilitate the meeting and reduce the preparation effort. Indeed, to some degree, game-based retrospectives are self-facilitated. For instance, during the second round of game-based retrospectives, when all the rules and steps were known, different developers voluntarily took over the role of a moderator and facilitated the meeting. This approach was especially welcomed by Dyna_A, whose some team members had complained that their Scrum Master had overcontrolled the retrospective meetings.

Based on our experience, we think that developers who perceive retrospective meetings as a waste of time probably will not change their minds after implementing retrospective games, especially if their companies focus too intently on short-term efficiency gains (indeed, in all participating teams, retrospective meetings were sometimes canceled or postponed because of a more important meeting or the time pressure to complete the work). Furthermore, retrospective games may be perceived as childish and be embarrassing for reserved people. Indeed, some participants who considered themselves as “serious developers” insisted that they “could have contributed more to the project by coding instead of playing retrospective games”. As such, we believe that it is better not to force them to take part in retrospectives, because they may bring negative energy and even sabotage the meeting. However, agile coaches may perceive our view to be too simplistic and even dangerous as conducting a retrospective without the whole team present goes against the Scrum Guide. Accordingly, this issue requires further investigation. We also found that there are usually some individuals who do not contribute to the meeting, but take a free ride on those who do contribute. Probably, the best solution for both issues would be to find the right people – those who can leverage agile practices in the way that they are intended – at the stage of team building. Our discussion may be supplemented by a reference to Ellis et al. (2014) who stated that the effectiveness of systematic reflection depends on person-based factors and is higher for learning-oriented people who are conscientious and emotionally stable, can accurately evaluate their performance, and enjoy effortful cognitive activity.

In our judgment, the feedback on our intervention obtained during the focus groups is more positive than that obtained through the questionnaires. The reason for this difference is probably that opinions put forward during a focus group must be supported by a reasonable explanation and hence they must be genuine. On the contrary, if no justification is required in a questionnaire, it is likely that some participants may try to sabotage the results or they simply do not reflect deeply on the results.

The variability in the responses of individuals as well as the teams to the games is quite high. This means that what works for some may not work for others and that the preference of the game probably depends on the personality of the person. Nonetheless, some general findings on the advantages and disadvantages of each investigated game are presented in Table 9, while below we present further guidelines for Scrum Masters. In our view, Sailboat is a good choice for teams who are starting their adventure with game-based retrospectives. It is simple, but it keeps all team members pulling in the same direction and helps them think outside of the box, which results in fresh new ideas. Besides, drawing a sailboat together promotes a sense of community. However, after a few sessions, we advise teams to switch to Mad/Sad/Glad or Starfish, because otherwise Sailboat may become boring. Mad/Sad/Glad has the most straightforward rules and the quickest play time. In contrast to Sailboat, it allows the team to discuss the past Sprint from the feeling perspective and release negative emotions. Nevertheless, according to the participants’ experience, the game does not cover all of the topics that should be discussed during a retrospective, while running the game does not directly generate constructive suggestions for improvements. Mood++ addresses the latter and provides an opportunity for team members to compliment each other. In contrast to Mad/Sad/Glad, Starfish covers all aspects of the traditional agile retrospective and allows for a straightforward application of the findings, as stated by some participants. Another game that also covers all aspects of the traditional agile retrospective is 5Ls. However, according to our findings, its category names may cause confusion and anger among participants, so we do not recommend it to teams who have just started running game-based retrospectives. Finally, a completely different type of game is 360 Degrees Appreciation. It helps the team to loosen up a bit and relax after a stressful sprint. It is our experience that the generated feedback is usually enthusiastically welcomed and boosts the team spirit. Nevertheless, the game is not a standalone retrospective technique.

Our observations agree with findings of other researchers (Nerur et al. 2005; Fruhling and Tarrell 2008; Cao et al. 2009; Hanslo and Mnkandla 2018; Gupta et al. 2019; VersionOne, 2020; Spiegler et al. 2021) that organizational culture is a significant barrier to the successful adoption of agile practices especially for bureaucratic organizations steeped in plan-driven methods. Due to the tayloristic past, project managers tend to stick with traditional viewpoints on hierarchy in terms of reporting, resource allocation, decision-making, etc. and are reluctant to grant leadership roles to the teams (Moe et al. 2012; Spiegler et al. 2021). However, agile methods require a shift from command-and-control project management to shared decision-making and self-management teams, where project managers are facilitators who coordinate collaborative efforts and create a supportive organizational context (Nerur et al. 2005; Dybå et al. 2014b; Theobald et al. 2020). Therefore, in order to enjoy the full benefits of Scrum, agile values and principles must be adopted not only at the team level but also at the organizational level.

Moreover, we found that continuous process improvement is hard to achieve even though inspect-and-adapt cycles and reflection are central concepts in agile development. In 5 out of 6 teams participating in the project, the implementation of Scrum deviated from the framework. In all cases, the deviations covered up organizational dysfunctions which, if addressed and removed, would allow the team to be more effective and efficient. What is more, in our opinion such dysfunctions can be discovered in the course of traditional retrospectives. In turn, the main reason for reducing the frequency of retrospective meetings was not the way in which the meetings were run, but unproductive discussions on recurring problems that were beyond the control of the team (e.g. unavailability of the Product Owner for requirements clarification, lack of communication between Scrum teams, or distribution of team members across multiple rooms). Indeed the phenomenon of repeating discussions was also experienced by teams investigated by Lehtinen et al. (2017). This again indicates that making the transition to the agile mindset (Schön et al. 2017; Miler and Gaida 2020; Özkan et al. 2020) and understanding agile values is still challenging for some companies. Taking into account that we were also not authorized to fix organizational dysfunctions (the scope of our agreement was limited to improving retrospectives), we decided not to start a new AR cycle, because we believed that this would not bring much added value.