Agile Processes in Software Engineering and Extreme Programming

Refactoring is one of the key practices in Extreme Programming and other agile methods. Duplicated code is one of the most pervasive and pungent smells to remove from source code through refactoring. Duplicated code has been largely studied in the literature, and different types of duplications, or “clones”, have been identified. Some studies analyzed in details the problems caused by clones in the code, others outlined also the difficulties in removing clones, and the cases in which it could be better not removing them. The refactoring cost for removing clones can be very high, also due to the different choices on the possible refactoring steps. In this paper, we describe our approach and tool developed with the aim to suggest the best refactorings to remove clones in Java code. Our approach is based on the classification of the clones in terms of their location in a class hierarchy, and allows to choose among a restricted set of refactorings, which are then evaluated using multiple criteria. We provide a validation of the effectiveness of the approach.

Agile software development is increasingly adopted by companies evolving and maintaining software products to support better planning and tracking the realization of user stories and features. While convincing success stories help to further spread the adoption of Agile, mechatronics-driven companies need guidance to implement Agile for non-software teams. In this comparative case study of three companies from the Nordic region, we systematically investigate expectations and challenges from scaling Agile in organizations dealing with mechatronics development by conducting on-site workshops and surveys. Our findings show that all companies have already successfully implemented Agile in their software teams. The expected main benefit of successfully scaling agile development is a faster time-to-market product development; however, the two main challenges are: (a) An inflexible test environment that inhibits fast feedback to changed or added features, and (b) the existing organizational structure including the company’s mind-set that needs to be opened-up for agile principles.

It is a conundrum that agile projects can succeed ‘without requirements’ when weak requirements engineering is a known cause for project failures. While Agile development projects often manage well without extensive requirements documentation, test cases are commonly used as requirements. We have investigated this agile practice at three companies in order to understand how test cases can fill the role of requirements. We performed a case study based on twelve interviews performed in a previous study. The findings include a range of benefits and challenges in using test cases for eliciting, validating, verifying, tracing and managing requirements. In addition, we identified three scenarios for applying the practice, namely as a mature practice, as a de facto practice and as part of an agile transition. The findings provide insights into how the role of requirements may be met in agile development including challenges to consider.

Background

: Agile software development has become a popular way of developing software. Scrum is the most frequently used agile framework, but it is often reported to be adapted in practice.

Objective

: Thus, we aim to understand how Scrum is adapted in different contexts and what are the reasons for these changes.

Method

: Using a structured interview guideline, we interviewed ten German companies about their concrete usage of Scrum and analysed the results qualitatively.

Results

: All companies vary Scrum in some way. The least variations are in the Sprint length, events, team size and requirements engineering. Many users varied the roles, effort estimations and quality assurance.

Conclusions

: Many variations constitute a substantial deviation from Scrum as initially proposed. For some of these variations, there are good reasons. Sometimes, however, the variations are a result of a previous non-agile, hierarchical organisation.

Software startups are newly created companies designed to grow fast. The uncertainty of new markets and development of cutting-edge technologies pose challenges different from those faced by more mature companies. In this study, we focus on exploring the key challenges that early-stage software startups have to cope with from idea conceptualization to the first time to market. To investigate the key challenges, we used a mixed-method research approach which includes both a large-scale survey of 5389 responses and an in-depth multiple-case study. The initial findings reveal that thriving in technology uncertainty and acquiring the first paying customer are among the top challenges, perceived and experienced by early-stage software startups. Our study implies deeper issues that early-stage software startups need to address effectively in validating the problem-solution fit.

As agile is maturing and becoming more widely adopted, it is important that researchers are aware of the challenges faced by practitioners and organisations. We undertook a thematic analysis of 193 agile challenges collected at a series of agile conferences and events during 2013 and 2014. Participants were mainly practitioners and business representatives along with some academics. The challenges were thematically analysed by separate authors, synthesised, and a list of seven themes and 27 sub-themes was agreed. Themes were Organisation, Sustainability, Culture, Teams, Scale, Value and Claims and Limitations. We compare our findings against previous attempts to identify and categorise agile challenges. While most themes have persisted we found a shift of focus towards sustainability, business engagement and transformation, as well as claims and limitations. We identify areas for further research and a need for innovative methods of conveying academic research to industry and industrial problems to academia.

Startups are creating innovative new products and services while seeking fast growth with little resources. The capability to produce software products with good user experience (UX) can help the startup to gain positive attention and revenue. Practices and needs for UX design in startups are not well understood. Research can provide insight on how to design UX with little resources as well as to gaps about what kind of better practices should be developed. In this paper we describe the results of an interview study with eight startups operating in Finland. Current UX practices, challenges and needs for the future were investigated. The results show that personal networks have a significant role in helping startups gain professional UX advice as well as user feedback when designing for UX. When scaling up startups expect usage data and analytics to guide them towards better UX design.

Many software development projects fail due to problems in requirements, scope, and collaboration. This paper presents a case study of the mismatch between the expectations of Product Owners and the outcome of the development in a large distributed Scrum organization. The data was collected in retrospective meetings involving a team of Product Owners and two software development teams. A focused root cause analysis of the problem “Why the expectations of Product Owners do not meet the outcome of development teams?” was conducted. The analysis aimed at explaining why the problem occurred and how the causes were related to one another. The outcomes were analyzed both quantitatively and qualitatively. Our results illustrate the challenges of implementing the Product Owner role in the context of complex, high-variability requirements and distributed development. We highlight the importance of true collaboration, effective requirements specification activities, and sufficient resources for the Product Owner role.

To help developers during the Scrum planning poker, in our previous work we ran a case study on a Moonlight Scrum process to understand if it is possible to introduce functional size metrics to improve estimation accuracy and to measure the accuracy of expert-based estimation. The results of this original study showed that expert-based estimations are more accurate than those obtained by means of models, calculated with functional size measures. To validate the results and to extend them to plain Scrum processes, we replicated the original study twice, applying an exact replication to two plain Scrum development processes. The results of this replicated study show that the accuracy of the effort estimated by the developers is very accurate and higher than that obtained through functional size measures. In particular, SiFP and IFPUG Function Points, have low predictive power and are thus not help to improve the estimation accuracy in Scrum.

An experiment-driven approach to software product and service development is gaining increasing attention as a way to channel limited resources to the efficient creation of customer value. In this approach, software functionalities are developed incrementally and validated in continuous experiments with stakeholders such as customers and users. The experiments provide factual feedback for guiding subsequent development. Although case studies on experimentation in industry exist, the understanding of the state of the practice and the encountered obstacles is incomplete. This paper presents an interview-based qualitative survey exploring the experimentation experiences of ten software development companies. The study found that although the principles of continuous experimentation resonated with industry practitioners, the state of the practice is not yet mature. In particular, experimentation is rarely systematic and continuous. Key challenges relate to changing organizational culture, accelerating development cycle speed, and measuring customer value and product success.

A successful software project is the result of a complex process involving, above all, people. Developers are the key factors for the success of a software development process and the Agile philosophy is developer-centred. Developers are not merely executors of tasks, but actually the protagonists and core of the whole development process. This paper aims to investigate social aspects among developers working together and the appeal of a software project developed with the support of Agile tools such as Agile boards. We studied 14 open source software projects developed using the Agile board of the JIRA repository. We analysed all the comments committed by the developers involved in the projects and we studied whether the politeness of the comments affected the number of developers involved over the years and the time required to fix any given issue. Our results show that the level of politeness in the communication process among developers does have an effect on the time required to fix issues and, in the majority of the analysed projects, it has a positive correlation with attractiveness of the project to both active and potential developers. The more polite developers were, the less time it took to fix an issue, and, in the majority of the analysed cases, the more the developers wanted to be part of project, the more they were willing to continue working on the project over time.

Agile software development projects depend upon roles located outside Agile teams such as User Experience Designers and Software Architects to support teams. As external specialists’ expertise is valuable to Agile teams, further investigation is needed to explore the relationships between Agile teams and external specialists in coordinating external expertise. Through a Grounded Theory study involving 47 Agile practitioners and external specialists, we discovered five factors that Agile teams and external specialists need to consider when coordinating external expertise:

Availability, Agile Mindset, Stability, Knowledge Retention

and

Effective Communication

. We present strategies for each factor as a guideline for coordinating external expertise in teams. This study helps Agile teams to manage and utilize external expertise resources effectively.

Delivering value to customers in real-time requires companies to utilize real-time deployment of software to expose features to users faster, and to shorten the feedback loop. This allows for faster reaction and helps to ensure that the development is focused on features providing real value. Continuous delivery is a development practice where the software functionality is deployed continuously to customer environment. Although this practice has been established in some domains such as B2C mobile software, the B2B domain imposes specific challenges. This article presents a case study that is conducted in a medium-sized software company operating in the B2B domain. The objective of this study is to analyze the challenges and benefits of continuous delivery in this domain. The results suggest that technical challenges are only one part of the challenges a company encounters in this transition. The company must also address challenges related to the customer and procedures. The core challenges are caused by having multiple customers with diverse environments and unique properties, whose business depends on the software product. Some customers require to perform manual acceptance testing, while some are reluctant towards new versions. By utilizing continuous delivery, it is possible for the case company to shorten the feedback cycles, increase the reliability of new versions, and reduce the amount of resources required for deploying and testing new releases.

As the interest in DevOps continues to grow, there is an increasing need for software organizations to understand how to adopt it successfully. This study has as objective to clarify the concept and provide insight into existing challenges of adopting DevOps. First, the existing literature is reviewed. A definition of DevOps is then formed based on the literature by breaking down the concept into its defining characteristics. We interview 13 subjects in a software company adopting DevOps and, finally, we present 11 impediments for the company’s DevOps adoption that were identified based on the interviews.

In software development organizations, large-scale distributed projects pose many challenges, such as hierarchical requirements, large team size, and managing workflow. Agile methods, like Scrum, seem to have limitations in addressing those issues. Kanban offers an interesting alternative in this concern by setting work-in-progress (WIP) limits to manage flow and establishing visibility of requirements using a visual signaling system. However, only few empirical studies have investigated scaling Kanban for large organizations. To address this concern, a case study was conducted in a large multisite company to identify the challenges and possible solutions in scaling Kanban. During the study, defining WIP limits, coordinating with distributed teams, and dividing features between teams were found as major challenges. Setting WIP limits by common agreement between teams and visualizing product backlogs and teams on electronic board, with teams pulling features from the board, were found as possible solutions for overcoming the mentioned challenges.

Uncertainty is an inevitable fact of software development that can determine success or failure of entire projects. Although often associated with risk or threat, uncertainty bears much overlooked qualities regarding market opportunities. Adopting a conceptualization of uncertainty that includes both threat and opportunity, we investigate different possibilities to manage the uncertainties preceding the creation of novel ideas and innovation. In this qualitative study, we empirically explore the concept of uncertainty beyond the focus on requirement uncertainty that prevails in the literature and explicitly emphasize its multidimensionality. We argue that the failure to distinguish between different dimensions of uncertainty can lead to their inadequate management at high cost. Our results show how teams exploit practices to manage different uncertainties in order to mitigate threats while remaining open to opportunities. We discuss the implications of our findings for product development teams as well as for the design of supportive organizational structures.

Organizational ambidexterity is the ability to be aligned and efficient in combining current resources and demands (exploitation) as well as adaptive and innovative due to changing conditions and demands (exploration). Maturity in software development is defined over exploitation – through processes definition and control. We argue in this study that mature agile software development is also exploratory – adaptive and innovative. Thus, our objective is to verify how ambidexterity occurs in mature agile software development. The research approach is a single case study with analysis of qualitative data. Our findings show how a mature team is managed by ambidextrous strategies.

Modern high-performing software product development organizations are nowadays more and more often transforming their operations towards continuous higher-level ends. In general, customer satisfaction (CS) is such a goal. This paper presents an approach of gauging and improving customer satisfaction in an industrial B2B product development project organization for continuous customer experience (CX) management. In order to do that, the Customer Satisfaction Index (CSI) used in the company was systematized with an applied impact analysis technique. The resulting artifact (called

CSI Impact Mapping Grid

) combines a set of satisfier improvement strategies derived from our initial work with the GQM

+

Strategies method coupled with the company assets (capabilities) and insights. It is furthermore aimed to be integrated with rich online measurement inputs for real-time predictive feedback. Such transparency across the whole organization enables employees to realize and insightfully support − even proactively in real time − the various cause-effect relationships of the CS/CX.

DevOps has been identified as an important aspect in the continuous deployment paradigm in practitioner communities and academic research circles. However, little has been presented to describe and formalize what it constitutes. The absence of such understanding means that the phenomenon will not be effectively communicated and its impact not understood in those two communities. This study investigates the elements that characterize the DevOps phenomenon using a literature survey and interviews with practitioners actively involved in the DevOps movement. Four main dimensions of DevOps are identified: collaboration, automation, measurement and monitoring. An initial conceptual framework is developed to communicate the phenomenon to practitioners and the scientific community as well as to facilitate input for future research.

To continuously deliver value both in short-term and long-term, a key goal for large product lines companies is to combine Agile Software Development with the continuous development and management of software architecture. We have conducted interviews involving several roles at 3 sites from 2 large companies employing Agile. We have identified current architect roles and gaps in the practices employed at the organizations. From such investigation, we have developed an organizational framework, CAFFEA, for Agile architecting, including roles, teams and practices.

This paper investigates if Code-Churn, Lines of Code (LoC), Duplicated Code, Complexity and Technical Debt (TD) can inform a coder where to prioritize refactoring. A mature code-base of 1 300 000 LoC in approximately 5000 files for a period of 20 months has been examined. The result is that code-churn is the strongest variable out of the studied variables followed by LoC and TD. Multiplying with code-churn strengthens LoC and TD even more, making them the strongest indicators of refactoring potential.

The presented study examines the prevailing espoused values of some of the most successful software organizations and evaluates the extent to which they are promoted in the agile ideology. Its objective is to determine the level of value congruence and whether it could be used to further explain the widespread adoption of agile software development. Its findings reveal a perfect match between the prevailing espoused values of the examined global software leaders and the agile ideology, and suggest that the popularity of the agile methods might be explained through their capability to increase customer value, facilitate collaboration and teamwork, secure continuous improvement and ensure high quality of delivered products and services.

Agile development methods are increasingly used in large projects, with many development teams. Because acceptance testing can require a large chain of features to be completed, testing is often carried out late in such projects. In this experience report, we describe a large project where 11 development teams delivered a system in 12 three week iterations. We also describe how the focus of test activities was shifted towards the earlier phases of development, what we call “shift left”. This involved shifting the focus both within the iterations, and in how the overall testing work was organized. We describe the results of this change, and provide recommendations for how to organize test work in future large-scale development projects.

Gathering early feedback on features is critical to many projects. Many Agile methodologies define feedback loops. Often, the feedback loop for completed features only closes after the iteration finishes. In this paper we will introduce a way of closing this feedback loop early, by means of continuous deployment. This also lowers the deployment effort for developers, increasing their happiness.

This experience report summarises the decision-making process used at Aptivate, focusing on the decision to create ‘standing teams’ with delegated authority to plan the distribution of their own work across multiple projects. The paper presents the company approach to consensus decision-making, and how this has supported company culture and values as well as enabled this change with full participation. This experience report covers the time period from September 2014 to March 2015.

This paper describes how lean elements have been applied in a large company to change existing agilean culture towards innovation culture. Innovation concentrates on radical, new business innovations but covers product and process innovations as well. The main motivation and need to build the innovation culture was an assumption that the company has a lot of competence and innovation potential not utilized. The final goal is to measure if the actions taken really have an impact to the amount and quality of new, radical innovations and business growth of the company. This paper is limited to the intermediary results achieved after the first year being (1) ideas-to-innovations value stream established (2) idea flow and positive pull created among personnel and (3) group of perfection practices has been stabilized for continuous improvement. We plan to describe, measure and analyze concrete examples of radical innovations in the future

This paper tells the story of my Agile Journey over the last 15 years. It is neither exceptionally rosy nor excessively pessimistic. It is an attempt to give an honest experience report.

For an organization wishing to be more agile after working waterfall for years, it is not enough to just start learning and implementing new ways of working. There must be a parallel activity, at least equally important, of dealing with the organizational culture changes required to support this transformation. In Cisco I deal with those on a daily basis. An organizational culture is much harder to change than work methods. It involves feelings, perceptions and fears, so it is advisable to be aware of the importance and invest in dealing with it. This experience report details examples of these culture aspects, how we deal with them and some tips that can help make such transformation successful.

The Guide Board is an artefact that supports continuous improvement of practices and interactions within a team, with the same materials as classical agile artefacts. It represents the conclusions of a team’s retrospectives as “guides” to make them tangible. By specifying how these guides are visualised and handled depending on their actual application, the Guide Board improves the production system efficiency by increasing the critical reuse of previous conclusions. A successful application substantially increases the team’s self-awareness of its culture, and makes its habits more visible to other stakeholders, thus improving communication. Finally, it improves the readability of its social rules to newcomers, thus supporting integration of new hires and therefore growth.

The Modtalk project is an effort to create a production ready tool chain for compiling Smalltalk programs into standalone executables. This development project entailed writing and testing code in a cross development environment, in a target executable environment, and a C based runtime that supports the compiled executable. We discovered that test-driven development supported team communication, focused design efforts, and produced code artifacts that documented the system. In the process, we also discovered that tests were often brittle and would break for a variety of reasons. We identify why some of our tests were brittle and ways in which we responded when tests failed.

The story is about how we decided to activate potential of the teams in a product portfolio with 21 products and more than 400 people in 5 countries and started to build a learning organization where product teams share experience and knowledge with each other and improve. And this all driven by their own motivation. How? By organizing a community a bit different way, using peer hands-on support in between community sessions and by gamifying personal growth. This paper describes how we designed the community, piloted it for 3 months and summarizes the results. Teams are now much more connected cross team/product borders and they actively help each other to improve.

This paper describes management strategies for continuous improvement in agile software development teams. We have applied these strategies in a Brazilian team, which was born in 2009 and now grew into a headquarter of the company with ninety people. We have currently reached lean flow state with constant throughput, reduced lead time and enhanced quality by cutting bugs rate in half. In a continuous improvement cycle, our management strategies are based on looking at the situation, sensemaking the situation and providing simple responses. We describe how we applied these strategies to learn how to correctly limit work in progress (WIP) and to face challenges with coaching, estimates, team motivation, sprints and pair programming.

At Unruly we are constantly trying to turn up the dial on our XP practices, and in the second half of 2014 we started to take the step from Pair Programming on all production code to Mob Programming with the entire team. This report shares experiences that Unruly has gained in pushing the boundaries of Extreme Programming.