Rapid quality assurance with Requirements Smells

Highlights

• We clarify the notion of Requirements Smells.

• We integrate Requirements Smells into quality assurance for RE artifacts.

• We present Smella, a prototypical realization of our Smell Detection approach.

• We conduct an empirical investigation on the usefulness of Requirements Smells.

• Smells can uncover many practically relevant defects in a reasonably precise way.

Abstract

Bad requirements quality can cause expensive consequences during the software development lifecycle, especially if iterations are long and feedback comes late. We aim at a light-weight static requirements analysis approach that allows for rapid checks immediately when requirements are written down. We transfer the concept of code smells to requirements engineering as Requirements Smells. To evaluate the benefits and limitations, we define Requirements Smells, realize our concepts for a smell detection in a prototype called Smella and apply Smella in a series of cases provided by three industrial and a university context. The automatic detection yields an average precision of 59% at an average recall of 82% with high variation. The evaluation in practical environments indicates benefits such as an increase of the awareness of quality defects. Yet, some smells were not clearly distinguishable. Lightweight smell detection can uncover many practically relevant requirements defects in a reasonably precise way. Although some smells need to be defined more clearly, smell detection provides a helpful means to support quality assurance in requirements engineering, for instance, as a supplement to reviews.

Introduction

Defects in requirements, such as ambiguities or incomplete requirements, can lead to time and cost overruns in a project (Méndez Fernández and Wagner, 2015). Some of the issues require specific domain knowledge to be uncovered. For example, it is very difficult to decide whether a requirements artifact is complete without domain knowledge. Other issues, however, can be detected more easily: If a requirement states that a sensor should work with sufficient accuracy without detailing what sufficient means in that context, the requirement is vague and consequently not testable. The same holds for other pitfalls such as loopholes: Phrasing that a certain property of the software under development should be fulfilled as far as possible leaves room for subjective (mis-)interpretation and, thus, can have severe consequences during the acceptance phase of a product (Femmer, Méndez Fernández, Juergens, Klose, Zimmer, Zimmer, 2014, ISO, IEC, IEEE).

To detect such quality defects, quality assurance processes often rely on reviews. Reviews of requirements artifacts, however, need to involve all relevant stakeholders (Salger, 2013), who must manually read and understand each requirements artifact. Moreover, they are difficult to perform. They require a high domain knowledge and expertise from the reviewers (Salger, 2013) and the quality of their outcome depends on the quality of the reviewer (Zelkowitz et al., 1983). On top of all this, reviewers could be distracted by superficial quality defects such as the aforementioned vague formulations or loopholes. We therefore argue that reviews are time-consuming and costly.

Therefore, quality assurance processes would benefit from faster feedback cycles in requirements engineering (RE), which support requirements engineers and project participants in immediately discovering certain types of pitfalls in requirements artifacts. Such feedback cycles could enable a lightweight quality assurance, e.g., as a complement to reviews.

Since requirements in industry are nearly exclusively written in natural language (Mich et al., 2004) and natural language has no formal semantics, quality defects in requirements artifacts are hard to detect automatically. To face this challenge of fast feedback and the imperfect knowledge of a requirement’s semantics, we created an approach that is based on what we call Requirements (Bad) Smells. These are concrete symptoms for a requirement artifact’s quality defect for which we enable rapid feedback through automatic smell detection.

In this paper, we contribute an analysis of whether and to what extent Requirements Smell analysis can support quality assurance in RE. To this end, we

• define the notion of Requirements Smells and integrate the Requirements Smells¹ concept into an analysis approach to complement (constructive and analytical) quality assurance in RE,

• present a prototypical realization of our smell detection approach, which we call Smella, and

• conduct an empirical investigation of our approach to better understand the usefulness of a Requirements Smell analysis in quality assurance.

Our empirical evaluation involves three industrial contexts: The companies Daimler AG as a representative for the automotive sector, Wacker Chemie AG as a representative for the chemical sector, and TechDivison GmbH as an agile-specialized company. We complement the industrial contexts with an academic one, where we apply Smella to 51 requirements artifacts created by students. With our evaluations, we aim at discovering the accuracy of our smell analysis taking both a technical and a practical perspective that determines the context-specific relevance of the detected smells. We further analyze which requirements quality defects can be detected with smells, and we conclude with a discussion of how smell detection could help in the (industrial) quality assurance (QA) process.

This article extends our previously published workshop paper (Femmer et al., 2014b) in the following aspects: We provide a richer discussion on the notion of Requirements Smell and give a precise definition. We introduce our (extended) tool-supported realization of our smell analysis approach and outline its integration into the QA process. We extend our first two case studies with another industrial one as well as with an investigation in an academic context to expand our initial empirical investigations by

• investigating the accuracy of our smell detection including precision, recall, and relevance from a practical perspective,

• analyzing which quality defects can be detected with smells and

• gathering practitioner’s feedback on how they would integrate smell detection in their QA process considering both formal and agile process environments.

The remainder of this paper is structured as follows. In Section 2, we describe previous work in the area. In Section 3, we define the concept of Requirements Smells and describe how we derived a set of Requirements Smells from ISO 29148. We introduce the tool realization in Section 4 and discuss the integration of smell detection in context of quality assurance in Section 5. In Section 6, we report on the empirical study that we set up to evaluate our approach, before concluding our paper in Section 7.