A framework for the design and verification of software measurement methods

Abstract

At the core of any engineering discipline is the use of measures, based on ISO standards or on widely recognized conventions, for the development and analysis of the artifacts produced by engineers. In the software domain, many alternatives have been proposed to measure the same attributes, but there is no consensus on a framework for how to analyze or choose among these measures. Furthermore, there is often not even a consensus on the characteristics of the attributes to be measured.

In this paper, a framework is proposed for a software measurement life cycle with a particular focus on the design phase of a software measure. The framework includes definitions of the verification criteria that can be used to understand the stages of software measurement design. This framework also integrates the different perspectives of existing measurement approaches. In addition to inputs from the software measurement literature the framework integrates the concepts and vocabulary of metrology. This metrological approach provides a clear definition of the concepts, as well as the activities and products, related to measurement. The aim is to give an integrated view, involving the practical side and the theoretical side, as well as the basic underlying concepts of measurement.

Introduction

Hundreds of software measures have been proposed in the software engineering literature for dozens of software attributes, including many alternative measures for the same attributes. However, understanding, classifying, evaluating and comparing them remains a challenge to both the industrial and the academic communities.

In the academic community, for example, there is no consensus on how to conduct an analysis of the measures proposed: that is, there is no recognized framework for doing so. Such a framework would allow researchers to carry out comparative studies of multiple alternative measures for measuring the same attributes, and then to publish their studies and recommendations so that industry has the necessary information on which to base their selection of a measurement method appropriate to their needs:

Some initial frameworks have been proposed (e.g. Fenton and Pfleeger, 1997, Jacquet et al., 1997, Kitchenham et al., 1995, Lopez et al., 2003, Schneidewind, 1992, Sellami et al., 2003, Weyuker, 1988, Zuse, 1999) to structure and/or to analyze measurement methods, as well as to discuss their validity issues. However, these frameworks all use different terminology. Our review highlights three needs:

• the need to fix a unified vocabulary for the main measurement concepts and to relate it coherently to major literature sources;

• the need to detail the very first phase of the measurement lifecycle, i.e. the design of a measurement method;

• the need to derive verification criteria for software measures.

On the basis of the existing literature, this paper presents a general framework for software measurement which aims at responding to the above needs. The first contribution of this paper is to provide a comprehensive glossary which goes beyond a simple literature survey to put up a global view integrating theoretical and practical issues. The second contribution is to develop in details the phase of the design of a measurement method and thus to derive verification criteria related to such designs of measurement methods. The intended use of this framework is to provide a structured analysis framework for understanding and evaluating existing measurement methods documented in the literature and in use in industry.

The paper is organized as follows. Section 2 presents a unified vocabulary for key concepts in measurements, many of them adopted from the metrology domain. Section 3 presents some of the peculiarities of software that have to be taken into account when designing measures for software. Section 4 presents the design phase within the measurement life cycle. The different activities of that phase will be detailed in Sections 5 Measurement design: defining the measurement principle, 6 Measurement design: determining the measurement method, 7 Measurement design: determining the measurement procedure. Section 8 summarizes the products of the measurement design phase and Section 9 the verification activities related to it. A summary and a discussion are presented in Section 10.