Identification of factors that influence defect injection and detection in development of software intensive products

https://doi.org/10.1016/j.infsof.2006.09.002Get rights and content

Abstract

The objective of this study is the identification of factors that influence defect injection and defect detection. The study is part of a broader research project with the goal to lower the number of residual defects in software intensive products, by using the influencing factors to decrease injection of defects and to increase detection of defects. As a first step, we performed an extensive literature search to find influencing factors and processed the factors to achieve consistently formulated sets of factors without duplications. As a second step, we used a cluster analysis to reduce the number influencing factors to manageable-sized sets for practical application. As a last step, final groupings of factors were obtained by expert interpretation of the cluster analysis results. These steps were separately performed for defect injection and detection influencing factors, resulting in sets of, respectively, 16 and 17 factors. Finally, the resulting factor groupings were evaluated.

The findings (1) are the basis for further research focusing on a framework for lowering residual defects, (2) already provide information to enable practitioners to devise strategies for lowering residual defects, and (3) may create awareness in organizations to reconsider policies regarding development and Verification & Validation.

Introduction

The objective of this paper is the identification of factors influencing defect injection (DI) and defect detection (DD) for software intensive products. This study is part of a larger research project with the ultimate goal being the design of a framework for obtaining higher product quality for software intensive products. The framework is particularly intended for, but not limited to, products developed by virtual teams (that is, teams distributed across space, time, and organizational boundaries).

Product quality is regarded as a main determinant of customer satisfaction [12]. Product quality is also of high economical importance because lack of product quality will keep burdening the development organization far beyond the termination of the development project; e.g., with high and prolonged maintenance costs, loss of market share, lawsuits, and costly product recalls [13].

However, product quality is a multidimensional concept. One can look at it from a variety of different viewpoints, with different stakeholders for the different viewpoints [9], [12]. Garvin carried out a wide-range study of (product) quality and recognized five different views: the transcendental view, the manufacturing view, the product view, the user’s view, and the value-for-money view [9]. The manufacturing view is the predominant view on product quality in many industries, including the software industry [12]. This view is also expressed as “conformance to specification” [5] or as “conformance to requirements” [12]; that is, conformance to both customer requirements (leading to customer satisfaction) and technical requirements (leading to product reliability, which is also an important constituent of customer satisfaction). Non-conformances to specifications (which we shall call defects) will typically have a negative impact on product quality whatever the point of view.

From elicitation of requirements for the product to the final delivery of the product, the development process is complex. It involves a series of stages, each with feedback paths. In each stage, an intermediate deliverable is produced for an intermediate user in the next stage. Each stage also receives an intermediate deliverable from the preceding stage. Each intermediate deliverable has certain quality attributes that affect the quality of the final product. If each stage of the development process meets the requirements of its intermediate user (the next stage), the final product thus developed and produced will meet the specified requirements. This statement, of course, is an oversimplification of reality because in each stage many factors exist that will influence the ability of the stage to fulfill its requirements completely. Consequently, defects may be injected in every stage of development. Hopefully, defects are detected (and subsequently removed) by the Verification & Validation process performed during all stages of development. The injection of defects should be minimized and the detection of defects should be maximized. The net effect would be a reduction of defects, in line with Boehm’s Defect Introduction and Removal Model [2], which is analogous the “tank and pipe model” introduced by Capers Jones [11], illustrated in Fig. 1.

The defect introduction pipes represent phases and activities of a project’s lifecycle. Defects injected add to those already injected in previous phases and might propagate to subsequent phases. The goal of defect detection is to minimize the amount of defects that propagates to the subsequent phases: defects are drained off through various defect removal pipes.

The usage of influencing factors for managing aspects of project performance, also referred to as drivers, has a long history in the software engineering discipline. For example, in the early design of the COCOMO software project cost model, Boehm cited already earlier works on drivers affecting project performance [2].

The influencing factors may cause people to make errors. Two basic types of errors can be distinguished [18]: errors of action (“good plan, but bad execution”) and errors of intention (“execution as planned, but bad plan”). We define a DI-influencing factor as a stimulant or circumstance that may cause people to make errors that may lead to the introduction of faults in work products. If faults are not detected, these may eventually manifest as a failure of the end product, that is, as a defect. Defect injection influencing factors may cause a bad execution of an otherwise sound development process (“errors of action”) or may cause a flawed development process in itself (“error of intention”). Likewise, we define a DD-influencing factor as a stimulant or circumstance affecting the ability to detect defects in the V&V (=Verification & Validation) of a product in any of its development stages. DD-influencing factors may cause errors in the execution of an otherwise sound V&V process or may cause a flawed V&V process in itself. In both cases, defects may and will be prevented from being found.

The influencing factors can be used to take measures during project initiation and execution to avoid errors of action and intention, or at least to limit their consequences. DI-influencing factors are to be used for taking measures to minimize the injection of defects while DD-influencing factors are to be used for taking measures to maximize defect detection (and subsequent defect removal).

As far as we are aware, very few research efforts have been conducted with respect to factors influencing detect injection and the ability to detect defects. The few studies that address these, to some degree at least, use several approaches to address the factors.

In the design of his software cost estimation model COCOMO, Boehm performed a literature search to find cost-influencing factors [2]. Unfortunately, he did not motivate his choice for a literature search, nor did he describe the details of his literature search. Chulani anad Boehm [3] used a factor-based approach in their CoQualMo model [4], an extension of the existing COCOMO II model. A Delphi procedure was applied to estimate the degree of influence of identified factors on defect injection. Factors addressing defect detection were not incorporated.

Zhang and Pham [28] used a survey instrument in which 23 representatives from 13 companies identified 32 factors impacting software reliability. However, they did not motivate their choice for the survey method. Dyba [6] utilized expert opinion to identify key factors of success in software process improvement. Li et al. [15] also utilized expert opinion to identify software engineering measures and, in addition, utilized a workshop procedure followed by individual interviews to rank the measures. These investigators motivated their choice for expert opinion by referring to the lack of maturity in the field and its lack of literature [15].

Though these studies provide valuable information about influencing factors, they do not distinguish between factors influencing defect injection and defect detection. Furthermore, from the studies mentioned above, the origin of the factors is not always evident. We therefore initiated this study to identify DI and DD factors, from initial conceptualization to a final set of evaluated factors.

The research questions for our study are formulated as:

  • W

    hat are the factors that influence the injection of defects?

  • W

    hat are the factors that influence the ability to detect defects?

The initial conceptualization – that is, the process of finding influencing factors – is crucial. Everything that follows with respect to our ultimate goal, the design of a framework for reducing the number of residual defects in software intensive products, depends on how well this conceptualization is performed. A general framework for structured conceptualization is given by Trochim and Linton [25] and further extended by Trochim [22] to a specific structured conceptualization method known as concept mapping. We applied this conceptualization method, of which the main steps are: (1) generation of concepts, (2) categorization of the concepts, (3) analysis and clustering of the concepts, and (4) interpretation of the clustering results.

Where Trochim [22] applies a group brainstorming technique to generate initial concepts – in the context of this paper influencing factors for defect injection and defect detection – we relied upon a literature search. De facto, literature search can be regarded as a brainstorming technique using a group of experts or direct stakeholders not directly instructed or controlled by the researchers. The rationale for the reliance upon literature search is the availability of a wide repository of accumulated knowledge and experience with respect to factors influencing defect injection and defect detection.

The next section, Section 2, describes the literature search for factors influencing DI and DD – covering the search approach and the recognition of the influencing factors. It includes outcomes of a pilot search intended to test and refine the search strategy and the recognition of influencing factors prior to the execution of the full literature search. Also described are pre-processing steps performed to clean up the raw factors resulting from the literature search into consistently formulated and full sets of DI and DD factors, respectively.

Section 3 reports on the reduction of the large number of DI- and DD-influencing factors to manageable sets for practical usage. The method comprises a categorization of factors by experts in the fields of DI and DD, using a card-sorting procedure, and a subsequent cluster analysis to aggregate the experts’ categorizations. It includes reliability estimates of the obtained results.

Section 4 describes the interpretation of the cluster analysis results using an expert judgment procedure. Section 5 presents the results of the literature search, categorization, cluster analysis, and interpretation of cluster groupings. Section 6 discusses reliability and validation considerations, while Section 7 summarizes the conclusions and presents directions for further research.

Section snippets

Search space

An extensive literature search was performed to identify influencing factors for defect injection and defect detection. The search embraced books, journals, conferences, theses, and technical reports in areas like virtual development, defect injection, defect detection, verification & validation, defect prediction, cost prediction, software engineering, product quality, etc.

The literature search was conducted through libraries and electronic libraries (like IEEE, ACM, ScienceDirect, Citeseer,

Clustering of factors

Eventually, the factors are intended to define measures to lower the number of residual defects in the development of software intensive products. Too many influencing factors were found for practical application. Therefore, a clustering procedure was performed to reduce the DI- and DD-influencing factor lists to manageable sets. This reduction proceeded along the lines of Trochim’s concept mapping method [22]: the pre-processed factors resulting from the literature search were categorized

Interpretation of cluster groupings

Since the optimal number of cluster groupings cannot be determined on the basis of mathematical criteria, cluster groupings are determined through interpretation of the cluster analysis data by teams of experts. The meaningfulness of cluster groupings is considered in the context of the intended usage of the groupings of DI- and DD-influencing factors, respectively. Meaningfulness is to be interpreted as the extent to which the groupings are recognized as areas of attention for which measures

Cluster analysis and interpretation

The results of the cluster analysis are shown in Appendix B (for DI factors) and in Appendix C (for DD factors).

The first column of the tables in Appendices Appendix B Defect injection factors and cluster groupings, Appendix C Defect detection factors and cluster groupings shows the factors as found from the literature search and after pre-processing. The factors are given in the order resulting from the cluster analysis. The participants in the interpretation sessions agreed that the number of

Reliability

Krippendorff discusses three types of reliability: stability, reproducibility, and accuracy [14]. In our study, only stability and reproducibility are relevant. Stability refers to the degree to which the same expert at different times categorizes the factors in the same manner. Reproducibility refers to the extent to which the same results can be reproduced in different times and locations and with different experts.

In addition, Krippendorff identified common reliability concerns of which two

Conclusions

We performed an extensive literature search to identify factors that are believed to influence defect injection and defect detection. The identified factors were pre-processed to achieve consistently formulated, full sets of DI and DD factors without duplications: 117 factors influencing defect injection and 113 influencing defect detection resulted. To reduce the factors to manageable sets for practical application, a cluster analysis was performed. A card-sorting procedure was applied in

Acknowledgements

We thank all participants in the tedious and time-consuming card-sorting task, and the subsequent interpretation sessions and/or evaluation rounds. Finally we are indebted to Larry Glick for proofreading and suggested corrections to this paper.

References (28)

  • T. Furuyama et al.

    Analysis of fault generation caused by stress during software development

    Journal of Systems and Software

    (1997)
  • W. Trochim et al.

    Conceptualization for evaluation and planning

    Evaluation and Program Planning

    (1986)
  • X. Zhang et al.

    An analysis of factors affecting software reliability

    The Journal of Systems and Software

    (2000)
  • H.H. Bock

    On some significance tests in cluster analysis

    Journal of Classification

    (1985)
  • B.W. Boehm

    Software Engineering Economics

    (1981)
  • B.W. Boehm et al.

    Cost Models for Future Software Life Cycle Processes: COCOMO 2.0

  • S. Chulani, B. Boehm, Modeling Software Defect Introduction and Removal: COQUALMO (COnstructive QUALity MOdel),...
  • P.B. Crosby

    Quality Is Free: The Art of Making Quality Certain

    (1979)
  • T. Dyba

    An instrument for measuring the key factors of success in software process improvement

    Empirical Software Engineering

    (2000)
  • B.S. Everitt

    Unresolved problems in cluster analysis

    Biometrics

    (1979)
  • D. Garvin

    What does “Product Quality” really mean? Sloan Management Review

    Fall

    (1984)
  • J.A. Hartigan

    Statistical theory in clustering

    Journal of Classification

    (1985)
  • T.C. Jones, Programming defect removal, in: Proceedings GUIDE 40, Miami, USA,...
  • S.H. Kan et al.

    Software quality: an overview from the perspective of total quality management

    IBM Systems Journal

    (1994)
  • View full text