A Bayesian network approach to assess and predict software quality using activity-based quality models

https://doi.org/10.1016/j.infsof.2010.03.016Get rights and content

Abstract

Context

Software quality is a complex concept. Therefore, assessing and predicting it is still challenging in practice as well as in research. Activity-based quality models break down this complex concept into concrete definitions, more precisely facts about the system, process, and environment as well as their impact on activities performed on and with the system. However, these models lack an operationalisation that would allow them to be used in assessment and prediction of quality. Bayesian networks have been shown to be a viable means for this task incorporating variables with uncertainty.

Objective

The qualitative knowledge contained in activity-based quality models are an abundant basis for building Bayesian networks for quality assessment. This paper describes a four-step approach for deriving systematically a Bayesian network from an assessment goal and a quality model.

Method

The four steps of the approach are explained in detail and with running examples. Furthermore, an initial evaluation is performed, in which data from NASA projects and an open source system is obtained. The approach is applied to this data and its applicability is analysed.

Results

The approach is applicable to the data from the NASA projects and the open source system. However, the predictive results vary depending on the availability and quality of the data, especially the underlying general distributions.

Conclusion

The approach is viable in a realistic context but needs further investigation in case studies in order to analyse its predictive validity.

Introduction

Despite the importance of software quality, its management is still an immature discipline in software engineering research and practice. Research work has gone in many directions and produced a variety of useful results. However, there is still no commonly agreed way for quality management. The practice varies strongly from a concentration on testing to a large-scale quality management process.

One main problem is that many of the tools and methods in quality assurance and management work isolated. For example, development teams usually perform peer reviews whose results are often not set into relation to results of integration or system tests performed by the quality assurance team. Hence, quality is tackled on many levels without a combined strategy [1]. What is missing is a clear integration of these single efforts. One prerequisite for such an integration is a quality management sub-process in the overall development process. The process defines roles, activities, and artefacts and how they work together. Hence, the maturity level of the organisation’s processes plays an important role. A second prerequisite is a clear quality model that specifies the quality of the software to be developed.

Current quality models such as the ISO 9126 [2] have widely acknowledged problems [3], [4], [5]. Especially as a basis for assessment and prediction, the defined “-ilities” are too abstract. A clear transition to measurements is therefore difficult in practice. Hence, quantitative quality assessment and prediction is usually done without direct use of such a quality model. This, in turn, leads to isolated solutions in quality management.

We use the previously proposed activity-based quality models (ABQM) [4] as a basis for quality assessment and prediction. They provide a clear structure of quality and detailed information about quality-influences. Activity-based quality models have proven useful in practice to structure quality and to generate corresponding guidelines and checklists. In this paper, we add a systematic transition from ABQMs to Bayesian networks in order to enhance their assessment and prediction capabilities. A four-step approach is defined that generates a Bayesian network using an activity-based quality model and an assessment or prediction goal. The approach is demonstrated in several examples.

We first motivate and introduce activity-based quality models in Section 2. In Section 3 the four-step approach for systematically constructing a Bayesian network from an activity-based quality model is proposed. The approach is then demonstrated in an initial evaluation in Section 4 using publicly available data from NASA projects and measured data from the open source system Tomcat. Related work is discussed in Section 5 and final conclusions are given in Section 6.

Section snippets

Activity-based quality models

Quality models describe in a structured way the meaning of a software’s quality. We introduce the use of general quality models and how the modelling of activities and facts helps to define quality more precisely.

Assessment and prediction approach

Although activity-based quality models have proven to be useful in practice, there is no systematic measurement approach for them. Hence, there are no quantitative assessments and predictions possible so far. We propose an approach that can be used for systematically deriving assessment and prediction models from an activity-based quality model.

Initial evaluation

The approach presented above can be used in various contexts to answer assessment and prediction questions. In this section, we provide an initial evaluation by applying the approach to several publicly available data sets as well as automatically collectable measures for an open source system.

Related work

The basic idea to use Bayesian networks for assessing and predicting software quality has been developed foremost by Fenton, Neil, and Littlewood. They introduced Bayesian networks as a tool in this area and applied it in various contexts related to software quality. In [21] they formulate a critique on current defect prediction models and suggest to use Bayesian networks. Other researchers also used Bayesian networks for software quality prediction similarly [22], [23].

The work closest to the

Conclusions

A high goal in software quality management is the reliable quantitative assessment and prediction of software quality. Many efforts in building assessment and prediction models have given insights in the usefulness but also the constraints of such models. However, these models have not been tightly integrated into other quality management activities. Activity-based quality models have proven in practice to be a solid foundation for defining quality on a detailed level. However, quantitative

Acknowledgments

This work has partially been supported by the German Federal Ministry of Education and Research (BMBF) in the project QuaMoCo (01 IS 08023B). I would like to thank the anonymous reviewers at PROMISE for feedback on an earlier version and Vic Basili, Sebastian Winter, and the anonymous IST reviewers for helpful suggestions for this version.

References (28)

  • S. Wagner, F. Deissenboeck, An integrated approach to quality modelling, in: Proc. 5th International Workshop on...
  • ISO 9126-1, Software Engineering – Product Quality – Part 1: Quality Model,...
  • B. Kitchenham et al.

    Software quality: the elusive target

    IEEE Software

    (1996)
  • F. Deissenboeck, S. Wagner, M. Pizka, S. Teuchert, J.-F. Girard, An activity-based quality model for maintainability,...
  • S. Wagner, K. Lochmann, S. Winter, A. Goeb, M. Klaes, Quality models in practice: a preliminary analysis, in: Proc. 3rd...
  • S. Winter et al.

    A comprehensive model of usability

  • S. Wagner, D. Méndez Fernández, S. Islam, K. Lochmann, A security requirements approach for web systems, in: Proc....
  • ISO/IEC 14764, IEEE Std. 14764-2006, Software Engineering – Software Life Cycle Processes – Maintenance,...
  • V.R. Basili et al.

    Goal question metric paradigm

  • M. Lindvall, P. Donzelli, S. Asgari, V. Basili. Towards reusable measurement patterns, in: Proc. 11th IEEE...
  • N. Fenton, M. Neil, Managing Risk in the Modern World, Applications of Bayesian Networks, Knowledge Transfer Report,...
  • N.E. Fenton et al.

    Using ranked nodes to model qualitative judgments in Bayesian networks

    IEEE Transactions on Knowledge and Data Engineering

    (2007)
  • IEEE Standard Glossary of Software Engineering Terminology, IEEE Std 610.12-1990,...
  • NASA IV& V Facility, Metrics Data Program....
  • Cited by (41)

    • A Method to Estimate Software Strategic Indicators in Software Development: An Industrial Application

      2021, Information and Software Technology
      Citation Excerpt :

      There is plenty of literature on indicators and metrics aimed to specify meaningful aspects of software development products and processes. One specific type of instrument that has been widely used in the literature to specify and create assessment models for software-related indicators refers to Quality Models (QMs) [39–42]. Although useful for specification, most of these works are academic and provide too abstract procedures to be operational [49].

    • Quantitative software reliability assessment methodology based on Bayesian belief networks and statistical testing for safety-critical software

      2020, Annals of Nuclear Energy
      Citation Excerpt :

      Therefore, through the analysis of relevant standards (such as IEEE 1012 (IEEE, 2017)) and guidelines (e.g. BTP 7–14 (U.S. NRC, 2007)), a series of indirect metrics are gained as indicator nodes (Fenton et al., 2007), which we can think of as providing observations for the parent node (Chu et al., 2018). In other words, indicator nodes represent metrics for the quality of the actual activity (Wagner, 2010). In this study, according to the degree of compliance with relevant standards and guidelines, the measurement results of indicator nodes can be divided into three levels, which are cited Chu et al. (2018):

    • Development of a cyber security risk model using Bayesian networks

      2015, Reliability Engineering and System Safety
    • Operationalised product quality models and assessment: The Quamoco approach

      2015, Information and Software Technology
      Citation Excerpt :

      Moreover, we can go the other way round. We can pick quality aspects, for example, ISO/IEC 25010 quality characteristics, which we consider important and costly for a specific software system, and trace what product factors affect them and what measures quantify them (cf. [73]). This allows us to put emphasis on the product factors with the largest impact on these quality aspects.

    View all citing articles on Scopus
    View full text