The cost of errors in software development: evidence from industry

https://doi.org/10.1016/S0164-1212(01)00130-3Get rights and content

Abstract

The search for and correction of errors in software are often time consuming and expensive components of the total cost of software development. The current research investigates to what extent these costs of software error detection and correction contribute to the total cost of software. We initiated the research reported here with the collection of a sample of transactions recording progress on one phase of development of a set of software programs. Each of these projects represented the completion of an identical phase of development (i.e., country localisation) for a different country. This enabled each project to be compared with the other, and provided an unusually high degree of control over the data collection and analysis in real-world empirical study. The research findings relied on programmers' self-assessment of the severity of errors discovered. It found that serious errors have less influence on total cost than errors that were classified as less serious but which occurred with more frequency once these less serious errors are actually resolved and corrected. The research suggests one explanation – that programmers have greater discretion in how and when to resolve these less severe errors. The data supports the hypothesis that errors generate significant software development costs if their resolution requires system redesign. Within the context of the research, it was concluded that uncorrected errors become exponentially more costly with each phase in which they are unresolved, which is consistent with earlier findings in the literature. The research also found that the number of days that a project is open is a log-linear predictor of the number of software errors that will be discovered, implying a bias in error discovery over time. This implies that testing results need to be interpreted in light of the length of testing, and that in practice, tests should take place both before and after systems release.

Section snippets

Software life cycle costs

This research develops a model for costs generated in software development, focusing on costs associated with software error correction and detection. It draws on empirical data extracted for one phase of the software development life cycle by a vendor of packaged microcomputer software.

In the past, empirical investigations of software costs have been rare, most likely due to the difficulty in obtaining sufficient data for analysis. Financial accounting systems do not require separate account

Data collection

Data was collected from a packaged software vendor with global operations and a number of competitive products. The vendor developed, researched, specified, and tested each software release at corporate headquarters before allowing global release. Each completed, tested, and stable product was then released in the vendors home market simultaneously with its being passed on for localisation (i.e., translation to the local language, culture and practice) to 31 strategically important global

Error cost

The vendor's internal studies concluded that the largest single contributor to localisation cost was the detection and correction of errors. Consequently, this research first investigated evidence on error costs.

Prior literature has suggested a growth model for software errors and reliability. Evolutionary growth models assume that the number of errors discovered depends on time and that the underlying error distribution is exponential. The most important of these was proposed by Jelinski and

Specification searches

This part of the research reports the findings from a general multiple variable analysis of software costs and cost drivers for development tasks during the `localisation' phase. Because part of the data used is accounting cost data, there is the possibility of results being confounded by double-entry bookkeeping. Double-entry bookkeeping, by definition, records information about the same transactions in multiple places around the corporate accounts simultaneously. This can weaken our

Conclusions and implications for the software industry

The dataset acquired for this research provides strong support for an evolutionary model of error occurrence with an exponential distribution of error count overtime. It also provided some additional specifications for enhanced models for software cost estimation. Specifically:

(1) The regression shows that the serious errors have less influence on cost than other errors, assuming that they are ultimately resolved. This likely reflects greater discretion available in how and when to resolve the

J. Christopher Westland is Professor and Head of the Department of Information Systems and Management at the Hong Kong University of Science and Technology. He received a BA in mathematics, and an MBA in accounting from Indiana University. He received his Ph.D. in information systems from the University of Michigan. He has professional experience in the US as a certified public accountant and as a consultant in information systems in the US, Europe, Latin America and Asia. He is a US CPA, an

References (21)

  • G Canfora et al.

    An integrated environment for reuse reengineering C code

    The Journal of Systems and Software

    (1998)
  • H Benyahia

    Costs and productivity estimation in computer engineering economics

    The Engineering Economist

    (1996)
  • R Bloor

    Software quality standards: do software quality initiatives such as ISO 9000 reduce the introduction of dangerous programmatic errors?

    DBMS

    (1993)
  • B.W Boehm

    Software and its impact: a quantitative assessment

    Datamation

    (1973)
  • B.W Boehm

    8th World computer congress developing small-scale application software products: some experimental results

  • B.W Boehm
  • G.E.P Box et al.

    An Analysis of Transformations

    Journal of the Royal Statistical Society, Series B

    (1964)
  • E Ferneley

    Coupling and control flow measures in practice

    The Journal of Systems and Software

    (2000)
  • C Inwood

    Formal methods can cut your error rate

    Computing Canada

    (1993)
  • C Inwood

    To err is human, to forgive uncommon in IS

    Computing Canada

    (1994)
There are more references available in the full text version of this article.

Cited by (70)

  • IFC+: Towards the integration of IoT into early stages of building design

    2022, Automation in Construction
    Citation Excerpt :

    The possibility of working with models is a very desirable characteristic as far as the support of IoT features in BIM processes is concerned, since each IoT infrastructure is managed by an underlying software. The use of models to represent IoT devices can helps to detect errors and inconsistencies in early stages of the software development, which in turn, has been repeatedly proved to reduce software lifecycle costs exponentially [23–25]. Similarly, there are starting to appear studies reflecting the positive economic impact of using BIM models in construction processes, although the lack of records and available data still difficulties to carry out these kinds of research [26,27].

  • The information content of Sarbanes-Oxley in predicting security breaches

    2020, Computers and Security
    Citation Excerpt :

    This is somewhat more frequent than the 5.6 years expected between breaches by the hazard structural model, but with more extensive data collection, a more robust estimate could be obtained, and we consider this estimate to be realistic for the occupancy structural model. Such failures where they do occur in the software field are apocryphally called 5000 errors and have been studied in Boehm et al. (1995), Westland (2000), Westland 2002, Westland (2004) . The probability of a breach at any given time is small, with a large enough population of firms one can expect many failures to show up every year across the full population of firms and industry wide risk of breach is thus substantial.

  • Supporting agent oriented requirement analysis with ontologies

    2016, International Journal of Human Computer Studies
    Citation Excerpt :

    Throughout these activities, cognitive limitations of modellers, communication gaps with the users, together with the interdependencies of models, can lead to errors. These errors can snowball in the course of the development process and become more costly to fix [Westland, 2002]. As such, capturing errors in the requirement models during the verification and validation activities before they propagate into the design and later phases is often pursued.

  • User Involvement in Software Development Processes

    2016, Procedia Computer Science

  • An Assessment of Rules of Thumb for Software Phase Management, and the Relationship Between Phase Effort and Schedule Success

    2024, IEEE Transactions on Software Engineering

  • Enhancing conceptual models with computational capabilities: A methodical approach to executable integrative modeling

    2024, Systems Engineering
View all citing articles on Scopus

J. Christopher Westland is Professor and Head of the Department of Information Systems and Management at the Hong Kong University of Science and Technology. He received a BA in mathematics, and an MBA in accounting from Indiana University. He received his Ph.D. in information systems from the University of Michigan. He has professional experience in the US as a certified public accountant and as a consultant in information systems in the US, Europe, Latin America and Asia. He is a US CPA, an active member of Mensa, and sits on the editorial boards of several of the leading academic journals in information technology and has written two books, The New Science of Wealth (forthcoming October 2001, Wiley) and Global Electronic Commerce (MIT Press, 2000). He has been a frequent commentator on Hong Kong's technology policy. A full biography and selected papers may be found at http://www.ismt.ust.hk/faculty/westland/.

View full text
Copyright © 2001 Elsevier Science Inc. All rights reserved.