Top

Empirical Software Engineering

Published in:

01-01-2023

An empirical study on a single company’s cost estimations of 338 software projects

Authors: Christian Schürhoff, Stefan Hanenberg, Volker Gruhn

Published in: Empirical Software Engineering | Issue 1/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

A recurring statement in the literature is that cost estimations in software projects are problematic, but the evidence for such a statement is often unclear. In this paper, we analyze a project repository consisting of 338 software projects from the German software company adesso where for each project the original estimation is available in addition to the actual and estimated remaining costs at different points in time. The results revealed that there is an underestimation of costs in the repository (12.6%), but this underestimation is not significant: the hypothesis \(\mu (\frac {estimatedCosts}{averageCosts})=1\) (respectively the hypothesis on the log-transformed ratios \(\mu (log(\frac {estimatedCosts}{averageCosts}))=0\)) could not be rejected. However, we found a significant underestimation in the largest 20% of projects. And finally, we found a strong correlation between the estimated costs after 50% project duration and the final costs.

previous article A controlled experiment on the impact of microtasking on programming

next article Leveraging Stack Overflow to detect relevant tutorial fragments of APIs

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

https://www.adesso.de/

We considered only software development projects, because, for example, maintenance projects in the repository do not necessarily describe software maintenance projects, i.e. the other kinds of projects in the repository could be pure consulting projects without the goal to write any software.

Note: The data set provided with this paper does not permit to reconstruct the absolute human days per project or per year. The reasons for this is the need to protect the company’s data and to not give others the opportunity to check what projects played what role in the company’s balance sheet.

We use here the previously used terminology of BAC to describe the initially estimated costs.

Note: We give these numbers in order to get an impression of the data underlying this study. The raw data provided with this paper does not contain absolute numbers, i.e., it is not possible to compute from the raw data provided with this paper the number of human days per project.

Due to the uncleanliness of the repository not all ETCs are set to 0, i.e., the computation of the global CPI requires to consider the ETCs of each project as well.

This number cannot be computed from the repository delivered with this paper, because the computation requires the concrete ACs, BACs, and ETCs.

In order to ease the reading of the paper, we directly state at the hypotheses, whether or not they were rejected.

Throughout this paper, we used SPSS v27 for the statistical analysis.

There is one project in the dataset with CPI = 0 (which cannot be log transformed). For this rather small project (total number of human days less than 10), the final BAC and ETC where equal, i.e. it is one of those project, where we assume that the ETCs (and ACs) were finally not added to the repository. As a consequence, we decided to remove the project for the comparison of log-transformed CPIs.

In order to have a comparable basis, we applied the t-test to the same 337 projects.

The original reference refers to the ratio \(\frac {BAC}{AC}\) which is at the end of the project the CPI.

Since not all projects had ETC = 0 in the end, we added the remaining planned estimates to complete to the actual costs before the analysis.

We intentionally do not describe the concrete values on the x-axis, because such concrete value would contradict the confidality of adesso’s data.

Actually, the sets of AC and the BAC determined upper 20% projects just differ by 6 projects.

Just to make sure that the used 20% are not the main drivers that no difference to CPI = 0 on all projects was identified in Section 4.3, we tested the difference to 0 on the smallest 20% of projects as well as on the resulting 80% of projects. In all cases, no significant difference from μ(log(CPI))_Largest20% = 0 was detected (p >.5).

We define the 0% quantile as the very first CPI for a project.

Addison T, Vallabh S (2002) Controlling software project risks: an empirical study of methods used by experienced project managers. In: Proceedings of the 2002 annual research conference of the South African Institute of Computer Scientists and Information Technologists on enablement through technology, SAICSIT ’02. South African Institute for Computer Scientists and Information Technologists, Republic of South Africa, pp 128–140

Albrecht A J (1979) Measuring application development productivity. In: Proceedings of joint share, guide, and IBM application development symposium

Aranda J, Easterbrook S (2005) Anchoring and adjustment in software estimation. In: Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on foundations of software engineering, ESEC/FSE-13. ACM, New York, pp 346–355

Bergeron F, St-Arnaud J-Y (1992) Estimation of information systems development efforts: a pilot study. Inf Manag 22(4):239–254CrossRef

Boehm B W (2017) Software cost estimation meets software diversity. In: 2017 IEEE/ACM 39th international conference on software engineering companion (ICSE-c), IEEE, pp 495–496

Boehm B W (1981) Software engineering economics. Prentice-hall advances in computing science and technology series. Prentice-Hall

Boehm B, Abts C, Chulani S (2000a) Software development cost estimation approaches—a survey. Ann Softw Eng 10(1–4):177–205CrossRefMATH

Boehm B W, Clark B K, Horowitz E, Brown A W, Reifer D J, Chulani S, Madachy R, Steece B (2000b) Software cost estimation with Cocomo II with Cdrom, 1st edn. Prentice Hall PTR, Upper Saddle River

Britto R, Freitas V, Mendes E, Usman M (2014) Effort estimation in global software development: a systematic literature review

Brooks FP Jr (1995) The mythical man-month (anniversary Ed.) Addison-Wesley Longman Publishing Co., Inc., Boston

Emam K E, Koru A G (2008) A replicated survey of it software project failures. IEEE Softw 25(5):84–90CrossRef

Garousi V, Coskuncay A, Betin-Can A, Demirors O (2014) A survey of software engineering practices in Turkey. J Syst Softw 108:148–177CrossRef

Glass R L (2006) The standish report: does it really describe a software crisis? Commun ACM 49(8):15–16CrossRef

Grimstad S, Jørgensen M (2007) Inconsistency of expert judgment-based estimates of software development effort. J Syst Softw 80(11):1770–1777CrossRef

Grimstad S, Jørgensen M (2008) A preliminary study of sequence effects in judgment-based software development work-effort estimation. In: Proceedings of the 12th international conference on evaluation and assessment in software engineering, EASE’08. BCS Learning & Development Ltd, Swindon, pp 129–135

Halkjelsvik T, Jørgensen M (2012) From origami to software development: a review of studies on judgment-based predictions of performance time. Psychol Bull 138(2):238–271CrossRef

Hill J, Thomas L C, Allen D E (2000) Experts’ estimates of task durations in software development projects. Int J Proj Manag 18(1):13–21CrossRef

Jenkins A M, Naumann J D, Wetherbe J C (1984) Empirical investigation of systems development practices and results. Inf Manag 7(2):73–82CrossRef

Jørgensen M (2004) Regression models of software development effort estimation accuracy and bias. Empir Softw Eng 9(4):297–314CrossRef

Jørgensen M, Shepperd M (2007) A systematic review of software development cost estimation studies. IEEE Trans Softw Eng 33(1):33–53CrossRef

Jørgensen M, Grimstad S (2010) Software development effort estimation—demystifying and improving expert estimation. Springer, Berlin, pp 381–403

Jørgensen M, Halkjelsvik T, Kitchenham B (2012) How does project size affect cost estimation error? Statistical artifacts and methodological challenges. Int J Proj Manag 30:839–849CrossRef

Kitchenham B, Pfleeger S L, McColl B, Eagan S (2002) An empirical study of maintenance and development estimation accuracy. J Syst Softw 64(1):57–77CrossRef

Kwak Y, Anbari F (2012) History, practices, and future of earned value management in government: perspectives from nasa. Project Manag J 43:77–90CrossRef

Langdon W B, Dolado J, Sarro F, Harman M (2016) Exact mean absolute error of baseline predictor, marp0. Inf Softw Technol 73:16–18CrossRef

Larman C, Basili V R (2003) Iterative and incremental developments. A brief history. Computer 36(6):47–56CrossRef

Larman C (2004) Agile and iterative development: a manager’s guide. Addison-Wesley

Lederer A L, Mirani R, Neo B S, Pollard C, Prasad J, Ramamurthy K (1990) Information system cost estimating: a management perspective. MIS Q 14(2):159–176CrossRef

Lo B, Gao X (1997) Assessing software cost estimation models: criteria for accuracy, consistency and regression. Australas J Inf Syst 5(1):11

Møloekken K, Jørgensen M (2003) A review of software surveys on software effort estimation. In: 2003 International symposium on empirical software engineering. ISESE 2003. Proceedings, pp 223–230

Møloekken-Østvold K, Jørgensen M, Tanilkan SS, Gallis H, Lien AC, Hove SW (2004) A survey on software estimation in the Norwegian industry. In: 10th International symposium on software metrics. Proceedings, pp 208–219

Moløkken K, Jørgensen M (2003) Software effort estimation: unstructured group discussion as a method to reduce individual biasis. In: Proceedings of the 15th annual workshop of the psychology of programming interest group, PPIG 2003, Keele, UK, April 8–10, 2003, p 4

Phan D, Vogel D, Nunamaker J (1988) The search for perfect project management. Computerworld 22(39):95–100

Putnam L (1978) A general empirical solution to the macro software sizing and estimating problem. IEEE Trans Softw Eng 4(04):345–361CrossRefMATH

Sarro F, Petrozziello A, Harman M (2016) Multi-objective software effort estimation. In: Dillon LK, Visser W, Williams LA (eds) Proceedings of the 38th international conference on software engineering, ICSE 2016, Austin, TX, USA, May 14–22, 2016. ACM, pp 619–630

Sauer C, Cuthbertson C (2003) The state of it project management in the UK 2002–2003. Computer Weekly

Shepperd M J (2014) Cost prediction and software project management. In: Ruhe G, Wohlin C (eds) Software project management in a changing world. Springer, pp 51–71

Shepperd M J, Macdonell SG (2012) Evaluating prediction systems in software project estimation. Inf Softw Technol 54(8):820–827CrossRef

Shepperd M, Schofield C, Kitchenham B (1996) Effort estimation using analogy. In: Proceedings of the 18th international conference on software engineering, ICSE ’96. IEEE Computer Society, Washington, DC, pp 170–178

Vicinanza S S, Mukhopadhyay T, Prietula M J (1991) Software-effort estimation: an exploratory study of expert performance. Info Sys Res 2 (4):243–262CrossRef

Whitfield D (2007) Cost overruns, delays and terminations in 105 outsourced public sector ict contracts. In: ESSU Research report no. 3. The European services strategy unit

Wohlin C, Runeson P, Höst M, Ohlsson M C, Regnell B, Wesslén A (2000) Experimentation in software engineering: an introduction. Kluwer Academic Publishers, NorwellCrossRefMATH

Zhu X, Zhou B (2010) An earned-value approach to assess and monitor software project uncertainty: a case study in software test execution. Inf Technol J 9:0CrossRef

Yang D, Wang Q, Li M, Yang Y, Ye K, Du J (2008) A survey on software cost estimation in the chinese software industry. In: Proceedings of the second ACM-IEEE international symposium on empirical software engineering and measurement, ESEM ’08. ACM, New York, pp 253–262

Yourdon E (1997) Death march: the complete software developer’s guide to surviving mission impossible projects. Prentice Hall PTR, Upper Saddle River

Title: An empirical study on a single company’s cost estimations of 338 software projects
Authors: Christian Schürhoff
Stefan Hanenberg
Volker Gruhn
Publication date: 01-01-2023
Publisher: Springer US
Published in: Empirical Software Engineering / Issue 1/2023
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-022-10245-z

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 1/2023

Towards understanding bugs in Python interpreters

Evaluating classifiers in SE research: the ECSER pipeline and two replication studies

Leveraging Stack Overflow to detect relevant tutorial fragments of APIs

Automatic prediction of rejected edits in Stack Overflow

Analyzing Techniques for Duplicate Question Detection on Q&A Websites for Game Developers

A controlled experiment on the impact of microtasking on programming

Premium Partner