nach oben

Erschienen in:

01.05.2021

A family of experiments on test-driven development

verfasst von: Adrian Santos, Sira Vegas, Oscar Dieste, Fernando Uyaguari, Ayşe Tosun, Davide Fucci, Burak Turhan, Giuseppe Scanniello, Simone Romano, Itir Karac, Marco Kuhrmann, Vladimir Mandić, Robert Ramač, Dietmar Pfahl, Christian Engblom, Jarno Kyykka, Kerli Rungi, Carolina Palomeque, Jaroslav Spisak, Markku Oivo, Natalia Juristo

Erschienen in: Empirical Software Engineering | Ausgabe 3/2021

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Context:

Test-driven development (TDD) is an agile software development approach that has been widely claimed to improve software quality. However, the extent to which TDD improves quality appears to be largely dependent upon the characteristics of the study in which it is evaluated (e.g., the research method, participant type, programming environment, etc.). The particularities of each study make the aggregation of results untenable.

Objectives:

The goal of this paper is to: increase the accuracy and generalizability of the results achieved in isolated experiments on TDD, provide joint conclusions on the performance of TDD across different industrial and academic settings, and assess the extent to which the characteristics of the experiments affect the quality-related performance of TDD.

Method:

We conduct a family of 12 experiments on TDD in academia and industry. We aggregate their results by means of meta-analysis. We perform exploratory analyses to identify variables impacting the quality-related performance of TDD.

Results:

TDD novices achieve a slightly higher code quality with iterative test-last development (i.e., ITL, the reverse approach of TDD) than with TDD. The task being developed largely determines quality. The programming environment, the order in which TDD and ITL are applied, or the learning effects from one development approach to another do not appear to affect quality. The quality-related performance of professionals using TDD drops more than for students. We hypothesize that this may be due to their being more resistant to change and potentially less motivated than students.

Conclusion:

Previous studies seem to provide conflicting results on TDD performance (i.e., positive vs. negative, respectively). We hypothesize that these conflicting results may be due to different study durations, experiment participants being unfamiliar with the TDD process, or case studies comparing the performance achieved by TDD vs. the control approach (e.g., the waterfall model), each applied to develop a different system. Further experiments with TDD experts are needed to validate these hypotheses.

Vorheriger Artikel An empirical study on changing leadership in agile teams

Nächster Artikel An automated framework for the extraction of semantic legal metadata from legal texts

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Nur mit Berechtigung zugänglich

For simplicity’s sake, we refer to quality and external quality interchangeably throughout the rest of the article. We acknowledge the limitations of this under the threats to validity.

Note that in our experiments the programming language is confounded with other variables: IDE, testing tools, and other programming environment related variables (the use of Java implies the use of Java-related technologies, while the use of C++/C# implies the use of C++/C#-related technologies). We have grouped all confounded variables under the programming environment name.

Due to space restrictions, we moved the references of the primary studies to the A.

It was not feasible to compute any response ratio synthesizing the quality achieved with TDD with respect to a control approach.

The outlier observed in [P35] may have been due to the small number of participants, and the larger variability of results expected in small sample sizes (Cumming 2013).

Throughout the rest of the paper we refer to the treatment—terminology commonly used in experimental design and data analysis (Brown and Prescott 2014; Higgins et al. 2008; Juristo and Moreno 2001; Wohlin et al. 2012)—and the development approach (i.e., either ITL or TDD) interchangeably.

https://github.com/GRISE-UPM/FiDiPro_ESEIL_TDD.

Note that the implementation style influences the size of the programs. We do not claim that these are gold implementations with the optimal design and coding practices. In fact, there are several implementations of these katas in public GitHub repositories.

Both measured with eclEmma: https://www.eclemma.org/

Measured with muJava: https://cs.gmu.edu/~offutt/mujava/

Note that the fact that participants do not have any ITL and TDD experience does not mean that they have no software testing experience. ITL and TDD have to do with knowledge of slicing and not with knowledge of testing. Therefore, participants with testing experience might conceivably have no experience with either ITL and/or TDD.

https://www.eui.eu/documents/servicesadmin/deanofstudies/researchethics/guide-data-protection-research.pdf

We analyzed the data with t-tests. Therefore, the mean difference (i.e., the slope of the line) provides useful information for evaluating experiment results.

In our experiments, the IDEs and testing tools used with C++ and C# are different. In this study, however, we make the simplification of considering them as being a part of the same group of technologies merely for the purposes of comparison.

A type of reactivity in which individuals modify an aspect of their behavior in response to their awareness of being observed.

Astels D (2003) Test driven development: A practical guide. Prentice Hall Professional Technical Reference

Baltes S, Diehl S (2018) Towards a theory of software development expertise. arXiv:1807.06087

Basili V R (1992) Software modeling and measurement: the goal/question/metric paradigm

Basili V R, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456–473CrossRef

Beck K (2003) Test-driven development: by example. Addison-Wesley Professional

Bergersen GR, Sjoberg DIK, Dyba T (2014) Construction and validation of an instrument for measuring programming skill. IEEE Trans Softw Eng 40 (12):1163–1184CrossRef

Bertolino A (2007) Software testing research: Achievements, challenges, dreams. In: 2007 Future of Software Engineering. IEEE Computer Society, pp 85–103

Bissi W, Neto A G S S, Emer M C F P (2016) The effects of test driven development on internal quality, external quality and productivity: A systematic review. Inf Softw Technol 74:45–54CrossRef

Borenstein M, Hedges L V, Higgins JPT, Rothstein H R (2011) Introduction to meta-analysis. Wiley

Brown H, Prescott R (2014) Applied mixed models in medicine. Wiley

Causevic A, Sundmark D, Punnekkat S (2011) Factors limiting industrial adoption of test driven development: A systematic review. In: 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation (ICST). IEEE, pp 337–346

Cooper H, Patall E A (2009) The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychol Methods 14(2):165CrossRef

Cumming G (2013) Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge

de Winter JCF (2013) Using the student’s t-test with extremely small sample sizes. Pract Assess Res Eval 18(10). [Online; accessed 28-August-2018]

Dieste O, Aranda A M, Uyaguari F, Turhan B, Tosun A, Fucci D, Oivo M, Juristo N (2017) Empirical evaluation of the effects of experience on code quality and programmer productivity: an exploratory study. Empir Softw Eng 22(5):2457–2542CrossRef

Falessi D, Juristo N, Wohlin C, Turhan B, Münch J, Jedlitschka A, Oivo M (2018) Empirical software engineering experts on the use of students and professionals in experiments. Empir Softw Eng 23(1):452–489CrossRef

Feigenspan J, Kästner C, Liebig J, Apel S, Hanenberg S (2012) Measuring programming experience. In: 2012 IEEE 20th International Conference on Program Comprehension (ICPC). IEEE, pp 73–82

Field A (2013) Discovering statistics using ibm spss statistics. Sage

Fisher DJ, Copas AJ, Tierney JF, Parmar MKB (2011) A critical review of methods for the assessment of patient-level interactions in individual participant data meta-analysis of randomized trials, and guidance for practitioners. J Clin Epidemiol 64(9):949–967CrossRef

Fucci D, Erdogmus H, Turhan B, Oivo M, Juristo N (2017) A dissection of the test-driven development process: does it really matter to test-first or to test-last?. IEEE Trans Softw Eng 43(7):597–614CrossRef

Gómez O S, Juristo N, Vegas S (2014) Understanding replication of experiments in software engineering: A classification. Inf Softw Technol 56 (8):1033–1048CrossRef

Gurevitch J, Koricheva J, Nakagawa S, Stewart G (2018) Meta-analysis and the science of research synthesis. Nature 555(7695):175CrossRef

Higgins JPT, Green S, et al. (2008) Cochrane handbook for systematic reviews of interventions, vol 5. Wiley Online Library

ISO/IEC 25010:2011 (2011) https://www.iso.org/obp/ui/#iso:std:iso-iec:25010:ed-1:v1:en

Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng 37(5):649–678CrossRef

Jung J, Hoefig K, Domis D, Jedlitschka A, Hiller M (2013) Experimental comparison of two safety analysis methods and its replication. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, pp 223–232

Juristo N, Moreno A M (2001) Basics of software engineering experimentation. Springer Science & Business Media

Juristo N, Vegas S (2009) Using differences among replications of software engineering experiments to gain knowledge. In: Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement. IEEE Computer Society, pp 356–366

Karac I, Turhan B (2018) What do we (really) know about test-driven development?. IEEE Softw 35(4):81–85CrossRef

Karac E I, Turhan B, Juristo N (2019) A controlled experiment with novice developers on the impact of task description granularity on software quality in test-driven development. IEEE Transactions on Software Engineering

Kitchenham B (2008) The role of replications in empirical software engineering, a word of warning. Empir Softw Eng 13(2):219–221CrossRef

Kollanus S (2010) Test-driven development-still a promising approach? In: Quality of Information and Communications Technology (QUATIC), 2010 Seventh International Conference on the. , pp 403–408

Kruger J, Dunning D (1999) Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Person Soc Psychol 77(6):1121CrossRef

Lau J, Ioannidis John PA, Schmid C H (1998) Summing up evidence: one answer is not always enough. Lancet 351(9096):123–127CrossRef

Lumley T, Diehr P, Emerson S, Chen L (2002) The importance of the normality assumption in large public health data sets. Ann Rev Public Health 23(1):151–169CrossRef

Mäkinen S, Münch J (2014) Effects of test-driven development: A comparative analysis of empirical studies. In: International Conference on Software Quality. Springer, pp 155–169

Martin CR (2001) Advanced principles, patterns and process of software development. Prentice Hall

Munir H, Moayyed M, Petersen K (2014) Considering rigor and relevance when evaluating test driven development: A systematic review. Inf Softw Technol 56(4):375–394CrossRef

Myers G J, Sandler C, Badgett T (2011) The art of software testing. Wiley

Norman G (2010) Likert scales, levels of measurement and the laws of statistics. Adv Health Sci Educ 15(5):625–632CrossRef

Offutt J (2018) Why don’t we publish more TDD research papers?. Softw Test Verif Reliab 28(4):e1670CrossRef

Quinn G P, Keough M J (2002) Experimental design and data analysis for biologists. Cambridge University Press

Rafique Y, Mišić V B (2013) The effects of test-driven development on external quality and productivity: A meta-analysis. IEEE Trans Softw Eng 39(6):835–856CrossRef

Riley R D, Lambert P C, Abo-Zaid G (2010) Meta-analysis of individual participant data: rationale, conduct, and reporting. Bmj 340:c221CrossRef

Rosenthal R (1991) Meta-analytic procedures for social research, vol 6. Sage

Santos A, Gomez O S, Juristo N (2018a) Analyzing families of experiments in SE: a systematic mapping study. IEEE Trans Softw Eng:1. https://doi.org/10.1109/TSE.2018.2864633

Santos A, Jarvinen J, Partanen J, Oivo M, Juristo N (2018b) Does the performance of tdd hold across software companies and premises? a group of industrial experiments on tdd. In: International Conference on Product-Focused Software Process Improvement. Springer, pp 227–242

Santos A, Vegas S, Oivo M, Juristo N (2018c) Guidelines for analyzing families of experiments in SE. Submitted to IEEE Transactions on Software Engineering

Schmider E, Ziegler M, Danay E, Beyer L, Bühner M (2010) Is it really robust? Methodology

Shull F, Melnik G, Turhan B, Layman L, Diep M, Erdogmus H (2010) What do we know about test-driven development?. IEEE Softw 27(6):16–19CrossRef

Sjøberg D IK, Bergersen G R (2018) The price of using students comments on empirical software engineering experts on the use of students and professionals in experiments. CoRR, arXiv:1810.10791

Thorlund K, Imberger G, Johnston B C, Walsh M, Awad T, Thabane L, Gluud C, Devereaux PJ, Wetterslev J (2012) Evolution of heterogeneity (i2) estimates and their 95% confidence intervals in large meta-analyses. PloS One 7(7):e39471CrossRef

Tosun A, Dieste O, Fucci D, Vegas S, Turhan B, Erdogmus H, Santos A, Oivo M, Toro K, Jarvinen J et al (2017) An industry experiment on the effects of test-driven development on external quality and productivity. Empir Softw Eng 22(6):2763–2805CrossRef

Tosun A, Dieste O, Vegas S, Pfahl D, Rungi K, Juristo N (In press) Investigating the impact of development task on external quality in test-driven development: An industry experiment. IEEE Transactions on Software Engineering

Vegas S, Dieste O, Juristo N (2015) Difficulties in running experiments in the software industry: experiences from the trenches. In: Proceedings of the Third International Workshop on Conducting Empirical Studies in Industry at ICSE. IEEE Press, pp 3–9

Vickers A J (2005) Parametric versus non-parametric statistics in the analysis of randomized trials with non-normally distributed data. BMC Med Res Methodol 5(1):35CrossRef

Williams L, Kessler R (2002) Pair programming illuminated. Addison-Wesley Longman Publishing Co., Inc.

Wohlin C, Runeson P, Höst M, Ohlsson M C, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Science & Business Media

Titel: A family of experiments on test-driven development
verfasst von: Adrian Santos
Sira Vegas
Oscar Dieste
Fernando Uyaguari
Ayşe Tosun
Davide Fucci
Burak Turhan
Giuseppe Scanniello
Simone Romano
Itir Karac
Marco Kuhrmann
Vladimir Mandić
Robert Ramač
Dietmar Pfahl
Christian Engblom
Jarno Kyykka
Kerli Rungi
Carolina Palomeque
Jaroslav Spisak
Markku Oivo
Natalia Juristo
Publikationsdatum: 01.05.2021
Verlag: Springer US
Erschienen in: Empirical Software Engineering / Ausgabe 3/2021
Print ISSN: 1382-3256
Elektronische ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-020-09895-8

Springer Professional

Abstract

Context:

Objectives:

Method:

Results:

Conclusion:

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Weitere Artikel der Ausgabe 3/2021

How to Better Distinguish Security Bug Reports (Using Dual Hyperparameter Optimization)

An automated framework for the extraction of semantic legal metadata from legal texts

Release synchronization in software ecosystems

Correction to: Wait for it: identifying “On-Hold” self-admitted technical debt

An empirical study on changing leadership in agile teams

Correction to: An empirical assessment of baseline feature location techniques

Premium Partner