Top

Empirical Software Engineering

Published in:

01-11-2022

Testing research software: a survey

Authors: Nasir U. Eisty, Jeffrey C. Carver

Published in: Empirical Software Engineering | Issue 6/2022

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Background

Research software plays an important role in solving real-life problems, empowering scientific innovations, and handling emergency situations. Therefore, the correctness and trustworthiness of research software are of absolute importance. Software testing is an important activity for identifying problematic code and helping to produce high-quality software. However, testing of research software is difficult due to the complexity of the underlying science, relatively unknown results from scientific algorithms, and the culture of the research software community.

Aims

The goal of this paper is to better understand current testing practices, identify challenges, and provide recommendations on how to improve the testing process for research software development.

Method

We surveyed members of the research software developer community to collect information regarding their knowledge about and use of software testing in their projects.

Results

We analysed 120 responses and identified that even though research software developers report they have an average level of knowledge about software testing, they still find it difficult due to the numerous challenges involved. However, there are a number of ways, such as proper training, that can improve the testing process for research software.

Conclusions

Testing can be challenging for any type of software. This difficulty is especially present in the development of research software, where software engineering activities are typically given less attention. To produce trustworthy results from research software, there is a need for a culture change so that testing is valued and teams devote appropriate effort to writing and executing tests.

previous article Predicting sensitive information leakage in IoT applications using flows-aware machine learning approach

next article Self-Admitted Technical Debt and comments’ polarity: an empirical study

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

https://se4science.org/workshops/

The survey data is available in a public repository but set to private until publication of this paper (Carver and Eisty 2021)

https://www.istqb.org/

Ackroyd K S, Kinder S H, Mant G R, Miller M C, Ramsdale C A, Stephenson P C (2008) Scientific software development at a research facility. IEEE Softw 25(4):44–51. https://doi.org/10.1109/MS.2008.93CrossRef

Ammann P, Offutt J (2016) Introduction to software testing, 2nd edn. Cambridge University Press, CambridgeCrossRef

Bourque P, Fairley RE (eds) (2014) SWEBOK: guide to the software engineering body of knowledge, version 3.0 edn. IEEE Computer Society, Los Alamitos. http://www.swebok.org/

Carver JC, Eisty N (2021) Testing research software: survey data. https://doi.org/10.6084/m9.figshare.16663561. https://figshare.com/articles/dataset/_/16663561/0

Clune T, Rood R (2011) Software testing and verification in climate model development. IEEE Softw 28(6):49–55. https://doi.org/10.1109/MS.2011.117CrossRef

Drake J B, Jones P W, George R Carr J (2005) Overview of the software design of the community climate system model. Int J High Perform Comput Appl 19 (3):177–186. https://doi.org/10.1177/1094342005056094https://doi.org/10.1177/1094342005056094CrossRef

Easterbrook S M (2010) Climate change: a grand software challenge. In: Proceedings of the FSE/SDP workshop on future of software engineering research, FoSER ’10. https://doi.org/10.1145/1882362.1882383. http://doi.acm.org/10.1145/1882362.1882383. ACM, pp 99–104

Easterbrook S M, Johns T C (2009) Engineering the software for understanding climate change. Comput Sci Eng 11(6):65–74. https://doi.org/10.1109/MCSE.2009.193CrossRef

Eddins S L (2009) Automated software testing for matlab. Comput Sci Eng 11(6):48–55. https://doi.org/10.1109/MCSE.2009.186https://doi.org/10.1109/MCSE.2009.186CrossRef

Eisty N U, Thiruvathukal G K, Carver J C (2018) A survey of software metric use in research software development. In: 2018 IEEE 14th international conference on e-science (e-science). https://doi.org/10.1109/eScience.2018.00036, pp 212–222

Farrell P E, Piggott M D, Gorman G J, Ham D A, Wilson C R, Bond T M (2011) Automated continuous verification for numerical simulation. Geosci Model Dev 4(2):435–449. https://doi.org/10.5194/gmd-4-435-2011https://doi.org/10.5194/gmd-4-435-2011. https://www.geosci-model-dev.net/4/435/2011/CrossRef

Hannay J E, MacLeod C, Singer J, Langtangen H P, Pfahl D, Wilson G (2009) How do scientists develop and use scientific software?. In: 2009 ICSE workshop on software engineering for computational science and engineering. https://doi.org/10.1109/SECSE.2009.5069155, pp 1–8

Heaton D, Carver J C (2015) Claims about the use of software engineering practices in science: a systematic literature review. Inf Softw Tech 67:207–219. https://doi.org/10.1016/j.infsof.2015.07.011. http://www.sciencedirect.com/science/article/pii/S0950584915001342CrossRef

Hill C (2016) Socio-economic status and computer use: designing software that supports low-income users. In: 2016 IEEE Symposium on visual languages and human-centric computing. https://doi.org/10.1109/VLHCC.2016.7739651, pp 1–1

Hochstein L, Basili V R (2008) The asc-alliance projects: a case study of large-scale parallel scientific code development. Computer 41(3):50–58. https://doi.org/10.1109/MC.2008.101CrossRef

Hook D, Kelly D (2009) Testing for trustworthiness in scientific software. In: 2009 ICSE Workshop on software engineering for computational science and engineering. https://doi.org/10.1109/SECSE.2009.5069163https://doi.org/10.1109/SECSE.2009.5069163, pp 59–64

Kanewala U, Bieman J M (2014) Testing scientific software: a systematic literature review. Inf Softw Technol 56(10):1219–1232. https://doi.org/10.1016/j.infsof.2014.05.006CrossRef

Katz D S, McInnes L C, Bernholdt D E, Mayes A C, Hong N P C, Duckles J, Gesing S, Heroux M A, Hettrick S, Jimenez R C, Pierce M, Weaver B, Wilkins-Diehr N (2019) Community organizations: changing the culture in which research software is developed and sustained. Comput Sci Eng 21(2):8–24. https://doi.org/10.1109/MCSE.2018.2883051CrossRef

Kelly D, Sanders R, Saint R, Floor P, Sanders R, Kelly D (2008) The challenge of testing scientific software. In: Proceedings of the conference for the association for software testing, pp 30–36

Kelly D, Hook D, Sanders R (2009) Five recommended practices for computational scientists who write software. Comput Sci Eng 11(5):48–53. https://doi.org/10.1109/MCSE.2009.139CrossRef

Kelly D, Thorsteinson S, Hook D (2011) Scientific software testing: analysis with four dimensions. IEEE Softw 28(3):84–90. https://doi.org/10.1109/MS.2010.88CrossRef

Lemos G S, Martins E (2012) Specification-guided golden run for analysis of robustness testing results. In: 2012 IEEE Sixth international conference on software security and reliability. https://doi.org/10.1109/SERE.2012.28, pp 157–166

Miller G (2006) A scientist’s nightmare: software problem leads to five retractions. Science 314(5807):1856–1857. https://doi.org/10.1126/science.314.5807.1856. https://science.sciencemag.org/content/314/5807/1856, https://science.sciencemag.org/content/314/5807/1856.full.pdfCrossRef

Murphy C, Raunak M, King A, Chen S, Imbraino C, Kaiser G, Lee I, Sokolsky O, Clarke L, Osterweil L (2011) On effective testing of health care simulation software. Technical Reports (CIS). https://doi.org/10.1145/1987993.1988003

Nguyen-Hoan L, Flint S, Sankaranarayana R (2010) A survey of scientific software development. In: Proceedings of the 2010 ACM-IEEE international symposium on empirical software engineering and measurement, ESEM ’10. https://doi.org/10.1145/1852786.1852802. http://doi.acm.org/10.1145/1852786.1852802. ACM, pp 12:1–12:10

Post D E, Kendall R P (2004) Software project management and quality engineering practices for complex, coupled multiphysics, massively parallel computational simulations: Lessons learned from asci. Int J High Perform Comput Appl 18(4):399–416. https://doi.org/10.1177/1094342004048534CrossRef

Sanders R, Kelly D (2008) Dealing with risk in scientific software development. IEEE Softw 25(4):21–28. https://doi.org/10.1109/MS.2008.84CrossRef

Segal J (2005) When software engineers met research scientists: a case study. Empir Softw Eng 10. https://doi.org/10.1007/s10664-005-3865-y

Segal J (2009) Software development cultures and cooperation problems: a field study of the early stages of development of software for a scientific community. Comput Supported Coop Work 18 (5):581. https://doi.org/10.1007/s10606-009-9096-9CrossRef

Vilkomir S A, Swain W T, Poore J H, Clarno K T (2008) Modeling input space for testing scientific computational software: a case study. In: Bubak M, van Albada GD, Dongarra J, Sloot PMA (eds) Computational science—ICCS 2008. Springer, Berlin, pp 291–300

Title: Testing research software: a survey
Authors: Nasir U. Eisty
Jeffrey C. Carver
Publication date: 01-11-2022
Publisher: Springer US
Published in: Empirical Software Engineering / Issue 6/2022
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-022-10184-9

Springer Professional

Abstract

Background

Aims

Method

Results

Conclusions

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 6/2022

Identifying self-admitted technical debt in issue tracking systems using machine learning

SSPCatcher: Learning to catch security patches

The sense of logging in the Linux kernel

Development effort estimation in free/open source software from activity in version control systems

On effort-aware metrics for defect prediction

Revisiting the debate: Are code metrics useful for measuring maintenance effort?

Premium Partner