Top

Empirical Software Engineering

Published in:

01-03-2023 | Editorial

Registered reports in software engineering

Authors: Neil A. Ernst, Maria Teresa Baldassarre

Published in: Empirical Software Engineering | Issue 2/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Registered reports are scientific publications which begin the publication process by first having the detailed research protocol, including key research questions, reviewed and approved by peers. Subsequent analysis and results are published with minimal additional review, even if there was no clear support for the underlying hypothesis, as long as the approved protocol is followed. Registered reports can prevent several questionable research practices and give early feedback on research designs. In software engineering research, registered reports were first introduced in the International Conference on Mining Software Repositories (MSR) in 2020. They are now established in three conferences and two pre-eminent journals, including this one (EMSE). We explain the motivation for registered reports, outline the way they have been implemented in software engineering, and outline some ongoing challenges for addressing high quality software engineering research.

previous article JEMMA: An extensible Java dataset for ML4Code applications

next article Evaluating ensemble imputation in software effort estimation

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

https://conf.researchr.org/track/icse-2022/icse-2022-papers?#Call-for-Papers

see Ghafari et al. (2020) for a more thorough discussion of TDD experiments

See Läkens’s excellent course https://lakens.github.io/statistical_inferences/pvalue.html for more on N-P inference.

Note these issues are often unconscious, and not deliberate.

https://cos.io/rr/

e.g., https://github.com/researchart/rose-fse18

https://github.com/emsejournal/openscience

https://dl.acm.org/journal/tosem/registered-papers

osf.io/rr

https://github.com/acmsigsoft/EmpiricalStandards/blob/master/Supplements/RegisteredReports.md

https://computinged.wordpress.com/2019/05/31/come-hang-out-with-wil-and-me-to-talk-about-new-research-ideas-acm-icer-2019-work-in-progress-workshop/

rr.peercommunityin.org

https://aspredicted.org

Chambers CD (2013) Registered reports: A new publishing initiative at cortex. Cortex 49(3):609–610. https://doi.org/10.1016/j.cortex.2012.12.016CrossRef

Chambers CD, Tzavella L (2022) The past, present, and future of registered reports. Nat Hum Behav 6:29–42. https://doi.org/10.1038/s41562-021-01193-7CrossRef

de Oliveira Neto FG, Torkar R, Feldt R et al (2019) Evolution of statistical analysis in empirical software engineering research: Current state and steps forward. J Syst Softw 156:246–267CrossRef

Gelman A, Basbøll T (2014) When do stories work? Evidence and illustration in the social sciences. Sociol Methods Res 43(4):547–570. https://doi.org/10.1177/0049124114526377MathSciNetCrossRef

Gelman A, Loken E (2014) The statistical crisis in science. Am Sci 102:460CrossRef

Ghafari M, Gross T, Fucci D et al (2020) Why research on test-driven development is inconclusive?. In: Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). Association for Computing Machinery, New York, NY, USA, ESEM ’20. https://doi.org/10.1145/3382494.3410687

Hannay JE, Sjøberg DI, Dybå T (2007) A systematic review of theory use in software engineering experiments. IEEE Trans Softw Eng :33

Heiden S, Grunske L, Kehrer T et al (2019) An evaluation of pure spectrum-based fault localization techniques for large-scale software systems. Softw Pract Exp 49:1197–1224CrossRef

John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 23(5):524–532. https://doi.org/10.1177/0956797611430953CrossRef

Karhulahti VM (2022) Registered reports for qualitative research. Nat Hum Behav 6(1):4–5. https://doi.org/10.1038/s41562-021-01265-8CrossRef

Kitchenham BA, Madeyski L, Brereton P (2019) Meta-analysis for families of experiments in software engineering: a systematic review and reproducibility and validity assessment. Empir Softw Eng 25:353–401CrossRef

Lundberg I, Johnson R, Stewart BM (2021) What is your estimand? Defining the target quantity connects statistical evidence to theory. Am Sociol Rev 86(3):532–565. https://doi.org/10.1177/00031224211004187CrossRef

Mayo DG, Spanos A (2006) Severe testing as a basic concept in a Neyman-Pearson philosophy of induction. Br J Philos Sci 57(2):323–357. https://doi.org/10.1093/bjps/axl003MathSciNetCrossRefMATH

Menzies T (2021) Shockingly simple: “KEYS” for better AI for SE. IEEE Softw 38(2):114–118. https://doi.org/10.1109/ms.2020.3043014CrossRef

Menzies T, Krishna R, Pryor D (2017) The SEACRAFT repository of empirical software engineering data. https://zenodo.org/communities/seacraft

Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density. In: Proceedings of the international conference on software engineering. https://doi.org/10.1145/1062455.1062514

Paige RF, Cabot J, Ernst NA (2017) Foreword to the special section on negative results in software engineering. Empir Softw Eng 22(5):2453–2456. https://doi.org/10.1007/s10664-017-9498-0CrossRef

Pham MT, Oh TT (2021) Preregistration is neither sufficient nor necessary for good science. J Consum Psychol 31(1):163–176. https://doi.org/10.1002/jcpy.1209CrossRef

Ralph P (2021) ACM SIGSOFT empirical standards released. ACM SIGSOFT Softw Eng Notes 46(1):19–19. https://doi.org/10.1145/3437479.3437483CrossRef

Rohrer JM (2018) Thinking clearly about correlations and causation: Graphical causal models for observational data. Adv Methods Pract Psychol Sci 1 (1):27–42. https://doi.org/10.1177/2515245917745629MathSciNetCrossRef

Simmons JP, Nelson LD, Simonsohn U (2011) False-positive psychology. Psychol Sci 22(11):1359–1366. https://doi.org/10.1177/0956797611417632CrossRef

Simmons JP, Nelson LD, Simonsohn U (2021) Pre-registration is a game changer. But, like random assignment, it is neither necessary nor sufficient for credible science. J Consum Psychol 31(1):177–180. https://doi.org/10.1002/jcpy.1207CrossRef

Storey MAD, Ernst NA, Williams C et al (2020) The who, what, how of software engineering research: A socio-technical framework. Empir Softw Eng 25:4097–4129CrossRef

Waldron S, Allen C (2022) Not all pre-registrations are equal. Neuropsychopharmacology 47(13):2181–2183. https://doi.org/10.1038/s41386-022-01418-xCrossRef

Title: Registered reports in software engineering
Authors: Neil A. Ernst
Maria Teresa Baldassarre
Publication date: 01-03-2023
Publisher: Springer US
Published in: Empirical Software Engineering / Issue 2/2023
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-022-10277-5

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 2/2023

Correction to: Advantages and disadvantages of (dedicated) model transformation languages

JEMMA: An extensible Java dataset for ML4Code applications

Evaluating state-of-the-art # SAT solvers on industrial configuration spaces

On the assignment of commits to releases

An empirical study of text-based machine learning models for vulnerability detection

Vulnerability management in Linux distributions

Premium Partner