Skip to main content
main-content
Top

Hint

Swipe to navigate through the articles of this issue

Published in: Journal of Business and Psychology 4/2021

11-06-2020 | Original Paper

Study Preregistration: An Evaluation of a Method for Transparent Reporting

Authors: Allison A. Toth, George C. Banks, David Mellor, Ernest H. O’Boyle, Ashleigh Dickson, Daniel J. Davis, Alex DeHaven, Jaime Bochantin, Jared Borns

Published in: Journal of Business and Psychology | Issue 4/2021

Login to get access
share
SHARE

Abstract

Study preregistration promotes transparency in scientific research by making a clear distinction between a priori and post hoc procedures or analyses. Management and applied psychology have not embraced preregistration in the way other closely related social science fields have. There may be concerns that preregistration does not add value and prevents exploratory data analyses. Using a mixed-method approach, in Study 1, we compared published preregistered samples against published non-preregistered samples. We found that preregistration effectively facilitated more transparent reporting based on criteria (i.e., confirmed hypotheses and a priori analysis plans). Moreover, consistent with concerns that the published literature contains elevated type I error rates, preregistered samples had fewer statistically significant results (48%) than non-preregistered samples (66%). To learn about the perceived advantages, disadvantages, and misconceptions of study preregistration, in Study 2, we surveyed authors of preregistered studies and authors who had never preregistered a study. Participants in both samples had positive inclinations towards preregistration yet expressed concerns about the process. We conclude with a review of best practices for management and applied psychology stakeholders.

Footnotes
1

We also originally excluded samples at the team/organizational level, as the expertise of our author team is more at the individual level. However, as one of our reviewers rightly noted, samples at the team/organization level are central to the management and applied psychology fields. As a result, we went back and coded the two samples that we initially excluded for being at the team/organizational level and included them in our analyses/results.

 
2

We would like to thank an anonymous reviewer for raising this issue.

 
Literature
  1. Agnoli, F., Wicherts, J. M., Veldkamp, C. L., Albiero, P., & Cubelli, R. (2017). Questionable research practices among Italian research psychologists. PLoS One, 12(3), e0172792.View Article
  2. Aguinis, H., Ramani, R. S., & Alabduljader, N. (2018). What you see is what you get? Enhancing methodological transparency in management research. Academy of Management Annals, 12(1), 83–110.View Article
  3. Anderson, M., & Magruder, J. (2017). Split-sample strategies for avoiding false discoveries (no. w23544). https://​doi.​org/​10.​3386/​w23544.
  4. Anderson, M. S., Martinson, B. C., & De Vries, R. (2007). Normative dissonance in science: Results from a national survey of US scientists. Journal of Empirical Research on Human Research Ethics, 2(4), 3–14.View Article
  5. Antonakis, J. (2017). On doing better science: From thrill of discovery to policy implications. The Leadership Quarterly, 28(1), 5–21.View Article
  6. Banks, G. C., O’Boyle Jr, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett., A. A, Adkins, C. L. (2016a). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42(1): 5–20.
  7. Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., & Rupp, D. E. (2016b). Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology, 31, 323–338.View Article
  8. Banks, G. C., Field, J. G., Oswald, F. L., O’Boyle, E. H., Landis, R. S., Rupp, D. E., & Rogelberg, S. G. (2018). Answers to 18 questions about Open Science practices. Journal of Business and Psychology, 1–14.
  9. Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. (2016). HARKing’s threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology, 69, 709–750.View Article
  10. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.View Article
  11. Butler, N., Delaney, H., & Spoelstra, S. (2017). The gray zone: Questionable research practices in the business school. Academy of Management Learning & Education, 16(1), 94–109.View Article
  12. Choi, M. (2011). Employees' attitudes toward organizational change: A literature review. Human Resource Management, 50(4), 479–500.View Article
  13. Cunliffe, A. L., & Alcadipani, R. (2016). The politics of access in fieldwork: Immersion, backstage dramas, and deception. Organizational Research Methods, 19(4), 535–561.View Article
  14. Dechartres, A., Ravaud, P., Atal, I., Riveros, C., & Boutron, I. (2016). Association between trial registration and treatment effect estimates: A meta-epidemiological study. BMC Medicine, 14(1), 100.View Article
  15. Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015). The reusable holdout: Preserving validity in adaptive data analysis. Science, 349(6248), 636–638.View Article
  16. Ebersole, C. R., Axt, J. R., & Nosek, B. A. (2016). Scientists’ reputations are based on getting it right, not being right. PLoS Biology, 14(5), e1002460.View Article
  17. Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010). Testing for the presence of positive-outcome bias in peer review: A randomized controlled trial. Archives of Internal Medicine, 170(21), 1934–1939.View Article
  18. Fafchamps, M., & Labonne, J. (2016). Using split samples to improve inference about causal effects (no. w21842). https://​doi.​org/​10.​3386/​w21842.
  19. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904.View Article
  20. Fleiss, J. L. (1981). Statistical methods for rates and proportions. 2nd ed. Wiley.
  21. Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018). Questionable research practices in ecology and evolution. PLoS One, 13(7), e0200303.View Article
  22. Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R. (2019a). COMPare: A prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials, 20(1), 118.View Article
  23. Goldacre, B., Drysdale, H., Marston, C., Mahtani, K. R., Dale, A., Milosevic, I., Slade, E., Hartley, P., & Heneghan, C. (2019b). COMPare: Qualitative analysis of researchers’ responses to critical correspondence on a cohort of 58 misreported trials. Trials, 20(1), 124.View Article
  24. Guest, G., MacQueen, K. M., & Namey, E. E. (2012). Validity and reliability (credibility and dependability) in qualitative research and data analysis. Applied Thematic Analysis: 79–106. Sage Publications.
  25. Haven, T. L., & Van Grootel, D. L. (2019). Preregistering qualitative research. Accountability in Research, 26(3), 229–244.View Article
  26. Hollenbeck, J. R., & Wright, P. M. (2016). Harking, sharking, and tharking: Making the case for post hoc analysis of scientific data. Journal of Management, 43, 5–18.View Article
  27. Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288.View Article
  28. Hunt, S. T. (2018). If robust science is relevant science, then make IO psychology research more relevant: Thoughts from a practitioner point of view. Industrial and Organizational Psychology, 11(1), 65–70.View Article
  29. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532.View Article
  30. Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of null effects of large NHLBI clinical trials has increased over time. PLoS One, 10(8), e0132382.View Article
  31. Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in I-O psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6(3), 252–268.View Article
  32. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.View Article
  33. Koenig, W. D. (2017). Striving for science that is transparent, credible—and enjoyable: A comment on Ihle et al. Behavioral Ecology, 28(2), 358–358.View Article
  34. Köhler, T., Gonzàlez-Morales, M. G., Banks, G.C., O’Boyle, E., Allen, J., Sinha, R., Woo, S. E., Gulick, L. (in press). Supporting robust, rigorous, and reliable reviewing as the cornerstone of our profession: Introducing a competency model for peer review. Industrial and Organizational Psychology: Perspectives on Science and Practice.
  35. Leavitt, K. (2013). Publication bias might make us untrustworthy, but the solutions may be worse. Industrial and Organizational Psychology, 6(3), 290–295.View Article
  36. Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. (2019, October 31). Questionable and open research practices in education research. https://​doi.​org/​10.​35542/​osf.​io/​f7srb.
  37. McAbee, S. T., Grubbs, J. B., & Zickar, M. J. (2018). Open science is robust science. Industrial and Organizational Psychology, 11(1), 54–61.View Article
  38. Mellor, D. T., & Nosek, B. A. (2018). Easy preregistration will benefit any research. Nature Human Behaviour, 1.
  39. Morey. (2019). You must tug that thread: Why treating preregistration as a gold standard might incentivize poor behavior. Psychonomic Society. https://​featuredcontent.​psychonomic.​org/​you-must-tug-that-thread-why-treating-preregistration-as-a-gold-standard-might-incentivize-poor-behavior/​.​
  40. Murphy, K. R., & Aguinis, H. (2017). HARKing: How badly can cherry-picking and question trolling produce bias in published results? Journal of Business and Psychology: 1–17.
  41. Murray, M. M., & Antonakis, J. (2019). An Introductory Guide to Organizational Neuroscience: SAGE Publications Sage CA, An Introductory Guide to Organizational Neuroscience.
  42. Navarro. (2019). Prediction, pre-specification and transparency. Psychonomic Society. https://​featuredcontent.​psychonomic.​org/​prediction-pre-specification-and-transparency/​
  43. Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23, 217–243.View Article
  44. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.View Article
  45. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Chistensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., et al. (2015). Promoting an open research culture: Author guidelines for journals to promote transparency, openness, and reproducibility. Science, 348, 1422–1425.View Article
  46. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences: 201708274.
  47. Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van't Veer, A. E, & Vazire, S. (2019). Preregistration is hard, and worthwhile. Trends in Cognitive Sciences, 23(10), 815–818.
  48. O’Boyle, E., Banks, G. C., Carter, K., Walter, S., & Yuan, Z. (2019). A 20-year review of outcome reporting bias in moderated multiple regression. Journal of Business and Psychology, 34(1), 1–19.View Article
  49. Oberauer. (2019). Preregistration of a forking path—what does it add to the garden of evidence? Psychonomic Society. https://​featuredcontent.​psychonomic.​org/​preregistration-of-a-forking-path-what-does-it-add-to-the-garden-of-evidence/​.​
  50. O'Boyle, E. H., Banks, G. C., & Gonzalez-Mule, E. (2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43, 400–425.
  51. Papageorgiou, S. N., Xavier, G. M., Cobourne, M. T., & Eliades, T. (2018). Registered trials report less beneficial treatment effects than unregistered ones: A meta-epidemiological study in orthodontics. Journal of Clinical Epidemiology, 100, 44–52.View Article
  52. Rogelberg, S. G., & Stanton, J. M. (2007). Introduction: Understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10(2), 195–209.View Article
  53. Schwarzkopf, S. (2015, March 21). Some questions about registered reports. https://​neuroneurotic.​net/​2015/​03/​21/​some-questions-about-registered-reports/​.​
  54. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.View Article
  55. Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015). Specification curve: Descriptive and inferential statistics on all reasonable specifications. Available at SSRN 2694998.
  56. Tonidandel, S., King, E. B., & Cortina, J. M. (2018). Big data methods: Leveraging modern data analytic techniques to build organizational science. Organizational Research Methods, 21(3), 525–547.View Article
  57. van Rooij. (2019). Psychological science needs theory development before preregistration. Psychonomic Society. https://​featuredcontent.​psychonomic.​org/​psychological-science-needs-theory-development-before-preregistration/​.​
  58. Veldkamp, C. L. S., Bakker, M., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Soderberg, C. K., Mellor, D., Nosek, B. A., & Wicherts, J. M. (in progress)(n.d.). Restriction of opportunistic use of researcher degrees of freedom in pre-registrations on the Open Science Framework.
  59. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7(6), 632–638.View Article
Metadata
Title
Study Preregistration: An Evaluation of a Method for Transparent Reporting
Authors
Allison A. Toth
George C. Banks
David Mellor
Ernest H. O’Boyle
Ashleigh Dickson
Daniel J. Davis
Alex DeHaven
Jaime Bochantin
Jared Borns
Publication date
11-06-2020
Publisher
Springer US
Published in
Journal of Business and Psychology / Issue 4/2021
Print ISSN: 0889-3268
Electronic ISSN: 1573-353X
DOI
https://doi.org/10.1007/s10869-020-09695-3

Other articles of this Issue 4/2021

Journal of Business and Psychology 4/2021 Go to the issue

Premium Partner