nach oben

Journal of Business and Psychology

Erschienen in:

01.09.2016 | Editorial

Editorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly

verfasst von: George C. Banks, Steven G. Rogelberg, Haley M. Woznyj, Ronald S. Landis, Deborah E. Rupp

Erschienen in: Journal of Business and Psychology | Ausgabe 3/2016

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Purpose

Questionable research or reporting practices (QRPs) contribute to a growing concern regarding the credibility of research in the organizational sciences and related fields. Such practices include design, analytic, or reporting practices that may introduce biased evidence, which can have harmful implications for evidence-based practice, theory development, and perceptions of the rigor of science.

Design/Methodology/Approach

To assess the extent to which QRPs are actually a concern, we conducted a systematic review to consider the evidence on QRPs. Using a triangulation approach (e.g., by reviewing data from observations, sensitivity analyses, and surveys), we identified the good, the bad, and the ugly.

Findings

Of the 64 studies that fit our criteria, 6 appeared to find little to no evidence of engagement in QRPs and the other 58 found more severe evidence (91 %).

Implications

Drawing upon the findings, we provide recommendations for future research related to publication practices and academic training.

Originality/value

We report findings from studies that suggest that QRPs are not a problem, that QRPs are used at a suboptimal rate, and that QRPs present a threat to the viability of organizational science research.

Nächster Artikel Four Research Designs and a Comprehensive Analysis Strategy for Investigating Common Method Variance with Self-Report Measures Using Latent Variables

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Nur mit Berechtigung zugänglich

Aliseda, A. (2006). Abductive reasoning: Logical investigations into discovery and explanation. Dordrecht: Springer.

Allen, P. J., Lourenco, A., & Roberts, L. D. (2015). Detecting duplication in students’ research data: A method and illustration. Ethics & Behavior. doi:10.1080/10508422.2015.1019070.

Bailey, C. D. (2015). Psychopathy, academic accountants’ attitudes toward unethical research practices, and publication success. The Accounting Review, 90(4), 1307–1332.CrossRef

Bakker, M., & Wicherts, J. M. (2014). Outlier removal and the relation with reporting errors and quality of psychological research. PLoS One, 9(7), e103360.CrossRefPubMedPubMedCentral

Bandura, A. (2001). Social cognitive theory: An agentic perspective. Annual Review of Psychology, 52, 1–26.CrossRefPubMed

Banks, G. C., et al. (2016). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42(1), 5–20.CrossRef

Banks, G. C., & O’Boyle, E. H. (2013). Why we need industrial-organizational psychology to fix industrial-organizational psychology. Industrial and Organizational Psychology, 6, 291–294.CrossRef

Banks, G. C., O’Boyle, E. H., White, C. D., & Batchelor, J. H. (2013). Tracking SMA papers to journal publication: An investigation into the phases of dissemination bias, Paper presented at the 2013 annual meeting of the Southern Management Association, New Orleans, LA.

Becker, T. E. (2005). Potential problems in the statistical control of variables in organizational research: A qualitative analysis with recommendations. Organizational Research Methods, 8, 274–289.CrossRef

Bedeian, A. G., Taylor, S. G., & Miller, A. N. (2010). Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning & Education, 9(4), 715–725. doi:10.5465/amle.2010.56659889.CrossRef

Berry, C. M., Carpenter, N. C., & Barratt, C. L. (2012). Do other reports of counterproductive work behavior provide an incremental contribution over self-reports? A meta-analytic comparison. Journal of Applied Psychology, 97, 613–636. doi:10.1037/a0026739.CrossRefPubMed

Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. (2015). HARKing’s threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology. doi:10.1111/peps.12111.

Braun, M., & Roussos, A. J. (2012). Psychotherapy researchers: Reported misbehaviors and opinions. Journal of Empirical Research on Human Research Ethics, 7(5), 25–29.CrossRefPubMed

Cortina, J. M. (2015). A revolution with a solution. Opening plenary presented at the meeting of the Society for Industrial/Organizational Psychology, Philadelphia, PA.

Davis, M. S., Riske-Morris, M., & Diaz, S. R. (2007). Causal factors implicated in research misconduct: Evidence from ORI case files. Science and Engineering Ethics, 13(4), 395–414.CrossRefPubMed

de Winter, J. C. F., & Dodou, D. (2015). A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too). PeerJ, 3, e733. doi:10.7717/peerj.733.CrossRefPubMedPubMedCentral

Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010). Testing for the presence of positive-outcome bias in peer review: A randomized controlled trial. Archives of Internal Medicine, 170, 1934–1939. doi:10.1001/archinternmed.2010.406.CrossRefPubMed

Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS One, 4(5), e5738.CrossRefPubMedPubMedCentral

Fanelli, D. (2010). Do pressures to publish increase scientists’ bias? An empirical support from US States Data. PloS One, 5(4), e10271.CrossRefPubMedPubMedCentral

Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904. doi:10.1007/s11192-011-0494-7.CrossRef

Fiedler, K., & Schwarz, N. (2015). Questionable research practices revisited. Social Psychological and Personality Science, 7, 45–52.CrossRef

Field, J. G., Mihm, D., O’Boyle, E. H., Bosco, F. A., Uggerslev, K., & Steel, P. (2015). An examination of the funding-finding relation in the field of management. Academy of Management Proceedings. Paper presented at the Academy of Management Annual Meeting, Vancouver, Canada (p. 17463).

Field et al. (2016). The extent of p-hacking in I/O psychology. Paper presented at the Society of Industrial/Organizational Psychology Annual Conference in Anaheim, CA.

Francis, G. (2014). The frequency of excess success for articles in Psychological Science. Psychonomic Bulletin & Review, 21(5), 1180–1187.CrossRef

Francis, G., Tanzman, J., & Matthews, W. J. (2014). Excess success for psychology articles in the journal Science. PLoS One, 9(12), e114255.CrossRefPubMedPubMedCentral

Franco, A., Malhotra, N., & Simonovits, G. (2016). Underreporting in psychology experiments evidence from a study registry. Social Psychological and Personality Science, 7(1), 8–12.CrossRef

Gerber, A., & Malhotra, N. (2008a). Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science, 3, 313–326. doi:10.1561/100.00008024.CrossRef

Gerber, A. S., & Malhotra, N. (2008b). Publication bias in empirical sociological research do arbitrary significance levels distort published results? Sociological Methods & Research, 37, 3–30. doi:10.1177/0049124108318973.CrossRef

Glick, J. L., & Shamoo, A. E. (1994). Results of a survey on research practices, completed by attendees at the third conference on research policies and quality assurance. Accountability in Research, 3(4), 275–280.CrossRef

Hambrick, D. C. (2007). The field of management’s devotion to theory: Too much of a good thing? Academy of Management Journal, 50, 1346–1352.CrossRef

Harrison, J. S., Banks, G. C., Pollack, J. M., O’Boyle Jr., E. H., & Short, J. C. (2014). Publication bias in strategic management research. Journal of Management. doi:10.1177/0149206314535438.

Hartgerink, C. H., van Aert, R. C., Nuijten, M. B., Wicherts, J. M., & van Assen, M. A. (2016). Distributions of p-values smaller than.05 in Psychology: What is going on? PeerJ, 4, e1935.CrossRefPubMedPubMedCentral

Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13(3), e1002106.CrossRefPubMedPubMedCentral

Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, 602–611. doi:10.2307/2392366.CrossRef

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. doi:10.1177/0956797611430953.CrossRefPubMed

Jørgensen, M., Dybå, T., Liestøl, K., & Sjøberg, D. I. (2015). Incorrect results in software engineering experiments: How to improve research practices. Journal of Systems and Software,. doi:10.1016/j.jss.2015.03.065.

Kattenbraker, M. (2007). Health education research and publication: ethical considerations and the response of health educators (Unpublished thesis). Southern Illinois University Carbondale, Carbondale, IL.

Kepes, S., Banks, G. C., McDaniel, M. A., & Whetzel, D. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624–662. doi:10.1177/1094428112452760.CrossRef

Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in I-O psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 252–268.CrossRef

Kepes, S., McDaniel, M. A., Brannick, M. T., & Banks, G. C. (2013). Meta-analytic reviews in the organizational sciences: Two meta-analytic schools on the way to MARS (the Meta-Analytic Reporting Standards). Journal of Business and Psychology, 28, 123–143.CrossRef

Kerr, N. L., & Harris, S. E. (1998). HARKing: hypothesizing after the results are known: Views from three disciplines. Unpublished manuscript, Michigan State University, East Lansing.

Krawczyk, M. (2015). The search for significance: A few peculiarities in the distribution of p-values in experimental psychology literature. PloS One, 10(6), e0127872.CrossRefPubMedPubMedCentral

Landis, R. S., Lance, C. E., Pierce, C. A., & Rogelberg, S. G. (2014). When is nothing something? Editorial for the null results special issue of Journal of Business and Psychology. Journal of Business and Psychology, 29, 163–167. doi:10.1007/s10869-014-9347-8.CrossRef

LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T. (2013). PsychDisclosure.org grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8(4), 424–432.CrossRefPubMed

Leggett, N. C., Thomas, N. A., Loetscher, T., & Nicholls, M. E. (2013). The life of p: “Just significant” results are on the rise. The Quarterly Journal of Experimental Psychology, 66(12), 2303–2309.CrossRefPubMed

List, J. A., & Gallet, C. A. (2001). What experimental protocol influence disparities between actual and hypothetical stated values? Environmental and Resource Economics, 20(3), 241–254.CrossRef

Locke, E. A. (2007). The case for inductive theory building. Journal of Management, 33, 867–890.CrossRef

Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57, 705–717.CrossRefPubMed

Martinson, B. C., Anderson, M. S., Crain, A. L., & De Vries, R. (2006). Scientists’ perceptions of organizational justice and self-reported misbehaviors. Journal of Empirical Research on Human Research Ethics, 1(1), 51–66.CrossRefPubMedPubMedCentral

Martinson, B. C., Anderson, M. S., & De Vries, R. (2005). Scientists behaving badly. Nature, 435(7043), 737–738.CrossRefPubMed

Martinson, B. C., Crain, A. L., Anderson, M. S., & De Vries, R. (2009). Institutions’ expectations for researchers’ self-funding, federal grant holding and private industry involvement: Manifold drivers of self-interest and researcher behavior. Academic Medicine: Journal of the Association of American Medical Colleges, 84(11), 1491–1499.CrossRef

Masicampo, E. J., & Lalande, D. R. (2012). A peculiar prevalence of p-values just below. 05. The Quarterly Journal of Experimental Psychology and Aging, 65(11), 2271–2279. doi:10.1080/17470218.2012.711335.CrossRef

Masters, E. A. (2012). Research misconduct in National Science Foundation funded research a mixed-methods analysis of 2007-2011 research awards (Unpublished doctoral dissertation). Northcentral University, Prescott Valley, AZ.

Matthes, J., Marquart, F., Naderer, B., Arendt, F., Schmuck, D., & Adam, K. (2015). Questionable research practices in experimental communication research: A systematic analysis from 1980 to 2013. Communication Methods and Measures, 9(4), 193–207.CrossRef

Mazzola, J. J., & Deuling, J. K. (2013). Forgetting what we learned as graduate students: HARKing and selective outcome reporting in I-O journal articles. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6(03), 279–284.CrossRef

Meyer, M. J., & McMahon, D. (2004). An examination of ethical research conduct by experienced and novice accounting academics. Issues in Accounting Education, 19(4), 413–442.CrossRef

Nagel, M., Wicherts, J. M., & Bakker, M. Participant exclusion in psychological research: A study of its effects on research results. Unpublished manuscript.

Necker, S. (2014). Scientific misbehavior in economics. Research Policy, 43(10), 1747–1759.CrossRef

Nosek, B. A., et al. (2015). Promoting an open research culture: Author guidelines for journals to promote transparency, openness, and reproducibility. Science, 348, 1422–1425. doi:10.1126/science.aab2374.CrossRefPubMedPubMedCentral

Nuijten, M. B., Hartgerink, C. H., van Assen, M. A., Epskamp, S., & Wicherts, J. M. (2015). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods. doi:10.3758/s13428-015-0664-2.

O’Boyle, E. H., Banks, G. C., & Gonzalez-Mule, E. (2014). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management. doi:10.1177/0149206314527133.

O’Boyle, E. H., Banks, G. C., Carter, K., Walter, S., & Yuan, Z. (2015). A 20-year review of outcome reporting bias in moderated multiple regression. Paper presented at the annual meeting of the Academy of Management, Vancouver, British Columbia.

Pigott, T. D., Valentine, J. C., Polanin, J. R., Williams, R. T., & Canada, D. D. (2013). Outcome-reporting bias in education research. Educational Researcher. doi:10.3102/0013189X13507104.

Rajah-Kanagasabai, C. J., & Roberts, L. D. (2015). Predicting self-reported research misconduct and questionable research practices in university students using an augmented Theory of Planned Behavior. Frontiers in Psychology, 6, 1–11.CrossRef

Reed, J. G., & Baxter, P. M. (2009). Using reference databases. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 74–101). New York: Russell Sage Foundation.

Riordan, C. A., & Marlin, N. A. (1987). Some good news about some bad practices. American Psychologist, 42(1), 104–106.CrossRef

Rogelberg, S. G., & Laber, M. (2002). Securing our collective future: Challenges facing those designing and doing research in Industrial and Organizational Psychology. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 479–485). London: Blackwell.

Rupp, D. E. (2011). Research and publishing ethics: Editor and reviewer responsibilities. Management and Organizational Review, 7, 481–493.CrossRef

Sackett, P. R., & Larson, J. R. (1990). Research strategies and tactics in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp. 419–489). Palo Alto, CA: Consulting Psychologists Press.

Schimmack, U. (2014). Quantifying statistical research integrity: The Replicabilty-Index. Unpublished manuscript.

Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Newbury Park, CA: Sage.

Spector, P. E., Rogelberg, S. G., Ryan, A. M., Schmitt, N., & Zedeck, S. (2014). Moving the pendulum back to the middle: Reflections on and introduction to the inductive research special issue of Journal of Business and Psychology. Journal of Business and Psychology, 29, 499–502. doi:10.1007/s10869-014-9372-7.CrossRef

Swazey, J. P., Anderson, M. S., Lewis, K. S., & Louis, K. S. (1993). Ethical problems in academic research. American Scientist, 81(6), 542–553.

Tangney, J. P. (1987). Fraud will out-or will it? New Scientist, 115, 62–63.PubMed

Titus, S. L., Wells, J. A., & Rhoades, L. J. (2008). Repairing research integrity. Nature, 453(7198), 980–982.CrossRefPubMed

Trainor, B. P. (2015). Incomplete reporting: Addressing the problem of outcome-reporting bias in educational research (Unpublished doctoral dissertation). Loyala University, Chigao, IL.

Vasilev, M. R. (2013). Negative results in European psychology journals. Europe’s Journal of Psychology, 9(4), 717–730.CrossRef

Veldkamp, C. L., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A., & Wicherts, J. M. (2014). Statistical reporting errors and collaboration on statistical analyses in psychological science. PloS One, 9(12), e114876.CrossRefPubMedPubMedCentral

Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4(3), 274–290.CrossRefPubMed

Wilson, K., Schreier, A., Griffin, A., & Resnik, D. (2007). Research records and the resolution of misconduct allegations at research universities. Accountability in Research, 14(1), 57–71.CrossRefPubMedPubMedCentral

Titel: Editorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly
verfasst von: George C. Banks
Steven G. Rogelberg
Haley M. Woznyj
Ronald S. Landis
Deborah E. Rupp
Publikationsdatum: 01.09.2016
Verlag: Springer US
Erschienen in: Journal of Business and Psychology / Ausgabe 3/2016
Print ISSN: 0889-3268
Elektronische ISSN: 1573-353X
DOI: https://doi.org/10.1007/s10869-016-9456-7

Springer Professional

Abstract

Purpose

Design/Methodology/Approach

Findings

Implications

Originality/value

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Weitere Artikel der Ausgabe 3/2016

Idiosyncratic Deals and Helping Behavior: The Moderating Role of I-deal Opportunity for Co-workers

Fostering Change-Oriented Behaviors: A Broaden-and-Build Model

Four Research Designs and a Comprehensive Analysis Strategy for Investigating Common Method Variance with Self-Report Measures Using Latent Variables

Domain Satisfaction as a Mediator of the Relationship Between Work–Family Spillover and Subjective Well-Being: A Longitudinal Study

Examining Ethicality Judgements of Theft Behavior: The Role of Moral Relativism

The Moderating Role of Intergroup Contact in Race Composition, Perceived Similarity, and Applicant Attraction Relationships