Skip to main content
SpringerLink
Log in

The Hawthorne effect in journal peer review

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Purpose—this paper aims to look at the Hawthorne effect in editorial peer review. Design/methodology/approach—discusses the quality evaluation of refereed scholarly journals. Findings—a key finding of this research was that in the peer review process of one and the same manuscript, reviewers or editors, respectively, arrive at different judgments. This phenomenon is named as “Hawthorne effect” because the different judgements are dependent on the specific conditions under which the peer review process at the individual journals takes place. Originality/value—provides a discussion on the quality evaluation of scholarly journals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  • Abelson, P. H. (1980). Scientific communication. Science, 209(4452), 60–62.

    Article  Google Scholar 

  • Bakanic, V., McPhail, C., & Simon, R. J. (1987). The manuscript review and decision-making process. American Sociological Review, 52(5), 631–642.

    Article  Google Scholar 

  • Benda, W. G. G., & Engels, T. C. E. (2011). The predictive validity of peer review: a selective review of the judgmental forecasting qualities of peers, and implications for innovation in science. International Journal of Forecasting, 27(1), 166–182. doi:10.1016/j.ijforecast.2010.03.003.

    Article  Google Scholar 

  • Bornmann, L. (2008). Scientific peer review. An analysis of the peer review process from the perspective of sociology of science theories. Human Architecture—Journal of the Sociology of Self-Knowledge, 6(2), 23–38.

    Google Scholar 

  • Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199–245.

    Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008a). The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173–7178. doi:10.1002/anie.200800513.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008b). Selecting manuscripts for a high impact journal through peer review: a citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59(11), 1841–1852. doi:10.1002/asi.20901.

    Article  Google Scholar 

  • Bunting, C. (2005, 25 February). Early careers spent grinding teeth, not cutting them. Times Higher Education Supplement, 18.

  • Callaham, M., & McCulloch, C. (2011). Longitudinal trends in the performance of scientific peer reviewers. [Proceedings Paper]. Annals of Emergency Medicine, 57(2), 141–148. doi:10.1016/j.annemergmed.2010.07.027.

    Article  Google Scholar 

  • Chew, F. S. (1991). Fate of manuscripts rejected for publication in the AJR. American Journal of Roentgenology, 156(3), 627–632.

    Google Scholar 

  • Chiesa, M., & Hobbs, S. (2008). Making sense of social research: how useful is the Hawthorne Effect? [Article]. European Journal of Social Psychology, 38(1), 67–74. doi:10.1002/ejsp.401.

    Article  Google Scholar 

  • Cole, S. (1992). Making science, between nature and society. Cambridge: Harvard University Press.

    Google Scholar 

  • Cronin, B., & McKenzie, G. (1992). The trajectory of rejection. Journal of Documentation, 48(3), 310–317.

    Article  Google Scholar 

  • Daniel, H.-D. (1993). Guardians of science, fairness and reliability of peer review. Weinheim: Wiley-VCH.

    Google Scholar 

  • de Vries, D. R., Marschall, E. A., & Stein, R. A. (2009). Exploring the peer review process: What is it, does it work, and can it be improved? Fisheries, 34(6), 270–279.

    Article  Google Scholar 

  • Eisenhart, M. (2002). The paradox of peer review: Admitting too much or allowing too little? Research in Science Education, 32(2), 241–255.

    Article  Google Scholar 

  • Fogg, L., & Fiske, D. W. (1993). Foretelling the judgments of reviewers and editors. American Psychologist, 48(3), 293–294.

    Article  Google Scholar 

  • French, J. R. P. (1953). Experiments in field settings. In L. Festinger & D. Katz (Eds.), Research methods in the behavioral sciences (pp. 98–135). New York: Holt, Rinehart and Winston.

    Google Scholar 

  • Gans, J. S., & Shepherd, G. B. (1994). How are the mighty fallen—rejected classic articles by leading economists. Journal of Economic Perspectives, 8(1), 165–179.

    Article  Google Scholar 

  • Gordon, M. D. (1984). How authors select journals—a test of the reward maximization model of submission behavior. Social Studies of Science, 14(1), 27–43.

    Article  Google Scholar 

  • Gorman, G. E. (2007). The Oppenheim effect in scholarly journal publishing. Online Information Review, 31(4), 417–419. doi:10.1108/14684520710780386.

    Article  Google Scholar 

  • Hojat, M., Gonnella, J. S., & Caelleigh, A. S. (2003). Impartial judgment by the “gatekeepers” of science: fallibility and accountability in the peer review process. Advances in Health Sciences Education, 8(1), 75–96.

    Article  Google Scholar 

  • Jayasinghe, U. W., Marsh, H. W., & Bond, N. (2001). Peer review in the funding of research in higher education: the Australian experience. Educational Evaluation and Policy Analysis, 23(4), 343–346.

    Article  Google Scholar 

  • Khosla, A., McDonald, R. J., Bornmann, L., & Kallmes, D. F. (2011). Getting to yes: the fate of neuroradiology manuscripts rejected by Radiology over a 2-year period. Radiology, 260(1), 3–5. doi:10.1148/radiol.11110490.

    Article  Google Scholar 

  • Lipworth, W. L., Kerridge, I. H., Carter, S. M., & Little, M. (2011). Journal peer review in context: a qualitative study of the social and subjective dimensions of manuscript review in biomedical publishing. [Review]. Social Science and Medicine, 72(7), 1056–1063. doi:10.1016/j.socscimed.2011.02.002.

    Article  Google Scholar 

  • Lock, S. (1985). A difficult balance: editorial peer review in medicine. Philadelphia: ISI Press.

    Google Scholar 

  • Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008). Improving the peer-review process for grant applications—reliability, validity, bias, and generalizability. American Psychologist, 63(3), 160–168. doi:10.1037/0003-066x.63.3.160.

    Article  Google Scholar 

  • Martin, B. (2000). Research grants: problems and options. Australian Universities’ Review, 43(2), 17–22.

    Google Scholar 

  • McCook, A. (2006). Is peer review broken? The Scientist, 20(2), 26.

    Google Scholar 

  • McDonald, R. J., Cloft, H. J., & Kallmes, D. F. (2007). Fate of submitted manuscripts rejected from the American Journal of Neuroradiology: outcomes and commentary. American Journal of Neuroradiology, 28(8), 1430–1434. doi:10.3174/Ajnr.A0766.

    Article  Google Scholar 

  • Opthof, T., Furstner, F., van Geer, M., & Coronel, R. (2000). Regrets or no regrets? No regrets! The fate of rejected manuscripts. Cardiovascular Research, 45(1), 255–258.

    Article  Google Scholar 

  • Owen, R. (1982). Reader bias. Journal of the American Medical Association, 247(18), 2533–2534.

    Article  Google Scholar 

  • Petty, R. E., & Fleming, M. A. (1999). The review process at PSPB: correlates of interreviewer agreement and manuscript acceptance. Personality and Social Psychology Bulletin, 25(2), 188–203.

    Article  Google Scholar 

  • Pruthi, S., Jain, A., Wahid, A., Mehra, K., & Nabi, S. A. (1997). Scientific community and peer review system—a case study of a central government funding scheme in India. Journal of Scientific & Industrial Research, 56(7), 398–407.

    Google Scholar 

  • Pulverer, B. (2010). Transparency showcases strength of peer review. [10.1038/468029a]. Nature, 468(7320), 29–31.

    Article  Google Scholar 

  • Ray, J., Berkwits, M., & Davidoff, F. (2000). The fate of manuscripts rejected by a general medical journal. American Journal of Medicine, 109(2), 131–135.

    Article  Google Scholar 

  • Ross, P. F. (1980). The sciences’ self-management: manuscript refereeing, peer review, and goals in science. Lincoln: The Ross Company.

    Google Scholar 

  • Sharp, D. W. (1990). What can and should be done to reduce publication bias—the perspective of an editor. Journal of the American Medical Association, 263(10), 1390–1391.

    Article  MathSciNet  Google Scholar 

  • Shatz, D. (2004). Peer review: a critical inquiry. Lanham: Rowman & Littlefield.

    Google Scholar 

  • Sternberg, R. J., Hojjat, M., Brigockas, M. G., & Grigorenko, E. L. (1997). Getting in: criteria for acceptance of manuscripts in Psychological Bulletin, 1993–1996. Psychological Bulletin, 121(2), 321–323.

    Article  Google Scholar 

  • van Rooyen, S., Godlee, F., Evans, S., Smith, R., & Black, N. (1998). Effect of blinding and unmasking on the quality of peer review—a randomized trial. Journal of the American Medical Association, 280(3), 234–237.

    Article  Google Scholar 

  • Weller, A. C. (1996). Editorial peer review: a comparison of authors publishing in two groups of US medical journals. Bulletin of the Medical Library Association, 84(3), 359–366.

    MathSciNet  Google Scholar 

  • Weller, A. C. (2002). Editorial peer review: its strengths and weaknesses. Medford: Information Today, Inc.

    Google Scholar 

  • Whitman, N., & Eyre, S. (1985). The pattern of publishing previously rejected articles in selected journals. Family Medicine, 17(1), 26–28.

    Google Scholar 

  • Wood, F. Q., & Wessely, S. (2003). Peer review of grant applications: a systematic review. In F. Godlee & T. Jefferson (Eds.), Peer review in health sciences (2nd ed., pp. 14–44). London: BMJ Books.

    Google Scholar 

  • Ziman, J. (1968). Public knowledge: an essay concerning the social dimension of science. Cambridge: Cambridge University Press.

    Google Scholar 

  • Ziman, J. (2000). Real science, what it is, and what it means. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Zuckerman, H., & Merton, R. K. (1971). Patterns of evaluation in science: institutionalisation, structure and functions of the referee system. Minerva, 9(1), 66–100. doi:10.1007/bf01553188.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Max Planck Society, Administrative Headquarters, Hofgartenstr. 8, 80539, Munich, Germany

    Lutz Bornmann

Authors
  1. Lutz Bornmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lutz Bornmann.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bornmann, L. The Hawthorne effect in journal peer review. Scientometrics 91, 857–862 (2012). https://doi.org/10.1007/s11192-011-0547-y

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-011-0547-y

Keywords

Search

Navigation