Quantifying perceived impact of scientific publications

Highlights

• Researchers tend to prefer their own publications to those by others. This bias is so strong that they prefer their own publications even to the most cited papers in their own field of research.

• When assessing papers written by others, researchers perceive the impact of these papers in way that is not strongly correlated with citation impact.

• Citation impact and perceived impact are significantly correlated only when researchers are asked to judge their own work.

Abstract

We report on an empirical verification of the degree to which citation numbers represent scientific impact as it is actually perceived by experts in their respective field. We run a survey of about 2000 corresponding authors who performed a pairwise impact assessment task across more than 20,000 scientific articles. Results of the survey show that citation data and perceived impact do not align well, unless one properly accounts for psychological biases that affect the opinions of experts with respect to their own papers vs. those of others. Researchers tend to prefer their own publications to the most cited papers in their field of research. There is only a mild positive correlation between the number of citations of top-cited papers and expert preference in pairwise comparisons. This also applies to pairs of papers with several orders of magnitude differences in their total number of accumulated citations. However, when researchers were asked to choose among pairs of their own papers, thus eliminating the bias favouring one's own papers over those of others, they did systematically prefer the most cited article. We conclude that, when scientists have full information and are making unbiased choices, expert opinion on impact is congruent with citation numbers.

Introduction

Metrics based on bibliographic data increasingly inform important decision-making processes in science, such as hiring, tenure, promotion, and funding (Bornmann & Daniel, 2006; Bornmann, Mutz, Marx, Schier, & Daniel, 2011; Bornmann, Wallon, & Ledin, 2008; Harzing, 2010; Hornbostel, Böhmer, Klingsporn, Neufeld, & von Ins, 2009; Lovegrove & Johnson, 2008). Many, if not most, of these bibliometric indicators are derived from article-level citation counts ([Bar-Ilan, 2008], [Van Noorden, 2010]), e.g. the h-index (Hirsch, 2005) and the journal impact factor (Garfield, 2006), due to the wide availability of extensive bibliographic records. However, they rest on the frequently undeclared assumption that citations are an accurate and reliable reflection of scientific impact. Bornmann and Daniel (2008) and Sarigöl, Pfitzner, Scholtes, Garas, and Schweitzer (2014) find that citations can serve as reasonable approximations of social indicators such as popularity or success, but they do not support the most important assumption underlying most work in the area of citation-based impact indicators, namely that citations quantify scientific impact. Here, we investigate this assumption directly by determining whether citation data truly reflect scientific impact as it is perceived by expert scholars in their respective fields. Such an assumption is central in the long-standing debate about the proper use of citation data (Garfield, 1979). One common argument against the use of citation data in assessment exercises rests in particular on the doubts about their validity as indicators of true scientific impact (Adler, Ewing, & Taylor, 2009; [MacRoberts and MacRoberts, 1996], [MacRoberts and MacRoberts, 2010]). In fact, the literature frequently confounds “impact”, “influence”, and “rank” with citation-derived metrics (Radicchi, Fortunato, & Castellano, 2008; Wang, Song, & Barabási, 2013; Wuchty, Jones, & Uzzi, 2007) although these metrics can vary along multiple distinct dimensions (Bollen, Van de Sompel, Hagberg, & Chute, 2009; Bornmann & Daniel, 2008).

Determining whether citations actually indicate scientific impact is an empirical question which cannot be resolved by theoretical discussions of the perceived or presumed benefits vs. demerits of citation data alone. For this reason, we decided to define and empirically quantify a novel post-publication metric, namely impact as perceived by the authors themselves. Our goal is to understand whether citation numbers “truly” reflect a particular dimension of impact: the influence or importance of papers in the daily practice of researchers.

We designed a large-scale survey at bigscience.soic.indiana.edu to collect responses from thousands of experienced scholars in a multitude of disciplines (Fig. A1). We asked researchers to make a pairwise decision with regards to their preference of one paper over the other. These decisions were made under two conditions: whether the articles were written by the expert scholars themselves or by other scholars. We then aggregated results over the entire population of respondents to quantify the degree of correlation between the pairwise preferences of respondents (i.e., perceived impact) and the actual difference in the number of citations accumulated (i.e., citation impact) for the pair of papers. Our results indicate that, from the perspective of individual researchers, their assessment of impact vs. impact judged from citation data are only related for pairs of their own papers. Every time that a paper not co-authored by the individual is involved in the estimation of perceived impact, this comparison shows null or negative correlation with citation impact.