Skip to main content
SpringerLink
Log in

Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Alternative metrics are currently one of the most popular research topics in scientometric research. This paper provides an overview of research into three of the most important altmetrics: microblogging (Twitter), online reference managers (Mendeley and CiteULike) and blogging. The literature is discussed in relation to the possible use of altmetrics in research evaluation. Since the research was particularly interested in the correlation between altmetrics counts and citation counts, this overview focuses particularly on this correlation. For each altmetric, a meta-analysis is calculated for its correlation with traditional citation counts. As the results of the meta-analyses show, the correlation with traditional citations for micro-blogging counts is negligible (pooled r = 0.003), for blog counts it is small (pooled r = 0.12) and for bookmark counts from online reference managers, medium to large (CiteULike pooled r = 0.23; Mendeley pooled r = 0.51).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Bar-Ilan, J. (2012a). JASIST 2001–2010. Bulletin of the American Society for Information Science and Technology, 38(6), 24–28. doi:10.1002/bult.2012.1720380607.

    Google Scholar 

  • Bar-Ilan, J. (2012b). JASIST@mendeley. Paper presented at the altmetrics12: An ACM Web Science Conference 2012 Workshop, Evanston, IL, USA.

  • Bar-Ilan, J., Haustein, S., Peters, I., Priem, J., Shema, H., & Terliesner, J. (2012). Beyond citations: Scholars’ visibility on the social Web. In É. Archambault, Y. Gingras, & V. Larivière (Eds.), Proceedings of the 17th international conference on science and technology indicators (pp. 98–109). Montreal, QC: Science-Metrix and OST.

    Google Scholar 

  • Bar-Ilan, J., Shema, H., & Thelwall, M. (2014). Bibliographic references in Web 2.0. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 307–325). Cambridge, MA: MIT Press.

    Google Scholar 

  • Bik, H. M., & Goldstein, M. C. (2013). An introduction to social media for scientists. PLoS Biology, 11(4), e1001535. doi:10.1371/journal.pbio.1001535.

    Article  Google Scholar 

  • Bonetta, L. (2007). Scientists enter the blogosphere. Cell, 129(3), 443–445.

    Article  Google Scholar 

  • Börner, K., Sanyal, S., & Vespignani, A. (2007). Network science. Annual Review of Information Science and Technology, 41(1), 537–607. doi:10.1002/aris.2007.1440410119.

    Article  Google Scholar 

  • Bornmann, L. (2012). Measuring the societal impact of research. EMBO Reports, 13(8), 673–676.

    Article  Google Scholar 

  • Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society of Information Science and Technology, 64(2), 217–233.

    Article  Google Scholar 

  • Bornmann, L. (2014). Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime. Journal of Informetrics, 8(4), 935–950.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80. doi:10.1108/00220410810844150.

    Article  Google Scholar 

  • Bornmann, L., Leydesdorff, L., & Mutz, R. (2013). The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits. Journal of Informetrics, 7(1), 158–165.

    Article  Google Scholar 

  • Bornmann, L., Stefaner, M., de Moya Anegón, F., & Mutz, R. (2014). What is the effect of country-specific characteristics on the research performance of scientific institutions? Using multi-level statistical models to rank and map universities and research-focused institutions worldwide. Journal of Informetrics, 8(3), 581–593. doi:10.1016/j.joi.2014.04.008.

    Article  Google Scholar 

  • Bradburn, M. J., Deeks, J. J., & Altman, D. G. (1998). Metan—An alternative meta-analysis command. Stata Technical Bulletin, 44, 4–15.

    Google Scholar 

  • Cheung, M. W. L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19(2), 211–229. doi:10.1037/A0032968.

    Article  Google Scholar 

  • Colledge, L. (2014). Snowball metrics recipe book. Amsterdam: Snowball Metrics program partners.

    Google Scholar 

  • Colson, V. (2011). Science blogs as competing channels for the dissemination of science news. Journalism, 12(7), 889–902. doi:10.1177/1464884911412834.

    Article  Google Scholar 

  • Costas, R., Zahedi, Z., & Wouters, P. (2014). Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Retrieved February 11, from http://arxiv.org/abs/1401.4321

  • Darling, E. S., Shiffman, D., Côté, I. M., & Drew, J. A. (2013). The role of Twitter in the life cycle of a scientific publication. PeerJ PrePrints, 1, e16v11. doi:10.7287/peerj.preprints.16v1.

    Google Scholar 

  • de Bellis, N. (2014). History and evolution of (biblio)metrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 23–44). Cambridge, MA: MIT Press.

    Google Scholar 

  • Dinsmore, A., Allen, L., & Dolby, K. (2014). Alternative perspectives on impact: The potential of ALMs and altmetrics to inform funders about research impact. PLoS Biology, 12(11), e1002003. doi:10.1371/journal.pbio.1002003.

    Article  Google Scholar 

  • Duggan, M., & Smith, A. (2014). Social Media Update 2013. Retrieved March 28, from http://pewinternet.org/Reports/2013/Social-Media-Update.aspx

  • Eagly, A. H. (2005). Refereeing literature review submissions to journals. In R. J. Sternberg (Ed.), Reviewing scientific works in psychology. Washington, DC: American Psychological Association (APA).

    Google Scholar 

  • Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J Med Internet Res, 13(4), e123.

    Article  Google Scholar 

  • Fausto, S., Machado, F. A., Bento, L. F. J., Iamarino, A., Nahas, T. R., & Munger, D. S. (2012). Research blogging: Indexing and registering the change in Science 2.0. PLoS ONE, 7(12), e50109. doi:10.1371/journal.pone.0050109.

    Article  Google Scholar 

  • Galloway, L. M., Pease, J. L., & Rauh, A. E. (2013). Introduction to altmetrics for science, technology, engineering, and mathematics (STEM) librarians. Science & Technology Libraries, 32(4), 335–345. doi:10.1080/0194262X.2013.829762.

    Article  Google Scholar 

  • Glass, G. V. (1976). Primary, secondary, and meta-analysis. Educational Researcher, 5, 3–8.

    Article  Google Scholar 

  • Groth, P., & Gurney, T. (2010). Studying scientific discourse on the Web using bibliometrics: A chemistry blogging case study. Paper presented at the WebSci10: Extending the Frontiers of Society On-Line, Raleigh, NC, USA. http://wiki.few.vu.nl/sms/images/9/9c/Websci10-FINAL-29-4-2010f.pdf

  • Gunn, W. (2013). Social signals reflect academic impact: What it means when a scholar adds a paper to mendeley. Information Standards Quarterly, 25(2), 33–39.

    Article  Google Scholar 

  • Hammarfelt, B. (2014). Using altmetrics for assessing research impact in the humanities. Scientometrics. doi:10.1007/s11192-014-1261-3.

    Google Scholar 

  • Haustein, S. (2014). Readership metrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 327–344). Cambridge, MA: MIT Press.

    Google Scholar 

  • Haustein, S., & Peters, I. (2012). Using social bookmarks and tags as alternative indicators of journal content description. firstmonday, 17(11).

  • Haustein, S., Peters, I., Bar-Ilan, J., Priem, J., Shema, H., & Terliesner, J. (2014a). Coverage and adoption of altmetrics sources in the bibliometric community. Scientometrics. doi:10.1007/s11192-013-1221-3.

    MATH  Google Scholar 

  • Haustein, S., Peters, I., Sugimoto, C. R., Thelwall, M., & Larivière, V. (2014b). Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Science and Technology, 65(4), 656–669. doi:10.1002/asi.23101.

    Article  Google Scholar 

  • Haustein, S., & Siebenlist, T. (2011). Applying social bookmarking data to evaluate journal usage. Journal of Informetrics, 5(3), 446–457. doi:10.1016/j.joi.2011.04.002.

    Google Scholar 

  • Henning, V. (2010). The top 10 journal articles published in 2009 by readership on Mendeley | Mendeley Blog. Retrieved June 27, 2014, from http://www.mendeley.com/blog/academic-features/the-top-10-journal-articles-published-in-2009-by-readership-on-mendeley/

  • Holmberg, K., & Thelwall, M. (2014). Disciplinary differences in Twitter scholarly communication. Scientometrics. doi:10.1007/s11192-014-1229-3.

    Google Scholar 

  • Kjellberg, S. (2010). I am a blogging researcher: Motivations for blogging in a scholarly context. First Monday, 15(8).

  • Kovic, I., Lulic, I., & Brumini, G. (2008). Examining the medical blogosphere: An online survey of medical bloggers. J Med Internet Res, 10(3), e28.

    Article  Google Scholar 

  • Li, X., & Thelwall, M. (2012). F1000, Mendeley and traditional bibliometric indicators. In E. Archambault, Y. Gingras, & V. Lariviere (Eds.), The 17th international conference on science and technology indicators (pp. 541–551). Montreal: Repro-UQAM.

    Google Scholar 

  • Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2), 461–471. doi:10.1007/s11192-011-0580-x.

    Article  Google Scholar 

  • Lin, J., & Fenner, M. (2013). The many faces of article-level metrics. Bulletin of the American Society for Information Science and Technology, 39(4), 27–30. doi:10.1002/bult.2013.1720390409.

    Article  Google Scholar 

  • Liu, C. L., Xu, Y. Q., Wu, H., Chen, S. S., & Guo, J. J. (2013). Correlation and interaction visualization of altmetric indicators extracted from scholarly social network activities: Dimensions and structure. Journal of Medical Internet Research, 15(11), 17. doi:10.2196/jmir.2707.

    Article  Google Scholar 

  • Luzón, M. J. (2013). Public communication of science in blogs: Recontextualizing scientific discourse for a diversified audience. Written Communication. doi:10.1177/0741088313493610.

    MATH  Google Scholar 

  • Mahrt, M., Weller, K., & Peters, I. (2012). Twitter in scholarly communication. In K. Weller, A. Bruns, J. Burgess, M. Mahrt, & C. Puschmann (Eds.), Twitter and society (pp. 399–410). New York, NY: Peter Lang.

    Google Scholar 

  • Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H. D., & O’Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79(3), 1290–1326. doi:10.3102/0034654309334143.

    Article  Google Scholar 

  • Matt, G. E., & Navarro, A. M. (1997). What meta-analyses have and have not taught us about psychotherapy effects: A review and future directions. Clinical Psychology Review, 17(1), 1–32.

    Article  Google Scholar 

  • Mewburn, I., & Thomson, P. (2013). Why do academics blog? An analysis of audiences, purposes and challenges. Studies in Higher Education, 38(8), 1105–1119. doi:10.1080/03075079.2013.835624.

    Article  Google Scholar 

  • Mohammadi, E., & Thelwall, M. (2013). Assessing the Mendeley readership of social science and humanities research. In J. Gorraiz, E. Schiebel, C. Gumpenberger, & M. Ho (Eds.), Proceedings of ISSI 2013 Vienna: 14th International society of scientometrics and informetrics conference (pp. 200–214). Vienna, Austria: Austrian Institute of Technology GmbH.

    Google Scholar 

  • Mohammadi, E., & Thelwall, M. (2014). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. Journal of the Association for Information Science and Technology, n/a–n/a. doi:10.1002/asi.23071.

    Google Scholar 

  • Neylon, C., Willmers, M., & King, T. (2014). Rethinking impact: Applying altmetrics to southern African research. Ottawa: International Development Research Centre.

    Google Scholar 

  • NISO Alternative Assessment Metrics Project. (2014). NISO Altmetrics Standards Project White Paper. Retrieved July 8, 2014, from http://www.niso.org/apps/group_public/document.php?document_id=13295&wg_abbrev=altmetrics

  • Priem, J. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance. Cambridge, MA: MIT Press.

    Google Scholar 

  • Priem, J., & Costello, K. L. (2010). How and why scholars cite on Twitter. Proceedings of the American Society for Information Science and Technology, 47(1), 1–4. doi:10.1002/meet.14504701201.

    Article  Google Scholar 

  • Priem, J., Costello, K., & Dzuba, T. (2012, 2012/12/16). Prevalence and use of Twitter among scholars. Retrieved June 23, 2014, from http://figshare.com/articles/Prevalence_and_use_of_Twitter_among_scholars/104629

  • Priem, J., & Hemminger, B. M. (2010). Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web. First Monday, 15(7).

  • Priem, J., Piwowar, H., & Hemminger, B. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. Retrieved March 27, from http://arxiv.org/html/1203.4745

  • Puschmann, C. (2014). (Micro)Blogging Science? Notes on potentials and constraints of new forms of scholarly communication. In S. Bartling & S. Friesike (Eds.), Opening science (pp. 89–106). Cham, Switzerland: Springer International Publishing.

    Chapter  Google Scholar 

  • Puschmann, C., & Mahrt, M. (2012). Scholarly blogging: A new form of publishing or science journalism 2.0? In A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, K. Weller, & T. van Treeck (Eds.), Science and the Internet (pp. 171–182). Düsseldorf University Press: Düsseldorf.

    Google Scholar 

  • Rodgers, E. P., & Barbrow, S. (2013). A look at altmetrics and its growing significance to research libraries. Ann Arbor, MI: The University of Michigan University Library.

    Google Scholar 

  • Schlögl, C., Gorraiz, J., Gumpenberger, C., Jack, K., & Kraker, P. (2013). Download vs. vitiation vs. readership data: The case of an information systems journal. In J. Gorraiz, E. Schiebel, C. Gumpenberger, M. Hörlesberger, & H. Moed (Eds.), Proceedings of the 14th international society of scientometrics and informetrics conference. Austria: Austrian Institute of Technology, Vienna.

    Google Scholar 

  • Schlögl, C., Gorraiz, J., Gumpenberger, C., Jack, K., & Kraker, P. (2014). A comparison of citations, downloads and readership data for an information systems journal. Research Trends (37), 14–18.

  • Schubert, A., & Braun, T. (1986). Relative indicators and relational charts for comparative assessment of publication output and citation impact. Scientometrics, 9(5–6), 281–291.

    Article  Google Scholar 

  • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.

    Google Scholar 

  • Shema, H. (2014). Scholarly blogs are a promising altmetric source. Research Trends(37), 11-13.

  • Shema, H., Bar-Ilan, J., & Thelwall, M. (2012a). Research blogs and the discussion of scholarly information. PLoS ONE, 7(5), e35869. doi:10.1371/journal.pone.0035869.

    Article  Google Scholar 

  • Shema, H., Bar-Ilan, J., & Thelwall, M. (2012b). Self-citation of bloggers in the science blogosphere. In A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, K. Weller, & T. van Treeck (Eds.), Science and the Internet (pp. 183–192). Düsseldorf: Düsseldorf University Press.

    Google Scholar 

  • Shema, H., Bar-Ilan, J., & Thelwall, M. (2014). Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics. Journal of the Association for Information Science and Technology, 65(5), 1018–1027. doi:10.1002/asi.23037.

    Article  Google Scholar 

  • Shema, H., Bar-Ilan, J., & Thelwall, M. (in press). How is research blogged? A content analysis approach. Journal of the Association for Information Science and Technology.

  • Shuai, X., Pepe, A., & Bollen, J. (2012). How the scientific community reacts to newly submitted preprints: Article downloads, Twitter mentions, and citations. Plos One, 7(11). doi: ARTN e47523 doi:10.1371/journal.pone.0047523.

  • StataCorp. (2013). Stata statistical software: Release 13. College Station, TX: Stata Corporation.

    Google Scholar 

  • Sud, P., & Thelwall, M. (in press). Not all international collaboration is beneficial: The Mendeley readership and citation impact of biochemical research collaboration. Journal of the Association for Information Science and Technology.

  • Taylor, M. (2013). Towards a common model of citation: Some thoughts on merging altmetrics and bibliometrics. Research Trends (35), 19–22.

  • Thelwall, M. (2014, January 2). Five recommendations for using alternative metrics in the future UK Research Excellence Framework. Retrieved from http://blogs.lse.ac.uk/impactofsocialsciences/2014/10/23/alternative-metrics-future-uk-research-excellence-framework-thelwall/

  • Thelwall, M., Haustein, S., Lariviere, V., & Sugimoto, C. R. (2013). Do altmetrics work? Twitter and ten other social web services. PLoS ONE,. doi:10.1371/journal.pone.0064841.

    Google Scholar 

  • Thelwall, M., & Maflahi, N. (in press). Are scholarly articles disproportionately read in their own country? An analysis of Mendeley readers. Journal of the Association for Information Science and Technology.

  • Thorsen, E. (2013). Blogging on the ice: Connecting audiences with climate-change sciences. International Journal of Media & Cultural Politics, 9(1), 87–101. doi:10.1386/macp.9.1.87_1.

    Article  Google Scholar 

  • Torres-Salinas, D., Cabezas-Clavijo, A., & Jimenez-Contreras, E. (2013). Altmetrics: New indicators for scientific communication in Web 2.0. Comunicar, 41, 53–60.

    Article  Google Scholar 

  • Tramer, M. R., Reynolds, D. J. M., Moore, R. A., & McQuay, H. J. (1997). Impact of covert duplicate publication on meta-analysis: A case study. British Medical Journal, 315(7109), 635–640.

    Article  Google Scholar 

  • Wainer, J., & Vieira, P. (2013). Correlations between bibliometrics and peer evaluation for all disciplines: The evaluation of Brazilian scientists. Scientometrics, 96(2), 395–410. doi:10.1007/s11192-013-0969-9.

    Article  Google Scholar 

  • Weller, K., Dröge, E., & Puschmann, C. (2011). Citation analysis in Twitter: Approaches for defining and measuring information flows within Tweets during scientific conferences. In M. Rowe, M. Stankovic, A.-S. Dadzie, & M. Hardey (Eds.), Making Sense of Microposts (MSM2011) (pp. 1–12). Heraklion: CEUR Workshop Proceedings.

    Google Scholar 

  • Weller, K., & Peters, I. (2012). Citations in Web 2.0. In A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, T. van Treeck, & K. Weller (Eds.), Science and the Internet (pp. 209–222). Düsseldorf: Düsseldorf University Press.

    Google Scholar 

  • Weller, K., & Puschmann, C. (2011, June 14–17 2011). Twitter for Scientific Communication: How Can Citations/References be Identified and Measured? Paper presented at the ACM WebSci’11, Koblenz, Germany.

  • White, H. D. (2005). On extending informetrics: An opinion paper. In P. Ingwersen & B. Larsen (Eds.), Proceedings of the 10th International conference of the international society for scientometrics and informetrics (Vol. 2, pp. 442–449). Stockholm, Sweden: Karolinska University Press.

    Google Scholar 

  • Wolinsky, H. (2011). More than a blog. EMBO Reports, 12(11), 1102–1105.

    Article  Google Scholar 

  • Wouters, P. (2014). The citation: From culture to infrastructure. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 47–66). Cambridge, MA: MIT Press.

    Google Scholar 

  • Yan, K. K., & Gerstein, M. (2011). The spread of scientific information: Insights from the web usage statistics in PLoS article-level metrics. PloS One, 6(5). doi: ARTN e19917 doi:10.1371/journal.pone.0019917

  • Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics. doi:10.1007/s11192-014-1264-0.

    Google Scholar 

  • Zubiaga, A., Spina, D., Martínez, R., & Fresno, V. (2014). Real-time classification of twitter trends. Journal of the Association for Information Science and Technology, n/a–n/a. doi:10.1002/asi.23186.

    Google Scholar 

Download references

Acknowledgments

I would like to thank Hadas Shema for discussing the concept of this paper.

Author information

Authors and Affiliations

  1. Division for Science and Innovation Studies, Administrative Headquarters of the Max Planck Society, Hofgartenstr. 8, 80539, Munich, Germany

    Lutz Bornmann

Authors
  1. Lutz Bornmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lutz Bornmann.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bornmann, L. Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics. Scientometrics 103, 1123–1144 (2015). https://doi.org/10.1007/s11192-015-1565-y

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-015-1565-y

Keywords

Search

Navigation