Skip to main content
SpringerLink
Log in

Journal impact evaluation: a webometric perspective

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

In theory, the web has the potential to provide information about the wider impact of academic research, beyond traditional scholarly impact. This is because the web can reflect non-scholarly uses of research, such as in online government documents, press coverage or public discussions. Nevertheless, there are practical problems with creating metrics for journals based on web data: principally that most such metrics should be easy for journal editors or publishers to manipulate. Nevertheless, two alternatives seem to have both promise and value: citations derived from digitised books and download counts for journals within specific delivery platforms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aguillo, I. F. (2009). Measuring the institution’s footprint in the web. Library Hi Tech, 27(4), 540–556.

    Article  Google Scholar 

  • Bar-Ilan, J. (2011). Articles tagged by ‘bibliometrics’ on Mendeley and CiteULike. Paper presented at the Metrics 2011 Symposium on Informetric and Scientometric Research.

  • Barjak, F., & Thelwall, M. (2008). A statistical analysis of the web presences of European life sciences research teams. Journal of the American Society for Information Science and Technology, 59(4), 628–643.

    Article  Google Scholar 

  • Becher, T., & Trowler, P. (2001). Academic tribes and territories (2nd ed.). Milton Keynes: Open University press.

    Google Scholar 

  • Bogers, T., Bosch, A. V. D. (2008). Recommending scientific articles using citeulike. In: Proceedings of the 2008 ACM conference on Recommender Systems (RecSys ‘08), New York, pp. 287–290.

  • Bollen, J., & Van de Sompel, H. (2008). Usage impact factor: the effects of sample characteristics on usage-based impact metrics. Journal of the American Society for Information Science and Technology, 59(1), 136–149.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behaviour. Journal of Documentation, 64(1), 45–80.

    Article  Google Scholar 

  • Brin, S., & Page, L. (1998). The anatomy of a large scale hyper textual web search engine. Computer Networks and ISDN Systems, 30(1–7), 107–117.

    Article  Google Scholar 

  • Brody, T., Harnad, S., & Carr, L. (2006). Earlier web usage statistics as predictors of later citation impact. Journal of the American Society for Information Science and Technology, 57(8), 1060–1072.

    Article  Google Scholar 

  • Brooks, T. A. (1986). Evidence of complex citer motivations. Journal of the American Society for Information Science, 37, 34–36.

    Google Scholar 

  • Chakrabarti, S., Dom, B., Raghavan, P., Rajagonpalan, S., Gibson, D., Kleinberg, J. M. (1998). Automatic resource compilation by analyzing hyperlink structure and associated text. Paper presented at the 7th International World Wide Web Conference, April 1998.

  • Chakrabarti, S., Joshi, M. M., Punera, K., Pennock, D. M. (2002). The structure of broad topics on the web. http://www2002.org/CDROM/refereed/338.

  • Chakrabarti, S., van den Berg, M., Dom, B. (1999). Focused crawling: a new approach to topic-specific web resource discovery. Paper presented at the 8th International World Wide Web Conference, May 1999.

  • Cole, S., & Cole, J. R. (1967). Scientific output and recognition: a study in the operation of the reward system in science. American Sociological Review, 32(3), 377–390.

    Article  Google Scholar 

  • Coleman, A. (2007). Assessing the value of a journal beyond the impact factor. Journal of the American Society for Information Science and Technology, 58(8), 1148–1161.

    Article  Google Scholar 

  • COUNTER. (2011a). Counting online usage of networked electronic resources. http://www.projectcounter.org/. Accessed 14 December 2011.

  • COUNTER. (2011b). Register of vendors providing usage reports compliant with release three of the code of practice for journals and databases. http://www.projectcounter.org/r2013/R2013CV_Dec2011.pdf. Accessed 14 December 2011.

  • Cronin, B. (1984). The citation process: the role and significance of citations in scientific communication. London: Taylor Graham.

    Google Scholar 

  • Davis, P. (2011). Gaming the impact factor puts journal in time-out. http://scholarlykitchen.sspnet.org/2011/2010/2017/gaming-the-impact-factor-puts-journal-in-time-out/. Accessed 14 December 2011.

  • De Costa, C. M. (2002). “The contagiousness of childbed fever”: a short history of puerperal sepsis and its treatment. The Medical Journal of Australia, 177(11/12), 668–671.

    Google Scholar 

  • Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research, 13(4), e123.

    Article  Google Scholar 

  • Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.

    Article  Google Scholar 

  • Garfield, E. (2005). The agony and the ecstasy: the history and the meaning of the journal impact factor. Fifth International Congress on Peer Review in Biomedical Publication, Chicago. USA. http://garfield.library.upenn.edu/papers/jifchicago2005.pdf. Accessed 27 September 2007.

  • Gyongyi, Z., Garcia-Molina, H., Pedersen, J. (2004). Combating web spam with TrustRank. Proceedings of the thirtieth international conference on very large data bases, 30, 576–587.

  • Han, S., Ahn, Y.-Y., Moon, S., Jeong, H. (2006). Collaborative blog spam filtering using adaptive percolation search, WWW2006 Workshop. http://www.blogpulse.com/www2006-workshop/papers/collaborative-blogspam-filtering.pdf. Accessed 5 May 2006.

  • Haustein, S., & Siebenlist, T. (2011). Applying social bookmarking data to evaluate journal usage. Journal of Informetrics, 5(3), 446–457.

    Google Scholar 

  • Henning, V., Reichelt, J. (2008). Mendeley—A last.fm for research? In: IEEE Fourth International Conference on eScience (eScience ‘08), Los Alamitos, pp. 327–328.

  • Huizenga, J. R. (1992). Cold fusion: the scientific fiasco of the century. Rochester: University of Rochester press.

    Google Scholar 

  • Ingwersen, P. (1998). The calculation of web impact factors. Journal of Documentation, 54(2), 236–243.

    Article  Google Scholar 

  • Jacsó, P. (2005). Google Scholar: the pros and the cons. Online Information Review, 29(2), 208–214.

    Article  Google Scholar 

  • Kleinberg, J. M. (1999). Authoritative sources in a hyperlinked environment. Journal of the ACM, 46(5), 604–632.

    Article  MathSciNet  MATH  Google Scholar 

  • Kousha, K., & Horri, A. (2004). The relationship between scholarly publishing and the counts of academic in links to Iranian university web sites: exploring academic link creation motivations. Journal of Information Management and Scientometrics, 1(2), 13–22.

    Google Scholar 

  • Kousha, K., & Thelwall, M. (2007). Google Scholar citations and Google Web/URL citations: a multi-discipline exploratory analysis. Journal of the American Society for Information Science and Technology, 58(7), 1055–1065.

    Article  Google Scholar 

  • Kousha, K., & Thelwall, M. (2008). Assessing the impact of disciplinary research on teaching: an automatic analysis of online syllabuses. Journal of the American Society for Information Science and Technology, 59(13), 2060–2069.

    Article  Google Scholar 

  • Kousha, K., & Thelwall, M. (2009). Google Book search: citation analysis for social science and the humanities. Journal of the American Society for Information Science and Technology, 60(8), 1537–1549.

    Article  Google Scholar 

  • Kousha, K., Thelwall, M., & Rezaie, S. (2010). Using the web for research evaluation: the integrated online impact indicator. Journal of Informetrics, 4(1), 124–135.

    Article  Google Scholar 

  • Larson, R. R. (1996). Bibliometrics of the world wide web: an exploratory analysis of the intellectual structure of cyberspace. ASIS 59th annual meeting, Baltimore.

  • Li, X., Thelwall, M., Giustini, D. (2011). Validating online reference managers for scholarly impact measurement. Scientometrics.

  • Li, X., Thelwall, M., Wilkinson, D., & Musgrove, P. B. (2005). National and international university departmental web site interlinking, part 2: link patterns. Scientometrics, 64(2), 187–208.

    Article  Google Scholar 

  • Maxmen, A. (2010). Science networking gets serious. Cell, 141(3), 387–389.

    Article  Google Scholar 

  • Mayr, P., & Walter, A. K. (2007). An exploratory study of Google Scholar. Online Information Review, 31(6), 814–830.

    Article  Google Scholar 

  • Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: web of science vs. scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105–2125.

    Article  Google Scholar 

  • Merton, R. K. (1973). The sociology of science. Theoretical and empirical investigations. Chicago: University of Chicago press.

    Google Scholar 

  • Meyer, M. (2003). Academic patents as an indicator of useful research? A new approach to measure academic inventiveness. Research Evaluation, 12(1), 17–27.

    Article  Google Scholar 

  • Moed, H. F. (2005). Statistical relationships between downloads and citations at the level of individual documents within a single journal. Journal of the American Society for Information Science and Technology, 56(10), 1088–1097.

    Article  Google Scholar 

  • Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.

    Article  Google Scholar 

  • Newton, I. (1676; 1992 republication). Letter from Isaac Newton to Robert Hooke, 5 February 1676. In: J.-P. Maury (Ed.), Newton: Understanding the Cosmos. New York: New Horizons.

  • NISO. (2011). ANSI/NISO Z39.93-2007 the standardized usage statistics harvesting initiative (SUSHI) protocol. http://www.niso.org/apps/group_public/project/details.php?project_id=2021. Accessed 14 December 2011.

  • Oppenheim, C. (2000). Do patent citations count? In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: a festschrift in honor of Eugene Garfield (pp. 405–432). Metford: Information today Inc. ASIS monograph series.

    Google Scholar 

  • Priem, J., Costello, K. L. (2010). How and why scholars cite on Twitter. In: Proceedings of the American Society for Information Science and Technology (ASIST 2010), vol 47. pp. 1–4).

  • Priem, J., Hemminger, B. M. (2010). Scientometrics 2.0: toward new metrics of scholarly impact on the social web. http://www.uic.edu/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2874/2570. First Monday, 15(7), Accessed 7 December 2011.

  • Priem, J., Taraborelli, D., Groth, P., Neylon, C. (2011). Altmetrics: a manifesto. http://altmetrics.org/manifesto/. Accessed 14 December 2011.

  • Provos, N., Mavrommatis, P., Rajab, M. A., Monrose, F. (2008). All your iFRAMEs point to us. http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//archive/provos-2008a.pdf. Accessed 13 December 2011.

  • Qiu, J. P., Chen, J. Q., & Wang, Z. (2004). An analysis of back link counts and web impact factors for Chinese university websites. Scientometrics, 60(3), 463–473.

    Article  Google Scholar 

  • RCUK. (2011). Types of impact. http://www.rcuk.ac.uk/documents/impacts/TypologyofResearchImpacts.pdf. Accessed 12 December 2011.

  • Rodríguez, I., & Gairín, J. M. (1997). Valorando el impacto de la información en Internet: AltaVista, el “Citation Index” de la Red (Evaluating the impact of internet information: Altavista, the “Citation Index” of the web). Revista Española de Documentación Científica, 20(2), 175–181.

    Article  Google Scholar 

  • Rousseau, R. (1997). Sitations: an exploratory study. http://www.cindoc.csic.es/cybermetrics/articles/v2001i2001p2001.html. Cybermetrics, 1(1), Accessed 25 July 2006.

  • Shepherd, P. (2011). Journal usage factor: results, recommendations and next steps. http://www.uksg.org/sites/uksg.org/files/JournalUsageFactorReport080711.pdf. Accessed 14 December 2011.

  • Smith, A. G. (1999). A tale of two web spaces; comparing sites using web impact factors. Journal of Documentation, 55(5), 577–592.

    Google Scholar 

  • Thelwall, M. (2001). Extracting macroscopic information from web links. Journal of American Society for Information Science and Technology, 52(13), 1157–1168.

    Article  Google Scholar 

  • Thelwall, M. (2002). Conceptualizing documentation on the web: an evaluation of different heuristic-based models for counting links between university web sites. Journal of American Society for Information Science and Technology, 53(12), 995–1005.

    Article  Google Scholar 

  • Thelwall, M., & Harries, G. (2004). Do the web sites of higher rated scholars have significantly more online impact? Journal of the American Society for Information Science and Technology, 55(2), 149–159.

    Article  Google Scholar 

  • Thelwall, M., & Kousha, K. (2008). Online presentations as a source of scientific impact?: an analysis of powerpoint files citing academic journals. Journal of the American Society for Information Science and Technology, 59(5), 805–815.

    Article  Google Scholar 

  • Thomas, O., & Willett, P. (2000). Webometric analysis of departments of librarianship and information science. Journal of Information Science, 26(6), 421–428.

    Article  Google Scholar 

  • van Raan, A. F. J. (2001). Bibliometrics and internet: some observations and expectations. Scientometrics, 50(1), 59–63.

    Article  Google Scholar 

  • Vanclay, J. K. (2012). Impact factor: outdated artefact or stepping-stone to journal certification? Scientometrics.

  • Vaughan, L., & Hysen, K. (2002). Relationship between links to journal web sites and impact factors. ASLIB Proceedings, 54(6), 356–361.

    Article  Google Scholar 

  • Vaughan, L., & Shaw, D. (2003). Bibliographic and web citations: what is the difference? Journal of the American Society for Information Science and Technology, 54(14), 1313–1322.

    Article  Google Scholar 

  • Vaughan, L., & Shaw, D. (2005). Web citation data for impact assessment: a comparison of four science disciplines. Journal of the American Society for Information Science and Technology, 56(10), 1075–1087.

    Article  Google Scholar 

  • Vaughan, L., & Shaw, D. (2008). A new look at evidence of scholarly citation in citation indexes and from web sources. Scientometrics, 74(2), 317–330.

    Article  Google Scholar 

  • Vaughan, L., & Thelwall, M. (2003). Scholarly use of the web: what are the key inducers of links to journal web sites? Journal of American Society for Information Science and Technology, 54(1), 29–38.

    Article  Google Scholar 

  • Vaughan, L., & Thelwall, M. (2005). A modeling approach to uncover hyperlink patterns: the case of Canadian universities. Information Processing and Management, 41(2), 347–359.

    Article  Google Scholar 

  • Weller, K., Dröge, E., Puschmann, C. (2011). Citation analysis in Twitter: approaches for defining and measuring information flows within tweets during scientific conferences. 1st Workshop on Making Sense of Microposts. http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-2718/paper_2004.pdf. Accessed 16 June 2011.

  • Weller, K., Puschmann, C. (2011). Twitter for scientific communication: how can citations/references be identified and measured? In: Proceedings of the poster session at the Web Science Conference 2011, Koblenz, Germany. http://journal.webscience.org/2500/). Accessed 21 October 2011.

  • Whitley, R. (2000). The intellectual and social organization of the sciences (2nd ed.). Oxford: Oxford University press.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Statistical Cybermetrics Research Group, School of Technology, University of Wolverhampton, Wulfruna Street, Wolverhampton, WV1 1LY, UK

    Mike Thelwall

Authors
  1. Mike Thelwall
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mike Thelwall.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Thelwall, M. Journal impact evaluation: a webometric perspective. Scientometrics 92, 429–441 (2012). https://doi.org/10.1007/s11192-012-0669-x

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-012-0669-x

Keywords

Search

Navigation