Skip to main content
SpringerLink
Log in

On using the Shanghai ranking to assess the research performance of university systems

  • Published:
Scientometrics Aims and scope Submit manuscript

An Erratum to this article was published on 04 November 2010

Abstract

We take a new look at the Shanghai Jiao Tong Academic Ranking of World Universities to evaluate the performance of whole university systems. We deal with system aggregates by means of averaging scores taken over a number of institutions from each higher education system according to the Gross Domestic Product of its country. We treat the set of indicators (measures) at the country level as a scale, and investigate its reliability and dimensionality using appropriate statistical tools. After a Principal Component Analysis is performed, a clear picture emerges: at the aggregate level ARWU seems to be a very reliable one-dimensional scale, with a first component that explains more than 72% of the variance of the sample under analysis. The percentages of variance of the indicators explained by the first component do shed light on the fact that ARWU is in fact measuring the research quality (both at the individual and collective levels) of a university system. When the second principal component is taken into account, the two principal components contribute to explain more than 90% of the variance. The rotated solution facilitates the interpretation of the components and provides clear and interesting clustering information about the 32 higher education systems under analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. For a review of the nature of the indicators we refer to the web page of ARWU (http://www.arwu.org).

  2. http://www.isihighlycited.com.

  3. When the number of academic staff for the institution is not known, ARWU uses the weighted total scores of the other five indicators.

References

  • Bartlett, M. S. (1954). A note on the multiplying factors for various chi-square approximations. Journal of the Royal Statistical Society, 16(B), 296–98.

    MATH  MathSciNet  Google Scholar 

  • Billaut, J. C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shangai ranking: An MCDM view. Scientometrics, 84(1), 237–263.

    Article  Google Scholar 

  • Brown, T. A. (2006). Confirmatory factor analysis for applied research (3rd ed.). New York: The Guilford Press.

    Google Scholar 

  • Catell, R. B. (1966). The scree test for number of factors. Multivariate Behavioural Research, 1, 245–276.

    Article  Google Scholar 

  • Dehon, C., McCathie, A., & Verardi, V. (2010). Uncovering excellence in academic rankings: A closer look at the Shanghai ranking. Scientometrics, 83(2), 515–524.

    Article  Google Scholar 

  • DeVellis, R. (2003). Scale development: Theory and applications (2nd ed.). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Docampo, D. (2008). Rankings internacionales y calidad institucional. Revista de Educación, Número Extraordinario, 149–176.

  • Florian, R. V. (2007). Irreproducibility of the results of the Shangai academic ranking of world universities. Scientometrics, 72(1), 25–32.

    Article  MathSciNet  Google Scholar 

  • Guadagnoli, E., & Vellicer, W. (1988). Relation of sample size to the stability of component patterns. Psychological Bulletin, 103, 265–275.

    Article  Google Scholar 

  • Hsu, J. (1996). Multiple comparisons: Theory and methods. London: Chapman & Hall.

    MATH  Google Scholar 

  • IMF. (2009). World Economic Outlook (WEO) Database: Downloaded from the International Monetary Fund server on November 19th 2009. http://www.imf.org/external/pubs/ft/weo/2009/02/index.htm..

  • Jackson, J. E. (2003). A users guide to principal components. Hoboken, New Jersey: Wiley.

    Google Scholar 

  • Kaiser, H. (1974). An index of factorial simplicity. Psychometrika, 39, 31–36.

    Article  MATH  Google Scholar 

  • Liu, N. C., & Cheng, Y. (2005). Academic ranking of world universities: Methodologies and problems. Higher Education in Europe, 30(2), 127–136.

    Article  Google Scholar 

  • Marginson, S. (2005). There must be some way out of here. Tertiary Educ. Management Conference, Keynote address, Perth, Australia.

  • Morrison, D. (2000). Multivariate statistical methods (3rd ed.). New York: McGraw-Hill.

    Google Scholar 

  • Stevens, J. (1996). Applied multivariate statistics for the social sciences. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

  • Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston: Pearson Education, Inc./Allyn and Bacon.

  • Zitt, M., & Filliatreau, G. (2007). The world class universities and ranking: Aiming beyond status (pp. 141–160), Romania: UNESCO-CEPES, Cluj University Press, chap Big is (made) beautiful: Some comments about the Shangai ranking of world-class universities, Part Two.

Download references

Acknowledgments

The author would like to thank the financial support from Xunta de Galicia through the IMAN Program. The author would also thank the anonymous reviewer for the insightful comments and suggestions which helped to improve the manuscript.

Author information

Authors and Affiliations

  1. Universidad de Vigo, ETSE TELECOMUNICACION, Campus Universitario, 36310, Vigo, Spain

    Domingo Docampo

Authors
  1. Domingo Docampo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Domingo Docampo.

Additional information

An erratum to this article can be found at http://dx.doi.org/10.1007/s11192-010-0315-4

Rights and permissions

Reprints and permissions

About this article

Cite this article

Docampo, D. On using the Shanghai ranking to assess the research performance of university systems. Scientometrics 86, 77–92 (2011). https://doi.org/10.1007/s11192-010-0280-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-010-0280-y

Keywords

Search

Navigation