Abstract
To provide users insight into the value and limits of world university rankings, a comparative analysis is conducted of five ranking systems: ARWU, Leiden, THE, QS and U-Multirank. It links these systems with one another at the level of individual institutions, and analyses the overlap in institutional coverage, geographical coverage, how indicators are calculated from raw data, the skewness of indicator distributions, and statistical correlations between indicators. Four secondary analyses are presented investigating national academic systems and selected pairs of indicators. It is argued that current systems are still one-dimensional in the sense that they provide finalized, seemingly unrelated indicator values rather than offering a dataset and tools to observe patterns in multi-faceted data. By systematically comparing different systems, more insight is provided into how their institutional coverage, rating methods, the selection of indicators and their normalizations influence the ranking positions of given institutions.
This is a preview of subscription content, log in via an institution to check access.
(Data source: THE ranking 2016)
Similar content being viewed by others
References
AUBR. (2010). Assessment of University-Based Research Expert Group (AUBR). Assessing Europe’s University-Based Research, K1-NA-24187-EN-N, European Commission, Brussels (pp. 151). http://ec.europa.eu/research/era/docs/en/areas-of-actions-universities-assessing-europeuniversity-based-research-2010-en.pdf.
Calero-Medina, C., López-Illescas, C., Visser, M. S., & Moed, H. F. (2008). Important factors in the interpretation of bibliometric rankings of world universities. Research Evaluation, 17, 71–81.
EU Council. (2007). Council of the European Union. Council resolution on modernising universities for Europe’s competitiveness in a global knowledge economy. 16096/1/07 REV 1. http://www.consilium.europa.eu/ueDocs/cms_Data/docs/pressData/en/intm/97237.pdf.
EU Council. (2000). Lisbon European Council 23–23 March 2000. Presidency Conclusions. http://www.europarl.europa.eu/summits/lis1_en.htm.
Gringas, Y. (2014). How to boost your university up the rankings. University World News. 18 July 2014 Issue No: 329. http://www.universityworldnews.com/article.php?story=20140715142345754.
Hazelkorn, E. (2011). Rankings and the reshaping of higher education. the battle for world-class excellence. London: Palgrave.
Leiden Indicators. (n.d.). Indicators. http://www.leidenranking.com/information/indicators. Accessed 1 Nov 2016.
Paruolo, P., Saisana, M., & Saltelli, A. (2013). Ratings and rankings: Voodoo or science? Journal of the Royal Statistical Society Series A: Statistics in Society, 176(3), 609–634. doi:10.1111/j.1467-985X.2012.01059.x.
QS Normalization. (n.d.). Faculty Area Normalization. Technical Explanation. http://content.qs.com/qsiu/Faculty_Area_Normalization_-_Technical_Explanation.pdf. Accessed 1 Nov 2016.
QS Methodology. (n.d.). http://www.topuniversities.com/university-rankings-articles/world-university-rankings/qs-world-university-rankings-methodology. Accessed 1 Nov 2016.
Rauhvargers, A. (2011). Global rankings and their impact. EUA report on rankings 2011. Brussels: The European University Association.
Rauhvargers, A. (n.d.). Rankings criteria and their impact upon universities. http://www.unica-network.eu/sites/default/files/Rauhvargers_UNICA_IRO.pdf. Accessed 1 Nov 2016.
Salmi, J. (2009). The challenge of World class universities, World Bank, Washington, pp. 136. Retrieved 12 Jan. 2011 from: http://siteresources.worldbank.org/EDUCATION/Resources/278200-1099079877269/547664-1099079956815/547670-1237305262556/WCU.pdf.
Shin, J. C., Toutkoushian, R. K., & Teichler, U. (2011). University rankings. Theoretical basis, methodology and impacts on global higher education. Berlin: Springer. ISBN 978-94-007-1116-7.
Soh, K. (2013). Misleading university rankings: Cause and cure for discrepancies between nominal and attained weights. Journal of Higher Education Policy and Management, 35, 206–214. doi:10.1080/1360080X.2013.775929.
Soh, K. (2015a). What the overall doesn’t tell about world university rankings: Examples from ARWU, QSWUR, and THEWUR in 2013. Journal of Higher Education Policy and Management, 37, 295–307. doi:10.1080/1360080X.2015.1035523.
Soh, K. (2015b). Multicolinearity and indicator redundancy problem in world university rankings: An example using times higher education world university ranking 2013–2014 data. Higher Education Quarterly, 69, 158–174. doi:10.1111/hequ.12058. (Cited 1 time).
THE Ranking Methodology. (n.d.). https://www.timeshighereducation.com/news/ranking-methodology-2016. Accessed 1 Nov 2016.
Van Raan, A. F. J. (2005). Fatal attraction. Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62, 133–143.
Acknowledgements
The author wishes to thank two referees for their useful comments on an earlier version of this paper. The author is also grateful to the members of the Nucleo di Valutazione of the Sapienza University of Rome for stimulating discussions about the interpretation and the policy significance of world university rankings.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
See Table 10.
Rights and permissions
About this article
Cite this article
Moed, H.F. A critical comparative analysis of five world university rankings. Scientometrics 110, 967–990 (2017). https://doi.org/10.1007/s11192-016-2212-y
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-016-2212-y