Abstract
The Academic Ranking of World Universities (ARWU) published by researchers at Shanghai Jiao Tong University has become a major source of information for university administrators, country officials, students and the public at large. Recent discoveries regarding its internal dynamics allow the inversion of published ARWU indicator scores to reconstruct raw scores for 500 world class universities. This paper explores raw scores in the ARWU and in other contests to contrast the dynamics of rank-driven and score-driven tables, and to explain why the ARWU ranking is a score-driven procedure. We show that the ARWU indicators constitute sub-scales of a single factor accounting for research performance, and provide an account of the system of gains and non-linearities used by ARWU. The paper discusses the non-linearities selected by ARWU, concluding that they are designed to represent the regressive character of indicators measuring research performance. We propose that the utility and usability of the ARWU could be greatly improved by replacing the unwanted dynamical effects of the annual re-scaling based on raw scores of the best performers.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aghion, P., Dewatripont, M., Hoxby, C., Mas-Colell, A., & Sapir, A. (2008). Higher aspirations: An agenda for reforming european universities: Bruegel blueprint V, downloaded from aei.pitt.edu on March 2nd 2013. http://aei.pitt.edu/id/eprint/8714.
Aghion, P., Dewatripont, M., Hoxby, C., Mas-Colell, A., & Sapir, A. (2010). The governance and performance of universities: Evidence from Europe and the US. Economic Policy, pp 7–59.
Arrow, K. (1963). Social choice and individual values. New York: Wiley.
Bartlett, M. S. (1954). A note on the multiplying factors for various Chi-square approximations. Journal of the Royal Statistical Society, 16(B), 296–98.
Billaut, J. C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shangai ranking: An MCDM view. Scientometrics, 84(1), 237–263.
Butler, D. (2007). Academics strike back at spurious rankings: Universities seek reform of ratings. News; Nature 447, 514–515.
Butler, D. (2010). University rankings smarten up. Nature Special Report; Nature 464, pp 16–17.
Cliff, N., & Hamburger, C. (1967). The study of sampling errors in factor analysis by means of artificial experiments. Psychological Bulletin, 68, 430–435.
Comray, A. L., & Lee, H. (1992). A First Course in Factor Analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
Dasgupta, P., & Weale, M. (1992). On measuring the quality of life. World Development, 20(1), 119–131.
Dehon, C., McCathie, A., & Verardi, V. (2010). Uncovering excellence in academic rankings: a closer look at the Shanghai ranking. Scientometrics, 83(2), 515–524.
Docampo, D. (2011). On using the Shanghai ranking to assess the research performance of university systems. Scientometrics, 86(1), 77–92.
Docampo, D. (2013). Reproducibility of the Shanghai academic ranking of world universities results. Scientometrics, 94(2), 515–524.
Geese, R. (2004). Is a revision of the international scoring tables overdue?. New Studies in Athletics, 19(3), 9–19.
Grammaticos, B. (2007). The physical basis of scoring athletic performance. New Studies in Athletics, 22(3), 47–53.
Guadagnoli, E., & Vellicer, W. (1988). Relation of sample size to the stability of component patterns. Psychological Bulletin, 103, 265–275.
Hammond, T. (2007). Rank injustice?: How the scoring method for cross-country running competitions violates major social principles. Public Choice, 133, 359–375.
Harder, D. (2001). Apples to oranges. Princeton: Education Plus.
Hasktian, A., Rogers, W., & Catell, R. B. (1982). The behavior of numbers factors rules with simulated data. Multivariate Behavioural Research, 17, 193–219.
Hazelkorn, E. (2008). Learning to live with league tables and ranking: The experience of institutional leaders. Higher Education Policy, 21, 193–215.
IAAF. (2004). International association of athletics federations scoring tables for combined events: Downloaded from the I A A F server on march 14th 2013. URL:http://www.iaaf.org.
Ioannidis, J. P. A., Nikolaos, A., Patsopoulos, N., Kavvoura, F., Tatsioni, A., Evangelou, E., & Kouri, A., Contopoulos-Ioannidis, D. G., & Liberopoulos, G. (2007). International ranking systems for universities and institutions: A critical appraisal. BMC Medicine, 5: 30.
Kaiser, H. (1974). An index of factorial simplicity. Psychometrika, 39, 31–36.
Liu, N. C., & Cheng, Y. (2005). Academic ranking of world universities: Methodologies and problems. Higher Education in Europe, 30(2), 127–136.
London. (2012). Official results in 100 m (men): Downloaded from the Official London 2012 website server on March 22st 2013. URL http://www.london2012.com/athletics/event/men-100m/index.html.
Marginson, M., & van der Wende, M. (2007). To rank or to be ranked: The impact of global rankings in higher education. Journal of Studies in International Education, 11(3/4), 306–329.
Morrison, D. (2000). Multivariate Statistical Methods (3rd ed.). New York: McGraw-Hill.
Myerson, R., & Weber, R. (1993). A theory of voting equilibria. American Political Science Review, 87, 102–114.
Saari, D. (2001). Decisions and elections: explaining the unexpected. New York: Cambridge University Press.
Sass, D. (2010). Factor loading estimation error and stability using exploratory factor analysis. Educational and Psychological Measurement, 70:4, 557–577.
Sawyer, K., Sankey, H., & Lombardo, R. (2013). Measurability invariance, continuity and a portfolio representation. Measurement, 46, 89–96.
Stevens, J. (1996). Applied Multivariate Statistics for the Social Sciences. Mahwah, NJ: Lawrence Erlbaum Associates.
Tabachnick, B. G., & Fidell, L. S. (2007). Using Multivariate Statistics (5th ed.). Boston: Pearson Education, Inc. / Allyn and Bacon.
Thomson-Reuters. (2012). Methodology for identifying highly-cited researchers: Downloaded from the Thomson Reuters highlycited.com server on March 20th 2013. URL http://www.highlycited.com/methodology/.
Trkal, V. (2003). The development of combined event scoring tables and implications for the training of athletes. New Studies in Athletics, 18(4), 7–12.
Van Damme, R., Wilson, R., Vanhooydonk, B., & Aerts, P. (2002). Performance constraints in decathletes. Nature, 415, 755–756.
Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C. M., Tijssen, R. J. W., van Eck, N. J., van Leeuwen, T. N., van Raan, A. F. J., Visser, M. S., & Wouters, P. (2012). The Leiden ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology, 63(12), 2419–2432.
WBI. (2012). Knowledge Assessment Methodology: Downloaded from the World Bank server on March 7th 2013. URL http://www.worldbank.org/kam.
Westera, W. (2006). Decathlon; Towards a balanced and sustainable performance assessment method. New Studies in Athletics, 211, 39–51.
Zientek, L., & Thompson, B. (2007). Applying the bootstrap to the multivariate case: Bootstrap component/factor analysis. Behavior Research Methods, 39(2), 318–325.
Zitt, M., & Filliatreau, G. (2007). The world class universities and ranking: Aiming beyond status, Romania: UNESCO-CEPES, Cluj University Press, chap Big is (made) beautiful: Some comments about the Shangai ranking of world-class universities, Part Two, 141–160.