Top

Educational Assessment, Evaluation and Accountability

Published in:

08-02-2021

Making sense out of measurement non-invariance: how to explore differences among educational systems in international large-scale assessments

Authors: Edwin Cuellar, Ivailo Partchev, Robert Zwitser, Timo Bechger

Published in: Educational Assessment, Evaluation and Accountability | Issue 1/2021

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

International large-scale assessment in education aims to compare educational achievement across many countries. Differences between countries in language, culture, and education give rise to differential item functioning (DIF). For many decades, DIF has been regarded as a nuisance and a threat to validity. In this paper, we take a different stance and argue that DIF holds essential information about the differences between countries. To uncover this information, we explore the use of multivariate analysis techniques as ways to analyze DIF emphasizing visualization. PISA 2012 data are used for illustration.

previous article The integrity of educational outcome measures in international assessments

next article Re-reviewing item parameter equivalence in TIMSS 2015 from a sociocognitive perspective

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

Orthonormal means that \(\mathbf {U^{\prime }U}=\mathbf {V^{\prime }V}=\mathbf {I}\), implying that the columns of both U and V are mutually independent or perpendicular, and the sum-of-squares for each of their columns is 1.

This information is provided by the PISA technical report (see Appendix A, OECD2014)

Ackerman, T.A. (1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29(1), 67–91.CrossRef

Angoff, W., & Ford, S. (1973). Item-race interaction on a test of scholastic aptitude. Journal of Educational Measurement, 10(2), 95–105.CrossRef

Bechger, T., & Maris, G. (2015). A statistical test for differential item pair functioning. Psychometrika, 80(2), 317–340.CrossRef

Bechger, T., Hox, J., van den Wittenboer, G., & de Glopper, C. (1999). The validity of comparative educational studies. Educational Measurement: Issues and Practice, 18(3), 18–26.CrossRef

Behrisch, M., Bach, B., Henry Riche, N., Schreck, T., & Fekete, J. (2016). Matrix reordering methods for table and network visualization. Computer Graphics Forum, 35(3), 693–716. https://doi.org/10.1111/cgf.12935.CrossRef

Brazma, A., & Vilo, J. (2000). Gene expression data analysis. FEBS Letters, 480, 117–24.CrossRef

Brinkhuis, M.J., Bakker, M., & Maris, G. (2015). Filtering data for detecting differential development. Journal of Educational Measurement, 52(3), 319–338.CrossRef

Cadima, J., & Joliffe, I. (2009). On relationships between uncentred and column-centred principal component analysis. Pakistan Journal of Statistics, 25(4), 473–503.

Doebler, A. (2019). Looking at dif from a new perspective: A structure-based approach acknowledging inherent indefinability. Applied Psychological Measurement, 43(4), 303–321.CrossRef

Eckart, C., & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika 1211–218.

Everitt, B., Landau, S., Leese, M., & Stahl, D. (2011). Cluster analysis. Chichester: Wiley.CrossRef

Gabriel, K. (1971). The biplot graphical display of matrices with application to principal component analysis. Biometrika, 456–467.

Glas, C.A.W., & Verhelst, N.D. (1995). Testing the Rasch model. In Fischer, G. H., & Molenaar, I W (Eds.) Rasch models: Foundations, recent developments, and applications, chap 5 (pp. 69–95). New York: Springer.

Golub, G.H., & Van Loan, C.F. (1996). Matrix computations, 3rd edn. Johns Hopkins University Press.

Greenacre, M. (2010). Biplots in practice. Bilbao: BBVA Foundation. http://www.multivariatestatistics.org.

Hastie, T., Tibshirani, R., & Friedman, J. (2013). The elements of statistical learning: Data mining, inference and prediction. Springer Series in Statistics. New York: Springer.

Holland, P., & Wainer, H. (2012). Differential item functioning. Taylor & Francis. https://books.google.nl/books?id=6YAXJfswvfYC.

Jolliffe, I. (2002). Principal component analysis. Springer Series in Statistics, Springer, Berlin.

Kolde, R. (2019). pheatmap: Pretty Heatmaps. https://CRAN.R-project.org/package=pheatmap, r package version 1.0.12.

Koops, J., Bechger, T., & Maris, G. (in press). Research for practical issues and solutions in computerized multistage testing (chap 19). In von Davier, A., & Duanli, Y (Eds.) (pp. 201–216). London: Routledge.

Lele, S., & Richtsmeier, J. (2001). An invariant approach to statistical analysis of shapes. Chapman & Hall/CRC Interdisciplinary Statistics, CRC Press.

Lord, F. (1980). Applications of item response theory to practical testing problems. Hillsdale: Lawrence Erlbaum Associate.

Madeira, S., & Oliveira, A. (2004). Biclustering algorithms for biological data analysis: A survey. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 24–45.

Maris, G., Bechger, T., & San Martin, E. (2015). A Gibbs sampler for the (extended) marginal Rasch model. Psychometrika, 80(4), 859–879.CrossRef

Maris, G., Bechger, T., Koops, J., & Partchev, I. (2019). dexter: Data management and analysis of tests. https://CRAN.R-project.org/package=dexter, r package version 1.0.0.

Millsap, R. (2012). Statistical approaches to measurement invariance. Routledge.

OECD. (2014). Pisa 2012 technical report.

Oshima, T., & Miller, M.D. (1992). Multidimensionality and item bias in item response theory. Applied Psychological Measurement, 16(3), 237–248.CrossRef

Padilha, V., & Campello, R. (2017). A systematic comparative evaluation of biclustering techniques. BMC Bioinformatics, 18. https://doi.org/10.1186/s12859-017-1487-1.

R. Core Team. (2019). R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. https://www.R-project.org/.

San Martín, E., & Rolin, J. (2013). Identification of parametric Rasch-type models. Journal of Statistical Planning and Inference, 143(1), 116–130.CrossRef

Thompson, D.R., Huntley, M.A., & Suurtamm, C. (2017). International perspectives on mathematics curriculum. IAP.

Travers, K.J., & Westbury, I. (1989). The IEA study of mathematics I: Analysis of mathematics curricula. Pergamon Press.

Verhelst, N. (2012). Profile analysis: A closer look at the PISA 2000 reading data. Scandinavian Journal of Educational Research, 56(3), 315–332. https://doi.org/10.1080/00313831.2011.583937.CrossRef

Wang, T., Strobl, C., Zeileis, A., & Merkle, E. (2018). Score-based tests of differential item functioning via pairwise maximum likelihood estimation. Psychometrika, 83(1), 132–155.CrossRef

Zwitser, R., Glaser, S., & Maris, G. (2017). Monitoring countries in a changing world: A new look at DIF in international surveys. Psychometrika, 82 (1), 210–232.CrossRef

Title: Making sense out of measurement non-invariance: how to explore differences among educational systems in international large-scale assessments
Authors: Edwin Cuellar
Ivailo Partchev
Robert Zwitser
Timo Bechger
Publication date: 08-02-2021
Publisher: Springer Netherlands
Published in: Educational Assessment, Evaluation and Accountability / Issue 1/2021
Print ISSN: 1874-8597
Electronic ISSN: 1874-8600
DOI: https://doi.org/10.1007/s11092-021-09355-x

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 1/2021

Is Canada really an education superpower? The impact of non-participation on results from PISA 2015

Disentangling general achievement levels and subject-specific strengths and weaknesses in mathematics, reading, and science

Re-reviewing item parameter equivalence in TIMSS 2015 from a sociocognitive perspective

Early tracking and different types of inequalities in achievement: difference-in-differences evidence from 20 years of large-scale assessments

Measures of long-term trends in mathematics: linking large-scale assessments over 50 years

Assessing the comparability of teacher-related constructs in TIMSS 2015 across 46 education systems: an alignment optimization approach