Top

Educational Assessment, Evaluation and Accountability

Published in:

02-11-2020

Item position effects in listening but not in reading in the European Survey of Language Competences

Authors: Andrés Christiansen, Rianne Janssen

Published in: Educational Assessment, Evaluation and Accountability | Issue 1/2021

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

In contrast with the assumptions made in standard measurement models used in large-scale assessments, students’ performance may change during the test administration. This change can be modeled as a function of item position in case of a test booklet design with item-order manipulations. The present study used an explanatory item response theory (IRT) framework to analyze item position effects in the 2012 European Survey on Language Competences. Consistent item position effects were found for listening but not for reading. More specifically, for a large subset of items, item difficulty decreased along with item position, which is known as a practice effect. The effect was found across all tested languages, although the effect sizes varied across items, test levels, and countries.

previous article Re-reviewing item parameter equivalence in TIMSS 2015 from a sociocognitive perspective

next article Measures of long-term trends in mathematics: linking large-scale assessments over 50 years

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

Albano, A.D., Cai, L., Lease, E.M., McConnel, S.R. (2019). Computerized adaptive testing in early education: exploring the impact of item position effects on ability estimation. Journal of Educational Measurement, 56(2), 437–451.CrossRef

Bates, D., Mächler, M., Bolker, B., Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48.CrossRef

Bejar, I.I. (1985). Test seededness under number-right scoring: an analysis of the test of English as a foreign language (Report No. ETS-RR-85-11). Princeton: Educational Testing Service.

Bolt, D.M., Cohen, A.S., Wollack, J.A. (2002). Item parameter estimation under conditions of test speededness: application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39(4), 331–348.CrossRef

Cosgrove, J., Cartwright, F. (2014). Changes in achievement on PISA: the case of Ireland and implications for international assessment practice. Large-scale Assessments in Education, 2(1), 2.CrossRef

Davis, J., & Ferdous, A. (2005). Using item difficulty and item position to measure test fatigue. Montreal: Paper presentation at the Annual Meeting of the American Educational Research Association.CrossRef

Davis, M.H., & Johnsrude, I.S. (2003). Hierarchical processing in spoken language comprehension. The Journal of Neuroscience, 23(8), 3423–3431.CrossRef

De Boeck, P., Bakker, M., Zwitser, R., Nivard, M., Hofman, A., Tuerlinckx, F., Partchev, I. (2011). The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software, 39(12), 1–28.CrossRef

De Boeck, P., & Wilson, M. (2004). Explanatory item response models: a generalized linear and nonlinear approach. New York: Springer.CrossRef

Debeer, D., Buchholz, J., Hartig, J., Janssen, R. (2014). Student, school, and country differences in sustained test-taking effort in the 2009 PISA reading assessment. Journal of Educational and Behavioral Statistics, 39(6), 502–523.CrossRef

Debeer, D., & Janssen, R. (2013). Modeling item-position effects within an IRT framework. Journal of Educational Measurement, 50(2), 164–185.CrossRef

Erb, J., Henry, M.J., Eisner, F., Obleser, J. (2013). The brain dynamics of rapid perceptual adaptation to adverse listening conditions. The Journal of Neuroscience, 33(26), 10688–10697.CrossRef

European Commission. (2012). First European Survey on Language Competences: final report. Brussels: Author.

European Commission. (2012). First European Survey on Language Competences: technical report. Brussels: Author.

Goegebeur, Y., De Boeck, P., Wollack, J.A., Cohen, A.S. (2008). A speeded item response model with gradual process change. Psychometrika, 73 (1), 65–87.CrossRef

Guertin, W.H. (1954). The effect of instructions and item order on the arithmetic subtest of the Wechsler-Bellevue. Journal of Genetic Psychology, 85(1), 79–83.CrossRef

Hambleton, R.K., & Traub, R.E. (1974). The effects of item order in test performance and stress. Journal of Experimental Education, 43(1), 40–46.CrossRef

Hamilton, J.C., & Shuminsky, T.R. (1990). Self-awareness mediates the relationship between serial position and item reliability. Journal of Personality and Social Psychology, 59(6), 1301–1037.CrossRef

Hartig, J., & Buchholz, J. (2012). A multilevel item response model for item position effects and individual persistence. Psychological Test and Assessment Modeling, 54(4), 418–431.

Hohensinn, C., Kubinger, K.D., Reif, M., Holocher-Ertl, S., Khorramdel, L., Frebort, M. (2008). Examining item-position effects in large-scale assessment using the linear logistic test model. Psychology Science Quarterly, 50 (3), 391–402.

Hohensinn, C., Kubinger, K.D., Reif, M., Schleicher, E., Khorramdel, L. (2011). Analysing item position effects due to test booklet design within large-scale assessment. Educational Research and Evaluation, 17(6), 497–509.CrossRef

Jagodzinski, W., Kühnel, S.M., Schmidt, P. (1987). Is there a ‘Socratic effect’ in nonexperimental panel studies. Sociological Methods and Research, 15(3), 259–302.CrossRef

Kingston, N.M., & Dorans, N.J. (1982). The effect of the position of an item within a test on item response behaviour: an analysis based on item response theory. GRE Board Professional Report GREB No. 79-12 bp ETS Research Report.

Kingston, N.M., & Dorans, N.J. (1984). Item location effects and their implications for IRT equating and adaptive testing. Applied Psychological Measurement, 8(2), 147–154.CrossRef

Knowles, E.S. (1988). Item context effects on personality scales: measuring changes the measure. Journal of Personality and Social Psychology, 55(2), 312–320.CrossRef

Knowles, E.S., & Byers, B. (1996). Reliability shifts in measurement reactivity: driven by content engagement or self-engagement?. Journal of Personality and Social Psychology, 70(5), 1080–1090.CrossRef

McGuire, W.J. (1960). Context and serial-order effects in personality measurement: limits on the generality of measuring changes the measure. The Journal of Abnormal and Social Psychology, 60(3), 345–353.CrossRef

Meyers, J.L., Miller, G.E., Way, W.D. (2009). Item position and item difficulty change in an IRT-based common item equating design. Applied Measurement in Education, 22(1), 38–60.CrossRef

Nagy, G., Nagengast, B., Becker, M., Rose, N., Frey, A. (2018). Item position effects in a reading comprehension test: an IRT study of individual differences and individual correlates. Psychological Test and Assessment Modeling, 60(2), 165–187.

Nagy, G., Nagengast, B., Frey, A., Becker, M., Rose, N. (2018). A multilevel study of position effects in PISA achievement tests: student- and school-level predictors in the German tracked school system. Assessment in Education: Principles, Policy & Practice, 26(4), 422–443.

Oshima, T.C. (1994). The effect of speededness on parameter estimation in item response theory. Journal of Educational Measurement, 31(3), 200–219.CrossRef

R Core Team. (2019). R: a language and environment for statistical computing, R Foundation for Statistical Computing. Vienna.

Ren, X., Wang, T., Altmeyer, M., Schweizer, K. (2014). A learning-based account of fluid intelligence from the perspective of the position effect. Learning and Individual Differences, 31, 30–35.CrossRef

Schweizer, K., Schreiner, M., Gold, A. (2009). The confirmatory investigation of APM items with loadings as a function of the position and easiness of items: a two-dimensional model of APM. Psychology Science, 51(1), 47–64.

Schweizer, K., Troche, S.J., Rammsayer, T.H. (2011). On the special relationship between fluid and general intelligence: new evidence obtained by considering the position effect. Personality and Individual Differences, 50, 1249–1254.CrossRef

Steinberg, L. (1994). Context and serial-order effects in personality measurement: limits on the generality of measuring changes the measure. Journal of Personality and Social Psychology, 66(2), 341–349.CrossRef

Swinton, S.S., Wild, C.L., Wallmark, M.M. (1983). Investigation of practice effects on item types in the graduate record examinations aptitude test. Educational Testing Service, Princeton.

Verguts, T., & De Boeck, P. (2000). A Rasch model for detecting learning while solving an intelligence test. Applied Psychological Measurement, 24(2), 151–162.CrossRef

Whitely, S.E., & Dawis, R.V. (1976). The influence of test context on item difficulty. Educational and Psychological Measurement, 36, 329–337. CrossRef

Wild, C.J., Yusuf, A., Wilson, D.E., Peelle, J.E., Davis, M.H., Johnsrude, I.S. (2012). Effortful listening: the processing of degraded speech depends critically on attention. The Journal of Neuroscience, 32(40), 14010–14021.CrossRef

Wise, L.L. (1986). Latent trait models for partially speeded tests. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco.

Wu, Q., Debeer, D., Buchholz, J., Hartig, J., Janssen, R. (2019). Predictors of individual performance changes related to item positions in PISA assessments. Large-scale Assessments in Education, 7(5), 1–21.

Yamamoto, K., & Everson, H. (1997). Modeling the effects of test length and test time on parameter estimation using the hybrid model. In Rost, J., & Langeheine, R. (Eds.) Applications of latent trait and latent class models in the social sciences. New York: Waxmann.

Yen, W.M. (1980). The extent, causes and importance of context effects on item parameters for two latent trait models. Journal of Educational Measurement, 17(4), 297–311.CrossRef

Title: Item position effects in listening but not in reading in the European Survey of Language Competences
Authors: Andrés Christiansen
Rianne Janssen
Publication date: 02-11-2020
Publisher: Springer Netherlands
Published in: Educational Assessment, Evaluation and Accountability / Issue 1/2021
Print ISSN: 1874-8597
Electronic ISSN: 1874-8600
DOI: https://doi.org/10.1007/s11092-020-09335-7

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 1/2021

Re-reviewing item parameter equivalence in TIMSS 2015 from a sociocognitive perspective

Measures of long-term trends in mathematics: linking large-scale assessments over 50 years

Academic resilience: underlying norms and validity of definitions

Is Canada really an education superpower? The impact of non-participation on results from PISA 2015

Early tracking and different types of inequalities in achievement: difference-in-differences evidence from 20 years of large-scale assessments

The integrity of educational outcome measures in international assessments