Top

Educational Assessment, Evaluation and Accountability

Published in:

17-07-2020

Do student ratings provide reliable and valid information about teaching quality at the school level? Evaluating measures of science teaching in PISA 2015

Authors: Anindito Aditomo, Carmen Köhler

Published in: Educational Assessment, Evaluation and Accountability | Issue 3/2020

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Large-scale educational surveys, including PISA, often collect student ratings to assess teaching quality. Because of the sampling design in PISA, student ratings must be aggregated at the school level instead of the classroom level. To what extent does school-level aggregation of student ratings yield reliable and valid measures of teaching quality? We investigate this question for six scales measuring classroom management, emotional support, inquiry-based instruction, teacher-directed instruction, adaptive instruction, and feedback provided by PISA 2015. The sample consisted of 503,146 students from 17,678 schools in 69 countries/regions. Multilevel CFA and SEM were conducted for each scale in each country/region to evaluate school-level reliability (intraclass correlations 1 and 2), factorial validity, and predictive validity. In most countries/regions, school-level reliability was found to be adequate for the classroom management scale, but only low to moderate for the other scales. Examination of factorial and predictive validity indicated that the classroom management, emotional support, adaptive instruction, and teacher-directed instruction scales capture meaningful differences in teaching quality between schools. Meanwhile, the inquiry scale exhibited poor validity in almost all countries/regions. These findings suggest the possibility of using student ratings in PISA to investigate some aspects of school-level teaching quality in most countries/regions.

previous article Measuring teaching quality, designing tests, and transforming feedback targeting various education actors

next article The role of feedback acceptance and gaining awareness on teachers’ willingness to use inspection feedback

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

Aditomo, A. (2020). Science teaching practices in Indonesian secondary schools: a portrait of educational quality and equity based on PISA 2015. In A. Suryani, H. Masalam, & I. Tirtowalujo (Eds.), Preparing Indonesian youth: a review of educational research. Brill: Leiden.

Aditomo, A., & Klieme, E. (2020). Forms of inquiry-based science instruction and their relations with learning outcomes: evidence from high and low-performing education systems. International Journal of Science Education, 42(4), 504–525. https://doi.org/10.1080/09500693.2020.1716093.CrossRef

Aldrup, K., Klusmann, U., Lüdtke, O., Göllner, R., Trautwein, U., Aldrup, K., & Klusmann, U. (2018). Social support and classroom management are related to secondary students’ general school adjustment: a multilevel structural equation model using student and teacher ratings. Journal of Educational Psychology., 110, 1066–1083.CrossRef

Blazar, D. (2015). Effective teaching in elementary mathematics: identifying classroom practices that support student achievement. Economics of Education Review, 48, 16–29. https://doi.org/10.1016/j.econedurev.2015.05.005.CrossRef

Blazar, D., & Kraft, M. A. (2017). Teacher and teaching effects on students’ attitudes and behaviors. Educational Evaluation and Policy Analysis, 39(1), 146–170. https://doi.org/10.3102/0162373716670260.CrossRef

Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). How people learn: brain, mind, experience, and school. Washington DC: National Academies Press.

Breakspear, S. (2012). The policy impact of PISA: an exploration of the normative effects of international benchmarking in school system performance. OECD Education Working Papers, No. 71. OECD Publishing (NJ1), (71). Retrieved from http://eric.ed.gov/?id=ED530643.

Brown, T. A. (2015). Confirmatory factor analysis for applied research second edition (2nd ed.). New York: The Guilford Press.

Carroll, J. B. (1989). The Carroll Model: a 25-year retrospective and prospective view. Educational Researcher, 18(1), 26–31.CrossRef

Cooley, W., & Leinhardt, G. (1980). The instructional dimensions study. Educational Evaluation and Policy Analysis, 2, 7–25.CrossRef

Creemers, B. P. M. (1994). The effective classroom. New York: Cassell.

Deci, E. L., Ryan, R. M., Vallerand, R. J., & Pelletier, L. G. (1991). Motivation and education: the self-determination perspective. Educational Psychologist, 26(3–4), 325–346. https://doi.org/10.1080/00461520.1991.9653137.CrossRef

Deci, E. L., Ryan, R. M., & Williams, G. C. (1996). Need satisfaction and the self-regulation of learning. Learning and Individual Differences, 8(3), 165–183. https://doi.org/10.1016/S1041-6080(96)90013-8.CrossRef

Decristan, J., Kunter, M., Fauth, B., Büttner, G., Hardy, I., & Hertel, S. (2016). What role does instructional quality play for elementary school children’s science competence?: a focus on students at risk/Zur Bedeutung von Unterrichtsqualität für die naturwissenschaftliche Kompetenz von Grundschulkindern. Journal for Educational Research Online, 8(1), 66–89.

Derry, S. J. (1996). Cognitive schema theory in the constructivist debate. Educational Psychologist, 31(3–4), 163–174. https://doi.org/10.1080/00461520.1996.9653264.CrossRef

Egeberg, H. M., Mcconney, A., & Price, A. (2016). Classroom management and national professional standards for teachers: a review of the literature on theory and practice. Australian Journal of Teacher Education, 41(7), 1–18.CrossRef

Emmer, E. T., & Stough, L. M. (2001). Classroom management: a critical part of educational psychology, with implications for teacher education. Educational Psychologist, 36(2), 103–112. https://doi.org/10.1207/S15326985EP3602.CrossRef

Fauth, B., Decristan, J., Rieser, S., Klieme, E., & Büttner, G. (2014). Student ratings of teaching quality in primary school: dimensions and prediction of student outcomes. Learning and Instruction, 29, 1–9. https://doi.org/10.1016/j.learninstruc.2013.07.001.CrossRef

Fischer, J., Praetorius, A. K., & Klieme, E. (2019). The impact of linguistic similarity on cross-cultural comparability of students ’ perceptions of teaching quality. Educational Assessment, Evaluation and Accountability, 31, 201–220. https://doi.org/10.1007/s11092-019-09295-7.CrossRef

Gee, K. A., & Wong, K. K. (2012). A cross national examination of inquiry and its relationship to student performance in science: evidence from the Program for International Student Assessment (PISA) 2006. International Journal of Educational Research, 53, 303–318. https://doi.org/10.1016/j.ijer.2012.04.004.CrossRef

Gershenson, S. (2016). Linking teacher quality, student attendance, and student achievement. Education Finance and Policy, 11, 125–149.CrossRef

Goldstein, H. (1997). Methods in school effectiveness research. School Effectiveness and School Improvement, 8, 369–395.CrossRef

Grek, S. (2009). Governing by numbers: The PISA “effect” in Europe. Journal of Education Policy, 24(1), 23–37. https://doi.org/10.1080/02680930802412669.CrossRef

He, J., & van de Vijver, F. J. R. (2013). Methodological issues in cross-cultural studies in educational psychology. In G. A. D. Liem & A. B. I. Bernardo (Eds.), Advancing cross-cultural perspectives on educational psychology (pp. 39–56). Charlotte, NC: Information Age Publishing.

Hox, J. J. (2010). 从汶川地震到芦山地震. In Multilevel analysis: techniques and applications (2nd ed.). New York: Routledge. https://doi.org/10.1360/zd-2013-43-6-1064.CrossRef

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal1, 6(1), 1–55. https://doi.org/10.1080/10705519909540118.CrossRef

Hutchinson, D., & Healy, M. (2001). The effect on variance component estimates of ignoring a level in a multilevel model. Multilevel Modeling Newsletter, 13(2), 4–5.

Hwang, J., Choi, K. M., Bae, Y., & Shin, D. H. (2018). Do teachers’ instructional practices moderate equity in mathematical and scientific literacy?: an investigation of the PISA 2012 and 2015. International Journal of Science and Mathematics Education, 16, 25–45. https://doi.org/10.1007/s10763-018-9909-8.CrossRef

Jackson C. K. (2012). Non-cognitive ability, test scores, and teacher quality: evidence from 9th grade teachers in North Carolina (NBER Working Paper No. 18624). Cambridge, MA: National Bureau for Economic Research.

James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Estimating Within-Group Interrater Reliability with and without Response Bias, 69(1), 85–98.

Jennings, J. L., & Di Prete, T. A. (2010). Teacher effects on social and behavioral skills in early elementary school. Sociology of Education, 83, 135–159.CrossRef

Jiang, F., & McComas, W. F. (2015). The effects of inquiry teaching on student science achievement and attitudes: evidence from propensity score analysis of PISA data. International Journal of Science Education, 37(3), 554–576. https://doi.org/10.1080/09500693.2014.1000426.CrossRef

Kelloway, E. K. (2015). Using Mplus for structural equation modeling (2nd ed.). Thousand Oaks: Sage Publications.

Klein, K. J., Bliese, P. D., Kozlowski, S. W. J., Dansereau, F., Gavin, M. B., Griffin, M. A., … Bligh, M. C. (2000). Multilevel analytical techniques: commonalities, differences, and continuing questions. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations, and methods in organizations: foundations, extensions, and new directions (pp. 512–553). San Francisco: Jossey-Bass.

Klieme, E. (2013). The role of large-scale assessments in research on educational effectiveness and school development. In M. von Davier, E. Gonzalez, I. Kirsch, & K. Yamamoto (Eds.), The role of international large-scale assessments: perspectives from technology, economy, and educational research (pp. 115–148). Dordrecht: Springer.CrossRef

Klieme, E., Pauli, C., & Reusser, K. (2009). The Pythagoras study: investigating effects of teaching and learning in Swiss and German mathematics classrooms (pp. 137–160). Munster: Waxmann Verlag.

Kraft, M. A., & Grace, S. (2016). Teaching for tomorrow’s economy? Teacher effects on complex cognitive skills and social-emotional competencies (working paper). Providence, RI: Brown University Retrieved from http://scholar.harvard.edu/files/mkraft/files/teaching_for_tomorrows_economy_-_final_public.pdf.

Kunter, M., & Baumert, J. (2006). Who is the expert? Construct and criteria validity of student and teacher ratings of instruction. Learning Environments Research, 9(3), 231–251. https://doi.org/10.1007/s10984-006-9015-7.CrossRef

Kyriakides, L., Campbell, R. J., & Gagatsis, A. (2000). The significance of the classroom effect in primary schools: an application of Creemers’ comprehensive model of educational effectiveness. School Effectiveness and School Improvement, 11, 501–529.CrossRef

Lau, K., & Lam, T. Y. (2017). Instructional practices and science performance of 10 top-performing regions in PISA 2015. International Journal of Science Education, 39, 1–22. https://doi.org/10.1080/09500693.2017.1387947.CrossRef

Lazonder, A. W., & Harmsen, R. (2016). Meta-analysis of inquiry-based learning: effects of guidance. Review of Educational Research, 86(3), 681–718. https://doi.org/10.3102/0034654315627366.CrossRef

Lebreton, J. M., & Senter, J. L. (2008). Answers to 20 questions and interrater agreement. Organizational Research Methods, 11(4), 815–852. https://doi.org/10.1177/1094428106296642.CrossRef

Lüdtke, O., Trautwein, U., Kunter, M., & Baumert, J. (2006). Reliability and agreement of student ratings of the classroom environment: a reanalysis of TIMSS data. Learning Environments Research, 9(3), 215–230. https://doi.org/10.1007/s10984-006-9014-8.CrossRef

Lüdtke, O., Robitzsch, A., Trautwein, U., & Kunter, M. (2009). Assessing the impact of learning environments: how to use student ratings of classroom or school characteristics in multilevel modeling. Contemporary Educational Psychology, 34(2), 120–131. https://doi.org/10.1016/j.cedpsych.2008.12.001.CrossRef

Lüdtke, O., Marsh, H. W., Robitzsch, A., & Trautwein, U. (2011). A 2 x 2 taxonomy of multilevel latent contextual models: accuracy-bias trade-offs in full and partial error correction models. Psychological Methods, 16(4), 444–467. https://doi.org/10.1037/a0024376.CrossRef

Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: the critical issues of validity, bias, and utility. American Psychologist, 52(11), 1187–1197.CrossRef

Marsh, H. W., Hau, K-T., & Wen, Z. (2004). In Search of Golden Rules: Comment on Hypothesis-Testing Approaches to Setting Cutoff Values for Fit Indexes and Dangers in Overgeneralizing Hu and Bentler's (1999) Findings. Structural Equation Modeling: A Multidisciplinary Journal, 11(3), 320–341. https://doi.org/10.1207/s15328007sem1103_2.

Martínez, J.F. (2012). Consequences of omitting the classroom in multilevel models of schooling: an illustration using opportunity to learn and reading achievement. School Effectiveness and School Improvement, 23(13), 305–326.

McConney, A., Oliver, M. C., Conney, A. W.-M., Schibeci, R., & Maor, D. (2014). Inquiry, engagement, and literacy in science: a retrospective, cross-national analysis using PISA 2006. Science Education, 98, 963–980. https://doi.org/10.1002/sce.21135.CrossRef

Miller, A. D., & Murdock, T. B. (2007). Modeling latent true scores to determine the utility of aggregate student perceptions as classroom indicators in HLM: the case of classroom goal structures. Contemporary Educational Psychology, 32, 83–104. https://doi.org/10.1016/j.cedpsych.2006.10.006.CrossRef

Moerbeek, M. (2004). The consequence of ignoring a level of nesting in multilevel analysis. Multivariate Behavioral Research, 39, 127–149.CrossRef

Morin, A. J. S., Marsh, H. W., Nagengast, B., & Scalas, L. F. (2013). Doubly latent multilevel analyses of classroom climate: an illustration. Journal of Experimental Education., 82, 143–167. https://doi.org/10.1080/00220973.2013.769412.CrossRef

Müller, K., Prenzel, M., Seidel, T., Schiepe-tiska, A., & Kjærnsli, M. (2016). Science teaching and learning in schools: theoretical and empirical foundations for investigating classroom-level processes. In S. Kuger, E. Klieme, N. Jude, & D. Kaplan (Eds.), Assessing contexts of learning: an international perspective (pp. 423–446). Switzerland: Springer International. https://doi.org/10.1007/978-3-319-45357-6.CrossRef

Muthe’n, B. (1991). Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement, 28, 338–354.CrossRef

Muthén, L. K., & Muthén, B. O. (2017). Mplus user’s guide (8th ed.). Los Angeles: Muthén & Muthén.

OECD. (2016). PISA 2015 assessment and analytical framework: science, reading, mathematic and financial literacy. Paris: OECD Publishing. https://doi.org/10.1787/9789264255425-en.CrossRef

OECD. (2017). PISA 2015 technical report. Paris: OECD Publishing.

Perry, J. L., Nicholls, A. R., Clough, P. J., & Crust, L. (2015). Assessing model fit: caveats and recommendations for confirmatory factor analysis and exploratory structural equation modeling. Measurement in Physical Education and Exercise Science, 19(1), 12–21, https://doi.org/10.1080/1091367X.2014.952370.

Praetorius, A. K., Klieme, E., Herbert, B., & Pinger, P. (2018). Generic dimensions of teaching quality: the German framework of three basic dimensions. ZDM, 50(3), 407–426. https://doi.org/10.1007/s11858-018-0918-4.CrossRef

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: applications and data analysis methods. Thousand Oaks: Sage.

Raudensbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: applications and data analysis methods (2nd ed.). Thousand Oaks: Sage.

Rjosk, C., Richter, D., Hochweber, J., Lüdtke, O., Klieme, E., & Stanat, P. (2014). Socioeconomic and language minority classroom composition and individual reading achievement: the mediating role of instructional quality. Learning and Instruction, 32, 63–72. https://doi.org/10.1016/j.learninstruc.2014.01.007.CrossRef

Rudasill, K. M., Snyder, K. E., Levinson, H., & Adelson, J. L. (2018). Systems view of school climate: a theoretical framework for research. Educational Psychology Review, 30(35), 35–60. https://doi.org/10.1007/s10648-017-9401-y.CrossRef

Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78. https://doi.org/10.1037/0003-066X.55.1.68.CrossRef

Scheerens, J. & Creemers, BPM. (1989). Conceptualizing school effectiveness. International Journal of Educational Research, 13(7), 691–706.

Schiepe-tiska, A. (2019). School tracks as differential learning environments moderate the relationship between teaching quality and multidimensional learning goals in mathematics. Frontiers in Education, 4(January). https://doi.org/10.3389/feduc.2019.00004.

Schweig, J. (2014). Cross-level measurement invariance in school and classroom environment surveys: implications for policy and practice. Educational Evaluation and Policy Analysis, 36(3), 259–280. https://doi.org/10.3102/0162373713509880.CrossRef

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.CrossRef

Slater, H., Davies, N. M., & Burgess, S. (2012). Do Teachers Matter? Measuring the Variation in Teacher Effectiveness in England. Oxford Bulletin of Economics and Statistics, 75(4), 629–645. https://doi.org/10.1111/j.1468-0084.2011.00666.x.

Stapleton, L. M., & Hancock, G. R. (2016). Construct meaning in multilevel settings. Journal of Educational and Behavioral Statistics, 41(5), 481–520. https://doi.org/10.3102/1076998616646200.CrossRef

Stapleton, L. M., Yang, J. S., & Hancock, G. R. (2016). Construct meaning in multilevel settings. Journal of Educational and Behavioral Statistics, 41(5), 481–520. https://doi.org/10.3102/1076998616646200.CrossRef

Wagner, W., Göllner, R., Helmke, A., Trautwein, U., & Lüdtke, O. (2013). Construct validity of student perceptions of instructional quality is high, but not perfect: dimensionality and generalizability of domain-independent assessments. Learning and Instruction, 28, 1–11. https://doi.org/10.1016/j.learninstruc.2013.03.003.CrossRef

Wagner, W., Göllner, R., Werth, S., Voss, T., Schmitz, B., & Trautwein, U. (2016). Student and teacher ratings of instructional quality: consistency of ratings over time, agreement, and predictive power. Journal of Educational Psychology, 108(5), 705–721.CrossRef

Wallace, T. L., Kelcey, B., & Ruzek, E. (2016). What can student perception surveys tell us about teaching? Empirically testing the underlying structure of the tripod student perception survey. American Educational Research Journal, XX(X), 1–35. https://doi.org/10.3102/0002831216671864.CrossRef

Wang, M., & Degol, J. L. (2016). School climate: a review of the construct, measurement, and impact on student outcomes. Educational Psychology Review, 38, 315–352. https://doi.org/10.1007/s10648-015-9319-1.CrossRef

Wenger, M., Lüdtke, O., & Brunner, M. (2018). Übereinstimmung, Variabilität und Reliabilität von Schülerurteilen zur Unterrichtsqualität auf Schulebene: Ergebnisse aus 81 Ländern. Zeitschrift Fur Erziehungswiss, 21, 929–950. https://doi.org/10.1007/s11618-018-0813-3.CrossRef

Wittrock, M. C. (2010). Learning as a generative process. Educational Psychologist, 45(1), 40–45. https://doi.org/10.1080/00461520903433554.CrossRef

Zyphur, M. J., Kaplan, S. A., & Christian, M. S. (2008). Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: problems and solutions. Group Dynamics: Theory, Research, and Practice, 12(2), 127–140. https://doi.org/10.1037/1089-2699.12.2.127.CrossRef

Title: Do student ratings provide reliable and valid information about teaching quality at the school level? Evaluating measures of science teaching in PISA 2015
Authors: Anindito Aditomo
Carmen Köhler
Publication date: 17-07-2020
Publisher: Springer Netherlands
Published in: Educational Assessment, Evaluation and Accountability / Issue 3/2020
Print ISSN: 1874-8597
Electronic ISSN: 1874-8600
DOI: https://doi.org/10.1007/s11092-020-09328-6

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 3/2020

EFL reading test on mobile versus on paper: a study from metacognitive strategy use to test-media impacts

A model of interplay between student English achievement and the joint affective factors in a high-stakes test change context: model construction and validity

Correction to: Teachers responding to cultural diversity: case studies on assessment practices, challenges and experiences in secondary schools in Austria, Ireland, Norway and Turkey

The role of feedback acceptance and gaining awareness on teachers’ willingness to use inspection feedback

Teachers responding to cultural diversity: case studies on assessment practices, challenges and experiences in secondary schools in Austria, Ireland, Norway and Turkey

Measuring teaching quality, designing tests, and transforming feedback targeting various education actors