Abstract
Multilevel SEM was used to examine the extent to which student, instructor, and course characteristics affect student ratings. Data were gathered from 1867 students enrolled in 117 courses at a large teacher training college in Israel. Four alternative two-level models that differ in only the nature of the relationship among interest in the course subject, expected grade, and student ratings were tested. Two of the models were judged as less appropriate, one because it failed to support the spurious relationship assumed between expected grade and student ratings, and the other on grounds of poor model-data fit. The other two models were equally good both in terms of the model-data fit and the amount of variance in student ratings that is accounted for by each of them. Both models supported the mediation effect of expected grade in the relationship between interest in the course subject and student ratings.
Similar content being viewed by others
References
Aleamoni, M. L. (November, 1978). Are there differences in perceived teaching effectiveness between males and females in anthropology? Paper Presented at the Annual Meeting of the American Anthropological Association, Los Angeles, California
Avi-Itzhak T., Kremer L. (1986) An investigation into the relationship between university faculty attitudes toward student rating and organizational and background factors. Educational Research Quarterly 10(2): 31–38
Bachen M. C., McLoughlin M. M., Garcia S. S. (1999) Assessing the role of gender in college students’ evaluations of faculty. Communication Education 48(3): 193–210
Baron R. M., Kenny D. A. (1986) The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology 51(6): 1173–1182
Basow S. A., Howe K. G. (1987) Evaluation of college professors: Effects of professors’ sex type and sex, and student’s sex. Psychological Reports 60(2): 671–678
Bonetti S. (1994) On the use of student questionnaires. Higher Education Review 26(3): 57–64
Centra J. A. (1979) Determining Faculty Effectiveness. Jossey-Bass, San Francisco
Centra J. A. (1993) Reflective Faculty Evaluation. Jossey-Bass, San Francisco
Centra A. J., and Gaubatz B. N. (2000) Is there gender bias in student evaluations of teaching? Journal of Higher Education 71(1): 17–33
Chau C.-T. (1997) A Bootstrap experiment on the statistical properties of students’ ratings of teaching effectiveness. Research in Higher Education 38(4): 497–517
Delaney, L. Jr. E. (April, 1976). The relationships of student ratings of instruction to student, instructor and course characteristics. Paper Presented at the Annual Meeting of the American Educational Research Association, San Francisco, California
Eiszler C. F. (2002) College student evaluations of teaching and grade inflation. Research in Higher Education 43(4): 483–501
Feldman K. A. (1976a) Grades and college students’ evaluation of their courses and teachers. Research in Higher Education 4(1): 69–111
Feldman K. A. (1976b) The superior college teacher from the student’s view. Research in Higher Education 5(3): 243–288
Feldman K. A. (1983) Seniority and experience of college teachers as related to evaluations they receive from students. Research in Higher Education 18(1): 3–124
Feldman K. A. (1984) Class size and college students’ evaluations of teachers and courses: A closer look. Research in Higher Education 21(1): 45–116
Feldman K. A. (1993) College students’ views of male and female college teachers: Part II–evidence from students’ evaluations of their classroom teachers. Research in Higher Education 33(3): 317–375
Fernandez J., Mateo A. A., Muniz J. (1998) Is there a relationship between class size and student ratings of teacher quality? Educational and Psychological Measurement 58(4): 596–604
Fourie C. M. (2000) Class size and student ratings of teaching presentation. South African Journal of Higher Education 14(3): 132–138
Goldberg G., Callahan J. (1991) Objectivity of student evaluations of instructors. Journal of Education for Business 66(6): 377–378
Greenwald A. G., Gillmore G. M. (1997) Grading leniency is a removable contaminant of student ratings. American Psychologist 52(11): 1209–1217
Hancock R. G., Shannon M. D., Trentham L. L. (1992) Student and teacher gender in ratings of university faculty: results from five colleges of study. Journal of Personnel Evaluation in Education 6(3): 235–248
Haugen E. R. (1984) Educationists and academics: Ratings and community college instructors. Community/Junior College Quarterly of Research and Practice 8(1–4): 103–113
Howard G. C., Schmeck R. R. (1979) Relationship of changes in student motivation to student evaluations of instruction. Research in Higher Education 10: 305–315
Hu L. T., Bentler P. M. (1999) Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling 6(1): 1–55
Jackson D. L., Teal C. R., Raines S. J., Nansel T. R., Force R. C., Burdsal C. A. (1999) The dimensions of students’ perceptions of teaching effectiveness. Educational and Psychological Measurement 59(4): 580–596
Jöreskog K. G. (1977) Structural equation modeling in the social sciences: Specification, estimation, and testing. In: Krishnaiah P. R. (eds.), Applications of Statistics, North-Holland, Amsterdam, pp. 265–287
Jöreskog K.G. (1993) Testing structural equation model. In: Bollen K. A., Scott Long J. (eds) Testing Structural Equation Models. Sage, Newbury Park, CA, pp. 294–316
Jöreskog K. G., Sörbom D. (1996) LISREL 8: User’s Reference Guide. Scientific Software International, Chicago, IL
Kaplan D., Elliot R. P. (1997) A didactic example of multilevel structural equation modeling applicable to study of organizations. Structural Equation Modeling 4(1): 1–4
Keller L.M. (1983) Motivational design instruction. In: Reigeluth C.M. (eds) Instructional Design Theories and Models: An Overview of Their Current Status. Erlbaum, Hillsdale, NJ
Koon J. (1986) Teaching evaluation: Theory and practice for a student position. National Issues in Higher Education 22: 265–274
Koon J., Murray H. G. (1995) Using multiple outcomes to validate student ratings of overall teacher effectiveness. Journal of Higher Education 66(1): 61–81
Kwan K.-P. (1999) How fair are student ratings in assessing the teaching performance of university teachers? Assessment & Evaluation in Higher Education 24(2): 181–195
Krapp A. (2002) Structural and dynamic aspects of interest development: Theoretical considerations from an ontogenetic perspective. Learning and Instruction 12(4): 383–409
Lin W. (1992) Is class size a bias to student evaluations of faculty? A review. Chinese University Education Journal 20(1): 49–53
Marsh H. W. (1983) Multidimensional ratings of teaching effectiveness by students from different academic settings and their relation to student/course/instructor characteristics. Journal of Educational Psychology 75(1): 150–166
Marsh H. W. (1984) Student’s evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and utility. Journal of Educational Psychology 76(5): 707–754
Marsh H. W. (1987) Students’ evaluations of university teaching: Research findings, methodological issues, and directions for future research [Entire issue]. International Journal of Educational Research 11(3): 263–388
Marsh H. W. (1989) Responses to review of “students’ evaluations of university teaching: Research findings, methodological issues, and directions for future research”. Instructional Evaluation 10: 5–9
Marsh H. W., Cooper L. T. (1981) Prior subject interest, students’ evaluations, and instructional effectiveness. Multivariate Behavioral Research 16(1): 83–104
Marsh H. W., Dunkin M. (1997) Student evaluation of university teaching: A multidimensional perspective. In: Perry P. R., Smart J. C. (eds) Effective Teaching in Higher Education: Research and Practice. Agathon, New York, pp. 241–320
Marsh H. W., Rochi L. A. (1997) Making students’ evaluations of teaching effectiveness effective. American Psychologist 52(11): 1187–1197
Marsh H. W., Rochi L. A. (2000) Effects of grading leniency and low workload on students’ evaluations of popular myth, bias, validity, or innocent bystanders? Journal of Educational Psychology 92(1): 202–228
Marsh H. W., Overall U. J., Kesler P. S. (1979) Validity of student evaluation of instructional effectiveness: A comparison of faculty self-evaluations and evaluation by their students. Journal of Educational Psychology 71(2): 149–160
Mason P. M., Steagall J. W., Fabritius M. M. (1995) Student evaluations of faculty: A new procedure for aggregate measures of performance. Economics of Education Review 14(4): 403–416
Mateo A., Fernandez J. (1996) Incidence of class size on the evaluation of university teaching quality. Educational and Psychological Measurement 56(5): 771–778
McKee G. B., Dowalibly J. F. (1985) The relationship between student, course and instructor characteristics and hearing impaired students’ ratings of instruction. Volta Review 87(2): 77–86
Menges R. J. (1973) The new reporters: Students rate instruction. New Directions for Higher Education 1(4): 59–76
Murphy P. K., Alexander A. P. (2002) What counts? The predictive powers of subject-matter knowledge, strategic processing, and interest in domain-specific performance. Journal of Experimental Education 70(3): 197–214
Murray G. H., Rushton J. P., Paunonen V. S. (1990) Teacher personality traits and student instructional ratings in six types of university courses. Journal of Educational Psychology 82(2): 250–261
Muthén B. (1989) Latent variable modeling in heterogeneous populations. Psychometrika 54(4): 557–585
Muthén, B. (1991) Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement 28(4): 338–354
Muthén, B. (1994). Multilevel covariance structure analysis. In: Hox, J., and Kreft, I. (eds.), Multilevel Modeling, a special issue of Sociological Methods & Research 22: 376–398
Muthén L. K., Muthén B. O. (1998). Mplus User’s Guide. Muthén& Muthén, Los Angeles, CA
Muthén, L. K., and Muthén, B. O. (2003). Mplus Version 2.13. Addendum to the Mplus User’s Guide. Available at: http://www.statmodel.com/version2.html
Muthén B., Sattora A. (1989) Multilevel aspects of varying parameters in structural models. In: Bock R. D. (eds) Multilevel Analysis of Educational Data. Academic, San Diego, CA, pp. 87–99
Nasser F., Fresko B. (2002) Faculty views of student evaluation of college teaching. Assessment & Evaluation in Higher Education 27(2): 187–198
Olivares J. O. (2002) A conceptual and analytical critique of student ratings of teachers in the USA with implications for teacher effectiveness and student teaching. Teaching in Higher Education 8(2): 233–245
Petchers K. M., Chow C. J. (1988) Sources of variation in students’ evaluations of instruction in a graduate social work Program. Journal of Social Work Education 24(1): 35–42
Rabalais, J. M. (1977, July). The relationship between number of graduate semester hours, years of teaching experience and student evaluation of overall rating of instructors for a selected sample of junior college faculty. Paper Presented at the Annual Conference of the Southeast Region AERA Special Interest Group in Community College Research, New Orleans
Roundenbush S. W., Willms J. D. (eds) (1991) Schools, Classrooms, and Pupils: International Studies of Schooling from a Multilevel Perspective. Academic Press, Orlando, Fl
Sailor P., Worthen B., Shin E. H. (1997) Class level as a possible mediator of the relationship between grades and student ratings of teaching. Assessment & Evaluation in Higher Education 22(3): 261–269
Schmelkin-Padhazur L., Spencer K. J., Gellman E. S. (1997) Faculty perspectives on course and teacher evaluation. Research in Higher Education 38(5): 575–592
Stapleton J. R., Murkison G. (2001) Optimizing the fairness of student evaluations: A study of correlations between instructor excellence, study production, learning production, and expected grades. Journal of Management Education 25(3): 269–291
Tatro N. C. (1995) Gender effects on student evaluations of faculty. Journal of Research and Development in Education 28(3): 169–173
Tenko R., Marcoulides G. (2000) First Course in Structural Equation Modeling. Erlbaum, Mahwah, NJ
Theall M., Franklin J. (1990) Student Ratings in the Context of Complex Evaluation Systems. Jossey Bass, San Francisco, pp. 17–34
Tieman R. C., Ranklin-Ullock B. (1985) Student evaluations of teachers: An examination of the effect of sex and field study. Teaching Sociology 12(2): 1977–191
Young S., Shaw G. D. (1999) Profiles of effective college and university teachers. Journal of Higher Education 70(6): 670–686
Wachtel K. H. (1998) Student evaluation of college teaching effectiveness: A brief review. Assessment & Evaluation in Higher Education 23(2): 191–211
Wigington H., Tollefson N., Rodriguez E. (1989) Students’ ratings of instructors revisited: Interactions among class and instructor variables. Research in Higher Education 30(3): 331–344
Willet H. L. (1980) Comparison of institutional effectiveness of full-and part-time faculty. Community/Junior College Research Quarterly 5(1): 23–30
Worrell C. F., Kuterbach D. L. (2001) The use of student ratings of teacher behaviors with academically talented high school students. Journal of Secondary Gifted Education 12(4): 236–247
Worthington A. C. (2002) The impact of student perceptions and characteristics on teaching evaluations: A case study in finance education. Assessment & Evaluation in Higher Education 27(1): 49–64
Zahn K. D., Schramm M. R. (1992) Student perception of teacher effectiveness based on teacher employment and course skill level. Business Education Forum 46(3): 16–18
Author information
Authors and Affiliations
Corresponding author
Appendices
Appendix A
Items on the Evaluation Questionnaire
-
1.
The instructor teaches the course according to the syllabus.
-
2.
This course can contribute to my professional future.
-
3.
In this course, I am given the opportunity to demonstrate my knowledge and competence.
-
4.
The instructor creates a pleasant learning environment.
-
5.
The instructor defined course requirements clearly.
-
6.
Course content is stimulating.
-
7.
The instructor is tolerant of students’ views that differ from his/her own.
-
8.
In this course there are many opportunities for students to be active.
-
9.
The instructor provides clear instructions for all assignments.
-
10.
The instructor encourages the exchange of ideas in class.
-
11.
The content of the course is presented in an interesting way.
-
12.
Course assignments contribute to my understanding of the course materials.
-
13.
Class time is efficiently used.
-
14.
Discussions in class contribute to my understanding of course materials.
-
15.
Course assignments are too difficult.
-
16.
The instructor is not considerate of individual differences among students.
Rights and permissions
About this article
Cite this article
Nasser, F., Hagtvet, K. MULTILEVEL ANALYSIS OF THE EFFECTS OF STUDENT AND INSTRUCTOR/COURSE CHARACTERISTICS ON STUDENT RATINGS. Res High Educ 47, 559–590 (2006). https://doi.org/10.1007/s11162-005-9007-y
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11162-005-9007-y