Abstract
This study tested the Systematic Distortion Hypothesis by examining the factorial validity of student ratings of university teaching. Factorial validity is defined as the degree to which covariance among judged traits resembles the actual or true covariation of observable behaviors underlying these traits. Although many studies have examined the factorial validity of ratings, results are inconsistent. The present study used a more complete methodology to address some of the limitations of previous studies. Student ratings of teaching and measurements of actual teaching behaviors were obtained for 32 instructors. Student ratings were compared to frequency counts of actual teaching behaviors obtained from videotape and to students’ similarity judgments of teacher characteristics. It was found, first, that the structure of student ratings showed a moderately strong relation to the structure of actual behaviors, and a somewhat stronger relation to the structure of conceptual associations; and second, that the effects of systematic distortion were more pronounced for low-inference student ratings than for high-inference ratings.
Similar content being viewed by others
References
L. Albright D. A. Kenny T. R. Malloy (1988) ArticleTitleConsensus in personality judgment at zero acquaintance Journal of Personality and Social Psychology 55 387–395 Occurrence Handle10.1037/0022-3514.55.3.387
N. Ambady R. Rosenthal (1993) ArticleTitleHalf a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness Journal of Personality and Social Psychology 64 431–441 Occurrence Handle10.1037/0022-3514.64.3.431
J. S. Berman D. A. Kenny (1976) ArticleTitleCorrelational bias in observer ratings Journal of Personality and Social Psychology 34 263–273
J. Block D. S. Weiss A. Thorne (1979) ArticleTitleHow relevant is a semantic similarity interpretation of personality ratings? Journal of Personality and Social Psychology 37 1055–1074
P. Borkenau A. Liebler (1992) ArticleTitleTrait inferences: Sources of validity at zero acquaintance Journal of Personality and Social Psychology 62 645–657 Occurrence Handle10.1037/0022-3514.62.4.645
P. Borkenau F. Ostendorf (1987) ArticleTitleRetrospective estimates of act frequencies: How accurately do they reflect reality? Journal of Personality and Social Psychology 52 626–638 Occurrence Handle10.1037/0022-3514.52.3.626
B. M. Byrne (2001) Structural Equation Modeling with AMOS: Basic concepts. Applications, and programming Lawrence Erlbaum Associates Inc. Mahwah, NJ
J. Cadwell J. Jenkins (1985) ArticleTitleEffects of the semantic similarity of items on student ratings of instructors Journal of Educational Psychology 77 383–393 Occurrence Handle10.1037/0022-0663.77.4.383
P. Chadwick G. Taylor (2000) ArticleTitleAre deluded people unusually prone to illusory correlation? Behavior Modification 24 130–141
L. J. Chapman J. P. Chapman (1967) ArticleTitleGenesis of popular but erroneous psychodiagnostic observations Journal of Abnormal Psychology 72 193–204
L. J. Chapman J. P. Chapman (1969) ArticleTitleIllusory correlation as an obstacle to the use of valid psychodiagnostic signs Journal of Abnormal Psychology 74 271–280
P. A. Cohen (1981) ArticleTitleStudent ratings of instruction and student achievement: A meta-analysis of multisection validity studies Review of Educational Research 51 281–309
W. H. Cooper (1981) ArticleTitleConceptual similarity as a source of illusory halo Journal of Applied Psychology 66 302–307
P. A. Cranton W. Hilgartner (1981) ArticleTitleThe relationship between student ratings and instructor behavior: Implications for improving teaching Canadian Journal of Higher Education 11 73–81
L. J. Cronbach (1955) ArticleTitleProcesses affecting scores on “understanding of others” and “assumed similarity” Psychological Bulletin 52 177–193
Cronbach, L. J. (1958). Proposals leading to analytic treatment of social perception scores. In R. Tagiuri and L. Petrullo (eds.), Person Perception and Interpersonal Behavior. Stanford University Press, pp. 351–379.
J. R. Dewhirst J. S. Berman (1978) ArticleTitleSocial judgments of spurious and causal relations between attributes and outcomes Journal of Experimental Social Psychology 14 313–325 Occurrence Handle10.1016/0022-1031(78)90019-7
S. J. Dollinger L. Greening R. C. Radtke (2001) ArticleTitleReading too much between the lines: Illusory correlation and the word association implications test Personality and Individual Differences 30 799–808 Occurrence Handle10.1016/S0191-8869(00)00072-6
K. A. Feldman (1989a) ArticleTitleInstructional effectiveness of college teachers as judged by teachers themselves, current and former students, colleagues, administrators, and external (neutral) observers Research in Higher Education 30 137–194
K. A. Feldman (1989b) ArticleTitleThe association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies Research in Higher Education 30 583–645
D. L. Hamilton T. L. Rose (1980) ArticleTitleIllusory correlation and the maintenance of stereotypic beliefs Journal of Personality and Social Psychology 39 832–845
L.T. Hu P. M. Bentler (1995) Evaluating model fit R. H. Hoyle (Eds) Structural Equation Modeling: Concepts, Issues and Applications Sage Thousand Oaks, CA 76–99
D. N. Jackson D. W. Chan L. J. Stricker (1979) ArticleTitleImplicit personality theory: Is it illusory? Journal of Personality 47 1–10
D. A. Kenny C. Horner D. A. Kashy L. Chu (1992) ArticleTitleConsensus at zero acquaintance: Replication, behavioral cues, and stability Journal of Personality and Social Psychology 62 88–97 Occurrence Handle10.1037/0022-3514.62.1.88
R. N. King D. J. Koehler (2000) ArticleTitleIllusory correlations in graphical inference Journal of Experimental Psychology: Applied 6 336–348 Occurrence Handle10.1037/1076-898X.6.4.336
S. W. J. Kozlowski M. P. Kirsch (1987) ArticleTitleThe systematic distortion hypothesis, halo, and accuracy: An individual-level analysis Journal of Applied Psychology 72 252–261 Occurrence Handle10.1037/0021-9010.72.2.252
S. W. J. Kozlowski M. P. Kirsch G. T. Chao (1986) ArticleTitleJob knowledge, ratee familiarity, conceptual similarity, and halo error: An exploration Journal of Applied Psychology 71 45–49
Lodzinski, A. (1991). The Systematic Distortion Hypothesis and Diagnostically Meaningful Student Ratings of University Instructor Teaching Effectiveness. Unpublished doctoral dissertation, University of Windsor, Windsor Ontario.
A. Lodzinski F. Schneider H. G. Murray (1991) ArticleTitleThe systematic distortion hypothesis and student ratings of low-inference teaching behaviors Instructional Evaluation and Faculty Development 12 8–15
H. W. Marsh (1982) ArticleTitleSEEQ: A reliable, valid, and useful instrument for collecting students’ evaluations of university teaching British Journal of Educational Psychology 52 77–95
H. W. Marsh (1984) ArticleTitleStudents’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and utility Journal of Educational Psychology 76 707–754
H. W. Marsh K.-T. Hau (1999) Confirmatory factor analysis: Strategies for small sample sizes R. H. Hoyle (Eds) Statistical Strategies for Small Sample Research Sage Thousand Oaks, CA 251–284
H. G. Murray (1980) Evaluating University Teaching: A Review of Research Ontario Confederation of University Faculty Associations Toronto
H. G. Murray (1983) ArticleTitleLow-inference classroom teaching behaviors and student ratings of college teaching effectiveness Journal of Educational Psychology 75 138–149
F. T. Passini W. T. Norman (1966) ArticleTitleA universal conception of personality structure? Journal of Personality and Social Psychology 4 44–49
S. V. Paunonen (1991) ArticleTitleOn the accuracy of ratings of personality by strangers Journal of Personality and Social Psychology 61 471–477 Occurrence Handle10.1037/0022-3514.61.3.471
S. V. Paunonen D. N. Jackson (1979) ArticleTitleNonverbal trait inference Journal of Personality and Social Psychology 37 1645–1659
R. Riemann A. Angleitner (1993) ArticleTitleInferring interpersonal traits from behavior: Act prototypicality versus conceptual similarity of trait concepts Journal of Personality and Social Psychology 64 356–364 Occurrence Handle10.1037/0022-3514.64.3.356
D. Romer W. Revelle (1984) ArticleTitlePersonality traits: Fact or fiction? A critique of the Shweder and D’ndrade systematic distortion hypothesis Journal of Personality and Social Psychology 47 1028–1042
P. E. Shrout J. L. Fleiss (1979) ArticleTitleIntraclass correlations: Uses in assessing rater reliability Psychological Bulletin 86 420–428
R. A. Shweder (1975) ArticleTitleHow relevant is an individual difference theory of personality? Journal of Personality 43 455–485
R. A. Shweder (1977) ArticleTitleIllusory correlation and the M.M.P.I. controversy Journal of Consulting and Clinical Psychology 45 917–924
R. A. Shweder (1982) ArticleTitleFact and artifact in trait perception: The systematic distortion hypothesis Progress in Experimental Personality Research 11 65–100
R. A. Shweder R. G. D’Andrade (1980) The systematic distortion hypothesis R. A. Shweder (Eds) Fallible Judgment in Behavioral Research: New directions for Methodology of Social and Behavioral Science Jossey-Bass San Francisco 37–58
R. Vonk (1998) ArticleTitleThe slime effect: suspicion and dislike of likeable behavior toward superiors Journal of Personality and Social Psychology 74 849–864 Occurrence Handle10.1037/0022-3514.74.4.849
D. Watson (1989) ArticleTitleStrangers’ ratings of the five robust personality factors: Evidence of a surprising convergence with self-reports Journal of Personality and Social Psychology 57 120–128 Occurrence Handle10.1037/0022-3514.57.1.120
D. S. Weiss G. A. Mendelsohn (1986) ArticleTitleAn empirical demonstration of the implausibility of the semantic similarity explanation of how trait ratings are made and what they mean Journal of Personality and Social Psychology 50 595–601
S. E. Whitely K. O. Doyle (1976) ArticleTitleImplicit theories in student ratings American Educational Research Journal 13 241–253
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Renaud, R.D., Murray, H.G. Factorial Validity of Student Ratings of Instruction. Res High Educ 46, 929–953 (2005). https://doi.org/10.1007/s11162-005-6934-6
Received:
Issue Date:
DOI: https://doi.org/10.1007/s11162-005-6934-6