Skip to main content
Log in

Factorial Validity of Student Ratings of Instruction

  • Published:
Research in Higher Education Aims and scope Submit manuscript

Abstract

This study tested the Systematic Distortion Hypothesis by examining the factorial validity of student ratings of university teaching. Factorial validity is defined as the degree to which covariance among judged traits resembles the actual or true covariation of observable behaviors underlying these traits. Although many studies have examined the factorial validity of ratings, results are inconsistent. The present study used a more complete methodology to address some of the limitations of previous studies. Student ratings of teaching and measurements of actual teaching behaviors were obtained for 32 instructors. Student ratings were compared to frequency counts of actual teaching behaviors obtained from videotape and to students’ similarity judgments of teacher characteristics. It was found, first, that the structure of student ratings showed a moderately strong relation to the structure of actual behaviors, and a somewhat stronger relation to the structure of conceptual associations; and second, that the effects of systematic distortion were more pronounced for low-inference student ratings than for high-inference ratings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • L. Albright D. A. Kenny T. R. Malloy (1988) ArticleTitleConsensus in personality judgment at zero acquaintance Journal of Personality and Social Psychology 55 387–395 Occurrence Handle10.1037/0022-3514.55.3.387

    Article  Google Scholar 

  • N. Ambady R. Rosenthal (1993) ArticleTitleHalf a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness Journal of Personality and Social Psychology 64 431–441 Occurrence Handle10.1037/0022-3514.64.3.431

    Article  Google Scholar 

  • J. S. Berman D. A. Kenny (1976) ArticleTitleCorrelational bias in observer ratings Journal of Personality and Social Psychology 34 263–273

    Google Scholar 

  • J. Block D. S. Weiss A. Thorne (1979) ArticleTitleHow relevant is a semantic similarity interpretation of personality ratings? Journal of Personality and Social Psychology 37 1055–1074

    Google Scholar 

  • P. Borkenau A. Liebler (1992) ArticleTitleTrait inferences: Sources of validity at zero acquaintance Journal of Personality and Social Psychology 62 645–657 Occurrence Handle10.1037/0022-3514.62.4.645

    Article  Google Scholar 

  • P. Borkenau F. Ostendorf (1987) ArticleTitleRetrospective estimates of act frequencies: How accurately do they reflect reality? Journal of Personality and Social Psychology 52 626–638 Occurrence Handle10.1037/0022-3514.52.3.626

    Article  Google Scholar 

  • B. M. Byrne (2001) Structural Equation Modeling with AMOS: Basic concepts. Applications, and programming Lawrence Erlbaum Associates Inc. Mahwah, NJ

    Google Scholar 

  • J. Cadwell J. Jenkins (1985) ArticleTitleEffects of the semantic similarity of items on student ratings of instructors Journal of Educational Psychology 77 383–393 Occurrence Handle10.1037/0022-0663.77.4.383

    Article  Google Scholar 

  • P. Chadwick G. Taylor (2000) ArticleTitleAre deluded people unusually prone to illusory correlation? Behavior Modification 24 130–141

    Google Scholar 

  • L. J. Chapman J. P. Chapman (1967) ArticleTitleGenesis of popular but erroneous psychodiagnostic observations Journal of Abnormal Psychology 72 193–204

    Google Scholar 

  • L. J. Chapman J. P. Chapman (1969) ArticleTitleIllusory correlation as an obstacle to the use of valid psychodiagnostic signs Journal of Abnormal Psychology 74 271–280

    Google Scholar 

  • P. A. Cohen (1981) ArticleTitleStudent ratings of instruction and student achievement: A meta-analysis of multisection validity studies Review of Educational Research 51 281–309

    Google Scholar 

  • W. H. Cooper (1981) ArticleTitleConceptual similarity as a source of illusory halo Journal of Applied Psychology 66 302–307

    Google Scholar 

  • P. A. Cranton W. Hilgartner (1981) ArticleTitleThe relationship between student ratings and instructor behavior: Implications for improving teaching Canadian Journal of Higher Education 11 73–81

    Google Scholar 

  • L. J. Cronbach (1955) ArticleTitleProcesses affecting scores on “understanding of others” and “assumed similarity” Psychological Bulletin 52 177–193

    Google Scholar 

  • Cronbach, L. J. (1958). Proposals leading to analytic treatment of social perception scores. In R. Tagiuri and L. Petrullo (eds.), Person Perception and Interpersonal Behavior. Stanford University Press, pp. 351–379.

  • J. R. Dewhirst J. S. Berman (1978) ArticleTitleSocial judgments of spurious and causal relations between attributes and outcomes Journal of Experimental Social Psychology 14 313–325 Occurrence Handle10.1016/0022-1031(78)90019-7

    Article  Google Scholar 

  • S. J. Dollinger L. Greening R. C. Radtke (2001) ArticleTitleReading too much between the lines: Illusory correlation and the word association implications test Personality and Individual Differences 30 799–808 Occurrence Handle10.1016/S0191-8869(00)00072-6

    Article  Google Scholar 

  • K. A. Feldman (1989a) ArticleTitleInstructional effectiveness of college teachers as judged by teachers themselves, current and former students, colleagues, administrators, and external (neutral) observers Research in Higher Education 30 137–194

    Google Scholar 

  • K. A. Feldman (1989b) ArticleTitleThe association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies Research in Higher Education 30 583–645

    Google Scholar 

  • D. L. Hamilton T. L. Rose (1980) ArticleTitleIllusory correlation and the maintenance of stereotypic beliefs Journal of Personality and Social Psychology 39 832–845

    Google Scholar 

  • L.T. Hu P. M. Bentler (1995) Evaluating model fit R. H. Hoyle (Eds) Structural Equation Modeling: Concepts, Issues and Applications Sage Thousand Oaks, CA 76–99

    Google Scholar 

  • D. N. Jackson D. W. Chan L. J. Stricker (1979) ArticleTitleImplicit personality theory: Is it illusory? Journal of Personality 47 1–10

    Google Scholar 

  • D. A. Kenny C. Horner D. A. Kashy L. Chu (1992) ArticleTitleConsensus at zero acquaintance: Replication, behavioral cues, and stability Journal of Personality and Social Psychology 62 88–97 Occurrence Handle10.1037/0022-3514.62.1.88

    Article  Google Scholar 

  • R. N. King D. J. Koehler (2000) ArticleTitleIllusory correlations in graphical inference Journal of Experimental Psychology: Applied 6 336–348 Occurrence Handle10.1037/1076-898X.6.4.336

    Article  Google Scholar 

  • S. W. J. Kozlowski M. P. Kirsch (1987) ArticleTitleThe systematic distortion hypothesis, halo, and accuracy: An individual-level analysis Journal of Applied Psychology 72 252–261 Occurrence Handle10.1037/0021-9010.72.2.252

    Article  Google Scholar 

  • S. W. J. Kozlowski M. P. Kirsch G. T. Chao (1986) ArticleTitleJob knowledge, ratee familiarity, conceptual similarity, and halo error: An exploration Journal of Applied Psychology 71 45–49

    Google Scholar 

  • Lodzinski, A. (1991). The Systematic Distortion Hypothesis and Diagnostically Meaningful Student Ratings of University Instructor Teaching Effectiveness. Unpublished doctoral dissertation, University of Windsor, Windsor Ontario.

  • A. Lodzinski F. Schneider H. G. Murray (1991) ArticleTitleThe systematic distortion hypothesis and student ratings of low-inference teaching behaviors Instructional Evaluation and Faculty Development 12 8–15

    Google Scholar 

  • H. W. Marsh (1982) ArticleTitleSEEQ: A reliable, valid, and useful instrument for collecting students’ evaluations of university teaching British Journal of Educational Psychology 52 77–95

    Google Scholar 

  • H. W. Marsh (1984) ArticleTitleStudents’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and utility Journal of Educational Psychology 76 707–754

    Google Scholar 

  • H. W. Marsh K.-T. Hau (1999) Confirmatory factor analysis: Strategies for small sample sizes R. H. Hoyle (Eds) Statistical Strategies for Small Sample Research Sage Thousand Oaks, CA 251–284

    Google Scholar 

  • H. G. Murray (1980) Evaluating University Teaching: A Review of Research Ontario Confederation of University Faculty Associations Toronto

    Google Scholar 

  • H. G. Murray (1983) ArticleTitleLow-inference classroom teaching behaviors and student ratings of college teaching effectiveness Journal of Educational Psychology 75 138–149

    Google Scholar 

  • F. T. Passini W. T. Norman (1966) ArticleTitleA universal conception of personality structure? Journal of Personality and Social Psychology 4 44–49

    Google Scholar 

  • S. V. Paunonen (1991) ArticleTitleOn the accuracy of ratings of personality by strangers Journal of Personality and Social Psychology 61 471–477 Occurrence Handle10.1037/0022-3514.61.3.471

    Article  Google Scholar 

  • S. V. Paunonen D. N. Jackson (1979) ArticleTitleNonverbal trait inference Journal of Personality and Social Psychology 37 1645–1659

    Google Scholar 

  • R. Riemann A. Angleitner (1993) ArticleTitleInferring interpersonal traits from behavior: Act prototypicality versus conceptual similarity of trait concepts Journal of Personality and Social Psychology 64 356–364 Occurrence Handle10.1037/0022-3514.64.3.356

    Article  Google Scholar 

  • D. Romer W. Revelle (1984) ArticleTitlePersonality traits: Fact or fiction? A critique of the Shweder and D’ndrade systematic distortion hypothesis Journal of Personality and Social Psychology 47 1028–1042

    Google Scholar 

  • P. E. Shrout J. L. Fleiss (1979) ArticleTitleIntraclass correlations: Uses in assessing rater reliability Psychological Bulletin 86 420–428

    Google Scholar 

  • R. A. Shweder (1975) ArticleTitleHow relevant is an individual difference theory of personality? Journal of Personality 43 455–485

    Google Scholar 

  • R. A. Shweder (1977) ArticleTitleIllusory correlation and the M.M.P.I. controversy Journal of Consulting and Clinical Psychology 45 917–924

    Google Scholar 

  • R. A. Shweder (1982) ArticleTitleFact and artifact in trait perception: The systematic distortion hypothesis Progress in Experimental Personality Research 11 65–100

    Google Scholar 

  • R. A. Shweder R. G. D’Andrade (1980) The systematic distortion hypothesis R. A. Shweder (Eds) Fallible Judgment in Behavioral Research: New directions for Methodology of Social and Behavioral Science Jossey-Bass San Francisco 37–58

    Google Scholar 

  • R. Vonk (1998) ArticleTitleThe slime effect: suspicion and dislike of likeable behavior toward superiors Journal of Personality and Social Psychology 74 849–864 Occurrence Handle10.1037/0022-3514.74.4.849

    Article  Google Scholar 

  • D. Watson (1989) ArticleTitleStrangers’ ratings of the five robust personality factors: Evidence of a surprising convergence with self-reports Journal of Personality and Social Psychology 57 120–128 Occurrence Handle10.1037/0022-3514.57.1.120

    Article  Google Scholar 

  • D. S. Weiss G. A. Mendelsohn (1986) ArticleTitleAn empirical demonstration of the implausibility of the semantic similarity explanation of how trait ratings are made and what they mean Journal of Personality and Social Psychology 50 595–601

    Google Scholar 

  • S. E. Whitely K. O. Doyle (1976) ArticleTitleImplicit theories in student ratings American Educational Research Journal 13 241–253

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Robert D. Renaud.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Renaud, R.D., Murray, H.G. Factorial Validity of Student Ratings of Instruction. Res High Educ 46, 929–953 (2005). https://doi.org/10.1007/s11162-005-6934-6

Download citation

  • Received:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11162-005-6934-6

Keywords

Navigation