Testing! testing! one, two, three – Testing the theory in structural equation models!

https://doi.org/10.1016/j.paid.2006.10.001Get rights and content

Abstract

Barrett (2007) presents minor revisions to statements previously posted on Barrett’s website, and discussed on SEMNET (a web discussion group about structural equation modeling). Unfortunately, Barrett’s “recommendations” remain seriously statistically and methodologially flawed. Unlike Barrett, we see scientific value in reporting models that challenge or discredit theories. We critique Barrett’s way of proceeding in the context of both especially small and large samples, and we urge greater attention to the χ2 significance test.

Introduction

In “Structural equation modeling: adjudging model fit” Barrett (2007) provides minor revisions to advice previously presented on Barrett’s web site, and critiqued by one of us (Hayduk). Despite some revisions, Barrett’s recommendations remain seriously problematic. We begin with a brief context for Barrett’s article, before considering specific problematic statements. We argue that journal reviewers should pay greater attention to χ2 significance tests, and to the theory being tested.

Section snippets

Testing structural equation models

Structural equation modeling (SEM) grew out of the confluence of a path analytic tradition and a factor analytic tradition. The path analytic tradition had a history of seeking to test theories (Duncan, 1975). The exploratory factor analytic tradition had almost no theory1 and non-testing was the factor

Researchers must test theories!

Serious problems begin right in Barrett’s title. Do we want to “adjudge” fit, or do we want to test our theorizing? Barrett’s title says judge fit. Our view is that researchers ought to do their best to test their theorizing. Our scientific bias is to carefully test our structural equation models/theories. Barrett’s bias displaces or avoids model/theory testing by replacing testing with adjudication, and by focusing on fit rather than on the model providing the fit. A statistician who focuses

Structural equation models as theoretical models

Structural equation models represent specific theory-based causal connections between latent variables and between those latents and relevant indicator variables. Estimates of the model’s parameters are those values which, when placed in the model’s equations, imply an indicator variance/covariance matrix that is as similar as possible to the data variance/covariance matrix. The similarity, or dissimilarity, of these matrices is usually expressed as the likelihood of observing the data

Barrett on testing

Barrett (2007, p. 816) claims the model χ2 test is a “conventional null hypothesis significance test (NHST)”. The χ2 test has a null hypothesis and is a significance test, but it is rendered importantly unconventional by the possibility of mis-specified covariance-equivalent (and nearly covariance-equivalent) models. Both the proper causal model, and seriously causally-misspecified covariance-equivalent models can provide zero, or near zero, residual covariances. The existence of

Barrett section 1

We agree that structural equation model testing via χ2, degrees of freedom, and the associated probability must be reported for all manuscripts reporting SEM results. We disagree that N < 200 or N > 10,000 provide reasonable boundaries for altering this requirement. The researcher simply MUST report this information.

Barrett says that “if the model fits”, the researcher might “proceed to report and discuss features of the model” (2007, p. 820). Should researchers report, discuss, and publish failing

Conclusion

We conclude that Barrett’s article is too statistically ill-founded to constitute reasonable advice on structural equation model testing. Barrett seems to think he has adopted a “hard line”. We think he is still circumnavigating, and not addressing, the hard-point of structural equation model testing. Barrett has provided sporadically strong assertions, but these do not nullify his culpability for making other deficient statistical recommendations. The occasional strong assertion in favor of

Acknowledgement

The authors thank Dionne Pohler for participating in discussions of this paper.

References (13)

  • P. Barrett

    Structural equation modelling: adjudging model fit

    Personality and Individual Differences

    (2007)
  • K.A. Bollen

    Overall fit in covariance structure models: two types of sample size effects

    Psychological Bulletin

    (1990)
  • M.W. Browne et al.

    When fit indices and residuals are incompatible

    Psychological Methods

    (2002)
  • K.P. Burnham et al.

    Model selection and multimodel inference: A practical information-theoretic approach

    (2002)
  • G.G. Cummings et al.

    Is the nursing work index measuring up? Moving beyond estimating reliability to testing validity

    Nursing Research

    (2006)
  • O.D. Duncan

    Introduction to structural equation models

    (1975)
There are more references available in the full text version of this article.

Cited by (255)

  • Common methodological mistakes

    2023, Leadership Quarterly
View all citing articles on Scopus
View full text