Testing! testing! one, two, three – Testing the theory in structural equation models!
Introduction
In “Structural equation modeling: adjudging model fit” Barrett (2007) provides minor revisions to advice previously presented on Barrett’s web site, and critiqued by one of us (Hayduk). Despite some revisions, Barrett’s recommendations remain seriously problematic. We begin with a brief context for Barrett’s article, before considering specific problematic statements. We argue that journal reviewers should pay greater attention to χ2 significance tests, and to the theory being tested.
Section snippets
Testing structural equation models
Structural equation modeling (SEM) grew out of the confluence of a path analytic tradition and a factor analytic tradition. The path analytic tradition had a history of seeking to test theories (Duncan, 1975). The exploratory factor analytic tradition had almost no theory1 and non-testing was the factor
Researchers must test theories!
Serious problems begin right in Barrett’s title. Do we want to “adjudge” fit, or do we want to test our theorizing? Barrett’s title says judge fit. Our view is that researchers ought to do their best to test their theorizing. Our scientific bias is to carefully test our structural equation models/theories. Barrett’s bias displaces or avoids model/theory testing by replacing testing with adjudication, and by focusing on fit rather than on the model providing the fit. A statistician who focuses
Structural equation models as theoretical models
Structural equation models represent specific theory-based causal connections between latent variables and between those latents and relevant indicator variables. Estimates of the model’s parameters are those values which, when placed in the model’s equations, imply an indicator variance/covariance matrix that is as similar as possible to the data variance/covariance matrix. The similarity, or dissimilarity, of these matrices is usually expressed as the likelihood of observing the data
Barrett on testing
Barrett (2007, p. 816) claims the model χ2 test is a “conventional null hypothesis significance test (NHST)”. The χ2 test has a null hypothesis and is a significance test, but it is rendered importantly unconventional by the possibility of mis-specified covariance-equivalent (and nearly covariance-equivalent) models. Both the proper causal model, and seriously causally-misspecified covariance-equivalent models can provide zero, or near zero, residual covariances. The existence of
Barrett section 1
We agree that structural equation model testing via χ2, degrees of freedom, and the associated probability must be reported for all manuscripts reporting SEM results. We disagree that N < 200 or N > 10,000 provide reasonable boundaries for altering this requirement. The researcher simply MUST report this information.
Barrett says that “if the model fits”, the researcher might “proceed to report and discuss features of the model” (2007, p. 820). Should researchers report, discuss, and publish failing
Conclusion
We conclude that Barrett’s article is too statistically ill-founded to constitute reasonable advice on structural equation model testing. Barrett seems to think he has adopted a “hard line”. We think he is still circumnavigating, and not addressing, the hard-point of structural equation model testing. Barrett has provided sporadically strong assertions, but these do not nullify his culpability for making other deficient statistical recommendations. The occasional strong assertion in favor of
Acknowledgement
The authors thank Dionne Pohler for participating in discussions of this paper.
References (13)
Structural equation modelling: adjudging model fit
Personality and Individual Differences
(2007)Overall fit in covariance structure models: two types of sample size effects
Psychological Bulletin
(1990)- et al.
When fit indices and residuals are incompatible
Psychological Methods
(2002) - et al.
Model selection and multimodel inference: A practical information-theoretic approach
(2002) - et al.
Is the nursing work index measuring up? Moving beyond estimating reliability to testing validity
Nursing Research
(2006) Introduction to structural equation models
(1975)
Cited by (255)
Teachers’ motivation: Exploring the success expectancies, values and costs of the promotion of self-regulated learning
2023, Teaching and Teacher EducationMeasuring risk culture in finance: Development of a comprehensive measure
2023, Journal of Banking and FinanceCommon methodological mistakes
2023, Leadership QuarterlyDriving mechanism for farmers' participation in improving farmland ecosystem: Evidence from China
2022, Journal of Cleaner ProductionFifty years of structural equation modeling: A history of generalization, unification, and diffusion
2022, Social Science Research