Abstract
This paper studies three models for cognitive diagnosis, each illustrated with an application to fraction subtraction data. The objective of each of these models is to classify examinees according to their mastery of skills assumed to be required for fraction subtraction. We consider the DINA model, the NIDA model, and a new model that extends the DINA model to allow for multiple strategies of problem solving. For each of these models the joint distribution of the indicators of skill mastery is modeled using a single continuous higher-order latent trait, to explain the dependence in the mastery of distinct skills. This approach stems from viewing the skills as the specific states of knowledge required for exam performance, and viewing these skills as arising from a broadly defined latent trait resembling the θ of item response models. We discuss several techniques for comparing models and assessing goodness of fit. We then implement these methods using the fraction subtraction data with the aim of selecting the best of the three models for this application. We employ Markov chain Monte Carlo algorithms to fit the models, and we present simulation results to examine the performance of these algorithms.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B.N. Petrov & F. Csaki (Eds.), Proceedings of the Second International Symposium on Information Theory (pp. 267–281). Budapest: Akad. Kiado.
Brooks, S.P., & Gelman, A. (1998). General methods of monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7, 434–455.
Casella, G., & George, E.I. (1992). Explaining the Gibbs sampler. The American Statistician, 46, 167–174.
Celeux, G., Forbers, F., Robert, C.P., & Titterington, D.M. (2006). Deviance information criteria for missing data models. Bayesian Analysis, 1, 651–674.
Chib, S., & Greenberg, E. (1995). Understanding the Metropolis-Hastings algorithm. The American Statistician, 49, 327–335.
de la Torre, J. (2008, in press). An empirically-based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement.
de la Torre, J., & Douglas, J. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.
DiBello, L.V., Stout, W.F., & Roussos, L.A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In P.D. Nichols, S.F. Chipman, & R.L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361–389). Hillsdale, NJ: Erlbaum.
Doignon, J.P., & Falmagne, J.C. (1999). Knowledge spaces. New York: Springer.
Doornik, J.A. (2003). Object-Oriented Matrix Programming using Ox (Version 3.1) (Computer software). London: Timberlake Consultants Press.
Embretson, S. (1984). A general latent trait model for response processes. Psychometrika, 49, 175–186.
Embretson, S. (1997). Multicomponent response models. In W.J. van der Linden & R.K. Hambleton (Eds.), Handbook of modern item response theory (pp. 305–321). New York: Springer.
Geman, S., & Geman, D. (1984). Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 721–741.
Haertel, E.H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333–352.
Junker, B.W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.
Kass, R.E. (1993). Bayes factor in practice. The Statistician, 42, 551–560.
Kass, R.E., & Raftery, A.E. (1995). Bayes factor. Journal of the American Statistical Association, 430, 773–795.
LeFevre, J., Bisanz, J., Daley, K., Buffone, L., Greenham, S.L., & Sadesky, G.S. (1996). Multiple routes to solution of single-digit multiplication problems. Journal of Experimental Psychology: General, 125, 284–306.
Macready, G.B., & Dayton, C.M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 33, 379–416.
Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187–212.
Mislevy, R.J. (1996). Test theory reconceived. Journal of Education al Measurement, 33, 379–416.
Opfer, J.E., & Siegler, R.S. (2007). Representational change and children’s numerical estimation. Cognitive Psychology, 55, 169–195.
Patz, R.J., & Junker, B.W. (1999a). A straightforward approach to Markov chain Monte Carlo methods for item response theory. Journal of Educational and Behavioral Statistics, 24, 146–178.
Patz, R.J., & Junker, B.W. (1999b). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24, 342–366.
Raftery, A.E. (1996). Hypothesis testing and model selection. In R.W. Gilks, S. Richardson, & D.J. Spiegelhalter (Eds.), Markov chain Monte Carlo in practice (pp. 163–187). London: Chapman & Hall.
Reder, L.M. (1987). Strategy selection in question answering. Cognitive Psychology, 19, 90–138.
Shultz, T., Fisher, G., Pratt, C., & Rulf, S. (1986). Selection of causal rules. Child Development, 57, 143–152.
Siegler, R.S. (1988). Individual differences in strategy choices: Good students, not-so-good students, and perfectionists. Child Development, 59, 833–851.
Siegler, R.S., Adolph, K.E., & Lemaire, P. (1996). Strategy choices across the lifespan. In L. Reder (Ed.), Implicit memory and metacognition (pp. 79–121). Hillsdale, NJ: Erlbaum.
Siegler, R.S., & Shrager, J. (1984). Strategy choices in addition and subtraction: How do children know what to do? In C. Sophian (Ed.), The origins of cognitive skills (pp. 229–293). Hillsdale, NJ: Erlbaum.
Spiegelhalter, D.J., Best, N.G., Carlin, B.P., & van der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society Series B, 64, 583–539.
Stout, W., Roussos, L., & Hartz, S. (2003). A demonstration of the Fusion Model skills diagnostic system: An analysis of mixed-number subtraction data. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.
Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.
Tatsuoka, K. (1985). A probabilistic model for diagnosing misconceptions in the pattern classification approach. Journal of Educational Statistics, 12, 55–73.
Tatsuoka, C. (2002). Data-analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society Series C (Applied Statistics), 51, 337–350.
Tatsuoka, K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Safto (Eds.), Monitoring skills and knowledge acquisition (pp. 453–488). Hillsdale, NJ: Erlbaum.
Yan, D., Almond, R., Mislevy, R., & Simpson, M.A. (2003). A rule space approach to modeling skills-pattern of two different pedagogical groups in mathematics assessment. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.
Author information
Authors and Affiliations
Corresponding author
Additional information
The work reported here was performed under the auspices of the External Diagnostic Research Team funded by Educational Testing Service. Views expressed in this paper does not necessarily represent the views of Educational Testing Service.
Rights and permissions
About this article
Cite this article
de la Torre, J., Douglas, J.A. Model Evaluation and Multiple Strategies in Cognitive Diagnosis: An Analysis of Fraction Subtraction Data. Psychometrika 73, 595–624 (2008). https://doi.org/10.1007/s11336-008-9063-2
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-008-9063-2