Abstract
The focus of cognitive diagnosis (CD) is on evaluating an examinee’s strengths and weaknesses in terms of cognitive skills learned and skills that need study. Current methods for fitting CD models (CDMs) work well for large-scale assessments, where the data of hundreds or thousands of examinees are available. However, the development of CD-based assessment tools that can be used in small-scale test settings, say, for monitoring the instruction and learning process at the classroom level has not kept up with the rapid pace at which research and development proceeded for large-scale assessments. The main reason is that the sample sizes of the small-scale test settings are simply too small to guarantee the reliable estimation of item parameters and examinees’ proficiency class membership. In this article, a general nonparametric classification (GNPC) method that allows for assigning examinees to the correct proficiency classes with a high rate when sample sizes are at the classroom level is proposed as an extension of the nonparametric classification (NPC) method (Chiu and Douglas in J Classif 30:225–250, 2013). The proposed method remedies the shortcomings of the NPC method and can accommodate any CDM. The theoretical justification and the empirical studies are presented based on the saturated general CDMs, supporting the legitimacy of using the GNPC method with any CDM. The results from the simulation studies and real data analysis show that the GNPC method outperforms the general CDMs when samples are small.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ayers, E., Nugent, R., & Dean, N. (2008). Skill set profile clustering based on student capability vectors computed from online tutoring data. In R. S. J. de Baker, T. Barnes, & J. E. Beck (Eds.), Educational data mining 2008: Proceedings of the 1st international conference on educational data mining, Montreal, Quebec, Canada (pp. 210–217). Retrieved from http://www.educationaldatamining.org/EDM2008/uploads/proc/full%20proceedings.pdf
Chiu, C.-Y., & Douglas, J. A. (2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns. Journal of Classification, 30, 225–250.
Chiu, C.-Y., Douglas, J. A., & Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633–665.
Chiu, C.-Y., Köhn, H.-F., & Wu, H.-M. (2016). Fitting the Reduced RUM with Mplus: A Tutorial. International Journal of Testing, 16, 331–351.
de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76, 179–199.
DiBello, L. V., Roussos, L. A., & Stout, W. F. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Psychometrics (Vol. 26, pp. 979–1030). Amsterdam: Elsevier.
Fu, J., & Li, Y. (2007). An integrative review of cognitively diagnostic psychometric models. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.
Garrett, E., & Zeger, S. L. (2000). Latent class model diagnosis. Biometrics, 56, 1055–1067.
Hartz, S. M. (2002). A Bayesian framework for the Unified Model for assessing cognitive abilities: Blending theory with practicality (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3044108)
Hartz, S. M., & Roussos, L. A. (2008). The fusion model for skill diagnosis: blending theory with practicality. Princeton, NJ: Educational Testing Service. (Research report No. RR-08-71).
Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.
Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.
Köhn, H.-F., & Chiu, C.-Y. (2017). A procedure for assessing the completeness of the Q-matrices of cognitively diagnostic tests. Psychometrika, 82, 112–132.
Macready, G. B., & Dayton, C. M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2, 99–120.
Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187–212.
Richardson, S., & Green, P. J. (1997). On Bayesian analysis of mixtures with an unknown number of components. Journal of Royal Statistical Society Series B, 59, 731–792.
Rupp, A. A., Templin, J. L., & Henson, R. A. (2010). Diagnostic measurement. Theory, methods, and applications. New York: Guilford.
Stephens, M. (2000). Dealing with label switching in mixture models. Journal of Royal Statistical Society Series B, 62, 795–809.
Stout, W. (2002). Psychometrics: From practice to theory and back. Psychometrika, 67, 485–518.
Tatsuoka, K. K. (1985). A probabilistic model for diagnosing misconception by the pattern classification approach. Journal of Educational Statistics, 10, 55–73.
Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.
Tueller, S. J., Drotar, S., & Lubke, G. H. (2011). Addressing the problem of switched class labels in latent variable mixture model simulation studies. Structural Equation Modeling A Multidisciplinary Journal, 18, 110–131.
von Davier, M. (2005). A general diagnostic model applied to language testing data. Princeton, NJ: Educational Testing Service. (Research report No. RR-05-16).
von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287–301.
von Davier, M. (2014). The DINA model as a constrained general diagnostic model: Two variants of a model equivalency. British Journal of Mathematical and Statistical Psychology, 67, 49–71.
Wang, S., & Douglas, J. (2015). Consistency of nonparametric classification in cognitive diagnosis. Psychometrika, 80, 85–100.
Willse, J., Henson, R., & Templin, J. (2007). Using sum scores or IRT in place of cognitive diagnosis models: Can existing or more familiar models do the job? Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.
Zhang, S. & Culpepper, S. A. (2017). Bayesian estimation of restricted latent class models. Paper presented at the annual meeting of the National Council on Measurement in Education, San Antonio, Texas.
Acknowledgements
Funding was provided by National Science Foundation (Grant No. 1552563).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Chiu, CY., Sun, Y. & Bian, Y. Cognitive Diagnosis for Small Educational Programs: The General Nonparametric Classification Method. Psychometrika 83, 355–375 (2018). https://doi.org/10.1007/s11336-017-9595-4
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-017-9595-4