The Invalidity (And Then Some) of Rank Correlation for Describing the Accuracy of Multiple-Choice Question Difficulty Taxonomies

The difficulty of multiple-choice questions is one of their fundamental properties and that property serves many purposes. With this, the questions in published multiple-choice question banks accompanying textbooks are often classified into levels of difficulty. In light of the many purposes difficulty level may serve, the accuracy of those published classifications is important. Measuring that accuracy is the function of a recently created system of analyses, the Taxonomy Index (TaxI) Program (Dickinson 2013). It has been suggested (Bacon 2013), though, that rank correlation may perform this same function better than TaxI, better to the point of discounting the latter altogether. The present research demonstrates the invalidity of rank correlation for this application, the validity of TaxI for this application, and the numerous beneficial features of the TaxI Program not available with rank correlation (e.g., Spearman’s rho, Kendall’s tau_b).