Card Sorting Data Collection Methodology: How Many Participants Is Most Efficient?

Pairwise similarity judgments and card sorting methodologies are different ways of generating data for similarity matrices used in various analyses such as multidimensional scaling and cluster analysis. Pairwise similarity judgments are considered the gold standard methodology, but can be cumbersome for large numbers of stimuli given the geometric increase in number of judgments necessary to fill the matrix. Card sorting methods provide a more expedient means of gathering this information, although they typically generate only binary data. Nonetheless, aggregated matrices generated from card sorts approximate pairwise similarity matrices. The current study used pairwise similarity and card sorting results from two existing studies that used the same stimuli to determine the optimal number of participants needed in a card sorting task to approximate the similarity matrix of pairwise data collection. In these studies, approximately 10–15 participants provided optimal estimation of the similarity matrix, with minimal increases for higher numbers of participants.