Multilabel classification using crowdsourcing under budget constraints

Multilabel classification has excelled in several distinct fields during the past few decades but still has significant limitations. One of the critical concerns is the lack or insufficient availability of label instances, and data labelling also needs time and budget, which is a challenge. Crowdsourcing overcomes the problem of label availability, yet, it has drawbacks such as label quality and budget limitations. The paper introduced the multilabel reverse auction framework to address the lack of crowd worker's issue. Each crowd worker must provide cost and confidence for each task for a specific domain. Furthermore, two methods for systematic budget selection are presented to address the insufficient domain coverage within the budget limitation: Greedy bid selection and Multi cover bid selection. Both approaches choose the most inexpensive crowd workers while considering worker expertise and domain coverage. Crowd version binary relevance and multilabel k-nearest neighbours are also introduced to support label aggregation and reduce low-quality workers' impact while considering the domain. An experimental study shows the effectiveness of our approach on seven multilabel datasets using diverse crowds. It delivers more than 16% improvement compared to the random selection with a majority voting baseline technique. The proposed method is compared against five benchmark algorithms and provides promising results when minimal availability of data and workers.