Finding High-Quality Unstructured Submissions in General Crowdsourcing Tasks

The quality of crowdsourced work varies drastically from superior to inferior. As a consequence, the problem of automatically finding high-quality crowdsourced work is of great importance. A variety of aggregation methods have been proposed for multiple-choice tasks such as item labeling with structured claims. However, they do not apply to more general tasks, such as article writing and brand design, with unstructured submissions that cannot be aggregated. Recent work tackles this problem by asking another set of crowd workers to review and grade each submission, essentially transforming unstructured submissions into structured ratings that can be aggregated. Nevertheless, such an approach incurs unnecessary monetary cost and delay. In this paper, we address this problem by exploiting task requesters’ historical feedback and directly modeling the submission quality, without the need of additional crowdsourced ratings. We first propose three sets of features, which try to characterize the submission quality from various perspectives, including the submissions themselves, the workers who make the submissions, and the interactions between task requesters and workers. We then propose two quality models, where one judges the submission quality independently and the other judges comparatively. These models not only incorporate features, but also take worker-specific factors into consideration. Experimental results on three large-scale data sets demonstrate that our models outperform general-purpose learning-to-rank methods such as Logistic Regression, RankBoost, and ListNet for finding high-quality crowdsourced submissions.