Effectiveness of Diverse Evidence for Developing Convincing Proofs with Crowdsourcing

Crowdsourcing techniques have been developed over the past decade to leverage the ‘wisdom of crowds’ to solve real-world problems and human intelligence tasks. An example of such a task is fact-checking, that asks workers to check whether the shown claim is true or not. Fact-checking tasks are used in many applications such as services to identify fake news. This context motivates us to investigate a crowdsourcing-based framework to find convincing proofs for fact-checking. This study investigates a good crowdsourcing strategy to obtain convincing proofs with evidence. We focus on an iterative workflow in which workers improve the proofs provided by other crowd workers. In this study, we show the results of our experiment, in which we compare the results of two conditions, by which we see the effects of the diversification of shown evidence and proofs. The experimental results with real-world workers show that showing diverse evidence and proofs to crowd workers in the iterative workflow helps them develop more convincing proofs.