Yashar Moshfeghi
Alvaro F. Huertas-Rosero
ABSTRACT
In this paper we introduce a game scenario for crowdsourcing (CS) using incentives as a bait for careless (gambler) workers, who respond to them in a characteristic way. We hypothesise that careless workers are risk-inclined and can be detected in the game scenario by their use of time, and test this hypothesis in two steps: first, we formulate and prove a theorem stating that a risk-inclined worker will react to competition with shorter Task Completion Time (TCT) than a risk-neutral or risk-averse worker. Second, we check if the game scenario introduces a link between TCT and performance, by performing a crowdsourced evaluation using 35 topics from the TREC-8 collection. Experimental evidence confirms our hypothesis, showing that TCT can be used as a powerful discrimination factor to detect careless workers. This is a valuable result in the quest for quality assurance in CS-based micro tasks such as relevance assessment.
- O. Alonso. Implementing crowdsourcing-based relevance experimentation: an industrial perspective. Inf. Retr., pages 1--20, 2013. Google Scholar
Digital Library
- B. Carterette and I. Soboroff. The Effect of Assessor Error on IR System Evaluation. In SIGIR '10, pages 539--546, 2010. Google ScholarDigital Library
- C. Eickhoff, C. G. Harris, A. P. de Vries, and P. Srinivasan. Quality through flow and immersion: gamifying crowdsourced relevance assessments. In SIGIR '12, pages 871--880, 2012. Google ScholarDigital Library
- C. Grady and M. Lease. Crowdsourcing document relevance assessment with Mechanical Turk. In CSLDAMT '10, pages 172--179, 2010. Google ScholarDigital Library
- P. G. Ipeirotis, F. Provost, and J. Wang.Google Scholar
- G. Kazai, J. Kamps, and N. Milic-Frayling. The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy. In CIKM '12, pages 2583--2586, 2012. Google ScholarDigital Library
- G. Kazai, J. Kamps, and N. Milic-Frayling. An Analysis of Human Factors and Label Accuracy in Crowdsourcing Relevance Judgments. Inf. Retr., 16(2):138--178, Apr. 2013. Google ScholarDigital Library
- J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 workshop on crowdsourcing for search evaluation, pages 21--26, 2010.Google Scholar
- Y. Moshfeghi, A. F. H. Rosero, and J. M. Jose. A Game-Theory Approach for Effective Crowdsource-Based Relevance Assessment. ACM Trans. Intell. Syst. Technol., 7(4):55:1--55:25, Mar. 2016. Google ScholarDigital Library
- T. Straub, H. Gimpel, and F. Teschner. The negative effect of feedback on performance in crowd labor tournaments. Collective Intelligence 2014, 2014.Google Scholar
- M. Szilagyi. Agent-Based Simulation of the N-Person Chicken Game. In Advances in Dynamic Game Theory, volume 9, pages 696--703. Birkhäuser Boston, 2007.Google ScholarCross Ref
- L. Von Ahn and L. Dabbish. Designing games with a purpose. Communications of the ACM, 51(8):58--67, 2008. Google ScholarDigital Library
- E. Voorhees, D. Harman, N. I. of Standards, and T. (US). TREC: Experiment and evaluation in information retrieval, volume 63. MIT press Cambridge, 2005. Google ScholarDigital Library
- Y. Zhao and Q. Zhu. Evaluation on crowdsourcing research: Current status and future direction. Information Systems Frontiers, pages 1--18, 2012. Google ScholarDigital Library
Index Terms
- Identifying Careless Workers in Crowdsourcing Platforms: A Game Theory Approach
Recommendations
A Game Theory Approach for Estimating Reliability of Crowdsourced Relevance Assessments
In this article, we propose an approach to improve quality in crowdsourcing (CS) tasks using Task Completion Time (TCT) as a source of information about the reliability of workers in a game-theoretical competitive scenario. Our approach is based on the ...
A Game-Theory Approach for Effective Crowdsource-Based Relevance Assessment
Special Issue on Crowd in Intelligent Systems, Research Note/Short Paper and Regular PapersDespite the ever-increasing popularity of crowdsourcing (CS) in both industry and academia, procedures that ensure quality in its results are still elusive. We hypothesise that a CS design based on game theory can persuade workers to perform their tasks ...
Differences between the iterated prisoner's dilemma and the chicken game under noisy conditions
SAC '02: Proceedings of the 2002 ACM symposium on Applied computingThe prisoner's dilemma has evolved into a standard game for analyzing the success of cooperative strategies in repeated games. With the aim of investigating the behavior of strategies in some alternative games we analyzed the outcome of iterated games ...
Comments