Abstract
In this article, we aim to gain a better understanding into how paid microtask crowdsourcing could leverage its appeal and scaling power by using contests to boost crowd performance and engagement. We introduce our microtask-based annotation platform Wordsmith, which features incentives such as points, leaderboards, and badges on top of financial remuneration. Our analysis focuses on a particular type of incentive, contests, as a means to apply crowdsourcing in near-real-time scenarios, in which requesters need labels quickly. We model crowdsourcing contests as a continuous-time Markov chain with the objective to maximise the output of the crowd workers, while varying a parameter that determines whether a worker is eligible for a reward based on their present rank on the leaderboard. We conduct empirical experiments in which crowd workers recruited from CrowdFlower carry out annotation microtasks on Wordsmith—in our case, to identify named entities in a stream of Twitter posts. In the experimental conditions, we test different reward spreads and record the total number of annotations received. We compare the results against a control condition in which the same annotation task was completed on CrowdFlower without a time or contest constraint. The experiments show that rewarding only the best contributors in a live contest could be a viable model to deliver results faster, though quality might suffer for particular types of annotation tasks. Increasing the reward spread leads to more work being completed, especially by the top contestants. Overall, the experiments shed light on possible design improvements of paid microtasks platforms to boost task performance and speed and make the overall experience more fair and interesting for crowd workers.
- Nikolay Archak. 2010. Money, glory, and cheap talk: Analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on TopCoder.com. In Proceedings of the 19th International Conference on World Wide Web. ACM, 21--30. Google ScholarDigital Library
- Zahra Ashktorab, Christopher Brown, Manojit Nandi, and Aron Culotta. 2014. Tweedr: Mining Twitter to inform disaster response. In Proceedings of the Information Systems for Crisis Response and Management Conference (ISCRAM’14).Google Scholar
- James Bennett and Stan Lanning. 2007. The Netflix prize. In Proceedings of the Knowledge Discovery and Data Mining Cup and Workshop (KDD’07). 35.Google Scholar
- Michael S. Bernstein, Joel Brandt, Robert C. Miller, and David R. Karger. 2011. Crowds in two seconds: Enabling real-time crowd-powered interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM, 33--42.Google Scholar
- Michael S. Bernstein, David R. Karger, Robert C. Miller, and Joel Brandt. 2012. Analytic methods for optimizing realtime crowdsourcing. Collect. Intell. (2012). http://www.ci2012.org/.Google Scholar
- Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. 2015. Soylent: A word processor with a crowd inside. Commun. ACM 58, 8 (2015), 85--94. Google ScholarDigital Library
- Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. 2010. VizWiz: Nearly real-time answers to visual questions. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology. ACM, 333--342. Google ScholarDigital Library
- Kevin J. Boudreau, Nicola Lacetera, and Karim R. Lakhani. 2011. Incentives and problem uncertainty in innovation contests: An empirical analysis. Manage. Sci. 57, 5 (2011), 843--863. Google ScholarDigital Library
- Randy Burkett. 2013. An Alternative Framework for Agent Recruitment: From MICE to RASCLS. Retrieved from https://calhoun.nps.edu/handle/10945/43831.Google Scholar
- Amparo Elizabeth Cano Basave, Andrea Varga, Matthew Rowe, Milan Stankovic, and Aba-Sah Dadzie. 2013. Making sense of microposts (# MSM2013) concept extraction challenge. In Proceedings of the Concept Extraction Challenge at the Workshop on ‘Making Sense of Microposts’ Co-Located with the 22nd International World Wide Web Conference (WWW’13), Amparo E. Cano, Matthew Rowe, Milan Stankovic, and Aba-Sah Dadzie (Eds.). CEUR-WS.org, 1--15.Google Scholar
- Shuchi Chawla, Jason D. Hartline, and Balasubramanian Sivan. 2015. Optimal crowdsourcing contests. Games Econ. Behav. 113 (2015), 80--96.Google ScholarCross Ref
- S. Cooper, F. Khatib, A. Treuille, J. Barbero, J. Lee, M. Beenen, A. Leaver-Fay, D. Baker, Z. Popović, et al. 2010. Predicting protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756--760.Google ScholarCross Ref
- Peng Dai, Jeffrey M. Rzeszotarski, Praveen Paritosh, and Ed H. Chi. 2015. And now for something completely different: Improving crowdsourcing workflows with micro-diversions. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 628--638.Google Scholar
- Emmanuel Dechenaux, Dan Kovenock, and Roman M. Sheremeta. 2015. A survey of experimental research on contests, all-pay auctions and tournaments. Exper. Econ. 18, 4 (2015), 609--669.Google ScholarCross Ref
- Edward L. Deci and Richard M. Ryan. 2010. Self-Determination. John Wiley 8 Sons.Google Scholar
- Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux. 2013. Large-scale linked data integration using probabilistic reasoning and crowdsourcing. Very Large Data Base J. 22, 5 (2013), 665--687. Google ScholarDigital Library
- Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011. From game design elements to gamefulness: Defining gamification. In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments. ACM, 9--15.Google ScholarDigital Library
- Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G. Ipeirotis, and Philippe Cudré-Mauroux. 2015. The dynamics of micro-task crowdsourcing: The case of Amazon MTurk. In Proceedings of the 24th International Conference on World Wide Web. 238--247. Google ScholarDigital Library
- Oluwaseyi Feyisetan, Thomas Drake, Borja Balle, and Tom Diethe. 2019. Privacy-preserving active learning on sensitive data for user intent classification. arXiv preprint arXiv:1903.11112.Google Scholar
- Oluwaseyi Feyisetan, Markus Luczak-Rösch, Elena Simperl, Ramine Tinati, and Nigel Shadbolt. 2015. Towards hybrid NER: A study of content and crowdsourcing-related performance factors. In The Semantic Web: Latest Advances and New Domains. Springer, 525--540.Google Scholar
- Oluwaseyi Feyisetan and Elena Simperl. 2016. Please stay vs let’s play: Social pressure incentives in paid collaborative crowdsourcing. In Proceedings of the International Conference on Web Engineering. Springer, 405--412.Google ScholarCross Ref
- Oluwaseyi Feyisetan and Elena Simperl. 2017. Social incentives in paid collaborative crowdsourcing. ACM Trans. Intell. Syst. Technol. 8, 6 (2017), Article no 73.Google ScholarDigital Library
- Oluwaseyi Feyisetan, Elena Simperl, Markus Luczak-Roesch, Ramine Tinati, and Nigel Shadbolt. {n.d.}. An extended study of content and crowdsourcing-related performance factors in named entity annotation. Semantic Web 1--25.Google Scholar
- Oluwaseyi Feyisetan, Elena Simperl, Ramine Tinati, Markus Luczak-Roesch, and Nigel Shadbolt. 2014. Quick-and-clean extraction of linked data entities from microblogs. In Proceedings of the 10th International Conference on Semantic Systems (SEM’14). ACM, 5--12.Google ScholarDigital Library
- Oluwaseyi Feyisetan, Elena Simperl, Max Van Kleek, and Nigel Shadbolt. 2015. Improving paid microtasks through gamification and adaptive furtherance incentives. In Proceedings of the 24th International Conference on World Wide Web. 333--343.Google ScholarDigital Library
- Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating named entities in Twitter data with crowdsourcing. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. Association for Computational Linguistics, 80--88.Google ScholarDigital Library
- Brent Frei. 2009. Paid crowdsourcing. Current State 8 Progress toward Mainstream Business Use, Smartsheet.com Report, Smartsheet.com 9 (2009).Google Scholar
- Bruno S. Frey and Reto Jegen. 2001. Motivation crowding theory. J. Econ. Surveys 15, 5 (2001), 589--611.Google ScholarCross Ref
- Drew Fudenberg and Jean Tirole. 1986. A theory of exit in duopoly. Econometr.: J. Econometr. Soc. (1986), 943--960.Google Scholar
- Ujwal Gadiraju, Ricardo Kawase, and Stefan Dietze. 2014. A taxonomy of microtasks on the web. In Proceedings of the 25th ACM Conference on Hypertext and Social Media. ACM, 218--223. Google ScholarDigital Library
- Huiji Gao, Geoffrey Barbier, and Rebecca Goolsby. 2011. Harnessing the crowdsourcing power of social media for disaster relief. IEEE Intell. Syst. 3 (2011), 10--14.Google ScholarDigital Library
- Benjamin M. Good and Andrew I. Su. 2011. Games with a scientific purpose. Genome Biol. 12, 12 (2011), 1.Google ScholarCross Ref
- Jeff Howe. 2006. The rise of crowdsourcing. Wired Mag. 14, 6 (2006), 1--4.Google Scholar
- Lilly C. Irani and M. Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 611--620.Google Scholar
- Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker motivation in crowdsourcing-A study on mechanical turk. In Proceedings of the Americas Conference on Information Systems (AMCIS’11), Vol. 11. 1--11.Google Scholar
- Aniket Kittur. 2010. Crowdsourcing, collaboration and creativity. ACM Crossroads 17, 2 (2010), 22--26.Google ScholarDigital Library
- Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. ACM, 1301--1318. Google ScholarDigital Library
- Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM, 43--52. Google ScholarDigital Library
- Markus Krause and René Kizilcec. 2015. To play or not to play: Interactions between response quality and task complexity in games and paid crowdsourcing. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing.Google Scholar
- Robert E. Kraut, Paul Resnick, Sara Kiesler, Moira Burke, Yan Chen, Niki Kittur, Joseph Konstan, Yuqing Ren, and John Riedl. 2012. Building Successful Online Communities: Evidence-based Social Design. MIT Press. Google ScholarDigital Library
- Vijay Krishna and John Morgan. 1997. An analysis of the war of attrition and the all-pay auction. J. Econ. Theory 72, 2 (1997), 343--362.Google ScholarCross Ref
- Walter S. Lasecki, Christopher Homan, and Jeffrey P. Bigham. 2014. Architecting real-time crowd-powered systems. Hum. Comput. 1, 1 (2014).Google Scholar
- Walter S. Lasecki, Christopher Miller, Adam Sadilek, Andrew Abumoussa, Donato Borrello, Raja Kushalnagar, and Jeffrey Bigham. 2012. Real-time captioning by groups of non-experts. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology. ACM, 23--34. Google ScholarDigital Library
- Walter S. Lasecki, Christopher D. Miller, and Jeffrey P. Bigham. 2013. Warping time for more effective real-time crowdsourcing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2033--2036. Google ScholarDigital Library
- Walter S. Lasecki, Kyle I. Murray, Samuel White, Robert C. Miller, and Jeffrey P. Bigham. 2011. Real-time crowd control of existing interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM, 23--32. Google ScholarDigital Library
- Edith Law, Ming Yin, Kevin Chen Joslin Goh, Michael Terry, and Krzysztof Z. Gajos. 2016. Curiosity killed the cat, but makes crowdwork better. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4098--4110. Google ScholarDigital Library
- Tracy Xiao Liu, Jiang Yang, Lada A. Adamic, and Yan Chen. 2011. Crowdsourcing with all-pay auctions: A field experiment on Taskcn. Proc. Assoc. Info. Sci. Technol. 48, 1 (2011), 1--4.Google ScholarCross Ref
- Thomas W. Malone, Robert Laubacher, and Chrysanthos Dellarocas. 2010. The collective intelligence genome. MIT Sloan Manage. Rev. 51, 3 (2010), 21.Google Scholar
- David Martin, Benjamin V. Hanrahan, Jacki O’Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 224--235. Google ScholarDigital Library
- Winter Mason and Duncan J. Watts. 2010. Financial incentives and the performance of crowds. ACM SigKDD Explor. Newslett. 11, 2 (2010), 100--108. Google ScholarDigital Library
- Elisa D. Mekler, Florian Brühlmann, Klaus Opwis, and Alexandre N. Tuch. 2013. Disassembling gamification: The effects of points and meaning on user motivation and performance. In Proceedings of the Conference on Human Factors in Computing Systems—Extended Abstracts (CHI’13). ACM, 1137--1142.Google Scholar
- Pietro Michelucci. 2016. Handbook of Human Computation. Springer. Google ScholarDigital Library
- Benny Moldovanu and Aner Sela. 2001. The optimal allocation of prizes in contests. Amer. Econ. Rev. (2001), 542--558.Google Scholar
- Benny Moldovanu, Aner Sela, and Xianwen Shi. 2012. Carrots and sticks: Prizes and punishments in contests. Econ. Inquiry 50, 2 (2012), 453--462.Google ScholarCross Ref
- B. Norrander. 2006. The attrition game: Initial resources, initial contests and the exit of candidates during the U.S. presidential primary season. Brit. J. Polit. Sci. 36, 3 (2006), 487--507.Google ScholarCross Ref
- Oded Nov, Ofer Arazy, and David Anderson. 2014. Scientists@ home: What drives the quantity and quality of online citizen science participation? PloS One 9, 4 (2014), e90375.Google ScholarCross Ref
- Jordan Raddick, Georgia Bracey, Pamela L. Gay, Chris J. Lintott, Phil Murray, Kevin Schawinski, Alexander S. Szalay, and Jan Vandenberg. 2010. Galaxy zoo: Exploring the motivations of citizen science volunteers. Astron. Edu. Rev. 9, 1 (2010).Google Scholar
- Neal Reeves, Peter West, and Elena Simperl. 2018. “A game without competition is hardly a game”: The impact of competitions on player activity in a human computation game. In Proceedings of the 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP’18). AAAI.Google Scholar
- Alan Ritter, Sam Clark, Oren Etzioni, et al. 2011. Named entity recognition in tweets: An experimental study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1524--1534. Google ScholarDigital Library
- Markus Rokicki, Sergiu Chelaru, Sergej Zerr, and Stefan Siersdorfer. 2014. Competitive game designs for improving the cost effectiveness of crowdsourcing. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. ACM, 1469--1478.Google ScholarDigital Library
- Markus Rokicki, Sergej Zerr, and Stefan Siersdorfer. 2015. Groupsourcing: Team competition designs for crowdsourcing. In Proceedings of the 24th International Conference on World Wide Web. 906--915. Google ScholarDigital Library
- Razvan Rughinis. 2013. Gamification for productive interaction: Reading and working with the gamification debate in education. In Proceedings of the 8th Iberian Conference on Information Systems and Technologies (CISTI’13). IEEE, 1--5.Google Scholar
- Katie Seaborn and Deborah I. Fels. 2015. Gamification in theory and action: A survey. Int. J. Hum.-Comput. Studies 74 (2015), 14--31. Google ScholarDigital Library
- Burr Settles. 2012. Active learning. Synth. Lect. Artific. Intell. Mach. Learn. 6, 1 (2012), 1--114.Google ScholarCross Ref
- Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 614--622. Google ScholarDigital Library
- John C. Tang, Manuel Cebrian, Nicklaus A. Giacobe, Hyun-Woo Kim, Taemie Kim, and Douglas Beaker Wickert. 2011. Reflecting on the DARPA red balloon challenge. Commun. ACM 54, 4 (2011), 78--85. Google ScholarDigital Library
- Ramine Tinati, Markus Luczak-Roesch, Elena Simperl, and Wendy Hall. 2017. An investigation of player motivations in Eyewire, a gamified citizen science project. Comput. Hum. Behav. 73 (2017), 527--540. Google ScholarDigital Library
- Sudheendra Vijayanarasimhan and Kristen Grauman. 2014. Large-scale live active learning: Training object detectors with crawled data and crowds. Int. J. Comput. Vis. 108, 1--2 (2014), 97--114. Google ScholarDigital Library
- Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’04). ACM, New York, NY, 319--326.Google ScholarDigital Library
- Luis Von Ahn and Laura Dabbish. 2008. Designing games with a purpose. Commun. ACM 51, 8 (2008), 58--67. Google ScholarDigital Library
- Xianzhong Xie, Helin Yang, Athanasios V. Vasilakos, and Lu He. 2014. Fair power control using game theory with pricing scheme in cognitive radio networks. J. Commun. Netw. 16, 2 (2014), 183--192.Google ScholarCross Ref
- Yan Yan, Romer Rosales, Glenn Fung, and Jennifer G. Dy. 2011. Active learning from crowds. In Proceedings of the International Conference on Machine Learning (ICML’11), Vol. 11. 1161--1168.Google ScholarDigital Library
- Dejun Yang, Xi Fang, and Guoliang Xue. 2012. Game theory in cooperative communications. IEEE Wireless Commun. 19, 2 (2012).Google Scholar
- Gabe Zichermann. 2011. Gamification has issues, but they aren’t the ones everyone focuses on. O’ReillyRadar.Google Scholar
- Matthew Zook, Mark Graham, Taylor Shelton, and Sean Gorman. 2010. Volunteered geographic information and crowdsourcing disaster relief: A case study of the Haitian earthquake. World Med. Health Policy 2, 2 (2010), 7--33.Google ScholarCross Ref
Index Terms
- Beyond Monetary Incentives: Experiments in Paid Microtask Contests
Recommendations
Social Incentives in Paid Collaborative Crowdsourcing
Survey Paper, Regular Papers and Special Issue: Social Media ProcessingPaid microtask crowdsourcing has traditionally been approached as an individual activity, with units of work created and completed independently by the members of the crowd. Other forms of crowdsourcing have, however, embraced more varied models, which ...
Improving Paid Microtasks through Gamification and Adaptive Furtherance Incentives
WWW '15: Proceedings of the 24th International Conference on World Wide WebCrowdsourcing via paid microtasks has been successfully applied in a plethora of domains and tasks. Previous efforts for making such crowdsourcing more effective have considered aspects as diverse as task and workflow design, spam detection, quality ...
Optimal Crowds Contest Model for Crowdsourcing
ICCSE'17: Proceedings of the 2nd International Conference on Crowd Science and EngineeringWith the increasing frequency of participation in social networking activities, tremendous value has been created by crowds. Thus some emerging industries come along with it to collect these values. At the same time, crowds require some compensation ...
Comments