Joel Ross
Bill Tomlinson
ABSTRACT
Amazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is increasingly popular with researchers and developers. Here we extend previous studies of the demographics and usage behaviors of MTurk workers. We describe how the worker population has changed over time, shifting from a primarily moderate-income, U.S.-based workforce towards an increasingly international group with a significant population of young, well-educated Indian workers. This change in population points to how workers may treat Turking as a full-time job, which they rely on to make ends meet.
- Amazon Mechanical Turk -- Participation Agreement. https://www.mturk.com/mturk/conditionsofuse.Google Scholar
- von Ahn, L. Human computation. Pro. K-CAP, ACM (2007), 5--6. Google ScholarDigital Library
- von Ahn, L. and Dabbish, L. Designing games with a purpose. Commun. ACM 51, 8 (2008), 58--67. Google ScholarDigital Library
- Alonso, O., Rose, D.E., and Stewart, B. Crowdsourcing for relevance evaluation. SIGIR Forum 42, 2 (2008), 9--15. Google ScholarDigital Library
- Barley, S.R. and Kunda, G. Contracting: A new form of professional practice. Perspectives -- Academy of Management 20, 1 (2006), 45.Google Scholar
- Barr, J. and Cabrera, L.F. AI gets a brain. Queue 4, 4 (2006), 24--29. Google ScholarDigital Library
- Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. Proc. CVPR 2009, IEEE (2009), 248--255.Google ScholarCross Ref
- Evans, B.M. and Chi, E.H. Towards a model of understanding social search. Proc. CSCW 2008, ACM (2008), 485--494. Google ScholarDigital Library
- Howe, J. The rise of crowdsourcing. Wired Magazine 14, 6 (2006).Google Scholar
- Ipeirotis, P. Mechanical Turk: The Demographics. A Computer Scientist in a Business School, 2008. http://behind-the-enemy-lines.blogspot.com/2008/03/mechanical-turk-demographics.html.Google Scholar
- Ipeirotis, P. Turker Demographics vs. Internet Demographics. A Computer Scientist in a Business School, 2009. http://behind-the-enemy-lines.blogspot.com/2009/03/turker-demographics-vs-internet.html.Google Scholar
- Irani, L. and Silberman, M.S. "Agency and Exploitation in Mechanical Turk" at Internet as Playground and Factory Conference at The New School. 2009. http://www.slideshare.net/lirani/agency-and-exploitation-in-amazon-mechanical-turk.Google Scholar
- Kittur, A., Chi, E.H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. Proc. CHI 2008, ACM (2008), 453--456. Google ScholarDigital Library
- Paolacci, G. and Massimo, W. Experimental Turk. http://experimentalturk.wordpress.com/.Google Scholar
- Sala, M.C., Partridge, K., Jacobson, L., and Begole, J. An exploration into activity-informed physical advertising using pest. Proc. Pervasive Computing, (2007), 73. Google ScholarDigital Library
- Smith, V. New Forms of Work Organization. Annual Review of Sociology 23, 1 (1997), 315--339.Google Scholar
- Snow, R., O'Connor, B., Jurafsky, D., and Ng, A.Y. Cheap and Fast-But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. Proc. EMNLP 2008, ACL (2008), 254--263. Google ScholarDigital Library
- Sorokin, A. and Forsyth, D. Utility data annotation with Amazon Mechanical Turk. Proc. CVPRW 2008, IEEE (2008), 1--8.Google ScholarCross Ref
- Zittrain, J. Ubiquitous human computing. Phil. Trans. R. Soc. A.: 366, 1881 (2008), 3813--3821.Google ScholarCross Ref
Index Terms
- Who are the crowdworkers?: shifting demographics in mechanical turk
Recommendations
How many crowdsourced workers should a requester hire?
Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a ...
CrowdForge: crowdsourcing complex work
CHI EA '11: CHI '11 Extended Abstracts on Human Factors in Computing SystemsMicro-task markets such as Amazon's Mechanical Turk represent a new paradigm for accomplishing work, in which employers can tap into a large population of workers around the globe to accomplish tasks in a fraction of the time and money of more ...
Pricing mechanisms for crowdsourcing markets
WWW '13: Proceedings of the 22nd international conference on World Wide WebEvery day millions of crowdsourcing tasks are performed in exchange for payments. Despite the important role pricing plays in crowdsourcing campaigns and the complexity of the market, most platforms do not provide requesters appropriate tools for ...
Comments