Neha Gupta
ABSTRACT
Previous studies on Amazon Mechanical Turk (AMT), the most well-known marketplace for microtasks, show that the largest population of workers on AMT is U.S. based, while the second largest is based in India. In this paper, we present insights from an ethnographic study conducted in India to introduce some of these workers or "Turkers" -- who they are, how they work and what turking means to them. We examine the work they do to maintain their reputations and their work-life balance. In doing this, we illustrate how AMT's design practically impacts on turk-work. Understanding the "lived work" of crowdwork is a valuable first step for technology design.
- Bederson, B. B. and Quinn, A. J. 2011. Web workers unite! addressing challenges of online laborers. In Proceedings of CHI EA '11. ACM (2011), 97--106. Google Scholar
Digital Library
- Bourne, K. A., and Forman, P. J. (2013). Living in a Culture of Overwork: An Ethnographic Study of Flexibility. Journal of Management Inquiry, 25 March 2013, DOI: 10.1177/1056492613481245Google Scholar
- Crabtree, A. 2003. Designing Collaborative Systems: A Practical Guide to Ethnography, Springer. Google ScholarDigital Library
- Crabtree, A., Nichols, D. M., O'Brien, J., Rouncefield, M., & Twidale, M. B. 2000. Ethnomethodologically informed ethnography and information system design. Journal of the American Society for Information Science, 51(7), 666--682. Google ScholarDigital Library
- Felsteiner, A. 2011. Working the crowd: employment and labor law in the crowdsourcing industry. Berkeley Journal of Employment & Labor Law 32(1), 143--204.Google Scholar
- Felstiner, A. 2013. The weakness of crowds. Limn, Crowds and Clouds (2), Retrieved from http://limn.it/the-weakness-ofcrowds/.Google Scholar
- Garfinkel, H., 1967. Studies in ethnomethodology. Englewood Cliffs, N.J.: Prentice-Hall.Google Scholar
- Gawade, M., Vaish, R., Waihumbu, M, N., Davis, J., 2012. Exploring Employment Opportunities through Microtasks via Cybercafes. In Proceedings of IEEE GHTC'12 77--82. Google ScholarDigital Library
- Howe, J., 2006. The rise of crowdsourcing. Wired Magazine Issue 14.06. Available at http://www.wired.com/wired/archive/14.06/crowds.htmlGoogle Scholar
- Hughes: Hughes, J. A., Randall, D., & Shapiro, D. (1992, December). Faltering from ethnography to design. In Proceedings of the 1992 ACM conference on Computer-supported cooperative work, 115--122. Google ScholarDigital Library
- Hughes, J., King, V., Rodden, T., Andersen, H. (1994) Moving Out from the Control Room: Ethnography in System Design. In Proceedings of ACM CSCW'94 Conference on Computer-Supported Cooperative Work, 429--439. Google ScholarDigital Library
- Ipeirotis, P. 2010. Analyzing the Amazon Mechanical Turk marketplace. CeDER Working Papers No. CeDER-10-04. Available at http://hdl.handle.net/2451/29801Google Scholar
- Ipeirotis, P. 2010. Demographics of Mechanical Turk. CeDER Working Papers No. CeDER-10-01. http://hdl.handle.net/2451/29585Google Scholar
- Irani, L. and Silberman, M. S. 2013. Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. In Proceedings of CHI 2013, ACM Press. Google ScholarDigital Library
- Jensen, R. E., & Bjørn, P. (2012). Divergence and Convergence in Global Software Development: Cultural Complexities as Social Worlds. From research to practice in the design of cooperative systems. pp. 123--136Google Scholar
- Kaufmann, N., Schulze, T., and Veit, D. 2011. More than fun and money. Worker Motivation in Crowdsourcing -- A Study on Mechanical Turk. In Proceedings of AMCIS 2011.Google Scholar
- Khanna, S., Ratan, A., Davis, J., Thies, W. 2010. Evaluating and Improving the Usability of Mechanical Turk for Low-Income Workers in India. In Proceedings of ACM DEV 2010. Google ScholarDigital Library
- Kittur,A., Chi, E. H., and Suh, B. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the CHI '08. ACM, 453--456. Google ScholarDigital Library
- Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M. and Horton, J. 2013. The future of crowd work. In Proceedings of CSCW '13, ACM press, 1301--1318. Google ScholarDigital Library
- Kochhar, S., Mazzochi, S., Paritosh, P. 2010. The Anatomy of a Large-Scale Human Computation Engine. HCOMP'10. ACM press. 10--17. Google ScholarDigital Library
- Le, J., Edmonds, A., Hester, V., Biewald, L. 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. SIGIR'10. 17--20.Google Scholar
- Little, G., Chilton, L. B., Goldman, M. and Miller, R. C. 2009. Turkit: tools for iterative tasks on mechanical turk. In Proceedings of KDD-HCOMP '09, 2009. Google ScholarDigital Library
- Martin, D., Hanrahan, B. J., O'Neill, J., Gupta, N., 2014. Being a Turker. In Proceedings of ACM CSCW '14, 224--235. Google ScholarDigital Library
- O'Neill, J., Roy, S., Grasso, A., Martin, D., 2013. Form Digitization in BPO: From outsourcing to crowdsourcing? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'13). Google ScholarDigital Library
- O'Neill J., Martin D. 2013. Relationship-based Business Process Crowdsourcing? In Proceedings of IFIP Conference on Human-Computer Interaction'13. 429--446.Google ScholarCross Ref
- Pontin, J., 2007. Artificial Intelligence, With Help From the Humans. The New York Times. Available at http://www.nytimes.com/2007/03/25/business/yourmoney/25Stream.htmlGoogle Scholar
- Randall, D., Harper, R. & Rouncefield, M. (2007). Fieldwork for Design: Theory and Practice. Springer Verlag, New York. Google ScholarDigital Library
- Ratan, A. L., Satpathy, S., Zia, L., Toyama, K., Blagsvedt, S., Pawar, U. S., Subramaniam, T., 2009. Kelsa+: digital literacy for low-income office workers. Proceedings of the 3rd International Conference on Information and Communication Technologies and Development, 2009. Google ScholarDigital Library
- Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., and Tomlinson, B. 2010. Who are the crowdworkers': shifting demographics in mechanical turk. In Proceedings of CHI EA '10, ACM (2010), 2863--2872. Google ScholarDigital Library
- Shaw, A., 2010. For love or for money? A list experiment on the motivations behind crowdsourcing work. Available at https://crowdflower.com/blog/2010/08/for-love-or-for-money-a-list-experiment-on-the-motivations-behind-crowdsourcing-work/Google Scholar
- Silberman, M. S. 2010. What's fair? Rational action and its residuals in an electronic market. Unpublished manuscript. http://www.scribd.com/doc/86592724/Whats-FairGoogle Scholar
- Silberman, M. S., Irani, L., and Ross, J. 2010. Ethics and Tactics of Professional Crowdwork. ACM XRDS, 17, 2. Google ScholarDigital Library
- Sommerville, I., Rodden, T., Sawyer, P. & Bentley, R. 1992. Sociologists can be surprisingly useful in interactive systems design. In proceedings of HCI '92, CUP. People and computers, 341--341. Google ScholarDigital Library
- Star, S.L. and Strauss, A.L. 1998. Layers of silence, arenas of voice: The ecology of visible and invisible work?. CSCW Journal, 8: 9--30. Google ScholarDigital Library
- Tanney, J. (2013) Rules, Reason and Self-knowledge. Harvard University PressGoogle Scholar
- Thies, W., Ratan, A., Davis, J.: 2011. Paid crowdsourcing as a vehicle for global development. In: CHI '11 Workshop on Crowdsourcing and Human ComputationGoogle Scholar
Index Terms
- Turk-Life in India
Recommendations
Turking in a Global Labour Market
This paper examines how working in the global labour market of Amazon Mechanical Turk (AMT) impacts upon and is understood by two different groups of workers. To do this we report on two qualitative studies; one of US and another of Indian crowdworkers (...
Mechanical turk as an ontology engineer?: using microtasks as a component of an ontology-engineering workflow
WebSci '13: Proceedings of the 5th Annual ACM Web Science ConferenceOntology evaluation has proven to be one of the more difficult problems in ontology engineering. Researchers proposed numerous methods to evaluate logical correctness of an ontology, its structure, or coverage of a domain represented by a corpus. ...
Turker Tales: Integrating Tangential Play into Crowd Work
DIS '19: Proceedings of the 2019 on Designing Interactive Systems ConferenceWhile past work has admirably supported crowd workers in improving their work performance, we argue that there is also value in designing for enjoyment untied from work outcomes--- what we call "tangential play.'' To this end, we present Turker Tales, a ...
Comments