Benjamin V. Hanrahan
Saiph Savage
Skip Abstract Section
Abstract
Crowdworkers depend on Amazon Mechanical Turk (AMT) as an important source of income and it is left to workers to determine which tasks on AMT are fair and worth completing. While there are existing tools that assist workers in making these decisions, workers still spend significant amounts of time finding fair labor. Difficulties in this process may be a contributing factor in the imbalance between the median hourly earnings ($2.00/hour) and what the average requester pays ($11.00/hour). In this paper, we study how novices and experts select what tasks are worth doing. We argue that differences between the two populations likely lead to the wage imbalances. For this purpose, we first look at workers' comments in TurkOpticon (a tool where workers share their experience with requesters on AMT). We use this study to start to unravel what fair labor means for workers. In particular, we identify the characteristics of labor that workers consider is of "good quality'' and labor that is of "poor quality'' (e.g., work that pays too little.) Armed with this knowledge, we then conduct an experiment to study how experts and novices rate tasks that are of both good and poor quality. Through our research we uncover that experts and novices both treat good quality labor in the same way. However, there are significant differences in how experts and novices rate poor quality labor, and whether they believe the poor quality labor is worth doing. This points to several future directions, including machine learning models that support workers in detecting poor quality labor, and paths for educating novice workers on how to make better labor decisions on AMT.
- Antonio Aloisi. 2015. Commoditized workers: Case study research on labor law issues arising from a set of on-demand/gig economy platforms. Comp. Lab. L. & Pol'y J. 37 (2015), 653.Google Scholar
- Janine Berg, Marianne Furrer, Ellie Harmon, Uma Rani, and S Silberman. 2018. Digital labour platforms and the future of work. Towards Decent Work in the Online World. Rapport de l'OIT (2018).Google Scholar
- Birgitta Bergvall-Kåreborn and Debra Howcroft. 2014. A mazon Mechanical Turk and the commodification of labour. New Technology, Work and Employment 29, 3 (2014), 213--223.Google ScholarCross Ref
- Alice M Brawley and Cynthia LS Pury. 2016. Work experiences on MTurk: Job satisfaction, turnover, and information sharing. Computers in Human Behavior 54 (2016), 531--546.Google ScholarDigital Library
- Amy Bruckman. 2002. Studying the amateur artist: A perspective on disguising data collected in human subjects research on the Internet. Ethics and Information Technology 4, 3 (2002), 217--231.Google ScholarDigital Library
- Michael D Buhrmester, Sanaz Talaifar, and Samuel D Gosling. 2018. An evaluation of Amazon's Mechanical Turk, its rapid rise, and its effective use. Perspectives on Psychological Science 13, 2 (2018), 149--154.Google ScholarCross Ref
- Chris Callison-Burch. 2014. Crowd-Workers: Aggregating Information Across Turkers To Help Them Find Higher Paying Work. In HCOMP-2014.Google Scholar
- Chun-Wei Chiang, Anna Kasunic, and Saiph Savage. 2018. Crowd Coach: Peer Coaching for Crowd Workers' Skill Growth. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 37.Google ScholarDigital Library
- Djellel Difallah, Elena Filatova, and Panos Ipeirotis. 2018. Demographics and Dynamics of Mechanical Turk Workers. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (WSDM '18). ACM, New York, NY, USA, 135--143. https://doi.org/10.1145/3159652.3159661Google ScholarDigital Library
- Kinda El Maarry, Kristy Milland, and Wolf-Tilo Balke. 2018. A Fair Share of the Work?: The Evolving Ecosystem of Crowd Workers. In Proceedings of the 10th ACM Conference on Web Science. ACM, 145--152.Google ScholarDigital Library
- Christian Fieseler, Eliane Bucher, and Christian Pieter Hoffmann. 2019. Unfairness by Design? The Perceived Fairness of Digital Labor on Crowdworking Platforms. Journal of Business Ethics 156, 4 (01 Jun 2019), 987--1005. https: //doi.org/10.1007/s10551-017-3607-2Google ScholarCross Ref
- Ujwal Gadiraju and Gianluca Demartini. 2019. Understanding Worker Moods and Reactions to Rejection in Crowd-sourcing. In Proceedings of the 30th ACM Conference on Hypertext and Social Media (HT '19). Association for Computing Machinery, New York, NY, USA, 211--220. https://doi.org/10.1145/3342220.3343644Google ScholarDigital Library
- Ujwal Gadiraju, Jie Yang, and Alessandro Bozzon. 2017. Clarity is a Worthwhile Quality: On the Role of Task Clarity in Microtask Crowdsourcing. In Proceedings of the 28th ACM Conference on Hypertext and Social Media (HT '17). ACM, New York, NY, USA, 5--14. https://doi.org/10.1145/3078714.3078715Google ScholarDigital Library
- SR Gouravajhala, YOUXUAN Jiang, Preetraj Kaur, Jarir Chaar, and Walter S Lasecki. 2018. Finding mnemo: Hybrid intelligence memory in a crowd-powered dialog system. In Collective Intelligence Conference (CI 2018). Zurich, Switzerland.Google Scholar
- Mary L Gray, Siddharth Suri, Syed Shoaib Ali, and Deepti Kulkarni. 2016. The crowd is a collaborative network. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing. ACM, 134--147.Google ScholarDigital Library
- Neha Gupta, David Martin, Benjamin V Hanrahan, and Jacki O'Neill. 2014. Turk-life in India. In Proceedings of the 18th International Conference on Supporting Group Work. ACM, 1--11.Google ScholarDigital Library
- Lei Han, Eddy Maddalena, Alessandro Checco, Cristina Sarasua, Ujwal Gadiraju, Kevin Roitero, and Gianluca Demartini. 2020. Crowd Worker Strategies in Relevance Judgment Tasks. In Proceedings of the 13th International Conference on Web Search and Data Mining (WSDM '20). Association for Computing Machinery, New York, NY, USA, 241--249. https://doi.org/10.1145/3336191.3371857Google ScholarDigital Library
- Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini. 2019. All those wasted hours: On task abandonment in crowdsourcing. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 321--329.Google ScholarDigital Library
- Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini. 2019. The impact of task abandonment in crowdsourcing. IEEE Transactions on Knowledge and Data Engineering (2019).Google ScholarCross Ref
- Benjamin V Hanrahan, David Martin, Jutta Willamowski, and John M Carroll. 2018. Investigating the Amazon Mechanical Turk Market Through Tool Design. Computer Supported Cooperative Work (CSCW) 27, 3-6 (2018), 1255--1274.Google ScholarDigital Library
- Benjamin V Hanrahan, Jutta K Willamowski, Saiganesh Swaminathan, and David B Martin. 2015. TurkBench: Rendering the market for Turkers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1613--1616.Google ScholarDigital Library
- Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. 2018. A data-driven analysis of workers' earnings on amazon mechanical turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 449.Google ScholarDigital Library
- Larry V Hedges. 1981. Distribution theory for Glass's estimator of effect size and related estimators. journal of Educational Statistics 6, 2 (1981), 107--128.Google Scholar
- Paul Hitlin. 2016. Research in the Crowdsourcing Age, a Case Study. Pew Research Center. https://www.pewinternet. org/2016/07/11/research-in-the-crowdsourcing-age-a-case-study/Google Scholar
- Chien-Ju Ho, Aleksandrs Slivkins, Siddharth Suri, and Jennifer Wortman Vaughan. 2015. Incentivizing high quality crowdwork. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 419--429.Google ScholarDigital Library
- Joses Ho, Tayfun Tumkaya, Sameer Aryal, Hyungwon Choi, and Adam Claridge-Chang. 2019. Moving beyond P values: data analysis with estimation graphics. Nature Methods 16, 7 (2019), 565--566. https://doi.org/10.1038/s41592-019-0470-3Google ScholarCross Ref
- Debra Howcroft and Birgitta Bergvall-Kåreborn. 2018. A typology of crowdwork platforms. Work, Employment and Society (2018), 0950017018760136.Google Scholar
- Lilly Irani. 2015. Difference and dependence among digital workers: The case of Amazon Mechanical Turk. South Atlantic Quarterly 114, 1 (2015), 225--234.Google ScholarCross Ref
- Lilly C Irani and M Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 611--620.Google ScholarDigital Library
- Ayush Jain, Akash Das Sarma, Aditya Parameswaran, and Jennifer Widom. 2017. Understanding Workers, Developing Effective Tasks, and Enhancing Marketplace Dynamics: A Study of a Large Crowdsourcing Marketplace. Proc. VLDB Endow. 10, 7 (March 2017), 829--840. https://doi.org/10.14778/3067421.3067431Google ScholarDigital Library
- David Johnstone, Mary Tate, and Erwin Fielt. 2018. Taking rejection personally: An ethical analysis of work rejection on Amazon Mechanical Turk. (2018).Google Scholar
- Toni Kaplan, Susumu Saito, Kotaro Hara, and Jeffrey P Bigham. 2018. Striving to earn more: a survey of work strategies and tool use among crowd workers. In Sixth AAAI Conference on Human Computation and CrowdsourcingGoogle ScholarCross Ref
- Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 1301--1318.Google ScholarDigital Library
- Leib Litman, Jonathan Robinson, and Cheskie Rosenzweig. 2015. The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on Mechanical Turk. Behavior research methods 47, 2 (2015), 519--528.Google Scholar
- Bingjie Liu and S Shyam Sundar. 2018. Microworkers as research participants: Does underpaying Turkers lead to cognitive dissonance? Computers in Human Behavior 88 (2018), 61--69.Google ScholarDigital Library
- Matt Lovett, Saleh Bajaba, Myra Lovett, and Marcia J Simmering. 2018. Data Quality from Crowdsourced Surveys: A Mixed Method Inquiry into Perceptions of Amazon's Mechanical Turk Masters. Applied Psychology 67, 2 (2018), 339--366.Google ScholarCross Ref
- VK Chaithanya Manam and Alexander J. Quinn. 2018. Wingit: Efficient refinement of unclear task instructions. In In Sixth AAAI Conference on Human Computation and Crowdsourcing.Google Scholar
- David Martin, Benjamin V Hanrahan, Jacki O'Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 224--235.Google ScholarDigital Library
- David Martin, Jacki O'Neill, Neha Gupta, and Benjamin V Hanrahan. 2016. Turking in a global labour market. Computer Supported Cooperative Work (CSCW) 25, 1 (2016), 39--77.Google ScholarDigital Library
- Ted Matherly. 2019. A panel for lemons? Positivity bias, reputation systems and data quality on MTurk. European Journal of Marketing 53, 2 (2019), 195--223. https://doi.org/10.1108/EJM-07-2017-0491 arXiv:https://doi.org/10.1108/EJM-07- 2017-0491Google ScholarCross Ref
- Brian McInnis, Dan Cosley, Chaebong Nam, and Gilly Leshed. 2016. Taking a HIT: Designing around rejection, mistrust, risk, and workers' experiences in Amazon Mechanical Turk. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, 2271--2282.Google ScholarDigital Library
- Eyal Peer, Joachim Vosgerau, and Alessandro Acquisti. 2014. Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior research methods 46, 4 (2014), 1023--1031.Google Scholar
- Lisa Posch, Arnim Bleier, Clemens Lechner, Daniel Danner, Fabian Flöck, and Markus Strohmaier. 2017. Measuring Motivations of Crowdworkers: The Multidimensional Crowdworker Motivation Scale. (2017). arXiv:cs.SI/1702.01661Google Scholar
- Susumu Saito, Chun-Wei Chiang, Saiph Savage, Teppei Nakano, Tetsunori Kobayashi, and Jeffrey Bigham. 2019. TurkScanner: Predicting the Hourly Wage of Microtasks. arXiv preprint arXiv:1903.07032 (2019).Google Scholar
- Shruti Sannon and Dan Cosley. 2018. It was a shady HIT: Navigating Work-Related Privacy Concerns on MTurk. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, LBW507.Google ScholarDigital Library
- Shruti Sannon and Dan Cosley. 2019. Privacy, Power, and Invisible Labor on Amazon Mechanical Turk. (2019).Google Scholar
- Saiph Savage, Chun Wei Chiang, Susumu Saito, Carlos Toxtli, and Jeffrey Bigham. 2020. Becoming the Super Turker: Increasing Wages via a Strategy from High Earning Workers. In Proceedings of The Web Conference 2020. 1241--1252.Google ScholarDigital Library
- Kim Bartel Sheehan. 2018. Crowdsourcing research: data collection with Amazon's Mechanical Turk. Communication Monographs 85, 1 (2018), 140--156.Google ScholarCross Ref
- M. Silberman and Lilly Irani. 2016. Operating an employer reputation system: lessons from TurkOpticon, 2008 - 2015. Comparative Labor Law & Policy Journal, Forthcoming. https://ssrn.com/abstract=2729498Google Scholar
- M Six Silberman, Bill Tomlinson, Rochelle LaPlante, Joel Ross, Lilly Irani, and Andrew Zaldivar. 2018. Responsible research with crowds: pay crowdworkers at least minimum wage. Commun. ACM 61, 3 (2018), 39--41.Google ScholarDigital Library
- Vanessa Williamson. 2016. On the ethics of crowdsourced research. PS: Political Science & Politics 49, 1 (2016), 77--81.Google ScholarCross Ref
- Alex J. Wood, Vili Lehdonvirta, and Mark Graham. [n. d.]. Workers of the Internet unite? Online freelancer organisation among remote gig economy workers in six Asian and African countries. New Technology, Work and Employment 33, 2 ([n. d.]), 95--112. https://doi.org/10.1111/ntwe.12112 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/ntwe.12112Google ScholarCross Ref
- Meng-Han Wu and Alexander James Quinn. 2017. Confusing the crowd: Task instruction quality on amazon mechanical turk. In Fifth AAAI Conference on Human Computation and Crowdsourcing.Google ScholarCross Ref
- Jie Yang, Carlo van der Valk, Tobias Hoßfeld, Judith Redi, and Alessandro Bozzon. 2018. How Do Crowdworker Communities and Microtask Markets Influence Each Other? A Data-Driven Study on Amazon Mechanical Turk. In Sixth AAAI Conference on Human Computation and CrowdsourcingGoogle Scholar
Index Terms
- The Expertise Involved in Deciding which HITs are Worth Doing on Amazon Mechanical Turk
Recommendations
A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsA growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 ...
Privacy, Power, and Invisible Labor on Amazon Mechanical Turk
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsTasks on crowdsourcing platforms such as Amazon Mechanical Turk often request workers' personal information, raising privacy risks that may be exacerbated by requester-worker power dynamics. We interviewed 14 workers to understand how they navigate ...
Worker Demographics and Earnings on Amazon Mechanical Turk: An Exploratory Analysis
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing SystemsPrior research reported that workers on Amazon Mechanical Turk (AMT) are underpaid, earning about $2/h. But the prior research did not investigate the difference in wage due to worker characteristics (e.g., country of residence). We present the first ...
Comments