Danula Hettiachchi
Zhanna Sarsenbayeva
Niels van Berkel
Tilman Dingler
Vassilis Kostakos
ABSTRACT
Inspired by the increasing prevalence of digital voice assistants, we demonstrate the feasibility of using voice interfaces to deploy and complete crowd tasks. We have developed Crowd Tasker, a novel system that delivers crowd tasks through a digital voice assistant. In a lab study, we validate our proof-of-concept and show that crowd task performance through a voice assistant is comparable to that of a web interface for voice-compatible and voice-based crowd tasks for native English speakers. We also report on a field study where participants used our system in their homes. We find that crowdsourcing through voice can provide greater flexibility to crowd workers by allowing them to work in brief sessions, enabling multi-tasking, and reducing the time and effort required to initiate tasks. We conclude by proposing a set of design guidelines for the creation of crowd tasks for voice and the development of future voice-based crowdsourcing systems.
- Utku Günay Acer, Marc van den Broeck, Claudio Forlivesi, Florian Heller, and Fahim Kawsar. 2019. Scaling Crowdsourcing with Mobile Workforce: A Case Study with Belgian Postal Service. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 2, Article 35 (June 2019), 32 pages. DOI: http://dx.doi.org/10.1145/3328906Google Scholar
Digital Library
- Fraser Allison, Joshua Newn, Wally Smith, Marcus Carter, and Martin Gibbs. 2019. Frame Analysis of Voice Interaction Gameplay. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 393, 14 pages. DOI: http://dx.doi.org/10.1145/3290605.3300623Google ScholarDigital Library
- Tiago A. Almeida, José María Gómez Hidalgo, and Tiago P. Silva. 2013. Towards SMS Spam Filtering: Results under a New Dataset. International Journal of Information Security Science 2, 1 (2013), 1 -- 18.Google Scholar
- Frank Bentley, Chris Luvogt, Max Silverman, Rushani Wirasinghe, Brooke White, and Danielle Lottridge. 2018. Understanding the Long-Term Use of Smart Speaker Assistants. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 3, Article 91 (Sept. 2018), 24 pages. DOI: http://dx.doi.org/10.1145/3264901Google ScholarDigital Library
- Janine Berg. 2015. Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comp. Lab. L. & Pol'y J. 37 (2015), 543.Google Scholar
- Michael Braun, Anja Mainz, Ronee Chadowitz, Bastian Pfleging, and Florian Alt. 2019. At Your Service: Designing Voice Assistant Personalities to Improve Automotive User Interfaces. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 40, 11 pages. DOI: http://dx.doi.org/10.1145/3290605.3300270Google ScholarDigital Library
- Tara McAllister Byun, Peter F. Halpin, and Daniel Szeredi. 2015. Online crowdsourcing for efficient rating of speech: A validation study. Journal of Communication Disorders 53 (2015), 70 -- 83. DOI: http://dx.doi.org/10.1016/j.jcomdis.2014.11.003Google ScholarCross Ref
- Chris Callison-Burch and Mark Dredze. 2010. Creating Speech and Language Data with Amazon's Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk (CSLDAMT '10). Association for Computational Linguistics, Stroudsburg, PA, USA, 1--12.Google ScholarDigital Library
- Veena Chattaraman, Wi-Suk Kwon, Juan E. Gilbert, and Kassandra Ross. 2019. Should AI-Based, conversational digital assistants employ social- or task-oriented interaction style? A task-competency and reciprocity perspective for older adults. Computers in Human Behavior 90 (2019), 315 -- 330. DOI: http://dx.doi.org/10.1016/j.chb.2018.08.048Google ScholarCross Ref
- Justin Cheng, Jaime Teevan, Shamsi T. Iqbal, and Michael S. Bernstein. 2015. Break It Down: A Comparison of Macro- and Microtasks. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, NY, NY, USA, 4061--4064. DOI: http://dx.doi.org/10.1145/2702123.2702146Google ScholarDigital Library
- Lydia B. Chilton, John J. Horton, Robert C. Miller, and Shiri Azenkot. 2010. Task Search in a Human Computation Market. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '10). ACM, NY, NY, USA, 1--9. DOI: http://dx.doi.org/10.1145/1837885.1837889Google ScholarDigital Library
- Leigh Clark. 2018. Social Boundaries of Appropriate Speech in HCI: A Politeness Perspective. In Proceedings of the 32nd International BCS Human Computer Interaction Conference (HCI '18). BCS Learning & Development Ltd., Swindon, UK, Article 76, 5 pages. DOI: http://dx.doi.org/10.14236/ewic/HCI2018.76Google ScholarDigital Library
- Leigh Clark, João Cabral, and Benjamin Cowan. 2018. The CogSIS Project: Examining the Cognitive Effects of Speech Interface Synthesis. In Proceedings of the 32nd International BCS Human Computer Interaction Conference (HCI '18). BCS Learning & Development Ltd., Swindon, UK, Article 169, 3 pages. DOI: http://dx.doi.org/10.14236/ewic/HCI2018.170Google ScholarDigital Library
- Leigh Clark, Philip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl, Jens Edlund, Matthew Aylett, João Cabral, Cosmin Munteanu, Justin Edwards, and Benjamin R. Cowan. 2019a. The State of Speech in HCI: Trends, Themes and Challenges. Interacting with Computers (09 2019). DOI: http://dx.doi.org/10.1093/iwc/iwz016Google ScholarCross Ref
- Leigh Clark, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin Edwards, Brendan Spillane, Emer Gilmartin, Christine Murad, Cosmin Munteanu, Vincent Wade, and Benjamin R. Cowan. 2019b. What Makes a Good Conversation?: Challenges in Designing Truly Conversational Agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 475, 12 pages. DOI: http://dx.doi.org/10.1145/3290605.3300705Google ScholarDigital Library
- Benjamin R. Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017. "What Can I Help You with?": Infrequent Users' Experiences of Intelligent Personal Assistants. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, NY, NY, USA, Article 43, 12 pages. DOI: http://dx.doi.org/10.1145/3098279.3098539Google ScholarDigital Library
- Fred J. Damerau. 1964. A Technique for Computer Detection and Correction of Spelling Errors. Commun. ACM 7, 3 (1964), 171--176. DOI: http://dx.doi.org/10.1145/363958.363994Google ScholarDigital Library
- Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G. Ipeirotis, and Philippe Cudré-Mauroux. 2015. The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk. In Proceedings of the 24th International Conference on World Wide Web (WWW '15). ACM, NY, NY, USA, 617--617. DOI: http://dx.doi.org/10.1145/2740908.2744109Google ScholarDigital Library
- Djellel Eddine Difallah, Gianluca Demartini, and Philippe Cudré-Mauroux. 2013. Pick-a-crowd: Tell Me What You Like, and I'll Tell You What to Do. In Proceedings of the 22nd International Conference on World Wide Web (WWW '13). ACM, NY, NY, USA, 367--374. DOI: http://dx.doi.org/10.1145/2488388.2488421Google ScholarDigital Library
- Djellel Eddine Difallah, Gianluca Demartini, and Philippe Cudré-Mauroux. 2016. Scheduling Human Intelligence Tasks in Multi-Tenant Crowd-Powered Systems. In Proceedings of the 25th International Conference on World Wide Web (WWW '16). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 855--865. DOI: http://dx.doi.org/10.1145/2872427.2883030Google ScholarDigital Library
- Maxine Eskenazi, Gina-Anne Levow, Helen Meng, Gabriel Parent, and David Suendermann. 2013. Crowdsourcing for speech processing: Applications to data collection, transcription and assessment. John Wiley & Sons.Google ScholarDigital Library
- Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederic Font, Dmitry Bogdanov, Andrés Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra. 2017. Freesound Datasets: a platform for the creation of open audio datasets. In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR 2017). Suzhou, China, 486--493.Google Scholar
- Ujwal Gadiraju, Gianluca Demartini, Ricardo Kawase, and Stefan Dietze. 2018. Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection. Computer Supported Cooperative Work (CSCW) (Jun 2018). DOI: http://dx.doi.org/10.1007/s10606-018--9336-yGoogle ScholarCross Ref
- Ujwal Gadiraju, Ricardo Kawase, and Stefan Dietze. 2014. A Taxonomy of Microtasks on the Web. In Proceedings of the 25th ACM Conference on Hypertext and Social Media (HT '14). ACM, NY, NY, USA, 218--223. DOI: http://dx.doi.org/10.1145/2631775.2631819Google ScholarDigital Library
- Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio Set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 776--780. DOI: http://dx.doi.org/10.1109/ICASSP.2017.7952261Google ScholarDigital Library
- Jorge Goncalves, Michael Feldman, Subingqian Hu, Vassilis Kostakos, and Abraham Bernstein. 2017. Task Routing and Assignment in Crowdsourcing Based on Cognitive Abilities. In Proceedings of the 26th International Conference on World Wide Web (WWW '17). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 1023--1031. DOI: http://dx.doi.org/10.1145/3041021.3055128Google ScholarDigital Library
- Jorge Goncalves, Denzil Ferreira, Simo Hosio, Yong Liu, Jakob Rogstadius, Hannu Kukka, and Vassilis Kostakos. 2013. Crowdsourcing on the Spot: Altruistic Use of Public Displays, Feasibility, Performance, and Behaviours. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '13). ACM, NY, NY, USA, 753--762. DOI: http://dx.doi.org/10.1145/2493432.2493481Google ScholarDigital Library
- Jorge Goncalves, Simo Hosio, Niels van Berkel, Furqan Ahmed, and Vassilis Kostakos. 2017. CrowdPickUp: Crowdsourcing Task Pickup in the Wild. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 51 (Sept. 2017), 22 pages. DOI: http://dx.doi.org/10.1145/3130916Google ScholarDigital Library
- Jorge Goncalves, Hannu Kukka, Iván Sánchez, and Vassilis Kostakos. 2016. Crowdsourcing Queue Estimations in Situ. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, NY, NY, USA, 1040--1051. DOI: http://dx.doi.org/10.1145/2818048.2819997Google ScholarDigital Library
- Lisa Graham. 2008. Gestalt theory in interactive media design. Journal of Humanities & Social Sciences 2, 1 (2008).Google Scholar
- Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P. Bigham. 2018. A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, Article 449, 14 pages. DOI: http://dx.doi.org/10.1145/3173574.3174023Google ScholarDigital Library
- Seyyed Hadi Hashemi, Kyle Williams, Ahmed El Kholy, Imed Zitouni, and Paul A. Crook. 2018. Measuring User Satisfaction on Smart Speaker Intelligent Assistants Using Intent Sensitive Query Embeddings. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM '18). ACM, NY, NY, USA, 1183--1192. DOI: http://dx.doi.org/10.1145/3269206.3271802Google ScholarDigital Library
- Danula Hettiachchi, Niels van Berkel, Tilman Dingler, Fraser Allison, Vassilis Kostakos, and Jorge Goncalves. 2019a. Enabling Creative Crowd Work through Smart Speakers. In Workshop on Designing Crowd-powered Creativity Support Systems (CHI '19 Workshop). 1--5. DOI: http://dx.doi.org/10.5281/zenodo.2648986Google ScholarCross Ref
- Danula Hettiachchi, Niels van Berkel, Simo Hosio, Vassilis Kostakos, and Jorge Goncalves. 2019b. Effect of Cognitive Abilities on Crowdsourcing Task Performance. In Human-Computer Interaction -- INTERACT 2019. Springer International Publishing, Cham, 442--464. DOI: http://dx.doi.org/10.1007/978--3-030--29381--9_28Google ScholarDigital Library
- Simo Hosio, Jorge Goncalves, Vili Lehdonvirta, Denzil Ferreira, and Vassilis Kostakos. 2014. Situated Crowdsourcing Using a Market Model. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, NY, NY, USA, 55--64. DOI: http://dx.doi.org/10.1145/2642918.2647362Google ScholarDigital Library
- Thivya Kandappu, Nikita Jaiman, Randy Tandriansyah, Archan Misra, Shih-Fen Cheng, Cen Chen, Hoong Chuin Lau, Deepthi Chander, and Koustuv Dasgupta. 2016. TASKer: Behavioral Insights via Campus-based Experimental Mobile Crowd-sourcing. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). ACM, NY, NY, USA, 392--402. DOI: http://dx.doi.org/10.1145/2971648.2971690Google ScholarDigital Library
- Gabriella Kazai, Jaap Kamps, and Natasa Milic-Frayling. 2012. The Face of Quality in Crowdsourcing Relevance Labels: Demographics, Personality and Labeling Accuracy. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management (CIKM '12). ACM, NY, NY, USA, 2583--2586. DOI: http://dx.doi.org/10.1145/2396761.2398697Google ScholarDigital Library
- Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Hassan Awadallah, Aidan C. Crook, Imed Zitouni, and Tasos Anastasakos. 2016. Understanding User Satisfaction with Intelligent Assistants. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (CHIIR '16). ACM, NY, NY, USA, 121--130. DOI: http://dx.doi.org/10.1145/2854946.2854961Google ScholarDigital Library
- Ahmet Baki Kocaballi, Liliana Laranjo, and Enrico Coiera. 2019. Understanding and Measuring User Experience in Conversational Interfaces. Interacting with Computers 31, 2 (05 2019), 192--207. DOI: http://dx.doi.org/10.1093/iwc/iwz015Google ScholarCross Ref
- Arne Köhn, Florian Stegen, and Timo Baumann. 2016. Mining the Spoken Wikipedia for Speech Data and Beyond. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016) (23--28). European Language Resources Association (ELRA), Paris, France.Google Scholar
- Il-Youp Kwak, Jun Ho Huh, Seung Taek Han, Iljoo Kim, and Jiwon Yoon. 2019. Voice Presentation Attack Detection Through Text-Converted Voice Command Analysis. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 598, 12 pages. DOI: http://dx.doi.org/10.1145/3290605.3300828Google ScholarDigital Library
- Josephine Lau, Benjamin Zimmerman, and Florian Schaub. 2018. Alexa, Are You Listening?: Privacy Perceptions, Concerns and Privacy-seeking Behaviors with Smart Speakers. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 102 (Nov. 2018), 31 pages. DOI: http://dx.doi.org/10.1145/3274371Google ScholarDigital Library
- Adrian Leemann, Marie-José Kolly, and David Britain. 2018. The English Dialects App: The creation of a crowdsourced dialect corpus. Ampersand 5 (2018), 1 -- 17. DOI: http://dx.doi.org/10.1016/j.amper.2017.11.001Google ScholarCross Ref
- Irene Lopatovska, Katrina Rink, Ian Knight, Kieran Raines, Kevin Cosenza, Harriet Williams, Perachya Sorsche, David Hirsch, Qi Li, and Adrianna Martinez. 2019. Talk to me: Exploring user interactions with the Amazon Alexa. Journal of Librarianship and Information Science 51, 4 (2019), 984--997. DOI: http://dx.doi.org/10.1177/0961000618759414Google ScholarCross Ref
- Michal Luria, Guy Hoffman, and Oren Zuckerman. 2017. Comparing Social Robot, Screen and Voice Interfaces for Smart-Home Control. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, NY, NY, USA, 580--628. DOI: http://dx.doi.org/10.1145/3025453.3025786Google ScholarDigital Library
- Christine Murad, Cosmin Munteanu, Leigh Clark, and Benjamin R. Cowan. 2018. Design Guidelines for Hands-free Speech Interaction. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI '18). ACM, NY, NY, USA, 269--276. DOI: http://dx.doi.org/10.1145/3236112.3236149Google ScholarDigital Library
- Christine Murad, Cosmin Munteanu, Benjamin R. Cowan, and Leigh Clark. 2019. Revolution or Evolution? Speech Interaction and HCI Design Guidelines. IEEE Pervasive Computing 18, 02 (apr 2019), 33--45. DOI: http://dx.doi.org/10.1109/MPRV.2019.2906991Google ScholarDigital Library
- Chelsea Myers, Anushay Furqan, Jessica Nebolsky, Karina Caro, and Jichen Zhu. 2018. Patterns for How Users Overcome Obstacles in Voice User Interfaces. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, Article 6, 7 pages. DOI: http://dx.doi.org/10.1145/3173574.3173580Google ScholarDigital Library
- Chelsea M. Myers, Anushay Furqan, and Jichen Zhu. 2019. The Impact of User Characteristics and Preferences on Performance with an Unfamiliar Voice User Interface. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 47, 9 pages. DOI: http://dx.doi.org/10.1145/3290605.3300277Google ScholarDigital Library
- Christi Olson and Kelli Kemery. 2019. Voice Report: From answers to action: customer adoption of voice technology and digital assistants. (2019). https://advertiseonbing-blob.azureedge.net/blob/ bingads/media/insight/whitepapers/2019/04%20apr/voice-report/bingads_2019voicereport.pdfGoogle Scholar
- Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 527--536. DOI: http://dx.doi.org/10.18653/v1/P19--1050Google ScholarCross Ref
- Aung Pyae and Paul Scifleet. 2018. Investigating Differences Between Native English and Non-native English Speakers in Interacting with a Voice User Interface: A Case of Google Home. In Proceedings of the 30th Australian Conference on Computer-Human Interaction (OzCHI '18). ACM, NY, NY, USA, 548--553. DOI: http://dx.doi.org/10.1145/3292147.3292236Google ScholarDigital Library
- Julie Rico. 2010. Evaluating the Social Acceptability of Multimodal Mobile Interactions. In CHI '10 Extended Abstracts on Human Factors in Computing Systems (CHI EA '10). ACM, NY, NY, USA, 2887--2890. DOI: http://dx.doi.org/10.1145/1753846.1753877Google ScholarDigital Library
- Jeffrey M. Rzeszotarski and Aniket Kittur. 2011. Instrumenting the Crowd: Using Implicit Behavioral Measures to Predict Task Performance. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, NY, NY, USA, 13--22. DOI: http://dx.doi.org/10.1145/2047196.2047199Google ScholarDigital Library
- Clay Shirky. 2010. Cognitive surplus: How technology makes consumers into collaborators. Penguin, UK.Google Scholar
- Ben Shneiderman. 2000. The Limits of Speech Recognition. Commun. ACM 43, 9 (Sept. 2000), 63--65. DOI: http://dx.doi.org/10.1145/348941.348990Google ScholarDigital Library
- Kinga Skorupska, Manuel Nunez, Wieslaw Kopec, and Radoslaw Nielek. 2018. Older Adults and Crowdsourcing: Android TV App for Evaluating TEDx Subtitle Quality. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 159 (Nov. 2018), 23 pages. DOI: http://dx.doi.org/10.1145/3274428Google ScholarDigital Library
- David R. Thomas. 2006. A General Inductive Approach for Analyzing Qualitative Evaluation Data. American Journal of Evaluation 27, 2 (2006), 237--246. DOI: http://dx.doi.org/10.1177/1098214005283748Google ScholarCross Ref
- Cassia Valentini-Botinhao, Xin Wang, Shinji Takaki, and Junichi Yamagishi. 2016. Speech Enhancement for a Noise-Robust Text-to-Speech Synthesis System Using Deep Recurrent Neural Networks. In Interspeech. 352--356. DOI: http://dx.doi.org/10.21437/Interspeech.2016--159Google ScholarCross Ref
- Aditya Vashistha, Abhinav Garg, and Richard Anderson. 2019. ReCall: Crowdsourcing on Basic Phones to Financially Sustain Voice Forums. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 169, 13 pages. DOI: http://dx.doi.org/10.1145/3290605.3300399Google ScholarDigital Library
- Aditya Vashistha, Pooja Sethi, and Richard Anderson. 2017. Respeak: A Voice-based, Crowd-powered Speech Transcription System. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, NY, NY, USA, 1855--1866. DOI: http://dx.doi.org/10.1145/3025453.3025640Google ScholarDigital Library
- Aditya Vashistha, Pooja Sethi, and Richard Anderson. 2018. BSpeak: An Accessible Voice-based Crowdsourcing Marketplace for Low-Income Blind People. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, Article 57, 13 pages. DOI: http://dx.doi.org/10.1145/3173574.3173631Google ScholarDigital Library
- Yan Yan, Romer Rosales, Glenn Fung, and Jennifer G. Dy. 2011. Active Learning from Crowds. In Proceedings of the 28th International Conference on International Conference on Machine Learning (ICML'11). ACM, NY, NY, USA, 1161--1168.Google ScholarDigital Library
Index Terms
- "Hi! I am the Crowd Tasker" Crowdsourcing through Digital Voice Assistants
Recommendations
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments
The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Make Hay While the Crowd Shines: Towards Efficient Crowdsourcing on the Web
WWW '15 Companion: Proceedings of the 24th International Conference on World Wide WebWithin the scope of this PhD proposal, we set out to investigate two pivotal aspects that influence the effectiveness of crowdsourcing: (i) microtask design, and (ii) workers behavior. Leveraging the dynamics of tasks that are crowdsourced on the one ...
Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection
AbstractThe suitability of crowdsourcing to solve a variety of problems has been investigated widely. Yet, there is still a lack of understanding about the distinct behavior and performance of workers within microtasks. In this paper, we first introduce a ...
Comments