Lixiu Yu
Aniket Kittur
Robert Kraut
ABSTRACT
A significant challenge for crowdsourcing has been increasing worker engagement and output quality. We explore the effects of social, learning, and financial strategies, and their combinations, on increasing worker retention across tasks and change in the quality of worker output. Through three experiments, we show that 1) using these strategies together increased workers' engagement and the quality of their work; 2) a social strategy was most effective for increasing engagement; 3) a learning strategy was most effective in improving quality. The findings of this paper provide strategies for harnessing the crowd to perform complex tasks, as well as insight into crowd workers' motivation.
- Amabile, T. M. Motivational synergy: Toward new conceptualizations of intrinsic and extrinsic motivation in the workplace. Human Resource Management Review, 3(3), (1993), 185--20Google Scholar
Cross Ref
- Baumeister, R.F. and Leary, M.R. The need to belong: desire for interpersonal attachments as a fundamental human motivation. Psychological bulletin 117, 3 (1995), 497.Google Scholar
- Bowen, D.E. and Lawler, E.E. Total quality-oriented human resources management. Organizational Dynamics 20, 4 (1992), 29--41.Google ScholarCross Ref
- Chandler, D. and Kapelner, A. Breaking monotony with meaning: Motivation in crowdsourcing markets. University of Chicago mimeo, 2010.Google Scholar
- Choi, B., Alexander, K., Kraut, R. E. and Levine J. M., Socialization tactics in Wikipedia and their effects. In Proc. CSCW 2010. Google ScholarDigital Library
- Collins, N. L. and Miller, L. C., Self-disclourse and Liking: a meta-analytic review. Psychological bulletin 116, 3 (1994), 457--475.Google Scholar
- Deci, E.L., Koestner, R., and Ryan, R.M. A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological bulletin 125, 6 (1999), 62Google Scholar
- Dow, S., Kulkarni, A., Bunge, B., Nguyen, T., Klemmer, S., and Hartmann, B. Shepherding the crowd: managing and providing feedback to crowd workers. In Proc. CHI EA 2011. Google ScholarDigital Library
- Ertmer, P. A., Richardson, J. C., Belland, B., Camin, D., Connolly, P., Coulthard, G., Lei, K., and Mong, C., Using peer feedback to enhance the quality of student online postings: an exploratory study. JCMC, 12(2), (2007).Google Scholar
- Jehn, K. A. The dynamic nature of conflict: a longitudinal study of intragroup conflict and group performance. Academy of Management Journal 44, 2(2001), 238--251.Google Scholar
- Galton, F. The geometric mean, in vital and social statistics. Proceedings of the Royal Society of London 29, 196--199 (1879), 365--367.Google Scholar
- Hackman, J.R. and Oldham, G.R. Motivation through the design of work: Test of a theory. Organizational behavior and human quality 16, 2 (1976), 250--279.Google Scholar
- Heckman, J.J. Sample selection bias as a specification error. Econometrica: Journal of the econometric society, (1979), 153--161.Google Scholar
- Hill, C.A. Affiliation motivation: People who need people? but in different ways. Journal of personality and social psychology 52, 5 (1987), 1008.Google Scholar
- Hsieh, G., Hudson, S.E., and Kraut, R.E. Donate for credibility: how contribution strategies can improve credibility. In Proc. CHI 2011, 3435--3438. Google ScholarDigital Library
- Huang, S.-W. and Fu, W.-T. Don't Hide in the Crowd! Increasing Social Transparency Between Peer Workers Improves Crowdsourcing Outcomes. In Proc. CHI 2013. Google ScholarDigital Library
- Jenkins, G. D., Mitra, A., Gupta, N. and Shaw, J. D., Are financial incentives related to performance? A meta-analytic review of empirical research. Journal of Applied Psychology 83, 5(1998), 777--787.Google ScholarCross Ref
- Jensen, M. and Murphy, K., Competition and Incentives: Practice vs Theory. Journal of Finance 43 (1988), 593--616.Google ScholarCross Ref
- Kinnaird, P., Dabbish, L., Kiesler, S., and Faste, H. Co-worker transparency in a microtask marketplace. In Proc. CSCW 2013, 1285--1290. Google ScholarDigital Library
- Kittur, A., Nickerson, J. V., Bernstein, M. S., Gerber, E. M., Shaw, A., Zimmerman, J., Lease, M., and Horton, J. J. The future of crowd work. In Proc. CSCW 2013. Google ScholarDigital Library
- Kittur, A., Smus, B., Khamkar, S., and Kraut, R.E. Crowdforge: Crowdsourcing complex work. In Proc. UIST 2011. Google ScholarDigital Library
- Lasecki, W., Murray, K., White, S., Miller, R. C., and Bigham,J. P. Real-time crowd control of existing interfaces. In Proc. UIST 2011, 23--32. Google ScholarDigital Library
- Lave, J. and Wenger, E. Situated Learning: Legitimate Peripheral Participation. Cambridge University Press, 1991.Google ScholarCross Ref
- Lawler III, E.E. 3. Job design and employee motivation. Personnel Psychology 22, 4 (1969), 426--435.Google ScholarCross Ref
- Leana, C. R., & Van Buren, H. J. Organizational social capital and employment practices. Academy of Management Review 24, (1999)., 538--555.Google Scholar
- Mason, W. and Watts, D.J. Financial strategies and the quality of crowds. ACM SigKDD Explorations Newsletter 11, 2 (2010), 100--108. Google ScholarDigital Library
- Meyer, J. P., Paunonen, S. V., Gellatly, I. R., Goffin, R. D. and Jackson, D. N. Organizational commitment and job performance: It's the nature of the commitment that counts. Journal of Applied Psychology 74(1), 1989, 152--156.Google ScholarCross Ref
- Nov, O. What motivates Wikipedians? Communications of the ACM 50, 11(2007), 60--64. Google ScholarDigital Library
- Prentice, D. A., Miller, D. T., and LightdaleI,J. R. Asymmetries in attachments to groups and to their members: Distinguishing between common-identity and common-bond groups. Personality and Social Psychology Bulletin 20, 5 (1994), 484--49.Google ScholarCross Ref
- Podolny, J. M., & Baron, J. N. Relationships and resources: Social networks and mobility in the workplace. American Sociological Review, 62, (1997), 673--693.Google ScholarCross Ref
- Randel, A.E. and Ranft, A. L. Motivations to maintain social ties with coworkers. Group and Organization Management 32, 2(2007), 208--232.Google ScholarCross Ref
- Ren, Y., Kraut, R. E., Kiesler, S., & Resnick, P. (2012). Encouraging commitment in online communities. In R. E. Kraut & P. Resnick (Eds.), Building successful online communities: Evidence-based social design (pp. 77--124). Cambridge MA: MIT Press.Google Scholar
- Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., and Vukovic, M. An assessment of intrinsic and extrinsic motivation on task quality in crowdsourcing markets. In Proc. ICWSM 2011.Google Scholar
- Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., and Tomlinson, B. (2010). Who are the crowdworkers': shifting demographics in mechanical turk. In CHI EA'10: Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, pp. 2863--2872, New York, NY, USA. ACM. Google ScholarDigital Library
- Rousseau, D.M. Technological differences in job characteristics, employee satisfaction, and motivation: A synthesis of job design research and sociotechnical systems theory. Organizational Behavior and Human Performance 19, 1 (1977), 18--42.Google ScholarCross Ref
- Salganik, M.J., Dodds, P.S., and Watts, D.J. Experimental study of inequality and unpredictability in an artificial cultural market. Science 311, 5762 (2006), 854--856.Google ScholarCross Ref
- Shaw, A.D., Horton, J.J., and Chen, D.L. Designing strategies for inexpert human raters. In Proc. CSCW 2011. Google ScholarDigital Library
- Tang, T. L. and Sarsfield-Baldwin, L. The effects of self-esteem, task label, and performance feedback on task liking and intrinsic motivation. The Journal of Social Psychology 131, 4 (1991), 567--572.Google ScholarCross Ref
- Tang, J.C., Cebrian, M., Giacobe, N.A., Kim, H.W., Kim, T., and Wickert, D.B. Reflecting on the Darpa red balloon challenge. Comm. of the ACM 54, 4 (2011), 78--85. Google ScholarDigital Library
- Wang, Y., Kraut, R., and Levine, J. To stay or leave? The incentive of emotional and informational support to commitment in online health support groups. In Proc. CSCW 2011. Google ScholarDigital Library
- Wiggins, G., Educative assessment. Designing assessments to inform and improve student quality. Jossey-Bass Publishers, CA, 1998.Google Scholar
- Zhu, H., Kraut, R., and Kittur, A. Effectiveness of shared leadership in online communities. In Proc. CSCW 2012. Google ScholarDigital Library
- Zhu, H., Kraut, R.E., & Kittur, A. Effects of Peer Feedback on Contribution: A Field Experiment in Wikipedia. In Proc. CHI 2013. Google ScholarDigital Library
Index Terms
- A comparison of social, learning, and financial strategies on crowd engagement and output quality
Recommendations
Exploring Engagement in a 'Social Crowd' on Twitter
CSCW '16 Companion: Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing CompanionPeople use Twitter to communicate with companies and friends, and ask for or share information. Here, we explore participation and engagement in an information-gathering 'social crowd'. We first present a study of engagement behavior through both ...
Strategies to Enhance Technology-Based Learning Experiences in Older Adults: A Field Study
Human Aspects of IT for the Aged Population. Social Media, Games and Assistive EnvironmentsAbstractThe aim of this study is to explore the main aspects that affect pre/during and post-learning experiences in later age. Specifically, the study examined the problems adult learners encountered in learning how to use digital technologies, ...
Strategies for Crowdworkers to Overcome Barriers in Competition-based Software Crowdsourcing Development
ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering WorkshopsCrowdsourcing in software development uses a large pool of developers on-demand to outsource parts or the entire software project to a crowd. To succeed, this requires a continuous influx of developers, or simply crowdworkers. However, crowdworkers face ...
Comments