nach oben

Erschienen in:

2019 | OriginalPaper | Buchkapitel

Effect of Cognitive Abilities on Crowdsourcing Task Performance

verfasst von : Danula Hettiachchi, Niels van Berkel, Simo Hosio, Vassilis Kostakos, Jorge Goncalves

Erschienen in: Human-Computer Interaction – INTERACT 2019

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Matching crowd workers to suitable tasks is highly desirable as it can enhance task performance, reduce the cost for requesters, and increase worker satisfaction. In this paper, we propose a method that considers workers’ cognitive ability to predict their suitability for a wide range of crowdsourcing tasks. We measure cognitive ability via fast-paced online cognitive tests with a combined average duration of 6.2 min. We then demonstrate that our proposed method can effectively assign or recommend workers to five different popular crowd tasks: Classification, Counting, Proofreading, Sentiment Analysis, and Transcription. Using our approach we demonstrate a significant improvement in the expected overall task accuracy. While previous methods require access to worker history or demographics, our work offers a quick and accurate way to determine which workers are more suitable for which tasks.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel A Literature Review of the Practice of Educating Children About Technology Making

Nächstes Kapitel Insights on Older Adults’ Attitudes and Behavior Through the Participatory Design of an Online Storytelling Platform

https://www.mturk.com.

https://www.metmuseum.org/art/collection.

https://www.flickr.com.

Alagarai Sampath, H., Rajeshuni, R., Indurkhya, B.: Cognitively inspired task design to improve user performance on crowdsourcing platforms. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2014, pp. 3665–3674. ACM, New York (2014). https://doi.org/10.1145/2556288.2557155

Bailey, C.E.: Cognitive accuracy and intelligent executive function in the brain and in business. Ann. N. Y. Acad. Sci. 1118, 122–141 (2007). https://doi.org/10.1196/annals.1412.011CrossRef

Borella, E., Carretti, B., Pelegrina, S.: The specific role of inhibition in reading comprehension in good and poor comprehenders. J. Learn. Disabil. 43(6), 541–552 (2010). https://doi.org/10.1177/0022219410371676CrossRef

Chilton, M.A., Hardgrave, B.C., Armstrong, D.J.: Person-job cognitive style fit for software developers: the effect on strain and performance. J. Manag. Inf. Syst. 22(2), 193–226 (2005). https://doi.org/10.1080/07421222.2005.11045849CrossRef

Clair-Thompson, H.L.S., Gathercole, S.E.: Executive functions and achievements in school: shifting, updating, inhibition, and working memory. Q. J. Exp. Psychol. 59(4), 745–759 (2006). https://doi.org/10.1080/17470210500162854CrossRef

Crump, M.J.C., McDonnell, J.V., Gureckis, T.M.: Evaluating Amazon’s mechanical Turk as a tool for experimental behavioral research. PLoS ONE 8(3), 1–18 (2013). https://doi.org/10.1371/journal.pone.0057410CrossRef

Damerau, F.J.: A technique for computer detection and correction of spelling errors. Commun. ACM 7(3), 171–176 (1964)CrossRef

Deshpande, M., Karypis, G.: Item-based Top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1), 143–177 (2004). https://doi.org/10.1145/963770.963776CrossRef

Diamond, A.: Executive functions. Annu. Rev. Psychol. 64(1), 135–168 (2013). https://doi.org/10.1146/annurev-psych-113011-143750CrossRef

10.

Difallah, D.E., Catasta, M., Demartini, G., Ipeirotis, P.G., Cudré-Mauroux, P.: The dynamics of micro-task crowdsourcing: the case of Amazon MTurk. In: Proceedings of the 24th International Conference on World Wide Web, WWW 2015, pp. 238–247. IW3C2, Switzerland (2015). https://doi.org/10.1145/2736277.2741685

11.

Difallah, D.E., Demartini, G., Cudré-Mauroux, P.: Pick-a-crowd: tell me what you like, and I’ll tell you what to do. In: Proceedings of the 22nd International Conference on World Wide Web, WWW 2013, pp. 367–374. ACM, New York (2013). https://doi.org/10.1145/2488388.2488421

12.

Dingler, T., Schmidt, A., Machulla, T.: Building cognition-aware systems: a mobile toolkit for extracting time-of-day fluctuations of cognitive performance. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1(3) (2017). https://doi.org/10.1145/3132025CrossRef

13.

Downs, J.S., Holbrook, M.B., Sheng, S., Cranor, L.F.: Are your participants gaming the system?: Screening mechanical Turk workers. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2010, pp. 2399–2402. ACM, New York (2010). https://doi.org/10.1145/1753326.1753688

14.

Edwards, J.R.: Person-Job Fit: A Conceptual Integration, Literature Review, and Methodological Critique. Wiley, New York (1991)

15.

Eickhoff, C.: Cognitive biases in crowdsourcing. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, pp. 162–170. ACM, New York (2018). https://doi.org/10.1145/3159652.3159654

16.

Ekstrom, R.B., Dermen, D., Harman, H.H.: Manual for Kit of Factor-referenced Cognitive Tests, vol. 102. Educational Testing Service, Princeton (1976)

17.

Eriksen, B.A., Eriksen, C.W.: Effects of noise letters upon the identification of a target letter in a nonsearch task. Percept. Psychophys. 16(1), 143–149 (1974)CrossRef

18.

Fan, J., Li, G., Ooi, B.C., Tan, K.l., Feng, J.: iCrowd: an adaptive crowdsourcing framework. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD 2015, pp. 1015–1030. ACM, New York (2015). https://doi.org/10.1145/2723372.2750550

19.

Federico, P.A., Landis, D.B.: Cognitive styles, abilities, and aptitudes: are they dependent or independent? Contemp. Educ. Psychol. 9(2), 146–161 (1984). https://doi.org/10.1016/0361-476X(84)90016-XCrossRef

20.

Gadiraju, U., Kawase, R., Dietze, S.: A taxonomy of microtasks on the web. In: Proceedings of the 25th ACM Conference on Hypertext and Social Media, HT 2014, pp. 218–223. ACM, New York (2014)

21.

Germine, L., Nakayama, K., Duchaine, B.C., Chabris, C.F., Chatterjee, G., Wilmer, J.B.: Is the web as good as the lab? Comparable performance from web and lab in cognitive/perceptual experiments. Psychon. Bull. Rev. 19(5), 847–857 (2012). https://doi.org/10.3758/s13423-012-0296-9CrossRef

22.

Goncalves, J., Feldman, M., Hu, S., Kostakos, V., Bernstein, A.: Task routing and assignment in crowdsourcing based on cognitive abilities. In: Proceedings of the 26th International Conference on World Wide Web, WWW 2017, pp. 1023–1031. IW3C2, Switzerland (2017). https://doi.org/10.1145/3041021.3055128

23.

Goncalves, J., et al.: Crowdsourcing on the spot: altruistic use of public displays, feasibility, performance, and behaviours. In: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 2013, pp. 753–762 (2013). https://doi.org/10.1145/2493432.2493481

24.

Goncalves, J., Hosio, S., van Berkel, N., Ahmed, F., Kostakos, V.: CrowdPickUp: crowdsourcing task pickup in the wild. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1(3), 51:1–51:22 (2017). https://doi.org/10.1145/3130916CrossRef

25.

Goncalves, J., Hosio, S., Rogstadius, J., Karapanos, E., Kostakos, V.: Motivating participation and improving quality of contribution in ubiquitous crowdsourcing. Comput. Netw. 90(C), 34–48 (2015). https://doi.org/10.1016/j.comnet.2015.07.002CrossRef

26.

Gureckis, T.M., et al.: psiTurk: an open-source framework for conducting replicable behavioral experiments online. Behav. Res. Methods 48(3), 829–842 (2016). https://doi.org/10.3758/s13428-015-0642-8CrossRef

27.

Hair, J., Black, W., Babin, B., Anderson, R.: Multivariate Data Analysis. Prentice-Hall, Upper Saddle River (2010)

28.

Han, S., Dai, P., Paritosh, P., Huynh, D.: Crowdsourcing human annotation on web page structure: infrastructure design and behavior-based quality control. ACM Trans. Intell. Syst. Technol. 7(4), 56:1–56:25 (2016)CrossRef

29.

Hoffman, B.J., Woehr, D.J.: A quantitative review of the relationship between person-organization fit and behavioral outcomes. J. Vocat. Behav. 68(3), 389–399 (2006). https://doi.org/10.1016/j.jvb.2005.08.003CrossRef

30.

Hosio, S., Goncalves, J., Lehdonvirta, V., Ferreira, D., Kostakos, V.: Situated crowdsourcing using a market model. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST 2014, pp. 55–64. ACM, New York (2014). https://doi.org/10.1145/2642918.2647362

31.

Jain, A., Sarma, A.D., Parameswaran, A., Widom, J.: Understanding workers, developing effective tasks, and enhancing marketplace dynamics: a study of a large crowdsourcing marketplace. Proc. VLDB Endow. 10(7), 829–840 (2017). https://doi.org/10.14778/3067421.3067431CrossRef

32.

Kazai, G.: In search of quality in crowdsourcing for search engine evaluation. In: Clough, P., et al. (eds.) ECIR 2011. LNCS, vol. 6611, pp. 165–176. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20161-5_17CrossRef

33.

Kazai, G., Kamps, J., Milic-Frayling, N.: Worker types and personality traits in crowdsourcing relevance labels. In: Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM 2011, pp. 1941–1944. ACM, New York (2011). https://doi.org/10.1145/2063576.2063860

34.

Kazai, G., Kamps, J., Milic-Frayling, N.: The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, CIKM 2012, pp. 2583–2586. ACM, New York (2012). https://doi.org/10.1145/2396761.2398697

35.

Kazai, G., Zitouni, I.: Quality management in crowdsourcing using gold judges behavior. In: Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, WSDM 2016, pp. 267–276. ACM, New York (2016). https://doi.org/10.1145/2835776.2835835

36.

Kittur, A., et al.: The future of crowd work. In: Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW 2013, pp. 1301–1318. ACM, New York (2013). https://doi.org/10.1145/2441776.2441923

37.

Kristof, A.L.: Person-organization fit: an integrative review of its conceptualizations, measurement, and implications. Pers. Psychol. 49(1), 1–49 (1996)MathSciNetCrossRef

38.

Kristof-Brown, A.L., Zimmerman, R.D., Johnson, E.C.: Consequences of individuals’ fit at work: a meta-analysis of person-job, person-organization, person-group, and person-supervisor fit. Pers. Psychol. 58(2), 281–342 (2005)CrossRef

39.

de Leeuw, J.R.: jsPsych: a javascript library for creating behavioral experiments in a web browser. Behav. Res. Methods 47(1), 1–12 (2015)CrossRef

40.

Liu, X., Lu, M., Ooi, B.C., Shen, Y., Wu, S., Zhang, M.: CDAS: a crowdsourcing data analytics system. Proc. VLDB Endow. 5(10), 1040–1051 (2012)CrossRef

41.

Lykourentzou, I., Antoniou, A., Naudet, Y., Dow, S.P.: Personality matters: balancing for personality types leads to better outcomes for crowd teams. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW 2016, pp. 260–273. ACM, New York (2016)

42.

MacLeod, C.M.: Half a century of research on the stroop effect: an integrative review. Psychol. Bull. 109(2), 163 (1991)CrossRef

43.

Mavridis, P., Gross-Amblard, D., Miklós, Z.: Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing. In: Proceedings of the 25th International Conference on World Wide Web, WWW 2016, Switzerland, pp. 843–853. IW3C2 (2016). https://doi.org/10.1145/2872427.2883070

44.

McInnis, B., Cosley, D., Nam, C., Leshed, G.: Taking a hit: designing around rejection, mistrust, risk, and workers’ experiences in Amazon mechanical Turk. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI 2016, pp. 2271–2282. ACM, New York (2016)

45.

Mioshi, E., Dawson, K., Mitchell, J., Arnold, R., Hodges, J.R.: The Addenbrooke’s cognitive examination revised (ACE-R): a brief cognitive test battery for dementia screening. Int. J. Geriatr. Psychiatry 21(11), 1078–1085 (2006)CrossRef

46.

Mo, K., Zhong, E., Yang, Q.: Cross-task crowdsourcing. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, pp. 677–685. ACM, New York (2013)

47.

Monsell, S.: Task switching. Trends Cogn. Sci. 7(3), 134–140 (2003)CrossRef

48.

Owen, A.M., et al.: Putting brain training to the test. Nature 465, 775 (2010). https://doi.org/10.0.4.14/nature09042CrossRef

49.

Owen, A.M., McMillan, K.M., Laird, A.R., Bullmore, E.: N-back working memory paradigm: a meta-analysis of normative functional neuroimaging studies. Hum. Brain Mapp. 25(1), 46–59 (2005). https://doi.org/10.1002/hbm.20131CrossRef

50.

Petrides, M., Alivisatos, B., Evans, A.C., Meyer, E.: Dissociation of human mid-dorsolateral from posterior dorsolateral frontal cortex in memory processing. Proc. Natl. Acad. Sci. 90(3), 873–877 (1993)CrossRef

51.

Robbins, T.W., James, M., Owen, A.M., Sahakian, B.J., McInnes, L., Rabbitt, P.: Cambridge Neuropsychological Test Automated Battery (CANTAB): a factor analytic study of a large sample of normal elderly volunteers. Dement. Geriatr. Cogn. Disord. 5(5), 266–281 (1994). https://doi.org/10.1159/000106735CrossRef

52.

Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., Vukovic, M.: An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In: Proceedings of the Fifth International AAAI Conference on Web and Social Media, ICWSM, California, USA, vol. 11, pp. 17–21. AAAI (2011)

53.

Ruble, T.L., Cosier, R.A.: Effects of cognitive styles and decision setting on performance. Organ. Behav. Hum. Decis. Process. 46(2), 283–295 (1990). https://doi.org/10.1016/0749-5978(90)90033-6CrossRef

54.

Rzeszotarski, J.M., Kittur, A.: Instrumenting the crowd: using implicit behavioral measures to predict task performance. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST 2011, pp. 13–22. ACM, New York (2011). https://doi.org/10.1145/2047196.2047199

55.

Schmidt, F.L., Hunter, J.: General mental ability in the world of work: occupational attainment and job performance. J. Pers. Soc. Psychol. 86(1), 162 (2004)CrossRef

56.

Shaw, A.D., Horton, J.J., Chen, D.L.: Designing incentives for inexpert human raters. In: Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, CSCW 2011, pp. 275–284. ACM, New York (2011)

57.

Verquer, M.L., Beehr, T.A., Wagner, S.H.: A meta-analysis of relations between person-organization fit and work attitudes. J. Vocat. Behav. 63(3), 473–489 (2003). https://doi.org/10.1016/S0001-8791(02)00036-2CrossRef

58.

Washington, G.: George Washington papers, series 5, financial papers: Copybook of invoices and letters, 1754-1766 (1766). https://www.loc.gov/item/mgw500003

59.

West, R.F., Toplak, M.E., Stanovich, K.E.: Heuristics and biases as measures of critical thinking: associations with cognitive ability and thinking dispositions. J. Educ. Psychol. 100(4), 930 (2008)CrossRef

60.

Zheng, Y., Wang, J., Li, G., Cheng, R., Feng, J.: QASCA: a quality-aware task assignment system for crowdsourcing applications. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD 2015, pp. 1031–1046. ACM, New York (2015)

Titel: Effect of Cognitive Abilities on Crowdsourcing Task Performance
verfasst von: Danula Hettiachchi
Niels van Berkel
Simo Hosio
Vassilis Kostakos
Jorge Goncalves
Verlag: Springer International Publishing
Buch: Human-Computer Interaction – INTERACT 2019
Print ISBN: 978-3-030-29380-2

Electronic ISBN: 978-3-030-29381-9

Copyright-Jahr: 2019
DOI: https://doi.org/10.1007/978-3-030-29381-9_28

Neuer Inhalt

Bildnachweise

VDI-Icon, Profil Icon, inhalt2, Springer Professional Modul/© Springer Fachmedien Wiesbaden GmbH, Die Gewinner und Laudatoren des Sustainability Award in Automotive 2024/© Uli Regenscheit | ATZlive, Search Icon, Banner Hanser, Additiv gefertigte Teile/© Marina_Skoropadskaya | Getty Images | iStock, Warnschild "Land unter"/© Bluedesign / Fotolia, Gardiner von Trapp/© Alpega Group, Zeitschrift Wissensmanagement Cover, PatentFit-Logo/© Springer Fachmedien Wiesbaden GmbH, ATZ-Webinar: Prototypenfreie Entwicklung durch Offline- und Driver-in-the-Loop-HiL-Tests /© (c) VI-grade, chassis.tech plus 2023/© [M] ATZlive / TÜV SÜD PRODUCT SERVICE GMBH, adäsion-Webinar-Matinee/© krystiannawrocki_ Getty Images

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Neuer Inhalt

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.