Top

Journal of the Academy of Marketing Science

Published in:

09-05-2018 | Editorial

“Keep on Turkin’”?

Authors: John Hulland, Jeff Miller

Published in: Journal of the Academy of Marketing Science | Issue 5/2018

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Excerpt

Over the past decade, there has been tremendous growth in academic research use of online samples, coupled with a shift away from traditional sample groups (especially undergraduate college students). In particular, researchers are turning to crowdsourcing platforms like Mechanical Turk (MTurk) to obtain convenient and inexpensive samples. Launched by Amazon in 2005, MTurk is “an online crowdsourcing labor market” (Levay et al. 2016). In their review of work published over four years in the Journal of Consumer Research (volumes 39–42), Goodman and Paolacci (2017) observed that 27% of all surveys and experiments in the journal were conducted using MTurk. Even more strikingly, this proportion of MTurk-based empirical work grew from 9% in the first of the four years they reviewed to 43% in the final year. Similar growth has been observed in other social science fields, including (Chandler and Paolacci 2017, p. 500) “more than 40% of papers published in the Journal of Personality and Social Psychology and more than 20%” in Psychological Science. …

next article Identity change vs. strategy change: the effects of rebranding announcements on stock returns

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

One primary reason for these differences in MTurk use is rooted in the fact that both JMR and especially JCR publish more experimental-based research. Nonetheless, increased use of MTurk can be observed in all four journals over time.

All three authors (McDonald, Mohebbi, Sostek) work for Google. However, their conclusions are based on substantial, credible statistical comparisons that demonstrate good measurement accuracy.

The sample sizes we employ in our field study are quite small. However, they are large enough to illustrate that significant differences in response incidence can be observed through the use of different data sources. The SSI sample size is larger in order to more accurately represent the underlying population.

Before GS became available, Burke used surveys of its own employees to obtain incidence estimates. It did this for cost reasons, despite recognizing the skewed nature of its workforce.

Levay et al. (2016) provide evidence that it may be possible to statistically correct—at least in part—for MTurk sample biases. Specifically, they suggest the use of nine “broad, measureable features”: age, race and ethnicity, gender, income, education, marital status, religion, ideology, and partisanship. In some cases this approach may provide a cost effective way to collect a representative sample while using MTurk.

With apologies to Robert Crumb.

Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor Markets for Experimental Research: Amazon.com’s mechanical Turk. Political Analysis, 20(3), 351–368.CrossRef

Chandler, J. J., & Paolacci, G. (2017). Lie for a dime: When most prescreening responses are honest but most study participants are imposters. Social Psychological and Personality Science, 8(5), 500–508.CrossRef

Chandler, J. J., Gabriele Paolacci, E., Peer, P. M., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26, 1131–1139.CrossRef

Downes-Le Guin, T., J. Meehling, and R. Baker (2006), “Great Results from Ambiguous Sources: Cleaning Internet Panel Data,” in ESOMAR World Research Conference: Panel Research 2006, Amsterdaam, The Netherlands: ESOMAR.

Goodman, J. K., & Paolacci, G. (2017). Crowdsourcing consumer research. Journal of Consumer Research, 44(1), 196–210.CrossRef

Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213–224.CrossRef

Jones, M. S., House, L. A., & Gao, Z. (2015). Respondent screening and revealed preference axioms: Testing quarantining methods for enhanced data quality in web panel surveys. Public Opinion Quarterly, 79, 687–709.CrossRef

Levay, K. E., Freese, J., & Druckman, J. N. (2016). The demographic and political composition of mechanical Turk samples. Sage Open, 6(1), 1–17.CrossRef

McDonald, Paul, Matt Mohebbi, and Brett Slatkin (2013), “Comparing Google Consumer Surveys to Existing Probability and Non-Probability Based Internet Surveys,” White paper (https://www.google.com/insights/consumersurveys/static/consumer_surveys_whitepaper_v2.pdf), Google Inc.

Miller, Jeff (2006), “Research Reveals Alarming Incidence of ‘Undesirable’ Online Panelists,” Research Conference Report, RFL Communications, In.c (Skokie, IL), September–October issue. (Retrieved from http://www.burke.com/Library/Articles/Jeff%20Miller%20RCR%20PDF.pdf).

Sostek, Katrina, and Brett Slatkin (2017), “How Google Surveys Works,” White paper (http://services.google.com/fh/files/misc/google-surveys-whitepaper.pdf), Google Inc.

Wessling, S., Kathryn, J. H., & Netzer, O. (2017). MTurk character misrepresentation: Assessment and solutions. Journal of Consumer Research, 44(1), 211–230.CrossRef

Zwaan, Rolf A., Diane Pecher, Gabriele Paolacci, Samantha Bouwmeester, Peter Verkoeijen, Katinka Dijkstra, and René Zeelenberg (2018), “Participant Nonnaiveté and the Reproducability of cognitive psychology,” Psychonomic Bulletin & Review, in press.

Title: “Keep on Turkin’”?
Authors: John Hulland
Jeff Miller
Publication date: 09-05-2018
Publisher: Springer US
Published in: Journal of the Academy of Marketing Science / Issue 5/2018
Print ISSN: 0092-0703
Electronic ISSN: 1552-7824
DOI: https://doi.org/10.1007/s11747-018-0587-4

Springer Professional

Excerpt

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 5/2018

Scanning for discounts: examining the redemption of competing mobile coupons

An escalation of commitment perspective on allocation-of-effort decisions in professional selling

Customer query handling in sales interactions

Are conservative approaches to new product selling a blessing in disguise?

Initiating value co-creation: Dealing with non-receptive customers

Identity change vs. strategy change: the effects of rebranding announcements on stock returns