Top

Quality & Quantity

Published in:

15-09-2021

College students’ and Mechanical Turk workers’ environmental factors while completing online surveys

Authors: Arne Weigold, Ingrid K. Weigold, Migyeong Jang, Emily M. Thornton

Published in: Quality & Quantity | Issue 4/2022

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Few studies have investigated the environmental factors of participants completing online self-report surveys, such as the presence of others and engagement in multitasking. The current study examined the environmental factors of 1023 participants drawn from two popular convenience populations—college students (n = 512) and Mechanical Turk workers (n = 511). All participants completed online measures of computer self-efficacy, the five-factor model of personality, and social desirability, as well as a questionnaire about their environment and activities when completing the measures. The two samples differed in terms of environmental factors, with college students significantly more likely to report: (1) being in a location other than their place of residence, (2) interacting with others, (3) being interrupted, (4) engaging in multitasking, (5) being under the influence of alcohol or other substances, and (6) completing the study using a laptop computer. The presence of others interacted with sample to predict personality and social desirability scores. Additionally, both presence of others and engaging in multitasking predicted longer completion time.

previous article Mining excursion tourist profile through classification algorithms

next article RETRACTED ARTICLE: Psychological aspect of the training process of preschool education students

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Ansolabehere, S., Schaffner, B.F.: Distractions: the incidence and consequences of interruptions for survey respondents. J. Surv. Stat. Methodol. (2015). https://doi.org/10.1093/jssam/smv003CrossRef

Antoun, C., Couper, M.P., Conrad, F.G.: Effects of mobile versus PC Web on survey response quality: a crossover experiment in a probability web panel. Public Opin. q. (2017). https://doi.org/10.1093/poq/nfw088CrossRef

Behrend, T.S., Sharek, D.J., Meade, A.W., Wiebe, E.N.: The viability of crowdsourcing for survey research. Behav. Res. Methods (2011). https://doi.org/10.3758/s13428-011-0081-0CrossRef

Berinsky, A.J., Huber, G.A., Lenz, G.S.: Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Polit. Anal. (2012). https://doi.org/10.1093/pan/mpr057CrossRef

Buchanan, T., Johnson, J.A., Goldberg, L.R.: Implementing a five-factor personality inventory for use on the Internet. Eur. J. Psychol. Assess. (2005). https://doi.org/10.1027/1015-5759.18.1.115CrossRef

Buhrmester, M., Kwang, T., Gosling, S.D.: Amazon’s Mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. (2011). https://doi.org/10.1177/1745691610393980CrossRef

Cima, R.: Mechanical Turk: the new face of behavioral science? Priceonomics (2014). http://priceonomics.com/mechanical-turk-new-face-of-behavioral-science/

Couper, M.P., Peterson, G.J.: Why do web surveys take longer on smartphones? Soc. Sci. Comput. Rev. (2017). https://doi.org/10.1177/0894439316629932CrossRef

Cronk, B.C., West, J.L.: Personality research on the Internet: a comparison of web-based and traditional instruments in take-home and in-class settings. Behav. Res. Methods Instrum. Comput. (2002). https://doi.org/10.3758/BF03195440CrossRef

Dadey, N., Lyons, S., DePascale, C.: The comparability of scores from different digital devices: a literature review and synthesis with recommendations for practice. Appl. Meas. Educ. (2018). https://doi.org/10.1080/08957347.2017.1391262CrossRef

DeCastellarnau, A.: A classification of response scale characteristics that affect data quality: a literature review. Qual. Quant. (2018). https://doi.org/10.1007/s11135-017-0533-4CrossRef

Farrell, D., Petersen, J.C.: The growth of Internet research methods and the reluctant sociologist. Sociol. Inq. (2010). https://doi.org/10.1111/j.1475-682X.2009.00318.xCrossRef

Francavilla, N.M., Meade, A.W., Young, A.L.: Social interaction and Internet-based surveys: examining the effects of virtual and in-person proctors on careless response. Appl. Psychol. (2019). https://doi.org/10.1111/apps.12159CrossRef

Goldberg, L.R., Johnson, J.A., Eber, H.W., Hogan, R., Ashton, M.C., Cloninger, C.R., Gough, H.G.: The international personality item pool and the future of public-domain personality measures. J. Res. Pers. (2006). https://doi.org/10.1016/j.jrp.2005.08.007CrossRef

Goodman, J.K., Cryder, C.E., Cheema, A.: Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples. J. Behav. Decis. Mak. (2013). https://doi.org/10.1002/bdm.1753CrossRef

Hart, C.M., Ritchie, T.D., Hepper, E.G., Gebauer, J.E.: The balanced inventory of desirable responding short form (BIDR-16). SAGE Open (2015). https://doi.org/10.1177/2158244015621113CrossRef

Hauser, D.J., Schwarz, N.: Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behav. Res. Methods (2016). https://doi.org/10.3758/s13428-015-0578-zCrossRef

Henry, P.J.: College sophomores in the laboratory redux: influences of a narrow database on social psychology’s view of the nature of prejudice. Psychol. Inq. (2008). https://doi.org/10.1080/10478400802049936CrossRef

Holden, C.J., Dennie, T., Hicks, A.D.: Assessing the reliability of the M5-120 on Amazon’s Mechanical Turk. Comput. Hum. Behav. (2013). https://doi.org/10.1016/j.chb.2013.02.020CrossRef

International Personality Item Pool: A Scientific Collaboratory for the Development of Advanced Measures of Personality and Other Individual Differences (2019). http://ipip.ori.org/

Iperiotis, P.: Demographics of Mechanical Turk (2010). https://archivefda.dlib.nyu.edu/jspui/bitstream/2451/29585/2/CeDER-10-01.pdf

Kardes, F.R.: In defense of experimental consumer psychology. J. Consum. Psychol. (1996). https://doi.org/10.1207/s15327663jcp0503_04CrossRef

Lambert, A.D., Miller, A.L.: Living with smartphones: does completion device affect survey responses? Res. High. Educ. (2015). https://doi.org/10.1007/s11162-014-9354-7CrossRef

Lucas, J.W.: Theory-testing, generalization, and the problem of external validity. Soc. Theory (2003). https://doi.org/10.1111/1467-9558.00187CrossRef

Marakas, G.M., Johnson, R.D., Clay, P.F.: The evolving nature of the computer self-efficacy construct: an empirical investigation of measurement construction, validity, reliability and stability over time. J. Assoc. Inf. Syst. (2007). https://doi.org/10.17705/1jais.00112CrossRef

Meade, A.W., Craig, S.B.: Identifying careless responses in survey data. Psychol. Methods (2012). https://doi.org/10.1037/a0028085CrossRef

Miyazaki, A.D., Taylor, K.A.: Researcher interaction biases and business ethics research: respondent reactions to researcher characteristics. J. Bus. Ethics (2008). https://doi.org/10.1007/s10551-007-9547-5CrossRef

Necka, E.A., Cacioppo, S., Norman, G.J., Cacioppo, J.T.: Measuring the prevalence of problematic respondent behaviors among MTurk, campus, and community participants. PLoS ONE (2016). https://doi.org/10.1371/journal.pone.0157732CrossRef

Paolacci, G., Chandler, J.: Inside the Turk: understanding Mechanical Turk as a participant pool. Curr. Dir. Psychol. Sci. (2014). https://doi.org/10.1177/0963721414531598CrossRef

Paolacci, G., Chandler, J., Ipeirotis, P.G.: Running experiments on Amazon Mechanical Turk. Judgm. Decis. Mak. 5, 411–419 (2010)

Paulhus, D.L.: Manual for the Paulhus Deception Scales: BIDR Version 7. Multi-Health Systems, Toronto (1998)

Peterson, R.A., Merunka, D.R.: Convenience samples of college students and research reproducibility. J. Bus. Res. (2014). https://doi.org/10.1016/j.jbusres.2013.08.010CrossRef

Richardson, J.T.E.: Eta squared and partial eta squared as measures of effect size in educational research. Educ. Res. Rev. (2011). https://doi.org/10.1016/j.edurev.2010.12.001CrossRef

Rosenthal, R.: Experimenter Effects in Behavioral Research. Halsted Press, New York (1966)

Sears, D.O.: College sophomores in the laboratory: influences of a narrow data base on social psychology’s view of human nature. J. Pers. Soc. Psychol. (1986). https://doi.org/10.1037/0022-3514.51.3.515CrossRef

Sendelbah, A., Vejovar, V., Slavec, A., Petrovčič, A.: Investigating respondent multitasking in web surveys using paradata. Comput. Hum. Behav. (2016). https://doi.org/10.1016/j.chb.2015.10.028CrossRef

Shih, T.-H., Fan, X.: Comparing response rates from web and mail surveys: a meta-analysis. Field Methods (2008). https://doi.org/10.1177/1525822X08317085CrossRef

Skitka, L.J., Sargis, E.G.: The Internet as psychological laboratory. Annu. Rev. Psychol. (2006). https://doi.org/10.1146/annurev.psych.57.102904.190048CrossRef

Stieger, S., Reips, U.-D.: What are participants doing while filling in an online questionnaire: a paradata collection tool and an empirical study. Comput. Hum. Behav. 26, 1488–1495 (2010)CrossRef

Tourangeau, R.: Measurement properties of web surveys. In: Statistics Canada Symposium, Gatineau, Québec (2014)

Tourangeau, R., Sun, H., Yan, T., Maitland, A., Rivero, G., Williams, D.: Web surveys by smartphones and tablets: effects on data quality. Soc. Sci. Comput. Rev. (2018). https://doi.org/10.1177/0894439317719438CrossRef

Weigold, A., Weigold, I.K.: Traditional and modern convenience samples: an investigation of college student, Mechanical Turk, and Mechanical Turk college student samples. Soc. Sci. Comput. Rev. (2021). https://doi.org/10.1177/08944393211006847CrossRef

Weigold, A., Weigold, I.K., Natera, S.N.: Mean scores for self-report surveys completed using paper-and-pencil and computers: a meta-analytic test of equivalence. Comput. Hum. Behav. 86, 153–164 (2018)CrossRef

Weigold, A., Weigold, I.K., Natera, S.N.: Response rates for surveys completed with paper-and-pencil and computers: using meta-analysis to assess equivalence. Soc. Sci. Comput. Rev. (2019). https://doi.org/10.1177/0894439318783435CrossRef

Wintre, M.G., North, C., Sugar, L.A.: Psychologists’ response to criticisms about research based on undergraduate participants: a developmental perspective. Can. Psychol. Psychol. Can. (2001). https://doi.org/10.1037/h0086893CrossRef

Wood, E., Nosko, A., Desmarais, S., Ross, C., Irvine, C.: Online and traditional paper-and-pencil survey administration: examining experimenter presence, sensitive material and long surveys. Can. J. Hum. Sex. (2006). https://www.semanticscholar.org/paper/Online-and-Traditional-Paper-and-Pencil-Survey-and-Wood-Nosko/9052bd6eba7deae19dd0119f521e731f4844607e

Zwarun, L., Hall, A.: What’s going on? Age, distraction, and multitasking during online survey taking. Comput. Hum. Behav. (2014). https://doi.org/10.1016/j.chb.2014.09.041CrossRef

Title: College students’ and Mechanical Turk workers’ environmental factors while completing online surveys
Authors: Arne Weigold
Ingrid K. Weigold
Migyeong Jang
Emily M. Thornton
Publication date: 15-09-2021
Publisher: Springer Netherlands
Published in: Quality & Quantity / Issue 4/2022
Print ISSN: 0033-5177
Electronic ISSN: 1573-7845
DOI: https://doi.org/10.1007/s11135-021-01237-0

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 4/2022

Role of tariffs, imports substitution and investment efficiency in economic growth of Pakistan

Ensuring survey research data integrity in the era of internet bots

Different algorithms, different models

Systematic review of declarative tactical knowledge evaluation tools based on game-play scenarios in soccer

Cost of doing business index in Latin America

The impact of fiscal centralization policies on education expenditure among Chinese local governments