Top

Quality & Quantity

Published in:

07-09-2023 | Original paper

The inaccuracy of data from online surveys: A cautionary analysis

Authors: Jennifer P. Agans, Serena A. Schade, Steven R. Hanna, Shou-Chun Chiang, Kimia Shirzad, Sunhye Bai

Published in: Quality & Quantity | Issue 3/2024

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Online recruitment methods for survey-based studies have become increasingly common in social science research. However, they are susceptible to a high rate of participation by fraudulent research subjects. The current study identified fraudulent (i.e., “fake”) participants in an online research study of parents of 13 to 18-year-old adolescents, and compared demographic, anthropometric, and subjective health data between “fake” (N = 1084) and “real” (N = 197) participants. Of 1,281 subjects who started the eligibility survey, 84.6% were coded as “fake.” “Fake” participants were less diverse in race/ethnicity and more diverse in gender. Their depression symptoms were inflated, but ratings of perceived health were comparable to “real” participants. Well-established correlations, such as that between BMI and perceived health, were not replicated with “fake” participants. Online surveys are highly vulnerable to fraudulent research subjects whose participation compromises the validity and interpretability of results. The discussion provides a guide and recommendations for improving data quality in online survey research.

previous article Quantifying inequity in research assistant selection procedures at Turkish universities

next article Evaluation of urban transportation preferences with analytical hierarchy process method

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

Amazon Mechanical Turk:. Qualifications and worker task quality best practices. Medium. (2021)., February 16 https://blog.mturk.com/qualifications-and-worker-task-quality-best-practices-886f1f4e03fc

Appelbaum, M., Cooper, H., Kline, R.B., Mayo-Wilson, E., Nezu, A.M., Rao, S.M.: Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. Am. Psychol. 73(1), 3–25 (2018). https://doi.org/10.1037/amp0000191CrossRef

Ball, H.L.: Conducting online surveys. J. Hum. Lactation. 35(3), 413–417 (2019). https://doi.org/10.1177/0890334419848734CrossRef

Barratt, M.J., Ferris, J.A., Lenton, S.: Hidden populations, online purposive sampling, and external validity: Taking off the blindfold. Field Methods. 27(1), 3–21 (2015). https://doi.org/10.1177/1525822X14526838CrossRef

Bentler, P.M., Bonett, D.G.: Significance tests and goodness of fit in the analysis of covariance structures. Psychol. Bull. 88(3), 588–606 (1980). https://doi.org/10.1037/0033-2909.88.3.588CrossRef

Buchanan, E.M., Scofield, J.E.: Methods to detect low quality data and its implication for psychological research. Behav. Res. Methods. 50(6), 2586–2596 (2018)CrossRef

Chen, F.F.: Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct. Equation Modeling: Multidisciplinary J. 14(3), 464–504 (2007). https://doi.org/10.1080/10705510701301834CrossRef

Chmielewski, M., Kucker, S.C.: An mturk crisis? Shifts in data quality and the impact on study results. Social Psychol. Personality Sci. 11(4), 464–473 (2019). https://doi.org/10.1177/1948550619875149CrossRef

Chung, S.-Y., Hacker, E.D., Rawl, S., Ellis, R., Bakas, T., Jones, J., Welch, J.: Using facebook in recruiting kidney transplant recipients for a redcap study. West. J. Nurs. Res. 41(12), 1790–1812 (2019). https://doi.org/10.1177/0193945919832600CrossRef

Dermody, S., [@SarahSDermody]: Because I wish someone warned me… Twitter. Retrieved July 22, 2022, from (2022)., July 14 https://twitter.com/SarahSDermody/status/1547606450905591810?t=u0Y0DqQhEbkEbh-AzVC0QA&s=19

DeSalvo, K.B., Fisher, W.P., Tran, K., Bloser, N., Merrill, W., Peabody, J.: Assessing measurement properties of two single-item general health measures. Qual. Life Res. 15(2), 191–201 (2006). https://doi.org/10.1007/s11136-005-0887-2CrossRef

Dupuis, M., Meier, E., Cuneo, F.: Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices. Behav. Res. Methods. 51(5), 2228–2237 (2019)CrossRef

Dworkin, J., Hessel, H., Gliske, K., Rudi, J.H.: A comparison of three online recruitment strategies for engaging parents. Fam. Relat. 65(4), 550–561 (2016). https://doi.org/10.1111/fare.12206CrossRef

Eynon, R., Fry, J., Schroeder, R.: The ethics of online research. In: Fielding, N., Lee, R., Blank, G. (eds.) The SAGE Handbook of Online Research Methods, pp. 19–37. SAGE Publications Ltd (2017). https://doi.org/10.4135/9781473957992.n2

Fryar, C.D., Carroll, M.D., Gu, Q., Afful, J., Ogden, C.L.: Anthropometric reference data for children and adults: United States, 2015–2018. National Center for Health Statistics. Vital Health Stat, 3(46). (2021). https://www.cdc.gov/nchs/data/series/sr_03/sr03-046-508.pdf

Greenwood, D.: How one teen’s TikTok created a crisis for the paid survey industry i-D. (2021)., September 27 https://i-d.vice.com/en_uk/article/7kvnmy/tiktok-prolific-survey

Griffin, M., Martino, R.J., LoSchiavo, C., Comer-Carruthers, C., Krause, K.D., Stults, C.B., Halkitis, P.N.: Ensuring survey research data integrity in the era of internet bots. Qual. Quant. (2021). https://doi.org/10.1007/s11135-021-01252-1CrossRef

Howell, B.: Dealing with bots, randoms and satisficing in online research. Psychstudio. (2019)., October 31 https://www.psychstudio.com/articles/bots-randoms-satisficing/

Hu, L., Bentler, P.M.: Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct. Equation Modeling: Multidisciplinary J. 6(1), 1–55 (1999). https://doi.org/10.1080/10705519909540118CrossRef

Jibunu: (n.d.). Preventing Bad Data: How to Catch Survey Speeders, Cheaters, & Bots. Retrieved October 1, 2021 from: https://cdn2.hubspot.net/hubfs/4115290/Content (after March 19, 2018)/Preventing Bad Data/Preventing Bad Data.pdf

Kaczmirek, L.: Online survey software. In: Fielding, N., Lee, R., Blank, G. (eds.) The SAGE Handbook of Online Research Methods, pp. 203–219. SAGE Publications Ltd (2017). https://doi.org/10.4135/9781473957992.n12

Karcher, M.J., Sass, D.: A multicultural assessment of adolescent connectedness: Testing measurement invariance across gender and ethnicity. J. Couns. Psychol. 57(3), 274–289 (2010). https://doi.org/10.1037/a0019357CrossRef

Krosnick, J.A., Narayan, S., Smith, W.R.: Satisficing in surveys: Initial evidence. New Dir. Evaluation. 1996(70), 29–44 (1996). https://doi.org/10.1002/ev.1033CrossRef

Lee, R., Fielding, N., Blank, G.: Online research methods in the social sciences: An editorial introduction. In: Fielding, N., Lee, R., Blank, G. (eds.) The SAGE Handbook of Online Research Methods, pp. 3–16. SAGE Publications Ltd (2017). https://doi.org/10.4135/9781473957992.n1

Levi, R., Ridberg, R., Akers, M., Seligman, H.: Survey fraud and the integrity of web-based Survey Research. Am. J. Health Promotion. 36(1), 18–20 (2021). https://doi.org/10.1177/08901171211037531CrossRef

Longworth, J.: VPN: From an obscure network to a widespread solution. Comput. Fraud Secur. 2018(4), 14–15 (2018). https://doi.org/10.1016/s1361-3723(18)30034-4CrossRef

McLeroy, K.R., Northridge, M.E., Balcazar, H., Greenberg, M.R., Landers, S.J.: Reporting guidelines and the American Journal of Public Health’s adoption of preferred reporting items for systematic reviews and meta-analyses. Am. J. Public Health. 102(5), 780–784 (2012). https://doi.org/10.2105/ajph.2011.300630CrossRef

Meredith, W.: Measurement invariance, factor analysis and factorial invariance. Psychometrika. 58(4), 525–543 (1993). https://doi.org/10.1007/bf02294825CrossRef

Mitchell, J.W., Chavanduka, T.M., Sullivan, S., Stephenson, R.: Recommendations from a descriptive evaluation to improve screening procedures for web-based studies with couples: Cross-sectional study. JMIR Public. Health and Surveillance. 6(2), e15079 (2020). https://doi.org/10.2196/15079CrossRef

Peer, E., Rothschild, D., Gordon, A., Evernden, Z., Damer, E.: Data quality of platforms and panels for online behavioral research. Behav. Res. Methods. 1–20 (2021). https://doi.org/10.3758/s13428-021-01694-3

Penn State Social Science Research Institute:. New redcap feature protects from ‘bot’ respondents. Population Research Institute. (2019)., October 18 https://pop.psu.edu/news/new-redcap-feature-protects-bot-respondents

Pozzar, R., Hammer, M.J., Underhill-Blazey, M., Wright, A.A., Tulsky, J.A., Hong, F., Gundersen, D.A., Berry, D.L.: Threats of bots and other bad actors to data quality following research participant recruitment through social media: Cross-sectional questionnaire. J. Med. Internet. Res. 22(10) (2020). https://doi.org/10.2196/23021

Pratt-Chapman, M., Moses, J., Arem, H.: Strategies for the identification and prevention of survey fraud: Data analysis of a web-based survey. JMIR Cancer. 7(3) (2021). https://doi.org/10.2196/30730

PRISMA. (n.d.). Welcome to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) website! Retrieved February 13:, from (2022). http://www.prisma-statement.org/

Qualtrics: Response Quality. (2021)., May 21 https://www.qualtrics.com/support/survey-platform/survey-module/survey-checker/response-quality/

Radloff, L.S.: The CES-D scale: A self report depression scale for research in the general population. Appl. Psychol. Measurements. 1, 385–401 (1977). https://doi.org/10.1177/014662167700100306CrossRef

Rasmussen, K.: Data quality in online environments. In: Fielding, N., Lee, R., Blank, G. (eds.) The SAGE Handbook of Online Research Methods, pp. 38–54. SAGE Publications Ltd (2017). https://doi.org/10.4135/9781473957992.n3

Rand Health. 36-Item Short Form Survey (SF-36) Scoring Instructions. (2016).

REDCap Security Overview (n.d.). IHR REDCap. Retrieved January 20, from (2022). http://kpco-ihr.org/redcap/citing.html#redcapgeneralsecurity

Roberts, C., Gilbert, E., Allum, N., Eisner, L.: Research synthesis: Satisficing in surveys: A systematic review of the literature. Pub. Opin. Q. 83(3), 598–626 (2019). https://doi.org/10.1093/poq/nfz035CrossRef

Russomanno, J., Patterson, J.G., Tree, J.M.J.: Social media recruitment of marginalized, hard-to-reach populations: Development of recruitment and monitoring guidelines. JMIR Public. Health and Surveillance. 5(4), e14886 (2019). https://doi.org/10.2196/14886CrossRef

Simone, M.: [@m_simonephd]. My online #researchstudy was recently infiltrated by bot. Twitter. Retrieved October 1, 2021, from (2019a)., September 17 https://twitter.com/m_simonephd/status/1174010078632009728

Simone, M.: November 25). How to battle the bots wrecking your online study. Behav. Sci. (2019b). https://behavioralscientist.org/how-to-battle-the-bots-wrecking-your-online-study/

Society for Research in Child Development SRCD. (n.d.). Child Development Submission Guidelines. Retrieved February 18, 2022, from: https://www.srcd.org/research/journals/child-development/child-development-submission-guidelines#SCP

Stephens, B., Shaghaghi, A., Doss, R., Kanhere, S.S.: Detecting internet of things bots: A comparative study. IEEE Access. 9, 160391–160401 (2021). https://doi.org/10.1109/access.2021.3130714CrossRef

Sterzing, P.R., Gartner, R.E., McGeough, B.L.: Conducting anonymous, incentivized, online surveys with sexual and gender minority adolescents: Lessons learned from a national polyvictimization study. J. Interpers. Violence. 33(5), 740–761 (2018). https://doi.org/10.1177/0886260517744845CrossRef

Storozuk, A., Ashley, M., Delage, V., Maloney, E.A.: Got bots? Practical recommendations to protect online survey data from BOT attacks. Quant. Methods Psychol. 16(5), 472–481 (2020). https://doi.org/10.20982/tqmp.16.5.p472CrossRef

Teitcher, J.E., Bockting, W.O., Bauermeister, J.A., Hoefer, C.J., Miner, M.H., Klitzman, R.L.: Detecting, preventing, and responding to fraudsters in internet research: Ethics and tradeoffs. J. Law Med. Ethics. 43(1), 116–133 (2015)CrossRef

Thornton, L., Batterham, P.J., Fassnacht, D.B., Kay-Lambkin, F., Calear, A.L., Hunt, S.: Recruiting for health, medical or psychosocial research using Facebook: Systematic review. Internet Interventions. 4, 72–81 (2016). https://doi.org/10.1016/j.invent.2016.02.001CrossRef

Van Selm, M., Jankowski, N.W.: Conducting online surveys. Qual. Quantity. 40(3), 435–456 (2006). https://doi.org/10.1007/s11135-005-8081-8CrossRef

Vannette, D.: Survey straightlining: What it is & how to fight it. Qualtrics. (2018)., May 27 https://www.qualtrics.com/blog/straightlining-what-is-it-how-can-it-hurt-you-and-how-to-protect-against-it/

Vilagut, G., Forero, C.G., Barbaglia, G., Alonso, J.: Screening for depression in the general population with the Center for epidemiologic studies Depression (CES-D): A systematic review with meta-analysis. PLOS ONE. 11(5) (2016). https://doi.org/10.1371/journal.pone.0155431

Vu, M., Huynh, V.N., Bednarczyk, R.A., Escoffery, C., Ta, D., Nguyen, T.T., Berg, C.J.: Experience and lessons learned from multi-modal internet-based recruitment of US Vietnamese into research. PloS One, 16(8), e0256074. (2021)

Wong, J.J., Roubinov, D.S., Gonzales, N.A., Dumka, L.E., Millsap, R.E.: Father enrollment and participation in a parenting intervention: Personal and contextual predictors. Fam. Process. 52(3), 440–454 (2013). https://doi.org/10.1111/famp.12024CrossRef

Wright, A.: REDCap: A tool for the electronic capture of research data. J. Electron. Resour. Med. Libr. 13(4), 197–201 (2016). https://doi.org/10.1080/15424065.2016.1259026CrossRef

Yarrish, C., Groshon, L., Mitchell, J.D., Appelbaum, A., Klock, S., Winternitz, T., Friedman-Wheeler, D.G.: Finding the signal in the noise: Minimizing responses from bots and inattentive humans in online research. Behav. Therapist. 42(7), 235–242 (2019)

Title: The inaccuracy of data from online surveys: A cautionary analysis
Authors: Jennifer P. Agans
Serena A. Schade
Steven R. Hanna
Shou-Chun Chiang
Kimia Shirzad
Sunhye Bai
Publication date: 07-09-2023
Publisher: Springer Netherlands
Published in: Quality & Quantity / Issue 3/2024
Print ISSN: 0033-5177
Electronic ISSN: 1573-7845
DOI: https://doi.org/10.1007/s11135-023-01733-5

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 3/2024

An overview of mindfulness theories applied to tourism: systematic review update and bibliometric analysis

Process capability analysis for simple linear profiles

Research hotspots, emerging patterns, and intellectual structure of homestay tourism: a bibliometric analysis

Modelling labour productivity and the role of research intensity in 129 years: evidence from a new dynamic instrumental variable estimation approach

Disentangling work–family conflict, support, and turnover intentions: quanti–quali mixed method approach

Assessing heterogeneity in MOOC student performance through composite-based path modelling

Premium Partner