Abstract
Objective
Crowdsourcing research allows investigators to engage thousands of people to provide either data or data analysis. However, prior work has not documented the use of crowdsourcing in health and medical research. We sought to systematically review the literature to describe the scope of crowdsourcing in health research and to create a taxonomy to characterize past uses of this methodology for health and medical research.
Data sources
PubMed, Embase, and CINAHL through March 2013.
Study eligibility criteria
Primary peer-reviewed literature that used crowdsourcing for health research.
Study appraisal and synthesis methods
Two authors independently screened studies and abstracted data, including demographics of the crowd engaged and approaches to crowdsourcing.
Results
Twenty-one health-related studies utilizing crowdsourcing met eligibility criteria. Four distinct types of crowdsourcing tasks were identified: problem solving, data processing, surveillance/monitoring, and surveying. These studies collectively engaged a crowd of >136,395 people, yet few studies reported demographics of the crowd. Only one (5 %) reported age, sex, and race statistics, and seven (33 %) reported at least one of these descriptors. Most reports included data on crowdsourcing logistics such as the length of crowdsourcing (n = 18, 86 %) and time to complete crowdsourcing task (n = 15, 71 %). All articles (n = 21, 100 %) reported employing some method for validating or improving the quality of data reported from the crowd.
Limitations
Gray literature not searched and only a sample of online survey articles included.
Conclusions and implications of key findings
Utilizing crowdsourcing can improve the quality, cost, and speed of a research project while engaging large segments of the public and creating novel science. Standardized guidelines are needed on crowdsourcing metrics that should be collected and reported to provide clarity and comparability in methods.
Similar content being viewed by others
References
Sobel D. Longitude: The True Story of a Lone Genius Who Solved the Greatest Scientific Problem of His Time. New York: Walker Publishing Company; 1995.
National Audubon Society. History of the Christmas Bird Count [Internet]. 2012 [cited 2012 Dec 24]. Available from: http://birds.audubon.org/history-christmas-bird-count
ICT Data and Statistics Division. Measuring the Information Society 2012 [Internet]. Geneva, Switzerland: International Telecommunication Union; 2012 [cited 2012 Oct 14]. Available from: http://www.itu.int/ITU-D/ict/publications/idi/material/2012/MIS2012_without_Annex_4.pdf
Lintott C, Schawinski K, Bamford S, Slosar A, Land K, Thomas D, et al. Galaxy Zoo 1: data release of morphological classifications for nearly 900 000 galaxies. Monthly Notices of the Royal Astronomical Society. 2011;410(1):166–78. doi:10.1111/j.1365-2966.2010.17432.x.
Sayigh L, Quick N, Hastie G, Tyack P. Repeated call types in short-finned pilot whales, Globicephala macrorhynchus. Marine Mammal Science. 2012;29(2):312–24. doi:10.1111/j.1748-7692.2012.00577.x.
Nature Publishing Group. Scientific American launches Citizen Science Whale-Song Project, Whale FM [Internet]. 2011 [cited 2012 Oct 14]. Available from: http://www.nature.com/press_releases/sciam-whale.html
Sullivan BL, Wood CL, Iliff MJ, Bonney RE, Fink D, Kelling S. eBird: A citizen-based bird observation network in the biological sciences. Biological Conservation. 2009;142(10):2282–92. doi:10.1016/j.biocon.2009.05.006.
Marris E. Supercomputing for the birds. Nature. 2010;466(7308):807. doi:10.1038/466807a. PubMed PMID: 20703280.
Preamble to the Constitution of the World Health Organization as adopted by the International Health Conference, New York, 19–22 June, 1946; signed on 22 July 1946 by the representatives of 61 States (Official Records of the World Health Organization, no. 2, p. 100) and entered into force on 7 April 1948. [cited 2012 Jul 21]. Available from: http://www.who.int/about/definition/en/print.html
Reips U-D, Birnbaum MH. Behavioral research and data collection via the Internet. In: Proctor RW, Vu K-PL, eds. Handbook of human factors in Web design. 2nd ed. Mahwah, New Jersey: Lawrence Erlbaum Associates; 2011:563–85.
Higgins JPT, Deeks JJ (editors). Chapter 7: Selecting studies and collecting data. In: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [Internet]. The Cochrane Collaboration; 2011 [updated 2011 Mar; cited 2012 Aug 1]. Available from www.cochrane-handbook.org
Cooper S, Treuille A, Barbero J, Leaver-Fay A, Tuite K, Khatib F, et al. The challenge of designing scientific discovery games. California: Proceedings of the Fifth International Conference on the Foundations of Digital Games; Monterey; 2010:40–7. 1822354: ACM.
Cooper S, Khatib F, Treuille A, Barbero J, Lee J, Beenen M, et al. Predicting protein structures with a multiplayer online game. Nature. 2010;466(7307):756–60. doi:10.1038/nature09304. PubMed PMID: 20686574; PubMed Central PMCID: PMC2956414.
Freifeld CC, Chunara R, Mekaru SR, Chan EH, Kass-Hout T, Ayala Iacucci A, et al. Participatory epidemiology: use of mobile phones for community-based health reporting. PLoS medicine. 2010;7(12):e1000376. doi:10.1371/journal.pmed.1000376. PubMed PMID: 21151888; PubMed Central PMCID: PMC2998443.
Behrend TS, Sharek DJ, Meade AW, Wiebe EN. The viability of crowdsourcing for survey research. Behavior research methods. 2011;43(3):800–13. doi:10.3758/s13428-011-0081-0. PubMed PMID: 21437749.
Bender J, O'Grady L, Desphande A, Cortinois A, Saffie L, Husereau D, et al. Collaborative authoring: a case study of the use of a wiki as a tool to keep systematic reviews up to date. 2011.
Cooper S, Khatib F, Makedon I, Lu H, Barbero J, Baker D, et al. Analysis of social gameplay macros in the Foldit cookbook. France: Proceedings of the 6th International Conference on Foundations of Digital Games; Bordeaux; 2011:9–14. 2159367: ACM.
Khatib F, DiMaio F, Foldit Contenders G, Foldit Void Crushers G, Cooper S, Kazmierczyk M, et al. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature structural & molecular biology. 2011;18(10):1175–7. doi:10.1038/nsmb.2119. PubMed PMID: 21926992.
Khatib F, Cooper S, Tyka MD, Xu K, Makedon I, Popovic Z, et al. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences of the United States of America. 2011;108(47):18949–53. doi:10.1073/pnas.1115898108. PubMed PMID: 22065763; PubMed Central PMCID: PMC3223433.
Chunara R, Chhaya V, Bane S, Mekaru SR, Chan EH, Freifeld CC, et al. Online reporting for malaria surveillance using micro-monetary incentives, in urban India 2010–2011. Malaria journal. 2012;11:43. doi:10.1186/1475-2875-11-43. PubMed PMID: 22330227; PubMed Central PMCID: PMC3305483.
Eiben CB, Siegel JB, Bale JB, Cooper S, Khatib F, Shen BW, et al. Increased Diels-Alderase activity through backbone remodeling guided by Foldit players. Nature biotechnology. 2012;30(2):190–2. doi:10.1038/nbt.2109. PubMed PMID: 22267011.
Jarmolowicz DP, Bickel WK, Carter AE, Franck CT, Mueller ET. Using crowdsourcing to examine relations between delay and probability discounting. Behavioural processes. 2012;91(3):308–12. doi:10.1016/j.beproc.2012.09.001. PubMed PMID: 22982370.
Kawrykow A, Roumanis G, Kam A, Kwak D, Leung C, Wu C, et al. Phylo: a citizen science approach for improving multiple sequence alignment. PLoS One. 2012;7(3):e31362. doi:10.1371/journal.pone.0031362. PubMed PMID: 22412834; PubMed Central PMCID: PMC3296692.
Luengo-Oroz MA, Arranz A, Frean J. Crowdsourcing malaria parasite quantification: an online game for analyzing images of infected thick blood smears. Journal of medical Internet research. 2012;14(6):e167. doi:10.2196/jmir.2338. PubMed PMID: 23196001; PubMed Central PMCID: PMC3510720.
Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, et al. Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLoS One. 2012;7(5):e37245. doi:10.1371/journal.pone.0037245. PubMed PMID: 22606353; PubMed Central PMCID: PMC3350488.
Mavandadi S, Dimitrov S, Feng S, Yu F, Yu R, Sikora U, et al. Crowd-sourced BioGames: managing the big data problem for next-generation lab-on-a-chip platforms. Lab on a chip. 2012;12(20):4102–6. doi:10.1039/c2lc40614d. PubMed PMID: 22918378; PubMed Central PMCID: PMC3477593.
McKenna MT, Wang S, Nguyen TB, Burns JE, Petrick N, Summers RM. Strategies for improved interpretation of computer-aided detections for CT colonography utilizing distributed human intelligence. Medical image analysis. 2012;16(6):1280–92. doi:10.1016/j.media.2012.04.007. PubMed PMID: 22705287; PubMed Central PMCID: PMC3443285.
Nguyen TB, Wang S, Anugu V, Rose N, McKenna M, Petrick N, et al. Distributed Human Intelligence for Colonic Polyp Classification in Computer-aided Detection for CT Colonography. Radiology. 2012;262(3):824–33. doi:10.1148/radiol.11110938. PubMed PMID: 22274839; PubMed Central PMCID: PMC3285221.
Turner AM, Kirchhoff K, Capurro D. Using crowdsourcing technology for testing multilingual public health promotion materials. Journal of medical Internet research. 2012;14(3):e79. doi:10.2196/jmir.2063. PubMed PMID: 22664384.
Crump MJ, McDonnell JV, Gureckis TM. Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS One. 2013;8(3):e57410. doi:10.1371/journal.pone.0057410. PubMed PMID: 23516406; PubMed Central PMCID: PMC3596391.
Hipp JA, Adlakha D, Eyler AA, Chang B, Pless R. Emerging technologies: webcams and crowd-sourcing to identify active transportation. American journal of preventive medicine. 2013;44(1):96–7. doi:10.1016/j.amepre.2012.09.051. PubMed PMID: 23253658.
Merchant RM, Asch DA, Hershey JC, Griffis HM, Hill S, Saynisch O, et al. A crowdsourcing innovation challenge to locate and map automated external defibrillators. Circulation Cardiovascular quality and outcomes. 2013;6(2):229–36. doi:10.1161/CIRCOUTCOMES.113.000140. PubMed PMID: 23481522.
Carpenter J. May the best analyst win. Science. 2011;331(6018):698–9. doi:10.1126/science.331.6018.698. PubMed PMID: 21311005.
Hussein A. Brokering knowledge in biosciences with InnoCentive. Interview by Semahat S. Demir. IEEE engineering in medicine and biology magazine: the quarterly magazine of the Engineering in Medicine & Biology Society. 2003;22(4):26–7. PubMed PMID: 14515687.
Wheat RE, Wang Y, Byrnes JE, Ranganathan J. Raising money for scientific research through crowdfunding. Trends in ecology & evolution. 2013;28(2):71–2. doi:10.1016/j.tree.2012.11.001. PubMed PMID: 23219380.
Mason W, Suri S. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods. 2012;44(1):1–23. doi:10.3758/s13428-011-0124-6. PubMed PMID: 21717266.
Signorini A, Segre AM, Polgreen PM. The use of Twitter to track levels of disease activity and public concern in the US during the influenza A H1N1 pandemic. PLoS One. 2011;6(5):e19467.
Swan M. Crowdsourced health research studies: an important emerging complement to clinical trials in the public health research ecosystem. Journal of medical Internet research. 2012;14(2):e46. doi:10.2196/jmir.1988. PubMed PMID: 22397809.
Acknowledgments
Contributors
The authors thank Hope Lappen, MLIS, for her assistance with designing the systematic literature search.
Funders
NIH, K23 grant 10714038 (Merchant).
Prior presentations
None
Conflict of Interest
The authors declare that they do not have any conflicts of interest.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Ranard, B.L., Ha, Y.P., Meisel, Z.F. et al. Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review. J GEN INTERN MED 29, 187–203 (2014). https://doi.org/10.1007/s11606-013-2536-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11606-013-2536-8