Skip to main content

Advertisement

Log in

Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review

  • Review Article
  • Published:
Journal of General Internal Medicine Aims and scope Submit manuscript

A Capsule Commentary to this article was published on 03 October 2013

Abstract

Objective

Crowdsourcing research allows investigators to engage thousands of people to provide either data or data analysis. However, prior work has not documented the use of crowdsourcing in health and medical research. We sought to systematically review the literature to describe the scope of crowdsourcing in health research and to create a taxonomy to characterize past uses of this methodology for health and medical research.

Data sources

PubMed, Embase, and CINAHL through March 2013.

Study eligibility criteria

Primary peer-reviewed literature that used crowdsourcing for health research.

Study appraisal and synthesis methods

Two authors independently screened studies and abstracted data, including demographics of the crowd engaged and approaches to crowdsourcing.

Results

Twenty-one health-related studies utilizing crowdsourcing met eligibility criteria. Four distinct types of crowdsourcing tasks were identified: problem solving, data processing, surveillance/monitoring, and surveying. These studies collectively engaged a crowd of >136,395 people, yet few studies reported demographics of the crowd. Only one (5 %) reported age, sex, and race statistics, and seven (33 %) reported at least one of these descriptors. Most reports included data on crowdsourcing logistics such as the length of crowdsourcing (n = 18, 86 %) and time to complete crowdsourcing task (n = 15, 71 %). All articles (n = 21, 100 %) reported employing some method for validating or improving the quality of data reported from the crowd.

Limitations

Gray literature not searched and only a sample of online survey articles included.

Conclusions and implications of key findings

Utilizing crowdsourcing can improve the quality, cost, and speed of a research project while engaging large segments of the public and creating novel science. Standardized guidelines are needed on crowdsourcing metrics that should be collected and reported to provide clarity and comparability in methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1.

Similar content being viewed by others

References

  1. Sobel D. Longitude: The True Story of a Lone Genius Who Solved the Greatest Scientific Problem of His Time. New York: Walker Publishing Company; 1995.

    Google Scholar 

  2. National Audubon Society. History of the Christmas Bird Count [Internet]. 2012 [cited 2012 Dec 24]. Available from: http://birds.audubon.org/history-christmas-bird-count

  3. ICT Data and Statistics Division. Measuring the Information Society 2012 [Internet]. Geneva, Switzerland: International Telecommunication Union; 2012 [cited 2012 Oct 14]. Available from: http://www.itu.int/ITU-D/ict/publications/idi/material/2012/MIS2012_without_Annex_4.pdf

  4. Lintott C, Schawinski K, Bamford S, Slosar A, Land K, Thomas D, et al. Galaxy Zoo 1: data release of morphological classifications for nearly 900 000 galaxies. Monthly Notices of the Royal Astronomical Society. 2011;410(1):166–78. doi:10.1111/j.1365-2966.2010.17432.x.

    Article  Google Scholar 

  5. Sayigh L, Quick N, Hastie G, Tyack P. Repeated call types in short-finned pilot whales, Globicephala macrorhynchus. Marine Mammal Science. 2012;29(2):312–24. doi:10.1111/j.1748-7692.2012.00577.x.

    Article  Google Scholar 

  6. Nature Publishing Group. Scientific American launches Citizen Science Whale-Song Project, Whale FM [Internet]. 2011 [cited 2012 Oct 14]. Available from: http://www.nature.com/press_releases/sciam-whale.html

  7. Sullivan BL, Wood CL, Iliff MJ, Bonney RE, Fink D, Kelling S. eBird: A citizen-based bird observation network in the biological sciences. Biological Conservation. 2009;142(10):2282–92. doi:10.1016/j.biocon.2009.05.006.

    Article  Google Scholar 

  8. Marris E. Supercomputing for the birds. Nature. 2010;466(7308):807. doi:10.1038/466807a. PubMed PMID: 20703280.

    Article  CAS  PubMed  Google Scholar 

  9. Preamble to the Constitution of the World Health Organization as adopted by the International Health Conference, New York, 19–22 June, 1946; signed on 22 July 1946 by the representatives of 61 States (Official Records of the World Health Organization, no. 2, p. 100) and entered into force on 7 April 1948. [cited 2012 Jul 21]. Available from: http://www.who.int/about/definition/en/print.html

  10. Reips U-D, Birnbaum MH. Behavioral research and data collection via the Internet. In: Proctor RW, Vu K-PL, eds. Handbook of human factors in Web design. 2nd ed. Mahwah, New Jersey: Lawrence Erlbaum Associates; 2011:563–85.

    Chapter  Google Scholar 

  11. Higgins JPT, Deeks JJ (editors). Chapter 7: Selecting studies and collecting data. In: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [Internet]. The Cochrane Collaboration; 2011 [updated 2011 Mar; cited 2012 Aug 1]. Available from www.cochrane-handbook.org

  12. Cooper S, Treuille A, Barbero J, Leaver-Fay A, Tuite K, Khatib F, et al. The challenge of designing scientific discovery games. California: Proceedings of the Fifth International Conference on the Foundations of Digital Games; Monterey; 2010:40–7. 1822354: ACM.

    Google Scholar 

  13. Cooper S, Khatib F, Treuille A, Barbero J, Lee J, Beenen M, et al. Predicting protein structures with a multiplayer online game. Nature. 2010;466(7307):756–60. doi:10.1038/nature09304. PubMed PMID: 20686574; PubMed Central PMCID: PMC2956414.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  14. Freifeld CC, Chunara R, Mekaru SR, Chan EH, Kass-Hout T, Ayala Iacucci A, et al. Participatory epidemiology: use of mobile phones for community-based health reporting. PLoS medicine. 2010;7(12):e1000376. doi:10.1371/journal.pmed.1000376. PubMed PMID: 21151888; PubMed Central PMCID: PMC2998443.

    Article  PubMed Central  PubMed  Google Scholar 

  15. Behrend TS, Sharek DJ, Meade AW, Wiebe EN. The viability of crowdsourcing for survey research. Behavior research methods. 2011;43(3):800–13. doi:10.3758/s13428-011-0081-0. PubMed PMID: 21437749.

    Article  PubMed  Google Scholar 

  16. Bender J, O'Grady L, Desphande A, Cortinois A, Saffie L, Husereau D, et al. Collaborative authoring: a case study of the use of a wiki as a tool to keep systematic reviews up to date. 2011.

  17. Cooper S, Khatib F, Makedon I, Lu H, Barbero J, Baker D, et al. Analysis of social gameplay macros in the Foldit cookbook. France: Proceedings of the 6th International Conference on Foundations of Digital Games; Bordeaux; 2011:9–14. 2159367: ACM.

    Google Scholar 

  18. Khatib F, DiMaio F, Foldit Contenders G, Foldit Void Crushers G, Cooper S, Kazmierczyk M, et al. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature structural & molecular biology. 2011;18(10):1175–7. doi:10.1038/nsmb.2119. PubMed PMID: 21926992.

    Article  CAS  Google Scholar 

  19. Khatib F, Cooper S, Tyka MD, Xu K, Makedon I, Popovic Z, et al. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences of the United States of America. 2011;108(47):18949–53. doi:10.1073/pnas.1115898108. PubMed PMID: 22065763; PubMed Central PMCID: PMC3223433.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  20. Chunara R, Chhaya V, Bane S, Mekaru SR, Chan EH, Freifeld CC, et al. Online reporting for malaria surveillance using micro-monetary incentives, in urban India 2010–2011. Malaria journal. 2012;11:43. doi:10.1186/1475-2875-11-43. PubMed PMID: 22330227; PubMed Central PMCID: PMC3305483.

    Article  PubMed Central  PubMed  Google Scholar 

  21. Eiben CB, Siegel JB, Bale JB, Cooper S, Khatib F, Shen BW, et al. Increased Diels-Alderase activity through backbone remodeling guided by Foldit players. Nature biotechnology. 2012;30(2):190–2. doi:10.1038/nbt.2109. PubMed PMID: 22267011.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  22. Jarmolowicz DP, Bickel WK, Carter AE, Franck CT, Mueller ET. Using crowdsourcing to examine relations between delay and probability discounting. Behavioural processes. 2012;91(3):308–12. doi:10.1016/j.beproc.2012.09.001. PubMed PMID: 22982370.

    Article  PubMed  Google Scholar 

  23. Kawrykow A, Roumanis G, Kam A, Kwak D, Leung C, Wu C, et al. Phylo: a citizen science approach for improving multiple sequence alignment. PLoS One. 2012;7(3):e31362. doi:10.1371/journal.pone.0031362. PubMed PMID: 22412834; PubMed Central PMCID: PMC3296692.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  24. Luengo-Oroz MA, Arranz A, Frean J. Crowdsourcing malaria parasite quantification: an online game for analyzing images of infected thick blood smears. Journal of medical Internet research. 2012;14(6):e167. doi:10.2196/jmir.2338. PubMed PMID: 23196001; PubMed Central PMCID: PMC3510720.

    Article  PubMed Central  PubMed  Google Scholar 

  25. Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, et al. Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLoS One. 2012;7(5):e37245. doi:10.1371/journal.pone.0037245. PubMed PMID: 22606353; PubMed Central PMCID: PMC3350488.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  26. Mavandadi S, Dimitrov S, Feng S, Yu F, Yu R, Sikora U, et al. Crowd-sourced BioGames: managing the big data problem for next-generation lab-on-a-chip platforms. Lab on a chip. 2012;12(20):4102–6. doi:10.1039/c2lc40614d. PubMed PMID: 22918378; PubMed Central PMCID: PMC3477593.

    Article  CAS  PubMed  Google Scholar 

  27. McKenna MT, Wang S, Nguyen TB, Burns JE, Petrick N, Summers RM. Strategies for improved interpretation of computer-aided detections for CT colonography utilizing distributed human intelligence. Medical image analysis. 2012;16(6):1280–92. doi:10.1016/j.media.2012.04.007. PubMed PMID: 22705287; PubMed Central PMCID: PMC3443285.

    Article  PubMed Central  PubMed  Google Scholar 

  28. Nguyen TB, Wang S, Anugu V, Rose N, McKenna M, Petrick N, et al. Distributed Human Intelligence for Colonic Polyp Classification in Computer-aided Detection for CT Colonography. Radiology. 2012;262(3):824–33. doi:10.1148/radiol.11110938. PubMed PMID: 22274839; PubMed Central PMCID: PMC3285221.

    Article  PubMed  Google Scholar 

  29. Turner AM, Kirchhoff K, Capurro D. Using crowdsourcing technology for testing multilingual public health promotion materials. Journal of medical Internet research. 2012;14(3):e79. doi:10.2196/jmir.2063. PubMed PMID: 22664384.

    Article  PubMed Central  PubMed  Google Scholar 

  30. Crump MJ, McDonnell JV, Gureckis TM. Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS One. 2013;8(3):e57410. doi:10.1371/journal.pone.0057410. PubMed PMID: 23516406; PubMed Central PMCID: PMC3596391.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  31. Hipp JA, Adlakha D, Eyler AA, Chang B, Pless R. Emerging technologies: webcams and crowd-sourcing to identify active transportation. American journal of preventive medicine. 2013;44(1):96–7. doi:10.1016/j.amepre.2012.09.051. PubMed PMID: 23253658.

    Article  PubMed  Google Scholar 

  32. Merchant RM, Asch DA, Hershey JC, Griffis HM, Hill S, Saynisch O, et al. A crowdsourcing innovation challenge to locate and map automated external defibrillators. Circulation Cardiovascular quality and outcomes. 2013;6(2):229–36. doi:10.1161/CIRCOUTCOMES.113.000140. PubMed PMID: 23481522.

    Article  PubMed  Google Scholar 

  33. Carpenter J. May the best analyst win. Science. 2011;331(6018):698–9. doi:10.1126/science.331.6018.698. PubMed PMID: 21311005.

    Article  CAS  PubMed  Google Scholar 

  34. Hussein A. Brokering knowledge in biosciences with InnoCentive. Interview by Semahat S. Demir. IEEE engineering in medicine and biology magazine: the quarterly magazine of the Engineering in Medicine & Biology Society. 2003;22(4):26–7. PubMed PMID: 14515687.

    Article  Google Scholar 

  35. Wheat RE, Wang Y, Byrnes JE, Ranganathan J. Raising money for scientific research through crowdfunding. Trends in ecology & evolution. 2013;28(2):71–2. doi:10.1016/j.tree.2012.11.001. PubMed PMID: 23219380.

    Article  Google Scholar 

  36. Mason W, Suri S. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods. 2012;44(1):1–23. doi:10.3758/s13428-011-0124-6. PubMed PMID: 21717266.

    Article  PubMed  Google Scholar 

  37. Signorini A, Segre AM, Polgreen PM. The use of Twitter to track levels of disease activity and public concern in the US during the influenza A H1N1 pandemic. PLoS One. 2011;6(5):e19467.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  38. Swan M. Crowdsourced health research studies: an important emerging complement to clinical trials in the public health research ecosystem. Journal of medical Internet research. 2012;14(2):e46. doi:10.2196/jmir.1988. PubMed PMID: 22397809.

    Article  PubMed Central  PubMed  Google Scholar 

Download references

Acknowledgments

Contributors

The authors thank Hope Lappen, MLIS, for her assistance with designing the systematic literature search.

Funders

NIH, K23 grant 10714038 (Merchant).

Prior presentations

None

Conflict of Interest

The authors declare that they do not have any conflicts of interest.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Raina M. Merchant MD, MSHP.

Appendix

Appendix

Figure 2.
figure 2

Crowdsourcing framework and categories.

Table 4 Category of Research Task Crowdsourcing was Utilized to Accomplish
Table 5 Potential Data to Report for Health Research Using Crowdsourcing

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ranard, B.L., Ha, Y.P., Meisel, Z.F. et al. Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review. J GEN INTERN MED 29, 187–203 (2014). https://doi.org/10.1007/s11606-013-2536-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11606-013-2536-8

KEY WORDS

Navigation