Skip to main content

2018 | OriginalPaper | Buchkapitel

Role of Task Complexity and Training in Crowdsourced Image Annotation

verfasst von : Nadine S. Schaadt, Anne Grote, Germain Forestier, Cédric Wemmert, Friedrich Feuerhake

Erschienen in: Computational Pathology and Ophthalmic Medical Image Analysis

Verlag: Springer International Publishing

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Accurate annotation of anatomical structures or pathological changes in microscopic images is an important task in computational pathology. Crowdsourcing holds promise to address this demand, but so far feasibility has only be shown for simple tasks and not for high-quality annotation of complex structures which is often limited by shortage of experts. Third-year medical students participated in solving two complex tasks, labeling of images and delineation of relevant image objects in breast cancer and kidney tissue. We evaluated their performance and addressed the requirements of task complexity and training phases. Our results show feasibility and a high agreement between students and experts. The training phase improved accuracy of image labeling.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imag. 35, 1313–1321 (2016)CrossRef Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imag. 35, 1313–1321 (2016)CrossRef
2.
Zurück zum Zitat Della Mea, V., Maddalena, E., Mizzaro, S., Machin, P., Beltrami, C.A.: Preliminary results from a crowdsourcing experiment in immunohistochemistry. Diagn. Pathol. 9, S6 (2014)CrossRef Della Mea, V., Maddalena, E., Mizzaro, S., Machin, P., Beltrami, C.A.: Preliminary results from a crowdsourcing experiment in immunohistochemistry. Diagn. Pathol. 9, S6 (2014)CrossRef
3.
Zurück zum Zitat Hoßfeld, T., et al.: Best practices and recommendations for crowdsourced qoe-lessons learned from the qualinet task force crowdsourcing. In: QUALINET (2014) Hoßfeld, T., et al.: Best practices and recommendations for crowdsourced qoe-lessons learned from the qualinet task force crowdsourcing. In: QUALINET (2014)
4.
Zurück zum Zitat Irshad, H., et al.: Crowdsourcing scoring of immunohistochemistry images: evaluating performance of the crowd and an automated computational method. Sci. Rep. 7 (2017) Irshad, H., et al.: Crowdsourcing scoring of immunohistochemistry images: evaluating performance of the crowd and an automated computational method. Sci. Rep. 7 (2017)
5.
Zurück zum Zitat Kim, E., Mente, S., Keenan, A., Gehlot, V.: Digital pathology annotation data for improved deep neural network classification. In: SPIE Medical Imaging, p. 101380D (2017) Kim, E., Mente, S., Keenan, A., Gehlot, V.: Digital pathology annotation data for improved deep neural network classification. In: SPIE Medical Imaging, p. 101380D (2017)
6.
Zurück zum Zitat Lawson, J., et al.: Crowdsourcing for translational research: analysis of biomarker expression using cancer microarrays. Br. J. Cancer 116, 237–245 (2017)CrossRef Lawson, J., et al.: Crowdsourcing for translational research: analysis of biomarker expression using cancer microarrays. Br. J. Cancer 116, 237–245 (2017)CrossRef
7.
Zurück zum Zitat Liu, S., Xia, F., Zhang, J., Wang, L., Wang, L.: How crowdsourcing risks affect performance: an exploratory model. Manag. Decis. 54, 2235–2255 (2016)CrossRef Liu, S., Xia, F., Zhang, J., Wang, L., Wang, L.: How crowdsourcing risks affect performance: an exploratory model. Manag. Decis. 54, 2235–2255 (2016)CrossRef
8.
Zurück zum Zitat Marée, R.: Collaborative analysis of multi-gigapixel imaging data using cytomine. Bioinformatics 32, 1395–1401 (2016)CrossRef Marée, R.: Collaborative analysis of multi-gigapixel imaging data using cytomine. Bioinformatics 32, 1395–1401 (2016)CrossRef
9.
Zurück zum Zitat Mavandadi, S., et al.: Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PloS One 7, e37245 (2012)CrossRef Mavandadi, S., et al.: Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PloS One 7, e37245 (2012)CrossRef
10.
Zurück zum Zitat Redi, J., Povoa, I.: Crowdsourcing for rating image aesthetic appeal: better a paid or a volunteer crowd? In: Proceedings of 2014 International ACM Workshop Crowdsourcing Multimedia, pp. 25–30. ACM (2014) Redi, J., Povoa, I.: Crowdsourcing for rating image aesthetic appeal: better a paid or a volunteer crowd? In: Proceedings of 2014 International ACM Workshop Crowdsourcing Multimedia, pp. 25–30. ACM (2014)
11.
Zurück zum Zitat dos Reis, F.J.C., et al.: Crowdsourcing the general public for large scale molecular pathology studies in cancer. EBioMedicine 2, 681–689 (2015)CrossRef dos Reis, F.J.C., et al.: Crowdsourcing the general public for large scale molecular pathology studies in cancer. EBioMedicine 2, 681–689 (2015)CrossRef
Metadaten
Titel
Role of Task Complexity and Training in Crowdsourced Image Annotation
verfasst von
Nadine S. Schaadt
Anne Grote
Germain Forestier
Cédric Wemmert
Friedrich Feuerhake
Copyright-Jahr
2018
DOI
https://doi.org/10.1007/978-3-030-00949-6_6