Skip to main content
main-content

Tipp

Weitere Kapitel dieses Buchs durch Wischen aufrufen

2018 | OriginalPaper | Buchkapitel

Radiology Objects in COntext (ROCO): A Multimodal Image Dataset

verfasst von: Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, Christoph M. Friedrich

Erschienen in: Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis

Verlag: Springer International Publishing

share
TEILEN

Abstract

This work introduces a new multimodal image dataset, with the aim of detecting the interplay between visual elements and semantic relations present in radiology images. The objective is accomplished by retrieving all image-caption pairs from the open-access biomedical literature database PubMedCentral, as these captions describe the visual content in their semantic context. All compound, multi-pane, and non-radiology images were eliminated using an automatic binary classifier fine-tuned with a deep convolutional neural network system. Radiology Objects in COntext (ROCO) dataset contains over 81k radiology images with several medical imaging modalities including Computer Tomography, Ultrasound, X-Ray, Fluoroscopy, Positron Emission Tomography, Mammography, Magnetic Resonance Imaging, Angiography. All images in ROCO have corresponding caption, keywords, Unified Medical Language Systems Concept Unique Identifiers and Semantic Type. An out-of-class set with 6k images ranging from synthetic radiology figures to digital arts is provided, to improve prediction and classification performance. Adopting ROCO, systems for caption and keywords generation can be modeled, which allows multimodal representation for datasets lacking text representation. Systems with the goal of image structuring and semantic information tagging can be created using ROCO, which is beneficial and of assistance for image and information retrieval purposes.
Literatur
1.
Zurück zum Zitat Abadi, M., et al.: Tensorflow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, Berkeley, CA, USA. USENIX Association (2016) Abadi, M., et al.: Tensorflow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, Berkeley, CA, USA. USENIX Association (2016)
4.
Zurück zum Zitat García Seco de Herrera, A., Kalpathy-Cramer, J., Demner Fushman, D., Antani, S., Müller, H.: Overview of the ImageCLEF 2013 medical tasks. In: Working Notes of CLEF 2013 - Conference and Labs of the Evaluation forum. CEUR-WS Proceedings Notes, vol. 1179, Valencia, Spain, 23–26 September 2013 (2013) García Seco de Herrera, A., Kalpathy-Cramer, J., Demner Fushman, D., Antani, S., Müller, H.: Overview of the ImageCLEF 2013 medical tasks. In: Working Notes of CLEF 2013 - Conference and Labs of the Evaluation forum. CEUR-WS Proceedings Notes, vol. 1179, Valencia, Spain, 23–26 September 2013 (2013)
5.
Zurück zum Zitat García Seco de Herrera, A., Müller, H., Bromuri, S.: Overview of the ImageCLEF 2015 medical classification task. In: Working Notes of CLEF 2015 - Conference and Labs of the Evaluation Forum. CEUR-WS Proceedings Notes, vol. 1391, Toulouse, France, 8–11 September 2015 (2015) García Seco de Herrera, A., Müller, H., Bromuri, S.: Overview of the ImageCLEF 2015 medical classification task. In: Working Notes of CLEF 2015 - Conference and Labs of the Evaluation Forum. CEUR-WS Proceedings Notes, vol. 1391, Toulouse, France, 8–11 September 2015 (2015)
6.
Zurück zum Zitat García Seco de Herrera, A., Schaer, R., Bromuri, S., Müller, H.: Overview of the ImageCLEF 2016 medical task. In: Working Notes of CLEF 2016 - Conference and Labs of the Evaluation forum, Évora. CEUR-WS Proceedings Notes, vol. 1609, Portugal, 5–8 September 2016 (2016) García Seco de Herrera, A., Schaer, R., Bromuri, S., Müller, H.: Overview of the ImageCLEF 2016 medical task. In: Working Notes of CLEF 2016 - Conference and Labs of the Evaluation forum, Évora. CEUR-WS Proceedings Notes, vol. 1609, Portugal, 5–8 September 2016 (2016)
8.
Zurück zum Zitat Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, vol. 1, pp. 1097–1105. Curran Associates Inc., USA (2012) Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, vol. 1, pp. 1097–1105. Curran Associates Inc., USA (2012)
11.
Zurück zum Zitat Müller, H., Kalpathy-Cramer, J., Demner-Fushman, D., Antani, S.: Creating a classification of image types in the medical literature for visual categorization. In: Proceedings of SPIE 8319, Medical Imaging 2012: Advanced PACS-Based Imaging Informatics and Therapeutic Applications, 83190P, 23 February 2012, vol. 8425, p. 194 (2012). https://​doi.​org/​10.​1117/​12.​911186 Müller, H., Kalpathy-Cramer, J., Demner-Fushman, D., Antani, S.: Creating a classification of image types in the medical literature for visual categorization. In: Proceedings of SPIE 8319, Medical Imaging 2012: Advanced PACS-Based Imaging Informatics and Therapeutic Applications, 83190P, 23 February 2012, vol. 8425, p. 194 (2012). https://​doi.​org/​10.​1117/​12.​911186
13.
14.
Zurück zum Zitat Pelka, O., Friedrich, C.M.: Keyword generation for biomedical image retrieval with recurrent neural networks. In: Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum. CEUR-WS Proceedings Notes, vol. 1866, Dublin, Ireland, 11–14 September 2017 (2017) Pelka, O., Friedrich, C.M.: Keyword generation for biomedical image retrieval with recurrent neural networks. In: Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum. CEUR-WS Proceedings Notes, vol. 1866, Dublin, Ireland, 11–14 September 2017 (2017)
15.
Zurück zum Zitat Pelka, O., Nensa, F., Friedrich, C.M.: Adopting semantic information of grayscale radiographs for image classification and retrieval. In: Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018). BIOIMAGING, vol. 2, Funchal, Madeira, Portugal, 19–21 January 2018, pp. 179–187 (2018). https://​doi.​org/​10.​5220/​0006732301790187​ Pelka, O., Nensa, F., Friedrich, C.M.: Adopting semantic information of grayscale radiographs for image classification and retrieval. In: Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018). BIOIMAGING, vol. 2, Funchal, Madeira, Portugal, 19–21 January 2018, pp. 179–187 (2018). https://​doi.​org/​10.​5220/​0006732301790187​
20.
Zurück zum Zitat Soldaini, L., Goharian, N.: QuickUMLS: a fast, unsupervised approach for medical concept extraction. In: MedIR Workshop, SIGIR (2016) Soldaini, L., Goharian, N.: QuickUMLS: a fast, unsupervised approach for medical concept extraction. In: MedIR Workshop, SIGIR (2016)
22.
Zurück zum Zitat Tommasi, T., Caputo, B., Welter, P., Güld, M.O., Deserno, T.M.: Overviewof the CLEF 2009 medical image annotation track. In: Multilingual Information Access Evaluation II. Multimedia Experiments - 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, 30 September–2 October 2009, pp. 85–93 (2009). https://​doi.​org/​10.​1007/​978-3-642-15751-6_​9 Tommasi, T., Caputo, B., Welter, P., Güld, M.O., Deserno, T.M.: Overviewof the CLEF 2009 medical image annotation track. In: Multilingual Information Access Evaluation II. Multimedia Experiments - 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, 30 September–2 October 2009, pp. 85–93 (2009). https://​doi.​org/​10.​1007/​978-3-642-15751-6_​9
24.
Zurück zum Zitat Xu, Y., Mo, T., Feng, Q., Zhong, P., Lai, M., Chang, E.I.: Deep learning of feature representation with multiple instance learning for medical image analysis. In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2014, Florence, Italy, 4–9 May 2014, pp. 1626–1630 (2014). https://​doi.​org/​10.​1109/​ICASSP.​2014.​6853873 Xu, Y., Mo, T., Feng, Q., Zhong, P., Lai, M., Chang, E.I.: Deep learning of feature representation with multiple instance learning for medical image analysis. In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2014, Florence, Italy, 4–9 May 2014, pp. 1626–1630 (2014). https://​doi.​org/​10.​1109/​ICASSP.​2014.​6853873
Metadaten
Titel
Radiology Objects in COntext (ROCO): A Multimodal Image Dataset
verfasst von
Obioma Pelka
Sven Koitka
Johannes Rückert
Felix Nensa
Christoph M. Friedrich
Copyright-Jahr
2018
DOI
https://doi.org/10.1007/978-3-030-01364-6_20

Premium Partner