Skip to main content

2019 | OriginalPaper | Buchkapitel

A Computational Framework Towards Medical Image Explanation

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

In this paper, a unified computational framework towards medical image explanation is proposed to promote the ability of computers on understanding and interpreting medical images. Four complementary modules are included, such as the construction of Medical Image-Text Joint Embedding (MITE) based on large-scale medical images and related texts; a Medical Image Semantic Association (MISA) mechanism based on the MITE multimodal knowledge representation; a Hierarchical Medical Image Caption (HMIC) module that is visually understandable to radiologists; and a language-independent medical imaging report generation prototype system by integrating the HMIC and transfer learning method. As an initial study of automatic medical image explanation, preliminary experiments were carried out to verify the feasibility of the proposed framework, including the extraction of large scale medical image-text pairs, semantic concept detection from medical images, and automatic medical imaging reports generation. However, there is still a great challenge to produce medical image interpretations clinically usable, and further research is needed to empower machines explaining medical images like a human being.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Interagency Working Group on Medical Imaging Committee on Science, National Science And Technology Council, Roadmap for medical imaging research and development, 12 (2017) Interagency Working Group on Medical Imaging Committee on Science, National Science And Technology Council, Roadmap for medical imaging research and development, 12 (2017)
2.
Zurück zum Zitat Ma, Q., Kong, D.: A new variational model for joint restoration and segmentation based on the Mumford-Shah model. J. Vis. Commun. Image Represent. 53, 224–234 (2018)CrossRef Ma, Q., Kong, D.: A new variational model for joint restoration and segmentation based on the Mumford-Shah model. J. Vis. Commun. Image Represent. 53, 224–234 (2018)CrossRef
3.
Zurück zum Zitat Wang, X., Peng, Y., Lu, L., et al.: ChestX-ray8.: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: IEEE CVPR 2017, pp. 2097–2106 (2017) Wang, X., Peng, Y., Lu, L., et al.: ChestX-ray8.: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: IEEE CVPR 2017, pp. 2097–2106 (2017)
4.
Zurück zum Zitat Demnerfushman, D., Kohli, M.D., Rosenman, M.B., et al.: Preparing a collection of radiology examinations for distribution and retrieval. J. Am. Med. Inform. Assoc. Jamia 23(2), 304–310 (2016) Demnerfushman, D., Kohli, M.D., Rosenman, M.B., et al.: Preparing a collection of radiology examinations for distribution and retrieval. J. Am. Med. Inform. Assoc. Jamia 23(2), 304–310 (2016)
5.
Zurück zum Zitat Iii, S.G.A., Mclennan, G., Bidaut, L., et al.: The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans. Med. Phys. 38(2), 9–15 (2011) Iii, S.G.A., Mclennan, G., Bidaut, L., et al.: The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans. Med. Phys. 38(2), 9–15 (2011)
6.
Zurück zum Zitat Eickhoff, C., Schwall, I., Garca íSeco de Herrera, A., Müller, H.: Overview of ImageCLEFcaption 2017 - the image caption prediction and concept extraction tasks to understand biomedical images. In: CLEF 2017 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Dublin, Ireland (2017) Eickhoff, C., Schwall, I., Garca íSeco de Herrera, A., Müller, H.: Overview of ImageCLEFcaption 2017 - the image caption prediction and concept extraction tasks to understand biomedical images. In: CLEF 2017 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Dublin, Ireland (2017)
7.
Zurück zum Zitat Garca íSeco de Herrera, A., Eickhoff, C., Andrearczyk, V., Müller, H.: Overview of the ImageCLEF 2018 caption prediction tasks. In: CLEF 2018 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Avignon, France (2018) Garca íSeco de Herrera, A., Eickhoff, C., Andrearczyk, V., Müller, H.: Overview of the ImageCLEF 2018 caption prediction tasks. In: CLEF 2018 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Avignon, France (2018)
8.
Zurück zum Zitat Pelka, O., Friedrich, C.M., Garca íSeco de Herrera, A., Müller, H.: Overview of the ImageCLEFmed 2019 concept detection task. In: CLEF 2019 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Lugano, Switzerland (2019). ISSN 1613-0073 Pelka, O., Friedrich, C.M., Garca íSeco de Herrera, A., Müller, H.: Overview of the ImageCLEFmed 2019 concept detection task. In: CLEF 2019 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Lugano, Switzerland (2019). ISSN 1613-0073
10.
Zurück zum Zitat Wang, X., Zhang, Y., Guo, Z., Li, J.: Identifying concepts from medical images via transfer learning and image retrieval. Math. Biosci. Eng. 16(4), 1978–1991 (2019) Wang, X., Zhang, Y., Guo, Z., Li, J.: Identifying concepts from medical images via transfer learning and image retrieval. Math. Biosci. Eng. 16(4), 1978–1991 (2019)
11.
Zurück zum Zitat Shin, H., Roberts, K., Lu, L., et al.: Learning to read chest X-rays: recurrent neural cascade model for automated image annotation. In: Computer Vision and Pattern Recognition, pp. 2497–2506 (2016) Shin, H., Roberts, K., Lu, L., et al.: Learning to read chest X-rays: recurrent neural cascade model for automated image annotation. In: Computer Vision and Pattern Recognition, pp. 2497–2506 (2016)
13.
Zurück zum Zitat Krause, J., Johnson, J., Krishna, R., Li, F.: A hierarchical approach for generating descriptive image paragraphs. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) Krause, J., Johnson, J., Krishna, R., Li, F.: A hierarchical approach for generating descriptive image paragraphs. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
14.
Zurück zum Zitat Liu, C., Wang, C., Sun, F., et al: Image2Text: a multimodal image captioner. In: ACM multimedia, pp. 746–748 (2016) Liu, C., Wang, C., Sun, F., et al: Image2Text: a multimodal image captioner. In: ACM multimedia, pp. 746–748 (2016)
15.
Zurück zum Zitat You, Q., Jin, H., Wang, Z., et al: Image captioning with semantic attention. In: Computer Vision and Pattern Recognition, pp. 4651–4659 (2016) You, Q., Jin, H., Wang, Z., et al: Image captioning with semantic attention. In: Computer Vision and Pattern Recognition, pp. 4651–4659 (2016)
16.
Zurück zum Zitat Liang, X., Hu, Z., Zhang, H., Gan, C., Xing, E.P.: Recurrent topic-transition GAN for visual paragraph generation. In: The IEEE International Conference on Computer Vision (ICCV) (2017) Liang, X., Hu, Z., Zhang, H., Gan, C., Xing, E.P.: Recurrent topic-transition GAN for visual paragraph generation. In: The IEEE International Conference on Computer Vision (ICCV) (2017)
17.
Zurück zum Zitat Xu, K., Ba, J., Kiros, R., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning, pp. 2048–2057 (2016) Xu, K., Ba, J., Kiros, R., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning, pp. 2048–2057 (2016)
18.
Zurück zum Zitat Hasan, S.A., Ling, Y., Liu, J., Sreenivasan, R., Anand, S., Arora, T.R., Datla, V., Lee, K., Qadir, A., Swisher, C., Farri, O.: Attention-based medical caption generation with image modality classification and clinical concept mapping. In: Bellot, P., et al. (eds.) CLEF 2018. LNCS, vol. 11018, pp. 224–230. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98932-7_21CrossRef Hasan, S.A., Ling, Y., Liu, J., Sreenivasan, R., Anand, S., Arora, T.R., Datla, V., Lee, K., Qadir, A., Swisher, C., Farri, O.: Attention-based medical caption generation with image modality classification and clinical concept mapping. In: Bellot, P., et al. (eds.) CLEF 2018. LNCS, vol. 11018, pp. 224–230. Springer, Cham (2018). https://​doi.​org/​10.​1007/​978-3-319-98932-7_​21CrossRef
19.
Zurück zum Zitat Zhang, Z., Xie, Y., Xing, F., McGough, M., Yang, L.: MDNet: a semantically and visually interpretable medical image diagnosis network, pp. 3549–3557 (2017) Zhang, Z., Xie, Y., Xing, F., McGough, M., Yang, L.: MDNet: a semantically and visually interpretable medical image diagnosis network, pp. 3549–3557 (2017)
20.
Zurück zum Zitat Jing, B., Xie, P., Eric, X.: On the automatic generation of medical imaging reports. In: Conference 2018, CVPR, pp. 2577–2586, Melbourne, Australia (2018) Jing, B., Xie, P., Eric, X.: On the automatic generation of medical imaging reports. In: Conference 2018, CVPR, pp. 2577–2586, Melbourne, Australia (2018)
21.
Zurück zum Zitat Wang, X., Peng, Y., Lu, L., et al: TieNet: text-image embedding network for common thorax disease classification and reporting in chest X-rays. In: Conference 2018, CVPR (2018) Wang, X., Peng, Y., Lu, L., et al: TieNet: text-image embedding network for common thorax disease classification and reporting in chest X-rays. In: Conference 2018, CVPR (2018)
23.
Zurück zum Zitat Hsu, W., Glass, J.: Disentangling by partitioning: a representation learning framework for multimodal sensory data, p.1805. arXiv (2018) Hsu, W., Glass, J.: Disentangling by partitioning: a representation learning framework for multimodal sensory data, p.1805. arXiv (2018)
24.
Zurück zum Zitat Angelov, P.P., Gu, X.: Deep rule based classifier with human-level performance and characteristics. Inf. Sci. 463–464, 196–213 (2018)CrossRef Angelov, P.P., Gu, X.: Deep rule based classifier with human-level performance and characteristics. Inf. Sci. 463–464, 196–213 (2018)CrossRef
25.
Zurück zum Zitat Gu, X., Angelov, P.P.: Semi-supervised deep rule-based approach for image classification. Appl. Soft Comput. 68, 53–68 (2018)CrossRef Gu, X., Angelov, P.P.: Semi-supervised deep rule-based approach for image classification. Appl. Soft Comput. 68, 53–68 (2018)CrossRef
29.
Zurück zum Zitat Lux, M., Chatzichristofis, S.A.: Lire: Lucene image retrieval: an extensible Java CBIR library. In: Proceedings of the 16th ACM International Conference on Multimedia. British Columbia, Canada (2008) Lux, M., Chatzichristofis, S.A.: Lire: Lucene image retrieval: an extensible Java CBIR library. In: Proceedings of the 16th ACM International Conference on Multimedia. British Columbia, Canada (2008)
30.
Zurück zum Zitat Guo, Z., Wang, X., Zhang, Y., Li, J.: ImageSem at ImageCLEFmed caption 2019 task: a two-stage medical concept detection strategy. In: CLEF 2019 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Lugano, Switzerland (2019) Guo, Z., Wang, X., Zhang, Y., Li, J.: ImageSem at ImageCLEFmed caption 2019 task: a two-stage medical concept detection strategy. In: CLEF 2019 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Lugano, Switzerland (2019)
31.
Zurück zum Zitat Zhang, Y.: Automatic generation of medical imaging report generation based on deep learning, Peking Union Medical College (2019) Zhang, Y.: Automatic generation of medical imaging report generation based on deep learning, Peking Union Medical College (2019)
32.
Zurück zum Zitat Demner-Fushman, D., Antani, S., Simpson, M., et al.: Design and development of a multimodal biomedical information retrieval system. J. Comput. Sci. Eng. 6(2), 168–177 (2012)CrossRef Demner-Fushman, D., Antani, S., Simpson, M., et al.: Design and development of a multimodal biomedical information retrieval system. J. Comput. Sci. Eng. 6(2), 168–177 (2012)CrossRef
33.
Zurück zum Zitat Papineni, K., Roukos, S., Ward, T., et al.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318 (2002) Papineni, K., Roukos, S., Ward, T., et al.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318 (2002)
34.
Zurück zum Zitat Denkowski, M., Lavie, A.: Meteor universal: language specific translation evaluation for any target language. In: Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 376–380 (2014) Denkowski, M., Lavie, A.: Meteor universal: language specific translation evaluation for any target language. In: Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 376–380 (2014)
35.
Zurück zum Zitat Lin, C.: Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches out: Proceedings of the ACL-04 Workshop, vol. 8, Barcelona, Spain (2004) Lin, C.: Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches out: Proceedings of the ACL-04 Workshop, vol. 8, Barcelona, Spain (2004)
36.
Zurück zum Zitat Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015) Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015)
Metadaten
Titel
A Computational Framework Towards Medical Image Explanation
verfasst von
Xuwen Wang
Yu Zhang
Zhen Guo
Jiao Li
Copyright-Jahr
2019
DOI
https://doi.org/10.1007/978-3-030-37446-4_10