Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
A Practical Exercise on Re-engineering Clinical Guideline Models Using Different Representation Languages Read chapter Read first chapter

2019 | OriginalPaper | Chapter

A Computational Framework Towards Medical Image Explanation

Authors : Xuwen Wang, Yu Zhang, Zhen Guo, Jiao Li

Published in: Artificial Intelligence in Medicine: Knowledge Representation and Transparent and Explainable Systems

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

In this paper, a unified computational framework towards medical image explanation is proposed to promote the ability of computers on understanding and interpreting medical images. Four complementary modules are included, such as the construction of Medical Image-Text Joint Embedding (MITE) based on large-scale medical images and related texts; a Medical Image Semantic Association (MISA) mechanism based on the MITE multimodal knowledge representation; a Hierarchical Medical Image Caption (HMIC) module that is visually understandable to radiologists; and a language-independent medical imaging report generation prototype system by integrating the HMIC and transfer learning method. As an initial study of automatic medical image explanation, preliminary experiments were carried out to verify the feasibility of the proposed framework, including the extraction of large scale medical image-text pairs, semantic concept detection from medical images, and automatic medical imaging reports generation. However, there is still a great challenge to produce medical image interpretations clinically usable, and further research is needed to empower machines explaining medical images like a human being.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Local vs. Global Interpretability of Machine Learning Models in Type 2 Diabetes Mellitus Screening
next chapter A Computational Framework for Interpretable Anomaly Detection and Classification of Multivariate Time Series with Application to Human Gait Data Analysis
Literature
1.
Interagency Working Group on Medical Imaging Committee on Science, National Science And Technology Council, Roadmap for medical imaging research and development, 12 (2017)
2.
Ma, Q., Kong, D.: A new variational model for joint restoration and segmentation based on the Mumford-Shah model. J. Vis. Commun. Image Represent. 53, 224–234 (2018)CrossRef
3.
Wang, X., Peng, Y., Lu, L., et al.: ChestX-ray8.: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: IEEE CVPR 2017, pp. 2097–2106 (2017)
4.
Demnerfushman, D., Kohli, M.D., Rosenman, M.B., et al.: Preparing a collection of radiology examinations for distribution and retrieval. J. Am. Med. Inform. Assoc. Jamia 23(2), 304–310 (2016)
5.
Iii, S.G.A., Mclennan, G., Bidaut, L., et al.: The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans. Med. Phys. 38(2), 9–15 (2011)
6.
Eickhoff, C., Schwall, I., Garca íSeco de Herrera, A., Müller, H.: Overview of ImageCLEFcaption 2017 - the image caption prediction and concept extraction tasks to understand biomedical images. In: CLEF 2017 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Dublin, Ireland (2017)
7.
Garca íSeco de Herrera, A., Eickhoff, C., Andrearczyk, V., Müller, H.: Overview of the ImageCLEF 2018 caption prediction tasks. In: CLEF 2018 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Avignon, France (2018)
8.
Pelka, O., Friedrich, C.M., Garca íSeco de Herrera, A., Müller, H.: Overview of the ImageCLEFmed 2019 concept detection task. In: CLEF 2019 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Lugano, Switzerland (2019). ISSN 1613-0073
9.
10.
Wang, X., Zhang, Y., Guo, Z., Li, J.: Identifying concepts from medical images via transfer learning and image retrieval. Math. Biosci. Eng. 16(4), 1978–1991 (2019)
11.
Shin, H., Roberts, K., Lu, L., et al.: Learning to read chest X-rays: recurrent neural cascade model for automated image annotation. In: Computer Vision and Pattern Recognition, pp. 2497–2506 (2016)
12.
13.
Krause, J., Johnson, J., Krishna, R., Li, F.: A hierarchical approach for generating descriptive image paragraphs. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
14.
Liu, C., Wang, C., Sun, F., et al: Image2Text: a multimodal image captioner. In: ACM multimedia, pp. 746–748 (2016)
15.
You, Q., Jin, H., Wang, Z., et al: Image captioning with semantic attention. In: Computer Vision and Pattern Recognition, pp. 4651–4659 (2016)
16.
Liang, X., Hu, Z., Zhang, H., Gan, C., Xing, E.P.: Recurrent topic-transition GAN for visual paragraph generation. In: The IEEE International Conference on Computer Vision (ICCV) (2017)
17.
Xu, K., Ba, J., Kiros, R., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning, pp. 2048–2057 (2016)
18.
Hasan, S.A., Ling, Y., Liu, J., Sreenivasan, R., Anand, S., Arora, T.R., Datla, V., Lee, K., Qadir, A., Swisher, C., Farri, O.: Attention-based medical caption generation with image modality classification and clinical concept mapping. In: Bellot, P., et al. (eds.) CLEF 2018. LNCS, vol. 11018, pp. 224–230. Springer, Cham (2018). https://​doi.​org/​10.​1007/​978-3-319-98932-7_​21CrossRef
19.
Zhang, Z., Xie, Y., Xing, F., McGough, M., Yang, L.: MDNet: a semantically and visually interpretable medical image diagnosis network, pp. 3549–3557 (2017)
20.
Jing, B., Xie, P., Eric, X.: On the automatic generation of medical imaging reports. In: Conference 2018, CVPR, pp. 2577–2586, Melbourne, Australia (2018)
21.
Wang, X., Peng, Y., Lu, L., et al: TieNet: text-image embedding network for common thorax disease classification and reporting in chest X-rays. In: Conference 2018, CVPR (2018)
22.
23.
Hsu, W., Glass, J.: Disentangling by partitioning: a representation learning framework for multimodal sensory data, p.1805. arXiv (2018)
24.
Angelov, P.P., Gu, X.: Deep rule based classifier with human-level performance and characteristics. Inf. Sci. 463–464, 196–213 (2018)CrossRef
25.
Gu, X., Angelov, P.P.: Semi-supervised deep rule-based approach for image classification. Appl. Soft Comput. 68, 53–68 (2018)CrossRef
26.
27.
28.
29.
Lux, M., Chatzichristofis, S.A.: Lire: Lucene image retrieval: an extensible Java CBIR library. In: Proceedings of the 16th ACM International Conference on Multimedia. British Columbia, Canada (2008)
30.
Guo, Z., Wang, X., Zhang, Y., Li, J.: ImageSem at ImageCLEFmed caption 2019 task: a two-stage medical concept detection strategy. In: CLEF 2019 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Lugano, Switzerland (2019)
31.
Zhang, Y.: Automatic generation of medical imaging report generation based on deep learning, Peking Union Medical College (2019)
32.
Demner-Fushman, D., Antani, S., Simpson, M., et al.: Design and development of a multimodal biomedical information retrieval system. J. Comput. Sci. Eng. 6(2), 168–177 (2012)CrossRef
33.
Papineni, K., Roukos, S., Ward, T., et al.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318 (2002)
34.
Denkowski, M., Lavie, A.: Meteor universal: language specific translation evaluation for any target language. In: Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 376–380 (2014)
35.
Lin, C.: Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches out: Proceedings of the ACL-04 Workshop, vol. 8, Barcelona, Spain (2004)
36.
Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015)
Metadata
Title
A Computational Framework Towards Medical Image Explanation
Authors
Xuwen Wang
Yu Zhang
Zhen Guo
Jiao Li
Copyright Year
2019
Book
Artificial Intelligence in Medicine: Knowledge Representation and Transparent and Explainable Systems

Print ISBN: 978-3-030-37445-7

Electronic ISBN: 978-3-030-37446-4

Copyright Year: 2019

DOI
https://doi.org/10.1007/978-3-030-37446-4_10

Premium Partner

    Image Credits