Top

Published in:

2019 | OriginalPaper | Chapter

Deep Learning in Multimodal Medical Image Analysis

Author : Yan Xu

Published in: Health Information Science

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Various imaging modalities (CT, MRI, PET, etc.) encompass abundant information which is different and complementary to each other. It is reasonable to combine images from multiple modalities to make a more accurate assessment. Multimodal medical imaging has shown notable achievements in improving clinical accuracy. Deep learning has achieved great success in image recognition, and also shown huge potential for multimodal medical imaging analysis. This paper gives a review of deep learning in multimodal medical imaging analysis, aiming to provide a starting point for people interested in this field, and highlight gaps and challenges of this topic. Based on the introduction of basic ideas of deep learning and medical imaging, the state-of-the-art multimodal medical image analysis is given, with emphasis on the fusion technique and feature extraction deep models. Multimodal medical image applications, especially cross-modality related, are also summarized.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Research and Development of Three-Dimensional Brain Augmented Reality System Based on PACS and Medical Image Characteristics

next chapter A R-CNN Based Approach for Microaneurysm Detection in Retinal Fundus Images

Litjens, G., Kooi, T., Bejnordi, B.E., et al.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)CrossRef

Lundervold, A.S., Lundervold, A.: An overview of deep learning in medical imaging focusing on MRI. https://arxiv.org/abs/1811.10052 (2018)

Liu, S., et al.: Deep learning in medical ultrasound analysis: a review, engineering (2019)CrossRef

James, A.P., Dasarathy, B.V.: Medical image fusion: a survey of the state of the art. Inf. Fusion 19, 4–19 (2014)CrossRef

Guo, Z., Li, X., Huang, H., Guo, N., Li, Q.: Deep learning-based image segmentation on multimodal medical imaging. IEEE Trans. Radiat. Plasma Med. Sci. 3(2), 162–169 (2019)CrossRef

Zhang, W., Li, R., Deng, H., Wang, L., Lin, W., et al.: Deep convolutional neural networks for multimodality isointense infant brain image segmentation. NeuroImage 108, 214–224 (2015)CrossRef

Liu, S., et al.: Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer’s disease. IEEE Trans. Biomed. Eng. 62(4), 1132–1140 (2015)CrossRef

Suk, H., Shen, D.: Deep learning-based feature representation for AD/MCI classification. In: Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 583–590 (2013)CrossRef

Shi, B., Chen, Y., Zhang, P., Smith, C.D., Liu, J.: Nonlinear feature transformation and deep fusion for Alzheimer’s disease staging analysis. Pattern Recognit. 63, 487–498 (2017)CrossRef

10.

Suk, H.I., Lee, S.W., Shen, D.G., ADNI: Hierarchical feature representation and multimodal fusion with deep learning for AD MCI diagnosis. NeuroImage 101, 569–582 (2014)CrossRef

11.

Li, F., Tran, L., Thung, K.H., Ji, S.W., Shen, D.G., Li, J.: A robust deep model for improved classification of AD/MCI patients. IEEE J. Biomed. Health Inform. 19(5), 1610–1616 (2015)CrossRef

12.

Shi, J., Zheng, X., Li, Y., Zhang, Q., Ying, S.: Multimodal neuroimaging feature learning with multimodal stacked deep polynomial networks for diagnosis of Alzheimer’s disease. IEEE J. Biomed. Health Inform. 22(1), 173–183 (2018)CrossRef

13.

Gupta, A., Ayhan, M., Maida, A.: Natural image bases to represent neuroimaging data. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA, 16–21 June, pp. 987–994 (2013)

14.

Carneiro, G., Nascimento, J., Bradley, A.P.: Unregistered multiview mammogram analysis with pre-trained deep learning models. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 652–660. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_78CrossRef

15.

Hadad, O., Bakalo, R., Ben-Ari, R., et al.: Classification of breast lesions using cross-modal deep learning. In: IEEE International Symposium on Biomedical Imaging. IEEE (2017)

16.

Nie, D., Wang, L., Gao, Y., Shen, D.: Fully convolutional networks for multi-modality isointense infant brain image segmentation. In: Proceedings of the 13th IEEE International Symposium on Biomedical Imaging, Washington, DC, pp. 1342–1345 (2016)

17.

Zeng, G., Zheng, G.: Multi-stream 3D FCN with multi-scale deep supervision for multi-modality isointense infant brain MR image segmentation. https://arxiv.org/abs/1711.10212, March 2019

18.

Huo, Y., Xu, Z., Moon, H., et al.: SynSeg-Net: synthetic segmentation without target modality ground truth. IEEE Trans. Med. Imaging 38(4), 1016–1025 (2019)CrossRef

19.

Zhang, Z., Yang, L., Zheng, Y.: Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network. https://arxiv.org/abs/1802.09655 (2018)

20.

Nie, D., Zhang, H., Adeli, E., Liu, L., Shen, D.: 3D deep learning for multi-modal imaging-guided survival time prediction of brain tumor patients. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 212–220. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_25CrossRef

21.

Wolterink, J.M., Dinkla, A.M., Savenije, M.H.F., Seevinck, P.R., van den Berg, C.A.T., Išgum, I.: Deep MR to CT synthesis using unpaired data. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2017. LNCS, vol. 10557, pp. 14–23. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68127-6_2CrossRef

22.

Chartsias, A., Joyce, T., Dharmakumar, R., Tsaftaris, S.A.: Adversarial image synthesis for unpaired multi-modal cardiac data. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2017. LNCS, vol. 10557, pp. 3–13. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68127-6_1CrossRef

23.

Van Nguyen, H., Zhou, K., Vemulapalli, R.: Cross-domain synthesis of medical images using efficient location-sensitive deep network. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 677–684. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24553-9_83CrossRef

24.

Sevetlidis, V., Giuffrida, M.V., Tsaftaris, S.A.: Whole image synthesis using a deep encoder-decoder network. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2016. LNCS, vol. 9968, pp. 127–137. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46630-9_13CrossRef

25.

Xiang, L., et al.: Deep embedding convolutional neural network for synthesizing CT image from T1-weighted MR image. Med. Image Anal. 47, 31–44 (2018)CrossRef

26.

Nie, D., et al.: Medical image synthesis with context-aware generative adversarial networks. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 417–425. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_48CrossRef

27.

Costa, P., et al.: Towards adversarial retinal image synthesis. https://arxiv.org/abs/1701.08974 (2017)

Title: Deep Learning in Multimodal Medical Image Analysis
Author: Yan Xu
Publisher: Springer International Publishing
Book: Health Information Science
Print ISBN: 978-3-030-32961-7

Electronic ISBN: 978-3-030-32962-4

Copyright Year: 2019
DOI: https://doi.org/10.1007/978-3-030-32962-4_18

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner