Pablo Messina
Denis Parra
Skip Abstract Section
Abstract
Every year physicians face an increasing demand of image-based diagnosis from patients, a problem that can be addressed with recent artificial intelligence methods. In this context, we survey works in the area of automatic report generation from medical images, with emphasis on methods using deep neural networks, with respect to (1) Datasets, (2) Architecture Design, (3) Explainability, and (4) Evaluation Metrics. Our survey identifies interesting developments but also remaining challenges. Among them, the current evaluation of generated reports is especially weak, since it mostly relies on traditional Natural Language Processing (NLP) metrics, which do not accurately capture medical correctness.
- [1] . 2013. Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmology 131, 3 (2013), 351–357.Google Scholar
Cross Ref
- [2] . 2018. Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access 6 (2018), 52138–52160.Google ScholarCross Ref
- [3] . 2018. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems 31. Curran Associates, Inc., 9505–9515.Google Scholar
- [4] . 2018. Interpretable machine learning in healthcare. In Proc. of the 2018 ACM Intl. Conf. on Bioinformatics, Computational Biology, and Health Informatics (BCB’18). ACM, New York, NY, 559–560.Google Scholar
- [5] . 2019. Automated generation of radiologic descriptions on brain volume changes from T1-weighted MR images: Initial assessment of feasibility. Frontiers in Neurology 10 (2019), 7.Google ScholarCross Ref
- [6] . 2018. Automatic caption generation for medical images. In Proc. of the 3rd Intl. Conf. on Smart City Applications (SCA’18). ACM, New York, NY, Article
86 , 6 pages.Google Scholar - [7] . 2019. Captioning ultrasound images automatically. In Medical Image Computing and Computer Assisted Intervention (MICCAI’19). Springer Intl. Publishing, Cham, 338–346.Google Scholar
- [8] . 2019. Guidelines for human-AI interaction. In Proc. of the 2019 CHI Conf. on Human Factors in Computing Systems (CHI’19). ACM, 1–13.Google Scholar
- [9] . 2016. SPICE: Semantic propositional image caption evaluation. In Computer Vision (ECCV’16). Springer Intl. Publishing, Cham, 382–398.Google Scholar
- [10] . 2010. An overview of MetaMap: Historical perspective and recent advances. Journal of the American Medical Informatics Association 17, 3 (2010), 229–236.Google ScholarCross Ref
- [11] . 2021. Evaluating diagnostic content of AI-generated radiology reports of chest X-rays. Artificial Intelligence in Medicine 116 (2021), 102075. Google ScholarCross Ref
- [12] . 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. ACL, 65–72.Google Scholar
- [13] . 2020. Overview of the VQA-med task at ImageCLEF 2020: Visual question answering and generation in the medical domain. In CLEF 2020 Working Notes (CEUR Workshop Proceedings).Google Scholar
- [14] . 2019. VQA-Med: Overview of the medical visual question answering task at ImageCLEF 2019. In CLEF2019 Working Notes (CEUR Workshop Proceedings).Google Scholar
- [15] . 2009. Curriculum learning. In Proc. of the 26th Annual Intl. Conf. on Machine Learning (ICML’09). ACM, 41–48.Google Scholar
- [16] . 2020. CLARA: Clinical report auto-completion. In Proc. of the Web Conf. 2020 (WWW’20). ACM, New York, NY, 541–550.Google Scholar
- [17] . 2020. Baselines for chest X-ray report generation. In Proc. of the Machine Learning for Health NeurIPS Workshop (Proc. of Machine Learning Research), Vol. 116. PMLR, 126–140.Google Scholar
- [18] . 2010. Retrieval of bibliographic records using Apache Lucene. Electronic Library 28, 4 (
Jan. 2010), 525–539.Google ScholarCross Ref - [19] . 2019. Padchest: A large chest x-ray image dataset with multi-label annotated reports. arXiv:1901.07441 (2019).Google Scholar
- [20] . 2019. “Hello AI”: Uncovering the onboarding needs of medical practitioners for human-AI collaborative decision-making. Proc. of the ACM on Human-computer Interaction 3, CSCW (2019), 1–24.Google ScholarDigital Library
- [21] . 1997. Multitask learning. Machine Learning 28, 1 (1997), 41–75.Google ScholarDigital Library
- [22] . 2019. Machine learning interpretability: A survey on methods and metrics. Electronics 8, 8 (2019), 832.Google ScholarCross Ref
- [23] . 2014. A fast and accurate dependency parser using neural networks. In Proc. of the 2014 Conf. on Empirical Methods in Natural Language Processing (EMNLP’14). ACL, 740–750.Google Scholar
- [24] . 2005. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (CVPR’05), Vol. 1. IEEE, 539–546.Google Scholar
- [25] . 2017. Lits-liver tumor segmentation challenge. ISBI and MICCAI (2017).Google Scholar
- [26] . 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning.Google Scholar
- [27] . 2014. Feedback on a publicly distributed image database: The Messidor database. Image Analysis & Stereology 33, 3 (2014), 231–234.Google ScholarCross Ref
- [28] . 2015. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association 23, 2 (
2015), 304–310.Google Scholar Cross Ref - [29] . 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conf. on Computer Vision and Pattern Recognition. 248–255.Google Scholar
- [30] . 2010. Extending the meteor machine translation evaluation metric to the phrase level. In Human Language Technologies: The 2010 Annual Conf. of the North American Chapter of the ACL (HLT’10). ACL, 250–253.Google Scholar
- [31] . 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proc. of the 6th Workshop on Statistical Machine Translation (WMT’11). ACL, 85–91.Google Scholar
- [32] . 2014. Meteor universal: Language specific translation evaluation for any target language. In Proc. of the 9th Workshop on Statistical Machine Translation. ACL, 376–380.Google Scholar
- [33] . 2017. Towards a rigorous science of interpretable machine learning. stat 1050 (2017), 2.Google Scholar
- [34] . 2018. Explainable artificial intelligence: A survey. In 2018 41st Intl. Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO’18). 0210–0215.Google Scholar
- [35] . 2017. Overview of ImageCLEFcaption 2017 - The image caption prediction and concept extraction tasks to understand biomedical images. In CLEF2017 Working Notes (CEUR Workshop Proceedings).Google Scholar
- [36] . 2020. Automatic report generation for chest X-ray images: A multilevel multi-attention approach. In Computer Vision and Image Processing. Springer, Singapore, 174–182.Google Scholar
- [37] . 2017. Detecting hip fractures with radiologist-level performance using deep neural networks. arXiv:1711.06504 (2017).Google Scholar
- [38] . 2019. Producing radiologist-quality reports for interpretable deep learning. In 2019 IEEE 16th Intl. Symposium on Biomedical Imaging (ISBI’19). 1275–1279.Google Scholar
- [39] . 2018. Overview of the ImageCLEF 2018 caption prediction tasks. In CLEF2018 Working Notes (CEUR Workshop Proceedings).Google Scholar
- [40] . 2019. Automated enriched medical concept generation for chest X-ray images. In Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support. Springer Intl. Publishing, Cham, 83–92.Google ScholarDigital Library
- [41] . 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems 27. Curran Associates, Inc., 2672–2680.Google Scholar
- [42] . 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks 18, 5–6 (2005), 602–610.Google ScholarDigital Library
- [43] . 2018. Regression concept vectors for bidirectional explanations in histopathology. In Understanding and Interpreting Machine Learning in Medical Image Computing Applications. Springer Intl. Publishing, Cham, 124–132.Google ScholarCross Ref
- [44] . 2019. Automatic generation of pulmonary radiology reports with semantic tags. In 2019 IEEE 11th Intl. Conf. on Advanced Infocomm Technology (ICAIT’19). 162–167.Google Scholar
- [45] . 2018. A survey of methods for explaining black box models. ACM Computing Surveys (CSUR) 51, 5 (2018), 1–42.Google ScholarDigital Library
- [46] . 2017. Explainable artificial intelligence (XAI).
DOI: Google ScholarCross Ref - [47] . 2018. Towards automatic report generation in spine radiology using weakly supervised framework. In Medical Image Computing and Computer Assisted Intervention (MICCAI’18). Springer Intl. Publishing, Cham, 185–193.Google Scholar
- [48] . 2019. Addressing data bias problems for chest X-ray image report generation. arXiv abs/1908.02123 (2019).Google Scholar
- [49] . 2019. Automatic disease detection and report generation for gastrointestinal tract examination. In Proc. of the 27th ACM Intl. Conf. on Multimedia (MM’19). ACM, New York, NY, 2573–2577.Google Scholar
- [50] . 2018. Overview of the ImageCLEF 2018 medical domain visual question answering task. In CLEF2018 Working Notes (CEUR Workshop Proceedings).Google Scholar
- [51] . 2018. Attention-based medical caption generation with image modality classification and clinical concept mapping. In Experimental IR Meets Multilinguality, Multimodality, and Interaction. Springer Intl. Publishing, Cham, 224–230.Google Scholar
- [52] . 2016. Deep residual learning for image recognition. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’16). 770–778.Google Scholar
- [53] . 2020. PathVQA: 30000+ questions for medical visual question answering. arXiv:2003.10286 (2020).Google Scholar
- [54] . 2001. The digital database for screening mammography. In Proc of the 5th Intl. Workshop on Digital Mammography, Vol. 58, M. J. Yaffe, ed. Medical Physics Publishing, 212–218.Google Scholar
- [55] . 2019. ACM multimedia BioMedia 2019 grand challenge overview. In Proc. of the 27th ACM Intl. Conf. on Multimedia (MM’19). ACM, 2563–2567.Google Scholar
- [56] . 2018. Comprehensible reasoning and automated reporting of medical examinations based on deep learning analysis. In Proc of the 9th ACM Multimedia Systems Conference (MMSys’18). ACM, 490–493.Google ScholarDigital Library
- [57] . 1997. Long short-term memory. Neural Computation 9, 8 (1997), 1735–1780.Google ScholarDigital Library
- [58] . 1975. STARE database. http://www.ces.clemson.edu/ahoover/stare.Google Scholar
- [59] . 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 (2017).Google Scholar
- [60] . 2017. Densely connected convolutional networks. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’17). 4700–4708.Google Scholar
- [61] . 2019. Multi-attention and incorporating background information model for chest X-ray image report generation. IEEE Access 7 (2019), 154808–154817.Google ScholarCross Ref
- [62] . 2019. Development and validation of a deep learning–based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Network Open 2, 3 (
03 2019), e191095–e191095.Google ScholarCross Ref - [63] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. 2019. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proc. of the AAAI Conf. on Artificial Intelligence, Vol. 33. Association for the Advancement of Artificial Intelligence (AAAI), 590–597.Google Scholar
- [64] . 2018. Quantitative Imaging: The translation from research tool to clinical practice. Radiology 286, 2 (2018), 499.Google ScholarCross Ref
- [65] . 2021. RadGraph: Extracting clinical entities and relations from radiology reports. In 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). https://openreview.net/forum?id=pMWtc5NKd7V.Google Scholar
- [66] . 2019. Attention is not explanation. In Proc. of the 2019 Conf. of the North American Chapter of the ACL: Human Language Technologies, Volume 1 (Long and Short Papers). ACL.Google Scholar
- [67] . 2019. Show, describe and conclude: On exploiting the structure information of chest X-ray reports. In Proc of the 57th Annual Meeting of the ACL. ACL, 6570–6580.Google Scholar
- [68] . 2018. On the automatic generation of medical imaging reports. In Proc. of the 56th Annual Meeting of the ACL (Volume 1: Long Papers). ACL, 2577–2586.Google Scholar
- [69] . 2019. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv:1901.07042 (2019).Google Scholar
- [70] . 2019. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data 6, 1 (
Dec. 2019), 317.Google ScholarCross Ref - [71] . 1996. Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4 (1996), 237–285.Google ScholarDigital Library
- [72] . 2007. DIARETDB1 diabetic retinopathy database and evaluation protocol. In Medical Image Understanding and Analysis, Vol. 2007. Citeseer, 61.Google Scholar
- [73] . 2006. DIARETDB0: Evaluation database and methodology for diabetic retinopathy algorithms. Machine Vision and Pattern Recognition Research Group 73 (2006), 1–17.Google Scholar
- [74] . 2020. A survey of the recent architectures of deep convolutional neural networks. Artificial Intelligence Review (
April 2020), 1–62.Google Scholar - [75] . 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In
Proc. of Machine Learning Research , Vol. 80. PMLR, 2668–2677.Google Scholar - [76] . 2016. Medical image description using multi-task-loss CNN. In Deep Learning and Data Labeling for Medical Applications. Springer Intl. Publishing, Cham, 121–129.Google Scholar
- [77] . 2017. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR. Google Scholar
- [78] . 2019. Do better imagenet models transfer better? In 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR’19). IEEE Computer Society, Los Alamitos, CA, 2656–2666.Google Scholar
- [79] . 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25. Curran Associates, Inc., 1097–1105.Google ScholarDigital Library
- [80] . 2010. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems 23. Curran Associates, Inc., 1189–1197.Google Scholar
- [81] . 2006. RadLex: A new method for indexing online educational materials. Radiographics: A Review Publication of the Radiological Society of North America, Inc. 26, 6 (2006), 1595.Google ScholarCross Ref
- [82] . 2007. Meteor: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proc of the 2nd Workshop on Statistical Machine Translation (StatMT’07). ACL, 228–231.Google Scholar
- [83] . 2014. Distributed representations of sentences and documents. In Intl. Conf. on Machine Learning. 1188–1196.Google Scholar
- [84] . 2015. Challenges in clinical natural language processing for automated disorder normalization. Journal of Biomedical Informatics 57 (2015), 28–37.Google ScholarDigital Library
- [85] . 2018. Hybrid retrieval-generation reinforced agent for medical image report generation. In Proc. of the 32nd Intl. Conf. on Neural Information Processing Systems (NIPS’18). Curran Associates Inc., Red Hook, NY, 1537–1547.Google Scholar
- [86] . 2019. Knowledge-driven encode, retrieve, paraphrase for medical image report generation. In Proc. of the AAAI Conf. on Artificial Intelligence, Vol. 33. 6666–6673.Google Scholar
- [87] . 2019. Label generation system based on generative adversarial network for medical image. In Proc. of the 2nd Intl. Conf. on Artificial Intelligence and Pattern Recognition (AIPR’19). ACM, 78–82.Google Scholar
- [88] . 2015. A hierarchical neural autoencoder for paragraphs and documents. In Proc of the 53rd Annual Meeting of the ACL and the 7th Intl. Joint Conf. on Natural Language Processing (Volume 1: Long Papers). ACL, 1106–1115.Google Scholar
- [89] . 2019. Vispi: Automatic visual perception and interpretation of chest X-rays. arXiv:1906.05190 (2019).Google Scholar
- [90] . 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. ACL, 74–81.Google Scholar
- [91] . 2018. The mythos of model interpretability. Communications of the ACM 61, 10 (2018), 36–43.Google ScholarDigital Library
- [92] . 2019. Clinically accurate chest X-ray report generation. In Machine Learning for Healthcare Conference (Proc of Machine Learning Research), Vol. 106. PMLR, 249–269.Google Scholar
- [93] . 2020. Generate structured radiology report from CT images using image annotation techniques: Preliminary results with liver CT. Journal of Digital Imaging 33, 2 (
April 2020), 375–390.Google ScholarCross Ref - [94] . 2018. A pathology image diagnosis network with visual interpretability and structured diagnostic report. In Neural Information Processing. Springer Intl. Publishing, Cham, 282–293.Google Scholar
- [95] . 2019. CORAL8: Concurrent object regression for area localization in medical image panels. In Medical Image Computing and Computer Assisted Intervention (MICCAI’19). Springer Intl. Publishing, Cham, 432–441.Google Scholar
- [96] . 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 4984–4997. Google ScholarCross Ref
- [97] . 2020. Deep learning in generating radiology reports: A survey. Artificial Intelligence in Medicine 106 (2020), 101878.Google ScholarCross Ref
- [98] . 2016. A cross-modality neural network transform for semi-automatic medical image annotation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI’16). Springer Intl. Publishing, Cham, 300–307.Google Scholar
- [99] . 2012. Inbreast: Toward a full-field digital mammographic database. Academic Radiology 19, 2 (2012), 236–248.Google ScholarCross Ref
- [100] . 2013. The NLM medical text indexer system for indexing biomedical literature. In CEUR Workshop Proceedings, Vol. 1094.Google Scholar
- [101] . 2020. A survey of the usages of deep learning for natural language processing. IEEE Transactions on Neural Networks and Learning Systems 32, 2 (2020), 604–624. https://europepmc.org/article/med/32324570.Google Scholar
- [102] . 2002. Bleu: A method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the ACL. ACL, 311–318.Google Scholar
- [103] . 2013. On the difficulty of training recurrent neural networks. In Proc. of the 30th Intl. Conf. on Intl. Conf. on Machine Learning - Volume 28 (ICML’13). JMLR.org, III–1310–III–1318.Google Scholar
- [104] . 2019. A survey on biomedical image captioning. In Proc. of the 2nd Workshop on Shortcomings in Vision and Language. ACL, 26–36.Google Scholar
- [105] . 2018. Radiology objects in COntext (ROCO): A multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis. Springer Intl. Publishing, Cham, 180–189.Google Scholar
- [106] . 2018. Negbio: A high-performance tool for negation and uncertainty detection in radiology reports. AMIA Summits on Translational Science Proceedings 2018 (2018), 188.Google Scholar
- [107] . 2021. Clinically correct report generation from chest X-rays using templates. In Machine Learning in Medical Imaging, , , , , and (Eds.). Springer International Publishing, Cham, 654–663. Google ScholarCross Ref
- [108] . 2020. Inspecting state of the art performance and NLP metrics in image-based medical report generation. arXiv preprint arXiv:2011.09257 (2020).
In LXAI at NeurIPS 2020. Google Scholar - [109] . 2019. Transfusion: Understanding transfer learning for medical imaging. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 3347–3357.Google Scholar
- [110] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, Matthew P. Lungren, and Andrew Y. Ng. 2017. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv:1711.05225 (2017).Google Scholar
- [111] . 2018. A structured review of the validity of BLEU. Computational Linguistics 44, 3 (2018), 393–401. Google ScholarCross Ref
- [112] . 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28. Curran Associates, Inc., 91–99.Google Scholar
- [113] . 2020. On the interpretability of artificial intelligence in radiology: Challenges and opportunities. Radiology: Artificial Intelligence 2, 3 (2020), e190043.Google ScholarCross Ref
- [114] . 2016. “Why should I trust you?”: Explaining the predictions of any classifier. In Proc. of the 22nd ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (KDD’16). ACM, 1135–1144.Google Scholar
- [115] . 1963. Medical subject headings. Bulletin of the Medical Library Association 51, 1 (1963), 114–116.Google Scholar
- [116] . 2015. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI’15). Springer Intl. Publishing, Cham, 234–241.Google Scholar
- [117] . 2019. Developing diagnostic radiology training in low resource countries. Current Radiology Reports 7, 9 (2019), 27.Google ScholarCross Ref
- [118] . 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proc. of the IEEE Intl. Conf. on Computer Vision (ICCV’17). 618–626.Google Scholar
- [119] . 2017. Deep EHR: A survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE Journal of Biomedical and Health Informatics 22, 5 (2017), 1589–1604.Google ScholarCross Ref
- [120] . 2016. Learning to read chest X-rays: Recurrent neural cascade model for automated image annotation. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’16). 2497–2506.Google Scholar
- [121] . 2017. Learning important features through propagating activation differences. In Proc. of the 34th Intl. Conf. on Machine Learning - Volume 70 (ICML’17). JMLR.org, 3145–3153.Google Scholar
- [122] . 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014).Google Scholar
- [123] . 2019. From chest X-rays to radiology reports: A multimodal machine learning approach. In 2019 Digital Image Computing: Techniques and Applications (DICTA’19). IEEE, 1–8.Google Scholar
- [124] . 2017. Smoothgrad: Removing noise by adding noise. arXiv:1706.03825 (2017).Google Scholar
- [125] . 2016. Quickumls: A fast, unsupervised approach for medical concept extraction. In MedIR Workshop, Sigir. 1–4.Google Scholar
- [126] . 2019. Justifying diagnosis decisions by deep neural networks. Journal of Biomedical Informatics 96 (2019), 103248.Google ScholarDigital Library
- [127] . 2015. Striving for simplicity: The all convolutional net. In ICLR (Workshop Track).Google Scholar
- [128] . 2019. Study on medical image report generation based on improved encoding-decoding method. In Intelligent Computing Theories and Application. Springer Intl. Publishing, Cham, 686–696.Google Scholar
- [129] . 2015. Going deeper with convolutions. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’15). 1–9.Google Scholar
- [130] . 2016. Rethinking the inception architecture for computer vision. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’16). 2818–2826.Google Scholar
- [131] . 2018. A diagnostic report generator from CT volumes on liver tumor with semi-supervised attention mechanism. In Medical Image Computing and Computer Assisted Intervention (MICCAI’18). Springer Intl. Publishing, Cham, 702–710.Google Scholar
- [132] . 2019. Towards automatic diagnosis from multi-modal medical data. In Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support. Springer Intl. Publishing, Cham, 67–74.Google Scholar
- [133] . 2019. A survey on explainable artificial intelligence (XAI): Towards medical XAI. arXiv:1907.07374 (2019).Google Scholar
- [134] . 2019. What clinicians want: Contextualizing explainable machine learning for clinical end use. In Proc. of the 4th Machine Learning for Healthcare Conference (Proc of Machine Learning Research), Vol. 106. PMLR, 359–380.Google Scholar
- [135] . 2019. Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again (1st ed.). Basic Books, Inc.Google Scholar
- [136] . 2019. Machine learning based common radiologist-level pneumonia detection on chest X-rays. In 2019 13th Intl. Conf. on Signal Processing and Communication Systems (ICSPCS’19). IEEE, 1–7.Google Scholar
- [137] . 2021. Underreporting of errors in NLG output, and what to do about it. In Proc. of the 14th International Conference on Natural Language Generation. Association for Computational Linguistics, 140–153. https://aclanthology.org/2021.inlg-1.14.Google Scholar
- [138] . 2020. Gradations of error severity in automatic image descriptions. In Proc. of the 13th International Conference on Natural Language Generation. Association for Computational Linguistics, 398–411. https://aclanthology.org/2020.inlg-1.45.Google Scholar
- [139] . 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30. Curran Associates, Inc., 5998–6008.Google Scholar
- [140] . 2015. Cider: Consensus-based image description evaluation. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’15). 4566–4575.Google Scholar
- [141] . 2015. Show and tell: A neural image caption generator. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’15). 3156–3164.Google Scholar
- [142] . 2016. Discrimination of breast cancer with microcalcifications on mammography by deep learning. Scientific Reports (Nature Publisher Group) 6 (2016), 27327.Google ScholarCross Ref
- [143] . 2017. ChestX-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In The IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’17). 3462–3471.Google Scholar
- [144] . 2018. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’18). 9049–9058.Google Scholar
- [145] . 2019. A computational framework towards medical image explanation. In Artificial Intelligence in Medicine: Knowledge Representation and Transparent and Explainable Systems. Springer Intl. Publishing, Cham, 120–131.Google Scholar
- [146] . 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1, 2 (1989), 270–280.Google ScholarDigital Library
- [147] . 2018. Adaptive generation of structured medical report using NER regarding deep learning. In 2018 Conf. on Technologies and Applications of Artificial Intelligence (TAAI’18). 10–13.Google Scholar
- [148] . 2017. Generative caption for diabetic retinopathy images. In 2017 Intl. Conf. on Security, Pattern Analysis, and Cybernetics (SPAC’17). 515–519.Google Scholar
- [149] . 2020. A survey on domain knowledge powered deep learning for medical image analysis. arXiv:2004.12150 (2020).Google Scholar
- [150] . 2019. Attention-based abnormal-aware fusion network for radiology report generation. In Database Systems for Advanced Applications. Springer Intl. Publishing, Cham, 448–452.Google Scholar
- [151] . 2019. Reinforced transformer for medical image captioning. In Machine Learning in Medical Imaging. Springer Intl. Publishing, Cham, 673–680.Google Scholar
- [152] . 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proc. of the 32nd Intl. Conf. on Intl. Conf. on Machine Learning - Volume 37 (ICML’15). JMLR.org, 2048–2057.Google Scholar
- [153] . 2019. Improved disease classification in chest X-rays with transferred features from report generation. In Information Processing in Medical Imaging. Springer Intl. Publishing, Cham, 125–138.Google Scholar
- [154] . 2018. Multimodal recurrent model with attention for automated radiology report generation. In Intl. Conf. on Medical Image Computing and Computer-Assisted Intervention. Springer, 457–466.Google Scholar
- [155] . 2019. Automatic generation of medical imaging diagnostic report with hierarchical recurrent neural network. In 2019 IEEE Intl. Conf. on Data Mining (ICDM’19). 728–737.Google Scholar
- [156] . 2019. Automatic radiology report generation based on multi-view image fusion and medical concept enrichment. In Medical Image Computing and Computer Assisted Intervention (MICCAI’19). Springer Intl. Publishing, Cham, 721–729.Google Scholar
- [157] . 2020. Deep learning for ultrasound image caption generation based on object detection. Neurocomputing 392 (2020), 132–141.Google ScholarCross Ref
- [158] . 2018. Understanding and generating ultrasound image description. Journal of Computer Science and Technology 33, 5 (2018), 1086–1100.Google ScholarCross Ref
- [159] . 2017. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proc. of the IEEE Intl. Conf. on Computer Vision (ICCV’17). 5907–5915.Google Scholar
- [160] . 2015. Character-level convolutional networks for text classification. In Proc. of the 28th Intl. Conf. on Neural Information Processing Systems - Volume 1 (NIPS’15). MIT Press, Cambridge, MA, 649–657.Google Scholar
- [161] . 2020. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 5108–5120. Google ScholarCross Ref
- [162] . 2020. When radiology report generation meets knowledge graph. Proceedings of the AAAI Conference on Artificial Intelligence 34, 7 (
April 2020), 12910–12917. Google ScholarCross Ref - [163] . 2017. Mdnet: A semantically and visually interpretable medical image diagnosis network. In Proc of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’17). 3549–3557.Google Scholar
- [164] . 2017. Adversarially regularized autoencoders. arXiv:cs.LG/1706.04223.Google Scholar
- [165] . 2016. Learning deep features for discriminative localization. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’16). 2921–2929.Google Scholar
- [166] . 2017. Learning spatial regularization with image-level supervisions for multi-label image classification. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’17). 2027–2036.Google Scholar
- [167] . 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proc. of the IEEE Intl. Conf. on Computer Vision. 2223–2232.Google Scholar
Index Terms
- A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images
Recommendations
Automatic Caption Generation for Medical Images
SCA '18: Proceedings of the 3rd International Conference on Smart City ApplicationsWith the increasing availability of medical images coming from different modalities (X-Ray, CT, PET, MRI, ultrasound, etc.), and the huge advances in the development of incredibly fast, accurate and enhanced computing power with the current graphics ...
Automatic Generation of Medical Report with Knowledge Graph
ICCPR '21: Proceedings of the 2021 10th International Conference on Computing and Pattern RecognitionAs an important part of medical diagnosis, medical images are widely used in the diagnosis and treatment of diseases. Radiologists need to write reports for a large number of medical images every day, which usually occupies most of the radiologists’ ...
Managing Medical Images and Clinical Information: InCor's Experience
Patients usually get medical assistance in several clinics and hospitals during their lifetime, archiving vital information in a dispersed way. Clearly, a proper patient care should take into account that information in order to check for ...
Comments