Abstract
Modern medical diagnostic techniques facilitate accurate diagnosis and treatment recommendations in healthcare. Such diagnostics procedures are performed daily in large numbers, thus, the clinical interpretation workload of radiologists is very high. Identification of abnormalities is a predominantly manual task that is performed by radiologists before the medical scans are available to the patient’s referring doctor for further recommendations. On the other hand, for a radiologist to delineate the imaging study’s findings/observations as a textual report is also a tedious task. Automated methods for radiographic image examination for identifying abnormalities and generating reliable radiology report are thus a fundamental requirement in clinical workflow management applications. In this work, we present an automated approach for abnormality classification, localization and diagnostic report retrieval for identified abnormalities. We propose MSDNet, an ensemble of Convolutional Neural models for abnormality classification, which combines the features of multiple CNN models to enhance abnormality classification performance. The proposed model also is designed to localize and visualize the detected abnormality on the radiograph image, based on an abnormal region detection algorithm to further optimize the diagnosis quality. Furthermore, the extracted features generated by MSDNet are used to automatically generate the diagnosis text report using an automatic content-based report retrieval algorithm. The upper extremity musculo-skeletal images from the MURA dataset and chest X-ray images from Indiana dataset were used for the experimental evaluation of the proposed approach. The proposed model achieved promising results, with an accuracy of 82.69%, showing its significant impact on alleviating radiologists’ cognitive load, thus improving the overall efficiency.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Change history
14 May 2022
A Correction to this paper has been published: https://doi.org/10.1007/s12652-022-03906-w
Notes
Receiver Operating Characteristic.
References
Aowal MA, Minhaz AT, Ashraf K (2017) Abnormality detection and localization in chest x-rays using deep convolutional neural networks. arXiv preprint arXiv:1705.09850
Baadjou VAE, Roussel NA, Verbunt JAMCF, Smeets RJEM, de Bie RA (2016) Systematic review: risk factors for musculoskeletal disorders in musicians. Occup Med 66(8):614–622
Banga D, Waiganjo P (2019) Abnormality detection in musculoskeletal radiographs with convolutional neural networks (ensembles) and performance optimization. arXiv preprint arXiv:1908.02170
BMUS (2014) United States Bone and Joint Initiative: The Burden of Musculoskeletal Diseases in the United States (BMUS), Fourth Edition. http://www.boneandjointburden.org/2014-report. [Accessed on 1 May 2019]
Chada G (2019) Machine learning models for abnormality detection in musculoskeletal radiographs. Rep Med Cases Images Videos 2(4):26
Cheng C-T, Tsung-Ying H, Tao-Yi L, Chih-Chen C, Ching-Cheng C, I-Fang, Chien-Hung Liao, (2019) Application of a deep learning algorithm for detection and visualization of hip fractures on plain pelvic radiographs. Eur Radiol 29(10):5469–5477
Chung SW, Han SS, Lee JW, Kyung-Soo O, Kim NR, Yoon, et al (2018) Automated detection and classification of the proximal humerus fracture by using deep learning algorithm. Acta Orthop 89:4
Dalia Y, Bharath A, Mayya V, Sowmya Kamath S (2021) Deepoa: Clinical decision support system for early detection and severity grading of knee osteoarthritis. In 2021 5th International Conference on Computer, Communication and Signal Processing (ICCCSP), pages 250–255. IEEE
Demner-Fushman D, Kohli MD, Rosenman MB, Shooshan SE, Rodriguez L, Antani S, Thoma GR, McDonald CJ (2016) Preparing a collection of radiology examinations for distribution and retrieval. J Am Med Inform Assoc 23(2):304–310
Faes L, Wagner SK, Dun Jack F, Liu X et al (2019) Automated deep learning design for medical image classification by health-care professionals with no coding experience: a feasibility study. Lancet Digit Health 1(5):e232–e242
Gale W, Oakden-Rayner L, Carneiro G, Bradley AP, Palmer LJ (2017) Detecting hip fractures with radiologist-level performance using deep neural networks. arXiv preprint arXiv:1711.06504
García-Floriano A, Ferreira-Santiago Á, Camacho-Nieto O, Márquez C (2019) A machine learning approach to medical image classification: detecting age-related macular degeneration in fundus images. Comput Electr Eng 75:218–229
Harzig P, Chen YY, Chen F, Lienhart R (2019) Addressing data bias problems for chest x-ray image report generation. arXiv preprint arXiv:1908.02123,
He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In European conference on computer vision, pages 630–645. Springer
Ioppolo F, Rompe JD, Furia JP, Cacchio A (2014) Clinical application of shock wave therapy (swt) in musculoskeletal disorders. Eur J Phys Rehabil Med 50(2):217–30
Karthik K, Sowmya Kamath S (2021) A deep neural network model for content-based medical image retrieval with multi-view classification. Vis Comput 37(7):1837–1850
Karthik K, Sowmya Kamath S (2021) Automated view orientation classification for x-ray images using deep neural networks. Smart computational intelligence in biomedical and health informatics. CRC Press, Boca Raton, pp 61–72
Katara K, Sowmya K et al (2021) Deep neural models for automated multi-task diagnostic scan management-quality enhancement, view classification and report generation. Biomed Phys Eng Express 8:1
Kim DH, MacKinnon T (2018) Artificial intelligence in fracture detection: transfer learning from deep convolutional neural networks. Clin Radiol 73(5):439–445
Kitamura G, Chung CY, Moore BE (2019) Ankle fracture detection utilizing a convolutional neural network ensemble implemented with a small sample, de novo training, and multiview incorporation. J Digit Imaging 32(4):672–677
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 2:1097–1105
Krogue Justin D, Cheng Kaiyang V, Hwang Kevin M, Paul T et al (2020) Automatic hip fracture identification and functional subclassification with deep learning. Radiol Artif Intell 2:2
Krupinski EA, Berbaum KS, Caldwell RT, Schartz KM, Kim J (2010) Long radiology workdays reduce detection and accommodation accuracy. J Am Coll Radiol 7:9
Kumar A, Kim J, Lyndon D, Fulham M, Feng D (2016) An ensemble of fine-tuned convolutional neural networks for medical image classification. IEEE J Biomed Health Inform 21:1
Mandikal V, Anantharaman A, Suhas BS (2019) An approach for multimodal medical image retrieval using latent dirichlet allocation. In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, pages 44–51
Mayya V, Karthik K, Sowmya KS, Karadka K, Jeganathan J (2021) Coviddx: Ai-based clinical decision support system for learning covid-19 disease representations from multimodal patient data. In HEALTHINF, pages 659–666
McHugh ML (2012) Interrater reliability: the kappa statistic. Biochem Med Biochem Med 22(3):276–282
Mukesh BR, Harish T, Mayya V, Sowmya Kamath S (2021) Deep learning based detection of diabetic retinopathy from inexpensive fundus imaging techniques. In 2021 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT), pages 1–6. IEEE
Nedumkunnel IM, George LE et al (2021) Explainable deep neural models for covid-19 prediction from chest x-rays with region of interest visualization. In 2021 2nd International Conference on Secure Cyber Computing and Communications (ICSCCC), pages 96–101. IEEE
Rajpurkar P, Irvin J, Bagul A, Ding D, Duan T et al (2017) Mura dataset: Towards radiologist-level abnormality detection in musculoskeletal radiographs. arXiv preprint arXiv:1712.06957
Rajpurkar P, Irvin J, Zhu K, Yang B, Mehta H et al (2017) Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225
Saif AFM, Shahnaz C, Zhu WP, Omair AM (2019) Abnormality detection in musculoskeletal radiographs using capsule network. IEEE Access 7:81494–81503
Silvian SP, Maiya A, Resmi AT, Page T (2011) Antecedents of work related musculoskeletal disorders in software professionals. Int J Enterprise Netw Manag 4(3):247–260
Solovyova A, Solovyov I (2020) X-ray bone abnormalities detection using mura dataset. arXiv preprint arXiv:2008.03356
Soundalgekar P, Kulkarni M, Nagaraju D (2018) Medical image retrieval using manifold ranking with relevance feedback. In 2018 IEEE 12th International Conference on Semantic Computing (ICSC). IEEE,
Tataru C, Yi D, Shenoyas A, Ma A (2017) Deep learning for abnormality detection in chest x-ray images
Wærsted M, Hanvold TN, Veiersted KB (2010) Computer work and musculoskeletal disorders of the neck and upper extremity: a systematic review. BMC Musculoskelet Disord 11(1):79
Woolf AD, Pfleger B (2003) Burden of major musculoskeletal conditions. Bull World Health Organ 81:2
Yahalomi F, Chernofsky M, Werman M (2019) Detection of distal radius fractures trained by a small set of x-ray images and faster r-cnn. In Intelligent Computing-Proceedings of the Computing Conference, pages 971–981. Springer,
Ying J, Dutta J, Guo N, Chenhui H, Zhou D, Sitek A, Li Q (2016) Classification of exacerbation frequency in the copdgene cohort using deep learning with deep belief networks. IEEE J Biomed Health Inform 2:2
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest. This article does not contain any studies with human participants or animals performed by any of the authors and confirms that no ethical approval is required.
Data Availability Statement (DAS)
The datasets that were used and analyzed as part of the current study are openly and publicly available as: MURA datasethttps://stanfordmlgroup.github.io/competitions/mura/ and Indiana Datasethttps://www.kaggle.com/raddar/chest-xrays-indiana-university.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The original online version of this article was revised: “The missing images has been processed in table 2 of the original article”.
Rights and permissions
About this article
Cite this article
Karthik, K., Sowmya Kamath, S. MSDNet: a deep neural ensemble model for abnormality detection and classification of plain radiographs. J Ambient Intell Human Comput 14, 16099–16113 (2023). https://doi.org/10.1007/s12652-022-03835-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-022-03835-8