Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics

Automatic pancreas segmentation in radiology images, e.g., computed tomography (CT), and magnetic resonance imaging (MRI), is frequently required by computer-aided screening, diagnosis, and quantitative assessment. Yet, pancreas is a challenging abdominal organ to segment due to the high inter-patient anatomical variability in both shape and volume metrics. Recently, convolutional neural networks (CNN) have demonstrated promising performance on accurate segmentation of pancreas. However, the CNN-based method often suffers from segmentation discontinuity for reasons such as noisy image quality and blurry pancreatic boundary. In this chapter, we first discuss the CNN configurations and training objectives that lead to the state-of-the-art performance on pancreas segmentation. We then present a recurrent neural network (RNN) to address the problem of segmentation spatial inconsistency across adjacent image slices. The RNN takes outputs of the CNN and refines the segmentation by improving the shape smoothness.

Inflammatory myopathy (IM) is a kind of heterogeneous disease that relates to disorders of muscle functionalities. The identification of IM subtypes is critical to guide effective patient treatment since each subtype requires distinct therapy. Image analysis of hematoxylin and eosin (H&E)-stained whole-slide specimens of muscle biopsies are considered as a gold standard for effective IM diagnosis. Accurate segmentation of perimysium plays an important role in early diagnosis of many muscle inflammation diseases. However, it remains as a challenging task due to the complex appearance of the perimysium morphology and its ambiguity to the background area. The muscle perimysium also exhibits strong structure spanned in the entire tissue, which makes it difficult for current local patch-based methods to capture this long-range context information. In this book chapter, we propose a novel spatial clockwork recurrent neural network (spatial CW-RNN) to address those issues. Besides perimysium segmentation, we also introduce a fully automatic whole-slide image analysis framework for IM subtype classification using deep convolutional neural networks (DCNNs).

Deep neural networks have been widely adopted for automatic organ segmentation from abdominal CT scans. However, the segmentation accuracy of small organs (e.g., pancreas) or neoplasms (e.g., pancreatic cyst) is sometimes below satisfaction, arguably because deep networks are easily disrupted by the complex and variable background regions which occupy a large fraction of the input volume. In this chapter, we propose two coarse-to-fine mechanisms which use prediction from the first (coarse) stage to shrink the input region for the second (fine) stage. More specifically, the two stages in the first method are trained individually in a step-wise manner, so that the entire input region and the region cropped according to the bounding box are treated separately. While the second method inserts a saliency transformation module between the two stages so that the segmentation probability map from the previous iteration can be repeatedly converted as spatial weights to the current iteration. In training, it allows joint optimization over the deep networks. In testing, it propagates multi-stage visual information throughout iterations to improve segmentation accuracy. Experiments are performed on several CT datasets, including NIH pancreas, JHMI multi-organ, and JHMI pancreatic cyst dataset. Our proposed approach gives strong results in terms of DSC.

Although deep neural networks have been a dominant method for many 2D vision tasks, it is still challenging to apply them to 3D tasks, such as medical image segmentation, due to the limited amount of annotated 3D data and limited computational resources. In this chapter, by rethinking the strategy to apply 3D Convolutional Neural Networks to segment medical images, we propose a novel 3D-based coarse-to-fine framework to efficiently tackle these challenges. The proposed 3D-based framework outperforms their 2D counterparts by a large margin since it can leverage the rich spatial information along all three axes. We further analyze the threat of adversarial attacks on the proposed framework and show how to defend against the attack. We conduct experiments on three datasets, the NIH pancreas dataset, the JHMI pancreas dataset and the JHMI pathological cyst dataset, where the first two and the last one contain healthy and pathological pancreases, respectively, and achieve the current state of the art in terms of Dice-Sørensen Coefficient (DSC) on all of them. Especially, on the NIH pancreas dataset, we outperform the previous best by an average of over \(2\%\), and the worst case is improved by \(7\%\) to reach almost \(70\%\), which indicates the reliability of our framework in clinical applications.

Deep convolutional networks (ConvNets) have achieved the state-of-the-art performance and become the de facto standard for solving a wide variety of medical image analysis tasks. However, the learned models tend to present degraded performance when being applied to a new target domain, which is different from the source domain where the model is trained on. This chapter presents unsupervised domain adaptation methods using adversarial learning, to generalize the ConvNets for medical image segmentation tasks. Specifically, we present solutions from two different perspectives, i.e., feature-level adaptation and pixel-level adaptation. The first is to utilize feature alignment in latent space, and has been applied to cross-modality (MRI/CT) cardiac image segmentation. The second is to use image-to-image transformation in appearance space, and has been applied to cross-cohort X-ray images for lung segmentation. Experimental results have validated the effectiveness of these unsupervised domain adaptation methods with promising performance on the challenging task.

Glaucoma is a chronic eye disease that leads to irreversible vision loss. In this chapter, we introduce two state-of-the-art glaucoma detection methods based on deep learning technique. The first is the multi-label segmentation network, named M-Net, which solves the optic disc and optic cup segmentation jointly. M-Net contains a multi-scale U-shape convolutional network with the side-output layer to learn discriminative representations and produces segmentation probability map. Then the vertical cup to disc ratio (CDR) is calculated based on segmented optic disc and cup to assess the glaucoma risk. The second network is the disc-aware ensemble network, named DENet, which integrates the deep hierarchical context of the global fundus image and the local optic disc region. Four deep streams on different levels and modules are, respectively, considered as global image stream, segmentation-guided network, local disc region stream, and disc polar transformation stream. The DENet produces the glaucoma detection result from the image directly without segmentation. Finally, we compare two deep learning methods with other related methods on several glaucoma detection datasets.

Accurate identification and localization of abnormalities from radiology images play an integral part in clinical diagnosis and treatment planning. Building a highly accurate prediction model for these tasks usually requires a large number of images manually annotated with labels and finding sites of abnormalities. In reality, however, such annotated data are expensive to acquire, especially the ones with location annotations. We need methods that can work well with only a small amount of location annotations. To address this challenge, we present a unified approach that simultaneously performs disease identification and localization through the same underlying model for all images. We demonstrate that our approach can effectively leverage both class information as well as limited location annotation, and significantly outperforms the comparative reference baseline in both classification and localization tasks.

We present a detection model that is capable of accelerating the inference time of lesion detection from breast dynamically contrast-enhanced magnetic resonance images (DCE-MRI) at state-of-the-art accuracy. In contrast to previous methods based on computationally expensive exhaustive search strategies, our method reduces the inference time with a search approach that gradually focuses on lesions by progressively transforming a bounding volume until the lesion is detected. Such detection model is trained with reinforcement learning and is modeled by a deep Q-network (DQN) that iteratively outputs the next transformation to the current bounding volume. We evaluate our proposed approach in a breast MRI data set containing the T1-weighted and the first DCE-MRI subtraction volume from 117 patients and a total of 142 lesions. Results show that our proposed reinforcement learning based detection model reaches a true positive rate (TPR) of 0.8 at around three false positive detections and a speedup of at least 1.78 times compared to baselines methods.

Efficient and accurate vertebra labeling in medical images is important for longitudinal assessment, pathological diagnosis, and clinical treatment of the spinal diseases. In practice, the abnormal conditions in the images increase the difficulties to accurately identify the vertebrae locations. Such conditions include uncommon spinal curvature, bright imaging artifacts caused by metal implants, and limited field of the imaging view, etc. In this chapter, we propose an automatic vertebra localization and labeling method with high accuracy and efficiency for medical images. First, we introduce a deep image-to-image network (DI2IN) which generates the probability maps for vertebral centroids. The DI2IN adopts multiple prevailing techniques, including feature concatenation and deep supervision, to boost its performance. Second, a message-passing scheme is used to evolve the probability maps from DI2IN within multiple iterations, according to the spatial relationship of vertebrae. Finally, the locations of vertebra are refined and constrained with a learned sparse representation. We evaluate the proposed method on two categories of public databases, 3D CT volumes, and 2D X-ray scans, under various pathologies. The experimental results show that our method outperforms other state-of-the-art methods in terms of localization accuracy. In order to further boost the performance, we add 1000 extra 3D CT volumes with expert annotation when training the DI2IN for CT images. The results justify that large databases can improve the generalization capability and the performance of the deep neural networks. To the best of our knowledge, it is the first time that more than 1000 3D CT volumes are utilized for the anatomical landmark detection and the overall identification rate reaches 90% in spine labeling.

While deep convolutional neural networks (CNN) have been successfully applied for 2D image analysis, it is still challenging to apply them to 3D anisotropic volumes, especially when the within-slice resolution is much higher than the between-slice resolution and when the amount of 3D volumes is relatively small. On one hand, direct learning of CNN with 3D convolution kernels suffers from the lack of data and likely ends up with poor generalization; insufficient GPU memory limits the model size or representational power. On the other hand, applying 2D CNN with generalizable features to 2D slices ignores between-slice information. Coupling 2D network with LSTM to further handle the between-slice information is not optimal due to the difficulty in LSTM learning. To overcome the above challenges, 3D anisotropic hybrid network (AH-Net) transfers convolutional features learned from 2D images to 3D anisotropic volumes. Such a transfer inherits the desired strong generalization capability for within-slice information while naturally exploiting between-slice information for more effective modeling. We show the effectiveness of the 3D AH-Net on two example medical image analysis applications, namely, lesion detection from a digital breast tomosynthesis volume, and liver, and liver tumor segmentation from a computed tomography volume.

Content-based image retrieval (CBIR) has attracted considerable attention for histopathology image analysis because it can provide more clinical evidence to support the diagnosis. Hashing is an important tool in CBIR due to the significant gain in both computation and storage. Because of the tremendous success of deep learning, deep hashing simultaneously learning powerful feature representations and binary codes has achieved promising performance on microscopic images. This chapter presents several popular deep hashing techniques and their applications on histopathology images. It starts introducing the automated histopathology image analysis and explaining the reasons why deep hashing is a significant and urgent need for data analysis in histopathology images. Then, it specifically discusses three popular deep hashing techniques and mainly introduces pairwise-based deep hashing. Finally, it presents their applications on histopathology image analysis.

Prognostic tumor growth modeling via volumetric medical imaging observations is a challenging yet important problem in precision and predictive medicine. It can potentially imply and lead to better outcomes of tumor treatment management and surgical planning. Traditionally, this problem is tackled through mathematical modeling. Recent advances of convolutional neural networks (ConvNets) have demonstrated higher accuracy and efficiency than conventional mathematical models can be achieved in predicting tumor growth. This indicates that deep learning based data-driven techniques may have great potentials on addressing such problem. In this chapter, we first introduce a statistical group learning approach to predict the pattern of tumor growth that incorporates both the population trend and personalized data, where deep ConvNet is used to model the voxel-wise spatiotemporal tumor progression. We then present a two-stream ConvNets which directly model and learn the two fundamental processes of tumor growth, i.e., cell invasion and mass effect, and predict the subsequent involvement regions of a tumor. Experiments on a longitudinal pancreatic tumor data set show that both approaches substantially outperform a state-of-the-art mathematical model-based approach in both accuracy and efficiency.

Computed tomography perfusion (CTP) facilitates low-cost diagnosis and treatment of acute stroke. Cine scanning allows users to visualize brain anatomy and blood flow in virtually live time. However, effective visualization exposes patients to radiocontrast pharmaceuticals and extended scan times. Higher radiation dosage exposes patients to potential risks including hair loss, cataract formation, and cancer. To alleviate these risks, radiation dosage can be reduced along with tube current and/or X-ray radiation exposure time. However, resulting images may lack sufficient information or be affected by noise and/or artifacts. In this chapter, we propose a deep spatial-temporal convolutional neural network to preserve CTP image quality at reduced tube current, low spatial resolution, and shorter exposure time. This network structure extracts multi-directional features from low-dose and low-resolution patches at different cross sections of the spatial-temporal data and reconstructs high-quality CT volumes. We assess the performance of the network concerning image restoration at different tube currents and multiple resolution scales. The results indicate the ability of our network in restoring high-quality scans from data captured at as low as 21% of the standard radiation dose. The proposed network achieves an average improvement of 7% in perfusion maps compared to the state-of-the-art method.

The continuous development and extensive use of CT in medical practice have raised a public concern over the associated radiation dose to patients. Reducing the radiation dose may lead to increased noise and artifacts, which can adversely affect radiologists’ judgment and confidence. Hence, advanced image reconstruction from low-dose CT data is needed to improve the diagnostic performance, which is a challenging problem due to its ill-posed nature. Over the past years, various low-dose CT methods have produced impressive results. However, most of the algorithms developed for this application, including the recently popularized deep learning techniques, aim for minimizing the mean squared error (MSE) between a denoised CT image and the ground truth under generic penalties. Although the peak signal-to-noise ratio (PSNR) is improved, MSE- or weighted-MSE-based methods can compromise the visibility of important structural details after aggressive denoising. This paper introduces a new CT image denoising method based on the generative adversarial network (GAN) with Wasserstein distance and perceptual similarity. The Wasserstein distance is a key concept of the optimal transport theory, and promises to improve the performance of GAN. The perceptual loss suppresses noise by comparing the perceptual features of a denoised output against those of the ground truth in an established feature space, while the GAN focuses more on migrating the data noise distribution from strong to weak statistically. Therefore, our proposed method transfers our knowledge of visual perception to the image denoising task and is capable of not only reducing the image noise level but also trying to keep the critical information at the same time. Promising results have been obtained in our experiments with clinical CT images.

Cardiac magnetic resonance (CMR) images play a growing role in diagnostic imaging of cardiovascular diseases. MRI is arguably the most comprehensive imaging modality for noninvasive and nonionizing imaging of the heart and great vessels and, hence, most suited for population imaging cohorts. Ensuring full coverage of the left ventricle (LV) is a basic criterion of CMR image quality. Complete LV coverage, from base to apex, precedes accurate cardiac volume and functional assessment. Incomplete coverage of the LV is identified through visual inspection, which is time-consuming and usually done retrospectively in large imaging cohorts. In this chapter, we propose a novel automatic method to check the coverage of LV from CMR images by using Fisher discriminative and dataset invariance (FDDI) three-dimensional (3D) convolutional neural networks (CNN) independently of image-acquisition parameters, such as imaging device, magnetic field strength, variations in protocol execution, etc. The proposed model is trained on multiple cohorts of different provenance to learn the appearance and identify missing basal and apical slices. To address this, a two-stage framework is proposed. First, the FDDI 3D CNN extracts high-level features in the common representation from different CMR datasets using adversarial approach; then these image features are used to detect missing basal and apical slices. Compared with the traditional 3D CNN strategy, the proposed FDDI 3D CNN can minimize the within-class scatter and maximize the between-class scatter, which can be adapted to other CMR image data for LV coverage assessment.

Medical imaging registration is a critical step in a wide spectrum of medical applications from diagnosis to therapy and has been an extensively studied research field. Prior to the popularity of deep learning, image registration was commonly performed by optimizing an image matching metric as a cost function in search for the optimal registration. However, the optimization task is known to be challenging due to (1) the non-convex nature of the matching metric over the registration parameter space and (2) the lack of effective approaches for robust optimization. With the latest advance in deep learning and artificial intelligence, the field of medical image registration had a major paradigm shift, whereby learning-based image registration methods are developed to employ deep neural networks to analyze images in order to estimate plausible registrations. Among the latest advances in learning-based registration methods, agent-based methods have been shown to be effective in both 3-D/3-D and 2-D/3-D registrations with significant robustness advantage over conventional optimization-based methods. In this chapter, we give an overview of agent-based methods for medical image registration and its two applications on rigid-body 3-D/3-D and 2-D/3-D registrations.

The detection of behavioral disorders rooted in neurological structure and function is an important research goal for neuroimaging communities. Recently, deep learning has been used successfully in diagnosis and segmentation applications using anatomical magnetic resonance imaging (MRI). One of the reasons for its popularity is that with repeated nonlinear transformations, the algorithm is capable of learning complex patterns in the data. Another advantage is that the feature selection step commonly used with machine learning algorithms in neuroimaging applications is eliminated which could lead to less bias in the result. However, there has been little progress in the application of these black-box approaches to functional MRI (fMRI). In this study, we explore the use of deep learning methods in comparison with conventional machine learning classifiers as well as their ensembles to analyze fMRI scans. We compare the benefits of deep learning against an ensemble of classical machine learning classifiers with a suitable feature selection strategy. Specifically, we focus on a clinically important problem of Attention Deficit Hyperactivity Disorder (ADHD). Functional connectivity information is extracted from fMRI scans of ADHD and control patients (ADHD-200), and analysis is performed by applying a decision fusion of various classifiers—the support vector machine, support vector regression, elastic net, and random forest. We selectively include features by a nonparametric ranking method for feature selection. After initial classification is performed, the decisions are summed in various permutations for an ensemble classifier, and the final results are compared with the deep learning-based results. We achieved a maximum accuracy of 93.93% on the KKI dataset (a subset of the ADHD-200) and also identified significantly different connections in the brain between ADHD and control subjects. In the blind testing with different subsets of the target data (Peking-1), we achieved a maximum accuracy of 72.9%. In contrast, the deep learning-based approaches yielded a maximum accuracy of 70.5% on the Peking-1 dataset and 67.74% on the complete ADHD-200 dataset, significantly inferior to the classifier ensemble approach. With more data being made publicly available, deep learning in fMRI may show a strong potential but as of now deep learning does not provide a magical solution for fMRI-based diagnosis.

The chest X-ray is one of the most commonly accessible radiological examinations for screening and diagnosis of many lung diseases. A tremendous number of X-ray imaging studies accompanied by radiological reports are accumulated and stored in many modern hospitals’ picture archiving and communication systems (PACS). On the other side, it is still an open question how this type of hospital-size knowledge database containing invaluable imaging informatics (i.e., loosely labeled) can be used to facilitate the data-hungry deep learning paradigms in building truly large-scale high-precision computer-aided diagnosis (CAD) systems. In this chapter, we present a chest X-ray database, namely, “ChestX-ray”, which comprises 121,120 frontal-view X-ray images of 30,805 unique patients with the text-mined eight disease image labels (where each image can have multi-labels), from the associated radiological reports using natural language processing. Importantly, we demonstrate that these commonly occurring thoracic diseases can be detected and even spatially located via a unified weakly supervised multi-label image classification and disease localization framework, which is validated using our proposed dataset. Although the initial quantitative results are promising as reported, deep convolutional neural network-based “reading chest X-rays” (i.e., recognizing and locating the common disease patterns trained with only image-level labels) remains a strenuous task for fully automated high-precision CAD systems.

Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reports that accompany X-ray images in hospital picture and archiving communication systems can be utilized as a priori knowledge for tackling these two key problems. We propose a novel text-image embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together, with X-ray images being the only input. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI).

Radiologists in their daily work routinely find and annotate significant abnormalities on a large number of radiology images. Such abnormalities, or lesions, have collected over years and stored in hospitals’ picture archiving and communication systems. However, they are basically unsorted and lack semantic annotations like type and location. In this paper, we aim to organize and explore them by learning a deep feature representation for each lesion. A large-scale and comprehensive dataset, DeepLesion, is introduced for this task. DeepLesion contains bounding boxes and size measurements of over 32K lesions. To model their similarity relationship, we leverage multiple supervision information including types, self-supervised location coordinates, and sizes. They require little manual annotation effort but describe useful attributes of the lesions. Then, a triplet network is utilized to learn lesion embeddings with a sequential sampling strategy to depict their hierarchical similarity structure. Experiments show promising qualitative and quantitative results on lesion retrieval, clustering, and classification. The learned embeddings can be further employed to build a lesion graph for various clinically useful applications. An algorithm for intra-patient lesion matching is proposed and validated with experiments.

Multi-modality magnetic resonance imaging (MRI) has enabled significant progress to both clinical diagnosis and medical research. Applications range from differential diagnosis to novel insights into disease mechanisms and phenotypes. However, there exist many practical scenarios where acquiring high-quality multi-modality MRI is restricted, for instance, owing to limited scanning time. This imposes constraints on multi-modality MRI processing tools, e.g., segmentation and registration. Such limitations are not only recurrent in prospective data acquisition but also when dealing with existing databases with either missing or low-quality imaging data. In this work, we explore the problem of synthesizing high-resolution images corresponding to one MRI modality from a low-resolution image of another MRI modality of the same subject. This is achieved by introducing the cross-modality dictionary learning scheme and a patch-based globally redundant model based on sparse representations. We use high-frequency multi-modality image features to train dictionary pairs, which are robust, compact, and correlated in this multimodal feature space. A feature clustering step is integrated into the reconstruction framework speeding up the search involved in the reconstruction process. Images are partitioned into a set of overlapping patches to maintain the consistency between neighboring pixels and increase speed further. Extensive experimental validations on two multi-modality databases of real brain MR images show that the proposed method outperforms state-of-the-art algorithms in two challenging tasks: image super-resolution and simultaneous SR and cross-modality synthesis. Our method was assessed on both healthy subjects and patients suffering from schizophrenia with excellent results.