Machine Learning in Medical Imaging

Functional magnetic resonance imaging (fMRI) has been widely applied to analysis and diagnosis of brain diseases, including Alzheimer’s disease (AD) and its prodrome, i.e., mild cognitive impairment (MCI). Traditional methods usually construct connectivity networks (CNs) by simply calculating Pearson correlation coefficients (PCCs) between time series of brain regions, and then extract low-level network measures as features to train the learning model. However, the valuable observation information in network construction (e.g., specific contributions of different time points) and high-level (i.e., high-order) network properties are neglected in these methods. In this paper, we first define a novel weighted correlation kernel (called wc-kernel) to measure the correlation of brain regions, by which weighting factors are determined in a data-driven manner to characterize the contribution of each time point, thus conveying the richer interaction information of brain regions compared with the PCC method. Furthermore, we propose a wc-kernel based convolutional neural network (CNN) (called wck-CNN) framework for extracting the hierarchical (i.e., from low-order to high-order) functional connectivities for disease diagnosis, by using fMRI data. Specifically, we first define a layer to build dynamic CNs (DCNs) using the defined wc-kernels. Then, we define three layers to extract local (region specific), global (network specific) and temporal high-order properties from the constructed low-order functional connectivities as features for classification. Results on 174 subjects (a total of 563 scans) with rs-fMRI data from ADNI suggest that the our method can not only improve the performance compared with state-of-the-art methods, but also provide novel insights into the interaction patterns of brain activities and their changes in diseases.

This paper considers the actor-critic contextual bandit for the mobile health (mHealth) intervention. The state-of-the-art decision-making methods in the mHealth generally assume that the noise in the dynamic system follows the Gaussian distribution. Those methods use the least-square-based algorithm to estimate the expected reward, which is prone to the existence of outliers. To deal with the issue of outliers, we are the first to propose a novel robust actor-critic contextual bandit method for the mHealth intervention. In the critic updating, the capped- $$\ell _{2}$$ ℓ 2 norm is used to measure the approximation error, which prevents outliers from dominating our objective. A set of weights could be achieved from the critic updating. Considering them gives a weighted objective for the actor updating. It provides the ineffective sample in the critic updating with zero weights for the actor updating. As a result, the robustness of both actor-critic updating is enhanced. There is a key parameter in the capped- $$\ell _{2}$$ ℓ 2 norm. We provide a reliable method to properly set it by making use of one of the most fundamental definitions of outliers in statistics. Extensive experiment results demonstrate that our method can achieve almost identical results compared with the state-of-the-art methods on the dataset without outliers and dramatically outperform them on the datasets noised by outliers.

Deep-learning based labeling methods have gained unprecedented popularity in different computer vision and medical image segmentation tasks. However, to the best of our knowledge, these have not been used for cervical tumor segmentation. More importantly, while the majority of innovative deep-learning works using convolutional neural networks (CNNs) focus on developing more sophisticated and robust architectures (e.g., ResNet, U-Net, GANs), there is very limited work on how to aggregate different CNN architectures to improve their relational learning at multiple levels of CNN-to-CNN interactions. To address this gap, we introduce a Dynamic Multi-Scale CNN Forest (CK+1DMF), which aims to address three major issues in medical image labeling and ensemble CNN learning: (1) heterogeneous distribution of MRI training patches, (2) a bi-directional flow of information between two consecutive CNNs as opposed to cascading CNNs—where information passes in a directional way from current to the next CNN in the cascade, and (3) multiscale anatomical variability across patients. To solve the first issue, we group training samples into K clusters, then design a forest with $$ (K + 1) $$ ( K + 1 ) trees: a principal tree of CNNs trained using all data samples and subordinate trees, each trained using a cluster of samples. As for the second and third issues, we design each dynamic multiscale tree (DMT) in the forest such that each node in the tree nests a CNN architecture. Two successive CNN nodes in the tree pass bidirectional contextual maps to progressively improve the learning of their relational non-linear mapping. Besides, as we traverse a path from the root node to a leaf node in the tree, the architecture of each CNN node becomes shallower to take in smaller training patches. Our CK+1DMF significantly (p < 0.05) outperformed several conventional and ensemble CNN architectures, including conventional CNN (improvement by 10.3%) and CNN-based DMT (improvement by 5%).

The quality of fundus images is critical for diabetic retinopathy diagnosis. The evaluation of fundus image quality can be affected by several factors, including image artifact, clarity, and field definition. In this paper, we propose a multi-task deep learning framework for automated assessment of fundus image quality. The network can classify whether an image is gradable, together with interpretable information about quality factors. The proposed method uses images in both rectangular and polar coordinates, and fine-tunes the network from trained model grading of diabetic retinopathy. The detection of optic disk and fovea assists learning the field definition task through coarse-to-fine feature encoding. The experimental results demonstrate that our framework outperform single-task convolutional neural networks and reject ungradable images in automated diabetic retinopathy diagnostic systems.

Computer aided diagnostic (CAD) system is crucial for modern medical imaging. But almost all CAD systems operate on reconstructed images, which were optimized for radiologists. Computer vision can capture features that is subtle to human observers, so it is desirable to design a CAD system operating on the raw data. In this paper, we proposed a deep-neural-network-based detection system for lung nodule detection in computed tomography (CT). A primal-dual-type deep reconstruction network was applied first to convert the raw data to the image space, followed by a 3-dimensional convolutional neural network (3D-CNN) for the nodule detection. For efficient network training, the deep reconstruction network and the CNN detector was trained sequentially first, then followed by one epoch of end-to-end fine tuning. The method was evaluated on the Lung Image Database Consortium image collection (LIDC-IDRI) with simulated forward projections. With 144 multi-slice fanbeam projections, the proposed end-to-end detector could achieve comparable sensitivity with the reference detector, which was trained and applied on the fully-sampled image data. It also demonstrated superior detection performance compared to detectors trained on the reconstructed images. The proposed method is general and could be expanded to most detection tasks in medical imaging.

Automated lesion segmentation from computed tomography (CT) is an important and challenging task in medical image analysis. While many advancements have been made, there is room for continued improvements. One hurdle is that CT images can exhibit high noise and low contrast, particularly in lower dosages. To address this, we focus on a preprocessing method for CT images that uses stacked generative adversarial networks (SGAN) approach. The first GAN reduces the noise in the CT image and the second GAN generates a higher resolution image with enhanced boundaries and high contrast. To make up for the absence of high quality CT images, we detail how to synthesize a large number of low- and high-quality natural images and use transfer learning with progressively larger amounts of CT images. We apply both the classic GrabCut method and the modern holistically nested network (HNN) to lesion segmentation, testing whether SGAN can yield improved lesion segmentation. Experimental results on the DeepLesion dataset demonstrate that the SGAN enhancements alone can push GrabCut performance over HNN trained on original images. We also demonstrate that HNN + SGAN performs best compared against four other enhancement methods, including when using only a single GAN.

Non-rigid inter-modality registration can facilitate accurate information fusion from different modalities, but it is challenging due to the very different image appearances across modalities. In this paper, we propose to train a non-rigid inter-modality image registration network, which can directly predict the transformation field from the input multimodal images, such as CT and MRI. In particular, the training of our inter-modality registration network is supervised by intra-modality similarity metric based on the available paired data, which is derived from a pre-aligned CT and MRI dataset. Specifically, in the training stage, to register the input CT and MR images, their similarity is evaluated on the warped MR image and the MR image that is paired with the input CT. So that, the intra-modality similarity metric can be directly applied to measure whether the input CT and MR images are well registered. Moreover, we use the idea of dual-modality fashion, in which we measure the similarity on both CT modality and MR modality. In this way, the complementary anatomies in both modalities can be jointly considered to more accurately train the inter-modality registration network. In the testing stage, the trained inter-modality registration network can be directly applied to register the new multimodal images without any paired data. Experimental results have shown that, the proposed method can achieve promising accuracy and efficiency for the challenging non-rigid inter-modality registration task and also outperforms the state-of-the-art approaches.

In this paper, we propose a novel method for MRI-based AD/MCI diagnosis that systematically integrates voxel-based, region-based, and patch-based approaches in a unified framework. Specifically, we parcellate a brain into predefined regions by using anatomical knowledge, i.e., template, and find complex nonlinear relations among voxels, whose intensity denotes the volumetric measure in our case, within each region. Unlike the existing methods that mostly use a cubical or rectangular shape, we regard the anatomical shape of regions as atypical forms of patches. Using the complex nonlinear relations among voxels in each region learned by deep neural networks, we extract a regional abnormality representation. We then make a final clinical decision by integrating the regional abnormality representations over a whole brain. It is noteworthy that the regional abnormality representations allow us to interpret and understand the symptomatic observations of a subject with AD or MCI by mapping and visualizing them in a brain space individually. We validated the efficacy of our method in experiments with baseline MRI dataset in the ADNI cohort by achieving promising performances in three binary classification tasks.

Medical image registration and segmentation are complementary functions and combining them can improve each other’s performance. Conventional deep learning (DL) based approaches tackle the two problems separately without leveraging their mutually beneficial information. We propose a DL based approach for joint registration and segmentation (JRS) of chest Xray images. Generative adversarial networks (GANs) are trained to register a floating image to a reference image by combining their segmentation map similarity with conventional feature maps. Intermediate segmentation maps from the GAN’s convolution layers are used in the training stage to generate the final segmentation mask at test time. Experiments on chest Xray images show that JRS gives better registration and segmentation performance than when solving them separately.

White matter hyperintensities (WMH) and atrophy are common findings in neurodegenerative diseases as well as healthy aging. However, it is not clear whether their co-occurrence is due to shared risk factors. Previous work has analyzed univariate associations between individual brain regions but not joint patterns over multiple regions. We propose a new method that jointly analyzes all the regions to discover spatial association patterns between WMH and atrophy. Univariate analyses typically correct for shared risk factors at the level of individual WMH and atrophy variables. Our method incorporates a novel correction strategy at the level of the entire pattern over multiple regions. Furthermore, we enforce sparsity to yield interpretable results. Results in a cohort of 703 participants from the Rhineland Study reveal two consistent spatial association patterns. Correction of individual variables did not yield qualitatively different patterns. Our proposed multi-variate correction strategy yielded different patterns thus, suggesting that it might be more appropriate for multi-variate analysis.

A method is proposed for converting raw ultrasound signals of respiratory organ motion into high frame rate dynamic MRI using a long-term recurrent convolutional neural network. Ultrasound signals were acquired using a single-element transducer, referred to here as ‘organ-configuration motion’ (OCM) sensor, while sagittal MR images were simultaneously acquired. Both streams of data were used for training a cascade of convolutional layers, to extract relevant features from raw ultrasound, followed by a recurrent neural network, to learn its temporal dynamics. The network was trained with MR images on the output, and was employed to predict MR images at a temporal resolution of 100 frames per second, based on ultrasound input alone, without any further MR scanner input. The method was validated on 7 subjects.

Deep neural network architectures have traditionally been designed and explored with human expertise in a long-lasting trial-and-error process. This process requires huge amount of time, expertise, and resources. To address this tedious problem, we propose a novel algorithm to optimally find hyperparameters of a deep network architecture automatically. We specifically focus on designing neural architectures for medical image segmentation task. Our proposed method is based on a policy gradient reinforcement learning for which the reward function is assigned a segmentation evaluation utility (i.e., dice index). We show the efficacy of the proposed method with its low computational cost in comparison with the state-of-the-art medical image segmentation networks. We also present a new architecture design, a densely connected encoder-decoder CNN, as a strong baseline architecture to apply the proposed hyperparameter search algorithm. We apply the proposed algorithm to each layer of the baseline architectures. As an application, we train the proposed system on cine cardiac MR images from Automated Cardiac Diagnosis Challenge (ACDC) MICCAI 2017. Starting from a baseline segmentation architecture, the resulting network architecture obtains the state-of-the-art results in accuracy without performing any trial-and-error based architecture design approaches or close supervision of the hyperparameters changes.

We define and investigate the Local Rotation Invariance (LRI) and Directional Sensitivity (DS) of radiomics features. Most of the classical features cannot combine the two properties, which are antagonist in simple designs. We propose texture operators based on spherical harmonic wavelets (SHW) invariants and show that they are both LRI and DS. An experimental comparison of SHW and popular radiomics operators for classifying 3D textures reveals the importance of combining the two properties for optimal pattern characterization.

Automated analysis of free-hand ultrasound video sweeps is an important topic in diagnostic and interventional imaging, however, it is a notoriously challenging task for detecting the standard planes, due to the low-quality data, variability in contrast, appearance and placement of the structures. Conventionally, sequential data is usually modelled with heavy Recurrent Neural Networks (RNNs). In this paper, we propose to apply a convolutional architecture (CNNs) for the standard plane detection in free-hand ultrasound videos. Our contributions are twofolds, firstly, we show a simple convolutional architecture can be applied to characterize the long range dependencies in the challenging ultrasound video sequences, and outperform the canonical LSTMs and the recently proposed two-stream spatial ConvNet by a large margin (89% versus 83% and 84% respectively). Secondly, to get an understanding of what evidences have been used by the model for decision making, we experimented with the soft-attention layers for feature pooling, and trained the entire model end-to-end with only standard classification losses. As a result, we find the input-dependent attention maps can not only boost the network’s performance, but also indicate useful patterns of the data that are deemed important for certain structure, therefore provide interpretation while deploying the models.

Reconstruction of accurate cortical surfaces with minimal topological errors (i.e., handles and holes) from infant brain MR images is important in early brain development studies. However, infant brain MR images usually exhibit extremely low tissue contrast (especially from 3 to 9 months of age) and dynamic imaging appearance patterns. Thus, it is inevitable to have large amounts of topological errors in the infant brain tissue segmentation results, thus leading to inaccurate surface reconstruction. To address these issues, inspired by recent advances in deep learning methods, we propose an anatomically constrained U-Net method for topological correction of infant cortical surfaces. Specifically, in our method, we first extract candidate voxels with potential topological errors, by leveraging a topology-preserving level set method. Then, we propose a U-Net with anatomical constraints to correct those located candidate voxels. Due to the fact that infant cortical surfaces often contain large handles or holes, it is difficult to completely correct all errors using one-shot correction. Therefore, we further gather these two steps into an iterative framework to correct large topological errors gradually. To our knowledge, this is the first work introducing deep learning for infant cortical topological correction. We compare our method with the state-of-the-art method on infant cortical topology and show the superior performance of our method.

Self-taught learning aims at obtaining compact and latent representations from data them-selves without previously manual labeling, which would be time-consuming and laborious. This study proposes a novel self-taught learning for more accurately reconstructing the raw data based on the sparse autoencoder. It is well known that autoencoder is able to learn latent features via setting the target values to be equal to the input data, and can be stacked for pursuing high-level feature learning. Motivated by the natural sparsity of data representation, sparsity has been imposed on the hidden layer responses of autoencoder for more effective feature learning. Although the conventional autoencoder-based feature learning aims at obtaining the latent representation via minimizing the approximation error of the input data, it is unavoidable to produce reconstruction residual error of the input data and thus some tiny structures are unable to be represented, which may be essential information for fine-grained image task such as medical image analysis. Even with the multiple-layer stacking for high-level feature pursuing in autoencoder-based learning strategy, the lost tiny structure in the former layers can not be recovered evermore. Therefore, this study proposes a residual sparse autoencoder for learning the latent feature representation of more tiny structures in the raw input data. With the unavoidably generated reconstruction residual error, we exploit another sparse autoencoder to pursuing the latent feature of the residual tiny structures and this self-taught learning process can continue until the representation residual error is enough small. We evaluate the proposed residual sparse autoencoding for self-taught learning the latent representations of HEp-2 cell image, and prove that promising performance for staining pattern recognition can be achieved compared with the conventional sparse autoencoder and the-state-of-the-art methods.

In spite of the compelling achievements that deep neural networks (DNNs) have made in medical image computing, these deep models often suffer from degraded performance when being applied to new test datasets with domain shift. In this paper, we present a novel unsupervised domain adaptation approach for segmentation tasks by designing semantic-aware generative adversarial networks (GANs). Specifically, we transform the test image into the appearance of source domain, with the semantic structural information being well preserved, which is achieved by imposing a nested adversarial learning in semantic label space. In this way, the segmentation DNN learned from the source domain is able to be directly generalized to the transformed test image, eliminating the need of training a new model for every new target dataset. Our domain adaptation procedure is unsupervised, without using any target domain labels. The adversarial learning of our network is guided by a GAN loss for mapping data distributions, a cycle-consistency loss for retaining pixel-level content, and a semantic-aware loss for enhancing structural information. We validated our method on two different chest X-ray public datasets for left/right lung segmentation. Experimental results show that the segmentation performance of our unsupervised approach is highly competitive with the upper bound of supervised transfer learning.

Study on brain status prediction has recently received increasing attention from the research community. In this paper, we propose to tackle brain status prediction by learning a discriminative representation of the data with a novel non-negative projective dictionary learning (NPDL) approach. The proposed approach performs class-wise projective dictionary learning, which uses an analysis dictionary to generate non-negative coding vectors from the data, and a synthesis dictionary to reconstruct the data. We formulate the learning problem as a constrained non-convex optimization problem and solve it via an alternating direction method of multipliers (ADMM). To investigate the effectiveness of the proposed approach on brain status prediction, we conduct experiments on two datasets, ADNI and NIH Study of Normal Brain Development repository, and report superior results over comparison methods.

Classification of pancreatic cystic neoplasms (PCN) into subclasses is crucial since their treatments are different. However, accurate classification is very difficult even for radiologists, due to similar appearance and shape. We propose a network called PCN-Net which makes use of T1/T2 MRI of abdomen by its three stages design. The first and second stages are trained on T1 and T2 separately for detection and inter-modality registration. After a Z-Continuity Filter and modalities fusion, the third stage predict the results with registered image pairs. On a database of 48 patients, our method can predict with slice level accuracy of $$80.0\%$$ 80.0 % and patient level accuracy of $$92.3\%$$ 92.3 % , which are much better than other baseline methods.

Fully convolutional networks (FCNs) are well known to provide state-of-the-art results in various medical image segmentation tasks. However, these models usually need a tremendous number of training samples to achieve good performances. Unfortunately, this requirement is often difficult to satisfy in the medical imaging field, due to the scarcity of labeled images. As a consequence, the common tricks for FCNs’ training go from data augmentation and transfer learning to patch-based segmentation. In the latter, the segmentation of an image involves patch extraction, patch segmentation, then patch aggregation. This paper presents a framework that takes advantage of all these tricks by starting with a patch-level segmentation which is then extended to the image level by transfer learning. The proposed framework follows two main steps. Given a image database $$\mathcal {D}$$ D , a first network $$\mathcal {N}_P$$ N P is designed and trained using patches extracted from $$\mathcal {D}$$ D . Then, $$\mathcal {N}_P$$ N P is used to pre-train a FCN $$\mathcal {N}_I$$ N I to be trained on the full sized images of $$\mathcal {D}$$ D . Experimental results are presented on the task of retinal blood vessel segmentation using the well known publicly available DRIVE database.

Deep learning has thoroughly changed the field of image analysis yielding impressive results whenever enough annotated data can be gathered. While partial annotation can be very fast, manual segmentation of 3D biological structures is tedious and error-prone. Additionally, high-level shape concepts such as topology or boundary smoothness are hard if not impossible to encode in Feedforward Neural Networks. Here we present a modular strategy for the accurate segmentation of neural cell bodies from light-sheet microscopy combining mixed-scale convolutional neural networks and topology-preserving geometric deformable models. We show that the network can be trained efficiently from simple cell centroid annotations, and that the final segmentation provides accurate cell detection and smooth segmentations that do not introduce further cell splitting or merging.

Radial 2D MRI scans of the hip are routinely used for the diagnosis of the cam-type of femoroacetabular impingement (FAI) and of avascular necrosis (AVN) of the femoral head, which are considered causes of hip joint osteoarthritis in young and active patients. However, for computer assisted planning of surgical treatment, it is highly desired to have 3D models of the proximal femur. In this paper, we propose a novel volumetric convolutional neural network (CNN) based framework to fully automatically extract 3D models of the proximal femur from sparsely hip radial slices. Our framework starts with a spatial transform to interpolate sparse 2D radial MR images to a densely sampled 3D volume data. Automated segmentation of the interpolated 3D volume data is very challenging due to the poor image quality and the interpolation artifact. To tackle these challenges, we introduce a multi-level latent shape space constrained 3D U-net, referred as Latent3DU-net, to incorporate prior shape knowledge into voxelwise semantic segmentation of the interpolated 3D volume. Comprehensive results obtained from 25 patient data demonstrated the effectiveness of the proposed framework.

Robust and accurate alignment of multimodal medical images is a very challenging task, which however is very useful for many clinical applications. For example, magnetic resonance (MR) and transrectal ultrasound (TRUS) image registration is a critical component in MR-TRUS fusion guided prostate interventions. However, due to the huge difference between the image appearances and the large variation in image correspondence, MR-TRUS image registration is a very challenging problem. In this paper, an adversarial image registration (AIR) framework is proposed. By training two deep neural networks simultaneously, one being a generator and the other being a discriminator, we can obtain not only a network for image registration, but also a metric network which can help evaluate the quality of image registration. The developed AIR-net is then evaluated using clinical datasets acquired through image-fusion guided prostate biopsy procedures and promising results are demonstrated.

Tract-specific diffusion measures, as derived from brain diffusion MRI, have been linked to white matter tract structural integrity and neurodegeneration. As a consequence, there is a large interest in the automatic segmentation of white matter tract in diffusion tensor MRI data. Methods based on the tractography are popular for white matter tract segmentation. However, because of the limited consistency and long processing time, such methods may not be suitable for clinical practice. We therefore developed a novel convolutional neural network based method to directly segment white matter tract trained on a low-resolution dataset of 9149 DTI images. The method is optimized on input, loss function and network architecture selections. We evaluated both segmentation accuracy and reproducibility, and reproducibility of determining tract-specific diffusion measures. The reproducibility of the method is higher than that of the reference standard and the determined diffusion measures are consistent. Therefore, we expect our method to be applicable in clinical practice and in longitudinal analysis of white matter microstructure.

Increased information sharing through short and long-range skip connections between layers in fully convolutional networks have demonstrated significant improvement in performance for semantic segmentation. In this paper, we propose Competitive Dense Fully Convolutional Networks (CDFNet) by introducing competitive maxout activations in place of naïve feature concatenation for inducing competition amongst layers. Within CDFNet, we propose two architectural contributions, namely competitive dense block (CDB) and competitive unpooling block (CUB) to induce competition at local and global scales for short and long-range skip connections respectively. This extension is demonstrated to boost learning of specialized sub-networks targeted at segmenting specific anatomies, which in turn eases the training of complex tasks. We present the proof-of-concept on the challenging task of whole body segmentation in the publicly available VISCERAL benchmark and demonstrate improved performance over multiple learning and registration based state-of-the-art methods.

In this paper, we develop a two-stage neural network solution for the challenging task of white-matter lesion segmentation. To cope with the vast variability in lesion sizes, we sample brain MR scans with patches at three different dimensions and feed them into separate fully convolutional neural networks (FCNs). In the second stage, we process large and small lesion separately, and use ensemble-nets to combine the segmentation results generated from the FCNs. A novel activation function is adopted in the ensemble-nets to improve the segmentation accuracy measured by Dice Similarity Coefficient. Experiments on MICCAI 2017 White Matter Hyperintensities (WMH) Segmentation Challenge data demonstrate that our two-stage-multi-sized FCN approach, as well as the new activation function, are effective in capturing white-matter lesions in MR images.

The human cerebellum has been recognized as a key brain structure for motor control and cognitive function regulation. Investigation of brain functional development in the early life has recently been focusing on both cerebral and cerebellar development. Accurate segmentation of the infant cerebellum into different tissues is among the most important steps for quantitative development studies. However, this is extremely challenging due to the weak tissue contrast, extremely folded structures, and severe partial volume effect. To date, there are very few works touching infant cerebellum segmentation. We tackle this challenge by proposing a densely connected convolutional network to learn robust feature representations of different cerebellar tissues towards automatic and accurate segmentation. Specifically, we develop a novel deep neural network architecture by directly connecting all the layers to ensure maximum information flow even among distant layers in the network. This is distinct from all previous studies. Importantly, the outputs from all previous layers are passed to all subsequent layers as contextual features that can guide the segmentation. Our method achieved superior performance than other state-of-the-art methods when applied to Baby Connectome Project (BCP) data consisting of both 6- and 12-month-old infant brain images.

Nuclei detection is an important task in the histology domain as it is a main step toward further analysis such as cell counting, cell segmentation, study of cell connections, etc. This is a challenging task due to complex texture of histology image, variation in shape, and touching cells. To tackle these hurdles, many approaches have been proposed in the literature where deep learning methods stand on top in terms of performance. Hence, in this paper, we propose a novel framework for nuclei detection based on Mixture Density Networks (MDNs). These networks are suitable to map a single input to several possible outputs and we utilize this property to detect multiple seeds in a single image patch. A new modified form of a cost function is proposed for training and handling patches with missing nuclei. The probability maps of the nuclei in the individual patches are next combined to generate the final image-wide result. The experimental results show the state-of-the-art performance on complex colorectal adenocarcinoma dataset.

In this work, we exploit the task of joint classification and weakly supervised localization of thoracic diseases from chest radiographs, with only image-level disease labels coupled with disease severity-level (DSL) information of a subset. A convolutional neural network (CNN) based attention-guided curriculum learning (AGCL) framework is presented, which leverages the severity-level attributes mined from radiology reports. Images in order of difficulty (grouped by different severity-levels) are fed to CNN to boost the learning gradually. In addition, highly confident samples (measured by classification probabilities) and their corresponding class-conditional heatmaps (generated by the CNN) are extracted and further fed into the AGCL framework to guide the learning of more distinctive convolutional features in the next iteration. A two-path network architecture is designed to regress the heatmaps from selected seed samples in addition to the original classification task. The joint learning scheme can improve the classification and localization performance along with more seed samples for the next iteration. We demonstrate the effectiveness of this iterative refinement framework via extensive experimental evaluations on the publicly available ChestXray14 dataset. AGCL achieves over 5.7% (averaged over 14 diseases) increase in classification AUC and 7%/11% increases in Recall/Precision for the localization task compared to the state of the art.

Numerous methods have been proposed to capture early hippocampus alterations caused by Alzheimer’s disease. Among them, patch-based grading approach showed its capability to capture subtle structural alterations. This framework applied on hippocampus obtains state-of-the-art results for AD detection but is limited for its prediction compared to the same approaches based on whole-brain analysis. We assume that this limitation could come from the fact that hippocampus is a complex structure divided into different subfields. Indeed, it has been shown that AD does not equally impact hippocampal subfields. In this work, we propose a graph-based representation of the hippocampal subfields alterations based on patch-based grading feature. The strength of this approach comes from better modeling of the inter-related alterations through the different hippocampal subfields. Thus, we show that our novel method obtains similar results than state-of-the-art approaches based on whole-brain analysis with improving by 4 percent points of accuracy patch-based grading methods based on hippocampus.

Localization of chest pathologies in chest X-ray images is a challenging task because of their varying sizes and appearances. We propose a novel weakly supervised method to localize chest pathologies using class aware deep multiscale feature learning. Our method leverages intermediate feature maps from CNN layers at different stages of a deep network during the training of a classification model using image level annotations of pathologies. During the training phase, a set of layer relevance weights are learned for each pathology class and the CNN is optimized to perform pathology classification by convex combination of feature maps from both shallow and deep layers using the learned weights. During the test phase, to localize the predicted pathology, the multiscale attention map is obtained by convex combination of class activation maps from each stage using the layer relevance weights learned during the training phase. We have validated our method using 112000 X-ray images and compared with the state-of-the-art localization methods. We experimentally demonstrate that the proposed weakly supervised method can improve the localization performance of small pathologies such as nodule and mass while giving comparable performance for bigger pathologies e.g., Cardiomegaly.

Accurate segmentation of medical images is an important step towards analyzing and tracking disease related morphological alterations in the anatomy. Convolutional neural networks (CNNs) have recently emerged as a powerful tool for many segmentation tasks in medical imaging. The performance of CNNs strongly depends on the size of the training data and combining data from different sources is an effective strategy for obtaining larger training datasets. However, this is often challenged by heterogeneous labeling of the datasets. For instance, one of the dataset may be missing labels or a number of labels may have been combined into a super label. In this work we propose a cost function which allows integration of multiple datasets with heterogeneous label subsets into a joint training. We evaluated the performance of this strategy on thigh MR and a cardiac MR datasets in which we artificially merged labels for half of the data. We found the proposed cost function substantially outperforms a naive masking approach, obtaining results very close to using the full annotations.

Segmentation of cells in scanning electron microscopy images is a challenging problem due to the presence of inhomogeneous illumination. Classical pre-processing methods for illumination normalization destroy the texture and add noise to the image. In this paper, we present a deep cell segmentation method using adversarial training that is robust to inhomogeneous illumination. Specifically, we apply a model based on U-net as the segmenter and a deep ConvNet as the discriminator for the adversarial training called SoLiD: “Segmentation of clostridioides difficile cells in the presence of inhomogeneous iLlumInation using a Deep adversarial network”. We also present an image augmentation algorithm to obtain the training images required for SoLid. The results indicate that SoLiD is robust to inhomogeneous illumination. The segmentation performance is compared to the U-net and the dice score is improved by 44%.

Deformable image registration is a fundamental problem in medical image analysis. During the last years, several methods based on deep convolutional neural networks (CNN) proved to be highly accurate to perform this task. These models achieved state-of-the-art accuracy while drastically reducing the required computational time, but mainly focusing on images of specific organs and modalities. To date, no work has reported on how these models adapt across different domains. In this work, we ask the question: can we use CNN-based registration models to spatially align images coming from a domain different than the one/s used at training time? We explore the adaptability of CNN-based image registration to different organs/modalities. We employ a fully convolutional architecture trained following an unsupervised approach. We consider a simple transfer learning strategy to study the generalisation of such model to unseen target domains, and devise a one-shot learning scheme taking advantage of the unsupervised nature of the proposed method. Evaluation on two publicly available datasets of X-Ray lung images and cardiac cine magnetic resonance sequences is provided. Our experiments suggest that models learned in different domains can be transferred at the expense of a decrease in performance, and that one-shot learning in the context of unsupervised CNN-based registration is a valid alternative to achieve consistent registration performance when only a pair of images from the target domain is available.

Currently there are still no early biomarkers to detect infants with risk of autism spectrum disorder (ASD), which is mainly diagnosed based on behavior observations at three or four years old. Since intervention efforts may miss a critical developmental window after 2 years old, it is significant to identify imaging-based biomarkers for early diagnosis of ASD. Although some methods using magnetic resonance imaging (MRI) for brain disease prediction have been proposed in the last decade, few of them were developed for predicting ASD in early age. Inspired by deep multi-instance learning, in this paper, we propose a patch-level data-expanding strategy for multi-channel convolutional neural networks to automatically identify infants with risk of ASD in early age. Experiments were conducted on the National Database for Autism Research (NDAR), with results showing that our proposed method can significantly improve the performance of early diagnosis of ASD.

Parkinson’s disease (PD) is a neurodegenerative progressive disease that mainly affects the motor systems of patients. To slow this disease deterioration, robust and accurate diagnosis of PD is an effective way to alleviate mental and physical sufferings of clinical intervention. In this paper, we propose a new unsupervised feature selection method via joint embedding learning and sparse regression using longitudinal multi-modal neuroimaging data. Specifically, the proposed method performs feature selection and local structure learning, simultaneously, to adaptively determine the similarity matrix. Meanwhile, we constrain the similarity matrix to make it contains c connected components for gaining the most accurate information of the neuroimaging data structure. The baseline data is utilized to establish the feature selection model to select the most discriminative features. Namely, we exploit baseline data to train four regression models for the clinical scores prediction (depression, sleep, olfaction, and cognition scores) and a classification model for the classification of PD disease in the future time point. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method on the Parkinson’s Progression Markers Initiative (PPMI) dataset. The experimental results demonstrate that, our proposed method can enhance the performance in clinical scores prediction and class label identification in longitudinal data and outperforms the state-of-art methods as well.

Currently, non-invasive imaging techniques such as magnetic resonance imaging (MRI) are emerging as powerful diagnostic tools for prostate cancer (PCa) characterization. This paper focuses on automated PCa classification on VERDICT (Vascular, Extracellular and Restricted Diffusion for Cytometry in Tumors) diffusion weighted (DW)-MRI, which is a non-invasive microstructural imaging technique that comprises a rich imaging protocol and a tissue computational model to map in vivo histological indices. The contribution of the paper is two fold. Firstly, we investigate the potential of automated, model-free PCa classification on raw VERDICT DW-MRI. Secondly, we attempt to adapt and evaluate novel fully convolutional neural networks (FCNNs) for PCa characterization. We present two neural network architectures that adapt U-Net and ResNet-18 to the PCa classification problem. We train the networks end-to-end on DW-MRI data and evaluate the diagnostic performance employing a 10-fold cross validation approach using data acquired from 103 patients. ResNet-18 outperforms U-Net with an average AUC of $$86.7\%$$ 86.7 % . Our results show promise for the utilization of raw VERDICT DW-MRI data and FCNNs for automating the PCa diagnostic pathway.

Videofluoroscopic swallowing study (VFSS) is a standard diagnostic tool for dysphagia. Previous computer assisted analysis of VFSS required manual preparation to mark several anatomical structures and to select time intervals of interest such as a pharyngeal phase during swallowing. These processes were still costly and challenging for clinicians. In this study, we present a novel approach to detect the pharyngeal phase of swallowing through whole of VFSS video clips using Inflated 3D Convolutional Networks (I3D) without additional manual annotations.

As shown in computer vision, the power of deep learning lies in automatically learning relevant and powerful features for any perdition task, which is made possible through end-to-end architectures. However, deep learning approaches applied for classifying medical images do not adhere to this architecture as they rely on several pre- and post-processing steps. This shortcoming can be explained by the relatively small number of available labeled subjects, the high dimensionality of neuroimaging data, and difficulties in interpreting the results of deep learning methods. In this paper, we propose a simple 3D Convolutional Neural Networks and exploit its model parameters to tailor the end-to-end architecture for the diagnosis of Alzheimer’s disease (AD). Our model can diagnose AD with an accuracy of 94.1% on the popular ADNI dataset using only MRI data, which outperforms the previous state-of-the-art. Based on the learned model, we identify the disease biomarkers, the results of which were in accordance with the literature. We further transfer the learned model to diagnose mild cognitive impairment (MCI), the prodromal stage of AD, which yield better results compared to other methods.

Accurate and robust segmentation of small organs in whole-body MRI is difficult due to anatomical variation and class imbalance. Recent deep network based approaches have demonstrated promising performance on abdominal multi-organ segmentations. However, the performance on small organs is still suboptimal as these occupy only small regions of the whole-body volumes with unclear boundaries and variable shapes. A coarse-to-fine, hierarchical strategy is a common approach to alleviate this problem, however, this might miss useful contextual information. We propose a two-stage approach with weighting schemes based on auto-context and spatial atlas priors. Our experiments show that the proposed approach can boost the segmentation accuracy of multiple small organs in whole-body MRI scans.

Masticatory muscles are of significant aesthetic and functional importance to craniofacial developments. Automatic segmentation is a crucial step for shape and functional analysis of muscles. In this paper, we propose an automatic masseter segmentation framework using a deep neural network with coupled feature learning and label prediction pathways. The volumetric features are learned using the unsupervised convolutional auto-encoder and integrated with multi-level features in the label prediction pathway to augment features for segmentation. The label prediction pathway is built upon the nested residual network which is feasible for information propagation and fast convergence. The proposed method realizes the voxel-wise label inference of masseter muscles from the clinically captured computed tomography (CT) images. In the experiments, the proposed method outperforms the compared state-of-the-arts, achieving a mean Dice similarity coefficient (DSC) of $$93\pm 1.2\%$$ 93 ± 1.2 % for the segmentation of masseter muscles.

Automatic segmentation has great potential to facilitate morphological measurements while simultaneously increasing efficiency. Nevertheless often users want to edit the segmentation to their own needs and will need different tools for this. There has been methods developed to edit segmentations of automatic methods based on the user input, primarily for binary segmentations. Here however, we present an unique training strategy for convolutional neural networks (CNNs) trained on top of an automatic method to enable interactive segmentation editing that is not limited to binary segmentation. By utilizing a robot-user during training, we closely mimic realistic use cases to achieve optimal editing performance. In addition, we show that an increase of the iterative interactions during the training process up to ten improves the segmentation editing performance substantially. Furthermore, we compare our segmentation editing CNN (interCNN) to state-of-the-art interactive segmentation algorithms and show a superior or on par performance.

Craniofacial growths and developments play an important role in treatment planning of orthopedics and orthodontics. Traditional growth studies are mainly on longitudinal growth datasets of 2D lateral cephalometric radiographs (LCR). In this paper, we propose a temporal consistent 2D-3D registration technique enabling 3D growth measurements of craniofacial structures. We initialize the independent 2D-3D registration by the convolutional neural network (CNN)-based regression, which produces the dense displacement field of the cone-beam computed tomography (CBCT) image when given the LCR. The temporal constraints of the growth-stable structures are used to refine the 2D-3D registration. Instead of traditional independent 2D-3D registration, we jointly solve the nonrigid displacement fields of a series of input LCRs captured at different ages. The hierarchical pyramid of the digitally reconstructed radiographs (DRR) is introduced to fasten the convergence. The proposed method has been applied to the growth dataset in clinical orthodontics. The resulted 2D-3D registration is consistent with both the input LCRs concerning the structural contours and the 3D volumetric images regarding the growth-stable structures.

Autosomal dominant polycystic kidney disease (ADPKD) characterized by progressive growth of renal cysts is the most prevalent and potentially lethal monogenic renal disease, affecting one in every 500–1000 people. Total Kidney Volume (TKV) and its growth computed from Computed Tomography images has been accepted as an essential prognostic marker for renal function loss. Due to large variation in shape and size of kidney in ADPKD, existing methods to compute TKV (i.e. to segment ADKP) including those based on 2D convolutional neural networks are not accurate enough to be directly useful in clinical practice. In this work, we propose multi-task 3D Convolutional Neural Networks to segment ADPK and achieve a mean DICE score of 0.95 and mean absolute percentage TKV error of 3.86%. Additionally, to solve the challenge of class imbalance, we propose to simply bootstrap cross entropy loss and compare results with recently prevalent dice loss in medical image segmentation community.

We present and evaluate a new deep neural network architecture for automatic thoracic disease detection on chest X-rays. Deep neural networks has shown great success in a plethora of vision recognition tasks such as image classification and object detection by stacking multiple layers of convolutional neural networks (CNN) in a feed forward manner. However the performance gain by going deeper has reached bottlenecks as a result of the trade-off between model complexity and discrimination power. We address this problem by utilizing recently developed routing-by agreement mechanism in our architecture. A novel characteristic of our network structure is that it extends routing to two types of layer connections (1) connection between feature maps in dense layers, (2) connection between primary capsules and prediction capsules in final classification layer. We show that our networks achieves comparable results with much fewer layers in the measurement of AUC score. We further show the combined benefits of model interpretability by generating Gradient-weighted Class Activation Mapping (Grad-CAM) for localization. We demonstrate our results on the NIH chestX-ray14 dataset that consists of 112,120 images on 30,805 unique patients including 14 kinds of lung diseases.

Magnetic resonance fingerprinting (MRF) is a novel quantitative imaging technique that allows simultaneous measurements of multiple important tissue properties in human body, e.g., T1 and T2 relaxation times. While MRF has demonstrated better scan efficiency as compared to conventional quantitative imaging techniques, further acceleration is desired, especially for certain subjects such as infants and young children. However, the conventional MRF framework only uses a simple template matching algorithm to quantify tissue properties, without considering the underlying spatial association among pixels in MRF signals. In this work, we aim to accelerate MRF acquisition by developing a new post-processing method that allows accurate quantification of tissue properties with fewer sampling data. Moreover, to improve the accuracy in quantification, the MRF signals from multiple surrounding pixels are used together to better estimate tissue properties at the central target pixel, which was simply done with the signal only from the target pixel in the original template matching method. In particular, a deep learning model, i.e., U-Net, is used to learn the mapping from the MRF signal evolutions to the tissue property map. To further reduce the network size of U-Net, principal component analysis (PCA) is used to reduce the dimensionality of the input signals. Based on in vivo brain data, our method can achieve accurate quantification for both T1 and T2 by using only 25% time points, which are four times of acceleration in data acquisition compared to the original template matching method.

In the originally published version of this chapter, the Acknowledgements section was missing. This has been corrected and an Acknowledgements section has been added.