Patch-Based Techniques in Medical Imaging

Longitudinal reproducibility is an essential concern in automated medical image segmentation, yet has proven to be an elusive objective as manual brain structure tracings have shown more than 10% variability. To improve reproducibility, longitudinal segmentation (4D) approaches have been investigated to reconcile temporal variations with traditional 3D approaches. In the past decade, multi-atlas label fusion has become a state-of-the-art segmentation technique for 3D image and many efforts have been made to adapt it to a 4D longitudinal fashion. However, the previous methods were either limited by using application specified energy function (e.g., surface fusion and multi model fusion) or only considered temporal smoothness on two consecutive time points (t and t + 1) under sparsity assumption. Therefore, a 4D multi-atlas label fusion theory for general label fusion purpose and simultaneously considering temporal consistency on all time points is appealing. Herein, we propose a novel longitudinal label fusion algorithm, called 4D joint label fusion (4DJLF), to incorporate the temporal consistency modeling via non-local patch-intensity covariance models. The advantages of 4DJLF include: (1) 4DJLF is under the general label fusion framework by simultaneously incorporating the spatial and temporal covariance on all longitudinal time points. (2) The proposed algorithm is a longitudinal generalization of a leading joint label fusion method (JLF) that has proven adaptable to a wide variety of applications. (3) The spatial temporal consistency of atlases is modeled in a probabilistic model inspired from both voting based and statistical fusion. The proposed approach improves the consistency of the longitudinal segmentation while retaining sensitivity compared with original JLF approach using the same set of atlases. The method is available online in open-source.

Automatic labeling of anatomical structures in brain images plays an important role in neuroimaging analysis. Among all methods, multi-atlas based segmentation methods are widely used, due to their robustness in propagating prior label information. However, non-linear registration is always needed, which is time-consuming. Alternatively, the patch-based methods have been proposed to relax the requirement of image registration, but the labeling is often determined independently by the target image information, without getting direct assistance from the atlases. To address these limitations, in this paper, we propose a multi-atlas guided 3D fully convolutional networks (FCN) for brain image labeling. Specifically, multi-atlas based guidance is incorporated during the network learning. Based on this, the discriminative of the FCN is boosted, which eventually contribute to accurate prediction. Experiments show that the use of multi-atlas guidance improves the brain labeling performance.

Numerous brain disorders are associated with ventriculomegaly; normal pressure hydrocephalus (NPH) is one example. NPH presents with dementia-like symptoms and is often misdiagnosed as Alzheimer’s due to its chronic nature and nonspecific presenting symptoms. However, unlike other forms of dementia NPH can be treated surgically with an over 80% success rate on appropriately selected patients. Accurate assessment of the ventricles, in particular its sub-compartments, is required to diagnose the condition. Existing segmentation algorithms fail to accurately identify the ventricles in patients with such extreme pathology. We present an improvement to a whole brain segmentation approach that accurately identifies the ventricles and parcellates them into four sub-compartments. Our work is a combination of patch-based tissue segmentation and multi-atlas registration-based labeling. We include a validation on NPH patients, demonstrating superior performance against state-of-the-art methods.

The hippocampus is a brain structure that is involved in several cognitive functions such as memory and learning. It is a structure of great interest in the study of the healthy and diseased brain due to its relationship to several neurodegenerative pathologies. In this work, we propose a novel patch-based method that uses an ensemble of boosted neural networks to perform the hippocampus subfield segmentation on multimodal MRI. This new method minimizes both random and systematic errors using an overcomplete autocontext patch-based labeling using a novel boosting strategy. The proposed method works well on high resolution MRI but also on standard resolution images after superresolution. Finally, the proposed method was compared with a similar state-of-the-art methods showing better results in terms of both accuracy and efficiency.

Multi-atlas segmentation has shown promising results in the segmentation of biomedical images. In the most common approach, registration is used to warp the atlases to the target space and then the warped atlas labelmaps are fused into a consensus segmentation. Label fusion in target space has shown to produce very accurate segmentations although at the expense of registering all atlases to each target image. Moreover, appearance and label information used by label fusion is extracted from the warped atlases, which are subject to interpolation errors. This work explores the role of extracting this information from the native spaces and adapt two label fusion approaches to this scheme. Results on the segmentation of subcortical brain structures indicate that using atlases in their native space yields superior performance than warping the atlases to the target. Moreover, using the native space lessens the computational requirements in terms of number of registrations and learning.

We introduce a functional for the learning of an optimal database for patch-based image segmentation with application to coronary lumen segmentation from coronary computed tomography angiography (CCTA) data. The proposed functional consists of fidelity, sparseness and robustness to small-variations terms and their associated weights. Existing work address database optimization by prototype selection aiming to optimize the database by either adding or removing prototypes according to a set of predefined rules. In contrast, we formulate the database optimization task as an energy minimization problem that can be solved using standard numerical tools. We apply the proposed database optimization functional to the task of optimizing a database for patch-base coronary lumen segmentation. Our experiments using the publicly available MICCAI 2012 coronary lumen segmentation challenge data show that optimizing the database using the proposed approach reduced database size by 96% while maintaining the same level of lumen segmentation accuracy. Moreover, we show that the optimized database yields an improved specificity of CCTA based fractional flow reserve (0.73 vs 0.7 for all lesions and 0.68 vs 0.65 for obstructive lesions) using a training set of 132 (76 obstructive) coronary lesions with invasively measured FFR as the reference.

Recent innovations in tissue clearing and light sheet microscopy allow rapid acquisition of three-dimensional micron resolution images in fluorescently labeled brain samples. These data allow the observation of every cell in the brain, necessitating an accurate and high-throughput cell segmentation method in order to perform basic operations like counting number of cells within a region; however, large computational challenges given noise in the data and sheer number of features to identify. Inspired by the success of deep learning technique in medical imaging, we propose a supervised learning approach using convolution neural network (CNN) to learn the non-linear relationship between local image appearance (within an image patch) and manual segmentations (cell or background at the center of the underlying patch). In order to improve the segmentation accuracy, we further integrate high-level contextual features with low-level image appearance features. Specifically, we extract contextual features from the probability map of cells (output of current CNN) and train the next CNN based on both patch-wise image appearance and contextual features, extending previous methods into a cascaded approach. Using (a) high-level contextual features extracted from the cell probability map and (b) the spatial information of cell-to-cell locations, our cascaded CNN progressively improves the segmentation accuracy. We have evaluated the segmentation results on mouse brain images, and compared conventional image processing approaches. More accurate and robust segmentation results have been achieved with our cascaded CNN method, indicating the promising potential of our proposed cell segmentation method for use in large tissue cleared images.

It has been recently demonstrated that the local BOLD signals in resting-state fMRI (rs-fMRI) can be captured for the white matter (WM) by functional correlation tensors (FCTs). FCTs provide similar orientation information as diffusion tensors (DTs), and also functional information concerning brain dynamics. However, FCTs are susceptible to noise due to the low signal-to-noise ratio nature of WM BOLD signals. Here we introduce a robust FCT estimation method to facilitate individualized diagnosis. First, we develop a noise-tolerating patch-based approach to measure spatiotemporal correlations of local BOLD signals. Second, it is also enhanced by DTs predicted from the input rs-fMRI using a learning-based regression model. We evaluate our trained regressor using the high-resolution HCP dataset. The regressor is then applied to estimate the robust FCTs for subjects in the ADNI2 dataset. We demonstrate for the first time the disease diagnostic value of robust FCTs.

Alzheimer’s disease (AD) is characterized by a progressive decline in the cognitive functions accompanied by an atrophic process which can already be observed in the early stages using magnetic resonance images (MRI). Individualized prediction of future progression to AD, when patients are still in the mild cognitive impairment (MCI) stage, has potential impact for preventive treatment. Atrophy patterns extracted from longitudinal MRI sequences provide valuable information to identify MCI patients at higher risk of developing AD in the future. We present a novel descriptor that uses the similarity between local image patches to encode local displacements due to atrophy between a pair of longitudinal MRI scans. Using a conventional logistic regression classifier, our descriptor achieves \(76\%\) accuracy in predicting which MCI patients will progress to AD up to 3 years before conversion.

Alzheimer’s disease is a neurodegenerative process leading to irreversible mental dysfunctions. The development of new biomarkers is crucial to perform an early detection of this disease. Among new biomarkers proposed during the last decades, patch-based grading framework demonstrated state-of-the-art results. In this paper, we study the potential using texture information based on Gabor filters to improve patch-based grading method performance, with a focus on the hippocampal structure. We also propose a novel fusion framework to efficiently combine multiple grading maps derived from a Gabor filters bank. Finally, we compare our new texture-based grading biomarker with the state-of-the-art approaches to demonstrate the high potential of the proposed method.

Histological images are very important for diagnosis of cancer and other diseases. However, during the preparation of the histological slides for microscopy, the 3D information of the tissue specimen gets lost. Therefore, many 3D reconstruction methods for histological images have been proposed. However, most approaches rely on the histological 2D images alone, which makes 3D reconstruction difficult due to the large deformations introduced by cutting and preparing the histological slides. In this work, we propose an image-guided approach to 3D reconstruction of histological images. Before histological preparation of the slides, the specimen is imaged using X-ray microtomography (micro CT). We can then align each histological image back to the micro CT image utilizing non-rigid registration. Our registration results show that our method can provide smooth 3D reconstructions with micro CT guidance.

In the last decade, the technological progress of multi-slice CT imaging has turned CCTA into a valuable tool for coronary assessment in many low to medium risk patients. Nevertheless, CCTA protocols expose the patient to high radiation doses, imposed by image quality and multiple cardiac phase acquisition requirements. Widespread use of CCTA calls for significant reduction of radiation exposure while maintaining high image quality as required for coronary assessment. Denoising algorithms have been recently applied to low-dose CT scans after image reconstruction. In this work, a fast neural regression framework is proposed for the denoising of low-dose CCTA. For this purpose, regression networks are trained to synthesize high-SNR patches directly from low-SNR input patches. In contrast to published methods, the denoising network is trained on real noise directly learned from noisy CT data rather than assuming a known parametric noise model. The denoised value for each pixel is computed as a function of the synthesized patches overlapping the pixel. The proposed algorithm is compared to state-of-the-art published algorithms for synthetic and real noise. The feature similarity index (FSIM) achieved by the proposed method is superior in all the comparisons with other methods, for synthetic radiation dose reductions higher than 90%. The results are further supported qualitatively, by observing a significant improvement in subsequent coronary reconstruction performed by commercial software on denoised images. The fast and high quality denoising capability suggests the proposed algorithm as a promising method for low-dose CCTA denoising.

Ultrasound elastography is an imaging modality that computes the elasticity of tissue through measuring shear waves from a mechanical excitation using pulse-echo ultrasound. To better measure shear waves and reduce acquisition time, elastography would benefit from a higher framerate, which is limited by conventional focused line-by-line acquisition. This paper proposes a dictionary learning-based framework that increases the framerate of steady state elastography. The method uses patches extracted from images with higher scanline density to train a dictionary, and uses this dictionary to interpolate images with lower scanline density collected at a faster framerate. Experiments on a tissue mimicking phantom showed when the framerate is increased 8 times, the reconstructed image using the proposed method achieved a 17.6 dB Peak Signal-to-Noise Ratio. The method was also implemented on a steady state elastography system, where elasticity measurements similar to conventional methods were obtained with a shorter total acquisition time.

Detecting tumor regions in breast ultrasound images has always been an interesting topic. Due to the complex structure of breasts and the existence of noise in the ultrasound images, traditional handcraft feature based methods usually cannot achieve satisfactory results. With the recent advance of deep learning, the performance of object detection has been boosted to a great extent, especially for general object detection. In this paper, we aim to systematically evaluate the performance of several existing state-of-the-art object detection methods for breast tumor detection. To achieve that, we have collected a new dataset consisting of 579 benign and 464 malignant lesion cases with the corresponding ultrasound images manually annotated by experienced clinicians. Comprehensive experimental results clearly show that the recently proposed convolutional neural network based method, Single Shot MultiBox Detector (SSD), outperforms other methods in terms of both precision and recall.

Automatic detection of liver lesions in CT images poses a great challenge for researchers. In this work we present a deep learning approach that models explicitly the variability within the non-lesion class, based on prior knowledge of the data, to support an automated lesion detection system. A multi-class convolutional neural network (CNN) is proposed to categorize input image patches into sub-categories of boundary and interior patches, the decisions of which are fused to reach a binary lesion vs non-lesion decision. For validation of our system, we use CT images of 132 livers and 498 lesions. Our approach shows highly improved detection results that outperform the state-of-the-art fully convolutional network. Automated computerized tools, as shown in this work, have the potential in the future to support the radiologists towards improved detection.

This paper adapts the joint label fusion (JLF) multi-atlas image segmentation algorithm to the problem of multiple sclerosis (MS) lesion segmentation in multi-modal MRI. Conventionally, JLF requires a set of atlas images to be co-registered to the target image using deformable registration. However, given the variable spatial distribution of lesions in the brain, whole-brain deformable registration is unlikely to line up lesions between atlases and the target image. As a solution, we propose to first pre-segment the target image using an intensity regression based technique, yielding a set of “candidate” lesions. Each “candidate” lesion is then matched to a set of similar lesions in the atlas based on location and size; and deformable registration and JLF are applied at the level of the “candidate” lesion. The approach is evaluated on a dataset of 74 subjects with MS and shown to improve Dice similarity coefficient with reference manual segmentation by 12% over intensity regression technique.

The current gold standard for interpreting patient tissue samples is the visual inspection of whole–slide histopathology images (WSIs) by pathologists. They generate a pathology report describing the main findings relevant for diagnosis and treatment planning. Searching for similar cases through repositories for differential diagnosis is often not done due to a lack of efficient strategies for medical case–based retrieval. A patch–based multimodal retrieval strategy that retrieves similar pathology cases from a large data set fusing both visual and text information is explained in this paper. By fine–tuning a deep convolutional neural network an automatic representation is obtained for the visual content of weakly annotated WSIs (using only a global cancer score and no manual annotations). The pathology text report is embedded into a category vector of the pathology terms also in a non–supervised approach. A publicly available data set of 267 prostate adenocarcinoma cases with their WSIs and corresponding pathology reports was used to train and evaluate each modality of the retrieval method. A MAP (Mean Average Precision) of 0.54 was obtained with the multimodal method in a previously unseen test set. The proposed retrieval system can help in differential diagnosis of tissue samples and during the training of pathologists, exploiting the large amount of pathology data already existing digital hospital repositories.

In this work we introduce sparse representation techniques for classification of high-dimensional imaging patterns into healthy and diseased states. We also propose a spatial block decomposition methodology that is used for training an ensemble of classifiers to address irregularities of the approximation problem. We first apply this framework to classification of bone radiography images for osteoporosis diagnosis. The second application domain is separation of breast lesions into benign and malignant. These are challenging classification problems because the imaging patterns are typically characterized by high Bayes error rate in the original space. To evaluate the classification performance we use cross-validation techniques. We also compare our sparse-based classification with state-of-the-art texture-based classification techniques. Our results indicate that decomposition into patches addresses difficulties caused by ill-posedness and improves original sparse classification.