Medical Image Computing and Computer Assisted Intervention – MICCAI 2018

In recent years, brain network analysis has attracted considerable interests in the field of neuroimaging analysis. It plays a vital role in understanding biologically fundamental mechanisms of human brains. As the upward trend of multi-source in neuroimaging data collection, effective learning from the different types of data sources, e.g. multimodal and longitudinal data, is much in demand. In this paper, we propose a general coupling framework, the multimodal neuroimaging network fusion with longitudinal couplings (MMLC), to learn the latent representations of brain networks. Specifically, we jointly factorize multimodal networks, assuming a linear relationship to couple network variance across time. Experimental results on two large datasets demonstrate the effectiveness of the proposed framework. The new approach integrates information from longitudinal, multimodal neuroimaging data and boosts statistical power to predict psychometric evaluation measures.

In this paper, we introduce a penalized geodesic tractography (PGT) algorithm for mitigating gyral bias in cortical tractography, which is essential for improving cortical connectomics. Unlike deterministic and probabilistic tractography algorithms that perform one-way tracking, PGT solves a global optimization problem in estimating the pathways connecting multiple regions, instead of local step-by-step orientation tracing. PGT is unconfounded by local false-positive or false-negative fiber orientations and ensures that fiber streamlines that are intended to connect two regions do not terminate prematurely. We show that PGT reduces gyral bias by allowing streamlines to make sharper turns into the cortical gyral matter and results in a significantly more uniform spatial distribution of cortical connections.

Diffusion tractography suffers from a difficult sensitivity-specificity trade-off. We present an approach that leverages knowledge about anatomically well-known tracts (anchor tracts) in a tractogram to quantitatively assess the remaining tracts (candidate tracts) according to their plausibility in conjunction with this context information. We show that our approach has the potential for greatly reducing the number of false positive tracts in fiber tractography while maintaining high sensitivities using phantom experiments (AUC 0.91). To investigate the applicability of the approach in vivo, we analyze 110 subjects of the Human Connectome Project young adult study. We demonstrate how the approach may be used for structured analysis of in vivo tractography and show supporting evidence for tracts previously discussed in the literature, while potentially sparking discussions about the role of others.

Diffusion tensor imaging (DTI) based group analysis has helped uncover the impact of white matter injuries in a wide range of studies involving subjects from preterm neonates to adults. The application of these methods to fetal cohorts, however, has been hampered by the challenging nature of in utero fetal DTI caused by unconstrained fetal motion, limited scan times, and limited signal-to-noise ratio. We present a framework that addresses these issues to systematically evaluate group differences in fetal cohorts. A motion-robust DTI computation approach with a new unbiased DTI template construction method is unified with kernel-regression in age and tensor-specific registration to normalize DTI volumes in an unbiased space. A robust statistical approach is used to map region-specific group differences to the medial representation of the tracts of interest. The proposed approach was applied and showed, for the first time, differences in local white matter fractional anisotropy based on in utero DTI of fetuses with congenital heart disease and age-matched healthy controls. This paper suggests the need for fetal-specific pipelines to be used for DTI-based group analysis involving fetal cohorts.

While the major white matter tracts are of great interest to numerous studies in neuroscience and medicine, their manual dissection in larger cohorts from diffusion MRI tractograms is time-consuming, requires expert knowledge and is hard to reproduce. Tract orientation mapping (TOM) is a novel concept that facilitates bundle-specific tractography based on a learned mapping from the original fiber orientation distribution function (fODF) peaks to a list of tract orientation maps (also abbr. TOM). Each TOM represents one of the known tracts with each voxel containing no more than one orientation vector. TOMs can act as a prior or even as direct input for tractography. We use an encoder-decoder fully-convolutional neural network architecture to learn the required mapping. In comparison to previous concepts for the reconstruction of specific bundles, the presented one avoids various cumbersome processing steps like whole brain tractography, atlas registration or clustering. We compare it to four state of the art bundle recognition methods on 20 different bundles in a total of 105 subjects from the Human Connectome Project. Results are anatomically convincing even for difficult tracts, while reaching low angular errors, unprecedented runtimes and top accuracy values (Dice). Our code and our data are openly available.

In connectomics, tractography involves tracing connections across gray-white matter boundaries in gyral blades of complex cortical convolutions. To date, most tractography algorithms exhibit gyral bias with fiber streamlines preferentially terminating at gyral crowns rather than sulcal banks or fundi. In this work, we will demonstrate that a multi-tissue global estimation framework of the asymmetric fiber orientation distribution function (AFODF) will mitigate the effects of gyral bias and will allow fiber streamlines at gyral blades to make sharper turns into the cortical gray matter. This is validated using in-vivo data from the Human Connectome Project (HCP), showing that, in a typical gyral blade with high curvature, the fiber streamlines estimated using AFODFs bend more naturally into the cortex than FODFs. Furthermore, we show that AFODF tractography results in better cortico-cortical connectivity.

We propose a novel way of estimating fiber orientation distribution functions (fODFs) from diffusion MRI. Our method combines convex optimization with unsupervised learning in a way that preserves the relative benefits of both. In particular, we regularize constrained spherical deconvolution (CSD) with a prior that is derived from an fODF autoencoder, effectively encouraging solutions that are similar to fODFs observed in high-quality training data. Our method improves results on independent test data, especially when only few measurements or relatively weak diffusion weighting (low b values) are available.

A multiple sclerosis (MS) lesion at an early stage undergoes active blood brain barrier (BBB) breakdown. Identifying MS lesions in a patient which are undergoing active BBB breakdown is of critical importance for MS burden evaluation and treatment planning. However in non-contrast enhanced structural magnetic resonance imaging (MRI) the regions of the lesion undergoing active BBB breakdown cannot be distinguished from the other parts of the lesion. Hence gadolinium (Gd) contrast enhanced T1-weighted MR images are used for this task. However some side effects of Gd injection into patients have been increasingly reported recently. The BBB breakdown is reflected by the condition of tissue microstructure such as increased inflammation, presence of higher extra-cellular matter and debris. We thus propose a framework to predict enhancing regions in MS lesions using tissue microstructure information derived from T2 relaxometry and diffusion MRI (dMRI) multi-compartment models. We show that combination of the dMRI and T2 relaxometry microstructure information can distinguish the Gd enhancing lesion regions from the other regions in MS lesions.

The single diffusion tensor model for mapping the brain white matter microstructure has long been criticized as providing sensitive yet non-specific clinical biomarkers for neurodegenerative diseases because (i) voxels in diffusion images actually contain more than one homogeneous tissue population and (ii) diffusion in a single homogeneous tissue can be non-Gaussian. Analytic models for compartmental diffusion signals have thus naturally emerged but there is surprisingly little for processing such images (estimation, smoothing, registration, atlasing, statistical analysis). We propose to embed these signals into a Bayes Hilbert space that we properly define and motivate. This provides a unified framework for compartment diffusion image computing. Experiments show that (i) interpolation in Bayes space features improved robustness to noise compared to the widely used log-Euclidean space for tensors and (ii) it is possible to trace complex key pathways such as the pyramidal tract using basic deterministic tractography thanks to the combined use of Bayes interpolation and multi-compartment diffusion models.

Improved outcome in patients with ischemic stroke is achieved through acute diagnosis and early restoration of cerebral flow in appropriate patients. Diffusion-weighted MR imaging (DWI) plays a central role in these efforts by enabling rapid early localization and quantification of ischemic lesions. Automated detection and quantification can potentially accelerate diagnosis, improve treatment safety and efficacy and reduce costs. However, the manual quantification of acute ischemic stroke volumes for algorithm training is time consuming and imprecise. We present YNet as a novel fully-automated deep learning algorithm for detection and volumetric segmentation and quantification of acute cerebral ischemic lesions from DWI. The algorithm is a semi-supervised multi-tasking deep neural network architecture we developed that enables the combination of both weak labels derived from radiology report classification and manually delineated pixel level training data. The model is trained on a very large dataset of 10000 studies, achieves detection sensitivity 0.981, detection specificity 0.980 and segmentation Dice score 0.623 on a heterogeneous test set.

Primate brain evolution has been an intriguing research topic for centuries. Previous comparative studies focused on identification of species-preserved cortical landmarks or axonal pathways via approaches such as registration. However, because of huge cross-species variations, these studies dealt with only a few specific fasciculi and cortices or relied on a predefined brain parcellation shared among species. In this work, we used T1-weighted MRI data and diffusion MRI data to identify novel landmarks on entire cortices based on folding patterns on macaque and human brains and further proposed a pipeline to establish cross-species correspondence for them based on networks derived from streamline fibers. Our experimental results are consistent with the reports in the literature, demonstrating the effectiveness and promise of this framework. The merits of this work lie in not only the identification of a novel, large group of species-preserved cortical landmarks, but also new insights into the relationship between cortical folding patterns and axonal wiring diagrams along the evolution line.

Glioblastoma is the most common and aggressive brain tumor. In clinical practice, diffusion MRI (dMRI) enables tumor infiltration assessment, tumor recurrence prognosis, and identification of white-matter tracks close to the resection volume. However, the vasogenic edema (free-water) surrounding the tumor causes partial volume contamination, which induces a bias in the estimates of the diffusion properties and limits the clinical utility of dMRI.We introduce a voxel-based deep learning method to map and correct free-water partial volume contamination in dMRI. Our model learns from synthetically generated data a non-parametric forward model that maps free-water partial volume contamination to volume fractions. This is independent of the diffusion protocol and can be used retrospectively. We show its benefits in glioblastoma cases: first, a gain of statistical power; second, quantification of free-water and tissue volume fractions; and third, correction of free-water contaminated diffusion metrics. Free-water elimination yields more relevant information from the available data.

Stroke is the second most common cause of death in developed countries, where rapid clinical intervention can have a major impact on a patient’s life. To perform the revascularization procedure, the decision making of physicians considers its risks and benefits based on multi-modal MRI and clinical experience. Therefore, automatic prediction of the ischemic stroke lesion outcome has the potential to assist the physician towards a better stroke assessment and information about tissue outcome. Typically, automatic methods consider the information of the standard kinetic models of diffusion and perfusion MRI (e.g. Tmax, TTP, MTT, rCBF, rCBV) to perform lesion outcome prediction. In this work, we propose a deep learning method to fuse this information with an automated data selection of the raw 4D PWI image information, followed by a data-driven deep-learning modeling of the underlying blood flow hemodynamics. We demonstrate the ability of the proposed approach to improve prediction of tissue at risk before therapy, as compared to only using the standard clinical perfusion maps, hence suggesting on the potential benefits of the proposed data-driven raw perfusion data modelling approach.

Diffusion MRI (dMRI) data is increasingly being acquired on multiple scanners as part of large multi-center neuroimaging studies. However, diffusion imaging is particularly sensitive to scanner-specific differences in coil sensitivity, reconstruction algorithms, acquisition parameters as well as the scanner magnetic field strength, which precludes joint analysis of such multi-site data. Earlier works on dMRI data harmonization were limited to data acquired on different scanners but with the same magnetic field strength (3T). In this work, we explore the possibility of harmonizing dMRI data acquired on scanners with different magnetic field strengths, i.e., 3T and 7T. We propose a linear and several machine learning based non-linear mapping algorithms that use rotation invariant spherical harmonic (RISH) features to map the dMRI data (the raw signal) between scanners without changing the fiber orientations. We extensively validate our algorithms on in-vivo data from the Human Connectome Project (HCP) where we used data from 40 subjects with scans done on both 7T and 3T scanners (10 training + 30 test). Using several quantitative metrics such as the root mean squared error (RMSE) in the harmonized dMRI signal and diffusion measures as well as a fiber bundle overlap measure, our preliminary results on 30 test subjects shows that the convolutional neural network (CNN) based algorithm can reliably harmonize the raw dMRI signal across magnetic field strengths. The algorithms proposed are general and can be used for dMRI data harmonization in multi-site studies.

Normative modeling has recently been proposed as an alternative for the case-control approach in modeling heterogeneity within clinical cohorts. Normative modeling is based on single-output Gaussian process regression that provides coherent estimates of uncertainty required by the method but does not consider spatial covariance structure. Here, we introduce a scalable multi-task Gaussian process regression (S-MTGPR) approach to address this problem. To this end, we exploit a combination of a low-rank approximation of the spatial covariance matrix with algebraic properties of Kronecker product in order to reduce the computational complexity of Gaussian process regression in high-dimensional output spaces. On a public fMRI dataset, we show that S-MTGPR: (1) leads to substantial computational improvements that allow us to estimate normative models for high-dimensional fMRI data whilst accounting for spatial structure in data; (2) by modeling both spatial and across-sample variances, it provides higher sensitivity in novelty detection scenarios.

Understanding human brain functional development in the very early ages is of great importance for charting normative development and detecting early neurodevelopmental disorders, but it is very challenging. We propose a group-constrained, robust community detection method for better understanding of developing brain functional connectome from neonate to two-year-old. For such a multi-subject, multi-age-group network topology study, we build a multi-layer functional network by adding inter-subject edges, and detect modular structure (communities) to explore topological changes of multiple functional systems at different ages and across subjects. This “Multi-Layer Inter-Subject-Constrained Modularity Analysis (MLISMA)” can detect group consistent modules without losing individual information, thus allowing assessment of individual variability in the brain modular topology, a key metric for developmental individualized fingerprinting. We propose a heuristic parameter optimization strategy to wisely determine the necessary parameters that define the modular configuration. Our method is validated to be feasible using longitudinal 0–1–2 year’s old infant brain functional MRI data, and reveals novel developmental trajectories of brain functional connectome. This work was supported by the NIH grants, EB022880, 1U01MH110274, and MH100217.

Even though the number of longitudinal resting-state-fMRI studies is increasing, accurately characterizing the changes in functional connectivity across visits is a largely unexplored topic. To improve characterization, we design a Riemannian framework that represents the functional connectivity pattern of a subject at a visit as a point on a Riemannian manifold. Geodesic regression across the ‘sample’ points of a subject on that manifold then defines the longitudinal trajectory of their connectivity pattern. To identify group differences specific to regions of interest (ROI), we map the resulting trajectories of all subjects to a common tangent space via the Lie group action. We account for the uncertainty in choosing the common tangent space by proposing a test procedure based on the theory of latent p-values. Unlike existing methods, our proposed approach identifies sex differences across 246 subjects, each of them being characterized by three rs-fMRI scans.

Single subject task-based fMRI analyses generally suffer from low detection sensitivity with parameter estimates from the general linear model (GLM) lying below the significance threshold especially for similar contrasts or conditions. In this paper, we present a shape-based approach for alignment of condition-specific time course activity for single subject task-based fMRI. Our approach extracts signals for each condition from the entire time course, constructs an unbiased average of those signals, and warps each signal to the mean. As the warping is diffeomorphic, nonlinear and allows large deformations of time series if required, we term this approach as elastic functional registration. On a single subject level, our method significantly detects more clusters and more activated voxels in relevant subcortical regions in healthy controls.

We propose a matrix factorization technique that decomposes the resting state fMRI (rs-fMRI) correlation matrices for a patient population into a sparse set of representative subnetworks, as modeled by rank one outer products. The subnetworks are combined using patient specific non-negative coefficients; these coefficients are also used to model, and subsequently predict the clinical severity of a given patient via a linear regression. Our generative-discriminative framework is able to exploit the structure of rs-fMRI correlation matrices to capture group level effects, while simultaneously accounting for patient variability. We employ ten fold cross validation to demonstrate the predictive power of our model on a cohort of fifty eight patients diagnosed with Autism Spectrum Disorder. Our method outperforms classical semi-supervised frameworks, which perform dimensionality reduction on the correlation features followed by non-linear regression to predict the clinical scores.

Spatial distribution patterns of functional brain networks derived from resting state fMRI data have been widely examined in the literature. However, the spatial overlap patterns among those brain networks have been rarely investigated, though spatial overlap is a fundamental principle of functional brain network organization. To bridge this gap, this paper presents an effective 3D convolutional neural network (CNN) framework to derive discriminative and meaningful spatial brain network overlap patterns that can characterize and differentiate Autism Spectrum Disorder (ASD) from healthy controls. Our experimental results demonstrated that the spatial distribution patterns of connectome-scale functional network maps per se have little discrimination power in differentiating ASD from controls via the CNN framework. In contrast, the spatial overlap patterns instead of spatial patterns per se among these connectome-scale networks, learned via the same CNN framework, have remarkable differentiation power in separating ASD from controls. Our work suggested the promise of using CNN deep learning methodologies to discover discriminative and meaningful spatial network overlap patterns and their applications in functional connectomics of brain disorders such as ASD.

Simultaneous modeling of the spatio-temporal variation patterns of brain functional network from 4D fMRI data has been an important yet challenging problem for the field of cognitive neuroscience and medical image analysis. Inspired by the recent success in applying deep learning for functional brain decoding and encoding, in this work we propose a spatio-temporal convolutional neural network (ST-CNN) to jointly learn the spatial and temporal patterns of targeted network from the training data and perform automatic, pin-pointing functional network identification. The proposed ST-CNN is evaluated by the task of identifying the Default Mode Network (DMN) from fMRI data. Results show that while the framework is only trained on one fMRI dataset, it has the sufficient generalizability to identify the DMN from different populations of data as well as different cognitive tasks. Further investigation into the results show that the superior performance of ST-CNN is driven by the jointly-learning scheme, which capture the intrinsic relationship between the spatial and temporal characteristic of DMN and ensures the accurate identification.

The human brain is always in the process of constantly changing. Given that the conventional measurements of regional brain activity are less sensitive to changes over time, three dynamic measurements were proposed to capture the temporal variability of regional brain activity. In this study, dynamic amplitude of low frequency fluctuation (d-ALFF), dynamic fractional amplitude of low frequency fluctuation (d-fALFF) and dynamic regional homogeneity (d-ReHo) were obtained from resting-state functional magnetic resonance imaging (rs-fMRI) of both 238 ADHD and 239 typical developing (TD) subjects. Then, they were applied to detecting the regional activity differences between ADHD and TD group. Compared with the conventional measurements (ALFF, fALFF and ReHo), the dynamic measurements were more sensitive in exploring the differences of regional brain activity between ADHD and TD group. The three new measurements not only enrich the diversity of methods for investigating the dynamic variation of regional brain activity, but also emphasize the significance of detecting the temporal variability of regional brain activity.

Cross subject functional studies of cerebral cortex require cortical registration that aligns functional brain regions. While cortical folding patterns are approximate indicators of the underlying cytoarchitecture, coregistration based on these features alone does not accurately align functional regions in cerebral cortex. This paper presents a method for cortical surface registration (rfDemons) based on resting fMRI (rfMRI) data that uses curvature-based anatomical registration as an initialization. In contrast to existing techniques that use connectivity-based features derived from rfMRI, the proposed method uses ‘synchronized’ resting rfMRI time series directly. The synchronization of rfMRI data is performed using the BrainSync transform which applies an orthogonal transform to the rfMRI time series to temporally align them across subjects. The rfDemons method was applied to rfMRI from the Human Connectome Project and evaluated using task fMRI data to explore the impact of cortical registration performed using resting fMRI data on functional alignment of the cerebral cortex.

Autism spectrum disorder (ASD) is a complex neurodevelopmental disorder. Finding the biomarkers associated with ASD is extremely helpful to understand the underlying roots of the disorder and can lead to earlier diagnosis and more targeted treatment. Although Deep Neural Networks (DNNs) have been applied in functional magnetic resonance imaging (fMRI) to identify ASD, understanding the data driven computational decision making procedure has not been previously explored. Therefore, in this work, we address the problem of interpreting reliable biomarkers associated with identifying ASD; specifically, we propose a 2-stage method that classifies ASD and control subjects using fMRI images and interprets the saliency features activated by the classifier. First, we trained an accurate DNN classifier. Then, for detecting the biomarkers, different from the DNN visualization works in computer vision, we take advantage of the anatomical structure of brain fMRI and develop a frequency-normalized sampling method to corrupt images. Furthermore, in the ASD vs. control subjects classification scenario, we provide a new approach to detect and characterize important brain features into three categories. The biomarkers we found by the proposed method are robust and consistent with previous findings in the literature. We also validate the detected biomarkers by neurological function decoding and comparing with the DNN activation maps.

Since the introduction of BOLD (Blood Oxygen Level Dependent) imaging, the hemodynamic response model has remained the standard analysis approach to relate activated brain areas to extrinsic task conditions. Ongoing brain activity unrelated to the task is neglected and considered noise. By contrast, model-free blind source separation techniques such as Independent Component Analysis (ICA) have been used in intrinsic task-free experiments to reveal functional systems usually referred to as “resting-state” networks. However, matrix factorization techniques applied to BOLD imaging do not model the translation of neuronal activity into BOLD fluctuations and depend on arbitrarily chosen regularization measures such as statistical independence or sparsity. We present a novel neurobiologically-driven matrix factorization approach. Our matrix factorization model incorporates the hemodynamic response function that enables the estimation of underlying neural activity in individual brain networks that present during task- and task-free BOLD-fMRI experiments. We validate our model on the recently published Midnight Scanning Club dataset including five hours of task-free and six hours of various task experiments per subject. The resulting temporal and spatial activation patterns obtained from our matrix factorization technique resemble individual task profiles and known functional brain networks, which are either correlated with the task or spontaneously activating unrelated to the task.

We present a deep semi-nonnegative matrix factorization method for identifying subject-specific functional networks (FNs) at multiple spatial scales with a hierarchical organization from resting state fMRI data. Our method is built upon a deep semi-nonnegative matrix factorization framework to jointly detect the FNs at multiple scales with a hierarchical organization, enhanced by group sparsity regularization that helps identify subject-specific FNs without loss of inter-subject comparability. The proposed method has been validated for predicting subject-specific functional activations based on functional connectivity measures of the hierarchical multi-scale FNs of the same subjects. Experimental results have demonstrated that our method could obtain subject-specific multi-scale hierarchical FNs and their functional connectivity measures across different scales could better predict subject-specific functional activations than those obtained by alternative techniques.

Dynamic functional connectivity analysis provides valuable information for understanding brain functional activity underlying different cognitive processes. Besides sliding window based approaches, a variety of methods have been developed to automatically split the entire functional MRI scan into segments by detecting change points of functional signals to facilitate better characterization of temporally dynamic functional connectivity patterns. However, these methods are based on certain assumptions for the functional signals, such as Gaussian distribution, which are not necessarily suitable for the fMRI data. In this study, we develop a deep learning based framework for adaptively detecting temporally dynamic functional state transitions in a data-driven way without any explicit modeling assumptions, by leveraging recent advances in recurrent neural networks (RNNs) for sequence modeling. Particularly, we solve this problem in an anomaly detection framework with an assumption that the functional profile of one single time point could be reliably predicted based on its preceding profiles within a stable functional state, while large prediction errors would occur around change points of functional states. We evaluate the proposed method using both task and resting-state fMRI data obtained from the human connectome project and experimental results have demonstrated that the proposed change point detection method could effectively identify change points between different task events and split the resting-state fMRI into segments with distinct functional connectivity patterns.

In this study, a personalized computer aided diagnosis system for autism spectrum disorder is introduced. The proposed system uses resting state functional MRI data to build local classifiers, global classifier, and correlate the classification findings with ADOS behavioral reports. This system is composed of 3 main phases: (i) Data preprocessing to overcome the motion and timing artifacts and normalize the data to standard MNI152 space, (ii) using a small subset (40 subjects) to extract significant activation components, and (iii) utilize the extracted significant components to build a deep learning based diagnosis system for each component, combine the probabilities for global diagnosis and calculate the correlation with ADOS reports. The deep learning based classification system showed accuracies of more than 80% in the significant components, moreover, the global diagnosis accuracy is 93%. Out of the significant components, 2 components are found to be correlated with neuro-circuits involved in autism related impairments as reported in ADOS reports.

Brain functional connectivity (FC) extracted from resting-state fMRI (RS-fMRI) has become a popular approach for disease diagnosis, where discriminating subjects with mild cognitive impairment (MCI) from normal controls (NC) is still one of the most challenging problems. Dynamic functional connectivity (dFC), consisting of time-varying spatiotemporal dynamics, may characterize “chronnectome” diagnostic information for improving MCI classification. However, most of the current dFC studies are based on detecting discrete major “brain status” via spatial clustering, which ignores rich spatiotemporal dynamics contained in such chronnectome. We propose Deep Chronnectome Learning for exhaustively mining the comprehensive information, especially the hidden higher-level features, i.e., the dFC time series that may add critical diagnostic power for MCI classification. To this end, we devise a new Fully-connected bidirectional Long Short-Term Memory (LSTM) network (Full-BiLSTM) to effectively learn the periodic brain status changes using both past and future information for each brief time segment and then fuse them to form the final output. We have applied our method to a rigorously built large-scale multi-site database (i.e., with 164 data from NCs and 330 from MCIs, which can be further augmented by 25 folds). Our method outperforms other state-of-the-art approaches with an accuracy of 73.6% under solid cross-validations. We also made extensive comparisons among multiple variants of LSTM models. The results suggest high feasibility of our method with promising value also for other brain disorder diagnoses.

Machine learning-based accurate diagnosis of psychiatric disorders is expected to find their biomarkers and to evaluate the treatments. For this purpose, neuroimaging datasets have required special procedures including feature-selections and dimensional-reductions since they are still composed of a limited number of high-dimensional samples. Recent studies reported a certain success by applying generative models to fMRI data. Generative models can classify small datasets more accurately than discriminative models as long as their assumptions are appropriate. Leveraging our prior knowledge of fMRI signal and the flexibility of deep neural networks, we propose a structured deep generative model, which takes into account fMRI images, disorder, and individual variability. The proposed model estimates the subjects’ conditions more accurately than existing diagnostic procedures, general discriminative models, and recently-proposed generative models. Also, it identifies brain regions related to the disorders.

Previous studies [1, 2] have shown that slow variations in the cardiac cycle are coupled with signal changes in the blood-oxygen level dependent (BOLD) contrast. The detection of neurophysiological hemodynamic changes, driven by neuronal activity, is hampered by such physiological noise. It is therefore of great importance to model and remove these physiological artifacts. The cardiac cycle causes pulsatile arterial blood flow. This pulsation is translated into brain tissue and fluids bounded by the cranial cavity [3]. We exploit this pulsality effect in BOLD fMRI volumes to build a reliable cardio surrogate estimate. We propose a Gaussian Process (GP) heart rate model to build physiological noise regressors for the General Linear Model (GLM) used in fMRI analysis. The proposed model can also incorporate information from physiological recordings such as photoplethysmogram or electrocardiogram, and is able to learn the temporal interdependence of individual modalities.

In analyzing rs-fMRI, blind source separation has been studied extensively and various machine-learning techniques have been proposed in the literature. However, to our best knowledge, most of the existing methods do not explicitly separate noise components that naturally corrupt the observed BOLD signals, thus hindering from the understanding of underlying functional mechanisms in a human brain. In this paper, we formulate the problem of latent source separation in a probabilistic manner, where we explicitly separate the observed signals into a true source signal and a noise component. As for the inference of the latent source distribution with respect to an input regional mean signal, we use a stochastic variational Bayesian inference and implement it in a neural network framework. Further, in order for identification of a subject with early mild cognitive impairment (eMCI) rs-fMRI, we also propose to use the relations of the inferred source signals as features, i.e., potential imaging-biomarkers. We presented the validity of the proposed methods by conducting experiments on the publicly available ADNI2 dataset and comparing with the existing methods.

For decades, task-based functional magnetic resonance imaging (tfMRI) has been a powerful noninvasive tool to explore the organizational architecture of human brain function. Researchers have developed a variety of brain network analysis methods for tfMRI data, including the general linear model (GLM), independent component analysis (ICA) and sparse representation methods. However, these shallow models are limited in faithful reconstruction and modeling of the hierarchical and temporal structures of brain networks, as demonstrated in more and more studies. Recently, recurrent neural networks (RNNs) exhibit great ability of modeling hierarchical and temporal dependency features in the machine learning field, which might be suitable for tfMRI data modeling. To explore such possible advantages of RNNs for tfMRI data, we propose a novel framework of deep recurrent neural network (DRNN) to model the functional brain networks for tfMRI data. Experimental results on the motor task tfMRI data of Human Connectome Project 900 subjects data release demonstrated that the proposed DRNN can not only faithfully reconstruct functional brain networks, but also identify more meaningful brain networks with multiple time scales which are overlooked by traditional shallow models. In general, this work provides an effective and powerful approach to identifying functional brain networks of multiple time scales from tfMRI data.

Although alternations of brain functional networks (BFNs) derived from resting-state functional magnetic resonance imaging (rs-fMRI) have been considered as promising biomarkers for early Alzheimer’s disease (AD) diagnosis, it is still challenging to perform individualized diagnosis, especially at the very early stage of preclinical stage of AD, i.e., early mild cognitive impairment (eMCI). Recently, convolutional neural networks (CNNs) show powerful ability in computer vision and image analysis applications, but there is still a gap for directly applying CNNs to rs-fMRI-based disease diagnosis. In this paper, we propose a novel multiple-BFN-based 3D CNN framework that can automatically and deeply learn complex, high-level, hierarchical diagnostic features from various independent component analysis-derived BFNs. More importantly, the embedded features of different BFNs could comprehensively support each other towards a more accurate eMCI diagnosis in a unified model. The performance of the proposed method is validated by a large-sample, multisite, rigorously controlled publicly accessible dataset. The proposed framework can also be conveniently and straightforwardly applied to individualized diagnosis of various neurological and psychiatric diseases.

We study how human brains activate to process input information and execute necessary cognitive tasks. Understanding the process is crucial in improving our diagnostic and treatment of different neurological disorders. Given functional MRI images recorded when human subjects execute tasks with different levels of information uncertainty, we need to identify the similarity and difference between brain activities at different regions of interest (ROIs), and thus gain insights into the underlying mechanism. To achieve this goal, we propose a new ROI-reweight 3D convolutional neural network (CNN). Our CNN not only learns to classify the task-evoked fMRIs with a high accuracy, but also locates crucial ROIs based on a reweight layer. Our findings reveal several brain regions to be crucial in differentiating brain activity patterns facing tasks of different uncertainty levels.

This work introduces a novel framework for the deconvolution of the BOLD signal in multi-echo functional MRI (ME-fMRI) data: Multi-echo Sparse Paradigm Free Mapping (ME-SPFM). Building upon a physical model of the ME-fMRI signal and using a sparsity-promoting regularized least squares estimator, this algorithm obtains time-varying maps of the changes in the transverse relaxation rate ( $$ R_{2}^{*} $$ R 2 ∗ ) and the net magnetization ( $$ S_{0} $$ S 0 ) of the signal at the subject level without prior knowledge of the timing of the individual BOLD events. Our results with experimental data demonstrate that the maps of $$ R_{2}^{*} $$ R 2 ∗ changes obtained with ME-SPFM at the times of the stimulus trials closely resemble the maps obtained with standard model-based methods that are aware of the onsets and durations of the experimental events, and considerably improves the accuracy of the deconvolution compared with the SPFM algorithm developed for single-echo fMRI. Furthermore, this method yields estimates of $$ R_{2}^{*} $$ R 2 ∗ changes in physiologically interpretable units (s−1), which is a step towards deciphering the dynamic nature of brain activity in a pseudo-quantitative manner in naturalistic paradigms, resting-state or clinical applications with unknown event-timing.

Decoding brain functional states underlying different cognitive processes using multivariate pattern recognition techniques has attracted increasing interests in brain imaging studies. Promising performance has been achieved using brain functional connectivity or brain activation signatures for a variety of brain decoding tasks. However, most of existing studies have built decoding models upon features extracted from imaging data at individual time points or temporal windows with a fixed interval, which might not be optimal across different cognitive processes due to varying temporal durations and dependency of different cognitive processes. In this study, we develop a deep learning based framework for brain decoding by leveraging recent advances in sequence modeling using long short-term memory (LSTM) recurrent neural networks (RNNs). Particularly, functional profiles extracted from task functional imaging data based on their corresponding subject-specific intrinsic functional networks are used as features to build brain decoding models, and LSTM RNNs are adopted to learn decoding mappings between functional profiles and brain states. We evaluate the proposed method using task fMRI data from the HCP dataset, and experimental results have demonstrated that the proposed method could effectively distinguish brain states under different task events and obtain higher accuracy than conventional decoding models.

Deep learning has become the new state-of-the-art for many problems in image analysis. However, large datasets are often required for such deep networks to learn effectively. This poses a difficult challenge for many medical image analysis problems in which only a small number of subjects are available, e.g., patients undergoing a new treatment. In this work, we propose a number of approaches for learning generalizable recurrent neural networks from smaller task-fMRI datasets: (1) a resampling method for ROI-based fMRI analysis to create augmented data; (2) inclusion of a small number of non-imaging variables to provide subject-specific initialization of the recurrent neural network; and (3) selection of the most generalizable model from multiple reinitialized training runs using criteria based on only training loss. Using cross-validation to assess model performance, we demonstrate the effectiveness of the proposed methods to train recurrent neural networks from small datasets to predict treatment outcome for children with autism spectrum disorder ( $$N=21$$ N = 21 ) and classify autistic vs. typical control subjects ( $$N=40$$ N = 40 ) from task-fMRI scans.

The resection of brain tumors beneath eloquent areas of the human brain requires precise delineation of eloquent areas for maximum removal of tumor mass while minimizing the risk for postoperative functional deficits. Non-invasive mapping of eloquent areas can be carried out by intraoperative thermal imaging since neural activity alters the cortical temperature distribution. These characteristic changes in cortical temperature can be modeled by a response function. A prominent choice for this response function is the haemodynamic response function. However, the signal is typically superimposed by various effects such as motion artifacts, physiological effects, sensor drifts as well as autoregulation which have to be compensated.In this paper, we contribute a regularized semiparametric regression framework that recognizes the response function while it compensates for arbitrary background signals. We achieve this by learning a tailored L2 penalty that basically regularizes the estimated background signal such that it doesn’t comprise the characteristics of the response function. The evaluation of this approach is carried out by augmented semisynthetic resting-state- as well as intraoperative thermal imaging data.

Generating models from functional connectivity data that predict behavioral measures holds great clinical potential. While the majority of the literature has focused on using only connectivity data from a single source, there is ample evidence that different cognitive conditions amplify individual differences in functional connectivity in a distinct, complementary manner. In this work, we introduce a computational model, labeled multidimensional Connectome-based Predictive Modeling (mCPM), that combines connectivity matrices collected from different task conditions in order to improve behavioral prediction by using complementary information found in different cognitive tasks. We apply our algorithm to data from the Human Connectome Project and UCLA Consortium for Neuropsychiatric Phenomics (CNP) LA5c Study. Using data from multiple tasks, mCPM generated models that better predicted IQ than models generated from any single task. Our results suggest that prediction of behavior can be improved by including multiple task conditions in computational models, that different tasks provide complementary information for prediction, and that mCPM provides a principled method for modeling such data.

Studying human brain connectome has been an important, yet challenging problem due to the intrinsic complexity of the brain function and structure. Many studies have been done to map the brain connectome, like Human Connectome Project (HCP). However, multi-modality (DTI and fMRI) brain connectome analysis is still under-studied. One challenge is the lack of a framework to efficiently link different modalities together. In this paper, we integrate two research efforts including sparse dictionary learning derived functional networks and structural connectivity into a joint representation of brain connectome. This joint representation then guided the identification of the main skeletons of whole-brain fiber connections, which contributes to a better understanding of brain architecture of structural connectome and its local pathways. We applied our framework on the HCP multimodal DTI/fMRI data and successfully constructed the main skeleton of whole-brain fiber connections. We identified 14 local fiber skeletons that are functionally and structurally consistent across individual brains.

Modern resting-state functional magnetic resonance imaging (rs-fMRI) provides a wealth of information about the inherent functional connectivity of the human brain. However, understanding the role of negative correlations and the nonlinear topology of rs-fMRI remains a challenge. To address these challenges, we propose a novel graph embedding technique, phase angle spatial embedding (PhASE), to study the “intrinsic geometry” of the functional connectome. PhASE both incorporates negative correlations as well as reformulates the connectome modularity problem as a kernel two-sample test, using a kernel method that induces a maximum mean discrepancy (MMD) in a reproducing kernel Hilbert space (RKHS). By solving a graph partition that maximizes this MMD, PhASE identifies the most functionally distinct brain modules. As a test case, we analyzed a public rs-fMRI dataset to compare male and female connectomes using PhASE and minimum spanning tree inferential statistics. These results show statistically significant differences between male and female resting-state brain networks, demonstrating PhASE to be a robust tool for connectome analysis.

In this work, we propose an edema-informed anatomically constrained tractography paradigm that enables reconstructing larger spatial extent of white matter bundles as well as increased cortical coverage in the presence of edema. These improvements will help surgeons maximize the extent of the resection while minimizing the risk of cognitive deficits. The new paradigm is based on a segmentation of the brain into gray matter, white matter, corticospinal fluid, edema and tumor regions which utilizes a tumor growth model. Using this segmentation, a valid tracking domain is generated and, in combination with anatomically constrained particle filter tractography, allows streamlines to cross the edema region and reach the cortex. Using subjects with brain tumors, we show that our edema-informed anatomically constrained tractography paradigm increases the cortico-cortical connections that cross edema-contaminated regions when compared to traditional fractional anisotropy thresholded tracking.

The thalamus is a deep grey matter structure that plays an important role in propagating nerve impulses between subcortical regions and the cerebral cortex. It is composed of distinct nuclei that have unique long-range connectivity. Accurate thalamic nuclei segmentation provides insights about structural connectivity and the neurodegeneration mechanisms occurring in distinct brain disorders, for instance Alzheimer’s disease and Frontotemporal dementia (FTD). In this work, we propose a novel thalamic nuclei segmentation approach that relies on tractography, thalamic nuclei priors and local fibre orientation. Validation was performed in a cohort of healthy controls and FTD patients against other thalamus connectivity-based parcellation methods. Results showed that the proposed strategy led to anatomical plausible thalamic nuclei segmentations and was able to detect connectivity differences between controls and FTD patients.

In diffusion MRI, fiber tracts, represented by densely distributed 3D curves, can be estimated from diffusion weighted images using tractography. The spatial geometric structure of white matter fiber tracts is known to be complex in human brain, but it carries intrinsic information of human brain. In this paper, inspired by studies of liquid crystals, we propose tract-based director field analysis (tDFA) with total six rotationally invariant scalar indices to quantify local geometric structures of fiber tracts. The contributions of tDFA include: (1) We propose orientational order (OO) and orientational dispersion (OD) indices to quantify the degree of alignment and dispersion of fiber tracts; (2) We define the local orthogonal frame for a set of unoriented curves, which is proved to be a generalization of the Frenet frame defined for a single oriented curve; (3) With the local orthogonal frame, we propose splay, bend, and twist indices to quantify three types of orientational distortion of local fiber tracts, and a total distortion index to describe distortions of all three types. The proposed tDFA for fiber tracts is a generalization of the voxel-based DFA (vDFA) which was recently proposed for a spherical function field (i.e., an ODF field). To our knowledge, this is the first work to quantify orientational distortion (splay, bend, twist, and total distortion) of fiber tracts. Experiments show that the proposed scalar indices are useful descriptors of local geometric structures to visualize and analyze fiber tracts.

Image voxel/vertex-wise feature in the brain is widely used for automatic classification or significant region detection of various dementia syndromes. In these studies, the non-imaging variables, such as age, will affect the results, but may be uninterested to the clinical applications. Imaging data can be considered as a combination of the confound variable (e.g. age) and the variable of clinical interest (e.g. AD diagnosis). However, non-imaging confound variable is not well dealt in each voxel. In this paper, we proposed a spatial-anatomical regularized parametric function fitting approach that explicitly modeling the relationship between the voxelwise feature and the confound variable. By adding the spatial-anatomical regularization, our model not only obtains a better voxelwise feature estimation, but also generates a more interpretable parameter map to help understand the effect of confound variable on imaging features. Besides the commonly used linear model, we also develop a spatial-anatomical regularized voxelwise general logistic model to investigate deeper of the aging process in gray matter and white matter density map.

Autism spectrum disorder (ASD) is mainly diagnosed by the observation of core behavioral symptoms. Due to the absence of early biomarkers to detect infants either with or at-risk of ASD during the first postnatal year of life, diagnosis must rely on behavioral observations long after birth. As a result, the window of opportunity for effective intervention may have passed when the disorder is detected. Therefore, it is clinically urgent to identify imaging-based biomarkers for early diagnosis and intervention. In this paper, for the first time, we proposed a volume-based analysis of infant subjects with risk of ASD at very early age, i.e., as early as at 6 months of age. A critical part of volume-based analysis is to accurately segment 6-month-old infant brain MRI scans into different regions of interest, e.g., white matter, gray matter, and cerebrospinal fluid. This is actually very challenging since the tissue contrast at 6-month-old is extremely low, caused by inherent ongoing myelination and maturation. To address this challenge, we propose an anatomy-guided, densely-connected network for accurate tissue segmentation. Based on tissue segmentations, we further perform brain parcellation and statistical analysis to identify those significantly different regions between autistic and normal subjects. Experimental results on National Database for Autism Research (NDAR) show the advantages of our proposed method in terms of both segmentation accuracy and diagnosis accuracy over state-of-the-art results.

Cortical thickness analysis of brain magnetic resonance images is an important technique in neuroimaging research. There are two main computational paradigms, namely voxel-based and surface-based methods. Recently, a tetrahedron-based volumetric morphometry (TBVM) approach involving proper discretization methods was proposed. The multi-scale and physics-based geometric features generated through such methods may yield stronger statistical power. However, several challenges, such as the lack of well-defined thickness statistics and the difficulty in filling tetrahedrons into the thin and curvy cortex structure, impede the broad application of TBVM. In this paper, we present a universal cortical thickness morphometry analysis approach called tetrahedron-based Heat Flux Signature (tHFS) to address these challenges. We define the tetrahedron-based weak form heat equation and Laplace-Beltrami eigen decomposition and give an explicit FEM-based discretization formulation to compute the tHFS. We further show a tHFS metric space with which cortical morphometric distances can be directly visualized. Additionally, we optimize the cortical tetrahedral mesh generation pipeline and fill dense high-quality tetrahedra in the grey matters without sacrificing data integrity. Compared with existing cortical thickness analysis approaches, our experimental results of distinguishing among Alzheimer’s disease (AD), cognitively normal (CN) and mild cognitive impairment (MCI) subjects shows that tHFS yields a more accurate representation of cortical thickness morphometry. The tHFS metric experiment provides a more vivid visualization of tHFS’s power in separating different clinical groups.

Alzheimer’s disease is the most common dementia leading to an irreversible neurodegenerative process. To date, subject revealed advanced brain structural alterations when the diagnosis is established. Therefore, an earlier diagnosis of this dementia is crucial although it is a challenging task. Recently, many studies have proposed biomarkers to perform early detection of Alzheimer’s disease. Some of them have proposed methods based on inter-subject similarity while other approaches have investigated framework using intra-subject variability. In this work, we propose a novel framework combining both approaches within an efficient graph of brain structures grading. Subsequently, we demonstrate the competitive performance of the proposed method compared to state-of-the-art methods.

Despite the large body of existing neuroimaging-based studies on brain dementia, in particular mild cognitive impairment (MCI), modeling and predicting the early dynamics of dementia onset and development in healthy brains is somewhat overlooked in the literature. The majority of computer-aided diagnosis tools developed for classifying healthy and demented brains mainly rely on either using single timepoint or longitudinal neuroimaging data. Longitudinal brain imaging data offer a larger time window to better capture subtle brain changes in early MCI development, and its utilization has been shown to improve classification and prediction results. However, typical longitudinal studies are challenged by a limited number of acquisition timepoints and the absence of inter-subject matching between timepoints. To address this limitation, we propose a novel framework that learns how to predict the developmental trajectory of a brain image from a single acquisition timepoint (i.e., baseline), while classifying the predicted trajectory as ‘healthy’ or ‘demented’. To do so, we first rigidly align all training images, then extract ‘landmark patches’ from training images. Next, to predict the patch-wise trajectory evolution from baseline patch, we propose two novel strategies. The first strategy learns in a supervised manner to select a few training atlas patches that best boost the classification accuracy of the target testing patch. The second strategy learns in an unsupervised manner to select the set of most similar training atlas patches to the target testing patch using multi-kernel patch manifold learning. Finally, we train a linear classifier for each predicted patch trajectory. To identify the final label of the target subject, we use majority voting to aggregate the labels assigned by our model to all landmark patches’ trajectories. Our image prediction model boosted the classification performance by 14% point without further leveraging any enhancing methods such as feature selection.

In Alzheimer’s research, Mild Cognitive Impairment (MCI) is an important intermediate stage between normal aging and Alzheimer’s. How to distinguish MCI samples that finally convert to AD from those do not is an essential problem in the prevention and diagnosis of Alzheimer’s. Traditional methods use various classification models to distinguish MCI converters from non-converters, while the performance is usually limited by the small number of available data. Moreover, previous methods only use the data at baseline time for training but ignore the longitudinal information at other time points along the disease progression. To tackle with these problems, we propose a novel deep learning framework that uncovers the temporal correlation structure between adjacent time points in the disease progression. We also construct a generative framework to learn the inherent data distribution so as to produce more reliable data to strengthen the training process. Extensive experiments on the ADNI cohort validate the superiority of our model.

Multi-modal neuroimages (e.g., MRI and PET) have been widely used for diagnosis of brain diseases such as Alzheimer’s disease (AD) by providing complementary information. However, in practice, it is unavoidable to have missing data, i.e., missing PET data for many subjects in the ADNI dataset. A straightforward strategy to tackle this challenge is to simply discard subjects with missing PET, but this will significantly reduce the number of training subjects for learning reliable diagnostic models. On the other hand, since different modalities (i.e., MRI and PET) were acquired from the same subject, there often exist underlying relevance between different modalities. Accordingly, we propose a two-stage deep learning framework for AD diagnosis using both MRI and PET data. Specifically, in the first stage, we impute missing PET data based on their corresponding MRI data by using 3D Cycle-consistent Generative Adversarial Networks (3D-cGAN) to capture their underlying relationship. In the second stage, with the complete MRI and PET (i.e., after imputation for the case of missing PET), we develop a deep multi-instance neural network for AD diagnosis and also mild cognitive impairment (MCI) conversion prediction. Experimental results on subjects from ADNI demonstrate that our synthesized PET images with 3D-cGAN are reasonable, and also our two-stage deep learning method outperforms the state-of-the-art methods in AD diagnosis.

The plethora of data from neuroimaging studies provide a rich opportunity to discover effects and generate hypotheses through exploratory data analysis. Brain pathologies often manifest in changes in shape along with deterioration and alteration of brain matter, i.e., changes in mass. We propose a morphometry approach using unbalanced optimal transport that detects and localizes changes in mass and separates them from changes due to the location of mass. The approach generates images of mass allocation and mass transport cost for each subject in the population. Voxelwise correlations with clinical variables highlight regions of mass allocation or mass transfer related to the variables. We demonstrate the method on the white and gray matter segmentations from the OASIS brain MRI data set. The separation of white and gray matter ensures that optimal transport does not transfer mass between different tissues types and separates gray and white matter related changes. The OASIS data set includes subjects ranging from healthy to mild and moderate dementia, and the results corroborate known pathology changes related to dementia that are not discovered with traditional voxel-based morphometry. The transport-based morphometry increases the explanatory power of regression on clinical variables compared to traditional voxel-based morphometry, indicating that transport cost and mass allocation images capture a larger portion of pathology induced changes.

Machine learning classifiers are frequently trained on heterogeneous multi-modal imaging data, where some patients have missing modalities. We address the problem of synthesising arterial spin labelling magnetic resonance imaging (ASL-MRI) - derived cerebral blood flow (CBF) - features in a heterogeneous data set. We synthesise ASL-MRI features using T1-weighted structural MRI (sMRI) and carotid ultrasound flow features. To deal with heterogeneous data, we extend the kernel partial least squares regression (kPLSR) - method to the case where both input and output data have partial coverage. The utility of the synthetic CBF features is tested on a binary classification problem of mild cognitive impairment patients vs. controls. Classifiers based on sMRI and synthetic ASL-MRI features are combined using a maximum probability rule, achieving a balanced accuracy of 92% (sensitivity 100 %, specificity 80 %) in a separate validation set. Comparison is made against support vector machine-classifiers from literature.

We propose a novel Coupled Hidden Markov Model to detect epileptic seizures in multichannel electroencephalography (EEG) data. Our model defines a network of seizure propagation paths to capture both the temporal and spatial evolution of epileptic activity. To address the intractability introduced by the coupled interactions, we derive a variational inference procedure to efficiently infer the seizure evolution from spectral patterns in the EEG data. We validate our model on EEG aquired under clinical conditions in the Epilepsy Monitoring Unit of the Johns Hopkins Hospital. Using 5-fold cross validation, we demonstrate that our model outperforms three baseline approaches which rely on a classical detection framework. Our model also demonstrates the potential to localize seizure onset zones in focal epilepsy.

Focal cortical dysplasia (FCD) is a prevalent surgically-amenable epileptogenic malformation of cortical development. On MRI, FCD typically presents with cortical thickening, hyperintensity, and blurring of the gray-white matter interface. These changes may be visible to the naked eye, or subtle and be easily overlooked. Despite advances in MRI analytics, current surface-based algorithms fail to detect FCD in 50% of cases. Moreover, arduous data pre-processing and specialized expertise preclude widespread use. Here we propose a novel algorithm that harnesses feature-learning capability of convolutional neural networks (CNNs) with minimal data pre-processing. Our classifier, trained on a patch-based augmented dataset derived from patients with histologically-validated FCD operates directly on MRI voxels to distinguish the lesion from healthy tissue. The algorithm was trained and cross-validated on multimodal MRI data from a single site (S1) and evaluated on independent data from S1 and six other sites worldwide (S2–S7; 3 scanner manufacturers and 2 field strengths) for a total of 107 subjects. The classifier showed excellent sensitivity (S1: 87%, 35/40 lesions detected; S2–S7: 91%, 61/67 lesions detected) and specificity (S1: 95%, no findings in 36/38 healthy controls; 90%, no findings in 57/63 disease controls). Easy implementation, minimal pre-processing, high performance and generalizability make this classifier an ideal platform for large-scale clinical use, particularly in “MRI-negative” FCD.

Determining the malignancy of glioma is highly important for initial therapy planning. In current clinical practice, often a biopsy is performed to verify tumour grade which involves risks and can negatively impact overall survival. To avoid biopsy, non-invasive tumour characterisation based on MRI is preferred and to improve accuracy and efficiency, the use of computer-aided diagnosis (CAD) systems is investigated. Existing radiomics CAD techniques often rely on manual segmentation and are trained and evaluated on data from one clinical centre. Therefore, there is a need for accurate and automatic CAD systems that are robust to large variations in imaging protocols between different institutions. In this study, we extract features from T1ce MRI with a pre-trained CNN and compare their predictive power with hand-engineered radiomics features for binary grade prediction. Performance was evaluated on the BRATS 2017 database containing MRI and manual segmentation data of 285 patients from multiple institutions. State-of-the-art performance with an AUC of $$96.4\%$$ 96.4 % was achieved with radiomics features extracted from manually segmented tumour volumes. Pre-trained CNN features had a strong predictive value as well and an AUC score of $$93.5\%$$ 93.5 % could be obtained when propagating the tumour region of interest (ROI). Additionally, using a pre-trained CNN as feature extractor, we were able to design an accurate, automatic, fast and robust binary glioma grading system achieving an AUC score of $$91.1\%$$ 91.1 % without requiring ROI annotations.

We propose a novel approach named one-time sampling irregularity age map (OTS-IAM) to detect any irregular texture in FLAIR brain MRI without any human supervision or interaction. In this study, we show that OTS-IAM is able to detect FLAIR’s brain tissue irregularities (i.e. hyperintensities) without any manual labelling. One-time sampling (OTS) scheme is proposed in this study to speed up the computation. The proposed OTS-IAM implementation on GPU successfully speeds up IAM’s computation by more than 17 times. We compared the performance of OTS-IAM with two unsupervised methods for hyperintensities’ detection; the original IAM and the Lesion Growth Algorithm from public toolbox Lesion Segmentation Toolbox (LST-LGA), and two conventional supervised machine learning algorithms; support vector machine (SVM) and random forest (RF). Furthermore, we also compared OTS-IAM’s performance with three supervised deep neural networks algorithms; Deep Boltzmann machine (DBM), convolutional encoder network (CEN) and 2D convolutional neural network (2D Patch-CNN). Based on our experiments, OTS-IAM outperformed LST-LGA, SVM, RF and DBM while it was on par with CEN.

Multiple sclerosis (MS) is a demyelinating disease of the central nervous system (CNS). A reliable measure of the tissue myelin content is therefore essential to understand the physiopathology of MS, track progression and assess treatment efficacy. Positron emission tomography (PET) with $$[^{11} \text {C}] \text {PIB}$$ [ 11 C ] PIB has been proposed as a promising biomarker for measuring myelin content changes in-vivo in MS. However, PET imaging is expensive and invasive due to the injection of a radioactive tracer. On the contrary, magnetic resonance imaging (MRI) is a non-invasive, widely available technique, but existing MRI sequences do not provide, to date, a reliable, specific, or direct marker of either demyelination or remyelination. In this work, we therefore propose Sketcher-Refiner Generative Adversarial Networks (GANs) with specifically designed adversarial loss functions to predict the PET-derived myelin content map from a combination of MRI modalities. The prediction problem is solved by a sketch-refinement process in which the sketcher generates the preliminary anatomical and physiological information and the refiner refines and generates images reflecting the tissue myelin content in the human brain. We evaluated the ability of our method to predict myelin content at both global and voxel-wise levels. The evaluation results show that the demyelination in lesion regions and myelin content in normal-appearing white matter (NAWM) can be well predicted by our method. The method has the potential to become a useful tool for clinical management of patients with MS.

We propose a deep neural network for supervised learning on neuroanatomical shapes. The network directly operates on raw point clouds without the need for mesh processing or the identification of point correspondences, as spatial transformer networks map the data to a canonical space. Instead of relying on hand-crafted shape descriptors, an optimal representation is learned in the end-to-end training stage of the network. The proposed network consists of multiple branches, so that features for multiple structures are learned simultaneously. We demonstrate the performance of our method on two applications: (i) the prediction of Alzheimer’s disease and mild cognitive impairment and (ii) the regression of the brain age. Finally, we visualize the important parts of the anatomy for the prediction by adapting the occlusion method to point clouds.

Most neurodegenerative diseases are caused by pathogenic proteins. Pathogenic protein behaviour is governed by neurobiological mechanisms which cause them to spread and accumulate in the brain, leading to cellular death and eventually atrophy. Patient data suggests atrophy loosely follows a number of spatiotemporal patterns, with different patterns associated with each neurodegenerative disease variant. It is hypothesised that the behaviour of different pathogenic protein variants is governed by different mechanisms, which could explain the pattern variety. Machine learning approaches take advantage of the pattern predictability for differential diagnosis and prognosis, but are unable to reveal new information on the underlying mechanisms, which are still poorly understood. We propose a framework where computational models of these mechanisms were created based on neurobiological literature. Competing hypotheses regarding the mechanisms were modelled and the outcomes evaluated against empirical data of Alzheimer’s disease. With this approach, we are able to characterise the impact of each mechanism on the neurodegenerative process. We also demonstrate how our framework could evaluate candidate therapies.

This paper presents a general framework for obtaining interpretable multivariate discriminative models that allow efficient statistical inference for neuroimage analysis. The framework, termed generative discriminative machine (GDM), augments discriminative models with a generative regularization term. We demonstrate that the proposed formulation can be optimized in closed form and in dual space, allowing efficient computation for high dimensional neuroimaging datasets. Furthermore, we provide an analytic estimation of the null distribution of the model parameters, which enables efficient statistical inference and p-value computation without the need for permutation testing. We compared the proposed method with both purely generative and discriminative learning methods in two large structural magnetic resonance imaging (sMRI) datasets of Alzheimer’s disease (AD) (n = 415) and Schizophrenia (n = 853). Using the AD dataset, we demonstrated the ability of GDM to robustly handle confounding variations. Using Schizophrenia dataset, we demonstrated the ability of GDM to handle multi-site studies. Taken together, the results underline the potential of the proposed approach for neuroimaging analyses.

Automatic computation of cortical thickness is a critical step when investigating neuroanatomical population differences and changes associated with normal development and aging, as well as in neuro-degenerative diseases including Alzheimer’s and Parkinson’s. Limited spatial resolution and partial volume effects, in which more than one tissue type is represented in each voxel, have a significant impact on the accuracy of thickness estimates, particularly if a hard intensity threshold is used to delineate cortical boundaries. We describe a novel method based on the anisotropic heat equation that explicitly accounts for the presence of partial tissue volumes to more accurately estimate cortical thickness. The anisotropic term uses gray matter fractions to incorporate partial tissue voxels into the thickness calculation, as demonstrated through simulations and experiments. We also show that the proposed method is robust to the effects of finite voxel resolution and blurring. In comparison to methods based on hard intensity thresholds, the heat equation based method yields results with in-vivo data that are more consistent with histological findings reported in the literature. We also performed a test-retest study across scanners that indicated improved consistency and robustness to scanner differences.

Ultrasound (US) can be used to assess brain development in newborns, as MRI is challenging due to immobilization issues, and may require sedation. Dilatation of the lateral ventricles in the brain is a risk factor for poorer neurodevelopment outcomes in infants. Hence, 3D US has the ability to assess the volume of the lateral ventricles similar to clinically standard MRI, but manual segmentation is time consuming. The objective of this study is to develop an approach quantifying the ratio of lateral ventricular dilatation with respect to total brain volume using 3D US, which can assess the severity of macrocephaly. Automatic segmentation of the lateral ventricles is achieved with a multi-atlas deformable registration approach using locally linear correlation metrics for US-MRI fusion, followed by a refinement step using deformable mesh models. Total brain volume is estimated using a 3D ellipsoid modeling approach. Validation was performed on a cohort of 12 infants, ranging from 2 to 8.5 months old, where 3D US and MRI were used to compare brain volumes and segmented lateral ventricles. Automatically extracted volumes from 3D US show a high correlation and no statistically significant difference when compared to ground truth measurements. Differences in volume ratios was $$6.0 \pm 4.8\%$$ 6.0 ± 4.8 % compared to MRI, while lateral ventricular segmentation yielded a mean Dice coefficient of $$70.8\pm 3.6\%$$ 70.8 ± 3.6 % and a mean absolute distance (MAD) of $$0.88\pm 0.2$$ 0.88 ± 0.2 mm, demonstrating the clinical benefit of this tool in paediatric ultrasound.

The neonatal brain cortex is marked with complex and high-convoluted morphology, that undergoes dramatic changes over the first year of postnatal development. A large body of existing research works investigating ‘the developing brain’ have focused on looking at changes in cortical morphology and charting the developmental trajectories of the cortex. However, the relationship between neonatal cortical morphology and its postnatal growth trajectory was poorly investigated. Notably, understanding the multi-scale shape-growth relationship may help identify early neurodevelopmental disorders that affect it. Here, we unprecedentedly explore the question: “Do cortices that look alike in shape at birth have similar kinetic growth patterns?”. To this aim, we propose to analyze shape-growth relationship at three different scales. On a global scale, we found that neonatal cortices similar in geometric closeness are significantly correlated with their postnatal overall growth dynamics from birth till 1-year-old ( $$r=0.27$$ r = 0.27 ). This finding was replicated when using shape similarity in morphology ( $$r=0.20$$ r = 0.20 ). On a local scale, for both hemispheres, 20% of cortical regions displayed a significant high correlation ( $$r > 0.4$$ r > 0.4 ) between their similarities in morphology and dynamics. On a connectional scale, we identified hubs of cortical regions that were consistently similar in morphology and developed similarly across subjects including the cingulate cortex using a novel integral shape-growth brain graph representation.

Study of the untoward effects associated with the comorbidity of multiple diseases on brain morphology requires identifying differences across multiple diagnostic groupings. To identify such effects and differentiate between groups of patients and normal subjects, conventional methods often compare each patient group with healthy subjects using binary or multi-class classifiers. However, testing inferences across multiple diagnostic groupings of complex disorders commonly yield inconclusive or conflicting findings when the classifier is confined to modeling two cohorts at a time or considers class labels mutually-exclusive (as in multi-class classifiers). These shortcomings are potentially caused by the difficulties associated with modeling compounding factors of diseases with these approaches. Multi-label classifiers, on the other hand, can appropriately model disease comorbidity, as each subject can be assigned to two or more labels. In this paper, we propose a multi-label transductive (MLT) method based on low-rank matrix completion that is able not only to classify the data into multiple labels but also to identify patterns from MRI data unique to each cohort. To evaluate the method, we use a dataset containing individuals with Alcohol Use Disorder (AUD) and human immunodeficiency virus (HIV) infection (specifically 244 healthy controls, 227 AUD, 70 HIV, and 61 AUD+HIV). On this dataset, our proposed method is more accurate in correctly labeling subjects than common approaches. Furthermore, our method identifies patterns specific to each disease and AUD+HIV comorbidity that shows that the comorbidity is characterized by a compounding effect of AUD and HIV infection.

Despite the digital nature of magnetic resonance imaging, the resulting observations are most frequently reported and stored in text documents. There is a trove of information untapped in medical health records, case reports, and medical publications. In this paper, we propose to mine brain medical publications to learn the spatial distribution associated with anatomical terms. The problem is formulated in terms of minimization of a risk on distributions which leads to a least-deviation cost function. An efficient algorithm in the dual then learns the mapping from documents to brain structures. Empirical results using coordinates extracted from the brain-imaging literature show that (i) models must adapt to semantic variation in the terms used to describe a given anatomical structure, (ii) voxel-wise parameterization leads to higher likelihood of locations reported in unseen documents, (iii) least-deviation cost outperforms least-square. As a proof of concept for our method, we use our model of spatial distributions to predict the distribution of specific neurological conditions from text-only reports.

Image segmentation based on convolutional neural networks is proving to be a powerful and efficient solution for medical applications. However, the lack of annotated data, presence of artifacts and variability in appearance can still result in inconsistencies during the inference. We choose to take advantage of the invariant nature of anatomical structures, by enforcing a semantic constraint to improve the robustness of the segmentation. The proposed solution is applied on a brain structures segmentation task, where the output of the network is constrained to satisfy a known adjacency graph of the brain regions. This criteria is introduced during the training through an original penalization loss named NonAdjLoss. With the help of a new metric, we show that the proposed approach significantly reduces abnormalities produced during the segmentation. Additionally, we demonstrate that our framework can be used in a semi-supervised way, opening a path to better generalization to unseen data.

We propose the autofocus convolutional layer for semantic segmentation with the objective of enhancing the capabilities of neural networks for multi-scale processing. Autofocus layers adaptively change the size of the effective receptive field based on the processed context to generate more powerful features. This is achieved by parallelising multiple convolutional layers with different dilation rates, combined by an attention mechanism that learns to focus on the optimal scales driven by context. By sharing the weights of the parallel convolutions we make the network scale-invariant, with only a modest increase in the number of parameters. The proposed autofocus layer can be easily integrated into existing networks to improve a model’s representational power. We evaluate our mod els on the challenging tasks of multi-organ segmentation in pelvic CT and brain tumor segmentation in MRI and achieve very promising performance.

With the introduction of fully convolutional neural networks, deep learning has raised the benchmark for medical image segmentation on both speed and accuracy, and different networks have been proposed for 2D and 3D segmentation with promising results. Nevertheless, most networks only handle relatively small numbers of labels (<10), and there are very limited works on handling highly unbalanced object sizes especially in 3D segmentation. In this paper, we propose a network architecture and the corresponding loss function which improve segmentation of very small structures. By combining skip connections and deep supervision with respect to the computational feasibility of 3D segmentation, we propose a fast converging and computationally efficient network architecture for accurate segmentation. Furthermore, inspired by the concept of focal loss, we propose an exponential logarithmic loss which balances the labels not only by their relative sizes but also by their segmentation difficulties. We achieve an average Dice coefficient of 82% on brain segmentation with 20 labels, with the ratio of the smallest to largest object sizes as 0.14%. Less than 100 epochs are required to reach such accuracy, and segmenting a $$128 \times 128 \times 128$$ 128 × 128 × 128 volume only takes around 0.4 s.

Fetal ventriculomegaly (VM) is a condition with dilation of one or both lateral ventricles, and is diagnosed as an atrial diameter larger than 10 mm. Evidence of altered cortical folding associated with VM has been shown in the literature. However, existing studies use a holistic approach (i.e., ventricle as a whole) based on diagnosis or ventricular volume, thus failing to reveal the spatially-heterogeneous association patterns between cortex and ventricle. To address this issue, we develop a novel method to identify spatially fine-scaled association maps between cortical development and VM by leveraging vertex-wise correlations between the growth patterns of both ventricular and cortical surfaces in terms of area expansion and curvature information. Our approach comprises multiple steps. In the first step, we define a joint graph Laplacian matrix using cortex-to-ventricle correlations. Next, we propose a spectral embedding of the cortex-to-ventricle graph into a common underlying space where their joint growth patterns are projected. More importantly, in the joint ventricle-cortex space, the vertices of associated regions from both cortical and ventricular surfaces would lie close to each other. In the final step, we perform clustering in the joint embedded space to identify associated sub-regions between cortex and ventricle. Using a dataset of 25 healthy fetuses and 23 fetuses with isolated non-severe VM within the age range of 26–29 gestational weeks, our results show that the proposed approach is able to reveal clinically relevant and meaningful regional associations.

Brain extraction is a fundamental step for most brain imaging studies. In this paper, we investigate the problem of skull stripping and propose complementary segmentation networks (CompNets) to accurately extract the brain from T1-weighted MRI scans, for both normal and pathological brain images. The proposed networks are designed in the framework of encoder-decoder networks and have two pathways to learn features from both the brain tissue and its complementary part located outside of the brain. The complementary pathway extracts the features in the non-brain region and leads to a robust solution to brain extraction from MRIs with pathologies, which do not exist in our training dataset. We demonstrate the effectiveness of our networks by evaluating them on the OASIS dataset, resulting in the state of the art performance under the two-fold cross-validation setting. Moreover, the robustness of our networks is verified by testing on images with introduced pathologies and by showing its invariance to unseen brain pathologies. In addition, our complementary network design is general and can be extended to address other image segmentation problems with better generalization.

The model cascade strategy that runs a series of deep models sequentially for coarse-to-fine medical image segmentation is becoming increasingly popular, as it effectively relieves the class imbalance problem. This strategy has achieved state-of-the-art performance in many segmentation applications but results in undesired system complexity and ignores correlation among deep models. In this paper, we propose a light and clean deep model that conducts brain tumor segmentation in a single-pass and solves the class imbalance problem better than model cascade. First, we decompose brain tumor segmentation into three different but related tasks and propose a multi-task deep model that trains them together to exploit their underlying correlation. Second, we design a curriculum learning-based training strategy that trains the above multi-task model more effectively. Third, we introduce a simple yet effective post-processing method that can further improve the segmentation performance significantly. The proposed methods are extensively evaluated on BRATS 2017 and BRATS 2015 datasets, ranking first on the BRATS 2015 test set and showing top performance among 60+ competing teams on the BRATS 2017 validation set.

Variational Level Set (VLS) has been a widely used method in medical segmentation. However, segmentation accuracy in the VLS method dramatically decreases when dealing with intervening factors such as lighting, shadows, colors, etc. Additionally, results are quite sensitive to initial settings and are highly dependent on the number of iterations. In order to address these limitations, the proposed method incorporates VLS into deep learning by defining a novel end-to-end trainable model called as Deep Recurrent Level Set (DRLS). The proposed DRLS consists of three layers, i.e., Convolutional layers, Deconvolutional layers with skip connections and LevelSet layers. Brain tumor segmentation is taken as an instant to illustrate the performance of the proposed DRLS. Convolutional layer learns visual representation of brain tumor at different scales. Brain tumors occupy a small portion of the image, thus, deconvolutional layers are designed with skip connections to obtain a high quality feature map. Level-Set Layer drives the contour towards the brain tumor. In each step, the Convolutional Layer is fed with the LevelSet map to obtain a brain tumor feature map. This in turn serves as input for the LevelSet layer in the next step. The experimental results have been obtained on BRATS2013, BRATS2015 and BRATS2017 datasets. The proposed DRLS model improves both computational time and segmentation accuracy when compared to the classic VLS-based method. Additionally, a fully end-to-end system DRLS achieves state-of-the-art segmentation on brain tumors.

Accurate automatic segmentation of brain anatomy from $$T_1$$ T 1 -weighted ( $$T_1$$ T 1 -w) magnetic resonance images (MRI) has been a computationally intensive bottleneck in neuroimaging pipelines, with state-of-the-art results obtained by unsupervised intensity modeling-based methods and multi-atlas registration and label fusion. With the advent of powerful supervised convolutional neural networks (CNN)-based learning algorithms, it is now possible to produce a high quality brain segmentation within seconds. However, the very supervised nature of these methods makes it difficult to generalize them on data different from what they have been trained on. Modern neuroimaging studies are necessarily multi-center initiatives with a wide variety of acquisition protocols. Despite stringent protocol harmonization practices, it is not possible to standardize the whole gamut of MRI imaging parameters across scanners, field strengths, receive coils etc., that affect image contrast. In this paper we propose a CNN-based segmentation algorithm that, in addition to being highly accurate and fast, is also resilient to variation in the input $$T_1$$ T 1 -w acquisition. Our approach relies on building approximate forward models of $$T_1$$ T 1 -w pulse sequences that produce a typical test image. We use the forward models to augment the training data with test data specific training examples. These augmented data can be used to update and/or build a more robust segmentation model that is more attuned to the test data imaging properties. Our method generates highly accurate, state-of-the-art segmentation results (overall Dice overlap = 0.94), within seconds and is consistent across a wide-range of protocols.

Cytoarchitectonic parcellations of the human brain serve as anatomical references in multimodal atlas frameworks. They are based on analysis of cell-body stained histological sections and the identification of borders between brain areas. The de-facto standard involves a semi-automatic, reproducible border detection, but does not scale with high-throughput imaging in large series of sections at microscopical resolution. Automatic parcellation, however, is extremely challenging due to high variation in the data, and the need for a large field of view at microscopic resolution. The performance of a recently proposed Convolutional Neural Network model that addresses this problem especially suffers from the naturally limited amount of expert annotations for training. To circumvent this limitation, we propose to pre-train neural networks on a self-supervised auxiliary task, predicting the 3D distance between two patches sampled from the same brain. Compared to a random initialization, fine-tuning from these networks results in significantly better segmentations. We show that the self-supervised model has implicitly learned to distinguish several cortical brain areas – a strong indicator that the proposed auxiliary task is appropriate for cytoarchitectonic mapping.

Automatic parcellation of infant cortical surfaces into anatomical regions of interest (ROIs) is of great importance in brain structural and functional analysis. Conventional cortical surface parcellation methods suffer from two main issues: (1) Cortical surface registration is needed for establishing the atlas-to-individual correspondences; (2) The mapping from cortical shape to the parcellation labels requires designing of specific hand-crafted features. To address these issues, in this paper, we propose a novel cortical surface parcellation method, which is free of surface registration and designing of hand-crafted features, based on deep convolutional neural network (DCNN). Our main idea is to formulate surface parcellation as a patch-wise classification problem. Briefly, we use DCNN to train a classifier, whose inputs are the local cortical surface patches with multi-channel cortical shape descriptors such as mean curvature, sulcal depth, and average convexity; while the outputs are the parcellation label probabilities of cortical vertices. To enable effective convolutional operation on the surface data, we project each spherical surface patch onto its intrinsic tangent plane by a geodesic-distance-preserving mapping. Then, after classification, we further adopt the graph cuts method to improve spatial consistency of the parcellation. We have validated our method based on 90 neonatal cortical surfaces with manual parcellations, showing superior accuracy and efficiency of our proposed method.

Cerebral infarct volume observed from follow-up non contrast CT (NCCT) scans is an important radiologic outcome measurement of the effectiveness of acute ischemic patient treatment. Post-treatment infarct typically includes ischemic infarct only. But in around 10% of ischemic stroke patients, intracerebral hemorrhage is present along with ischemic infarction. Post-treatment infarct is currently segmented manually, making it tedious and error prone. In order to measure post-treatment infarct volume more efficiently from follow-up NCCT images, a novel joint segmentation approach is proposed for segmenting ischemic infarct and hemorrhage simultaneously, which makes use of multi-region time-implicit contour evolution combined with random forest (RF) learned semantic information. The quantitative evaluation using 16 patient images shows that the proposed segmentation approach is delivering favorable results compared to the gold standard of manual segmentation in terms of dice similarity coefficient (DSC) and the mean (MAD) and maximum (MAXD) absolute surface distance. To the best of our knowledge, this paper reports the first attempt of simultaneously segmenting ischemic infarct and hemorrhage from follow-up NCCT images of post-treatment acute stroke patients.

The significance of the transentorhinal (TE) cortex has been well known for the early diagnosis of Alzheimer’s disease (AD). However, precise mapping of the TE cortex for the detection of local changes in the region was not well established mostly due to significant geometric variations around TE. In this paper, we propose a novel framework for automated patch generation of the TE cortex, patch-based mapping, and construction of an atlas with a distributed network. We locate the TE cortex and extract a small patch surrounding the TE cortex from a cortical surface using a coarse map by FreeSurfer. We apply a recently developed intrinsic surface mapping algorithm based on Riemannian metric optimization on surfaces (RMOS) in the Laplace-Beltrami embedding space to compute fine maps between the small patches. We also develop a distributed atlas of the TE cortex, formed by a shortest path tree whose nodes are atlas subjects, to reduce anatomical misalignments by mapping only between similar patches. In our experimental results, we construct the distributed atlas of the TE cortex using 50 subjects from the Human Connectome Project (HCP), and show that detailed correspondences within the distributed network are established. Using a large-scale dataset of 380 subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI), we demonstrate that our patch-based mapping with the distribute atlas outperforms the conventional centralized mapping (direct mapping to a single atlas) for detecting atrophy of the TE cortex in the early stage of AD.

Whole brain segmentation on a structural magnetic resonance imaging (MRI) is essential in non-invasive investigation for neuroanatomy. Historically, multi-atlas segmentation (MAS) has been regarded as the de facto standard method for whole brain segmentation. Recently, deep neural network approaches have been applied to whole brain segmentation by learning random patches or 2D slices. Yet, few previous efforts have been made on detailed whole brain segmentation using 3D networks due to the following challenges: (1) fitting entire whole brain volume into 3D networks is restricted by the current GPU memory, and (2) the large number of targeting labels (e.g., >100 labels) with limited number of training 3D volumes (e.g., <50 scans). In this paper, we propose the spatially localized atlas network tiles (SLANT) method to distribute multiple independent 3D fully convolutional networks to cover overlapped sub-spaces in a standard atlas space. This strategy simplifies the whole brain learning task to localized sub-tasks, which was enabled by combing canonical registration and label fusion techniques with deep learning. To address the second challenge, auxiliary labels on 5111 initially unlabeled scans were created by MAS for pre-training. From empirical validation, the state-of-the-art MAS method achieved mean Dice value of 0.76, 0.71, and 0.68, while the proposed method achieved 0.78, 0.73, and 0.71 on three validation cohorts. Moreover, the computational time reduced from >30 h using MAS to $$\approx $$ ≈ 15 min using the proposed method. The source code is available online ( https://github.com/MASILab/SLANTbrainSeg ).

Convolutional neural networks (CNNs) have been successfully used for brain tumor segmentation, specifically, fully convolutional networks (FCNs). FCNs can segment a set of voxels at once, having a direct spatial correspondence between units in feature maps (FMs) at a given location and the corresponding classified voxels. In convolutional layers, FMs are merged to create new FMs, so, channel combination is crucial. However, not all FMs have the same relevance for a given class. Recently, in classification problems, Squeeze-and-Excitation (SE) blocks have been proposed to re-calibrate FMs as a whole, and suppress the less informative ones. However, this is not optimal in FCN due to the spatial correspondence between units and voxels. In this article, we propose feature recombination through linear expansion and compression to create more complex features for semantic segmentation. Additionally, we propose a segmentation SE (SegSE) block for feature recalibration that collects contextual information, while maintaining the spatial meaning. Finally, we evaluate the proposed methods in brain tumor segmentation, using publicly available data.

Deep learning for clinical applications is subject to stringent performance requirements, which raises a need for large labeled datasets. However, the enormous cost of labeling medical data makes this challenging. In this paper, we build a cost-sensitive active learning system for the problem of intracranial hemorrhage detection and segmentation on head computed tomography (CT). We show that our ensemble method compares favorably with the state-of-the-art, while running faster and using less memory. Moreover, our experiments are done using a substantially larger dataset than earlier papers on this topic. Since the labeling time could vary tremendously across examples, we model the labeling time and optimize the return on investment. We validate this idea by core-set selection on our large labeled dataset and by growing it with data from the wild.