Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016

Brain connectivity networks have been widely used for diagnosis of brain-related diseases, e.g., Alzheimer’s disease (AD), mild cognitive impairment (MCI), and attention deficit hyperactivity disorder (ADHD). Although several network descriptors have been designed for representing brain connectivity networks, most of them not only ignore the important weight information of edges, but also cannot capture the modular local structures of brain connectivity networks by only focusing on individual brain regions. In this paper, we propose a new network descriptor (called ordinal pattern) for brain connectivity networks, and apply it for brain disease diagnosis. Specifically, we first define ordinal patterns that contain sequences of weighted edges based on a functional connectivity network. A frequent ordinal pattern mining algorithm is then developed to identify those frequent ordinal patterns in a brain connectivity network set. We further perform discriminative ordinal pattern selection, followed by a SVM classification process. Experimental results on both the ADNI and the ADHD-200 data sets demonstrate that the proposed method achieves significant improvement compared with state-of-the-art brain connectivity network based methods.

The cortical folding of the human brain is highly complex and variable across individuals. Mining the major patterns of cortical folding from modern large-scale neuroimaging datasets is of great importance in advancing techniques for neuroimaging analysis and understanding the inter-individual variations of cortical folding and its relationship with cognitive function and disorders. As the primary cortical folding is genetically influenced and has been established at term birth, neonates with the minimal exposure to the complicated postnatal environmental influence are the ideal candidates for understanding the major patterns of cortical folding. In this paper, for the first time, we propose a novel method for discovering the major patterns of cortical folding in a large-scale dataset of neonatal brain MR images (N=677). In our method, first, cortical folding is characterized by the distribution of sulcal pits, which are the locally deepest points in cortical sulci. Because deep sulcal pits are genetically related, relatively consistent across individuals, and also stable during brain development, they are well suitable for representing and characterizing cortical folding. Then, the similarities between sulcal pit distributions of any two subjects are measured from spatial, geometrical, and topological points of view. Next, these different measurements are adaptively fused together using a similarity network fusion technique, to preserve their common information and also catch their complementary information. Finally, leveraging the fused similarity measurements, a hierarchical affinity propagation algorithm is used to group similar sulcal folding patterns together. The proposed method has been applied to 677 neonatal brains (the largest neonatal dataset to our knowledge) in the central sulcus, superior temporal sulcus, and cingulate sulcus, and revealed multiple distinct and meaningful folding patterns in each region.

Cortical gyrification is one of the most prominent features of human brain. A variety of studies in the brain mapping field have demonstrated the specific structural and functional differences between gyral and sulcal regions. However, previous studies of gyri/sulci function analysis based on the fMRI data assume the temporal stationarity over the entire fMRI scan, while the possible temporal dynamics of gyri/sulci function is largely unknown. We present a computational framework to model the functional dynamics of cortical gyri and sulci based on task fMRI data. Specifically, the whole-brain fMRI signals’ temporal segments are derived via the sliding time window approach. The spatial overlap patterns among functional networks (SOPFNs), which are crucial for characterizing brain functions, are then measured within each time window via a group-wise sparse representation approach. Finally, the temporal dynamics of SOPFNs distribution on gyral/sulcal regions across all time windows are assessed. Experimental results based on the publicly released Human Connectome Project task fMRI data demonstrated that the proposed framework identified meaningful temporal dynamics difference of the SOPFNs distribution between gyral and sulcal regions which are reproducible across different subjects and task fMRI datasets. Our results provide novel understanding of functional dynamics mechanisms of human cerebral cortex.

In clinical neuroscience, task-based fMRI (tfMRI) is a popular method to explore the brain network activation difference between healthy controls and brain diseases like Prenatal Alcohol Exposure (PAE). Traditionally, most studies adopt the general linear model (GLM) to detect task-evoked activations. However, GLM has been demonstrated to be limited in reconstructing concurrent heterogeneous networks. In contrast, sparse representation based methods have attracted increasing attention due to the capability of automatically reconstructing concurrent brain activities. However, this data-driven strategy is still challenged in establishing accurate correspondence across individuals and characterizing group-wise consistent activation maps in a principled way. In this paper, we propose a novel multi-stage sparse coding framework to identify group-wise consistent networks in a structured method. By applying this novel framework on two groups of tfMRI data (healthy control and PAE), we can effectively identify group-wise consistent activation maps and characterize brain networks/regions affected by PAE.

Analysis of brain functional connectivity network (BFCN) has shown great potential in understanding brain functions and identifying biomarkers for neurological and psychiatric disorders, such as Alzheimer’s disease and its early stage, mild cognitive impairment (MCI). In all these applications, the accurate construction of biologically meaningful brain network is critical. Due to the sparse nature of the brain network, sparse learning has been widely used for complex BFCN construction. However, the conventional $$l_1$$-norm penalty in the sparse learning equally penalizes each edge (or link) of the brain network, which ignores the link strength and could remove strong links in the brain network. Besides, the conventional sparse regularization often overlooks group structure in the brain network, i.e., a set of links (or connections) sharing similar attribute. To address these issues, we propose to construct BFCN by integrating both link strength and group structure information. Specifically, a novel correlation-weighted sparse group constraint is devised to account for and balance among (1) sparsity, (2) link strength, and (3) group structure, in a unified framework. The proposed method is applied to MCI classification using the resting-state fMRI from ADNI-2 dataset. Experimental results show that our method is effective in modeling human brain connectomics, as demonstrated by superior MCI classification accuracy of 81.8 %. Moreover, our method is promising for its capability in modeling more biologically meaningful sparse brain networks, which will benefit both basic and clinical neuroscience studies.

Resting state fMRI (rsfMRI) has been a useful imaging modality for network level understanding and diagnosis of brain diseases, such as mild traumatic brain injury (mTBI). However, there call for effective methodologies which can detect group-wise and longitudinal changes of network interactions in mTBI. The major challenges are two folds: (1) There lacks an individualized and common network system that can serve as a reference platform for statistical analysis; (2) Networks and their interactions are usually not modeled in the same algorithmic structure, which results in bias and uncertainty. In this paper, we propose a novel temporal concatenated sparse coding (TCSC) method to address these challenges. Based on the sparse graph theory the proposed method can model the commonly shared spatial maps of networks and the local dynamics of the networks in each subject in one algorithmic structure. Obviously, the local dynamics are not comparable across subjects in rsfMRI or across groups; however, based on the correspondence established by the common spatial profiles, the interactions of these networks can be modeled individually and statistically assessed in a group-wise fashion. The proposed method has been applied on an mTBI dataset with acute and sub-acute stages, and experimental results have revealed meaningful network interaction changes in mTBI.

Investigating functional brain networks and activities using sparse representation of fMRI data has received significant interests in the neuroimaging field. It has been reported that sparse representation is effective in reconstructing concurrent and interactive functional brain networks. However, previous data-driven reconstruction approaches rarely simultaneously take consideration of anatomical structures, which are the substrate of brain function. Furthermore, it has been rarely explored whether structured sparse representation with anatomical guidance could facilitate functional networks reconstruction. To address this problem, in this paper, we propose to reconstruct brain networks using the anatomy-guided structured multi-task regression (AGSMR) in which 116 anatomical regions from the AAL template as prior knowledge are employed to guide the network reconstruction. Using the publicly available Human Connectome Project (HCP) Q1 dataset as a test bed, our method demonstrated that anatomical guided structure sparse representation is effective in reconstructing concurrent functional brain networks.

Gene coexpression patterns carry rich information of complex brain structures and functions. Characterization of these patterns in an unbiased and integrated manner will illuminate the higher order transcriptome organization and offer molecular foundations of functional circuitry. Here we demonstrate a data-driven method that can effectively extract coexpression networks from transcriptome profiles using the Allen Mouse Brain Atlas dataset. For each of the obtained networks, both genetic compositions and spatial distributions in brain volume are learned. A simultaneous knowledge of precise spatial distributions of specific gene as well as the networks the gene plays in and the weights it carries can bring insights into the molecular mechanism of brain formation and functions. Gene ontologies and the comparisons with published data reveal interesting functions of the identified coexpression networks, including major cell types, biological functions, brain regions, and/or brain diseases.

Integrative analysis based on quantitative representation of whole slide images (WSIs) in a large histology cohort may provide predictive models of clinical outcome. On one hand, the efficiency and effectiveness of such representation is hindered as a result of large technical variations (e.g., fixation, staining) and biological heterogeneities (e.g., cell type, cell state) that are always present in a large cohort. On the other hand, perceptual interpretation/validation of important multi-variate phenotypic signatures are often difficult due to the loss of visual information during feature transformation in hyperspace. To address these issues, we propose a novel approach for integrative analysis based on cellular morphometric context, which is a robust representation of WSI, with the emphasis on tumor architecture and tumor heterogeneity, built upon cellular level morphometric features within the spatial pyramid matching (SPM) framework. The proposed approach is applied to The Cancer Genome Atlas (TCGA) lower grade glioma (LGG) cohort, where experimental results (i) reveal several clinically relevant cellular morphometric types, which enables both perceptual interpretation/validation and further investigation through gene set enrichment analysis; and (ii) indicate the significantly increased survival rates in one of the cellular morphometric context subtypes derived from the cellular morphometric context.

Understanding brain volumetry is essential to understand neuro-development and disease. Historically, age-related changes have been studied in detail for specific age ranges (e.g., early childhood, teen, young adults, elderly, etc.) or more sparsely sampled for wider considerations of lifetime aging. Recent advancements in data sharing and robust processing have made available considerable quantities of brain images from normal, healthy volunteers. However, existing analysis approaches have had difficulty addressing (1) complex volumetric developments on the large cohort across the life time (e.g., beyond cubic age trends), (2) accounting for confound effects, and (3) maintaining an analysis framework consistent with the general linear model (GLM) approach pervasive in neuroscience. To address these challenges, we propose to use covariate-adjusted restricted cubic spline (C-RCS) regression within a multi-site cross-sectional framework. This model allows for flexible consideration of nonlinear age-associated patterns while accounting for traditional covariates and interaction effects. As a demonstration of this approach on lifetime brain aging, we derive normative volumetric trajectories and 95 % confidence intervals from 5111 healthy patients from 64 sites while accounting for confounding sex, intracranial volume and field strength effects. The volumetric results are shown to be consistent with traditional studies that have explored more limited age ranges using single-site analyses. This work represents the first integration of C-RCS with neuroimaging and the derivation of structural covariance networks (SCNs) from a large study of multi-site, cross-sectional data.

We present a graph-theoretical algorithm to extract the connected core structural connectivity network of a subject population. Extracting this core common network across subjects is a main problem in current neuroscience. Such network facilitates cognitive and clinical analyses by reducing the number of connections that need to be explored. Furthermore, insights into the human brain structure can be gained by comparing core networks of different populations. We show that our novel algorithm has theoretical and practical advantages. First, contrary to the current approach our algorithm guarantees that the extracted core subnetwork is connected agreeing with current evidence that the core structural network is tightly connected. Second, our algorithm shows enhanced performance when used as feature selection approach for connectivity analysis on populations.

Diffusion magnetic resonance imaging (dMRI) enables in vivo investigation of white matter tracts, where the estimation of fiber orientations (FOs) is a crucial step. Dictionary-based methods have been developed to compute FOs with a lower number of dMRI acquisitions. To reduce the effect of noise that is inherent in dMRI acquisitions, spatial consistency of FOs between neighbor voxels has been incorporated into dictionary-based methods. Because many fiber tracts are tube- or sheet-shaped, voxels belonging to the same tract could share similar FO configurations even when they are not adjacent to each other. Therefore, it is possible to use nonlocal information to improve the performance of FO estimation. In this work, we propose an FO estimation algorithm, Fiber Orientation Reconstruction using Nonlocal and Local Information (FORNLI), which adds nonlocal information to guide FO computation. The diffusion signals are represented by a set of fixed prolate tensors. For each voxel, we compare its patch-based diffusion profile with those of the voxels in a search range, and its nonlocal reference voxels are determined as the k nearest neighbors in terms of diffusion profiles. Then, FOs are estimated by iteratively solving weighted $$\ell _{1}$$-norm regularized least squares problems, where the weights are determined using local neighbor voxels and nonlocal reference voxels. These weights encourage FOs that are consistent with the local and nonlocal information. FORNLI was performed on simulated and real brain dMRI, which demonstrates the benefit of incorporating nonlocal information for FO estimation.

Functional magnetic resonance imaging (fMRI) provides a non-invasive way to investigate brain activity. Recently, convergent evidence shows that the correlations of spontaneous fluctuations between two distinct brain regions dynamically change even in resting state, due to the condition-dependent nature of brain activity. Thus, quantifying the patterns of functional connectivity (FC) in a short time period and changes of FC over time can potentially provide valuable insight into both individual-based diagnosis and group comparison. In light of this, we propose a novel computational method to robustly estimate both static and dynamic spatial-temporal connectivity patterns from the observed noisy signals of individual subject. We achieve this goal in two folds: (1) Construct static functional connectivity across brain regions. Due to low signal-to-noise ratio induced by possible non-neural noise, the estimated FC strength is very sensitive and it is hard to define a good threshold to distinguish between real and spurious connections. To alleviate this issue, we propose to optimize FC which is in consensus with not only the low level region-to-region signal correlations but also the similarity of high level principal connection patterns learned from the estimated link-to-link connections. Since brain network is intrinsically sparse, we also encourage sparsity during FC optimization. (2) Characterize dynamic functional connectivity along time. It is hard to synchronize the estimated dynamic FC patterns and the real cognitive state changes, even using learning-based methods. To address these limitations, we further extend above FC optimization method into the spatial-temporal domain by arranging the FC estimations along a set of overlapped sliding windows into a tensor structure as the window slides. Then we employ low rank constraint in the temporal domain assuming there are likely a small number of discrete states that the brain transverses during a short period of time. We applied the learned spatial-temporal patterns from fMRI images to identify autism subjects. Promising classification results have been achieved, suggesting high discrimination power and great potentials in computer assisted diagnosis.

The study of the human connectome is becoming more popular due to its potential to reveal the brain function and structure. A critical step in connectome analysis is to parcellate the cortex into coherent regions that can be used to build graphical models of connectivity. Computing an optimal parcellation is of great importance, as this stage can affect the performance of the subsequent analysis. To this end, we propose a new parcellation method driven by structural connectivity estimated from diffusion MRI. We learn a manifold from the local connectivity properties of an individual subject and identify parcellation boundaries as points in this low-dimensional embedding where the connectivity patterns change. We compute spatially contiguous and non-overlapping parcels from these boundaries after projecting them back to the native cortical surface. Our experiments with a set of 100 subjects show that the proposed method can produce parcels with distinct patterns of connectivity and a higher degree of homogeneity at varying resolutions compared to the state-of-the-art methods, hence can potentially provide a more reliable set of network nodes for connectome analysis.

Brain evolution has been an intriguing research topic for centuries. Efforts have been denoted to identifying structural connectome preserved between macaques and humans and the one exclusive to one species. However, recent studies mainly focus on one specific fasciculus or one region. The similarity and difference of global structural connection network in macaque and human are still largely unknown. In this work, we used diffusion MRI (dMRI) to estimate the whole brain large-scale white matter pathways and Brodmann areas as a test bed to construct a global connectome for the two species. We adopted sparse canonical correlation analysis (SCCA) algorithm to yield the weights which can be applied to the connectome to produce the components strongly correlated between the two species. Joint analysis of the weights helped to identify the preserved white matter pathways and those exclusive to a specific species. The results are consistent with the reports in the literatures, demonstrating the effectiveness and promise of this framework.

Functional subnetwork extraction is commonly employed to study the brain’s modular structure. However, reliable extraction from functional magnetic resonance imaging (fMRI) data remains challenging. As representations of brain networks, brain graph estimates are typically noisy due to the pronounced noise in fMRI data. Also, confounds, such as region size bias, motion artifacts, and signal dropout, introduce region-specific bias in connectivity, e.g. a node in a signal dropout area tends to display lower connectivity. The traditional approach of global thresholding might thus remove relevant edges that have low connectivity due to confounds, resulting in erroneous subnetwork extraction. In this paper, we present a modularity reinforcement strategy that deals with the above two challenges. Specifically, we propose a local thresholding scheme that accounts for region-specific connectivity bias when pruning noisy edges. From the resulting thresholded graph, we derive a node similarity measure by comparing the adjacency structure of each node, i.e. its connection fingerprint, with that of other nodes. Drawing on the intuition that nodes belonging to the same subnetwork should have similar connection fingerprints, we refine the brain graph with this similarity measure to reinforce its modularity structure. On synthetic data, our strategy achieves higher accuracy in subnetwork extraction compared to using standard brain graph estimates. On real data, subnetworks extracted with our strategy attain higher overlaps with well-established brain systems and higher subnetwork reproducibility across a range of graph densities. Our results thus demonstrate that modularity reinforcement with our strategy provides a clear gain in subnetwork extraction.

Integration of functional and structural brain connectivity is a topic receiving growing attention in the research community. Their fusion can, in fact, shed new light on brain functions. Targeting this issue, the manuscript proposes a constrained autoregressive model allowing to generate an “effective” connectivity matrix that model the structural connectivity integrating the functional activity. In practice, an initial structural connectivity representation is altered according to functional data, by minimizing the reconstruction error of an autoregressive model constrained by the structural prior. The proposed model has been tested in a community detection framework, where the brain is partitioned using the effective network across multiple subjects. Results showed that using the effective connectivity the resulting clusters better describe the functional interactions of different regions while maintaining the structural organization.

Parcellating the brain into a set of distinct subregions is an essential step for building and studying brain connectivity networks. Connectivity driven parcellation is a natural approach, but suffers from the lack of reliability of connectivity data. Combining modalities in the parcellation task has the potential to yield more robust parcellations, yet hasn’t been explored much. In this paper, we propose a graph-based multi-modal parcellation method that iteratively computes a set of modality specific parcellations and merges them using the concept of fusion moves. The merged parcellation initialises the next iteration, forcing all modalities to converge towards a set of mutually informed parcellations. Experiments on 50 subjects of the Human Connectome Project database show that the multi-modal setting yields parcels that are more reproducible and more representative of the underlying connectivity.

We present a continuous model for structural brain connectivity based on the Poisson point process. The model treats each streamline curve in a tractography as an observed event in connectome space, here a product space of cortical white matter boundaries. We approximate the model parameter via kernel density estimation. To deal with the heavy computational burden, we develop a fast parameter estimation method by pre-computing associated Legendre products of the data, leveraging properties of the spherical heat kernel. We show how our approach can be used to assess the quality of cortical parcellations with respect to connectivty. We further present empirical results that suggest the “discrete” connectomes derived from our model have substantially higher test-retest reliability compared to standard methods.

In this paper, we present a novel method for obtaining a low dimensional representation of a complex brain network that: (1) can be interpreted in a neurobiologically meaningful way, (2) emphasizes group differences by accounting for label information, and (3) captures the variation in disease subtypes/severity by respecting the intrinsic manifold structure underlying the data. Our method is a supervised variant of non-negative matrix factorization (NMF), and achieves dimensionality reduction by extracting an orthogonal set of subnetworks that are interpretable, reconstructive of the original data, and also discriminative at the group level. In addition, the method includes a manifold regularizer that encourages the low dimensional representations to be smooth with respect to the intrinsic geometry of the data, allowing subjects with similar disease-severity to share similar network representations. While the method is generalizable to other types of non-negative network data, in this work we have used structural connectomes (SCs) derived from diffusion data to identify the cortical/subcortical connections that have been disrupted in abnormal neurological state. Experiments on a traumatic brain injury (TBI) dataset demonstrate that our method can identify subnetworks that can reliably classify TBI from controls and also reveal insightful connectivity patterns that may be indicative of a biomarker.

We present a new method to identify anatomical subnetworks of the human white matter connectome that are predictive of neurodevelopmental outcomes. We employ our method on a dataset of 168 preterm infant connectomes, generated from diffusion tensor images (DTI) taken shortly after birth, to discover subnetworks that predict scores of cognitive and motor development at 18 months. Predictive subnetworks are extracted via sparse linear regression with weights on each connectome edge. By enforcing novel backbone network and connectivity based priors, along with a non-negativity constraint, the learned subnetworks are simultaneously anatomically plausible, well connected, positively weighted and reasonably sparse. Compared to other state-of-the-art subnetwork extraction methods, we found that our approach extracts subnetworks that are more integrated, have fewer noisy edges and that are also better predictive of neurodevelopmental outcomes.

Diffusion MRI tractography produces massive sets of streamlines that contain a wealth of information on brain connections. The size of these datasets creates a need for automated clustering methods to group the streamlines into anatomically meaningful bundles. Conventional clustering techniques group streamlines based on their spatial coordinates. Neuroanatomists, however, define white-matter bundles based on the anatomical structures that they go through or next to, rather than their spatial coordinates. Thus we propose a similarity metric for clustering streamlines based on their position relative to cortical and subcortical brain regions. We incorporate this metric into a hierarchical clustering algorithm and compare it to a metric that relies on Euclidean distance, using data from the Human Connectome Project. We show that the anatomical similarity metric leads to a $$20\,\%$$ improvement in the agreement of clustering results with manually labeled tracts, without introducing prior information from a tract atlas into the clustering.

A key question in learning from clinical routine imaging data is whether we can identify coherent patterns that re-occur across a population, and at the same time are linked to clinically relevant patient parameters. Here, we present a feature learning and clustering approach that groups 3D imaging data based on visual features at corresponding anatomical regions extracted from clinical routine imaging data without any supervision. On a set of 7812 routine lung computed tomography volumes, we show that the clustering results in a grouping linked to terms in radiology reports which were not used for clustering. We evaluate different visual features in light of their ability to identify groups of images with consistent reported findings.

While tractography is widely used in brain imaging research, its quantitative validation is highly difficult. Many fiber systems, however, have well-known topographic organization which can even be quantitatively mapped such as the retinotopy of visual pathway. Motivated by this previously untapped anatomical knowledge, we develop a novel tractography method that preserves both topographic and geometric regularity of fiber systems. For topographic preservation, we propose a novel likelihood function that tests the match between parallel curves and fiber orientation distributions. For geometric regularity, we use Gaussian distributions of Frenet-Serret frames. Taken together, we develop a Bayesian framework for generating highly organized tracks that accurately follow neuroanatomy. Using multi-shell diffusion images of 56 subjects from Human Connectome Project, we compare our method with algorithms from MRtrix. By applying regression analysis between retinotopic eccentricity and tracks, we quantitatively demonstrate that our method achieves superior performance in preserving the retinotopic organization of optic radiation.

Dramatic changes of the human brain during the first year of postnatal development are poorly understood due to their multifold complexity. In this paper, we present the first attempt to jointly predict, using neonatal data, the dynamic growth pattern of brain cortical surfaces (collection of 3D triangular faces) and fiber tracts (collection of 3D lines). These two entities are modeled jointly as a multishape (a set of interlinked shapes). We propose a hybrid learning-based multishape prediction framework that captures both the diffeomorphic evolution of the cortical surfaces and the non-diffeomorphic growth of fiber tracts. In particular, we learn a set of geometric and dynamic cortical features and fiber connectivity features that characterize the relationships between cortical surfaces and fibers at different timepoints (0, 3, 6, and 9 months of age). Given a new neonatal multishape at 0 month of age, we hierarchically predict, at 3, 6 and 9 months, the postnatal cortical surfaces vertex-by-vertex along with fibers connected to adjacent faces to these vertices. This is achieved using a new fiber-to-face metric that quantifies the similarity between multishapes. For validation, we propose several evaluation metrics to thoroughly assess the performance of our framework. The results confirm that our framework yields good prediction accuracy of complex neonatal multishape development within a few seconds.

Reconstruction of topologically correct and accurate cortical surfaces from infant MR images is of great importance in neuroimaging mapping of early brain development. However, due to rapid growth and ongoing myelination, infant MR images exhibit extremely low tissue contrast and dynamic appearance patterns, thus leading to much more topological errors (holes and handles) in the cortical surfaces derived from tissue segmentation results, in comparison to adult MR images which typically have good tissue contrast. Existing methods for topological correction either rely on the minimal correction criteria, or ad hoc rules based on image intensity priori, thus often resulting in erroneous correction and large anatomical errors in reconstructed infant cortical surfaces. To address these issues, we propose to correct topological errors by learning information from the anatomical references, i.e., manually corrected images. Specifically, in our method, we first locate candidate voxels of topologically defected regions by using a topology-preserving level set method. Then, by leveraging rich information of the corresponding patches from reference images, we build region-specific dictionaries from the anatomical references and infer the correct labels of candidate voxels using sparse representation. Notably, we further integrate these two steps into an iterative framework to enable gradual correction of large topological errors, which are frequently occurred in infant images and cannot be completely corrected using one-shot sparse representation. Extensive experiments on infant cortical surfaces demonstrate that our method not only effectively corrects the topological defects, but also leads to better anatomical consistency, compared to the state-of-the-art methods.

With the advance of human connectome research, there are great interests in computing diffeomorphic maps of brain surfaces with rich connectivity features. In this paper, we propose a novel framework for connectivity-driven surface mapping based on Riemannian metric optimization on surfaces (RMOS) in the Laplace-Beltrami (LB) embedding space. The mathematical foundation of our method is that we can use the pullback metric to define an isometry between surfaces for an arbitrary diffeomorphism, which in turn results in identical LB embeddings from the two surfaces. For connectivity-driven surface mapping, our goal is to compute a diffeomorphism that can match a set of connectivity features defined over anatomical surfaces. The proposed RMOS approach achieves this goal by iteratively optimizing the Riemannian metric on surfaces to match the connectivity features in the LB embedding space. At the core of our framework is an optimization approach that converts the cost function of connectivity features into a distance measure in the LB embedding space, and optimizes it using gradients of the LB eigen-system with respect to the Riemannian metric. We demonstrate our method on the mapping of thalamic surfaces according to connectivity to ten cortical regions, which we compute with the multi-shell diffusion imaging data from the Human Connectome Project (HCP). Comparisons with a state-of-the-art method show that the RMOS method can more effectively match anatomical features and detect thalamic atrophy due to normal aging.

Typical studies of the geometry of the cerebral cortical structure focus on either cortical folding or thickness. They rely on spatial normalization, but use cortical descriptors that are sensitive to misregistration arising from the well-known problems of partial homologies between subject brains and local optima in nonlinear registration. In contrast to these approaches, we propose a novel framework for studying the geometry of the entire cortical sheet, subsuming its folding and thickness characteristics. We propose a novel descriptor of local cortical geometry to increase robustness to partial homology and misregistration. The proposed descriptor lies on a Riemannian manifold, and we describe a method for hypothesis testing on manifolds for cross-sectional studies. Results on simulated and clinical data show the benefits of the proposed approach for detecting between-group differences with greater accuracy and consistency.

Understanding patterns of brain development before birth is of both high clinical and scientific interest. However, despite advances in reconstruction methods, the challenging setting of in-utero imaging renders precise, point-wise measurements of the rapidly changing fetal brain morphology difficult. This paper proposes a method to deal with bad measurement quality due to image noise, motion artefacts and ensuing segmentation and registration errors by enforcing spatial regularity during the estimation of parametric models of cortical expansion. Qualitative and quantitative analysis of the proposed method was performed on 88 clinical fetal MR volumes. We show that the resulting models accurately capture the morphological and temporal properties of fetal brain development by predicting gestational age on unseen cases at human-level accuracy.

Extremely preterm birth (less than 32 weeks completed gestation) overlaps with a period of rapid brain growth and development. Investigating longitudinal brain changes over the preterm period in these infants may allow the development of biomarkers for predicting neurological outcome. In this paper we investigate longitudinal changes in cortical thickness, cortical fractional anisotropy and cortical mean diffusivity in a groupwise space obtained using a novel multi-modal spectral matching technique. The novelty of this method consists in its ability to register surfaces with very little shape complexity, like in the case of the early developmental stages of preterm infants, by also taking into account their underlying biology. A multi-modal method also allows us to investigate interdependencies between the parameters. Such tools have great potential in investigating in depth the regions affected by preterm birth and how they relate to each other.

The diagnosis of Alzheimer’s disease (AD) from neuroimaging data at the pre-clinical stage has been intensively investigated because of the immense social and economic cost. In the past decade, computational approaches on longitudinal image sequences have been actively investigated with special attention to Mild Cognitive Impairment (MCI), which is an intermediate stage between normal control (NC) and AD. However, current state-of-the-art diagnosis methods have limited power in clinical practice, due to the excessive requirements such as equal and immoderate number of scans in longitudinal imaging data. More critically, very few methods are specifically designed for the early alarm of AD uptake. To address these limitations, we propose a flexible spatial-temporal solution for early detection of AD by recognizing abnormal structure changes from longitudinal MR image sequence. Specifically, our method is leveraged by the non-reversible nature of AD progression. We employ temporally structured SVM to accurately alarm AD at early stage by enforcing the monotony on classification result to avoid unrealistic and inconsistent diagnosis result along time. Furthermore, in order to select best features which can well collaborate with the classifier, we present as joint feature selection and classification framework. The evaluation on more than 150 longitudinal subjects from ADNI dataset shows that our method is able to alarm the conversion of AD 12 months prior to the clinical diagnosis with at least 82.5 % accuracy. It is worth noting that our proposed method works on widely used MR images and does not have restriction on the number of scans in the longitudinal sequence, which is very attractive to real clinical practice.

Alzheimer’s Disease (AD), a severe type of neurodegenerative disorder with progressive impairment of learning and memory, has threatened the health of millions of people. How to recognize AD at early stage is crucial. Multiple models have been presented to predict cognitive impairments by means of neuroimaging data. However, traditional models did not employ the valuable longitudinal information along the progression of the disease. In this paper, we proposed a novel longitudinal feature learning model to simultaneously uncover the interrelations among different cognitive measures at different time points and utilize such interrelated structures to enhance the learning of associations between imaging features and prediction tasks. Moreover, we adopted Schatten p-norm to identify the interrelation structures existing in the low-rank subspace. Empirical results on the ADNI cohort demonstrated promising performance of our model.

To boost the power of classifiers, studies often increase the size of existing samples through the addition of independently collected data sets. Doing so requires harmonizing the data for demographic and acquisition differences based on a control cohort before performing disease specific classification. The initial harmonization often mitigates group differences negatively impacting classification accuracy. To preserve cohort separation, we propose the first model unifying linear regression for data harmonization with a logistic regression for disease classification. Learning to harmonize data is now an adaptive process taking both disease and control data into account. Solutions within that model are confined by group cardinality to reduce the risk of overfitting (via sparsity), to explicitly account for the impact of disease on the inter-dependency of regions (by grouping them), and to identify disease specific patterns (by enforcing sparsity via the $$l_0$$-‘norm’). We test those solutions in distinguishing HIV-Associated Neurocognitive Disorder from Mild Cognitive Impairment of two independently collected, neuroimage data sets; each contains controls and samples from one disease. Our classifier is impartial to acquisition difference between the data sets while being more accurate in diseases seperation than sequential learning of harmonization and classification parameters, and non-sparsity based logistic regressors.

Graph-based Transductive Learning (GTL) is a powerful tool in computer-assisted diagnosis, especially when the training data is not sufficient to build reliable classifiers. Conventional GTL approaches first construct a fixed subject-wise graph based on the similarities of observed features (i.e., extracted from imaging data) in the feature domain, and then follow the established graph to propagate the existing labels from training to testing data in the label domain. However, such a graph is exclusively learned in the feature domain and may not be necessarily optimal in the label domain. This may eventually undermine the classification accuracy. To address this issue, we propose a progressive GTL (pGTL) method to progressively find an intrinsic data representation. To achieve this, our pGTL method iteratively (1) refines the subject-wise relationships observed in the feature domain using the learned intrinsic data representation in the label domain, (2) updates the intrinsic data representation from the refined subject-wise relationships, and (3) verifies the intrinsic data representation on the training data, in order to guarantee an optimal classification on the new testing data. Furthermore, we extend our pGTL to incorporate multi-modal imaging data, to improve the classification accuracy and robustness as multi-modal imaging data can provide complementary information. Promising classification results in identifying Alzheimer’s disease (AD), Mild Cognitive Impairment (MCI), and Normal Control (NC) subjects are achieved using MRI and PET data.

Computer assisted imaging aims to characterize disease processes by contrasting healthy and pathological populations. The sensitivity of these analyses is hindered by the variability in the neuroanatomy of the normal population. To alleviate this shortcoming, it is necessary to define a normative range of controls. Moreover, elucidating the structure in outliers may be important in understanding diverging individuals and characterizing prodromal disease states. To address these issues, we propose a novel geometric concept called minimal convex polytope (MCP). The proposed approach is used to simultaneously capture high probability regions in datasets consisting of normal subjects, and delineate outliers, thus characterizing the main directions of deviation from the normative range. We validated our method using simulated datasets before applying it to an imaging study of elderly subjects consisting of 177 controls, 123 Alzheimer’s disease (AD) and 285 mild cognitive impairment (MCI) patients. We show that cerebellar degeneration is a major type of deviation among the controls. Furthermore, our findings suggest that a subset of AD patients may be following an accelerated type of deviation that is observed among the normal population.

Effectively utilizing incomplete multi-modality data for diagnosis of Alzheimer’s disease (AD) is still an area of active research. Several multi-view learning methods have recently been developed to deal with missing data, with each view corresponding to a specific modality or a combination of several modalities. However, existing methods usually ignore the underlying coherence among views, which may lead to sub-optimal learning performance. In this paper, we propose a view-aligned hypergraph learning (VAHL) method to explicitly model the coherence among the views. Specifically, we first divide the original data into several views based on possible combinations of modalities, followed by a sparse representation based hypergraph construction process in each view. A view-aligned hypergraph classification (VAHC) model is then proposed, by using a view-aligned regularizer to model the view coherence. We further assemble the class probability scores generated from VAHC via a multi-view label fusion method to make a final classification decision. We evaluate our method on the baseline ADNI-1 database having 807 subjects and three modalities (i.e., MRI, PET, and CSF). Our method achieves at least a $$4.6\,\%$$ improvement in classification accuracy compared with state-of-the-art methods for AD/MCI diagnosis.

As a neurodegenerative disorder, the Alzheimer’s disease (AD) status can be characterized by the progressive impairment of memory and other cognitive functions. Thus, it is an important topic to use neuroimaging measures to predict cognitive performance and track the progression of AD. Many existing cognitive performance prediction methods employ the regression models to associate cognitive scores to neuroimaging measures, but these methods do not take into account the interconnected structures within imaging data and those among cognitive scores. To address this problem, we propose a novel multi-task learning model for minimizing the k smallest singular values to uncover the underlying low-rank common subspace and jointly analyze all the imaging and clinical data. The effectiveness of our method is demonstrated by the clearly improved prediction performances in all empirical AD cognitive scores prediction cases.

Mild Cognitive Impairment (MCI) is a transitional stage between normal age-related cognitive decline and Alzheimer’s disease (AD). Here we introduce a hyperbolic space sparse coding method to predict impending decline of MCI patients to dementia using surface measures of ventricular enlargement. First, we compute diffeomorphic mappings between ventricular surfaces using a canonical hyperbolic parameter space with consistent boundary conditions and surface tensor-based morphometry is computed to measure local surface deformations. Second, ring-shaped patches of TBM features are selected according to the geometric structure of the hyperbolic parameter space to initialize a dictionary. Sparse coding is then applied on the patch features to learn sparse codes and update the dictionary. Finally, we adopt max-pooling to reduce the feature dimensions and apply Adaboost to predict AD in MCI patients ($$N=133$$) from the Alzheimer’s Disease Neuroimaging Initiative baseline dataset. Our work achieved an accuracy rate of $$96.7\,\%$$ and outperformed some other morphometry measures. The hyperbolic space sparse coding method may offer a more sensitive tool to study AD and its early symptom.

Genome-wide association studies (GWAS) offer new opportunities to identify genetic risk factors for Alzheimer’s disease (AD). Recently, collaborative efforts across different institutions emerged that enhance the power of many existing techniques on individual institution data. However, a major barrier to collaborative studies of GWAS is that many institutions need to preserve individual data privacy. To address this challenge, we propose a novel distributed framework, termed Local Query Model (LQM) to detect risk SNPs for AD across multiple research institutions. To accelerate the learning process, we propose a Distributed Enhanced Dual Polytope Projection (D-EDPP) screening rule to identify irrelevant features and remove them from the optimization. To the best of our knowledge, this is the first successful run of the computationally intensive model selection procedure to learn a consistent model across different institutions without compromising their privacy while ranking the SNPs that may collectively affect AD. Empirical studies are conducted on 809 subjects with 5.9 million SNP features which are distributed across three individual institutions. D-EDPP achieved a 66-fold speed-up by effectively identifying irrelevant features.

With the advances of neuroimaging techniques and genome sequences understanding, the phenotype and genotype data have been utilized to study the brain diseases (known as imaging genetics). One of the most important topics in image genetics is to discover the genetic basis of phenotypic markers and their associations. In such studies, the linear regression models have been playing an important role by providing interpretable results. However, due to their modeling characteristics, it is limited to effectively utilize inherent information among the phenotypes and genotypes, which are helpful for better understanding their associations. In this work, we propose a structured sparse low-rank regression method to explicitly consider the correlations within the imaging phenotypes and the genotypes simultaneously for Brain-Wide and Genome-Wide Association (BW-GWA) study. Specifically, we impose the low-rank constraint as well as the structured sparse constraint on both phenotypes and phenotypes. By using the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset, we conducted experiments of predicting the phenotype data from genotype data and achieved performance improvement by 12.75 % on average in terms of the root-mean-square error over the state-of-the-art methods.

Ultrasound image guidance is widely used in minimally invasive procedures, including fetal surgery. In this context, maintaining visibility of medical devices is a significant challenge. Needles and catheters can readily deviate from the ultrasound imaging plane as they are inserted. When the medical device tips are not visible, they can damage critical structures, with potentially profound consequences including loss of pregnancy. In this study, we performed 3D ultrasonic tracking of a needle using a novel probe with a 1.5D array of transducer elements that was driven by a commercial ultrasound system. A fiber-optic hydrophone integrated into the needle received transmissions from the probe, and data from this sensor was processed to estimate the position of the hydrophone tip in the coordinate space of the probe. Golay coding was used to increase the signal-to-noise (SNR). The relative tracking accuracy was better than 0.4 mm in all dimensions, as evaluated using a water phantom. To obtain a preliminary indication of the clinical potential of 3D ultrasonic needle tracking, an intravascular needle insertion was performed in an in vivo pregnant sheep model. The SNR values ranged from 12 to 16 at depths of 20 to 31 mm and at an insertion angle of $$\mathrm{49^{o}}$$ relative to the probe surface normal. The results of this study demonstrate that 3D ultrasonic needle tracking with a fiber-optic hydrophone sensor and a 1.5D array is feasible in clinically realistic environments.

New methods for needle tip and shaft enhancement in 2D curvilinear ultrasound are proposed. Needle tip enhancement is achieved using an image regularization method that utilizes ultrasound signal transmission maps to model inherent signal loss due to attenuation. Shaft enhancement is achieved by optimizing the proposed signal transmission map using the information based on trajectory constrained boundary statistics derived from phase oriented features. The enhanced tip is automatically localized using spatially distributed image statistics from the estimated shaft trajectory. Validation results from 100 ultrasound images of bovine, porcine, kidney and liver ex vivo reveal a mean localization error of $$0.3\pm 0.06\,\mathrm {mm}$$, a $$43\,\%$$ improvement in localization over previous state of the art.

Ultrasound-based interventions require experience and good hand-eye coordination. Especially for non-experts, correctly guiding a handheld probe towards a target, and staying there, poses a remarkable challenge. We augment a commercial vision-based instrument guidance system with haptic feedback to keep operators on target. A user study shows significant improvements across deviation, time, and ease-of-use when coupling standard ultrasound imaging with visual feedback, haptic feedback, or both.

Deep Anterior Lamellar Keratoplasty using Big-Bubble technique (BB-DALK) is a delicate and complex surgical procedure with a multitude of benefits over Penetrating Keratoplasty (PKP). Yet the steep learning curve and challenges associated with BB-DALK prevents it from becoming the standard procedure for keratoplasty. Optical Coherence Tomography (OCT) aids surgeons to carry out BB-DALK in a shorter time with a higher success rate but also brings complications of its own such as image occlusion by the instrument, the constant need to reposition and added distraction. This work presents a novel real-time guidance system for BB-DALK which is practically a complete tool for smooth execution of the procedure. The guidance system comprises of modified 3D+t OCT acquisitions, advanced visualization, tracking of corneal layers and providing depth information using Augmented Reality. The system is tested by an ophthalmic surgeon performing BB-DALK on several ex vivo pig eyes. Results from multiple evaluations show a maximum tracking error of 8.8 micrometers.

In robotic surgery, tool tracking is important for providing safe tool-tissue interaction and facilitating surgical skills assessment. Despite recent advances in tool tracking, existing approaches are faced with major difficulties in real-time tracking of articulated tools. Most algorithms are tailored for offline processing with pre-recorded videos. In this paper, we propose a real-time 3D tracking method for articulated tools in robotic surgery. The proposed method is based on the CAD model of the tools as well as robot kinematics to generate online part-based templates for efficient 2D matching and 3D pose estimation. A robust verification approach is incorporated to reject outliers in 2D detections, which is then followed by fusing inliers with robot kinematic readings for 3D pose estimation of the tool. The proposed method has been validated with phantom data, as well as ex vivo and in vivo experiments. The results derived clearly demonstrate the performance advantage of the proposed method when compared to the state-of-the-art.

Transesophageal Echocardiography (TEE) and X-Ray fluoroscopy are two routinely used real-time image guidance modalities for interventional procedures, and co-registering them into the same coordinate system enables advanced hybrid image guidance by providing augmented and complimentary information. In this paper, we present an image-based system of co-registering these two modalities through real-time tracking of the 3D position and orientation of a moving TEE probe from 2D fluoroscopy images. The 3D pose of the TEE probe is estimated fully automatically using a detection based visual tracking algorithm, followed by intensity-based 3D-to-2D registration refinement. In addition, to provide high reliability for clinical use, the proposed system can automatically recover from tracking failures. The system is validated on over 1900 fluoroscopic images from clinical trial studies, and achieves a success rate of 93.4 % at 2D target registration error (TRE) less than 2.5 mm and an average TRE of 0.86 mm, demonstrating high accuracy and robustness when dealing with poor image quality caused by low radiation dose and pose ambiguity caused by probe self-symmetry.

An open problem in computer-assisted surgery is to robustly track soft-tissue 3D organ models in laparoscopic videos in real-time and over long durations. Previous real-time approaches use locally-tracked features such as SIFT or SURF to drive the process, usually with KLT tracking. However this is not robust and breaks down with occlusions, blur, specularities, rapid motion and poor texture. We have developed a fundamentally different framework that can deal with most of the above challenges and in real-time. This works by densely matching tissue texture at the pixel level, without requiring feature detection or matching. It naturally handles texture distortion caused by deformation and/or viewpoint change, does not cause drift, is robust to occlusions from tools and other structures, and handles blurred frames. It also integrates robust boundary contour matching, which provides tracking constraints at the organ’s boundaries. We show that it can track over long durations and can handles challenging cases that were previously unsolvable.

Tracking of curvilinear structures (CS), such as vessels and catheters, in X-ray images has become increasingly important in recent interventional applications. However, CS is often barely visible in low-dose X-ray due to overlay of multiple 3D objects in a 2D projection, making robust and accurate tracking of CS very difficult. To address this challenge, we propose a new tracking method that encodes the structure prior of CS in the rank-1 tensor approximation tracking framework, and it also uses the learned hierarchical features via a convolutional neural network (CNN). The three components, i.e., curvilinear prior modeling, high-order information encoding and automatic feature learning, together enable our algorithm to reduce the ambiguity rising from the complex background, and consequently improve the tracking robustness. Our proposed approach is tested on two sets of X-ray fluoroscopic sequences including vascular structures and catheters, respectively. In the tests our approach achieves a mean tracking error of 1.1 pixels for vascular structure and 0.8 pixels for catheter tracking, significantly outperforming state-of-the-art solutions on both datasets.

We propose a novel method for instrument tracking in Retinal Microsurgery (RM) which is apt to withstand the challenges of RM visual sequences in terms of varying illumination conditions and blur. At the same time, the method is general enough to deal with different background and tool appearances. The proposed approach relies on two random forests to, respectively, track the surgery tool and estimate its 2D pose. Robustness to photometric distortions and blur is provided by a specific online refinement stage of the offline trained forest, which makes our method also capable of generalizing to unseen backgrounds and tools. In addition, a peculiar framework for merging together the predictions of tracking and pose is employed to improve the overall accuracy. Remarkable advantages in terms of accuracy over the state-of-the-art are shown on two benchmarks.

We present a novel dynamic shape tracking (DST) method that solves for Lagrangian motion trajectories originating at the left ventricle (LV) boundary surfaces using a graphical structure and Dijkstra’s shortest path algorithm.These trajectories, which are temporally regularized and accrue minimal drift, are augmented with radio-frequency (RF) speckle tracking based mid-wall displacements and dense myocardial deformation fields and strains are calculated.We used this method on 4D Echocardiography (4DE) images acquired from 7 canine subjects and validated the strains using a cuboidal array of 16 sonomicrometric crystals that were implanted on the LV wall. The 4DE based strains correlated well with the crystal based strains. We also created an ischemia on the LV wall and evaluated how strain values change across ischemic, non-ischemic remote and border regions (with the crystals planted accordingly) during baseline, severe occlusion and severe occlusion with dobutamine stress conditions. We were able to observe some interesting strain patterns for the different physiological conditions, which were in good agreement with the crystal based strains.

Endoscopy enables high resolution visualization of tissue texture and is a critical step in many clinical workflows, including diagnosis and radiation therapy treatment planning for cancers in the nasopharynx. However, an endoscopic video does not provide explicit 3D spatial information, making it difficult to use in tumor localization, and it is inefficient to review. We introduce a pipeline for automatically reconstructing a textured 3D surface model, which we call an endoscopogram, from multiple 2D endoscopic video frames. Our pipeline first reconstructs a partial 3D surface model for each input individual 2D frame. In the next step (which is the focus of this paper), we generate a single high-quality 3D surface model using a groupwise registration approach that fuses multiple, partially overlapping, incomplete, and deformed surface models together. We generate endoscopograms from synthetic, phantom, and patient data and show that our registration approach can account for tissue deformations and reconstruction inconsistency across endoscopic video frames.

For early diagnosis of malignancies in the gastrointestinal tract, surveillance endoscopy is increasingly used to monitor abnormal tissue changes in serial examinations of the same patient. Despite successes with optical biopsy for in vivo and in situ tissue characterisation, biopsy retargeting for serial examinations is challenging because tissue may change in appearance between examinations. In this paper, we propose an inter-examination retargeting framework for optical biopsy, based on an image descriptor designed for matching between endoscopic scenes over significant time intervals. Each scene is described by a hierarchy of regional intensity comparisons at various scales, offering tolerance to long-term change in tissue appearance whilst remaining discriminative. Binary coding is then used to compress the descriptor via a novel random forests approach, providing fast comparisons in Hamming space and real-time retargeting. Extensive validation conducted on 13 in vivo gastrointestinal videos, collected from six patients, show that our approach outperforms state-of-the-art methods.

In many clinical procedures involving needle insertion, such as cryoablation, accurate navigation of the needle to the desired target is of paramount importance to optimize the treatment and minimize the damage to the neighboring anatomy. However, the force interaction between the needle and tissue may lead to needle deflection, resulting in considerable error in the intraoperative tracking of the needle tip. In this paper, we have proposed a Kalman filter-based formulation to fuse two sensor data — optical sensor at the base and magnetic resonance (MR) gradient-field driven electromagnetic (EM) sensor placed 10 cm from the needle tip — to estimate the needle deflection online. Angular springs model based tip estimations and EM based estimation without model are used to form the measurement vector in the Kalman filter. Static tip bending experiments show that the fusion method can reduce the error of the tip estimation by from 29.23 mm to 3.15 mm and from 39.96 mm to 6.90 mm at the MRI isocenter and 650 mm from the isocenter respectively.

This paper proposes a 3D local phase-symmetry-based bone enhancement technique to automatically identify weak bone responses in 3D ultrasound images of the wrist. The objective is to enable percutaneous fixation of scaphoid bone fractures, which occur in $$90\,\%$$ of all carpal bone fractures. For this purpose, we utilize 3D frequency spectrum variations to design a set of 3D band-pass Log-Gabor filters for phase symmetry estimation. Shadow information is also incorporated to further enhance the bone surfaces compared to the soft-tissue response. The proposed technique is then used to register a statistical wrist model to intraoperative ultrasound in order to derive a patient specific 3D model of the wrist bones. We perform a cadaver study of 13 subjects to evaluate our method. Our results demonstrate average mean surface and Hausdorff distance errors of 0.7 mm and 1.8 mm, respectively, showing better performance compared to two state-of-the art approaches. This study demonstrate the potential of the proposed technique to be included in an ultrasound-based percutaenous scaphoid fracture fixation procedure.

Inspired by the electrolocalization behavior of weakly electric fish, we introduce a novel catheter guidance system for interventional vascular procedures. Impedance measurements from electrodes on the catheter form an electric image of the internal geometry of the vessel. That electric image is then mapped to a pre-interventional model to determine the relative position of the catheter within the vessel tree. The catheter’s measurement of its surroundings is unaffected by movement of the surrounding tissue, so there is no need for deformable 2D/3D image registration. Experiments in a synthetic vessel tree and ex vivo biological tissue are presented. We employed dynamic time warping to map the empirical data to the pre-interventional simulation, and our system correctly identified the catheter’s path in 25/30 trials in a synthetic phantom and 9/9 trials in biological tissue. These first results demonstrated the capability and potential of Bioelectric Navigation as a non-ionizing technique to guide intravascular devices.

Throughout a patient’s stay in the Intensive Care Unit (ICU), accurate measurement of patient mobility, as part of routine care, is helpful in understanding the harmful effects of bedrest [1]. However, mobility is typically measured through observation by a trained and dedicated observer, which is extremely limiting. In this work, we present a video-based automated mobility measurement system called NIMS: Non-Invasive Mobility Sensor. Our main contributions are: (1) a novel multi-person tracking methodology designed for complex environments with occlusion and pose variations, and (2) an application of human-activity attributes in a clinical setting. We demonstrate NIMS on data collected from an active patient room in an adult ICU and show a high inter-rater reliability using a weighted Kappa statistic of 0.86 for automatic prediction of the highest level of patient mobility as compared to clinical experts.

Motion analysis is typically used for a range of diagnostic procedures in the hospital. While automatic pose estimation from RGB-D input has entered the hospital in the domain of rehabilitation medicine and gait analysis, no such method is available for bed-ridden patients. However, patient pose estimation in the bed is required in several fields such as sleep laboratories, epilepsy monitoring and intensive care units. In this work, we propose a learning-based method that allows to automatically infer 3D patient pose from depth images. To this end we rely on a combination of convolutional neural network and recurrent neural network, which we train on a large database that covers a range of motions in the hospital bed. We compare to a state of the art pose estimation method which is trained on the same data and show the superior result of our method. Furthermore, we show that our method can estimate the joint positions under a simulated occluding blanket with an average joint error of 7.56 cm.

During Cochlear Implant Surgery, the right placement of the implant and the minimization of the surgical trauma to the inner ear are an important issue with recurrent fails. In this study, we reproduced, using simulation, the mechanical insertion of the implant during the surgery. This simulation allows to have a better understanding of the failing cases: excessive contact force, buckling of the implant inside and outside the cochlea. Moreover, using a patient-specific geometric model of the cochlea in the simulation, we show that the insertion angle is a clinical parameter that has an influence on the forces endured by both the cochlea walls and the basilar membrane, and hence to post-operative trauma. The paper presents the mechanical models used for the implant, for the basilar membrane and the boundary conditions (contact, friction, insertion etc...) and discuss the obtained results in the perspective of using the simulation for planning and robotization of the implant insertion.

Many surgical assessment metrics have been developed to identify and rank surgical expertise; however, some of these metrics (e.g., economy of motion) can be difficult to understand and do not coach the user on how to modify behavior. We aim to standardize assessment language by identifying key semantic labels for expertise. We chose six pairs of contrasting adjectives and associated a metric with each pair (e.g., fluid/viscous correlated to variability in angular velocity). In a user study, we measured quantitative data (e.g., limb accelerations, skin conductivity, and muscle activity), for subjects (n = 3, novice to expert) performing tasks on a robotic surgical simulator. Task and posture videos were recorded for each repetition and crowd-workers labeled the videos by selecting one word from each pair. The expert was assigned more positive words and also had better quantitative metrics for the majority of the chosen word pairs, showing feasibility for automated coaching.

We describe a fast analytical method to estimate landmark-based 2D-3D registration accuracy to aid the planning of pancreatobiliary interventions in which ERCP images are combined with information from diagnostic 3D MR or CT images. The method analytically estimates a target registration error (TRE), accounting for errors in the manual selection of both 2D- and 3D landmarks, that agrees with Monte Carlo simulation to within 4.5 ± 3.6 % (mean ± SD). We also show how to analytically estimate a planning uncertainty incorporating uncertainty in patient positioning, and utilise it to support ERCP-guided procedure planning by selecting the optimal patient position and X-ray C-arm orientation that minimises the expected TRE. Simulated- and derived planning uncertainties agreed to within 17.9 ± 9.7 % when the root-mean-square error was less than 50°. We demonstrate the feasibility of this approach on clinical data from two patients.

Endovascular procedures are challenging to perform due to the complexity and difficulty in catheter manipulation. The simultaneous recovery of the 3D structure of the vasculature and the catheter position and orientation intra-operatively is necessary in catheter control and navigation. State-of-art Simultaneous Catheter and Environment Modelling provides robust and real-time 3D vessel reconstruction based on real-time intravascular ultrasound (IVUS) imaging and electromagnetic (EM) sensing, but still relies on accurate registration between EM and pre-operative data. In this paper, a registration-free vessel reconstruction method is proposed for endovascular navigation. In the optimisation framework, the EM-CT registration is estimated and updated intra-operatively together with the 3D vessel reconstruction from IVUS, EM and pre-operative data, and thus does not require explicit registration. The proposed algorithm can also deal with global (patient) motion and periodic deformation caused by cardiac motion. Phantom and in-vivo experiments validate the accuracy of the algorithm and the results demonstrate the potential clinical value of the technique.

Preoperative path planning for Deep Brain Stimulation (DBS) is a multi-objective optimization problem consisting in searching the best compromise between multiple placement constraints. Its automation is usually addressed by turning the problem into mono-objective thanks to an aggregative approach. However, despite its intuitiveness, this approach is known for its incapacity to find all optimal solutions. In this work, we introduce an approach based on multi-objective dominance to DBS path planning. We compare it to a classical aggregative weighted sum of the multiple constraints and to a manual planning thanks to a retrospective study performed by a neurosurgeon on 14 DBS cases. The results show that the dominance-based method is preferred over manual planning, and covers a larger choice of relevant optimal entry points than the traditional weighted sum approach which discards interesting solutions that could be preferred by surgeons.

Epilepsy is curable if the epileptogenic zone (EZ) can be identified within the brain and resected. Intracranial depth electrodes help identify the EZ and also map cortical function. In current clinical practice, 7–12 electrode trajectories typically needed, and are planned manually, requiring 2–3 h. Automated methods can reduce planning time and improve safety by computing suitable trajectories. We present anatomy driven multiple trajectory planning (ADMTP) to compute safe trajectories from anatomical regions of interest(ROIs). Trajectories are computed by (1) identifying targets within deep ROIs, (2) finding trajectories that traverse superficial ROIs and avoid critical structures (blood vessels, sulci), and (3) determining a feasible configuration of trajectories. ADMTP was evaluated on 20 patients (186 electrodes). Compared to manual planning, ADMTP lowered risk in $$78\,\%$$ of trajectories and increased GM sampling in $$56\,\%$$ of trajectories. ADMTP is computationally efficient, computing between 7–12 trajectories in 61 (15–279) s.

We apply recurrent neural networks to the task of recognizing surgical activities from robot kinematics. Prior work in this area focuses on recognizing short, low-level activities, or gestures, and has been based on variants of hidden Markov models and conditional random fields. In contrast, we work on recognizing both gestures and longer, higher-level activites, or maneuvers, and we model the mapping from kinematics to gestures/maneuvers with recurrent neural networks. To our knowledge, we are the first to apply recurrent neural networks to this task. Using a single model and a single set of hyperparameters, we match state-of-the-art performance for gesture recognition and advance state-of-the-art performance for maneuver recognition, in terms of both accuracy and edit distance. Code is available at https://github.com/rdipietro/miccai-2016-surgical-activity-rec.

It is clinically important to accurately predict facial soft tissue changes prior to orthognathic surgery. However, the current simulation methods are problematic, especially in clinically critical regions. We developed a two-stage finite element method (FEM) simulation model with realistic tissue sliding effects. In the 1st stage, the facial soft-tissue-change following bone movement was simulated using FEM with a simple sliding effect. In the 2nd stage, the tissue sliding effect was improved by reassigning the bone-soft tissue mapping and boundary condition. Our method has been quantitatively and qualitatively evaluated using 30 patient datasets. The two-stage FEM simulation method showed significant accuracy improvement in the whole face and the critical areas (i.e., lips, nose and chin) in comparison with the traditional FEM method.

A novel hand-held speed-of-sound (SoS) imaging method is proposed, which requires only minor hardware extensions to conventional ultrasound (US) B-mode systems. A hand-held reflector is used as a timing reference for US signals. A robust reflector-detection algorithm, based on dynamic programming (DP), achieves unambiguous timing even with 10 dB signal-to-noise ratio in real tissues, successfully detecting delays <100 ns introduced by SoS heterogeneities. An Anisotropically-Weighted Total-Variation (AWTV) regularization based on L1-norm smoothness reconstruction is shown to achieve significant improvements in the delineation of focal lesions. The Contrast-to-noise-ratio (CNR) is improved from 15 dB to 37 dB, and the axial resolution loss from >300 % to <15 %. Experiments with breast-mimicking phantoms and ex-vivo liver samples showed, for hard hypoechogenic inclusions not visible in B-mode US, a high SoS contrast (2.6 %) with respect to cystic inclusions (0.9 %) and the background SoS noise (0.6 %). We also tested our method on a healthy volunteer in a preliminary in-vivo test. The proposed technique demonstrates potential for low-cost and non-ionizing screening, as well as for diagnostics in daily clinical routine.

Biopsy under B-mode transrectal ultrasound (TRUS) is the gold standard for prostate cancer diagnosis. However, B-mode US shows only the boundary of the prostate, therefore biopsy is performed in a blind fashion, resulting in many false negatives. Although MRI or TRUS-MRI fusion is more sensitive and specific, it may not be readily available across a broad population, and may be cost prohibitive. In this paper, a limited-angle transmission US methodology is proposed, here called US tomosynthesis (USTS), for prostate imaging. This enables quantitative imaging of the prostate, such as generation of a speed of sound (SOS) map, which theoretically may improve detection, localization, or characterization of cancerous prostate tissue. Prostate USTS can be enabled by adding an abdominal probe aligned with the transrectal probe by utilizing a robotic arm. In this paper, we elaborate proposed methodology; then develop a setup and a technique to enable ex vivo USTS imaging of human prostate immediately after prostatectomy. Custom hardware and software were developed and implemented. Mock ex vivo prostate and lesions were made by filling a mold cavity with water, and adding a plastisol lesion. The time of flights were picked using a proposed center of mass method and corrected manually. The SOS map with a difference expectation-maximization reconstruction performed most accurately, with 2.69 %, 0.23 %, 0.06 % bias in estimating the SOS of plastisol, water, and mold respectively. Although USTS methodology requires further ex vivo validation, USTS has the potential to open up a new window in quantitative low-cost US imaging of the prostate which may meet a public health need.

Photoacoustic (PA) imaging requires channel data acquisition synchronized with a laser firing system. Unfortunately, the access to these channel data is only available on specialized research systems, and most clinical ultrasound scanners do not offer an interface to obtain this data. To broaden the impact of clinical PA imaging, we propose a vendor-independent PA imaging system utilizing ultrasound post-beamformed radio frequency (RF) data, which is readily accessible in some clinical scanners. In this paper, two PA beamforming algorithms that use the post-beamformed RF data as the input are introduced: inverse beamforming, and synthetic aperture (SA) based re-beamforming. Inverse beamforming recovers the channel data by taking into account the ultrasound beamforming delay function. The recovered channel data can then be used to reconstruct a PA image. SA based re-beamforming algorithm regards the defocused RF data as a set of pre-beamformed RF data received by virtual elements; an adaptive synthetic aperture beamforming algorithm is applied to refocus it. We demonstrated the concepts in simulation, and experimentally validated their applicability on a clinical ultrasound scanner using a pseudo-PA point source and in vivo data. Results indicate the full width at the half maximum (FWHM) of the point target using the proposed inverse beamforming and SA re-beamforming were 1.33 mm, and 1.08 mm, respectively. This is comparable to conventional delay-and-sum PA beamforming, for which the measured FWHM was 1.49 mm.

4D ultrasound imaging of the fetal heart relies on reconstructions from B-mode images. In the presence of fetal or mother’s motion, current approaches suffer from artifacts. We propose to use many sweeps and exploit the resulting redundancy to recover from motion by reconstructing a 4D image which is consistent in phase, space and time. We first quantified the performance of 7 formulations on simulated data. Reconstructions of the best and baseline approach were then visually compared for 10 in-vivo sequences. Ratings from 4 observers showed that the proposed consistent reconstruction significantly improved image quality.

Ultrasound (US) imaging is recommended for early detection of developmental dysplasia of the hip (DDH), which includes a spectrum of hip joint abnormalities in infants. However, the currently standard 2-dimensional (2D) US-based approach to measuring the dysplasia metric (DM), namely the $$\alpha $$ angle, suffers from high within-hip variability with standard deviations typically ranging between $$3^{\circ }-7^{\circ }$$. Such high variability leads to elevated over- and under-treatment rates in hip classification. To reduce this high variability inherent to the 2D $$\alpha $$ angle, $$\alpha _{2D}$$, we propose a 3D US-based DM in the form of a 3D $$\alpha $$ angle, $$\alpha _{3D}$$, that more accurately characterizes the morphology of an infant’s hip joint. Our method leverages phase symmetry features that automatically identify the 3D bone/cartilage structures to compute $$\alpha _{3D}$$. Validating on 30 clinical patient hip examinations, we demonstrate the within-hip variability of $$\alpha _{3D}$$ to be significantly smaller than $$\alpha _{2D}$$ ($$28.9\,\%$$ reduction, $$p<0.01$$). Our findings indicate that $$\alpha _{3D}$$ may be significantly more reproducible than the conventional 2D measure, which will likely reduce misclassification rates.

The goal of this paper is to develop a computer-aided diagnostic (CAD) system for early detection of prostate cancer from diffusion-weighted magnetic resonance imaging (DW-MRI) acquired at different b-values. The proposed system consists of three main steps. First, the prostate is segmented using a hybrid framework that integrates geometric deformable model (level-sets) and nonnegative matrix factorization (NMF). Secondly, the apparent diffusion coefficient (ADC) of the segmented prostate volume is first estimated at different b-values and is then normalized and refined using a generalized Gauss-Markov random field (GGMRF) image model. Then, the cumulative distribution function (CDF) of the refined ADCs at different b-values are constructed. Finally, a two-stage structure of stacked non-negativity constraint auto-encoder (SNCAE) is trained to classify the prostate tumor as benign or malignant based on the constructed CDFs. In the first stage, classification probabilities are estimated at each b-value and in the second stage, those probabilities are fused and fed into the prediction stage SNCAE to calculate the final classification. Preliminary experiments on 53 clinical DW-MRI datasets resulted in $$98.11\,\%$$ correct classification (sensitivity $$=96.15\,\%$$ and specificity = $$100\,\%$$), indicating the high performance of the proposed CAD system and holding promise of the proposed system as a reliable non-invasive diagnostic tool.

The number density of scatterers in tumor tissue contribute to a heterogeneous ultrasound speckle pattern that can be difficult to discern by visual observation. Such tumor stochastic behavior becomes even more challenging if the tumor texture heterogeneity itself is investigated for changes related to response to chemotherapy treatment. Here we define a new tumor texture heterogeneity model for evaluating response to treatment. The characterization of the speckle patterns is performed via state-of-the-art multi-orientation and multi-scale circular harmonic wavelet (CHW) frames analysis of the envelope of the radio-frequency signal. The lacunarity measure – corresponding to scatterer number density – is then derived from fractal dimension texture maps within the CHW decomposition, leading to a localized quantitative assessment of tumor texture heterogeneity. Results indicate that evaluating tumor heterogeneity in a multidimensional texture analysis approach could potentially impact on designing an early and effective chemotherapy treatment.

In this paper, we study the correlation of tissue (i.e. prostate) elasticity with the spread and aggression of prostate cancers. We describe an improved, in-vivo method that estimates the individualized, relative tissue elasticity parameters directly from medical images. Although elasticity reconstruction, or elastograph, can be used to estimate tissue elasticity, it is less suited for in-vivo measurements or deeply-seated organs like prostate. We develop a non-invasive method to estimate tissue elasticity values based on pairs of medical images, using a finite-element based biomechanical model derived from an initial set of images, local displacements, and an optimization-based framework. We demonstrate the feasibility of a statistically-based multi-class learning method that classifies a clinical T-stage and Gleason score using the patient’s age and relative prostate elasticity values reconstructed from computed tomography (CT) images.

Estrogren and progesterone hormone receptor status play a role in the treatment planning and prognosis of breast cancer. These are typically found after Immuno-Histo-Chemistry (IHC) analysis of the tumor tissues after surgery. Since breast cancer and hormone receptor status affect thermographic images, we attempt to estimate the hormone receptor status before surgery through non-invasive thermographic imaging. We automatically extract novel features from the thermographic images that would differentiate hormone receptor positive tumors from hormone receptor negative tumors, and classify them though machine learning. We obtained a good accuracy of 82 % and 79 % in classification of HR$$+$$ and HR− tumors, respectively, on a dataset consisting of 56 subjects with breast cancer. This shows a novel application of automatic thermographic classification in breast cancer prognosis.

Despite recent advances in clinical oncology, prostate cancer remains a major health concern in men, where current detection techniques still lead to both over- and under-diagnosis. More accurate prediction and detection of prostate cancer can improve disease management and treatment outcome. Temporal ultrasound is a promising imaging approach that can help identify tissue-specific patterns in time-series of ultrasound data and, in turn, differentiate between benign and malignant tissues. We propose a probabilistic-temporal framework, based on hidden Markov models, for modeling ultrasound time-series data obtained from prostate cancer patients. Our results show improved prediction of malignancy compared to previously reported results, where we identify cancerous regions with over 88 % accuracy. As our models directly represent temporal aspects of the data, we expect our method to be applicable to other types of cancer in which temporal-ultrasound can be captured.

We propose a cancer grading approach for transrectal ultrasound-guided prostate biopsy based on analysis of temporal ultrasound signals. Histopathological grading of prostate cancer reports the statistics of cancer distribution in a biopsy core. We propose a coarse-to-fine classification approach, similar to histopathology reporting, that uses statistical analysis and deep learning to determine the distribution of aggressive cancer in ultrasound image regions surrounding a biopsy target. Our approach consists of two steps; in the first step, we learn high-level latent features that maximally differentiate benign from cancerous tissue. In the second step, we model the statistical distribution of prostate cancer grades in the space of latent features. In a study with 197 biopsy cores from 132 subjects, our approach can effectively separate clinically significant disease from low-grade tumors and benign tissue. Further, we achieve the area under the curve of 0.8 for separating aggressive cancer from benign tissue in large tumors.

Computed tomography imaging is a standard modality for detecting and assessing lung cancer. In order to evaluate the malignancy of lung nodules, clinical practice often involves expert qualitative ratings on several criteria describing a nodule’s appearance and shape. Translating these features for computer-aided diagnostics is challenging due to their subjective nature and the difficulties in gaining a complete description. In this paper, we propose a computerized approach to quantitatively evaluate both appearance distinctions and 3D surface variations. Nodule shape was modeled and parameterized using spherical harmonics, and appearance features were extracted using deep convolutional neural networks. Both sets of features were combined to estimate the nodule malignancy using a random forest classifier. The proposed algorithm was tested on the publicly available Lung Image Database Consortium dataset, achieving high accuracy. By providing lung nodule characterization, this method can provide a robust alternative reference opinion for lung cancer diagnosis.