Tensors in Image Processing and Computer Vision

Tensors have been broadly used in mathematics and physics, since they are a generalization of scalars or vectors and allow to represent more complex properties. In this chapter we present an overview of some tensor applications, especially those focused on the image processing field. From a mathematical point of view, a lot of work has been developed about tensor calculus, which obviously is more complex than scalar or vectorial calculus. Moreover, tensors can represent the metric of a vector space, which is very useful in the field of differential geometry. In physics, tensors have been used to describe several magnitudes, such as the strain or stress of materials. In solid mechanics, tensors are used to define the generalized Hooke’s law, where a fourth order tensor relates the strain and stress tensors. In fluid dynamics, the velocity gradient tensor provides information about the vorticity and the strain of the fluids. Also an electromagnetic tensor is defined, that simplifies the notation of the Maxwell equations. But tensors are not constrained to physics and mathematics. They have been used, for instance, in medical imaging, where we can highlight two applications: the diffusion tensor image, which represents how molecules diffuse inside the tissues and is broadly used for brain imaging; and the tensorial elastography, which computes the strain and vorticity tensor to analyze the tissues properties. Tensors have also been used in computer vision to provide information about the local structure or to define anisotropic image filters.

The segmentation of tensor-valued images or 3D volumes is a relatively recent issue in image processing, but a significant effort has been made in the last years. Most of this effort has been focused on the segmentation of anatomical structures from DT-MRI (Diffusion Tensor Magnetic Resonance Imaging), and some contributions have also been made for the segmentation of 2D textured images using the Local Structure Tensor (LST). In this chapter, we carefully review the state of the art in the segmentation of tensor fields. We will discuss the main approaches that have been proposed in the literature, with particular emphasis on the importance of the different tensor dissimilarity measures. Also, we will highlight the key limitations of the segmentation techniques proposed so far, and will provide some insight on the directions of current research.

A variational framework for the registration of tensor-valued images is presented. The underlying energy functional consists of four terms: a data term modelled on a tensor constancy constraint, a compatibility term which couples domain deformations and tensor reorientation on the basis of a physical model, and regularity terms imposing smoothness of deformation and tensor reorientation fieldss in space. A specific feature of our model is the separation of data and compatibility terms which eases an adaptation to different physical models of tensor deformation. A multiscale gradient descent is used to minimise the energy functional with repect to both transformation fields involved. The viability and potential of the approach in the registration of tensor-valued images is demonstrated by experiments.

The evaluation of tensor image processing algorithms is an open problem that has not been broadly handled, and specific measures have not been described to assess the quality of tensor images. In this chapter, we propose the adaptation of quality measures that have been defined in the case of conventional scalar images to the tensor case, in order to evaluate the quality of the tensor images that are most frequently used in the image processing field. Special attention is paid to the tensor features that made this extension no straightforward. Some general concepts that should be taken into account for the definition of quality indexes for tensor images based on the well-known measures for conventional scalar images are detailed. Then, some of these measures are adapted to deal with tensor images and their behavior is analyzed by means of some examples. Thus, it is shown that structure based measures outperform point-wise measures, as well as the influence of handling all the tensor components.

Nonnegative Matrix Factorization (NMF) is a decomposition which incorporates nonnegativity constraints in both the weights and the bases of the representation. The nonnegativity constraints in NMF correspond better to the intuitive notion of combining parts in order to create a complete object, since the object is represented using only additions of weighted nonnegative basis images. NMF has proven to be very successful for image analysis, especially for imaged-based object representation, discovery of latent object variables and recognition. A drawback of NMF is that it requires the object tensor (with valence more than one) to be vectorized. This procedure may result in information loss since the local object structure is lost due to vectorization. Recently, in order to remedy this disadvantage of NMF methods, Nonnegative Tensor Factorization (NTF) algorithms that can be applied directly to the tensor representation of object collections, have been introduced. In this chapter, we demonstrate how various algorithms are formulated in order to treat arbitrary valence NTFs and we present the various cost functions that have been used for measuring the quality of the approximation. We discuss the optimization procedures that have been used for deriving the factors of the decomposition. Afterwards, we describe how additional constraints can be incorporated into the cost of the decomposition in order to either enhance the sparsity of the solution or to enhance the discrimination between object classes. The presented NTF schemes are described in a manner that can be easily implemented using, in most cases, only matrix multiplications and publicly available packages for treating tensor representations. Finally, we comment on the various applications of NTF algorithms in visual representation and recognition.

Tensor fields are important in digital imaging and computer vision. Hence there is a demand for morphological operations to perform e.g. shape analysis, segmentation or enhancement procedures. Recently, fundamental morphological concepts have been transferred to the setting of fields of symmetric positive definite matrices, which are symmetric rank two tensors. This has been achieved by a matrix-valued extension of the nonlinear morphological partial differential equations (PDEs) for dilation and erosion known for grey scale images. Having these two basic operations at our disposal, more advanced morphological operators such as top hats or morphological derivatives for matrix fields with symmetric, positive semidefinite matrices can be constructed. The approach realises a proper coupling of the matrix channels rather than treating them independently. However, from the algorithmic side the usual scalar morphological PDEs are transport equations that require special upwind-schemes or novel high-accuracy predictor-corrector approaches for their adequate numerical treatment. In this chapter we propose the non-trivial extension of these schemes to the matrix-valued setting by exploiting the special algebraic structure available for symmetric matrices. Furthermore we compare the performance and juxtapose the results of these novel matrix-valued high-resolution-type (HRT) numerical schemes by considering top hats and morphological derivatives applied to artificial and real world data sets.

In 3D image processing tensors play an important role. While rank-1 and rank-2 tensors are well understood and commonly used, higher rank tensors are rare. This is probably due to their cumbersome rotation behavior which prevents a computationally efficient use. In this chapter we want to introduce the notion of a spherical tensor which is based on the irreducible representations of the 3D rotation group. In fact, any ordinary cartesian tensor can be decomposed into a sum of spherical tensors, while each spherical tensor has a quite simple rotation behavior. We introduce so called tensorial harmonics that provide an orthogonal basis for spherical tensor fields of any rank. It is just a generalization of the well known spherical harmonics. Additionally we propose a spherical derivative which connects spherical tensor fields of different degree by differentiation. Based on the proposed theory we present two applications. We propose an efficient algorithm for dense tensor voting in 3D, which makes use of tensorial harmonics decomposition of the tensor-valued voting field. In this way it is possible to perform tensor voting by linear-combinations of convolutions in an efficient way. Secondly, we propose an anisotropic smoothing filter that uses a local shape and orientation adaptive filter kernel which can be computed efficiently by the use spherical derivatives.

The structure of images has been studied for decades and the use of local structure tensor fields appeared during the eighties [3, 14, 6, 9, 11]. Since then numerous varieties of tensors and estimation schemes have been developed. Tensors have for instance been used to represent orientation [7], velocity, curvature [2] and diffusion [19] with applications to adaptive filtering [8], motion analysis [10] and segmentation [17]. Even though sampling in non-Cartesian coordinate system are common, analysis and processing of local structure tensor fields in such systems is less developed. Previous work on local structure in non-Cartesian coordinate systems include [21, 16, 1, 18].

General camera models relax the constraint on central projection and characterize cameras as mappings between each pixel and the corresponding projection rays. This allows to describe most cameras types, including classical pinhole cameras, cameras with various optical distortions, catadioptric cameras and other acquisition devices. We deal with the structure from motion problem for such general models. We first consider an hierarchy of general cameras first introduced in [28] where the cameras are described according to the number of points and lines that have a non-empty intersection with all the projection rays. Then we propose a study of the multi-view geometry of such cameras and a new formulation of multi-view matching tensors working for projection rays crossing the same 3D line, the counterpart of the fundamental matrices and the multifocal tensors of the standard perspective cameras. We also delineate a method to estimate such tensors and recover the motion between the views.

In this chapter, we propose an approach to full-body pose recognition and body orientation estimation using multilinear analysis. We extract low-dimensional pose and body orientation coefficient vectors by performing tensor decomposition and projection on silhouette images obtained from wide baseline binocular cameras. The coefficient vectors are then used as feature vectors in pose recognition and body orientation estimation. To do pose recognition, pose coefficient vectors obtained from synthesized pose silhouettes are used to train a family of support vector machines as pose classifiers. Using orientation coefficient vectors, a 1-D orientation manifold is learned and further used for the estimation of body orientation. Experiment results obtained using both synthetic and real image data showed that the performance of our approach is comparable to existing pose recognition approaches, and that our approach outperformed the traditional tensor-based recognition approach in the comparative test.

This chapter is devoted to applications of multiview tensors, in higher dimension, to projective recostruction of segmented or dynamic scenes. Particular emphasis is placed on the analysis of critical configurations and their loci in this context, i.e. configurations of chosen scene-points and cameras that turn out to prevent successful reconstruction or allow for multiple possible solutions giving rise to ambiguities. A general geometric set up for higher dimensional spaces ad projections is firstly recalled. Examples of segmented and dynamic scenes, interpreted as static scenes in higher dimensional projective spaces, are then considered, following Shashua and Wolf. A theoretical approach to multiview tensors in higher dimension is presented, according to Hartley and Schaffalitzky. Using techniques of multilinear algebra and proper formalized language of algebraic geometry, a complete description of the geometric structure of the loci of critical configurations in any dimension is given. Supporting examples are supplied, both for reconstruction from one view and from multiple views. In an experimental context, the following two cases are realized as static scenes in P^4: 3D points lying on two bodies moving relatively to each other by pure translation and 3D points moving independently along parallel straight lines with constant velocities. More explicitly, algorithms to determine suitable tensors used to reconstruct a scene in P^4: from three views are implemented with MATLAB. A number of simulated experiments are finally performed in order to prove instability of reconstruction near critical loci in both cases described above.

In this chapter we give an account of two different methods to find constraints for the trifocal tensor Т, used in geometric computer vision. We also show how to single out a set of only eight equations that are generically complete, i.e. for a generic choice of Т, they suffice to decide whether Т is indeed trifocal. Note that eight is minimum possible number of constraints.

Image registration is a common image processing task, and therefore, many algorithms have been proposed and described to carry it out for different image modalities. However, the application of these algorithms to diffusion tensor imaging is not straightforward due to the special features of this kind of data, where a tensor is defined at each voxel. The information provided by the diffusion tensor is related to the anatomical structures in tissues, and this relation should be preserved, even though the image has been transformed by a registration procedure. On the other hand, the registration problem can be viewed as an optimization problem, where a similarity measure has to be maximized. The appropriate definition of this similarity measure is indeed an important issue for the registration of diffusion tensor images. In this paper, we compile the different approaches for the registration of diffusion tensor images that have been published. Special attention is paid to the aforementioned topics: how to preserve the coherence between the tensor and the underlying tissue structure, and how to measure the similarity between two diffusion tensors. Methods to evaluate results are also reviewed, since a reliable validation leads to more conclusive results, specially in the comparison of different techniques. Most challenging issues for diffusion tensor images registration are underlined, and open research lines about this topic are pointed out.

Abstract Recent work has outlined a framework for analyzing diffusion tensor gradient and covariance tensors in terms of invariant gradient and rotation tangents, which span local variations in tensor shape and orientation, respectively. This chapter hopes to increase the adoption of this framework by giving it a more intuitive conceptual description, as well as providing practical advice for its numeric implementation. Example applications are described, with an emphasis on decomposing the third-order gradient of a diffusion tensor field.

Diffusion MRI, which is sensitive to the Brownian motion of molecules, has become today an excellent medical tool for probing the tissue micro-structure of cerebral white matter in vivo and non-invasively. It makes it possible to reconstruct fiber pathways and segment major fiber bundles that reflect the structures in the brain which are not visible to other non-invasive imaging modalities. Since this is possible without operating on the subject, but by integrating partial information from Diffusion Weighted Images into a reconstructed ‘complete’ image of diffusion, Diffusion MRI opens a whole new domain of image processing. Here we shall explore the role that tensors play in the mathematical model. We shall primarily deal with Cartesian tensors and begin with 2nd order tensors, since these are at the core of Diffusion Tensor Imaging. We shall then explore higher and even ordered symmetric tensors, that can take into account more complex micro-geometries of biological tissues such as axonal crossings in the white matter.

Abstract Diffusion Tensor MRI (DTI) is a special MR imaging technique where the second order symmetric diffusion tensors that are correlated with the underlying fi-brous structure (eg. the nerves in brain), are computed based on DiffusionWeighted MR Images (DWI). DTI is the only in vivo imaging technique that provides information about the network of nerves in brain. The computed tensors describe the local diffusion pattern of water molecules via a 3D Gaussian distribution in space. The most common analysis and visualization technique is tractography, which is a numerical integration of the principal diffusion direction (PDD) that attempts to reconstruct fibers as streamlines. Despite its simplicity and ease of interpretation, tractography algorithms suffer from several drawbacks mainly due to ignoring the information in the underlying spatial distribution but using the PDD only. An alternative to tractography is connectivity which aims at computing probabilistic connectivity maps based on the above mentioned 3D Gaussian distribution as described by the DTI data. However, the computational cost is high and the resulting maps are usually hard to visualize and interpret. This chapter discusses these two approaches and introduces two new tractography techniques, namely the Lattice-of-Springs (LoS) method that exploits the connectivity approach and the Split & Merge Tractography (SMT) that attempts to combine the advantages of tractography and connectivity.

In this work we propose an alternative method to estimate and visualize the Strain Rate Tensor (SRT) in Magnetic Resonance Images (MRI) when Phase Contrast MRI (PCMRI) and Tagged MRI (TMRI) are not available. This alternative is based on image processing techniques. Concretely, image registration algorithms are used to estimate the movement of the myocardium at each point. Additionally, a consistency checking method is presented to validate the accuracy of the estimates when no golden standard is available. Results prove that the consistency checking method provides an upper bound of the mean squared error of the estimate. Our experiments with real data show that the registration algorithm provides a useful deformation field to estimate the SRT fields. A classification between regional normal and dysfunctional contraction patterns, as compared with experts diagnosis, points out that the parameters extracted from the estimated SRT can represent these patterns. Additionally, a scheme for visualizing and analyzing the local behavior of the SRT field is presented.

Elastography measures the elastic properties of soft tissues using principally ultrasound (US) or magnetic resonance (MR) signals. The elastic behavior of tissues can be analyzed with tensor signal processing. Different approaches have been developed to estimate and image the elastic properties in the tissue. In ultrasound elastography, the estimation of the displacement and strain fields is mostly based on measures computed from the Radio Frequency signals, such as time-domain cross-correlation. We propose to estimate the displacement field from two consecutive B-mode images using a multiscale optical flow method. The tensor strain field can then be plotted as ellipsoids, visualizing in a single image the standard scalar parameters that are usually represented separately. This technique can offer physicians the possibility of extracting new discriminant and useful parameters related to the elastic behavior of tissues. Although clinical validation is still needed, our experiments from finite element and ultrasound simulations display consistent and reliable results.

Abstract This chapter describes a framework for storage of tensor array data, useful to describe regularly sampled tensor fields. The main component of the framework, called Similar Tensor Array Core (STAC), is the result of a collaboration between research groups within the SIMILAR network of excellence. It aims to capture the essence of regularly sampled tensor fields using a minimal set of attributes and can therefore be used as a “greatest common divisor” and interface between tensor array processing algorithms. This is potentially useful in applied fields like medical image analysis, in particular in Diffusion Tensor MRI, where misinterpretation of tensor array data is a common source of errors. By promoting a strictly geometric perspective on tensor arrays, with a close resemblance to the terminology used in differential geometry, (STAC) removes ambiguities and guides the user to define all necessary information. In contrast to existing tensor array file formats, it is minimalistic and based on an intrinsic and geometric interpretation of the array itself, without references to other coordinate systems.

Nowadays there is a growing interest in tensor medical imaging modalities. In Diffusion Tensor Magnetic Resonance Imaging (DT-MRI), each pixel is valued with a symmetric second-order tensor describing the spatial properties of diffusion at that point. Therefore, it provides significantly more information than scalar modalities, but this causes the complexity of the interfaces dealing with them to grow. In this chapter, the current situation of user interfaces for tensor fields is reviewed. Tensor user interfaces are difficult to design, given the difficulty of mentally integrating data with so many parameters. This is why a considerable effort must be invested in order to achieve intuitive and easy-to-use interfaces. The display of tensor information plays an important role in this, and we review several existing visualization methods for tensor fields.We must point out that, although most of the applications are graphical interfaces, there are also examples of command-line tools and multimodal interfaces employing virtual environments. We study some of the urrent medical user interfaces for diffusion tensor fields.

abstract Tensor valued data are frequently used in medical imaging. For a 3-dimensional second order tensor such data imply at least six degrees of freedom for each voxel. The operators ability to perceive this information is of outmost importance and in many cases a limiting factor for the interpretation of the data. In this paper we propose a decomposition of such tensor fields using the Tflash tensor glyphs that intuitively conveys important tensor features to a human observer. A matlab implementation for visualization of single tensors are described in detail and a VTK/ITK implementation for visualization of tensor fields have been developed as a Medical Studio component.