Brain-inspired algorithms for retinal image analysis

Multi-scale analysis is now well established [15, 44, 58]. The center-surround receptive fields in the retina are the first multi-scale sampling step. Our model for V1’s ‘simple cells’ (so called by Hubel and Wiesel) is that these have the function of multi-scale, regularized spatial differential operators, i.e., Gaussian derivatives, possibly up to fourth order [44, 58‐60]. The simultaneous sampling of the multi-scale differential structure leads to ‘deep structure’ analysis [43], with applications as edge focusing, hierarchical top points and SIFT and SURF keypoints. Nonlinear adaptive multi-scale analysis is developed in the rich field of geometry-driven diffusion [32, 62], which introduced nonlinear PDEs and energy minimizing variational methods into this field of evolutionary geometric computing.

Hubel and Wiesel [37] discovered that the receptive fields in cat’s striate cortex have a strong orientation-selective property (see Fig. 2). A so-called cortical hyper-column, with the characteristic pinwheel structure of equi-orientation lines radiating from a central singularity, can be interpreted as a visual pixel computer, neatly decomposed into a complete set of orientations.

Synaptic physiological studies of horizontal pathways in cat striate cortex show that neurons with aligned receptive field sites excite each other on the cortex between different pinwheels, in an elongated area and over long distances, creating long-range contextual connections [2]. Therefore, the visual system not only constructs the aforementioned score of local orientations, but also accounts for long-range context and alignment by excitation and inhibition a priori, which can be modeled by so-called association fields [15, 20, 27], and left-invariant PDEs and ODEs for contour enhancement and contour completion directly on the score [23, 61].

This multi-orientation framework [14, 15, 21, 31] also allows us to generically deal with crossings, as we will show with the application to vessel tracking [6, 8, 9, 52] and segmentation [1, 33, 65]. Moreover, due to the neat organization of image data on the Lie-group SE(2), we are able to design effective detection algorithms [6, 7], geometric feature analysis techniques such as bifurcation detection/analysis [55] and local vessel curvature analysis [11], and enhancement methods [31, 66].

The domain \(\mathbb {R}^2 \rtimes S^1\) (2 spatial dimensions, and one orientation dimension) of an orientation score can be identified with the Euclidean motion group SE(2), equipped with the group productfor all \(g,g' \in \hbox {SE}(2)\), and with \(\textstyle \mathbf R _\theta = \left( \begin{array}{ccc} \cos \theta &{}\quad -\sin \theta \\ \sin \theta &{}\quad \cos \theta \\ \end{array} \right) \) a counterclockwise rotation over angle \(\theta \).

An orientation score \(U_f:{\hbox {SE}}(2) \rightarrow \mathbb {C}\) is obtained by convolving an input image f with a specially designed, anisotropic wavelet \(\psi \):where \(R_\theta \) denotes a 2D counterclockwise rotation matrix and \(\psi _\theta (\mathbf {x}) = \psi (R_\theta ^{-1}\mathbf {x})\). Exact image reconstruction is achieved bywhere \(\mathcal {F}[\cdot ]\) represents the unitary Fourier transform on \(\mathbb {L}_2(\mathbb {R}^2)\) and \(M_\psi \) is given by \(\int _0^{2\pi }|\mathcal {F}\left[ \psi _\theta \right] |^2 \mathrm {d}\theta \).

Well-posedness of the reconstruction is controlled by \(M_\Psi \) [10, 24]. \(M_\Psi \): \(\mathbb {R}^{2} \rightarrow \mathbb {R}^{+}\) is calculated byThe function \(M_\Psi \) provides a stability measure of the inverse transformation. Theoretically, reconstruction is well-posed, as long as \(0< \delta< M < M_\Psi<\), where \(\delta \) is arbitrarily small, since then the condition number of \(W_\Psi \) is bounded by \(M\delta ^{1}\) (for a detailed discussion see [22]).

Wavelets that meet this criterion are the so-called cake wavelets described by [6, 21, 31]. They uniformly cover the Fourier domain up to a radius of about the Nyquist frequency \(\rho _n\) (see Fig. 3). They have the advantage over other oriented wavelets (such as Gabor wavelets) that they allow for a stable inverse transformation \({\mathcal {W}}_\psi ^*\) from the orientation score back to the image. As such, cake wavelets ensure that no data-evidence is lost during the forward and backward transformation. The procedure for creating cake wavelets is illustrated in Fig. 3. In detail, the proposed wavelet takes the formwith \(G_{\hbox {s}}\) an isotropic Gaussian window in the spatial domain and \((\rho ,\varphi )^T\) denote polar coordinates in the Fourier domain, i.e., \(\mathbf {\omega }= (\rho \cos \varphi ,\rho \sin \varphi )^T\). The Fourier wavelet \(\tilde{\psi }(\mathbf {\omega }) = A(\varphi )B(\rho )\) has an angular component \(A(\varphi )\) resembling a ‘piece of cake’ (Fig. 3), and a Gaussian weighted radial component \(B(\rho )\):where \(B^k(x)\) denotes the kth order B-spline, \(N_\theta \) is the number of samples in the orientation direction and \(s_\theta =2\pi /N_\theta \) is the angular step size. The function \(B(\rho )\) is a Gaussian multiplied with the Taylor series of its inverse up to order N to enforce faster decay. The parameter t is given by \(t^2=2 \hat{\rho }/(1+2N)\) with the inflection point \(\hat{\rho }\) that determines the bending point of \(B(\rho )\).

As depicted in Fig. 3, the symmetric real part of the kernel picks up lines, whereas the asymmetric imaginary part responds to edges. In the context of vessel filtering, the real part is of primary interest, whereas vessel tracking in the orientation score additionally makes use of the imaginary part [6]. Elongated structures involved in a crossing can now be disentangled for crossing-preserving analysis.

Theoretically, because of the curved geometry of orientation scores, it is wrong to take the derivatives in orientation scores using the \(\{\partial _x,\partial _y,\partial _\theta \}\) derivative frame (we use shorthand notation \(\partial _i=\frac{\partial }{\partial _i}\)) [21]. Therefore, left-invariant differential operatorsare used in SE(2). The \(\partial _\xi \) and \(\partial _\eta \) are the spatial derivatives tangent and orthogonal to the orientation \(\theta \). It is important to mention that not all the left-invariant derivatives commute, e.g., \(\partial _\theta \partial _\xi U \ne \partial _\xi \partial _\theta U\) [22, 31]. Mathematically this is due to the fact that rotations and translations do not commute. We have the following commutators in the Lie-algebra of SE(2):Suitable combinations of derivatives have been widely used to pick up differential invariant features like edges, ridges, corners and so on [59]. However, obtaining derivatives directly is an ill-posed problem. Therefore, we regularize the orientation scores via convolutions with Gaussian kernels \(G_{\sigma _s,\sigma _o}(\mathbf x ,\theta )=G_{\sigma _s}(\mathbf x )G_{\sigma _o}(\theta )\), with a d—dimensional Gaussian given byand where \(\sigma _s\) and \(\sigma _o\) are used to define the spatial scale \(\frac{1}{2}\sigma _s^2\) and orientation scale \(\frac{1}{2}\sigma _o^2\) of the Gaussian kernel. Note that the spatial Gaussian distribution \(G_{\sigma _s}: \mathbb {R}^{2} \rightarrow \mathbb {R}^{+}\) must be isotropic to preserve commutator relations of the SE(2) group for scales \(\sigma _{s}>0\), i.e., to preserve left-invariance.

Detailed numerical approaches for linear left-invariant diffusions on SE(2) have been developed in [66].

Sometimes retinal images have a low quality, e.g., due to cataract or other pathology. To exclude non-diagnostic images automatically in the high-volume screening process, a method for image quality verification is developed [26], based on [50]. The supervised method is based on the assumption that sufficient image structure according to a pre-defined distribution must be available. Geometric differential invariants (expressed in 2D gauge coordinates [58]) up to second order and at 4 different scales are used to develop response vectors. Different combinations of features are trained by different classifiers on 100 normal and 100 low-quality images. The ground truth for normal or low quality images was specified by two expert ophthalmologists. Combining the image structure clusters with RGB color histogram features, the Random Forest classifier proved to be the best classifier, with a performance of 0.984 area under the curve (AUC) of the receiver operator characteristic (ROC), with 0.91 accuracy rate.

All pixels outside the camera field-of-view are masked, according to the camera type. Retinal images often suffer from non-uniform illumination and varying contrast, which may affect the later detection process. We exploit the luminosity and contrast normalization procedures proposed by Foracchia et al. [28].

The optic nerve head (ONH) or optical disk is a key landmark in retinal images, and many methods for detection have been proposed [4, 18, 45, 47, 54, 64]. For an extensive and recent overview see [51]. Correct segmentation of the ONH and its rim is an important biomarker for glaucoma identification [38]), and for establishing a metric for regions-of-interest on the retina [36].

Conventional fundus images show the ONH as a bright disk-like feature. However, scanning laser ophthalmoscopy (SLO) cameras generally show dark regions with considerably less contrast. In the presence of large pathologies, classical approaches typically show decreased performance, or fail. Better performance is obtained by including contextual information, e.g., by incorporating the characteristic pattern of large blood vessel arches in the upper and lower retina emerging from the ONH [45, 46, 64], but this comes at higher computational costs.

Exploiting the extra dimensions of orientation scores, we have developed a template matching application via cross-correlation including local orientations [7, 10] (see also [16]). We can now match patterns of orientation distributions, rather than pixel intensities.

The templates are constructed from (a) a disk filter, (b) a model of the vessels radiating from the ONH, and (c) through minimization of a suitable energy functional. In Ref. [7] we found that, in contrast to model-based templates [10] (e.g., disk shapes or vessel indicator functions), the best performing templates are trained by the minimization of an energy functional, consisting of a term where the desired inner product responses are trained, and a regularization term that ensures stability and smoothness of the template. For 2D spatial image templates t, we minimizewhere \(\hat{f}_i\) is one of P (\(P=100\)) normalized positive patches with ONH (with \(y_i = 1\)), or negative patches without ONH (with \(y_i = 0\)) with typically \(\lambda = 10^{-1.5}\). The regularization term enforces smoothness by punishing the squared gradient magnitude \(||\nabla t ||^2\), and prevents over-fitting. Examples of both positive and negative training patches are given in Fig. 4.

For the orientation score template T, a similar functional is minimized:with typically \(\lambda = 10\) and \(D_{\theta \theta } = 10^{-3.5}\) and with the left-invariant gradient \( \nabla T = \left( \frac{\partial T}{\partial \xi }, \frac{\partial T}{\partial \eta }, \frac{\partial T}{\partial \theta } \right) ^T \). The templates are represented in a B-spline basis, allowing for efficient and accurate optimization of the energy functionals. The B-spline coefficients are solved by a conjugate gradient approach.

The orientation score SE(2) template method outperforms state-of-the-art methods on publicly available benchmark databases, as it correctly identifies the ONH in 99.7 % of 1737 images of the well-known MESSIDOR, STARE (with a wide variety of pathological images) and DRIVE databases. For more details, see [7, 10].

Vessel tracking has the advantage over pixel classification that it guarantees connectedness of vessel segments. We have developed tracking via orientation scores [6], so classical difficulties as crossings, bifurcations, closely parallel vessels and vessels of varying width and high curvature can be dealt with naturally.

To handle bifurcations properly, one-sided kernels are exploited, constructed by decomposition of orientations scores in two opposite directions, weighted in the radial direction with an error function (cumulative Gaussian).

Two tracking methods are developed: Edge Tracking based on Orientation Scores (ETOS) and Centerline Tracking based on multi-scale Orientation Scores (CTOS). ETOS tracks both edges of the vessel simultaneously. The edge positions are detected in the orientation score, as a local minimum (left edge) and maximum (right edge) from the anti-symmetric imaginary part of the tangent plane. CTOS exploits the fact that the disturbing vessel central light reflex is filtered out by Gabor kernels with proper scales. The invertible orientation score has some distinct advantages over Gabor filtering: much lower computational costs as the orientation score filters for all scales simultaneously, whereas the Gabor filter only calculates a single spatial frequency at a time, and the orientation score filtering is more accurate.

Overall, ETOS outperforms CTOS. ETOS gives best results when applied on invertible orientation scores, can deal with many complex geometries, and gives reliable width measurements.

The algorithm performs well: ETOS detected 76 % (290/381) of the bifurcations and 96 % (109/114) of the crossings correctly. Most mistakes in bifurcations were actually crossings, only 5 % was misclassified. For more details, see [6].

The separation of vessels in arteries and veins is crucial, as their different physiology, flow and mechanical properties respond differently to disease development, e.g., arteriolar narrowing, a decrease of the artery calibers relatively to the vein calibers is an important early biomarker for diabetic retinopathy. Also tortuosity measures are expected to be different for arteries and veins [39]. An automated artery–vein classification is required for high-volume screening.

The ratio of the arteriolar and venular diameters is called the arteriovenous ratio (AVR) and is classically computed from the six widest arteries and veins in a restricted zone around the optic disk [42].

We developed a novel method for artery/vein classification [25] based on local and contextual feature analysis of retinal vessels. Features are (a) the color, as arteries appear brighter and veins darker due to the oxygen content of the blood, (b) the transverse intensity profile, as arteries have a more pronounced central reflex to the camera flash, and (c) graph path properties of crossings and bifurcations of vessels, as these provide contextual information, because arteries never cross arteries and veins always cross arteries [17].

A non-submodular energy function is defined, integrating the features, and optimized exactly by means of graph cuts. The method was validated with a ground truth data set of 150 fundus images. An accuracy of 88.0 % was achieved for all vessels and 94.0 % for the 6 widest vessels. The contextual information especially benefits the classification of smaller vessels. For more details see [25].

The concept of fractal dimension was initially defined and developed in mathematics. It measures the complexity of self-similar objects that have the same patterns across different scales, e.g., trees and snowflakes. The vascular tree on the human retina also has self-similar branching patterns over different scales. Therefore, there is a growing interest in retinal fractal analysis, exploiting fractal dimension as a biomarker for discriminating healthy from diabetic retinopathy.

Fractal dimensions have been widely investigated. However, conflicting findings are found [3, 12]. This motivated us to especially investigate the stability and reproducibility of fractal dimension measurements.

The retinal vascular network is extracted by the method described in Sect. 4.4.1. The region-of-interest (ROI) must be carefully specified for fractal dimension calculation, because it is a global measurement and different fundus cameras have different field-of-views. The ROI is determined via firstly locating the optic nerve head by the template-matching-based method as described in Sect. 4.3, and the optic disk radii are obtained. The fovea position is found at the global minimal intensity in a small region with 5*OD radii lateral distance to the optic nerve head. For the fovea resp. ONH-centered images, the circular ROI mask is centered at the fovea centralis, resp. ONH. For the fractal dimension calculation, we used the box counting method which paves the full retinal ROI image with square boxes (tiles) with different side-length (different scales) [48]. In each box, various measurements are done: counting the number of boxes that overlap with the vessels, calculating the entropy of each box and calculating the probability of finding vessels near each pixel. Finally, these measurements are plotted against the box side-length on a log-log plot, and the fractal dimension is the slope of the fitting regression line. See Fig. 10.

We examined the stability of the fractal dimension measurements with respect to a range of variable factors: (1) different vessel annotations obtained from human observers, (2) different automatic segmentation methods, (3) different regions-of-interest, (4) different accuracy of vessel segmentation methods, and (5) different imaging modalities. Our results demonstrate that the relative errors for the measurement of fractal dimensions are significant and vary considerably according to the image quality, modality and the technique used for measuring it. So automated and semi-automated methods for the measurement of fractal dimension are not stable enough, which makes fractal dimension not a proper biomarker in quantitative clinical applications [35].

Vessel curvature is an important biomarker [13, 34, 39, 53]. Most methods require pre-segmentation and center line extraction, but our brain-inspired orientation score method [11] works directly on the image data by fitting so-called exponential curves in the orientation score. An exponential curve is a curve whose spatial projection has a constant curvature (Fig. 13).

The curvature values are computed directly from tangent vectors of exponential curves that locally best fit the data:The coefficients of the tangent vector \({c_\xi ,c_\eta ,c_\theta }\) are expressed in the moving frame (see Eq. (1), [24, 30]).

Principal directions are calculated from the eigenvectors of the Hessian matrix, but now in a curved domain, for which we need left-invariant derivatives:As left-invariant derivatives do not commutate, e.g., \(\partial _\theta \partial _\xi U \ne \partial _\xi \partial _\theta U\), we need to make the asymmetric Hessian matrix symmetric viaso we can calculate the eigensystem of \(\mathcal {H}_\mu U\). The diagonal matrix \(M_{\mu ^{-2}}\) compensates for the difference in units between the orientation and spatial dimensions, for details see [11].

A confidence measure is acquired by calculating the Gaussian Laplacian in the plane orthogonal to the tangent direction of the vascular geodesic. From all pixelwise curvature measures, several global measures are derived, as the mean and standard deviation. The method was validated on synthetic and retinal images.

Tortuosity measures from the MESSIDOR database (1200 images) showed significant increase of tortuosity with the increasing severity of diabetic retinopathy (R0, R1, R2 and R3), see Fig. 14. For more details, see [11].