Human visual system inspired multi-modal medical image fusion framework

Abstract

Multi-modal medical image fusion, as a powerful tool for the clinical applications, has developed with the advent of various imaging modalities in medical imaging. The main motivation is to capture most relevant information from sources into a single output, which plays an important role in medical diagnosis. In this paper, a novel framework for medical image fusion based on framelet transform is proposed considering the characteristics of human visual system (HVS). The core idea behind the proposed framework is to decompose all source images by the framelet transform. Two different HVS inspired fusion rules are proposed for combining the low- and high-frequency coefficients respectively. The former is based on the visibility measure, and the latter is based on the texture information. Finally, the fused image is constructed by the inverse framelet transform with all composite coefficients. Experimental results highlight the expediency and suitability of the proposed framework. The efficiency of the proposed method is demonstrated by the different experiments on different multi-modal medical images. Further, the enhanced performance of the proposed framework is understood from the comparison with existing algorithms.

Introduction

With the rapid development in high-tech and advanced instrumentations, medical imaging has become a vital component of a large number of applications, including diagnosis, research, and treatment. This rife development has enabled radiologists to quickly acquire images of the human body and its internal structures with effective resolution and realism. These images are often known as multimodality medical images such as X-ray, computed tomography (CT), magnetic resonance imaging (MRI), magnetic resonance angiography (MRA), and positron emission tomography (PET) images (Maes, Vandermeulen, & Suetens, 2003). These multimodality medical images usually provide complementary and occasionally conflicting information. For example, X-ray and computed tomography (CT) can provide dense structures like bones and implants with less distortion, but it cannot detect physiological changes (Aguilar & Garrett, 2001). Similarly, normal and pathological soft tissue can be better visualized by MRI whereas PET can be used to provide better information on blood flow and flood activity with low spatial resolution. For medical diagnosis, treatment planning and evaluation, the complementary information obtained from different modality images is needed. For example, combined PET/CT imaging can concurrently visualize anatomical and physiological characteristics of the human body and can also be used to view tumor activity in conjunction with anatomical references in oncology. Also in organ diagnosis, the combined PET/CT imaging is very useful, where tumor boundaries are difficult to discern (Baum et al., 2008, Kamman et al., 1989).

Hence, the fusion of the multi-modal medical images is necessary and it has become a promising and very challenging research area nowadays. Image fusion can be defined as the process in which some important features of multiple input images are combined into a single image without any loss of information. Medical image fusion aims at integrating complementary as well as redundant information from multiple modality images to obtain a more complete and accurate description of the same object. It provides easy access to the PET/CT/MRI images at the same location where reading from all other modalities is done, allowing radiologists to quickly and efficiently report PET/CT/MRI studies (Rojas, Raff, Quintana, Huete, & Hutchinson, 2007).

So far, many effective techniques for image fusion have been proposed in the literature especially for medical images. The simplest ways are pixel-by-pixel (Bhatnagar et al., 2010, Jiang and Tian, 2011, Xydeas and Petrovic, 2002, Yang and Li, 2012) gray level average or selection of the source images but these ways lead to undesirable side effects such as reduced contrast. Other categories include image fusion methods based on statistical and numerical methods (Cardinali and Nason, 2005, Wang and Ma, 2008), intensity-hue-saturation (IHS) (Daneshvar and Ghassemian, 2010, Tu et al., 2001), principal component analysis (PCA) (Chavez & Kwarteng, 1989), independent component analysis (ICA) (McKeown et al., 1998), contrast pyramid (Burt & Kolczynski, 1993), gradient pyramid (Petrovic & Xydeas, 2004) and multiresolution methods (Boussion et al., 2008, Bhatnagar and Raman, 2009, Bhatnagar and Wu, 2012, Guihong et al., 2001, Li and Wang, 2011, Li et al., 1995, Miao et al., 2011, Nemec et al., 2010, Wu et al., 2012, Yang et al., 2008, Yang et al., 2010, Zhang and Guo, 2009). Statistical and numerical methods involve huge computation using floating-point arithmetic, therefore these methods are time and memory consuming. The IHS method is based on the representation of low spatial resolution images using IHS system and then substitution of the intensity component by a high-resolution image. In PCA/ICA method, original images are transformed into uncorrelated images and then fused by choosing the maximum value among all. PCA/ICA is frequently used for fusion because of its ability to compact the redundant data into fewer bands. In pyramid and multiresolution based methods, the source images are decomposed by applying pyramid or wavelet transform, then fusion operation is performed on the transformed images. These methods produce very good results in less computation time and less memory when compared to others. However, these methods often produce undesirable side effects like block artifacts, reduced contrast etc. which often result in the wrong diagnosis (Li, Yang, & Hu, 2011).

This paper is an attempt to rectify the drawbacks of multiresolution transform in medical image fusion. For this purpose, framelet transform is used in the proposed framework. First, all the input images are decomposed in low and high frequency bands and the fusion is employed by considering the physical meaning of the bands. For this purpose, two new HVS based fusion rules are developed for fusing low and high frequency bands respectively. The low frequency bands are fused by considering the visibility as fusion measure instead of simple averaging as used in the existing schemes whereas the high frequency bands are fused based on texture information obtained from HVS model. The used HVS model is the Smallest Univalue Segment Assimilating Nucleus (SUSAN) which is a feature extractor function. The fused medical image that is produced by this scheme presents a visually better representation than the input images. Moreover, the compatibility and superiority of the proposed method can be judged by the comparison with the existing methods. The salient contributions of the proposed framework over existing methods can be summarized as follows.

• This paper proposes a new image fusion framework for multimodal medical images, which relies on the human visual system characteristics in framelet domain.

• Two different fusion rules are proposed for combining low- and high-frequency coefficients considering their physical meaning.

• For fusing the low-frequency coefficients, a visibility based process is used. The main benefit of visibility is that it selects and combines the low-frequency coefficients from the focused part of images.

• On the contrary, the texture information obtained from SUSAN feature extractor is used to combine high-frequency coefficients. Using SUSAN features, the most prominent texture and edge information are selected from high-frequency coefficients and combined in the fused ones.

• The efficiency of the proposed framework is highlighted by different experiments on different medical images of different modality whereas superiority is achieved by the comparison with state-of-art algorithms.

• Further, two clinical examples of the persons affected with alzheimer and tumor is also done for more elaborated performance comparison analysis.

The rest of the paper is organized as follows. The framelet transform and its benefits in image fusion are illustrated in Sections 2 Framelet transform, 3 Benefits of framelet transform in image fusion respectively. The proposed multi-modal medical image fusion framework is explained in Section 4 followed by the experimental results and discussions in Section 5. Finally, the concluding remarks are given in Section 6.