Elsevier

Expert Systems with Applications

Volume 41, Issue 16, 15 November 2014, Pages 7425-7435
Expert Systems with Applications

A novel approach for multimodal medical image fusion

https://doi.org/10.1016/j.eswa.2014.05.043Get rights and content

Highlights

  • The CS-based fusion framework guarantees image quality in clinical diagnosis.

  • The CS-based fusion scheme has merits of good flexibility and low time consumption.

  • Two fusion rules are utilized to enhance the performance before linear projection.

  • A fusion rule is proposed to preserve edges, lines and contours of the fused image.

Abstract

Fusion of multimodal medical images increases robustness and enhances accuracy in biomedical research and clinical diagnosis. It attracts much attention over the past decade. In this paper, an efficient multimodal medical image fusion approach based on compressive sensing is presented to fuse computed tomography (CT) and magnetic resonance imaging (MRI) images. The significant sparse coefficients of CT and MRI images are acquired via multi-scale discrete wavelet transform. A proposed weighted fusion rule is utilized to fuse the high frequency coefficients of the source medical images; while the pulse coupled neural networks (PCNN) fusion rule is exploited to fuse the low frequency coefficients. Random Gaussian matrix is used to encode and measure. The fused image is reconstructed via Compressive Sampling Matched Pursuit algorithm (CoSaMP). To show the efficiency of the proposed approach, several comparative experiments are conducted. The results reveal that the proposed approach achieves better fused image quality than the existing state-of-the-art methods. Furthermore, the novel fusion approach has the superiority of high stability, good flexibility and low time consumption.

Introduction

Medical image processing attracts much attention in modern diagnostic and health-care over the past decade. The ultimate aim of medical image fusion is to fuse the complementary information from multimodal medical images to acquire a high-quality image. It is widely utilized in biomedical research and clinical diagnosis for doctors. Medical images have many kinds of species with respective application boundaries, including computed tomography (CT), magnetic resonance imaging (MRI), magnetic resonance angiography (MRA), and positron emission tomography (PET) images (He et al., 2013, Parmar and Kher, 2012, Parmar et al., 2012). Anatomical information is supplied by CT images, functional information is delivered by PET images, and MRI images are superior in presenting the normal and pathological soft tissue (Parmar and Kher, 2012, Parmar et al., 2012, Li et al., 2011). Medical image processing is concentrated on extracting significant information via synthesizing multiple images in a scenario (Deserno, 2011, He et al., 2013, Li et al., 2011, Parmar and Kher, 2012, Parmar et al., 2012, Serikawa et al., 2013). With the complementary information of CT and MRI images, fusion of CT and MRI images can preserve much more edge and component information. It provides a high-quality fused image for doctors to give a more accurate diagnosis. Fusion of multimodal medical images draws many attention of experts and scholars around the word.

The process of fusing medical images into a unitary image without introduction of distortion or loss of information is a promising research issue. Accompanied by the growth demand of accurate diagnosis from complementary information of multimodal medical images, many medical image fusion approaches have been exploited, including Intensity–Hue–Saturation (IHS) (Daneshvar and Ghassemian, 2010, Pradhan et al., 2006), Principal Component Analysis (PCA) (Nirosha and Selin, 2012, Pradhan et al., 2006), discrete wavelet transform (DWT) (Wan, Canagarajah, & Aohim, 2009), Weighted Score Level Fusion (Sim, Asmuni, Hassan, & Othman, 2014), Laplacian Pyramid (LP) (Burt and Adelson, 1983, Tan et al., 2013) and Synthetic Variable Ratio (SVR) (Rahman and Csaplovics, 2007, Wang et al., 2008) etc.

More recently, multi-scale transform methods have emerged as a well developed yet rapidly expanding mathematical foundation for multimodal medical image fusion. In order to improve the precision and performance of computer assisted diagnosis, an approach utilizes different features of redundant discrete wavelet transform, mutual information and entropy information to preserve edge and component information is proposed (Singh, Vatsa, & Noore, 2009). To capture most relevant information from multimodal medical images into a single output, the framelet transform and two improved human visual system (HVS) are utilized to enhance the performance of the fused image (Bhatnagar, Jonathan Wu, & Liu, 2013). To derive useful information from multimodality medical image data, many scholars discuss the appropriate scale representation in wavelet transform domain. Inspired by these work, a multi-scale medical image fusion approach based on DWT is proposed to capture all salient information into a single fused image with low computation cost and storage space (Parmar et al., 2012, Singh and Khare, 2013). The real valued wavelet transforms have the limitations of shift sensitivity, lack of phase information and poor directionality. Specific to these problems, an improved wavelet transform, Daubechies complex wavelet transform, is proposed (Singh & Khare, 2014). As a new multi-resolution analysis tool, a multimodal medical image fusion scheme based on extended contourlet transforms is presented and novel fusion rules are utilized to achieve better fused image quality (Serikawa et al., 2013). The general fusion approach for multimodal medical images is shown in Fig. 1 (Bhatnagar et al., 2013, Serikawa et al., 2013, Singh and Khare, 2013, Singh and Khare, 2014, Stathaki, 2008, Tan et al., 2013, Wan et al., 2009). In the conventional fusion approach, the CT and MRI images are decomposed via multi-scale geometric transform. The decomposition coefficients are fused by utilizing one fusion rule. The fused medical image is acquired by the corresponding multi-scale inverse transform. However, in the real practice applications, an increasing number of multimodal medical sources images loads to the problem of information overload in modern diagnostic and health-care. Furthermore, the selection of the proper level of multi-scale geometric transform to approximate the sources generally depends on the priori knowledge of the source images. The ill-suited threshold may lead to the problems of poor fidelity and blocking artifacts.

The principle of compressive sensing (CS) can accurately reconstruct the sparse image at a lower sampling rate than at the Nyquist rate. The sparsity of multimodal medical images in the transformation domain is the only constrained priori information. Fortunately, almost all the two-dimensional signals are sparsity under certain transforms. Inspired by this work, many CS-based image fusion methods for medical images have been proposed with the superiority of low sampling ratio and low computation complexity. Experts and scholars pay close attention to the CS-based multimodal medical image fusion scheme. Paper Han et al. (2010) propose a novel fusion approach based on compressive sensing and Discrete Cosine Transform (DCT) sampling model to preserves much richer texture information of the source images. Recently, paper Wan and Qin (2010) verify the sampling of multimodal source images without assuming any prior information to regard the applicability of CS to image fusion. It also discusses the properties of compressive sensing under different sampling patterns. Furthermore, the experimental results demonstrate that it achieves the promising performance. However, it loses spatial information due to the randomness of measurement matrix. Since the sensors can observe related phenomena, the ensemble of signals is expected to extract and separate some joint structure, or possess the correlation. Inspired by this work, a novel joint sparse representation-based image fusion method is proposed, which can overcome the drawback of the weighted fusion rule and achieve better fusion performance (Yu, Qiu, Bi, & Wang, 2011). Although these CS-based methods have achieved good results, there are two main limitations in the medical image processing domain. Firstly, high reconstruction error may be generated due to the randomness of measurement matrix. It is related with the number of the sensing measurements and the measurement matrix consistency. Given this, fusing the sensing measurements after non-adaptive linear projection can not achieve the promising performance. Secondly, the sparse coefficients decomposed by various multi-scale transforms contain different properties. Applying single fusion rule to fuse both high and low frequency information is difficult to approximate all the coefficients. It may result in the problems of blocking artifacts and poor fidelity. In addition, in some papers, although two proper fusion rules are used to fuse the sparse coefficients, these algorithms fuse high frequency information after linear projection as usual. It may lead to blur problems and the loss of spatial information.

In this paper, a multimodal medical image fusion approach based on compressive sensing specific to the mentioned crucial limitations is presented. Firstly, the sparse coefficients are directly fused before measured by the measurement matrix. It can greatly reduce the reconstruction error. Secondly, an improved average-gradient-based fusion rule is used to fuse the high frequency information; while PCNN model is utilized to fuse the low frequency information with the superiority of much closer to the mode of human visual processing. By those steps, the proposed approach can overcome the problems of blocking artifacts and poor fidelity. In brief, the novel CS-based medical image fusion approach can obtain a high-quality fused image by only fusing fewer non-zero sparse coefficients with the property of real time. In the practice, the computation complexity is greatly reduced and concurrently the quality of the fused image is guaranteed. In our proposed fusion approach, the sparse coefficients are obtained by DWT. Then, an improved average-gradient-based fusion rule and PCNN fusion rule are utilized to fuse the high and low frequency information, respectively. Finally, Compressive Sampling Matching Pursuit algorithm (CoSaMP) is used to reconstruct the fused image accurately. The main contributions of this work can be illustrated as follow:

  • (1)

    We present a novel fusion approach based on CS for multimodal medical images. This work can greatly reduce the computation complexity and remove the problems of blocking artifacts and poor fidelity. It has the merit of real time and simultaneously achieves the promising performance. Meanwhile, the property of high compression ratio can reduce the demand of the storage spaces and the transmission bandwidth.

  • (2)

    We employ two fusion rules to fuse the low and high frequency information before the procedure of nonadaptive linear projection, respectively. This improved scheme can reduce the reconstruction error and acquire satisfied image quality than other conventional methods.

  • (3)

    We propose a novel fusion rule based on average gradient and mutual information to fuse the high frequency information. This fusion rule can acquire much more detail information to avoid the problems of blocking artifacts and poor fidelity, such as edges, lines and contours.

This paper is organized as follow. In Section 2, a brief description of the proposed approach is given out with the detail of the presented medical image fusion approach. Several experiments are conducted for the general algorithms and the proposed method in Section 3. Discussions are summarized in Section 4.

Section snippets

The proposed fusion approach based on compressive sensing

The main superiority of CS is its low sampling ratio and low computation complexity. In the CS-based image fusion framework, the sparse image can be acquired and represented at a compression and sampling rate far fewer bellow the Nyquist rate (Candes, 2006, Donoho, 2006, Kutyniok, 2012). As a two-dimensional signal processing, medical image fusion can take full advantage of this theory. In this section, the significant multimodal medical image fusion based on compressive sensing is elaborated.

Experiments and analysis

The CT and MRI images generally contain complementary and occasionally inconsistent information. The CT image cannot detect physiological changes, although it affords dense structures like bones and implants without introduction of distortion or loss of information. The MRI image can support normal and pathological soft tissues information, but it cannot sustain the bones information. Multimodal medical image fusion based on compressive sensing can be competent to support accurate clinical

Conclusion

Multimodal medical image fusion plays an important role in clinical applications. In this paper, a novel fusion approach for CT and MRI images based on compressive sensing is proposed. There are insightful and practical improvements in comparison to the conventional fusion methods. Firstly, fusing the sparse coefficients before non-adaptive linear projection can greatly reduce the reconstruction error and remove the unknown noise. Secondly, the novel fusion rule based on average gradient and

Acknowledgments

We would like to thank the supports by National Natural Science Foundation of China (61203321, 61374135), China Postdoctoral Science Foundation (2012M521676) and China Central Universities Foundation (106112013CDJZR170005), Chongqing Special Funding in Postdoctoral Scientific Research Project (XM2013007).

References (45)

  • R.G. Baraniuk

    Compressive sensing

    IEEE Signal Processing Magazine

    (2007)
  • P.J. Burt et al.

    The Laplacian pyramid as a compact image code

    IEEE Communications

    (1983)
  • E. Candes

    Compressive sampling

    (2006)
  • E. Cands et al.

    Stable signal recovery from incomplete and inaccurate measurements

    Communications on Pure and Applied Mathematics

    (2006)
  • T.M. Deserno

    Biomedical image processing: Biomedical and medical physics, biomedical engineering

    (2011)
  • M. Deshmukh et al.

    Image fusion and image quality assessment of fused images

    International Journal of Image Processing

    (2010)
  • D.L. Donoho

    Compressed sensing

    IEEE Transactions on Information Theory

    (2006)
  • M.F. Duarte et al.

    Structured compressed sensing: From theory to applications

    IEEE Transactions on Signal Processing

    (2011)
  • W. Ge et al.

    Image fusion algorithm based on PCNN and wavelet transform

    Fifth International Symposium on Computational Intelligence and Design (ISCID)

    (2012)
  • Han et al., (2010). Multi image fusion based on compressive sensing. 2010 International conference on audio language...
  • C.T. He et al.

    Medical image fusion using guided filtering and pixel screening based weight averaging scheme

    Journal of Software Engineering

    (2013)
  • H. Jiang et al.

    Fuzzy image fusion based on modified self-generating neural network

    Expert Systems with Applications

    (2011)
  • Cited by (106)

    • A novel sparse representation based fusion approach for multi-focus images

      2022, Expert Systems with Applications
      Citation Excerpt :

      Image fusion can integrate information from different images of the same scene into a single image with all-focused. Nowadays, multi-focus image fusion is widely used in military surveillance, clinical medicine, remote sensing, image processing, and so on (Liu et al., 2014; Lei et al., 2021; Chien and Tsai, 2014; Li et al., 2013; Kong et al., 2014). Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent.

    • Vague set theory based segmented image fusion technique for analysis of anatomical and functional images

      2020, Expert Systems with Applications
      Citation Excerpt :

      Bhatnagar et al. (Bhatnagar et al., 2015) proposed an approach where low and high frequency components are fused on the basis of normalized Shannon entropy measure and directive contrast rule respectively. In the recent years, many researchers proposed reduced pulse coupled neural network (RPCNN) based fusion model (Jiang and Tian, 2011; Bhatnagar et al., 2013; Das and Kundu, 2013; Kavitha and Chellamuthu, 2014; Liu et al., 2014; Xu et al., 2016) to improve the feature saliency. Das et al. (Das and Kundu, 2013) introduced reduced pulse coupled neural network (RPCNN) for improving visual clarity and then applied local energy and entropy based fuzzy logic approach for combining the multiscale frequency subbands.

    View all citing articles on Scopus
    View full text