Elsevier

Information Fusion

Volume 4, Issue 3, September 2003, Pages 167-183
Information Fusion

Sensor noise effects on signal-level image fusion performance

https://doi.org/10.1016/S1566-2535(03)00035-6Get rights and content

Abstract

The aim of this paper is twofold: (i) to define appropriate metrics which measure the effects of input sensor noise on the performance of signal-level image fusion systems and (ii) to employ these metrics in a comparative study of the robustness of typical image fusion schemes whose inputs are corrupted with noise. Thus system performance metrics for measuring both absolute and relative degradation in fused image quality are proposed when fusing noisy input modalities. A third metric, which considers fusion of noise patterns, is also developed and used to evaluate the perceptual effect of noise corrupting homogenous image regions (i.e. areas with no salient features). These metrics are employed to compare the performance of different image fusion methodologies and feature selection/information fusion strategies operating under noisy input conditions. Altogether, the performance of seventeen fusion schemes is examined and their robustness to noise considered at various input signal-to-noise ratio values for three types of sensor noise characteristics.

Introduction

Significant research effort has been directed in recent years into the development of multisensor signal-level image fusion (MSL-IF) schemes. This is due to the key role MSL-IF plays in determining the performance of advanced multisensor imaging display/visual information processing systems, in applications such as avionics, medical imaging, earth observation and security/surveillance. Within these application areas, where there is a tendency to employ several sensing modalities under a wide range of operating conditions, the prospect arises of fusing input images of low visual quality. As a result, “noisy” input information associated with individual sensors may affect significantly fusion system performance. This is because input “noise” may be treated by the fusion system as valid information and transferred to the fused output image. Furthermore input image noise may affect the selection/fusion process of the MSL-IF system in a way that introduces additional unwanted artefacts and distortion into the fused image [1], [2], [3], [4], [5], [6].

The performance characteristics of image fusion algorithms, operating in noise free conditions, are considered in a number of papers. Zhang and Blum [6] present a thorough investigation into several multiresolution fusion methodologies for a digital camera application. Fusion schemes are categorised according to their basic multiresolution/pyramid image representation approach and mechanisms for pyramid coefficient fusion. Pohl and van Genderen [27] provide a comprehensive review of fusion techniques as applied to the field of remote sensing. However, despite the unquestionable usefulness of such contributions almost no effort has been allocated in measuring and comparing the effects of input noise on the performance of image fusion systems. Furthermore, no independent assessment is available of image fusion algorithms specifically designed to operate robustly under noisy input conditions [3], [5].

This paper presents an investigation into the effects of sensor noise on the performance of signal-level image fusion systems. Its aim is twofold:

  • (i)

    to develop appropriate metrics which measure the detrimental effect of input sensor noise on the fusion performance of MSL-IF schemes and

  • (ii)

    to employ these metrics in a comparative study of the robustness of several published image fusion schemes operating under noisy input conditions.


Three different experimental scenarios for fusing images are considered, using a two-sensors suite that is formed from visible light, infrared or image enhanced (low light) cameras. In the first scenario, only one input image suffers from sensor noise. This is analogous to the practical situation when “low contrast” conditions exist for one of the sensors, e.g. a visible light sensor operating in foggy conditions with contrast boosting applied which in turn amplifies sensor noise. Here uncorrupted, “true” scene, information is still available from the noise free output of the second sensor. In the second scenario, both sensors suffer from noticeable levels of sensor noise and the expected degradation in the quality of the fused image is greater. In the third noisy fusion scenario there is no “true” scene information. Instead, the fusion of noise patterns is examined in order to investigate the behaviour and performance of MSL-IF algorithms operating on homogenous image regions that are corrupted by sensor noise.

Thus novel “noisy fusion” metrics are developed and used, in the first two scenarios, to measure the effects of additive sensor noise on the performance of several signal-level image fusion algorithms operating across a range of input signal-to-noise ratio (SNR) values. These metrics evaluate: (i) the absolute degradation of information in the output fused image, with respect to true scene information and (ii) the relative degradation of information in the fused image, with respect to the noisy input image(s). The third scenario considers the effect different fusion methods have on the characteristics of fused image noise. Overall, the performance of seventeen different fusion schemes, operating under varying noisy input conditions, is examined. These schemes were specifically chosen to cover a broad range of image fusion methodologies and feature fusion (information fusion) techniques. Furthermore, two MSL-IF schemes with built-in noise suppression procedures are also investigated.

The next section refers briefly to sensor noise models and the generation of noisy input image data. In Section 3, several novel noisy fusion performance metrics are derived using the objective image fusion performance evaluation framework of Xydeas and Petrović [13], [14]. Noisy fusion performance results are presented, compared and discussed in Section 4. Finally, concluding remarks are offered in Section 5.

Section snippets

Sensor noise modelling and noisy image data

Passive multisensor imaging arrays often include (i) visible light sensors that measure scene illumination in the visible spectrum (0.45–0.7 μm) (ii) infrared sensors that measure the thermal radiance of scene objects in the infrared part of the spectrum (1.5–15 μm) and (iii) low light or image enhanced cameras [25]. In general, sensor noise is the result of several processes associated with the underlying physics of recording an observation [8], [9], [10], [11], [12]. Typically however,

Noisy image fusion performance evaluation

Signal-level image fusion performance evaluation has only recently received attention from the research community and certain MSL-IF evaluation methods have been proposed. Noise-free fusion system performance metrics such as the root mean square error (RMSE), mutual information and the percentage of correct decisions of Zhang and Blum [6], the standard deviation of Li et al. [2] and various similar schemes outlined in Pohl and van Genderen [27], compare the output fused image with a reference

MSL-IF system performance results

The noisy fusion performance metrics defined in the previous section were employed in a comparative study of seventeen different fusion schemes operating on the noisy input image sets described in Section 2. The overall QpAB/Fn, QpFF/Fn and Dnp scores for each input set were obtained as average values calculated over all input image pairs in that set. Thus for example, QpAB/Fn is estimated asQpAB/Fn=1Mi=1MQpAiBi/Fi,nwhere i indicates the ith input image pair and M is the total number of pairs

Summary

The performance of several signal-level image fusion systems was examined, for the case when their input signals are corrupted at varying levels of sensor noise. This was done using novel noisy fusion performance metrics developed via the subjectively meaningful, objective image fusion performance evaluation framework recently established by Xydeas and Petrović [13], [14]. The proposed metrics measure absolute fusion performance, relative degradation of fusion performance (robustness) and

Acknowledgements

The authors gratefully acknowledge BAE Systems for their financial support during the early stages of this project and the Defence Research Establishment Valcartier, Canada and the US government’s AMPS programme for some of the imagery used in this research.

References (26)

  • H. Li et al.

    Multisensor image fusion using the wavelet transform

    Graphical Models and Image Processing

    (1995)
  • P. Burt, R. Kolczynski, Enhanced image capture through fusion, Proceedings of the Fourth International Conference on...
  • L. Chipman et al.

    Wavelets and image fusion

    Proceedings of SPIE

    (1995)
  • L. Ramac et al.

    Morphological filters and wavelet based image fusion for concealed weapons detection

    Proceedings of SPIE

    (1998)
  • L. Grewe et al.

    Atmospheric attenuation reduction through multi-sensor fusion

    Proceedings of SPIE

    (1998)
  • Z. Zhang et al.

    A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application

    Proceedings of the IEEE

    (1999)
  • G. Waldman et al.

    Electro-optical systems performance modelling

    (1993)
  • A. Frenkel et al.

    Photon-noise-limited operation of intensified CCD cameras

    Applied Optics

    (1997)
  • G. Healey et al.

    Radiometric CCD camera calibration and noise estimation

    IEEE Transactions of PAMI

    (1994)
  • M. Nelson et al.

    General noise processes in hybrid infrared focal plane arrays

    Optical Engineering

    (1991)
  • R. Boie et al.

    An analysis of camera noise

    IEEE Transactions of PAMI

    (1992)
  • A. Prabala

    CCD vs. CMOS imagers for Sci/industrial cameras

    Advanced Imaging

    (2000)
  • C. Xydeas et al.

    Objective image fusion performance measure

    Electronic Letters

    (2000)
  • Cited by (0)

    View full text