Particle swarm optimization based fusion of near infrared and visible images for improved face verification

Abstract

This paper presents two novel image fusion schemes for combining visible and near infrared face images (NIR), aiming at improving the verification performance. Sub-band decomposition is first performed on the visible and NIR images separately. In both cases, we further employ particle swarm optimization (PSO) to find an optimal strategy for performing fusion of the visible and NIR sub-band coefficients. In the first scheme, PSO is used to calculate the optimum weights of a weighted linear combination of the coefficients. In the second scheme, PSO is used to select an optimal subset of features from visible and near infrared face images. To evaluate and compare the efficacy of the proposed schemes, we have performed extensive verification experiments on the IRVI database. This database was acquired in our laboratory using a new sensor that is capable of acquiring visible and near infrared face images simultaneously thereby avoiding the need for image calibration. The experiments show the strong superiority of our first scheme compared to NIR and score fusion performance, which already showed a good stability to illumination variations.

Introduction

Recent improvements in sensor technology have boosted the face as a reliable trait for successful identification of persons in security applications. However, face verification still offers poor performance in presence of high illumination, variation in pose and facial expressions [4]. Recently, some studies have demonstrated that the use of thermal/infrared sensors allows for acquiring images that are less sensitive to ambient light or illumination changes [10], facial expression [10], [2] and face pose [3]. Unfortunately, these images present less information related to the face shape and texture [4]. Some authors [15], [18], [4] have shown recently that near infrared (NIR) images which use an intermediate light spectrum range between infrared and visible light (790–900 nm) present some robustness to illumination variations while preserving meaningful aspects of face like appearance and structure. This way, face recognition in degraded illumination condition remains possible on this type of images [15], [18]. We have developed in our team an original acquisition device allowing to acquire almost simultaneously (20 ms delay) visible and NIR images [4]. This successive capture is done using the video protocol consisting of the capture of two frames (odd and even) to form an image. We acquire during the odd frame a near infrared image by adding a synchronized illuminator activated during the integration time of the CCD (coupled charged device). During the even frame we deactivate the illuminator and a normal acquisition is performed. During the odd frame, we reduced the acquisition time in order to acquire less ambient light and we activated the pulsed flash during the capture time. The resulting image is composed of the reflection of the flash's light. During the even frame we deactivated the flash. The output of this sensor is a video flux that combines both images. We therefore separate the odd and even frames to form two images of 640×480 pixels. Fig. 1 presents the acquisition sequence. This original device allows us to alleviate the calibration task if we want to use the two images at the same moment, whatever for comparison or fusion purposes. The detailed description of our sensor is given in [5].

In our previous work [4], our aim was first to compare the performance of different classical face verification approaches on both types visible and near infrared images acquired with our sensor. We then studied the respective degradation of performance due to illumination variations on visible images and we checked the stability of NIR images in bad illumination contexts. We therefore confirmed that the error rates of two algorithms, one chosen as representative of global face verification method and the other one as a variation of elastic graph matching (EGM) were very high on visible images while they remain limited on NIR images, when considering highly illuminated conditions. We performed these experiments using a database acquired with our sensor on which we defined several protocols considered so as to emphasize the difference between controlled and non-controlled conditions. We also performed some limited and non-satisfactory experiments of scores fusion. Indeed, due to the bad performance of the systems working on visible images, fusion performance were roughly identical to those of the best system, namely the system working on near infrared light.

In this study, we propose to perform fusion at the image level in order to exploit at an earlier stage the inherent complementarities of the two images. The problem in this context is to build from the two available images of the same object a single image. It appears from the image fusion literature related to visible and infrared images an interest in performing multi-resolution fusion and not direct fusion at the pixel level. This way it is possible to integrate more information at different level of decomposition. The general scheme when performing multi-resolution image fusion is the following [6], [13], [14], [16]: After performing a multi-scale transformation of each source image leading to a different set of features for each image, a composite feature set is built from these two distinct feature sets according to some specific rules. Finally an inverse transform allows reconstructing an image that can be seen as the fusion of the two original images. In the literature of infrared and visible image fusion, there are interesting propositions for defining these fusion rules which can be split into two groups. The first idea is to select the group of features from visible and thermal face images that will be retained to constitute the fused feature set. To this end, Singh et al. [16], for instance, employed genetic algorithm. The other idea is to assign weights to the two types of features (of course selection can be seen as a particular case of weighting when weights take values 0 and 1). Heo et al. [6] proposed to fuse the pixels of visible and thermal face images using an empirically designed weighted linear combination. In [14], the authors suggest another approach for weighting the features by relating the weights to the activity (energy) level of each image, labeled “good” or “bad” thanks to a granular SVM classifier.

In this work, we provide two novel solutions for fusing features from two different image sources; one corresponds to a weighting scheme while the other gives a selection scheme. Novelty here is in the use of particle swarm optimization (PSO) [8], a well-known optimization algorithm, which we have already used efficiently [12] in the context of biometric feature fusion. In the first proposed scheme, we will use the “real” version of PSO for finding optimal weights in the weighted summation of visible and infrared feature while in the second proposed scheme, we will use the “binary version” of PSO to select the features in each spectrum which will finally constitutes the fused feature vector. In both cases, PSO is used for finding the feature fusion strategy which optimizes the verification performance. The proposed image fusion techniques are evaluated on visible and NIR images acquired by our original sensor. One should note that this sensor device provides an automatic registration between the visible and NIR images as both images are acquired simultaneously. This acquisition framework renders therefore natural the idea of image fusion. More precisely, we will use IRVI database that was acquired previously in our laboratory and the associated protocols defined in [4] to this end. Our experiments allow comparing the two proposed image fusion schemes to the unimodal models and also to several systems performing fusion at different stages namely: image, feature and match score level. Results illustrate that the proposed image fusion offers substantial improvements in performance over all the other schemes.

The rest of the paper is organized as follows: Section 2 briefly describes the PSO algorithm principles including the definition of the real and binary implementations. Section 3 describes the two proposed image fusion schemes that we designed for the image fusion of two different images resulting from different sources and the related verification systems. Section 4 describes the experimental setup, the PSO parameters that were chosen for performing this fusion task and the experimental protocols defined on the IRVI database. Section 5 describes and analyses the results of the proposed schemes and finally Section 6 draws the conclusion.