Elsevier

Pattern Recognition

Volume 48, Issue 11, November 2015, Pages 3371-3384
Pattern Recognition

Face recognition under pose variation with local Gabor features enhanced by Active Shape and Statistical Models

https://doi.org/10.1016/j.patcog.2015.05.017Get rights and content

Highlights

  • A local matching Gabor method is improved with an Active Shape and a Statistical Model.

  • The enhanced model is applied to recognize faces with significant pose variation.

  • Comprehensive tests were performed on the FERET and CMU-PIE databases.

  • Results improved from 31.1% to 70.57% in the extreme poses of the databases.

  • We reached the highest results with pose variation compared to any previous 2D method.

Abstract

Face recognition is one of the most active areas of research in computer vision. Gabor features have been used widely in face identification because of their good results and robustness. However, the results of face identification strongly depend on how different are the test and gallery images, as is the case in varying face pose. In this paper, a new Gabor-based method is proposed which modifies the grid from which the Gabor features are extracted using a mesh to model face deformations produced by varying pose. Also, a statistical model of the scores computed by using the Gabor features is used to improve recognition performance across pose. Our method incorporates blocks for illumination compensation by a Local Normalization method, and entropy weighted Gabor features to emphasize those features that improve proper identification. The method was tested on the FERET and CMU-PIE databases. Our literature review focused on articles with face identification with wide pose variation. Our results, compared to those of the literature review, achieved the highest classification accuracy on the FERET database with 2D face recognition methods. The performance obtained in the CMU-PIE database is among those obtained by the best methods published.

Introduction

Face recognition has a wide range of possible applications from person identification and surveillance to electronic marketing and advertising for selected customers. Face recognition in real time is a topic of active research, and several methods have been proposed to perform this task [1], [2], [3], [4].

Most studies focus on frontal face recognition, reaching high accuracy on internationally available face databases. An important number of studies have focused directly on face recognition under the assumption that the face has already been localized [5], [6].

Among the most widely cited methods for face recognition based on feature extraction are Eigenfaces [7], based on Principal Component Analysis (PCA), Fisherfaces [8], based on Linear Discriminant Analysis (LDA), and methods based on Independent Component Analysis (ICA) [9], [10]. In [11], the Local Binary Pattern (LBP) method was proposed, in which the face image is divided into square windows where a binary code is generated whenever a pixel exceeds the value of the average within the window. Also some papers report addressing the problem of face recognition in low resolution images [12]. This method uses a multidimensional scaling approach where low resolution images are embedded in a Euclidean space which is used to perform the matching between gallery and test images. Gabor wavelets [13], [14], [15], [16] have been used to extract local features achieving outstanding results in face recognition.

Among the methods based on Gabor Wavelets are the Elastic Bunch Graph Matching (EBGM) method [17], the Gabor Fisher Classifier (GFC) [18], the Local Gabor Binary Pattern Histogram Sequence (LGBPHS) [19], the Histogram of Gabor Phase Patterns (HGPP) [20], Local Gabor Textons (LGT) [21], Learned Local Gabor Pattern (LLGP) [22], Local Gabor Binary Pattern Whitening PCA (LGBPWP) [23], and the Local Matching Gabor method (LMG) [24], [25], [26], [27]. In [28], the face was divided into patches without overlap, and then the best patches were selected and weighted with an LDA strategy in a greedy search. Finally, the local scores of the patches were combined with a global score obtained from the low frequency components of the FFT applied to the whole face, including its external boundary. Magnitude and phase Gabor features were combined in [29]. The LBP operator was used on the Gabor magnitude features and the LXP operator (Local Xor Pattern) on the Gabor phase features. Then, the face was divided into regions, and histograms of each region were computed on LGBP and LGXP features. Every region dimensionality was reduced using LDA, and finally the regions were compared with cosine distance. In [30], Gabor features in face images at higher and lower resolutions were used.

Face recognition under varying pose continues to be an area of active research. There are various approaches to solving this problem using 2D as well as 3D methods. Some methods use a single face as input and build a 3D model called the 3D Morphable Model [31], [32], [33]. The 3D Morphable Model is based on a vector space representation of faces built using vectors of shape and texture. The parameters of the models are computed using a set of Eigen vectors obtained previously by training with images from 3D scans. A fully automatic face frontalization method using a 3D model was introduced in [34]. It works for poses varying up to ±45° on the yaw axis and ±30° on the tilt axis. In [35], an automatic method was developed to find correspondences between 2D facial feature points and a 3D face model. The 3D face model built was then rotated to generate the frontal view.

In our literature review, we focused on 2D methods because they are widely used and are applicable in real time. Nevertheless, the goal of this work is to develop a more accurate recognition method for rotated faces, not to create the fastest possible implementation. There are several methods that use 2D techniques to perform face recognition across pose. A method that performs frontalization by dividing the face into different components is presented in [36]. Several methods use Active Appearance Models (AAM) to frontalize the face [37], [38], [39] and perform the match with a frontal face from a gallery set. In [40] the image is divided into non-overlapping patches and then a statistically aligned model is built for each patch to perform a warping in the region. Using the same idea of patches, the image is divided into non-overlapping patches and a statistical model is constructed on each one at the score level [41]. This method models how the matching score varies when the input face pose is at a certain angle. A face recognition Gabor-based method using a regressor with a coupled bias-variance tradeoff is proposed in [42]. In this method, a statistical model is built at the score level, as in [41]. Some methods use face representations in a latent space to perform the recognition [43], [44]. In [45] a dictionary learning method designed for face recognition is presented.

Several recent papers have shown that the Local Matching Gabor (LMG) method [24] and its variants [25], [26] reach the best results for frontal face recognition. In the LMG method, a total of 4172 Gabor jets are employed to extract features at five different spatial resolutions and eight different orientations. A Borda count method is used to compare the inner products between the Gabor jets from the input face image and the Gabor jets from faces in the gallery [46]. In our previous work based on Gabor-feature face classification [26], faces are normalized using the eye position for coronal axis rotations (on the same face plane). Also, our previously proposed method included weights for the Gabor jets using an entropy measure and a preprocessing step with Local Normalization (LN) yielding results that are among the best face classification results reported on the FERET and AR databases [25], [26], [47]. Our method performed competitively with other published methods on face occlusions and in the presence of noise. Nevertheless, for face poses with increasing angles out of the face plane, the face normalization step loses the correction effect and, as in most 2D face recognition methods, performance declines significantly. For example, methods reaching near 100% for frontal face recognition may drop by up to 40% with pose variations ±60° [40].

In this paper, a new method for face recognition under pose variation is proposed. This method uses Active Shape Models (ASM) [48] to reposition Gabor jets on the face according to face pose. Because of local changes in the 2D face image with varying pose, we also use a local statistical model [41] to compensate for face pose. These new extensions to our previous LMGEW method that uses entropy-weighted features and fusion among LMG and LBP features [26] yield significant improvements in face recognition under varying pose. We tested our method and its variants using the FERET and CMU-PIE databases, which are among the most used databases to evaluate methods of face recognition across pose. The FERET database has pose variation between ±60°, and has been used in many recent publications of face recognition across pose [33], [34], [35], [41], [42], [43], [49]. According to our literature review, our method reached the highest classification performance published in the FERET database with 2D face recognition methods. The CMU-PIE database has pose variations near ±90° and also has been used in most recent publications of face recognition across pose [33], [34], [35], [42], [43], [49], [50], [51]. The performance of our method on the CMU-PIE database is among those that reached the highest classification performance.

Section snippets

Methodology

Our proposed model for face recognition based on local matching Gabor consists of three main modules [24]: image alignment using ASM, feature extraction through Gabor jets computation, and classification using a statistical model, as shown in Fig. 1.

In the first module, an ASM is used to align the face and to determine its shape with the goal of extracting features from corresponding points relative to the gallery image. Also, an illumination compensation method called Local Normalization (LN)

Database and experiments

The FERET and CMU-PIE databases were used to evaluate our method. Fig. 6 shows figures from PIE (top) and FERET (bottom) databases for negative angle poses. A brief description of the databases is given below.

Conclusions

The LMG is a well-established method for frontal face identification with excellent performance. Nevertheless, as with most face identification methods, LMG performance declines significantly for large geometric deformations such as those produced by varying pose. In this study, a new method based on LMG is proposed using ASM to correct the position where Gabor jets are computed with pose changes. Also, the method incorporates a statistical model of the Borda count scores computed by using the

Conflict of interest

None declared.

Acknowledgements

This work was supported by FONDECYT 1120613, and by the Department of Electrical Engineering, Universidad de Chile.

Leonardo A. Cament received the B.S. and P.E. in Electrical Engineering from Universidad de Chile in March 2008 with the highest honors. He is a Ph.D. candidate at the Department of Electrical Engineering, Universidad de Chile. His main interests in research are pattern recognition applied to eye detection, face recognition and human machine interfaces.

References (62)

  • X. Xie et al.

    An efficient illumination normalization method for face recognition

    Pattern Recognit. Lett.

    (2006)
  • P. Campadelli et al.

    Precise eye and mouth localization

    Int. J. Pattern Recognit. Artif. Intell.

    (2009)
  • C. Perez et al.

    Real-time template based face and iris detection on rotated faces

    Int. J. Optomechatronics

    (2009)
  • C.A. Perez et al.

    Real-time iris detection on coronal-axis-rotated faces

    IEEE Trans. Syst. Man Cybern. Part C—Appl. Rev.

    (2007)
  • C.A. Perez et al.

    Genetic design of biologically inspired receptive fields for neural pattern recognition

    IEEE Trans. Syst. Man Cybern. Part B—Cybern.

    (2003)
  • C.A. Perez et al.

    Linear versus nonlinear neural modeling for 2-D pattern recognition

    IEEE Trans. Syst. Man Cybern. Part A—Syst. Hum.

    (2005)
  • M.A. Turk, A.P. Pentland, Face recognition using eigenfaces, in: IEEE Computer Society Conference on Computer Vision...
  • P.N. Belhumeur et al.

    Eigenfaces vs. Fisherfacesrecognition using class specific linear projection

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1997)
  • M.S. Bartlett et al.

    Face recognition by independent component analysis

    IEEE Trans. Neural Netw.

    (2002)
  • M. Sharkas, M. A. Elenien, Eigenfaces vs. Fisherfaces vs. ica for face recognition; a comparative study, in: 9th...
  • T. Ahonen et al.

    Face description with local binary patterns: application to face recognition

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2006)
  • S. Biswas et al.

    Pose-robust recognition of low-resolution face images

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2013)
  • D. Gabor

    Theory of communication

    J. Inst. Electr. Eng.

    (1946)
  • M. Lades et al.

    Distortion invariant object recognition in the dynamic link architecture

    IEEE Trans. Comput.

    (1993)
  • L. Wiskott et al.

    Face recognition by elastic bunch graph matching

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1997)
  • C. Liu et al.

    Gabor feature based classification using the enhanced Fisher linear discriminant model for face recognition

    IEEE Trans. Image Process.

    (2002)
  • W. Zhang, S. Shan, W. Gao, X. Chen, H. Zhang, Local Gabor binary pattern histogram sequence (LGBPHS): a novel...
  • B. Zhang et al.

    Histogram of Gabor phase patterns (HGPP)a novel object representation approach for face recognition

    IEEE Trans. Image Process.

    (2007)
  • Z. Lei, S.Z. Li, R. Chu, X. Zhu, Face recognition with local Gabor textons, in: 2th International Conference on...
  • H. V. Nguyen, L. Bai, L. Shen, Local Gabor binary pattern whitened PCA: a novel approach for face recognition from...
  • J. Zou et al.

    A comparative study of local matching approach for face recognition

    IEEE Trans. Image Process.

    (2007)
  • Cited by (50)

    • Robust face descriptor in unconstrained environments

      2024, Expert Systems with Applications
    • The decadal perspective of facial emotion processing and Recognition: A survey

      2022, Displays
      Citation Excerpt :

      Secondly, an iterative search method can be used to transform the structure of the example according to the two construction models. [71] used as the FER nose, describe the geometric displacement between the location of the predicted ASM feature point and the mean ASM shape. [72] have developed an enhanced version of ASM facial recognition in recent years, called active type and mathematical models (ASSM), which has computational applications for FER.

    • Learning 2D Gabor filters by infinite kernel learning regression

      2021, Journal of Computational Mathematics and Data Science
    • Automatic space object detection on all-sky images from a synoptic survey synthetic telescope array

      2020, Advances in Space Research
      Citation Excerpt :

      Thus, we apply an OR operator among detections from different zones. Since SOs are observed as line segments in the images, a Progressive Probabilistic Hough Transform Segment Detector (Galamhos et al., 1999; Cament et al., 2015; Castillo et al., 2014) was applied after the Canny edge detector. This transform is widely used to detect objects with known geometry.

    • Multiple feature descriptors based model for individual identification in group photos

      2019, Journal of King Saud University - Computer and Information Sciences
      Citation Excerpt :

      This feature space provides the rotation invariant and multi-scaled feature descriptor to attain robust facial recognition. Another improvement to Gabor features was provided by incorporating the block feature analysis using entropy weight (Cament et al., 2015) assignment. This improved local normalization method provided the compensation against pose variant and illumination variant feature enhancement.

    View all citing articles on Scopus

    Leonardo A. Cament received the B.S. and P.E. in Electrical Engineering from Universidad de Chile in March 2008 with the highest honors. He is a Ph.D. candidate at the Department of Electrical Engineering, Universidad de Chile. His main interests in research are pattern recognition applied to eye detection, face recognition and human machine interfaces.

    Francisco J. Galdames received the B.S. and P.E. in Electrical Engineering and the M.S degree in Biomedical Engineering all from Universidad de Chile in 2006. In 2012 he obtained the Ph.D from Universidad de Chile and Université Joseph Fourier. He is a postdoctoral fellow at Universidad de Chile. His interests include pattern recognition applied to face recognition and 3D modeling.

    Kevin W. Bowyer is the Chair of the Department of Computer Science and Engineering at the University of Notre Dame. Professor Bowyer received a 2014 Technical Achievement Award from the IEEE Computer Society, and is a Fellow of the IEEE and the IAPR. His main research interests are in computer vision and pattern recognition, including biometrics, data mining, object recognition and medical image analysis.

    Claudio A. Perez received the B.S. and P.E. in Electrical Engineering and the M.S. degree in Biomedical Engineering all from Universidad de Chile in 1980 and 1985, respectively. He was a Fulbright student at the Ohio State University where he obtained the Ph.D. in 1991. He was a visiting scholar at UC, Berkeley in 2002 through the Alumni Initiatives Award Program from Fulbright Foundation. Currently he is a Professor at the Department of Electrical Engineering, Universidad de Chile.

    View full text