Elsevier

Pattern Recognition

Volume 41, Issue 3, March 2008, Pages 1173-1185
Pattern Recognition

Hybrid-boost learning for multi-pose face detection and facial expression recognition

https://doi.org/10.1016/j.patcog.2007.08.010Get rights and content

Abstract

This paper proposes a hybrid-boost learning algorithm for multi-pose face detection and facial expression recognition. To speed-up the detection process, the system searches the entire frame for the potential face regions by using skin color detection and segmentation. Then it scans the skin color segments of the image and applies the weak classifiers along with the strong classifier for face detection and expression classification. This system detects human face in different scales, various poses, different expressions, partial-occlusion, and defocus. Our major contribution is proposing the weak hybrid classifiers selection based on the Harr-like (local) features and Gabor (global) features. The multi-pose face detection algorithm can also be modified for facial expression recognition. The experimental results show that our face detection system and facial expression recognition system have better performance than the other classifiers.

Introduction

Automatic face detection has many applications such as surveillance, human computer interface (HCI). Most of the published methods assume: front-view pose, minimum out-of-plane head motion, and constant illumination of which the illumination variation is the most difficult one. Accuracy and efficiency are two of the most important issues in evaluating a face detection system. Most of the previous face detection systems focus on the eyes as the most prominent feature of the face. Instead of treating the face detection as a binary classification problem, we propose a multi-class hybrid-boost learning algorithm which selects the most discriminative Gabor features and Harr-like features for multi-pose face detection and expression identification.

Most of the previous face detection researches [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12] have many restrictions, such as no varying pose nor noisy defocus problem. Human face detection algorithms rely on the extracted facial features. The detected feature vector can also be applied for identifying the face in different poses and expressions. Viola et al. [3] introduce Adaboost with a cascade scheme and apply an integral image concept for face detection. They propose two-class AdaBoost learning algorithm for training efficient classifiers and a cascaded structure for rejecting non-face images.

Huang et al. [7] propose a novel tree-structured multi-view face detector (MVFD) called Vector Boosting, using the coarse-to-fine strategy to divide the entire face space into smaller and smaller subspaces. They developed a Width-First-Search (WFS) tree structure to achieve higher performance in both speed and accuracy. Li et al. [8] introduce the FloatBoost by using the floating search algorithm. There are basically three kinds of feature selection methods: Sequential Forward Selection (SFS), Sequential Backward Selection (SBS), and Sequential Floating Search Method (SFSM). FloatBoost algorithm [11] uses SFSM to select features and the training time is five times longer than AdaBoost. Xu et al. [12] propose an MRC-boosting algorithm which may compute the most discriminative feature in close-form.

Besides face detection in the image, there are other research interests regarding to the facial expression extraction [13], [14], [15]. Datcu et al. [16] propose a novel facial expression recognition method by using Relevance Vector Machine (RVM) [17], [18]. In Ref. [19], Zhao et al. present a new texture modeling based on volume local binary patterns (VLBP) which can be applied for analyzing so-called dynamic textures and facial expression. The face expression extraction methods can be divided as local analysis [20], [24] and global analysis [21], [22]. The former focuses on some specific feature points and divides the face into three local graphical objects: right eye graph, left eye graph, and mouth graph. The latter determines the features by processing the entire face by boosting Harr feature based weak classifier.

Adaboost algorithm has also been widely applied for real time facial expression recognition [21], [22]. Silapachote et al. [23] proposes a classification technique for face expression recognition using Adaboost to select the relevant global and local appearance feature with the most discriminant information. The other methods [20], [24] analyze the internal representation of facial expressions based on collections of Action Units (AUs). The local analysis needs some additional verification step to avoid feature errors. Improper feature points deteriorate the recognition accuracy, and more feature points require more computation to fit the model to the face image. These restrictions make the systems more complicated and inadequate for real-time processing. Kanade et al. [25] present the CMU AU-coded face expression image database which is the most comprehensive testbed for comparative studies of facial expression.

Here, we propose a hybrid-boost learning which selects Gabor features (for global appearance) and Harr-like features (for local appearance) to provide the most discriminating information for the strong classifier in the final stage. Our face detection system locates the potential face regions by using skin color detection and segmentation, and then searches for the hybrid features for the multi-class strong classifier to detect the multi-pose face and different facial expressions. Our system is robust to various size, poses, expressions, and defocus problems. The experimental results show that our method has a better system performance than the other methods.

Section snippets

Segmentation of potential face regions

The 1st module, potential face regions segmentation, consists of skin color detection and segmentation. To identify the existence of human face, it scans the image to detect the skin color regions and remove unnecessary pixels. To reduce the search region, we need to locate the possible face region. In the captured human face images, we assume that the color distribution of human face is somehow different from that of the image background. Pixels belonging to face region exhibit similar

Hybrid-boost learning for face detection

Here, we propose the hybrid-boost learning which iteratively chooses the weak classifiers that minimize the exponential loss function. The weak classifiers consist of Gabor features and Harr-like features which characterize the salient visual properties such as spatial localization, orientation selectivity, and spatial frequency characteristics of the human faces. The Harr-like features are easier to be obtained than the Gabor features. Similar to Adaboost, the hybrid-boost learning algorithm

Multi-pose face detection and expression recognition

Many face detection techniques can detect the frontal upright faces in wide variety of images. However, most of the methods can only deal with the front-pose faces. Here, we introduce a multi-pose face detection method to detect the faces in various poses. Instead of treating the face detection or expression recognition as a cascade binary classification problem, we propose a multi-class classification algorithm to solve the multi-pose face detection problem. This algorithm selects a small

Experimental results and discussions

Our system can also be implemented by using AMD 3000+ CPU and the image size is 320×240 pixels. In the first frame of a video sequence, we apply the face detector to search the entire image for the presence of different scale faces and the corresponding poses simultaneously. Once the face is detected, the face tracking is used to search and identify the face in the following image frames. The face tracking is also a face detection process with much smaller search area based on the detected face

Conclusions and feature works

We have introduced a multi-posed face detection and expression identification system which is more robust than the other proposed face detection system and facial expression system. Our system is based on hybrid-boost multi-class learning algorithm as well as three decision rules which generates higher detection rate and lower false alarm rate. The experimental results show that the system has better performance than the others using Harr-like feature or Gabor feature.

About the authorHSIAO-YING CHEN received her B.S., degree in electrical and control engineering from the National Chaio-TungUniversity, Hsinchu, Taiwan, and MS degree in electrical engineering from the National Tsing-Hua University, Hsinchu, Taiwan, in 2004, and 2006, respectively. Currently, she is with Media Technology, Inc., Hsinchu. Her research interests are pattern recognition and computer vision.

References (32)

  • E. Hjelmas et al.

    Face detection: a survey

    Comput. Vision Image Understanding

    (2001)
  • B. Fasel et al.

    Automatic facial expression analysis: a survey

    Pattern Recognition

    (2003)
  • M.-H. Yang et al.

    Detecting faces in images: a survey

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2002)
  • P. Viola, M. Jones, Rapid object detection using a boosted cascade of simple features, IEEE CVPR, 2001, pp....
  • M. Jones, P. Viola, Fast multi-view face detection, Technical Report. TR2003-96, Mitsubishi Electric Research...
  • P. Moreno et al.

    Gabor parameter selection for local feature detection

    IbPRIA 2005, Lecture Notes in Computer Science

    (2005)
  • G. Guo, H.J. Zhang, Boosting for fast face recognition, IEEE ICCV Workshop on Recognition, Analysis, and Tracking of...
  • C. Huang, H. Ai, Y. Li, S. Lao, Vector boosting for rotation invariant multi-view face detection, Proceedings of the...
  • S.Z. Li et al.

    Floatboost learning and statistical face detection

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2004)
  • R. Xiao, M.-J. Li, H.-J. Zhang, Robust multi-pose face detection in images, IEEE Trans. Circuits Systems Video Technol....
  • R. Verschae, J. Ruiz del Solar, A hybrid face detector based on an asymmetrical adaboost cascade detector and a...
  • Y.Y. Lin, T.L. Liu, Robust face detection with multi-class boosting, Proceedings of IEEE CVPR,...
  • X.Xu, T. Huang, Face recognition with MRC-boosting, Proceedings of IEEE, ICCV,...
  • M. Pantic et al.

    Automatic analysis of facial expressions: the state of the art

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2000)
  • M.S. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, J. Movellan, Recognizing facial expression—machine...
  • D. Datcu, L.J.M. Rothkrantz, Facial expression recognition with relevance vector machines, ICME,...
  • Cited by (63)

    • MPMFFT based DCA-DBT integrated probabilistic model for face expression classification

      2020, Journal of King Saud University - Computer and Information Sciences
    • Multi-gradient features and elongated quinary pattern encoding for image-based facial expression recognition

      2017, Pattern Recognition
      Citation Excerpt :

      Furthermore, additional studies have combined local descriptors with complimentary descriptors, such as the work in [12] or previous work in [13] where the recognition performance is improved significantly. All the above works have been applied on different static or dynamic types of posed databases such as the Cohn-Kanade (CK) or Multimedia Understanding Group (MUG) databases that were collected under tightly controlled conditions [14]. Recently, spontaneous facial expression recognition has been gaining more attention than posed facial expression recognition.

    • Adaptive facial point detection and emotion recognition for a humanoid robot

      2015, Computer Vision and Image Understanding
      Citation Excerpt :

      Local Gabor binary patterns was especially evaluated in their work using images with different viewing angles. Chen et al. [37] also proposed a hybrid-boost learning mechanism to deal with face detection and emotion recognition against pose variations. Skin color detection and segmentation algorithms were used to first locate face region.

    • Spontaneous facial expression recognition: A robust metric learning approach

      2014, Pattern Recognition
      Citation Excerpt :

      The ability to interpret facial gestures is a key to a wide range of HCI applications. Researchers have achieved tremendous success in recognizing prototypical and posed facial expressions that are collected under tightly controlled conditions [1–3]. Since the most useful current and future face related applications lie in a more natural context, it is our goal in this paper to develop a system that can operate on spontaneous expressions characterizing the natural interaction between humans and computers.

    • Design of Face Recognition System Based on Convolutional Neural Network

      2019, Proceedings - 2019 Chinese Automation Congress, CAC 2019
    View all citing articles on Scopus

    About the authorHSIAO-YING CHEN received her B.S., degree in electrical and control engineering from the National Chaio-TungUniversity, Hsinchu, Taiwan, and MS degree in electrical engineering from the National Tsing-Hua University, Hsinchu, Taiwan, in 2004, and 2006, respectively. Currently, she is with Media Technology, Inc., Hsinchu. Her research interests are pattern recognition and computer vision.

    About the authorCHUNG-LIN, Dr. Huang received his B.S. degree in Nuclear Engineering from the National Tsing-Hua University, Hsin-Chu, Taiwan, ROC, in 1977, and M.S. degree in Electrical Engineering from National Taiwan University, Taipei, Taiwan, ROC, in 1979, respectively. He obtained his Ph.D. degree in Electrical Engineering from the University of Florida, Gainesville, FL, USA, in 1987. From 1987 to 1988, he worked for the Unisys Co., Orange County, CA, USA, as a Project Engineer. Since August 1988 he has been with the Electrical Engineering Department, National Tsing-Hua University, Hsin-Chu, Taiwan, ROC. Currently, he is a Professor in the same department. In 1993 and 1994, he had received the Distinguish Research Awards from the National Science Council, Taiwan, ROC. In November 1993, he received the best paper award from the ACCV, Osaka, Japan, and in August 1996, he received the best paper award from the CVGIP Society, Taiwan, ROC. In December 1997, he received the best paper award from IEEE ISMIP Conference held Academia Sinica, Taipei. In 2002, he received the best paper annual award from the Journal of Information Science and Engineering, Academia Sinica, Taiwan. His research interests are in the area of image processing, computer vision, and visual communication. Dr. Huang is a senior member of IEEE.

    About the authorCHIH-MING FU received his B.S., M.S., and Ph.D. degrees in electrical engineering from the National Tsing-Hua University, Hsinchu, Taiwan, ROC., in 1999, 2001, and 2006, respectively. Currently, he is with Cheertek Technology, Inc., Hsinchu. His research interests are wavelet analysis, signal processing, image/video processing, pattern recognition, and multimedia communication. In August 2006, he received the best paper award from the CVGIP Society, Taiwan.

    This revised version is submitted to Pattern Recognition for peer review.

    1

    The corresponding author is also with Department of Informatics, Fo-Guang University, I-Lan, Taiwan.

    View full text