Facial feature extraction by a cascade of model-based algorithms

Abstract

In this paper, we propose a cascaded facial feature-extraction framework employing a set of model-based algorithms. In this framework, the algorithms are arranged with increasing model flexibility and extraction accuracy, such that the cascaded algorithm can have an optimal performance in both robustness and extraction accuracy. Especially, we propose a set of guidelines to analyze and jointly optimize the performance relation between the constituting algorithms, such that the constructed cascade gives the best overall performance. Afterwards, we present an implementation of the cascaded framework employing three algorithms, namely, sparse-graph search, component-based texture fitting and component-based direct fitting. Special attention is paid on the search and optimization of the model parameters of each algorithm, such that the overall extraction performance is greatly improved with respect to both reliability and accuracy.

Introduction

Recent advances in image processing and vision computing enable automatic identification of human faces in visual scenes. One promising application for face recognition is consumer home applications, where face-recognition systems facilitate intelligent and convenient services for daily life.

Within the framework of the ITEA project HomeNet2Run [10], we have developed a face-identification system [22], which actively identifies home users and notifies other home devices (e.g. home server/television) to adapt services accordingly. Facial feature extraction is an indispensable step in the processing, which lays the basis for both face alignment to the database format and the subsequent recognition stage. The robustness and accuracy of the facial feature extraction directly influences the success of the final identification.

However, to achieve an automatic facial feature-extraction system, a number of challenges should be addressed.

• The facial feature appearances and their geometric relationships are considerably influenced by individual variations and also affected by various facial expressions.

• Especially for non-professional applications, the image-capturing environments are more variable than for professional cases. Particularly camera qualities and illumination conditions exert difficulties to the feature-extraction procedure.

A good feature-extraction algorithm should provide accurate feature locations while being robust against large feature variances. Furthermore, for embedded/consumer applications, computation cost is also an important factor that should be taken into account during the algorithm design.

In the following, we present a brief literature review of current feature-extraction techniques and point out their limitations.

In the past decade, a number of techniques have been proposed in literature on automatic facial feature extraction. Earlier techniques find the positions of facial features based on empirical knowledge about facial feature appearances (e.g. [12], [18]), such as eye symmetry and varying color distributions over the features. A more recent paper [8] makes use of edge properties around facial landmarks and reports quite good results. Some other techniques treat feature component (e.g. eyes and nose) localization as a two-class pattern-classification problem, where the classes are feature and non-feature. Typical examples include the eigenfeature-based techniques [19] and Adaboost-based techniques [5]. These techniques are relatively robust for large feature variances. However, they usually provide only coarse localization results.

Recently, model-based algorithms have received great attention in literature. A face model (e.g. a graph model or patch model) is usually devised to represent the salient facial features, and its parameters are optimized to fit to a new face. These techniques usually offer both accurate description of features and flexibility to adapt to individual feature variances. Some earlier examples of model-based techniques are active contour models [1] and deformable geometric templates [20]. These techniques incorporate object-specific a priori knowledge about facial features into pre-defined energy functions. Although these techniques may lead to quite accurate feature-extraction results, the fitting procedure may converge to incorrect local minima due to improper model initializations and large feature variances, resulting in a limited convergence performance.

The well-known active shape model (ASM) [3] and active appearance model (AAM) [4] implicitly define facial feature properties by using statistics, which automatically derives feature models by the use of a set of training samples. The statistics-based approach for feature extraction offers more flexibility, which also can be easily applied to other object structures. Later on, many derivatives are proposed. In direct AAM [11], it has been suggested that it is possible to predict the shape directly from the texture when the two are sufficiently correlated. In [17], a nonlinear extension has been proposed to enhance the performance of the conventional ASM.

From the above, we can see that it is generally difficult for a single technique to achieve high performance in both robustness and accuracy. Classification-based algorithms are not influenced by initializations and therefore are more robust. The statistical point models can give fairly accurate results, but they are more sensitive to model initializations. To this end, many recent proposals explore different properties of facial features in order to benefit from various algorithms. For example, it is well-known that ASM/AAM performs poorly if the model is not initialized properly. In [13], a rough estimation of major feature components (e.g. eyes and mouth) is used based on color information, in order to provide a better model initialization for ASM. In [6], the authors first apply a set of independent feature detectors for 17 feature points using the Adaboost classification technique. Following that, a conventional AAM is applied to refine the results.

In this paper we focus on using multiple algorithms to improve the facial feature-extraction performance with respect to both robustness and accuracy. As compared to previous approaches, the following differences become apparent.

• We have clearly defined a number of performance metrics (e.g. the capture range, accuracy threshold and localization error) to characterize a model-based algorithm. These concepts have not been precisely defined previously in literature. The performance metrics are used extensively later on in the implementation of a cascaded algorithm to guide the optimization of model parameters.

• In previous approaches, several techniques are combined heuristically without quantitatively addressing their relation in performance. In our approach, we quantitatively analyze the coupling between multiple algorithms and propose ways to optimize the coordination between the algorithms.

• We have proposed an implementation example using three cascaded algorithms. The proposed principles are applied to guide the optimization of the algorithm parameters such that the overall performance is enhanced.

The remainder of the paper is organized as follows. In Section 2, we first define several performance metrics to characterize model-based algorithms. Section 3 presents the cascaded feature-extraction framework, where two important principles are proposed to provide guidance for the subsequent algorithm design. Following that, Section 4 gives a brief overview of an implementation of the extraction cascade consisting of three component algorithms, namely, sparse-graph search (SGS), prediction-based texture fitting and appearance-based feature refinement. Sections 5–7 present the detailed illustrations of the three algorithms. Section 8 analyzes the overall performance of the system and Section 9 concludes the paper.