Hallucinating faces: LPH super-resolution and neighbor reconstruction for residue compensation

Abstract

A two-phase face hallucination approach is proposed in this paper to infer high-resolution face image from the low-resolution observation based on a set of training image pairs. The proposed locality preserving hallucination (LPH) algorithm combines locality preserving projection (LPP) and radial basis function (RBF) regression together to hallucinate the global high-resolution face. Furthermore, in order to compensate the inferred global face with detailed inartificial facial features, the neighbor reconstruction based face residue hallucination is used. Compared with existing approaches, the proposed LPH algorithm can generate global face more similar to the ground truth face efficiently, moreover, the patch structure and search strategy carefully designed for the neighbor reconstruction algorithm greatly reduce the computational complexity without diminishing the quality of high-resolution face detail. The details of synthetic high-resolution face are further improved by a global linear smoother. Experiments indicate that our approach can synthesize distinct high-resolution faces with various facial appearances such as facial expressions, eyeglasses efficiently.

Introduction

Video surveillance cameras are already prevalent in banks, stores and parking lots, but in many cases, the size of interested faces is often small because of the distance between the camera and the object. The low image resolution of human face becomes a primary obstacle to face identification and recognition. Therefore, in order to gain detailed facial features for recognition, it is necessary to infer high-resolution face image from the low-resolution one and this technique is called face hallucination [1], [2], [3], [4], [5], [6] which incorporates image super-resolution techniques into facial image synthesis. Currently, the image super-resolution techniques can be divided into two classes: learning-based techniques [1], [2], [7], [8] and reconstruction-based techniques [9], [10], [11], and usually the former generates better results than the latter does. This paper focuses on learning-based face hallucination technique.

A number of learning-based image hallucination techniques have been proposed in recent years [7], [8], [12]. The common gist shared by them is to learn hallucinated high-resolution faces from a training database which comprises pairs of high-resolution and low-resolution image samples. For example, Freeman [7] proposed a sample-based method; they trained Markov networks to learn the relationship between the low-resolution image and fine details of corresponding high-resolution image, and then used the learned relationship to infer fine details of other low-resolution images. Hertzmann [8] proposed a generalized local feature transform approach called “image analogies”. According to this approach, the high-resolution image was inferred from the low-resolution input image by the local similarity between the training pairs based on a multi-scale auto-regression. These above approaches were suitable for dealing with generic images rather than face image because they ignored the special property of face images. The term “face hallucination” was first proposed by Baker and Kanade [1]. They developed a hallucination approach based on the property of human faces which learned a prior on the spatial distribution of the image gradient for frontal face images and employed Bayesian theory to infer the high-resolution face image from the low-resolution one. The approach intrinsically relied on a complicated statistical model. Su et al. [13] used a steerable pyramid to extract multi-orientation and multi-scale information of low-level facial features both from the input low-resolution face and other high-resolution ones, and then adopted pyramid parent structure and local best match to optimize the prior for solving a Bayesian MAP problem. Wang and Tang [3] developed an efficient face hallucination approach based on an eigen-transformation algorithm. They used principal component analysis (PCA) to fit the input face image as a linear combination of the training low-resolution face images, and then the high-resolution image was synthesized by replacing the training low-resolution face images with the high-resolution counterparts while retaining the same combination weights. But the methods only focused on global information without paying attention to local details, so the results seemed unclear and lacked detailed features. Liu et al. [2] developed a two-step statistical approach which integrated a global parametric model and a local model with Markov random field (MRF) prior. This approach depended on explicit down-sampling function which is sometimes unavailable in practice. Enlightened by Liu's work, a number of existing works treated face hallucination as a two-step problem [4], [5], [6]. First, a global face image keeping the main characteristics of the high-resolution face was obtained which looks smooth and lacks some detailed features. Second, an optimal residue face image containing the high-frequency image information was synthesized and the residue image was piled onto the global face image to gain the final results.

Motivated by the two-step architecture, we propose a statistical learning-based face hallucination approach which includes two phases. In the first phase, instead of PCA used by Liu et al., locality preserving projections (LPP) is adopted to learn the intrinsic features of the training low-resolution face images and build a set of transformation vectors beforehand. Then the transformation between low- and high-resolution image pairs is achieved by a radial basis function (RBF) regression between the intrinsic features of low-resolution images and corresponding high-resolution images. The inference can be done by projecting the input low-resolution face onto the transformation vectors and using the gained intrinsic features to obtain global high-resolution face via RBF regression. We state locality preserving hallucination (LPH) as the name of this method. In the second phase, different from Liu's approach, we infer high-resolution residue image from the low-resolution residue image rather than the global high-resolution image based on the low- and high-resolution training residue image pairs. In this phase, enlightened by locally linear embedding (LLE), a neighbor reconstruction based method is used to fit the input low-resolution residue patch as a linear combination of the K-NN low-resolution patches in the training set, and then the high-resolution residue patch can be inferred by replacing the training low-resolution residue patches with the high-resolution counterparts while preserving the same combination weights. The overlapped high-resolution residue patches are then smoothed by a linear operator to form the high-resolution residue face. The final hallucinated face image is the composition of the global high-resolution image and the high-resolution residue image. The outline of our approach can be described in Fig. 1.

It is worthwhile to highlight several aspects of our proposed approach here:

• Unlike other “batch” manifold algorithms such as isometric feature mapping (ISOMAP) and LLE, LPP is a linear method and it is defined everywhere by explicitly giving a set of transformation vectors. This indicates that LPP can be easily applied to any new data point out of the training set. Furthermore, by preserving the local structure of the image space, LPP provides better results than PCA in many real world applications.

• We originally build a global face hallucination framework named LPH by combining the manifold learning algorithm (LPP) and neural network (RBF regression). In the framework, LPP captures the most intrinsic features of training images while RBF regression is bridging connection between the intrinsic image features and original images. Compared with linear PCA, LPH generates global high-resolution face more similar to the ground truth image with high computational efficiency.

• Unlike previous works [2], [14], our search strategy in the second phase is position-dependent; this means when fitting an input patch with specific image position, the search is localized at the same position throughout the training image set. This will not diminish the quality of face details because patches at same position denote roughly the same region of human face naturally, and the K-NNs are more probable to appear in these patches. In addition, our K-NN search is applied on the low-resolution image patches instead.So, our search strategy greatly reduces the computational complexity of the neighbor searching process.

The rest part of this paper is organized as follows: Section 2 shows the reviewing of related works, the proposed approach to face hallucination is discussed in detail in Section 3, convincible experiments are shown in 4 Experiments and discussion, 5 Conclusions and future work concludes this paper.