Deep Learning-Based Face Analytics

The need for face recognition has evolved from identifying a few hundred people to identifying hundreds of thousands of people in the last decade. Most of the progress in automatic face recognition has been driven by deep networks in the past few years. In this article, we provide an overview of recent progress in this area and discuss state-of-the-art CNN-based face recognition and verification systems. We also present some open questions and discuss avenues for research in the coming years.

Face swapping refers to the task of changing the appearance of a face appearing in an image by replacing it with the appearance of a face taken from another image, in an effort to produce an authentic-looking result. We describe a method for face swapping that does not require training on faces being swapped and can be easily applied even when face images are unconstrained and arbitrarily paired. Our method offers the following contributions: (a) Instead of tailoring systems for face segmentation, as others previously proposed, we show that a standard fully convolutional network (FCN) can achieve remarkably fast and accurate segmentation, provided that it is trained on a rich enough example set. For this purpose, we describe novel data collection and generation routines which provide challenging segmented face examples. (b) We use our segmentations for robust face swapping under unprecedented conditions, without requiring subject-specific data or training. (c) Unlike previous work, our swapping is robust enough to allow for extensive quantitative tests. To this end, we use the Labeled Faces in the Wild (LFW) benchmark and measure how intra- and inter-subject face swapping affect face recognition. We show that intra-subject swapped faces remain as recognizable as their sources, testifying to the effectiveness of our method. In line with established perceptual studies, we show that better face swapping produces less recognizable inter-subject results (see, e.g., Fig. 2.1). This is the first time this effect was quantitatively demonstrated by a machine vision method. Some of the material in this chapter previously appeared in [47].

The goal of every contemporary recognition approach is to learn robust and unambiguous object representations in feature space. These learned powerful disentangled representations make it possible to build effective classifiers and are an active research topic in many fields such as face analytics.

As a classic statistical model of 3D facial shape and albedo, 3D Morphable Model (3DMM) is widely used in facial analysis, e.g., model fitting, and image synthesis. Conventional 3DMM is learned from a set of 3D face scans with associated well-controlled 2D face images, and represented by two sets of PCA basis functions. Due to the type and amount of training data, as well as, the linear bases, the representation power of 3DMM can be limited.

Image deblurring entails the recovery of an unknown true image from a blurry image. Image deblurring is an ill-posed problem, and therefore, it is crucial to leverage additional properties of the data to successfully recover the lost facial details in the deblurred image. Priors such as sparsity [2, 13, 16], low-rank [14], manifold [10], and patch similarity [22] have been proposed in the literature to obtain a regularized solution.

Super-resolution (SR) refers to a class of techniques that derive a high resolution image from its low resolution (LR) counterpart. A vast amount of literature exists on SR spanning both multi and single image approaches.

Face recognition methods have made great progress in the recent years. These methods most of the time represent a face image as a high-dimensional real-valued feature, often obtained using a deep network. However, comparisons of this high-dimensional feature can be computationally expensive. Furthermore, when dealing with large face images database this representation can lead to prohibitive storage requirements. Also, in a context where the capture of the face image is performed on a mobile device or in a separate location from the face verification or search process, the amount of data that needs to be transmitted over the network should be minimized.

Accidental new born swapping, health-care tracking, and child-abduction cases are some of the scenarios where new born face recognition can prove to be extremely useful. With the help of the right biometric system in place, cases of swapping, for instance, can be evaluated much faster. In this chapter, we first discuss the various biometric modalities along with their advantages and limitations. We next discuss the face biometrics in detail and present all the datasets available and existing hand-crafted, learning-based, as well as deep-learning-based techniques which have been proposed for new born face recognition. Finally, we evaluate and compare these techniques. Our comparative analysis shows that the state-of-the-art SSF-CNN technique achieves an average of rank-1 new born accuracy of \(82.075\%\).

Data produced from a particular source often exhibit correlations with those arising from other sources. Common data sources include (i) sensors—that gather raw data, (ii) feature extractors—which process the raw data from sensors to generate features representing the original data, and (iii) evaluators- –which produce a score or a measure that conveys the likelihood of the provided features belonging to an application specific hypothesis.

This chapter is concerned with face recognition based on videos or, more generally, sets of images, using deep learning techniques. We first briefly review some naive yet commonly used strategies pertaining to using frame-level features extracted by deep convolutional neural networks (CNNs) for video-level face recognition. Representative strategies include naive feature pooling and pairwise feature distance computation. Then, we present a method named neural aggregation network (NAN), which is a deep learning framework tailored for video-based representation and recognition. NAN can automatically learn the quality of faces in a video/image set and aggregate the frame-level deep features accordingly, yielding more discriminative video-level features. We conduct experimental evaluation on three video face recognition datasets. The results indicate that while previous deep learning-based methods with naive pooling or pairwise distances have obtained substantial improvements over traditional methods, the NAN method further outperforms them by an appreciable margin.

Face is one of the most widely used biometrics. One key advantage of using faces as a biometric is that they do not require the cooperation of the test subject. Various face recognition (FR) systems have been developed over the last two decades. Recent advances in machine learning and computer vision methods have provided robust systems that achieve significant gains in performance of face recognition systems [5, 19]. Deep learning methods, enabled by the vast improvements in processing hardware coupled with the ubiquity of face data and algorithmic development, have led to significant improvements in face recognition accuracy, particularly in unconstrained scenarios [4, 5, 19, 20, 27]. Also, largely driven by social network companies, progress in face recognition research, development, and deployment have focused on faces collected in visible regimes of the electromagnetic spectrum.

Recent years have seen fast development in synthesizing realistic human faces using AI technologies. AI-synthesized fake faces can be weaponized to cause negative personal and social impact. In this work, we develop technologies to defend individuals from becoming victims of recent AI-synthesized fake videos by sabotaging would-be training data. This is achieved by disrupting deep neural network (DNN)-based face detection and facial landmark extraction method with specially designed imperceptible adversarial perturbations to reduce the quality of the detected faces. We empirically show the effectiveness of our methods in disrupting state-of-the-art DNN-based face detectors and facial landmark extractors on several datasets.

Face recognition has emerged as a widely used biometric modality. However, its vulnerability to presentation attacks remains a significant security threat. Although Presentation Attack Detection (PAD) methods attempt to remedy this problem, often they fail in generalizing to unseen attacks and environments. As the quality of presentation attack instruments improves over time, achieving reliable PA detection using only visual spectra remains a major challenge. We argue that multi-channel systems could help solve this problem. In this chapter, we first present an approach based on a multi-channel convolutional neural network for the detection of presentation attacks. We further extend this approach to a one-class classifier framework by introducing a novel loss function that forces the network to learn a compact embedding for the bonafide class while being far from the representation of attacks. The proposed framework introduces a novel way to learn a robust PAD system from bonafide and available (known) attack classes. The superior performance in unseen attack samples in publicly available multi-channel PAD database WMCA shows the effectiveness of the proposed approach. Software, data, and protocols for reproducing the results are made publicly available.

Person re-identification across cameras is an important problem as it enables associating targets over a wide area, which is likely to be viewed by multiple cameras. It is an extremely active area of research today. Most of the approaches are extensively supervised, in the sense that they require significant labeling effort to train re-identification models, usually based on deep networks. However, as in other problems in computer vision, it raises the question of scalability of the approaches as the number of people to be associated grows or the size of the network grows. In this chapter, we focus on two problems that hold the potential for developing highly scalable person re-identification approaches. In the first, we focus on the problem of how to limit the labeling effort even as the number of targets in the network grows. On the challenging Market-1501 dataset, we demonstrate that with only 8% labeling, we can achieve performance very close to that with full-set labeling. In the second problem, we focus on the size of the camera network and consider how to onboard new cameras into an existing network with little to no additional supervision. We leverage upon transfer learning approaches for this purpose and demonstrate the results on a benchmark dataset. Overall, the chapter provides some research directions and initial results in pushing person re-identification beyond fully supervised approaches and lays the groundwork for future research in this area.

Measuring algorithmic bias is crucial both to assess algorithmic fairness and to guide the improvement of algorithms. Current bias measurement methods in computer vision are based on observational datasets and so conflate algorithmic bias with dataset bias. To address this problem, we develop an experimental method for measuring algorithmic bias of face analysis algorithms, which directly manipulates the attributes of interest, e.g., gender and skin tone, in order to reveal causal links between attribute variation and performance change. Our method is based on generating synthetic image grids that differ along specific attributes while leaving other attributes constant. Crucially, we rely on the perception of human observers to control for synthesis inaccuracies when measuring algorithmic bias. We validate our method by comparing it to a traditional observational bias analysis study in gender classification algorithms. The two methods reach different conclusions. While the observational method reports gender and skin color biases, the experimental method reveals biases due to gender, hair length, age, and facial hair. We also show that our synthetic transects allow for a more straightforward bias analysis on minority and intersectional groups.

Face recognition by machines has improved markedly over the last decade. Machines now perform some face recognition tasks at the level of untrained humans and forensic face identification experts. In this chapter, first we review recent work on human and machine performance on face recognition tasks. Second, we consider the benefits of statistically fusing human and machine responses to improve performance. Third, we review strategic differences in how humans with various levels of expertise approach face identification tasks. We conclude by considering the challenging problem of human and machine performance on recognition of faces of different races. Understanding how humans and machines perform these tasks can lead to more effective and accurate face recognition in applied settings.

While face recognition research has been perennial and popular since its inception, there has been a marked escalation in this research in recent years due to the confluence of several factors, primarily the development of advanced machine learning algorithms, free and robust software implementations thereof, ever faster GPU processors for running them, vast web-scraped face image databases, open performance benchmarks, and a vibrant face recognition literature.