Review ArticleEditor's Choice ArticleComparison of human and computer performance across face recognition experiments☆
Graphical abstract
Introduction
Overall, humans are the most accurate face recognition systems. People recognize faces as part of social interactions, at a distance, in still and video imagery, and under a wide variety of poses, expressions, and illuminations. A holy grail in automatic face recognition is developing an algorithm that has performance equivalent to humans—this is equivalent to solving the general face recognition problem. While it is easy to state the problem, accuracy equivalent to humans, it is not obvious how to determine if an algorithm's recognition accuracy is better than a human. One of the key challenges is establishing a measurable goal line and knowing when the goal line is crossed.
Since 2005, human and computer performance has been systematically compared as part of face recognition competitions conducted by the National Institute of Standards and Technology (NIST) [1], [2], [3], [4]. The comparisons provided an assessment of accuracy for both humans and machines for each competition. However, there has not been a systematic analysis of these results across the competitions.
To analyze the results across experiments, we introduce the cross-modal performance analysis (CMPA) framework, which is demonstrated on the NIST competitions. CMPA was adapted from techniques in neuroscience that were developed to compare output from different sensing modalities of brain activity; e.g., functional magnetic resonance imaging (fMRI) and human perceptual judgments [5], [6]. These techniques can measure concordance between experimental data and computational models. In our study, the modalities compared are human and algorithm performance. In the psychology and neuroscience literature, face recognition algorithms can be referred to as computational models. The computational model can be designed to optimize performance or to model the human face recognition processes. The framework is sufficiently general that it provides a goal line for determining when machine performance reaches human levels.
On frontal faces in high quality still images, our analysis shows that machine performance is superior to humans. For these images, machines represent a person's identity primarily by encoding information extracted from the face; information from the body, hair, and head is generally ignored. For video and extremely difficult-to-recognize face pairs, experiments show that humans take advantage of all available identity cues when recognizing people [7], [8]. CMPA quantifies the potential for improving machine performance if all possible identity information is encoded by algorithms.
Comparing machine and human performance started with independent experiments in NIST competitions. The synthesis of the results across experiments gives a greater understanding of the relative strengths of machines and humans. The CMPA framework provides a goal line for determining if algorithm and human performance is comparable on the general face recognition problem.
Section snippets
Review of human and machine comparisons
We examine the relative performance of humans and machines for both still and video imagery. This review section presents the key details and conclusion for each study. The key details and conclusions were selected to lay the groundwork for the cross-experiment analysis in Section 3. The summary includes an overview of the images in the experiment, how the images were selected for measuring human performance, the key receiver operating characteristics (ROCs) comparing machine and humans, and
Cross experiment comparison
The next step is to take the experiments reviewed in the previous section and to analyze them as a group. The analysis is performed by using the cross-modal performance analysis (CMPA) framework, which we introduce.
Insights from structural comparisons
The analysis in Section 3 directly compared human and machine performance. There is more to learn from the interplay of machines and humans than what can be learned from relative performance comparisons. We will examine this interplay in the context of three topics. The first is the other-race effect, where algorithms have contributed to understanding the human face processing systems and human face processing has contributed to understanding machine performance. Second, it has been possible to
Future directions
The cross-modal performance analysis framework was designed to compare human and machine performance across a series of experiments. Although this framework is useful in its own right, we apply this technique to establish goals for advancing face recognition technology.
Over the last two decades, phenomenal progress has been made in automated face recognition from frontal images taken in mobile studio or mugshot environments. Results from the MBE 2010 report a false reject rates of 27 in 10,000
Acknowledgments
PJP was supported by the Federal Bureau of Investigation and AJO was supported by the Department of Defense. The identification of any commercial product or trade name does not imply endorsement or recommendation by NIST or U of Texas at Dallas.
References (45)
- et al.
Recognizing people from dynamic and stable faces and bodies: dissecting identity with a fusion approach
Vis. Res.
(2011) - et al.
Introduction to face recognition and evaluation of algorithm performance
Comput. Stat. Data Anal.
(2013) - et al.
Factors that influence algorithm performance in the Face Recognition Grand Challenge
Comput. Vis. Image Underst.
(2009) - et al.
Face recognition algorithms and the other-race effect: computational mechanisms for a developmental contact hypothesis
Cogn. Sci.
(2002) - et al.
Demographic effects on estimates of automatic face recognition performance
Image Vis. Comput.
(2012) - et al.
Are facial image analysis experts any better than the general public at identifying individuals from CCTV images?
Sci. Justice
(2009) - et al.
Variability in photos of the same face
Cognition
(2011) - et al.
Face recognition algorithms surpass humans matching faces across changes in illumination
IEEE Trans. PAMI
(2007) - et al.
Humans versus algorithms: comparisons from the FRVT
- et al.
FRVT 2006 and ICE 2006 large-scale results
IEEE Trans. PAMI
(2010)
Comparing face recognition algorithms to humans on challenging tasks
ACM Trans. Appl. Percept.
Untangling object recognition:which neuronal population codes can ex- 7Q1125 plain human object recognition performance? in: Neural Computation: Population 713 Coding of High-Level Representations
Representational similarity analysis—connecting the branches of systems neuroscience
Front. Syst. Neurosci.
Unaware person recognition from the body when face identification fails
Psychol. Sci.
Improving face recognition technology
IEEE Comput.
Overview of the face recognition grand challenge
An introduction to the good, the bad, and the ugly face recognition challenge problem
The CSU face identification evaluation system
Mach. Vis. Appl.
Computational and performance aspects of PCA-based face-recognition algorithms
Perception
Eigenfaces for recognition
J. Cogn. Neurosci.
Capitalize on dimensionality increasing techniques for improving face recognition performance
IEEE Trans. PAMI
Strategies and benefits of fusion of 2D and 3D face recognition
Cited by (110)
Hebbian dreaming for small datasets
2024, Neural NetworksArtificial Intelligence in academic library strategy in the United Kingdom and the Mainland of China
2023, Journal of Academic LibrarianshipBLUFADER: Blurred face detection & recognition for privacy-friendly continuous authentication
2023, Pervasive and Mobile ComputingStable individual differences in unfamiliar face identification: Evidence from simultaneous and sequential matching tasks
2023, CognitionCitation Excerpt :Trials in which a participant recognized an identity were not included in the analyses. Images were obtained from the following databases: The Face and Ocular Challenge Series (Phillips et al., 2011; Phillips & O′Toole, 2014), The Center for Vital Longevity Face Database (Minear & Park, 2004), and Brock University′s Let′s face it database. Images were also obtained from previous publications: Burton et al. (2010), Baker and Mondloch (2019), Matthews and Mondloch (2018), and Dowsett, Sandford, and Burton (2016).
- ☆
Editor's Choice Articles are invited and handled by a select rotating 12 member Editorial Board committee. This paper has been recommended for acceptance by Ioannis A. Kakadiaris.