Kinship verification from facial images by scalable similarity fusion

doi:10.1016/j.neucom.2016.02.039

Neurocomputing

Volume 197, 12 July 2016, Pages 136-142

https://doi.org/10.1016/j.neucom.2016.02.039 Get rights and content

Abstract

Kinship verification using face images (KVFI) is a relatively new and challenging problem in computer vision and biometrics, while kin relationship in psychology has been well studied over the past decades. Recent advances in KVFI have shown that learning an effective similarity metric plays a critical role in the verification problem. However, most existing distance metric learning (DML) based KVFI methods use batch learning techniques to seek an optimal kin similarity metric, making them less scalable in practical verification tasks. To address this, we propose in this paper a scalable similarity learning (SSL) method for KVFI. Unlike existing DML-based solutions, SSL aims to learn a diagonal bilinear similarity model by online truncated gradient learning, which enjoys superiority in scalability and computational efficiency for practical KVFI with high-dimensional data. We further derive a multiview SSL algorithm by optimal fusion of the diagonal similarity models from multiple feature representations in a coherent online process, such that the interactions and correlations in multiview kin data can be leveraged to obtain refined and high-level information. Empirically, we evaluate our proposed method on two benchmark datasets against the state-of-the-art DML-based solutions, and the results demonstrate that our method can achieve competitive or better verification performance, and enjoys the superiority in scalability and computational efficiency.

Introduction

The goal of kinship verification using face images (KVFI) is to determine whether a given pair of face images has a kinship relation. Recent evidence in psychology has indicated that face appearance is a reliable cue for measurement of the genetic similarity between children and their parents [1], [2], [3], [4]. Motivated by this observation, KVFI has attracted more attention from computer vision and biometrics societies [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18]. In practice, there exist some important and potential applications for KVFI, ranging from searching missing children to social media analysis [5], [9], [10], [12]. Fig. 1 presents some face examples (with kin relations) from the KinFaceW dataset [12].

While some encouraging results have been achieved over the past five years [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], KVFI still remains open to further improvement. On one hand, face images are often captured in wild conditions, and varying illumination, poses and expressions in such scenarios make the verification problem quite challenging. On the other hand, kinship verification aims to investigate the kin relationship between two different visual entities (e.g., mother and son), and hence the inherent appearance gap in kinship verification is generally much larger than that in traditional face recognition [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [38].

The existing methods for KVFI are either feature-based [5], [6], [7], [10], [14], [16], [18] or model-based [8], [9], [12], [13], [15], [17], [19]. Feature-based methods extract discriminative feature from face image either by hand-crafted descriptors [5], [10], [18] or feature learning [6], [7], [14], [16], to describe the genetic traits on human face. Model-based methods, however, aim to learn an appropriate distance metric or classifier based on some statistical learning techniques. Among them, distance metric learning (DML) [20], [21], [22], [23] has been a promising choice for learning the kin similarity metric on face [12], [13], [15], [17], [19]. Most DML-based kinship verification methods suffer from two limitations; on one hand, they aim to learn a Mahalanobis distance metric parameterized by a full matrix for each feature representation, making the training and testing computationally inefficient for high-dimensional kin data. On the other hand, they seek to learn the kinship similarity metric in batch learning mode, making the process less scalable in practical verification tasks with ever-growing amount of training data collected and stored in the kinship databases. To address these issues, we propose in this paper a scalable similarity learning (SSL) method for KVFI.

Different from existing kinship verification methods, our proposed SSL seeks to learn a diagonal bilinear similarity metric on the human face based on an online sparse learning strategy, which enjoys superiority in efficiency and scalability for KVFI with high-dimensional kin data. In practice, we have access to multiple views (feature representations) of face data, each of which can be individually used for kinship verification. Exploiting information from multiple views, we hope to find a fused similarity metric on face appearance that is more robust and accurate than the ones learned by using the individual views. As such, we further develop a multiview SSL method by optimal fusion of the diagonal similarity models from multiple feature representations in a coherent process, such that the interactions and correlations in multiview kin data can be leveraged to obtain the refined and high-level information. The contributions of this work are summarized as follows:

1)
A scalable similarity learning method is proposed for training the kin similarity metric on the human face by introducing the truncated gradient to induce sparsity in online learning of the diagonal bilinear similarity function, making it computationally efficient and scalable for practical KVFI with high-dimensional kin data.
2)
A multiview SSL algorithm is developed for effective fusion of kin similarity by an optimal combination of the diagonal bilinear similarity models from multiple feature representations in a coherent online process.
3)
We empirically evaluate our SSL method against the state-of-the-art DML-based solutions to KVFI on two widely used kinship datasets, and the experimental results demonstrate that our method can achieve competitive or better verification performance, and enjoys superiority in computational efficiency and scalability.

The remainder of this paper is organized as follows. We briefly review the related work on kinship verification methods in Section 2. In Section 3, we first elaborate the scalable similarity learning method for KVFI, and then the multiview SSL algorithm is derived. Experiments and evaluations of our method are conducted in Section 4. Finally, we conclude the paper in Section 5.

Section snippets

Related work

We briefly review the work related to kinship verification in this section. The existing kinship verification methods can be roughly divided into two categories: feature-based [5], [6], [7], [10], [14], [16], [18] and model-based [8], [9], [12], [13], [15], [17], [19].

Diagonal bilinear similarity model

Most existing DML-based kinship verification methods [12], [13], [15], [17] aim at learning the Mahalanobis distance metric that measures the squared distance between a pair of face examples $x_{i}$ and $x_{j}$ : $d_{M}^{2} (x_{i}, x_{j}) = {(x_{i} - x_{j})}^{T} M (x_{i} - x_{j})$ where $x_{i}, x_{j} \in R^{d}$ , and $M \underset{̲}{≻} 0$ is a positive semi-definite matrix. One limitation of these methods is that they learn a full parameterized matrix from low-dimensional data. Such methods, however, typically require $O (d^{2})$ for storage and $O (d^{3})$ for optimization, making them

Experiments

In this section, experimental evaluations for our proposed methods are performed on the two benchmark datasets, i.e., the KinFaceW-I and KinFaceW-II [12], [13]. We detail the experimental settings and results in the following subsections.

Conclusion

We have introduced a new method for kinship verification by learning a diagonal bilinear similarity model in an online manner, which obtains comparable or better accuracy performance against state-of-the-art DML-based solutions, and enjoys the superiority in scalability and computational efficiency for practical KVFI with high-dimensional data. We also derive a multi-view SSL algorithm by optimal fusion of the diagonal similarity functions from multiple feature representations in a coherent

Acknowledgments

This work was supported by the research grant from the National Natural Science Foundation of China under Grant 61373090.

Xiuzhuang Zhou received the Ph.D. degree from the School of Computer Science, Beijing Institute of Technology, Beijing, China, in 2011. He is currently an associate professor at the College of Information Engineering, Capital Normal University, Beijing, China. His research interests include computer vision, pattern recognition, and machine learning. He has authored more than 30 scientific papers in peer-reviewed journals and conferences including some top venues such as the IEEE Transactions on

References (42)

G. Kaminski, S. Dridi, C. Graff, E. Gentaz, Human ability to detect kinship in strangers׳ faces: effects of the degree...
G. Kaminski et al.
Firstborns׳ disadvantage in kinship detection
Psychol. Sci.
(2010)
L. DeBruine et al.
Kin recognition signals in adult faces
Vis. Res.
(2009)
A. Alvergne et al.
Cross-cultural perceptions of facial resemblance between kin
J. Vis.
(2009)
R. Fang, K. Tang, N. Snavely, T. Chen, Towards computational models of kinship verification, in: Proceedings of the...
X. Zhou, J. Hu, J. Lu, Y. Shang, Y. Guan, Kinship verification from facial images under uncontrolled conditions, in:...
X. Zhou, J. Lu, J. Hu, Y. Shang, Gabor-based gradient orientation pyramid for kinship verification under uncontrolled...
S. Xia, M. Shao, Y. Fu, Kinship verification through transfer learning, in: Proceedings of the IJCAI, 2011, pp....
S. Xia et al.
Understanding kin relationships in a photo
IEEE Trans. Multimed.
(2012)
G.-D. Guo et al.
Kinship measurement on salient facial features
IEEE Trans. Instrum. Meas.
(2012)

H. Dibeklioglu, A.A. Salah, T. Gevers, Like father, like son: facial expression dynamics for kinship verification, in:...

J. Lu et al.

Neighborhood repulsed metric learning for kinship verification

IEEE Trans. Pattern Anal. Mach. Intell.

(2014)

J. Lu, J. Hu, X. Zhou, Y. Shang, Y.-P. Tan, G. Wang, Neighborhood repulsed metric learning for kinship verification,...

H. Yan et al.

Prototype-based discriminative feature learning for kinship verification

IEEE Trans. Cybern.

(2015)

H. Yan et al.

Discriminative multi-metric learning for kinship verification

IEEE Trans. Inf. Forensics Secur.

(2014)

A. Dehghan, E.G. Ortiz, R. Villegas, M. Shah, Who do i look like? Determining parent-offspring resemblance via gated...

J. Hu, J. Lu, J. Yuan, Y.-P. Tan, Large margin multi-metric learning for face and kinship verification in the wild, in:...

N. Kohli, R. Singh, M. Vatsa, Self-similarity representation of weber faces for kinship classification, in: Proceedings...

X. Zhou et al.

Ensemble similarity learning for kinship verification from facial images in the wild

Inf. Fusion

(2015)

J. Davis, B. Kulis, P. Jain, S. Sra, I. Dhillon, Informationtheoretic metric learning, in: Proceedings of the ICML,...

J. Goldberger, S. Roweis, G. Hinton, R. Salakhutdinov, Neighborhood component analysis, in: Proceedings of the NIPS,...

Cited by (52)

Kinship verification using multi-level dictionary pair learning for multiple resolution images
2023, Pattern Recognition
Kinship verification using facial images is gaining substantial attention by computer vision researchers. The real challenge in kinship verification is to effectively represent the discriminative features to ease the differences between kinship image pairs. Further, existing kinship methods only focus on a single resolution, and ignore the variability of resolutions in practical scenarios. To address these issues, we propose a multi-level dictionary pair learning (MLDPL) method to learn dictionary pairs by incorporating multiple resolution images for kinship verification. We learn dictionary pairs jointly by transforming discriminative features of image pairs into different coding coefficients in the same space, thereby reducing the differences between them. Further, multiple resolution images are incorporated into dictionary pair learning to effectively deal with resolution variations in kinship verification. Extensive experiments are performed on different kinship datasets to validate the efficacy of proposed MLDPL method. Experimental results show that MLDPL achieves competitive performance on all kinship datasets.
A survey on kinship verification
2023, Neurocomputing
In this survey, kinship verification is defined as the automatic process of verifying whether two or more persons are blood relatives (kin) by analyzing images of their faces. Kinship verification is an important research field in computer vision with many applications such as finding missing persons, family album organization, and online image search. Although substantial progress has been made in kinship verification in the past decade, there are still challenges such as intrinsic (face i.e., differences in facial appearance) and extrinsic (acquisition i.e., varying imaging conditions) problems. And there is still a demand for more diverse datasets.
Therefore, this paper provides a survey on kinship verification methods and datasets. The survey starts with the definition of kinship verification and its corresponding intrinsic and extrinsic challenges. Then, an overview of kinship verification methods and datasets is given. Finally, a new multi-modal dataset (Nemo-Kinship Dataset) is proposed as a benchmark dataset addressing large inter-subject age variations consisting of 4216 videos of 248 persons from 85 families. The newly collected dataset is used to systematically test and analyze state-of-the-art methods.
Binary cross coupled discriminant analysis for visual kinship verification
2022, Signal Processing: Image Communication
Citation Excerpt :
In [24] Zhou et al. proposed the ensemble similarity learning (ESL) method, which was learned by merging multiple base similarity functions, which results in increase in the similarity among kin pairs. Zhou et al. [25] proposed scalable similarity learning (SSL) which was learned by bi linear similarity model. In [35] Laiadi et al. proposed side-information based weighted exponential discriminant analysis (SIWEDA) method.
Kinship verification refers to comparing similarities between two different individuals through their facial images. In this context, feature descriptors play a crucial role, and few feature descriptors are present in literature to extract kin features from facial images. In this paper, we propose a binary cross-coupled discriminant analysis $(B C^{2} D A)$ based feature descriptor which is able to extract effective kin features from input facial image pairs. This method reduces the discrimination between kin pairs at the feature extraction stage itself. $B C^{2} D A$ converts original kin image pairs to encoded image pairs to reduce the discrimination between them. To make better use of tri-subject kin relations, we further propose multi cross-coupled discriminant analysis $(M C^{2} D A)$ . This method reduces the discrimination between child and both parents’ images at the feature extraction stage. Extensive experiments were conducted on six kinship datasets such as KinfaceW-I/II, Cornell, FIW, TSKinface UBKinface to show the efficacy of the proposed algorithm.
Deep discriminant generation-shared feature learning for image-based kinship verification
2022, Signal Processing: Image Communication
Citation Excerpt :
Third, genetic diversity makes it very difficult to describe patterns of kinship recognition. To tackle these difficulties, many methods have been presented [9,12–17]. Generally, these works can be roughly classified into two categories: traditional methods and deep learning methods.
Kinship verification based on facial images in the wild is an interesting and challenging research topic in the fields of computer vision. It has many potential applications, e.g., missing children search, image annotation, etc. In practice, the biggest obstacle in kinship verification is that there usually exists large divergence between the images of parent and children. How to effectively tackle such challenge is important for improving kinship verification performance. To this end, we proposed a novel Deep Discriminant Generation-Shared Feature Learning for Image-Based Kinship Verification (D $^{2}$ GFL) method for kinship verification, which consists of a two-stream generation-specific feature learning module and a generation-shared discrepancy reducing module. The generation-specific feature learning module can learn parent-specific and child-specific features. The generation-shared module aims to reduce the divergence between facial images of parent and child. In order to further relieve the cross-generation difference and improve the discriminability of learned features, we also design a cross-generation difference loss term and an intra-generation discriminant loss term, which simultaneously make use of the local and holistic features, as well as the family identity information. Experimental results on several widely used kinship datasets are presented to validate the effectiveness of our proposed approach by comparing with the state-of-the-art kinship verification methods. Specially, our approach can improve the average verification accuracy at least by 3.5%, 1.5% and 0.8% on KinFaceW-I, Cornell, and TSKinFace datasets, respectively.
New metric learning model using statistical inference for kinship verification
2020, Applied Soft Computing Journal
Kinship verification aims to predict whether there is a kin relationship between a pair of facial images. Previous representative research has proved that learning an appropriate similarity metric plays an important role in this task. Most of the related metric learning methods focus on learning a single Mahalanobis distance metric under the assumption that kin data has a unimodal distribution. Instead, in this paper, we propose a novel nonlinear multi-metric learning (NMML) method to learn a similarity measure where the pairwise difference is drawn from a mixture distribution. Particularly, the similarity measure is derived based on a statistical inference perspective, which can indicate the characteristics of a likelihood ratio that captures the subtle resemblance between identities. To make better use of multiple estimators, we further propose an ensemble NMML (ENMML) method to perform multiple estimators fusion to improve the performance of kinship verification. Extensive experiments on three publicly available kinship datasets demonstrate the feasibility and effectiveness of our proposed methods.
A literature survey on kinship verification through facial images
2020, Neurocomputing
Citation Excerpt :
On the other hand, most metric learning-based kinship verification methods seek to learn the metric in batch learning mode, which is less efficient and scalable to practical applications even for medium-size training set. To tackle this issue, Zhou et al. [99] and Kou et al. [36] propose to learn a similarity model by online learning, which enjoys superiority in scalability and computational efficiency. Kin relationship is one of the most important and most basic in the human social relationship, thus accurately and efficiently verifying whether given entities have kin relationship is naturally becoming research focus with an extraordinary realistic and applicative value.
Kinship verification is an emerging task in computer vision which aims at finding out whether there is a kin relation between given identities through their facial images. Applications of kinship verification include image annotation, children adoption, and social media analysis, etc. However, kinship verification through facial images is challenging because facial images usually contain high intra-variances, which vary from genetic, age and gender difference. Over the past decade, more and more effective methods have emerged. This paper aims at categorizing and evaluating these methods systematically. We attach great importance to the difficulties in practical applications of kinship verification, and review the prominent algorithms from the perspective of learning more efficient models with more diverse kin relations. Then we further show how to develop an efficient and robust kinship verification system. Finally we present several potential directions for future research.

View all citing articles on Scopus

Haibin Yan received the B.Eng. and M.Eng. degrees from the Xi׳an University of Technology, Xi׳an, China, in 2004 and 2007, and the Ph.D. degree from the National University of Singapore, Singapore, in 2013, all in mechanical engineering. Currently, she is a research fellow at the Department of Mechanical Engineering, National University of Singapore, Singapore. Her research interests include social robotics, human-robotic interaction, and computer vision. She has authored more than 10 scientific papers in peer- reviewed journals and conferences including some top venues such as the IEEE Transactions on Cybernetics, IEEE Transactions on Information Forensics and Security, and IEEE ICRA.

Yuanyuan Shang received the Ph.D. degree from the National Astronomical Observatories, Chinese Academy of Sciences in 2005. She is currently a professor and vice dean (research) of the College of Information Engineering, Capital Normal University, Beijing, PR China. Her research interests include digital imaging sensor, image processing, and computer vision. She has authored more than 50 scientific papers in peer-reviewed journals and conferences, including some top venues such as the IEEE Transactions on Pattern Analysis and Machine Intelligence , IEEE Transactions on Information Forensics and Security, and ACM MM. She is a member of the IEEE.

View full text

Kinship verification from facial images by scalable similarity fusion

Abstract

Introduction

Section snippets

Related work

Diagonal bilinear similarity model

Experiments

Conclusion

Acknowledgments

Firstborns׳ disadvantage in kinship detection

Psychol. Sci.

Kin recognition signals in adult faces

Vis. Res.

Cross-cultural perceptions of facial resemblance between kin

J. Vis.

Understanding kin relationships in a photo

IEEE Trans. Multimed.

Kinship measurement on salient facial features

IEEE Trans. Instrum. Meas.