Skip to main content
Top
Published in: Neural Computing and Applications 8/2021

05-07-2020 | Original Article

How frontal is a face? Quantitative estimation of face pose based on CNN and geometric projection

Authors: Fei Gao, Shuai Li, Shufang Lu

Published in: Neural Computing and Applications | Issue 8/2021

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Face pose estimation has been widely used into various applications of human–computer interaction; however, it is yet a challenging work due to illumination, background, face orientations, appearance visibility, etc. In this paper, a novel coarse-to-fine method of face pose quantitative estimation based on convolutional neural networks (CNN) and geometric projection is proposed. In coarse classification, CNN is applied to classify the input image into a specific category and obtain a relevant weight. After that, geometric projections of 3D face landmarks projected into three planes, xy, xz and yz, of 3D coordinate systems are used to perform the fine estimation of face pose, which can get the offset angles of the face in the three directions of roll, yaw, and pitch. Finally, the final score of face pose is obtained by combining the results of two stages. Experiments on standard datasets show that the proposed method can get better results than some competitive algorithms, which proves the effectiveness of the proposed method.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Doshi A, Trivedi MM (2012) Head and eye gaze dynamics during visual attention shifts in complex environments. J Vis 12(2):1–16CrossRef Doshi A, Trivedi MM (2012) Head and eye gaze dynamics during visual attention shifts in complex environments. J Vis 12(2):1–16CrossRef
2.
3.
go back to reference Murphy-Chutorian E, Trivedi MM (2009) Head pose estimation in computer vision: a survey. IEEE Trans Pattern Anal Mach Intell 31(4):607–626CrossRef Murphy-Chutorian E, Trivedi MM (2009) Head pose estimation in computer vision: a survey. IEEE Trans Pattern Anal Mach Intell 31(4):607–626CrossRef
4.
go back to reference Beymer D (1994) Face recognition under varying pose. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 756–761 Beymer D (1994) Face recognition under varying pose. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 756–761
5.
go back to reference Ng J, Gong S (2002) Composite support vector machines for detection of faces across views and pose estimation. Image Vis Comput 20(5–6):359–368CrossRef Ng J, Gong S (2002) Composite support vector machines for detection of faces across views and pose estimation. Image Vis Comput 20(5–6):359–368CrossRef
6.
go back to reference Ng J, Gong S (1999) Multi-view face detection and pose estimation using a composite support vector machine across the view sphere. In: Proceedings international workshop on recognition, analysis, and tracking of faces and gestures in real-time systems, pp 14–21 Ng J, Gong S (1999) Multi-view face detection and pose estimation using a composite support vector machine across the view sphere. In: Proceedings international workshop on recognition, analysis, and tracking of faces and gestures in real-time systems, pp 14–21
7.
go back to reference Wang J, Sung E (2007) EM enhancement of 3D head pose estimated by point at infinity. Image Vis Comput 25(12):1864–1874CrossRef Wang J, Sung E (2007) EM enhancement of 3D head pose estimated by point at infinity. Image Vis Comput 25(12):1864–1874CrossRef
8.
go back to reference Heo J, Savvides M (2011) Generic 3D face pose estimation using facial shapes. In: 2011 international joint conference on biometrics (IJCB), pp 1–8 Heo J, Savvides M (2011) Generic 3D face pose estimation using facial shapes. In: 2011 international joint conference on biometrics (IJCB), pp 1–8
9.
go back to reference Hegde C, Sankaranarayanan AC, Baraniuk RG (2011) Learning manifolds in the wild. J Mach Learn Res 1(2):1–34 Hegde C, Sankaranarayanan AC, Baraniuk RG (2011) Learning manifolds in the wild. J Mach Learn Res 1(2):1–34
10.
go back to reference Sundararajan K, Woodard DL (2015) Head pose estimation in the wild using approximate view manifolds. In: 2015 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 50–58 Sundararajan K, Woodard DL (2015) Head pose estimation in the wild using approximate view manifolds. In: 2015 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 50–58
11.
go back to reference Zhang Z, Hu Y, Liu M, Huang T (2007) Head pose estimation in seminar room using multi view face detectors. In: International evaluation workshop on classification of events, activities and relationships, pp 299–304 Zhang Z, Hu Y, Liu M, Huang T (2007) Head pose estimation in seminar room using multi view face detectors. In: International evaluation workshop on classification of events, activities and relationships, pp 299–304
12.
go back to reference Ma B, Zhang W, Shan S, Chen X, Gao W (2006) Robust head pose estimation using LGBP. In: 18th international conference on pattern recognition (ICPR’06), pp 512–515 Ma B, Zhang W, Shan S, Chen X, Gao W (2006) Robust head pose estimation using LGBP. In: 18th international conference on pattern recognition (ICPR’06), pp 512–515
13.
go back to reference Murphy-Chutorian E, Trivedi MM (2007) Head pose estimation for driver assistance systems: a robust algorithm and experimental evaluation. In: 2007 IEEE intelligent transportation systems conference, pp 709–714 Murphy-Chutorian E, Trivedi MM (2007) Head pose estimation for driver assistance systems: a robust algorithm and experimental evaluation. In: 2007 IEEE intelligent transportation systems conference, pp 709–714
14.
go back to reference Ma Y, Konishi Y, Kinoshita K, Lao S, Kawade M (2006) Sparse Bayesian regression for head pose estimation. In: 18th International conference on pattern recognition (ICPR’06), pp 507–510 Ma Y, Konishi Y, Kinoshita K, Lao S, Kawade M (2006) Sparse Bayesian regression for head pose estimation. In: 18th International conference on pattern recognition (ICPR’06), pp 507–510
15.
go back to reference Han B, Lee S, Yang H (2014) Head pose estimation using image abstraction and local directional quaternary patterns for multiclass classification. Pattern Recogn Lett 45:145–153CrossRef Han B, Lee S, Yang H (2014) Head pose estimation using image abstraction and local directional quaternary patterns for multiclass classification. Pattern Recogn Lett 45:145–153CrossRef
16.
go back to reference Drouard V, Ba S, Evangelidis G, Deleforge A, Horaud R (2015) Head pose estimation via probabilistic high-dimensional regression. In: 2015 IEEE international conference on image processing (ICIP), pp 4624–4628 Drouard V, Ba S, Evangelidis G, Deleforge A, Horaud R (2015) Head pose estimation via probabilistic high-dimensional regression. In: 2015 IEEE international conference on image processing (ICIP), pp 4624–4628
17.
go back to reference Drouard V, Horaud R, Deleforge A, Ba S, Evangelidis G (2017) Robust head-pose estimation based on partially-latent mixture of linear regressions. IEEE Trans Image Process 26(3):1428–1440MathSciNetCrossRef Drouard V, Horaud R, Deleforge A, Ba S, Evangelidis G (2017) Robust head-pose estimation based on partially-latent mixture of linear regressions. IEEE Trans Image Process 26(3):1428–1440MathSciNetCrossRef
18.
go back to reference Aghajanian J, Prince S (2009) Face pose estimation in uncontrolled environments. BMVC 1(2):1–11 Aghajanian J, Prince S (2009) Face pose estimation in uncontrolled environments. BMVC 1(2):1–11
19.
go back to reference Torki M, Elgammal A (2011) Regression from local features for viewpoint and pose estimation. In: 2011 international conference on computer vision, pp 2603–2610 Torki M, Elgammal A (2011) Regression from local features for viewpoint and pose estimation. In: 2011 international conference on computer vision, pp 2603–2610
20.
go back to reference Zhu X, Ramanan D (2012) Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE conference on computer vision and pattern recognition, pp 2879–2886 Zhu X, Ramanan D (2012) Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE conference on computer vision and pattern recognition, pp 2879–2886
21.
go back to reference Wang Y, Liang W, Shen J, Jia Y, Yu L (2019) A deep Coarse-to-Fine network for head pose estimation from synthetic data. Pattern Recogn 94(10):196–206CrossRef Wang Y, Liang W, Shen J, Jia Y, Yu L (2019) A deep Coarse-to-Fine network for head pose estimation from synthetic data. Pattern Recogn 94(10):196–206CrossRef
22.
go back to reference Ranjan R, Patel VM, Chellappa R (2019) HyperFace: a deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans Pattern Anal Mach Intell 41(1):121–135CrossRef Ranjan R, Patel VM, Chellappa R (2019) HyperFace: a deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans Pattern Anal Mach Intell 41(1):121–135CrossRef
23.
go back to reference Liu X, Liang W, Wang Y, Li S, Pei M (2016) 3D head pose estimation with convolutional neural network trained on synthetic images. In: 2016 IEEE international conference on image processing (ICIP), pp 1289–1293 Liu X, Liang W, Wang Y, Li S, Pei M (2016) 3D head pose estimation with convolutional neural network trained on synthetic images. In: 2016 IEEE international conference on image processing (ICIP), pp 1289–1293
24.
go back to reference Ahn B, Jaesik P, Kweon I (2014) Real-time head orientation from a monocular camera using deep neural network. In: Asian conference on computer vision, pp 82–96 Ahn B, Jaesik P, Kweon I (2014) Real-time head orientation from a monocular camera using deep neural network. In: Asian conference on computer vision, pp 82–96
25.
go back to reference Patacchiola M, Cangelosi A (2017) Head pose estimation in the wild using convolutional neural networks and adaptive gradient methods. Pattern Recogn 71:132–143CrossRef Patacchiola M, Cangelosi A (2017) Head pose estimation in the wild using convolutional neural networks and adaptive gradient methods. Pattern Recogn 71:132–143CrossRef
26.
go back to reference Zavan F, Bellon OR, Silva L, Medioni GG (2019) Benchmarking parts based face processing in-the-wild for gender recognition and head pose estimation. Pattern Recogn Lett 123:104–110CrossRef Zavan F, Bellon OR, Silva L, Medioni GG (2019) Benchmarking parts based face processing in-the-wild for gender recognition and head pose estimation. Pattern Recogn Lett 123:104–110CrossRef
27.
go back to reference Ruiz N, Chong E, Rehg JM (2018) Fine-grained head pose estimation without keypoints. In: 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 2155–2164 Ruiz N, Chong E, Rehg JM (2018) Fine-grained head pose estimation without keypoints. In: 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 2155–2164
28.
go back to reference Kumar A, Alavi A, Chellappa R (2017) KEPLER: keypoint and pose estimation of unconstrained faces by learning efficient H-CNN regressors. In: 2017 12th IEEE international conference on automatic face and gesture recognition (FG 2017), pp 258–265 Kumar A, Alavi A, Chellappa R (2017) KEPLER: keypoint and pose estimation of unconstrained faces by learning efficient H-CNN regressors. In: 2017 12th IEEE international conference on automatic face and gesture recognition (FG 2017), pp 258–265
29.
go back to reference Zhang H, Ji Y, Huang W, Liu L (2019) Sitcom-star-based clothing retrieval for video advertising: a deep learning framework. Neural Comput Appl 31:7361–7380CrossRef Zhang H, Ji Y, Huang W, Liu L (2019) Sitcom-star-based clothing retrieval for video advertising: a deep learning framework. Neural Comput Appl 31:7361–7380CrossRef
30.
go back to reference Ji Y, Zhang H, Wu Q (2018) Salient object detection via multi-scale attention CNN. Neurocomputing 322:130–140CrossRef Ji Y, Zhang H, Wu Q (2018) Salient object detection via multi-scale attention CNN. Neurocomputing 322:130–140CrossRef
31.
go back to reference Hsu H, Wu T, Wan S, Wong W, Lee C (2019) QuatNet: quaternion-based head pose estimation with multiregression loss. IEEE Trans Multimed 21(4):1035–1046CrossRef Hsu H, Wu T, Wan S, Wong W, Lee C (2019) QuatNet: quaternion-based head pose estimation with multiregression loss. IEEE Trans Multimed 21(4):1035–1046CrossRef
32.
go back to reference Huang B, Chen R, Xu W, Zhou Q (2020) Improving head pose estimation using two-stage ensembles with top-k regression. Image Vis Comput 93:103827–103835CrossRef Huang B, Chen R, Xu W, Zhou Q (2020) Improving head pose estimation using two-stage ensembles with top-k regression. Image Vis Comput 93:103827–103835CrossRef
33.
go back to reference Wu H, Zhang K, Tian G (2018) Simultaneous face detection and pose estimation using convolutional neural network cascade. IEEE Access 6:49563–49575CrossRef Wu H, Zhang K, Tian G (2018) Simultaneous face detection and pose estimation using convolutional neural network cascade. IEEE Access 6:49563–49575CrossRef
34.
go back to reference Fanelli G, Dantone M, Gall J, Fossati A, Gool L (2013) Random forests for real time 3D face analysis. Int J Comput Vision 101(3):437–458CrossRef Fanelli G, Dantone M, Gall J, Fossati A, Gool L (2013) Random forests for real time 3D face analysis. Int J Comput Vision 101(3):437–458CrossRef
35.
go back to reference Gourier N, Hall D, Crowley JL (2004) Estimating face orientation from robust detection of salient facial structures. In: FG Net workshop on visual observation of deictic gestures, vol 6, pp 1–9 Gourier N, Hall D, Crowley JL (2004) Estimating face orientation from robust detection of salient facial structures. In: FG Net workshop on visual observation of deictic gestures, vol 6, pp 1–9
36.
go back to reference Kostinger M, Wohlhart P, Roth P, Bischof H (2011) Annotated facial landmarks in the wild: a large-scale, real-world database for facial landmark localization. In: 2011 IEEE international conference on computer vision workshops (ICCV Workshops), pp 2144–2151 Kostinger M, Wohlhart P, Roth P, Bischof H (2011) Annotated facial landmarks in the wild: a large-scale, real-world database for facial landmark localization. In: 2011 IEEE international conference on computer vision workshops (ICCV Workshops), pp 2144–2151
37.
go back to reference Szegedy C, Ioffe S, Vanhoucke V, Alemi A (2017) Inception-v4, Inception-ResNet and the impact of residual connections on learning. In: Thirty-First AAAI conference on artificial intelligence, pp 4278–4284 Szegedy C, Ioffe S, Vanhoucke V, Alemi A (2017) Inception-v4, Inception-ResNet and the impact of residual connections on learning. In: Thirty-First AAAI conference on artificial intelligence, pp 4278–4284
38.
go back to reference Zhang K-P, Zhang Z-P, Li Z-F, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503CrossRef Zhang K-P, Zhang Z-P, Li Z-F, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503CrossRef
39.
go back to reference Deng J, Dong W, Socher R, Li L-J, Li K, Li F-F (2009) ImageNet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255 Deng J, Dong W, Socher R, Li L-J, Li K, Li F-F (2009) ImageNet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255
40.
go back to reference Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp 249–256 Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp 249–256
41.
go back to reference Bulat A, Tzimiropoulos G (2017) How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In: 2017 IEEE international conference on computer vision (ICCV), pp 1021–1030 Bulat A, Tzimiropoulos G (2017) How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In: 2017 IEEE international conference on computer vision (ICCV), pp 1021–1030
42.
go back to reference Jourabloo A, Liu X (2016) Large-pose face alignment via CNN-based dense 3D model fitting. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 4188–4196 Jourabloo A, Liu X (2016) Large-pose face alignment via CNN-based dense 3D model fitting. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 4188–4196
43.
go back to reference Zhu X, Lei Z, Liu X (2016) Face alignment across large poses: a 3D solution. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 146–155 Zhu X, Lei Z, Liu X (2016) Face alignment across large poses: a 3D solution. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 146–155
45.
go back to reference Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C, Berg A (2016) SSD: single shot multibox detector. In: European conference on computer vision (ECCV), pp 21–37 Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C, Berg A (2016) SSD: single shot multibox detector. In: European conference on computer vision (ECCV), pp 21–37
Metadata
Title
How frontal is a face? Quantitative estimation of face pose based on CNN and geometric projection
Authors
Fei Gao
Shuai Li
Shufang Lu
Publication date
05-07-2020
Publisher
Springer London
Published in
Neural Computing and Applications / Issue 8/2021
Print ISSN: 0941-0643
Electronic ISSN: 1433-3058
DOI
https://doi.org/10.1007/s00521-020-05167-0

Other articles of this Issue 8/2021

Neural Computing and Applications 8/2021 Go to the issue

Premium Partner