Top

International Journal of Computer Vision

Published in:

01-10-2019

Tensor Decomposition and Non-linear Manifold Modeling for 3D Head Pose Estimation

Authors: Dmytro Derkach, Adria Ruiz, Federico M. Sukno

Published in: International Journal of Computer Vision | Issue 10/2019

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Head pose estimation is a challenging computer vision problem with important applications in different scenarios such as human–computer interaction or face recognition. In this paper, we present a 3D head pose estimation algorithm based on non-linear manifold learning. A key feature of the proposed approach is that it allows modeling the underlying 3D manifold that results from the combination of rotation angles. To do so, we use tensor decomposition to generate separate subspaces for each variation factor and show that each of them has a clear structure that can be modeled with cosine functions from a unique shared parameter per angle. Such representation provides a deep understanding of data behavior. We show that the proposed framework can be applied to a wide variety of input features and can be used for different purposes. Firstly, we test our system on a publicly available database, which consists of 2D images and we show that the cosine functions can be used to synthesize rotated versions from an object from which we see only a 2D image at a specific angle. Further, we perform 3D head pose estimation experiments using other two types of features: automatic landmarks and histogram-based 3D descriptors. We evaluate our approach on two publicly available databases, and demonstrate that angle estimations can be performed by optimizing the combination of these cosine functions to achieve state-of-the-art performance.

previous article Learning Human Pose Models from Synthesized Data for Robust RGB-D Action Recognition

next article LCEval: Learned Composite Metric for Caption Evaluation

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

Of course this applies to rotations about a single axis; the general 3D-rotation case, depending on 3 parameters, could be anyway decomposed in the product of 3 such rotation matrices, analogous to the product of \(\mathbf {f}^{(y)}(\omega ^{(y)}) \times \mathbf {f}^{(p)}( \omega ^{(p)} ) \times \mathbf {f}^{(r)}( \omega ^{(r)} )\) in Eq. 11.

This process is coined translation in the seminal work by Tenenbaum and Freeman (2000).

These samples cover approximately a viewpoint range from \(5^{\circ }\) to \(35^{\circ }\).

Such invariance, however, is only partially achieved in 3DSC since the orientation of the surface normal still leaves one degree of freedom undefined (the sphere’s azimuth) (Sukno et al. 2013).

The results obtained from the minimization are forced to comply with the rotation manifold after each iteration using nearest-neighbour search. Results without such correction would be worse than those reported and not meaningful for comparison, since this is a widespread practice. Notice that no constraints are applied to the identity subspace in any of the experiments in this paper.

We consider that a landmark is on the surface when its distance to it is relatively small as compared to the mesh resolution.

Depending on the way in which the data is captured and the extent of the considered rotations, self-occlusions may jeopardize this strategy.

All experiments in this paper have been performed following this strategy.

Ahn, B., Park, J., & Kweon, I. S. (2014). Real-time head orientation from a monocular camera using deep neural network. In Asian conference on computer vision (pp. 82–96). Springer.

Bakry, A., & Elgammal, A. (2014). Untangling object-view manifold for multiview recognition and pose estimation. In European conference on computer vision (pp. 434–449). Springer.

Balasubramanian, V. N., Ye, J., & Panchanathan, S. (2007). Biased manifold embedding: A framework for person-independent head pose estimation. In Computer vision and pattern recognition (CVPR) (pp. 1–7). IEEE.

Baltrušaitis, T., Robinson, P., & Morency, L. P. (2012). 3D constrained local model for rigid and non-rigid facial tracking. In Computer vision and pattern recognition (CVPR) (pp. 2610–2617). IEEE.

Barros, J. M. D., Mirbach, B., Garcia, F., Varanasi, K., & Stricker, D. (2018). Fusion of keypoint tracking and facial landmark detection for real-time head pose estimation. In Winter conference on applications of computer vision (WACV) (pp. 2028–2037). IEEE.

BenAbdelkader, C. (2010). Robust head pose estimation using supervised manifold learning. In European conference on computer vision (pp. 518–531). Springer.

Bergqvist, G., & Larsson, E. G. (2010). The higher-order singular value decomposition: Theory and an application [lecture notes]. IEEE Signal Processing Magazine, 27(3), 151–154.CrossRef

Borghi, G., Fabbri, M., Vezzani, R., Calderara, S., & Cucchiara, R. (2019). Face-from-depth for head pose estimation on depth images. IEEE Transactions on Pattern Analysis and Machine Intelligence (in press).

Borghi, G., Venturelli, M., Vezzani, R., & Cucchiara, R. (2017). Poseidon: Face-from-depth for driver pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4661–4670).

Breitenstein, M. D., Kuettel, D., Weise, T., Van Gool L, & Pfister, H. (2008). Real-time face pose estimation from single range images. In Computer vision and pattern recognition (pp. 1–8). IEEE.

Byrd, R. H., Nocedal, J., & Schnabel, R. B. (1994). Representations of quasi-newton matrices and their use in limited memory methods. Mathematical Programming, 63(1–3), 129–156.MathSciNetCrossRefMATH

Chen, J., Wu, J., Richter, K., Konrad, J., & Ishwar, P. (2016). Estimating head pose orientation using extremely low resolution images. In Southwest symposium on image analysis and interpretation (SSIAI) (pp. 65–68). IEEE

Comon, P. (2014). Tensors: A brief introduction. Signal Processing Magazine, 31(3), 44–53.CrossRef

De Lathauwer, L., De Moor, B., & Vandewalle, J. (2000). A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications, 21(4), 1253–1278.MathSciNetCrossRefMATH

Derkach, D., Ruiz, A., & Sukno, F. M. (2017). Head pose estimation based on 3-D facial landmarks localization and regression. In 12th IEEE international conference on automatic face and gesture recognition (FG 2017) (pp. 820–827). IEEE.

Derkach, D., Ruiz, A., & Sukno, F. M. (2018). 3D head pose estimation using tensor decomposition and non-linear manifold modeling. In: International conference on 3D Vision (3DV) (pp. 505–513). IEEE.

Fanelli, G., Dantone, M., Gall, J., Fossati, A., & Van Gool, L. (2013). Random forests for real time 3D face analysis. International Journal of Computer Vision, 101(3), 437–458.CrossRef

Fanelli, G., Weise, T., Gall, J., & Van Gool, L. (2011). Real time head pose estimation from consumer depth cameras. In Joint pattern recognition symposium (pp. 101–110). Springer.

Frome, A., Huber, D., Kolluri, R., Bulow, T., & Malik, J. (2004). Recognizing objects in range data using regional point descriptors. In European conference on computer vision (pp. 224–237). Springer.

Fu, Y., & Huang, T. S. (2006). Graph embedded analysis for head pose estimation. In International conference on automatic face and gesture recognition (pp. 6–8). IEEE.

Ghiass, R. S., Arandjelović, O., & Laurendeau, D. (2015). Highly accurate and fully automatic head pose estimation from a low quality consumer-level rgb-d sensor. In Proceedings of the 2nd workshop on computational models of social interactions: Human–Computer–Media communication (pp. 25–34). ACM.

Gu, J., Yang, X., De Mello, S., & Kautz, J. (2017). Dynamic facial analysis: From bayesian filtering to recurrent neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1548–1557).

Johnson, A., & Hebert, M. (1999). Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(5), 433–449.CrossRef

Kolda, T. G., & Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review, 51(3), 455–500.MathSciNetCrossRefMATH

Lathuiliére, S., Juge, R., Mesejo, P., Muñoz-Salinas, R., & Horaud, R. (2017). Deep mixture of linear inverse regressions applied to head-pose estimation. In Conference on computer vision and pattern recognition (vol. 3, pp. 4817–4825).

Lathuiliére, S., Mesejo, P., Alameda-Pineda, X., & Horaud, R. (2019). A comprehensive analysis of deep regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1–1 (in press).

Lee, D., Yang, M. H., & Oh, S. (2015). Fast and accurate head pose estimation via random projection forests. In International conference on computer vision (pp. 1958–1966). IEEE.

Lee, D., Yang, M. H., & Oh, S. (2017). Head and body orientation estimation using convolutional random projection forests. In IEEE transactions on pattern analysis and machine intelligence (pp. 1–14)

Li, D., & Pedrycz, W. (2014). A central profile-based 3D face pose estimation. Pattern Recognition, 47(2), 525–534.CrossRef

Li, S., Ngan, K. N., Paramesran, R., & Sheng, L. (2016). Real-time head pose tracking with online face template reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(9), 1922–1928.CrossRef

Liu, X., Liang, W., Wang, Y., Li, S., & Pei, M. (2016). 3D head pose estimation with convolutional neural network trained on synthetic images. In International conference on image processing (ICIP) (pp. 1289–1293). IEEE.

Liu, X., Lu, H., & Li, W. (2010). Multi-manifold modeling for head pose estimation. In International conference on image processing (ICIP) (pp. 3277–3280). IEEE.

Lüsi, I., Escalera, S., & Anbarjafari, G. (2016a). Human head pose estimation on SASE database using random hough regression forests. Video Analytics (pp. 137–150). Springer: Face and Facial Expression Recognition and Audience Measurement.

Lüsi, I., Escarela, S., & Anbarjafari, G. (2016b). SASE: RGB-depth database for human head pose estimation. In European conference on computer vision (pp. 325–336). Springer.

Lüsi, I., Jacques Junior, J. C. S., Gorbova, J., Baró X, Escalera, S., Demirel, H., Allik, J., Ozcinar, C., & Anbarjafari, G. (2017). Joint challenge on dominant and complementary emotion recognition using micro emotion features and head-pose estimation: Databases. In International conference on automatic face and gesture recognition (pp. 809–813). IEEE.

Martin, M., Van De Camp, F., & Stiefelhagen, R. (2014). Real time head model creation and head pose estimation on consumer depth cameras. In International conference on 3D vision (3DV) (vol. 1, pp. 641–648). IEEE.

Meyer, G. P., Gupta, S., Frosio, I., Reddy, D., & Kautz, J. (2015). Robust model-based 3D head pose estimation. In Proceedings of the IEEE international conference on computer vision (pp. 3649–3657). IEEE.

Murphy-Chutorian, E., & Trivedi, M. M. (2009). Head pose estimation in computer vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(4), 607–626.CrossRef

Nene, S. A., Nayar, S. K., Murase, H., et al. (1996). Columbia object image library (coil-20).

Padeleris, P., Zabulis, X., & Argyros, A. A. (2012). Head pose estimation on depth data based on particle swarm optimization. In Computer society conference on computer vision and pattern recognition workshops (CVPRW) (pp. 42–49). IEEE.

Papazov, C., Marks, T. K., & Jones, M. (2015). Real-time 3D head pose and facial landmark estimation from depth images using triangular surface patch features. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4722–4730).

Patacchiola, M., & Cangelosi, A. (2017). Head pose estimation in the wild using convolutional neural networks and adaptive gradient methods. Pattern Recognition, 71, 132–143.CrossRef

Peng, X., Huang, J., Hu, Q., Zhang, S., & Metaxas, D. N. (2014). Head pose estimation by instance parameterization. In International conference on pattern recognition (ICPR) (pp. 1800–1805). IEEE.

Raytchev, B., Yoda, I., & Sakaue, K. (2004). Head pose estimation by nonlinear manifold learning. In International conference on pattern recognition (ICPR) (vol. 4, pp. 462–466). IEEE.

Ruiz, N., Chong, E., & Rehg, J. M. (2018). Fine-grained head pose estimation without keypoints. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 2074–2083).

Rusu, R. B., Blodow, N., & Beetz, M. (2009). Fast point feature histograms (fpfh) for 3d registration. In International conference on robotics and automation, Citeseer (pp. 3212–3217).

Schmidt, M. (2012). Minfunc: Unconstrained differentiable multivariate optimization in matlab. Software available at http://www.cs.ubc.ca/schmidtm/Software/minFunc.htm.

Seemann, E., Nickel, K., & Stiefelhagen, R. (2004). Head pose estimation using stereo vision for human–robot interaction. In International conference on automatic face and gesture recognition (pp. 626–631). IEEE.

Sukno, F., Waddington, J., & Whelan, P. (2012). Comparing 3D descriptors for local search of craniofacial landmarks. In International symposium on visual computing (pp. 92–103). Springer.

Sukno, F., Waddington, J., & Whelan, P. (2013). Rotationally invariant 3D shape contexts using asymmetry patterns. International conference on computer graphics theory and applications (pp. 7–17).

Sukno, F. M., Waddington, J. L., & Whelan, P. F. (2015). 3-D facial landmark localization with asymmetry patterns and shape regression from incomplete local features. IEEE Transactions on Cybernetics, 45(9), 1717–1730.CrossRef

Sun, Y., & Yin, L. (2008). Automatic pose estimation of 3D facial models. In International conference on pattern recognition (pp. 1–4.).

Sundararajan, K., & Woodard, D. L. (2015). Head pose estimation in the wild using approximate view manifolds. In International conference on computer vision and pattern recognition workshops (pp. 50–58). IEEE.

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–9).

Takallou, H. M., & Kasaei, S. (2014). Head pose estimation and face recognition using a non-linear tensor-based model. IET Computer Vision, 8(1), 54–65.CrossRef

Tan, D. J., Tombari, F., & Navab, N. (2018). Real-time accurate 3d head tracking and pose estimation with consumer rgb-d cameras. International Journal of Computer Vision, 126(2–4), 158–183.MathSciNetCrossRef

Tenenbaum, J. B., & Freeman, W. T. (1997). Separating style and content. In Advances in neural information processing systems (pp. 662–668).

Tenenbaum, J. B., & Freeman, W. T. (2000). Separating style and content with bilinear models. Neural Computation, 12(6), 1247–1283.CrossRef

Tombari, F., Salti, S., & Di Stefano, L. (2010). Unique signatures of histograms for local surface description. In European conference on computer vision (pp. 356–369). Springer.

Tulyakov, S., Vieriu, R. L., Semeniuta, S., & Sebe, N. (2014). Robust real-time extreme head pose estimation. In International conference on pattern recognition (ICPR) (pp. 2263–2268). IEEE.

Vasilescu, M. A. O., & Terzopoulos, D. (2002). Multilinear analysis of image ensembles: Tensorfaces. In European conference on computer vision (pp. 447–460). Springer.

Wang, B., Liang, W., Wang, Y., & Liang, Y. (2013). Head pose estimation with combined 2D SIFT and 3D HOG features. In International conference on image and graphics (ICIG) (pp. 650–655). IEEE.

Wang, C., Guo, Y., & Song, X. (2017a). Head pose estimation via manifold learning. InTech: In Manifolds-current research areas.CrossRef

Wang, C., & Song, X. (2014). Robust head pose estimation via supervised manifold learning. Neural Networks, 53, 15–25.CrossRefMATH

Wang, K., Wu, Y., & Ji, Q. (2018). Head pose estimation on low-quality images. In International conference on automatic face and gesture recognition (FG 2018) (pp. 540–547). IEEE.

Wang, M., Panagakis, Y., Snape, P., Zafeiriou, S., et al. (2017b). Learning the multilinear structure of visual data. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4592–4600).

Wang, Y., Liang, W., Shen, J., Jia, Y., & Yu, L. F. (2019). A deep coarse-to-fine network for head pose estimation from synthetic data. Pattern Recognition, 94, 196–206.CrossRef

Xu, Y., Hao, R., Yin, W., & Su, Z. (2015). Parallel matrix factorization for low-rank tensor completion. Inverse Problems and Imaging, 9(2), 601–624.MathSciNetCrossRefMATH

Yu, Y., Mora, K. A. F., & Odobez, J. M. (2017). Robust and accurate 3D head pose estimation through 3dmm and online head model reconstruction. In International conference on automatic face and gesture recognition (FG 2017) (pp. 711–718). IEEE.

Zhang, H., El-Gaaly, T., Elgammal, A., & Jiang, Z. (2015). Factorization of view-object manifolds for joint object recognition and pose estimation. Computer Vision and Image Understanding, 139, 89–103.CrossRef

Zhao, Q., Zhang, L., & Cichocki, A. (2015). Bayesian cp factorization of incomplete tensors with automatic rank determination. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9), 1751–1763.CrossRef

Zhu, Y., Xue, Z., & Li, C. (2014). Automatic head pose estimation with synchronized sub manifold embedding and random regression forests. International Journal of Signal Processing, Image Processing and Pattern Recognition, 7(3), 123–134.CrossRef

Title: Tensor Decomposition and Non-linear Manifold Modeling for 3D Head Pose Estimation
Authors: Dmytro Derkach
Adria Ruiz
Federico M. Sukno
Publication date: 01-10-2019
Publisher: Springer US
Published in: International Journal of Computer Vision / Issue 10/2019
Print ISSN: 0920-5691
Electronic ISSN: 1573-1405
DOI: https://doi.org/10.1007/s11263-019-01208-x

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Other articles of this Issue 10/2019

Learning Transparent Object Matting

Video Question Answering with Spatio-Temporal Reasoning

LCEval: Learned Composite Metric for Caption Evaluation

Estimation of 3D Category-Specific Object Structure: Symmetry, Manhattan and/or Multiple Images

Motion-Compensated Spatio-Temporal Filtering for Multi-Image and Multimodal Super-Resolution

What Does 2D Geometric Information Really Tell Us About 3D Face Shape?

Premium Partner