Skip to main content
Top

2016 | OriginalPaper | Chapter

Robust Face Alignment Using a Mixture of Invariant Experts

Authors : Oncel Tuzel, Tim K. Marks, Salil Tambe

Published in: Computer Vision – ECCV 2016

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Face alignment, which is the task of finding the locations of a set of facial landmark points in an image of a face, is useful in widespread application areas. Face alignment is particularly challenging when there are large variations in pose (in-plane and out-of-plane rotations) and facial expression. To address this issue, we propose a cascade in which each stage consists of a mixture of regression experts. Each expert learns a customized regression model that is specialized to a different subset of the joint space of pose and expressions. The system is invariant to a predefined class of transformations (e.g., affine), because the input is transformed to match each expert’s prototype shape before the regression is applied. We also present a method to include deformation constraints within the discriminative alignment framework, which makes our algorithm more robust. Our algorithm significantly outperforms previous methods on publicly available face alignment datasets.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Appendix
Available only for authorised users
Footnotes
1
The CFAN [34] algorithm included the 330 test faces from Helen in its training data. Thus when testing CFAN, we had to omit these 330 faces from the 300 W test set.
 
Literature
1.
go back to reference Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Comput. Vis. Image Understand. 61(1), 38–59 (1995)CrossRef Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Comput. Vis. Image Understand. 61(1), 38–59 (1995)CrossRef
2.
go back to reference Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)CrossRef Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)CrossRef
3.
go back to reference Sauer, P., Cootes, T.F., Taylor, C.J.: Accurate regression procedures for active appearance models. In: BMVC, pp. 1–11 (2011) Sauer, P., Cootes, T.F., Taylor, C.J.: Accurate regression procedures for active appearance models. In: BMVC, pp. 1–11 (2011)
4.
go back to reference Sung, J., Kim, D.: Adaptive active appearance model with incremental learning. Pattern Recogn. Lett. 30(4), 359–367 (2009)CrossRef Sung, J., Kim, D.: Adaptive active appearance model with incremental learning. Pattern Recogn. Lett. 30(4), 359–367 (2009)CrossRef
5.
go back to reference Tzimiropoulos, G., Pantic, M.: Optimization problems for fast AAM fitting in-the-wild. In: Proceedings of IEEE International Conference on Computer Vision (ICCV) (2013) Tzimiropoulos, G., Pantic, M.: Optimization problems for fast AAM fitting in-the-wild. In: Proceedings of IEEE International Conference on Computer Vision (ICCV) (2013)
6.
go back to reference Romdhani, S., Gong, S., Psarrou, A., et al.: A multi-view nonlinear active shape model using kernel PCA. BMVC 10, 483–492 (1999) Romdhani, S., Gong, S., Psarrou, A., et al.: A multi-view nonlinear active shape model using kernel PCA. BMVC 10, 483–492 (1999)
7.
go back to reference Cootes, T.F., Wheeler, G.V., Walker, K.N., Taylor, C.J.: View-based active appearance models. Image Vis. Comput. 20(9), 657–664 (2002)CrossRef Cootes, T.F., Wheeler, G.V., Walker, K.N., Taylor, C.J.: View-based active appearance models. Image Vis. Comput. 20(9), 657–664 (2002)CrossRef
8.
go back to reference Asthana, A., Marks, T., Jones, M., Tieu, K., M.V., R.: Fully automatic pose-invariant face recognition via 3d pose normalization. In: IEEE International Conference on Computer Vision (ICCV), pp. 937–944, November 2011 Asthana, A., Marks, T., Jones, M., Tieu, K., M.V., R.: Fully automatic pose-invariant face recognition via 3d pose normalization. In: IEEE International Conference on Computer Vision (ICCV), pp. 937–944, November 2011
9.
go back to reference Cootes, T.F., Wheeler, G.V., Walker, K.N., Taylor, C.J.: Coupled-view active appearance models. In: BMVC, pp. 52–61 (2000) Cootes, T.F., Wheeler, G.V., Walker, K.N., Taylor, C.J.: Coupled-view active appearance models. In: BMVC, pp. 52–61 (2000)
10.
go back to reference Hu, C., Xiao, J., Matthews, I., Baker, S., Cohn, J.F., Kanade, T.: Fitting a single active appearance model simultaneously to multiple images. In: BMVC, pp. 1–10 (2004) Hu, C., Xiao, J., Matthews, I., Baker, S., Cohn, J.F., Kanade, T.: Fitting a single active appearance model simultaneously to multiple images. In: BMVC, pp. 1–10 (2004)
11.
go back to reference Cristinacce, D., Cootes, T.F.: Feature detection and tracking with constrained local models. In: BMVC (2006) Cristinacce, D., Cootes, T.F.: Feature detection and tracking with constrained local models. In: BMVC (2006)
12.
go back to reference Cristinacce, D., Cootes, T.F.: Boosted regression active shape models. In: BMVC, pp. 1–10 (2007) Cristinacce, D., Cootes, T.F.: Boosted regression active shape models. In: BMVC, pp. 1–10 (2007)
13.
go back to reference Zhou, F., Brandt, J., Lin, Z.: Exemplar-based graph matching for robust facial landmark localization. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1025–1032. IEEE (2013) Zhou, F., Brandt, J., Lin, Z.: Exemplar-based graph matching for robust facial landmark localization. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1025–1032. IEEE (2013)
14.
go back to reference Smith, B.M., Zhang, L.: Joint face alignment with non-parametric shape models. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 43–56. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33712-3_4 Smith, B.M., Zhang, L.: Joint face alignment with non-parametric shape models. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 43–56. Springer, Heidelberg (2012). doi:10.​1007/​978-3-642-33712-3_​4
15.
go back to reference Tzimiropoulos, G., Pantic, M.: Gauss-newton deformable part models for face alignment in-the-wild. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1851–1858. IEEE (2014) Tzimiropoulos, G., Pantic, M.: Gauss-newton deformable part models for face alignment in-the-wild. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1851–1858. IEEE (2014)
16.
go back to reference Tuzel, O., Porikli, F., Meer, P.: Learning on lie groups for invariant detection and tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, 2008, CVPR 2008, pp. 1–8. IEEE (2008) Tuzel, O., Porikli, F., Meer, P.: Learning on lie groups for invariant detection and tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, 2008, CVPR 2008, pp. 1–8. IEEE (2008)
17.
go back to reference Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 532–539. IEEE (2013) Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 532–539. IEEE (2013)
18.
go back to reference Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Incremental face alignment in the wild. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1859–1866. IEEE (2014) Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Incremental face alignment in the wild. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1859–1866. IEEE (2014)
19.
go back to reference Liu, X.: Discriminative face alignment. IEEE Trans. Pattern Anal. Machine Intell. 31(11), 1941–1954 (2009)CrossRef Liu, X.: Discriminative face alignment. IEEE Trans. Pattern Anal. Machine Intell. 31(11), 1941–1954 (2009)CrossRef
20.
go back to reference Kazemi, V., Sullivan, J.: Face alignment with part-based modeling. In: Proceedings of the British Machine Vision Conference, p. 27:1. BMVA Press (2011) Kazemi, V., Sullivan, J.: Face alignment with part-based modeling. In: Proceedings of the British Machine Vision Conference, p. 27:1. BMVA Press (2011)
21.
go back to reference Saragih, J.M., Lucey, S., Cohn, J.F.: Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vis. 91(2), 200–215 (2011)MathSciNetCrossRefMATH Saragih, J.M., Lucey, S., Cohn, J.F.: Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vis. 91(2), 200–215 (2011)MathSciNetCrossRefMATH
22.
go back to reference Dollár, P., Welinder, P., Perona, P.: Cascaded pose regression. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1078–1085. IEEE (2010) Dollár, P., Welinder, P., Perona, P.: Cascaded pose regression. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1078–1085. IEEE (2010)
23.
go back to reference Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 fps via regressing local binary features. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1685–1692. IEEE (2014) Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 fps via regressing local binary features. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1685–1692. IEEE (2014)
24.
go back to reference Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1867–1874. IEEE (2014) Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1867–1874. IEEE (2014)
25.
go back to reference Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression. Int. J. Comput. Vis. 107(2), 177–190 (2014)MathSciNetCrossRef Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression. Int. J. Comput. Vis. 107(2), 177–190 (2014)MathSciNetCrossRef
26.
go back to reference Tzimiropoulos, G.: Project-out cascaded regression with an application to face alignment. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015 Tzimiropoulos, G.: Project-out cascaded regression with an application to face alignment. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015
27.
go back to reference Yan, J., Lei, Z., Yi, D., Li, S.Z.: Learn to combine multiple hypotheses for accurate face alignment. In: 2013 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 392–396. IEEE (2013) Yan, J., Lei, Z., Yi, D., Li, S.Z.: Learn to combine multiple hypotheses for accurate face alignment. In: 2013 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 392–396. IEEE (2013)
28.
go back to reference Cootes, T.F., Ionita, M.C., Lindner, C., Sauer, P.: Robust and accurate shape model fitting using random forest regression voting. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7578, pp. 278–291. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33786-4_21 CrossRef Cootes, T.F., Ionita, M.C., Lindner, C., Sauer, P.: Robust and accurate shape model fitting using random forest regression voting. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7578, pp. 278–291. Springer, Heidelberg (2012). doi:10.​1007/​978-3-642-33786-4_​21 CrossRef
29.
go back to reference Alabort-i Medina, J., Antonakos, E., Booth, J., Snape, P., Zafeiriou, S.: Menpo: A comprehensive platform for parametric image alignment and visual deformable models. In: Proceedings of the ACM International Conference on Multimedia, MM 2014, pp. 679–682. ACM, New York (2014) Alabort-i Medina, J., Antonakos, E., Booth, J., Snape, P., Zafeiriou, S.: Menpo: A comprehensive platform for parametric image alignment and visual deformable models. In: Proceedings of the ACM International Conference on Multimedia, MM 2014, pp. 679–682. ACM, New York (2014)
30.
go back to reference Xiong, X., De la Torre, F.: Global supervised descent method. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015 Xiong, X., De la Torre, F.: Global supervised descent method. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015
31.
go back to reference Xiao, S., Yan, S., Kassim, A.A.: Facial landmark detection via progressive initialization. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, December 2015 Xiao, S., Yan, S., Kassim, A.A.: Facial landmark detection via progressive initialization. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, December 2015
32.
go back to reference Zhang, J., Kan, M., Shan, S., Chen, X.: Leveraging datasets with varying annotations for face alignment via deep regression network. In: The IEEE International Conference on Computer Vision (ICCV), December 2015 Zhang, J., Kan, M., Shan, S., Chen, X.: Leveraging datasets with varying annotations for face alignment via deep regression network. In: The IEEE International Conference on Computer Vision (ICCV), December 2015
33.
go back to reference Zhou, E., Fan, H., Cao, Z., Jiang, Y., Yin, Q.: Extensive facial landmark localization with coarse-to-fine convolutional network cascade. In: ICCV Workshop (2013) Zhou, E., Fan, H., Cao, Z., Jiang, Y., Yin, Q.: Extensive facial landmark localization with coarse-to-fine convolutional network cascade. In: ICCV Workshop (2013)
34.
go back to reference Zhang, J., Shan, S., Kan, M., Chen, X.: Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In: ECCV (2014) Zhang, J., Shan, S., Kan, M., Chen, X.: Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In: ECCV (2014)
35.
go back to reference Zhu, S., Li, C., Change Loy, C., Tang, X.: Face alignment by coarse-to-fine shape searching. In: CVPR (2015) Zhu, S., Li, C., Change Loy, C., Tang, X.: Face alignment by coarse-to-fine shape searching. In: CVPR (2015)
36.
go back to reference Burgos-Artizzu, X.P., Perona, P., Dollar, P.: Robust face landmark estimation under occlusion. In: The IEEE International Conference on Computer Vision (ICCV), December 2013 Burgos-Artizzu, X.P., Perona, P., Dollar, P.: Robust face landmark estimation under occlusion. In: The IEEE International Conference on Computer Vision (ICCV), December 2013
37.
go back to reference Yu, X., Lin, Z., Brandt, J., Metaxas, D.N.: Consensus of regression for occlusion-robust facial feature localization. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 105–118. Springer, Heidelberg (2014). doi:10.1007/978-3-319-10593-2_8 Yu, X., Lin, Z., Brandt, J., Metaxas, D.N.: Consensus of regression for occlusion-robust facial feature localization. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 105–118. Springer, Heidelberg (2014). doi:10.​1007/​978-3-319-10593-2_​8
38.
go back to reference Rao, A., Miller, D., Rose, K., Gersho, A.: Mixture of experts regression modeling by deterministic annealing. IEEE Trans. Sig. Process. 45(11), 2811–2820 (1997)CrossRef Rao, A., Miller, D., Rose, K., Gersho, A.: Mixture of experts regression modeling by deterministic annealing. IEEE Trans. Sig. Process. 45(11), 2811–2820 (1997)CrossRef
39.
go back to reference Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: A semi-automatic methodology for facial landmark annotation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR-Workshops), 5th Workshop on Analysis and Modeling of Faces and Gestures (AMFG2013), Portland Oregon, USA, June 2013 Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: A semi-automatic methodology for facial landmark annotation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR-Workshops), 5th Workshop on Analysis and Modeling of Faces and Gestures (AMFG2013), Portland Oregon, USA, June 2013
40.
go back to reference Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: Proceedings of IEEE International Conference on Computer Vision (ICCV-Workshops), 300 Faces in-the-Wild Challenge (300-W), Sydney, Australia, December 2013 Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: Proceedings of IEEE International Conference on Computer Vision (ICCV-Workshops), 300 Faces in-the-Wild Challenge (300-W), Sydney, Australia, December 2013
41.
go back to reference Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2879–2886. IEEE (2012) Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2879–2886. IEEE (2012)
42.
go back to reference Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRef Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRef
43.
go back to reference Čech, J., Franc, V., Uřičář, M., Matas, J.: Multi-view facial landmark detection by using a 3D shape model. Image Vis. Comput. 47, 60–70 (2016)CrossRef Čech, J., Franc, V., Uřičář, M., Matas, J.: Multi-view facial landmark detection by using a 3D shape model. Image Vis. Comput. 47, 60–70 (2016)CrossRef
44.
go back to reference Belhumeur, P.N., Jacobs, D.W., Kriegman, D., Kumar, N.: Localizing parts of faces using a consensus of exemplars. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 545–552. IEEE (2011) Belhumeur, P.N., Jacobs, D.W., Kriegman, D., Kumar, N.: Localizing parts of faces using a consensus of exemplars. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 545–552. IEEE (2011)
45.
go back to reference Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S.: Interactive facial feature localization. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 679–692. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33712-3_49 Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S.: Interactive facial feature localization. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 679–692. Springer, Heidelberg (2012). doi:10.​1007/​978-3-642-33712-3_​49
46.
go back to reference Yang, H., Jia, X., Loy, C.C., Robinson, P.: An empirical study of recent face alignment methods. CoRR abs/1511.05049 (2015) Yang, H., Jia, X., Loy, C.C., Robinson, P.: An empirical study of recent face alignment methods. CoRR abs/1511.05049 (2015)
Metadata
Title
Robust Face Alignment Using a Mixture of Invariant Experts
Authors
Oncel Tuzel
Tim K. Marks
Salil Tambe
Copyright Year
2016
DOI
https://doi.org/10.1007/978-3-319-46454-1_50

Premium Partner