nach oben

Erschienen in:

2021 | OriginalPaper | Buchkapitel

State-of-the-Art in 3D Face Reconstruction from a Single RGB Image

verfasst von : Haibin Fu, Shaojun Bian, Ehtzaz Chaudhry, Andres Iglesias, Lihua You, Jian Jun Zhang

Erschienen in: Computational Science – ICCS 2021

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Since diverse and complex emotions need to be expressed by different facial deformation and appearances, facial animation has become a serious and on-going challenge for computer animation industry. Face reconstruction techniques based on 3D morphable face model and deep learning provide one effective solution to reuse existing databases and create believable animation of new characters from images or videos in seconds, which greatly reduce heavy manual operations and a lot of time. In this paper, we review the databases and state-of-the-art methods of 3D face reconstruction from a single RGB image. First, we classify 3D reconstruction methods into three categories and review each of them. These three categories are: Shape-from-Shading (SFS), 3D Morphable Face Model (3DMM), and Deep Learning (DL) based 3D face reconstruction. Next, we introduce existing 2D and 3D facial databases. After that, we review 10 methods of deep learning-based 3D face reconstruction and evaluate four representative ones among them. Finally, we draw conclusions of this paper and discuss future research directions.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel Capsule Network Versus Convolutional Neural Network in Image Classification

Nächstes Kapitel Towards Understanding Time Varying Triangle Meshes

Zhao, W.-Y., Chellappa, R.: Symmetric shape-from-shading using self-ratio image. Int. J. Comput. Vis. 45(1), 55–75 (2001)CrossRef

Zhang, R., Tsai, P., Cryer, J.-E., Shah, M.: Shape-from-shading: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 21(8), 690–706 (1999)

Zhu, X., Lei, Z., Yan, J., Yi, D., Li, S.-Z.: High-fidelity pose and expression normalization for face recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 787–796 (2015)

Li, Y., Ma, L., Fan, H., Mitchell, K.: Feature-preserving detailed 3D face reconstruction from a single image. In: Proceedings of the 15th ACM SIGGRAPH European Conference on Visual Media Production, pp. 1–9 (2018)

Garrido, P., et al.: Reconstruction of personalized 3D face rigs from monocular video. ACM Trans. Graph. (TOG) 35(3), 1–15 (2016)CrossRef

Sengupta, S., Kanazawa, A., Castillo, C.-D., Jacobs, D.-W.: SfSNet: Learning shape, reflectance and illuminance of faces in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6296–6305 (2018)

Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (1999)

Tran, L., Liu, X.: Nonlinear 3D face morphable model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7346–7355 (2018)

Zhu, X., Lei, Z., Liu, X.: Face alignment across large poses: a 3D solution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 146–155 (2016)

10.

Huber, P.: Real-time 3D morphable shape model fitting to monocular in-the-wild videos. University of Surrey (2017)

11.

Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 296–301. IEEE (2009)

12.

Liu, P., Yu, H., Cang, S.: Adaptive neural network tracking control for underactuated systems with matched and mismatched disturbances. Nonlinear Dyn. 98(2), 1447–1464 (2019)CrossRef

13.

Gerig, T., et al.: Morphable face models-an open framework. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 75–82. IEEE (2018)

14.

Egger, B., Smith, W., Tewari, A., Wuhrer, A., Zollhoefer, M.: 3D morphable face models—past, present, and future. ACM Trans. Graph. (TOG) 39(5), 1–38 (2020)CrossRef

15.

Zollhöfer, M., Thies, J., Garrido, P.: State of the art on monocular 3D face reconstruction, tracking, and applications. Comput. Graph. Forum 37(2), 523–550 (2018)CrossRef

16.

Sun, L., Zhao, C., Yan, Z., Liu, P., Duckett, T., Stolkin, R.: A novel weakly-supervised approach for RGB-D-based nuclear waste object detection. IEEE Sens. J. (2018)

17.

Chen, A., Chen, Z., Zhang, G., Mitchell, K., Yu, J.: Photo-realistic facial details synthesis from single image. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9429–9439 (2019)

18.

Kasinski, A., Florek, A., Schmidt, A.: The PUT face database. Image Process. Commun. 13(3–4), 59–64 (2008)

19.

Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-pie. Image Vis. Comput. 28(5), 807–813 (2010)CrossRef

20.

Milborrow, S., Morkel, J., Nicolls, F.: The MUCT landmarked face database. Pattern Recognit. Assoc. S. Afr. 201(0) (2010)

21.

Grgic, M., Delac, K., Grgic, S.: SCface–surveillance cameras face database. Multimedia Tools Appl. 51(3), 863–879 (2011)CrossRef

22.

Koestinger, M., Wohlhart, P., Roth, P., Bischof, H.: Annotated facial landmarks in the wild: a large-scale, real-world database for facial landmark localization. In: 2011 IEEE International Conference on Computer Vision Workshops, pp. 2144–2151. IEEE (2011)

23.

Lui, Y.-M., Bolme, D., Phillips, P.-J., Beveridge, J.-R.: Preliminary studies on the good, the bad, and the ugly face recognition challenge problem. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 9–16. IEEE (2012)

24.

Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: Conference on Computer Vision and Pattern Recognition. IEEE (2012)

25.

Sagonas, C., Antonakos, E., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: database and results. Image Vis. Comput. 47, 3–18 (2016)CrossRef

26.

Ng, H.-W., Winkler, S.: A data-driven approach to cleaning large face datasets. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 343–347. IEEE (2014)

27.

Parkhi, O.-M., Vedaldi, A., Zisserman, A.: Deep face recognition (2015)

28.

Liu, Z., Luo, P., Wang, X., Tang. X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)

29.

Bansal, A., Nanduri, A., Castillo, C.-D., Ranjan, R., Chellappa, R.: UMDfaces: an annotated face dataset for training deep networks. In: 2017 IEEE International Joint Conference on Biometrics (IJCB), pp. 464–473. IEEE (2017)

30.

Shlizerman, I.-K., Seitz, S.-M., Miller, D., Brossard, E.: The MegaFace benchmark: 1 million faces for recognition at scale. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4873–4882 (2016)

31.

Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3D face alignment problem. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1021–1030 (2017)

32.

Huang, G.-B., Mattar, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments (2008)

33.

Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401–4410 (2019)

34.

Panetta, K., et al.: A comprehensive database for benchmarking imaging systems. IEEE Trans. Pattern Anal. Mach. Intell. 42(3), 509–520 (2020)

35.

Faltemier, T.-C., Bowyer, K.-W., Flynn, P.: Using a multi-instance enrollment representation to improve 3D face recognition. In: 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems, pp. 1–6. IEEE (2007)

36.

Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.-J.: A 3D facial expression database for facial behavior research. In: 7th International Conference on Automatic Face and Gesture Recognition, pp. 211–216. IEEE (2006)

37.

Yin, L., Chen, X, Sun, Y., Worm, T., Reale, M.: 3D dynamic facial expression database. In: 8th International Conference on Automatic Face & Gesture Recognition, pp. 1–6. IEEE (2008)

38.

Heseltine, T., Pears, N.: Three-dimensional face recognition using combinations of surface feature map subspace components. Image Vis. Comput. 26(3), 382–396 (2008)CrossRef

39.

Savran, A., et al.: Bosphorus database for 3D face analysis. In: Schouten, B., Juul, N.C., Drygajlo, A., Tistarelli, M. (eds.) BioID 2008. LNCS, vol. 5372, pp. 47–56. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89991-4_6CrossRef

40.

Yin, B., Sun, Y., Wang, C., Ge, Y.: BJUT-3D large scale 3D face database and information processing. J. Comput. Res. Dev. 46(6), 1009 (2009)

41.

Cosker, D., Krumhuber, E., Hilton, A.: A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. In: 2011 International Conference on Computer Vision, pp. 2296–2303. IEEE (2011)

42.

Bagdanov, A.-D., Bimbo, A.-D.: The florence 2D/3D hybrid face dataset. In: Proceedings of ACM Workshop on Human Gesture and Behavior Understanding, pp. 79–80 (2011)

43.

Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou., K.: FaceWarehouse: a 3D facial expression database for visual computing. IEEE Trans. Vis. Comput. Graph. 20(3), 413–425 (2013)

44.

Zhang, X., Yin, L., Cohn, J.-F.: BP4D-spontaneous: a high-resolution spontaneous 3D dynamic facial expression database. Image Vis. Comput. 32(10), 1–6 (2014)

45.

Zhang, Z., et al.: Multimodal spontaneous emotion corpus for human behavior analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3438–3446 (2016)

46.

Le, H.-A., Kakadiaris, I.-A.: UHDB31: a dataset for better understanding face recognition across pose and illumination variation. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 2555–2563 (2017)

47.

Cheng, S., Kotsia, I., Pantic, M., Zafeiriou, S.: 4DFAB: a large scale 4D database for facial expression analysis and biometric applications. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5117–5126 (2018)

48.

Ertugrul, I.-O., Cohn, J.-F., Jeni, L.-A., Zhang, A., Yin, L. Ji, Q.: Cross-domain au detection: domains, learning approaches, and measures. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 1–8. IEEE (2019)

49.

Yang, H., Zhu, H., Wang, Y., Huang, M., Shen, Q.: FaceScape: a large-scale high quality 3D face dataset and detailed riggable 3D face prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 601–610 (2020)

50.

Booth, J., Roussos, A., Ponniah, A., Dunaway, D., Zafeiriou, S.: Large scale 3D morphable models. Int. J. Comput. Vis. 126(2–4), 233–254 (2018)MathSciNetCrossRef

51.

Dai, H., Pears, N., Smith, W.-A., Duncan, C.: A 3D morphable model of craniofacial shape and texture variation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3085–3093 (2017)

52.

Chang, F., Tran, A.-T., Hassner, T., Masi, I., Nevatia, R., Medioni, G.: ExpNet: landmark-free, deep, 3D facial expressions. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 122–129. IEEE (2018)

53.

Booth, J., Roussos, A., Ververas, E., Antonakos, E., Ploumpis, S., Panagakis, Y.: 3D reconstruction of “in-the-wild” faces in images and videos. IEEE Trans. Pattern Anal. Mach. Intell. 40(11), 2638–2652 (2018)CrossRef

54.

Shang, J., Shen, T., Li, S., Zhou, L., Zhen, M.: Self-supervised monocular 3D face reconstruction by occlusion-aware multi-view geometry consistency (2020)

55.

Feng, Y., Wu, F., Shao, X., Wang, Y., Zhou, X.: Joint 3D face reconstruction and dense alignment with position map regression network. In: Proceedings of the European Conference on Computer Vision, pp. 534–551 (2018)

56.

Zeng, Xi., Peng, X., Qiao, Y.: DF2Net: a dense-fine-finer network for detailed 3D face reconstruction. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2315–2324 (2019)

57.

Zhang, J., Cai, H., Guo, Y., Peng, Z.: Landmark detection and 3D face reconstruction for caricature using a nonlinear parametric model (2020)

58.

Browatzki, B., Wallraven, C.: 3FabRec: fast few-shot face alignment by reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6110–6120 (2020)

59.

Wu, S., Rupprecht, C., Vedaldi, A.: Unsupervised learning of probably symmetric deformable 3D objects from images in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1–10 (2020)

60.

Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W., Frangi, A. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28CrossRef

Titel: State-of-the-Art in 3D Face Reconstruction from a Single RGB Image
verfasst von: Haibin Fu
Shaojun Bian
Ehtzaz Chaudhry
Andres Iglesias
Lihua You
Jian Jun Zhang
Verlag: Springer International Publishing
Buch: Computational Science – ICCS 2021
Print ISBN: 978-3-030-77976-4

Electronic ISBN: 978-3-030-77977-1

Copyright-Jahr: 2021
DOI: https://doi.org/10.1007/978-3-030-77977-1_3

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"