nach oben

Erschienen in:

2022 | OriginalPaper | Buchkapitel

Personalized Frame-Level Facial Expression Recognition in Video

verfasst von : Andrey V. Savchenko

Erschienen in: Pattern Recognition and Artificial Intelligence

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

In this paper, the personalization of the video-based frame-level facial expression recognition is studied for multi-user systems if a small amount of short videos are available for each user. At first, embeddings of each video frame are computed using deep convolutional neural network pre-trained on a large emotional dataset of static images. Next, a dataset of videos is used to train a subject-independent emotion classifier, such as feed-forward neural network or frame attention network. Finally, it is proposed to fine-tune this neural classifier on the videos of each user of interest. As a result, every user is associated with his or her own emotional model. The classifier in a multi-user system is chosen by an appropriate video-based face recognition method. The experimental study with the RAMAS dataset demonstrates the significant (up to 25%) increase in accuracy of the proposed approach when compared to a subject-independent facial expression recognition.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel Improving UWB Image Reconstruction for Breast Cancer Diagnosis by Doing an Iterative Analysis of Radar Signals

Nächstes Kapitel 3D Reconstruction of Medical Image Based on Improved Ray Casting Algorithm

https://github.com/HSE-asavchenko/face-emotion-recognition/blob/main/src/train_ramas.ipynb.

Pietikäinen, M., Silven, O.: Challenges of artificial intelligence-from machine learning and computer vision to emotional intelligence. arXiv preprint arXiv:2201.01466 (2022)

Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. (2020). https://doi.org/10.1109/TAFFC.2020.2981446CrossRef

Savchenko, A.V.: Facial expression and attributes recognition based on multi-task learning of lightweight neural networks. In: Proceedings of 19th International Symposium on Intelligent Systems and Informatics (SISY), pp. 119–124. IEEE (2021)

Cerezo, E., et al.: Real-time facial expression recognition for natural interaction. In: Martí, J., Benedí, J.M., Mendonça, A.M., Serrat, J. (eds.) IbPRIA 2007. LNCS, vol. 4478, pp. 40–47. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-72849-8_6

Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: Vggface2: a dataset for recognising faces across pose and age. In: Proceedings of 13th International Conference on Automatic Face & Gesture Recognition (FG), pp. 67–74. IEEE (2018)

Mollahosseini, A., Hasani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)CrossRef

Perepelkina, O., Kazimirova, E., Konstantinova, M.: RAMAS: Russian multimodal corpus of dyadic interaction for affective computing. In: Karpov, A., Jokisch, O., Potapova, R. (eds.) SPECOM 2018. LNCS (LNAI), vol. 11096, pp. 501–510. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99579-3_52CrossRef

Ryumina, E., Verkholyak, O., Karpov, A.: Annotation confidence vs. training sample size: trade-off solution for partially-continuous categorical emotion recognition. In: Proceedings of Interspeech 2021, pp. 3690–3694 (2021). https://doi.org/10.21437/Interspeech.2021-1636

Saleem, S.M., Zeebaree, S.R., Abdulrazzaq, M.B.: Real-life dynamic facial expression recognition: a review. J. Phys. Conf. Ser. 1963, 012010 (2021). IOP Publishing

10.

Ben, X., et al.: Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans. Pattern Anal. Mach. Intell. (2021)

11.

Saeed, A., Al-Hamadi, A., Niese, R., Elzobi, M.: Frame-based facial expression recognition using geometrical features. In: Advances in Human-Computer Interaction 2014 (2014)

12.

Bargal, S.A., Barsoum, E., Ferrer, C.C., Zhang, C.: Emotion recognition in the wild from videos using images. In: Proceedings of the 18th International Conference on Multimodal Interaction (ICMI), pp. 433–436. ACM (2016)

13.

Meng, D., Peng, X., Wang, K., Qiao, Y.: Frame attention networks for facial expression recognition in videos. In: Proceedings of the International Conference on Image Processing (ICIP), pp. 3866–3870. IEEE (2019)

14.

Demochkina, P., Savchenko, A.V.: Neural network model for video-based facial expression recognition in-the-wild on mobile devices. In: Proceedings of International Conference on Information Technology and Nanotechnology (ITNT), pp. 1–5. IEEE (2021)

15.

Savchenko, A.V., Khokhlova, Y.I.: About neural-network algorithms application in viseme classification problem with face video in audiovisual speech recognition systems. Optical Memory Neural Netw. 23(1), 34–42 (2014). https://doi.org/10.3103/S1060992X14010068CrossRef

16.

Zhou, H., et al.: Exploring emotion features and fusion strategies for audio-video emotion recognition. In: Proceedings of International Conference on Multimodal Interaction (ICMI), pp. 562–566. ACM (2019)

17.

Peña, A., Morales, A., Serna, I., Fierrez, J., Lapedriza, A.: Facial expressions as a vulnerability in face recognition. In: Proceedings of International Conference on Image Processing (ICIP), pp. 2988–2992. IEEE (2021)

18.

Shahabinejad, M., Wang, Y., Yu, Y., Tang, J., Li, J.: Toward personalized emotion recognition: a face recognition based attention method for facial emotion recognition. In: Proceedings of 16th International Conference on Automatic Face and Gesture Recognition (FG), pp. 1–5. IEEE (2021)

19.

Zhao, Y., Li, J., Zhang, S., Chen, L., Gong, Y.: Domain and speaker adaptation for Cortana speech recognition. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5984–5988. IEEE (2018)

20.

Savchenko, L.V., Savchenko, A.V.: Speaker-aware training of speech emotion classifier with speaker recognition. In: Karpov, A., Potapova, R. (eds.) SPECOM 2021. LNCS (LNAI), vol. 12997, pp. 614–625. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87802-3_55CrossRef

21.

Savchenko, A.V.: Phonetic words decoding software in the problem of Russian speech recognition. Autom. Remote. Control. 74(7), 1225–1232 (2013)CrossRef

22.

Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: additive angular margin loss for deep face recognition. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4690–4699. IEEE (2019)

23.

Naas, S.A., Sigg, S.: Real-time emotion recognition for sales. In: Proceedings of 16th International Conference on Mobility, Sensing and Networking (MSN), pp. 584–591. IEEE (2020)

24.

Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)CrossRef

25.

Makarov, I., Bakhanova, M., Nikolenko, S., Gerasimova, O.: Self-supervised recurrent depth estimation with attention mechanisms. PeerJ Comput. Sci. 8, e865 (2022)CrossRef

26.

Sokolova, A.D., Kharchevnikova, A.S., Savchenko, A.V.: Organizing multimedia data in video surveillance systems based on face verification with convolutional neural networks. In: van der Aalst, W.M.P., et al. (eds.) AIST 2017. LNCS, vol. 10716, pp. 223–230. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73013-4_20

27.

Perepelkina, O., Sterling, G., Konstantinova, M., Kazimirova, E.: RAMAS: the Russian acted multimodal affective set for affective computing and emotion recognition studies. In: Proceedings of European Society for Cognitive and Affective Neuroscience (ESCAN), pp. 86–86 (2018)

28.

Savchenko, A.V.: Efficient facial representations for age, gender and identity recognition in organizing photo albums using multi-output convnet. PeerJ Comput. Sci. 5, e197 (2019)

29.

Kollias, D., Zafeiriou, S.: Analysing affective behavior in the second ABAW2 competition. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 3652–3660. IEEE (2021)

Titel: Personalized Frame-Level Facial Expression Recognition in Video
verfasst von: Andrey V. Savchenko
Verlag: Springer International Publishing
Buch: Pattern Recognition and Artificial Intelligence
Print ISBN: 978-3-031-09036-3

Electronic ISBN: 978-3-031-09037-0

Copyright-Jahr: 2022
DOI: https://doi.org/10.1007/978-3-031-09037-0_37

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner