Top

Published in:

2024 | OriginalPaper | Chapter

Applying Transfer Testing to Identify Annotation Discrepancies in Facial Emotion Data Sets

Authors : Sarah Dreher, Jens Gebele, Philipp Brune

Published in: Mobile, Secure, and Programmable Networking

Publisher: Springer Nature Switzerland

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The field of Artificial Intelligence (AI) has a significant impact on the way computers and humans interact. The topic of (facial) emotion recognition has gained a lot of attention in recent years. Majority of research literature focuses on improvement of algorithms and Machine Learning (ML) models for single data sets. Despite the impressive results achieved, the impact of the (training) data quality with its potential biases and annotation discrepancies is often neglected. Therefore, this paper demonstrates an approach to detect and evaluate annotation label discrepancies between three separate (facial) emotion recognition databases by Transfer Testing with three ML architectures. The findings indicate Transfer Testing to be a new promising method to detect inconsistencies in data annotations of emotional states, implying label bias and/or ambiguity. Therefore, Transfer Testing is a method to verify the transferability of trained ML models. Such research is the foundation for developing more accurate AI-based emotion recognition systems, which are also robust in real-life scenarios.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Privacy-Preserving Tree-Based Inference with TFHE

next chapter Multiple Person Tracking Based on Gait Identification Using Kinect and OpenPose

Affectiva - Humanizing Technology (2021). https://www.affectiva.com/

NVIDIA Data Science Stack. NVIDIA Corporation (2021)

Replika (2021). https://replika.com

Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)

Darwin, C.: The Expression of the Emotions in Man and Animals. John Murray, London (1872)CrossRef

Davenport, T., Guha, A., Grewal, D., Bressgott, T.: How artificial intelligence will change the future of marketing. J. Acad. Mark. Sci. 48(1), 24–42 (2020)CrossRef

Davoli, L., et al.: On driver behavior recognition for increased safety: a roadmap. Safety 6(4) (2020). https://doi.org/10.3390/safety6040055

Ekman, P.: Basic emotions. In: Handbook of Cognition and Emotion, pp. 301–320. Wiley, New York (1999)

Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124 (1971)CrossRef

10.

Ekman, P., Friesen, W.V.: Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues, vol. 10. ISHK (2003)

11.

Ekundayo, O.S., Viriri, S.: Facial expression recognition: a review of trends and techniques. IEEE Access 9, 136944–136973 (2021). https://doi.org/10.1109/ACCESS.2021.3113464CrossRef

12.

Ekweariri, A.N., Yurtkan, K.: Facial expression recognition using enhanced local binary patterns. In: 2017 9th International Conference on Computational Intelligence and Communication Networks (CICN), pp. 43–47. IEEE (2017)

13.

Gebele, J., Brune, P., Faußer, S.: Face value: on the impact of annotation (in-)consistencies and label ambiguity in facial data on emotion recognition. In: IEEE 26th International Conference on Pattern Recognition (2022)

14.

Generosi, A., Ceccacci, S., Mengoni, M.: A deep learning-based system to track and analyze customer behavior in retail store. In: 2018 IEEE 8th International Conference on Consumer Electronics-Berlin (ICCE-Berlin), pp. 1–6. IEEE (2018)

15.

Goodfellow, I.J., et al.: Challenges in Representation Learning: A report on three machine learning contests. arXiv:1307.0414 (2013)

16.

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

17.

Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018). https://doi.org/10.1109/CVPR.2018.00745

18.

Huang, M.H., Rust, R.T.: Artificial intelligence in service. J. Serv. Res. 21(2), 155–172 (2018)CrossRef

19.

Huang, M.H., Rust, R.T.: A strategic framework for artificial intelligence in marketing. J. Acad. Mark. Sci. 49(1), 30–50 (2021). https://doi.org/10.1007/s11747-020-00749-9CrossRef

20.

Jaison, A., Deepa, C.: A review on facial emotion recognition and classification analysis with deep learning. Biosci. Biotechnol. Res. Commun. 14(5), 154–161 (2021). https://doi.org/10.21786/bbrc/14.5/29

21.

Khaireddin, Y., Chen, Z.: Facial Emotion Recognition: State of the Art Performance on FER2013. arXiv:2105.03588 (2021)

22.

Knyazev, B., Shvetsov, R., Efremova, N., Kuharenko, A.: Convolutional neural networks pretrained on large face recognition datasets for emotion classification from video. arXiv:1711.04598 (2017)

23.

Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural. Inf. Process. Syst. 25, 1097–1105 (2012)

24.

Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998). https://doi.org/10.1109/5.726791CrossRef

25.

Li, J., Nie, J., Guo, D., Hong, R., Wang, M.: Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers. http://arxiv.org/abs/2207.11081

26.

Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. 13(3), 1195–1215 (2020)CrossRef

27.

Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, pp. 2584–2593. IEEE (2017). https://doi.org/10.1109/CVPR.2017.277

28.

Liu, X., Kumar, B.V.K.V., You, J., Jia, P.: Adaptive deep metric learning for identity-aware facial expression recognition. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 522–531 (2017). https://doi.org/10.1109/CVPRW.2017.79

29.

Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 94–101 (2010). https://doi.org/10.1109/CVPRW.2010.5543262

30.

Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with Gabor wavelets. In: Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, pp. 200–205 (1998). https://doi.org/10.1109/AFGR.1998.670949

31.

Marín-Morales, J., et al.: Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors. Sci. Rep. 8(1), 1–15 (2018)CrossRef

32.

Mellouk, W., Handouzi, W.: Facial emotion recognition using deep learning: Review and insights. Procedia Comput. Sci. 175, 689–694 (2020)CrossRef

33.

Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)CrossRef

34.

Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010). https://doi.org/10.1109/TKDE.2009.191CrossRef

35.

Quinn, M.A., Sivesind, G., Reis, G.: Real-time emotion recognition from facial expressions. Standford University (2017)

36.

Rouast, P.V., Adam, M., Chiong, R.: Deep learning for human affect recognition: insights and new developments. IEEE Trans. Affect. Comput. 12(2), 524–543 (2019)CrossRef

37.

Shao, J., Qian, Y.: Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355, 82–92 (2019)CrossRef

38.

Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556 (2015)

39.

Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

40.

Tian, Y.I., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 97–115 (2001)CrossRef

41.

Yang, H., Ciftci, U., Yin, L.: Facial expression recognition by de-expression residue learning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2168–2177 (2018). https://doi.org/10.1109/CVPR.2018.00231

Title: Applying Transfer Testing to Identify Annotation Discrepancies in Facial Emotion Data Sets
Authors: Sarah Dreher
Jens Gebele
Philipp Brune
Publisher: Springer Nature Switzerland
Book: Mobile, Secure, and Programmable Networking
Print ISBN: 978-3-031-52425-7

Electronic ISBN: 978-3-031-52426-4

Copyright Year: 2024
DOI: https://doi.org/10.1007/978-3-031-52426-4_11

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner