Skip to main content
main-content
Top

Hint

Swipe to navigate through the chapters of this book

Published in:
Cover of the book

2018 | OriginalPaper | Chapter

Deep Learning Approaches for Facial Emotion Recognition: A Case Study on FER-2013

Authors: Panagiotis Giannopoulos, Isidoros Perikos, Ioannis Hatzilygeroudis

Published in: Advances in Hybridization of Intelligent Methods

Publisher: Springer International Publishing

share
SHARE

Abstract

Emotions constitute an innate and important aspect of human behavior that colors the way of human communication. The accurate analysis and interpretation of the emotional content of human facial expressions is essential for the deeper understanding of human behavior. Although a human can detect and interpret faces and facial expressions naturally, with little or no effort, accurate and robust facial expression recognition by computer systems is still a great challenge. The analysis of human face characteristics and the recognition of its emotional states are considered to be very challenging and difficult tasks. The main difficulties come from the non-uniform nature of human face and variations in conditions such as lighting, shadows, facial pose and orientation. Deep learning approaches have been examined as a stream of methods to achieve robustness and provide the necessary scalability on new type of data. In this work, we examine the performance of two known deep learning approaches (GoogLeNet and AlexNet) on facial expression recognition, more specifically the recognition of the existence of emotional content, and on the recognition of the exact emotional content of facial expressions. The results collected from the study are quite interesting.

To get access to this content you need the following product:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 69.000 Bücher
  • über 500 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Testen Sie jetzt 15 Tage kostenlos.

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 50.000 Bücher
  • über 380 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




Testen Sie jetzt 15 Tage kostenlos.

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 58.000 Bücher
  • über 300 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Testen Sie jetzt 15 Tage kostenlos.

Literature
1.
go back to reference Pantic, M.: Facial expression recognition. In: Encyclopedia of Biometrics, pp. 400–406. Springer, US (2009) Pantic, M.: Facial expression recognition. In: Encyclopedia of Biometrics, pp. 400–406. Springer, US (2009)
2.
go back to reference Ekman, P., Rosenberg, E.L. (eds.): What the face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System. Oxford University Press, Oxford, UK (2005) Ekman, P., Rosenberg, E.L. (eds.): What the face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System. Oxford University Press, Oxford, UK (2005)
3.
go back to reference Mehrabian, A.: Communication without words. Psychol. Today 2(4), 53–56 (1968) Mehrabian, A.: Communication without words. Psychol. Today 2(4), 53–56 (1968)
4.
go back to reference Heylen, D.: Head gestures, gaze and the principles of conversational structure. Int. J. Humanoid Rob. 3(03), 241–267 (2006) CrossRef Heylen, D.: Head gestures, gaze and the principles of conversational structure. Int. J. Humanoid Rob. 3(03), 241–267 (2006) CrossRef
5.
go back to reference Ochs, M., Niewiadomski, R., Pelachaud, C.: Facial Expressions of Emotions for Virtual Characters. The Oxford Handbook of Affective Computing, 261 (2014) Ochs, M., Niewiadomski, R., Pelachaud, C.: Facial Expressions of Emotions for Virtual Characters. The Oxford Handbook of Affective Computing, 261 (2014)
6.
go back to reference Liebold, B., Richter, R., Teichmann, M., Hamker, F.H., Ohler, P.: Human capacities for emotion recognition and their implications for computer vision. i-com, 14(2), pp. 126–137 (2015) Liebold, B., Richter, R., Teichmann, M., Hamker, F.H., Ohler, P.: Human capacities for emotion recognition and their implications for computer vision. i-com, 14(2), pp. 126–137 (2015)
7.
go back to reference Clavel, C.: Surprise and human-agent interactions. Rev. Cogn. Linguist. 13(2), 461–477 (2015) CrossRef Clavel, C.: Surprise and human-agent interactions. Rev. Cogn. Linguist. 13(2), 461–477 (2015) CrossRef
8.
go back to reference Liebold, B., Ohler, P.: Multimodal emotion expressions of virtual agents, mimic and vocal emotion expressions and their effects on emotion recognition. In: Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp. 405–410. IEEE (2013) Liebold, B., Ohler, P.: Multimodal emotion expressions of virtual agents, mimic and vocal emotion expressions and their effects on emotion recognition. In: Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp. 405–410. IEEE (2013)
9.
go back to reference Bahreini, K., Nadolski, R., Westera, W.: Towards multimodal emotion recognition in e-learning environments. Interact. Learning Environ. 24(3), 590–605 (2016) CrossRef Bahreini, K., Nadolski, R., Westera, W.: Towards multimodal emotion recognition in e-learning environments. Interact. Learning Environ. 24(3), 590–605 (2016) CrossRef
10.
go back to reference Akputu, K.O., Seng, K.P., Lee, Y.L.: Facial emotion recognition for intelligent tutoring environment. In: 2nd International Conference on Machine Learning and Computer Science (IMLCS’2013), pp. 9–13 (2013) Akputu, K.O., Seng, K.P., Lee, Y.L.: Facial emotion recognition for intelligent tutoring environment. In: 2nd International Conference on Machine Learning and Computer Science (IMLCS’2013), pp. 9–13 (2013)
11.
go back to reference Shen, L., Wang, M., Shen, R.: Affective e—learning: Using “emotional” data to improve learning in pervasive learning environment. Educ. Technol. Soc. 12(2), 176–189 (2009) Shen, L., Wang, M., Shen, R.: Affective e—learning: Using “emotional” data to improve learning in pervasive learning environment. Educ. Technol. Soc. 12(2), 176–189 (2009)
12.
go back to reference Koutlas, A., Fotiadis, D.I.: An automatic region based methodology for facial expression recognition. In: IEEE International Conference on Systems Man and Cybernetics SMC, pp. 662–666 (2008) Koutlas, A., Fotiadis, D.I.: An automatic region based methodology for facial expression recognition. In: IEEE International Conference on Systems Man and Cybernetics SMC, pp. 662–666 (2008)
13.
go back to reference Pantic, M., Rothkrantz, L.J.M.: Automatic analysis of facial expressions: The state of the art. Pattern Anal. Mach. Intell. IEEE Trans. 22(12), 1424–1445 (2000) CrossRef Pantic, M., Rothkrantz, L.J.M.: Automatic analysis of facial expressions: The state of the art. Pattern Anal. Mach. Intell. IEEE Trans. 22(12), 1424–1445 (2000) CrossRef
14.
go back to reference Arca, S., Campadelli, P., Lanzarotti, R.: An automatic feature-based face recognition system. In: Proceedings of the 5th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’04) (2004) Arca, S., Campadelli, P., Lanzarotti, R.: An automatic feature-based face recognition system. In: Proceedings of the 5th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’04) (2004)
15.
go back to reference Ekman, P.: Basic Emotions. Handbook of Cognition and Emotion, pp. 45–60 (1999) Ekman, P.: Basic Emotions. Handbook of Cognition and Emotion, pp. 45–60 (1999)
16.
go back to reference Ortony, A., Clore, G., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988) CrossRef Ortony, A., Clore, G., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988) CrossRef
17.
18.
go back to reference Wang, W., Yang, J., Xiao, J., Li, S., Zhou, D.: Face recognition based on deep learning. In: International Conference on Human Centered Computing, pp. 812–820. International Publishing, Springer (2014) Wang, W., Yang, J., Xiao, J., Li, S., Zhou, D.: Face recognition based on deep learning. In: International Conference on Human Centered Computing, pp. 812–820. International Publishing, Springer (2014)
19.
go back to reference Salakhutdinov, R., Hinton, G.: Deep boltzmann machines. In: Artificial Intelligence and Statistics, pp. 448–455 (2009) Salakhutdinov, R., Hinton, G.: Deep boltzmann machines. In: Artificial Intelligence and Statistics, pp. 448–455 (2009)
20.
go back to reference Deng, L.: A tutorial survey of architectures, algorithms, and applications for deep learning. APSIPA Trans. Signal Inf. Process. 3, e2 (2014) CrossRef Deng, L.: A tutorial survey of architectures, algorithms, and applications for deep learning. APSIPA Trans. Signal Inf. Process. 3, e2 (2014) CrossRef
21.
go back to reference Srivastava, N., Salakhutdinov, R.R.: Multimodal learning with deep boltzmann machines. In: Advances in Neural Information Processing Systems, pp. 2222–2230 (2012) Srivastava, N., Salakhutdinov, R.R.: Multimodal learning with deep boltzmann machines. In: Advances in Neural Information Processing Systems, pp. 2222–2230 (2012)
22.
go back to reference Căleanu, C.D.: Face expression recognition: A brief overview of the last decade. In: IEEE 8th International Symposium on Applied Computational Intelligence and Informatics (SACI), pp. 157–161 (2013) Căleanu, C.D.: Face expression recognition: A brief overview of the last decade. In: IEEE 8th International Symposium on Applied Computational Intelligence and Informatics (SACI), pp. 157–161 (2013)
23.
go back to reference Sariyanidi, E., Gunes, H., Cavallaro, A.: Automatic analysis of facial affect: A survey of registration, representation, and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015) CrossRef Sariyanidi, E., Gunes, H., Cavallaro, A.: Automatic analysis of facial affect: A survey of registration, representation, and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015) CrossRef
24.
go back to reference Danelakis, A., Theoharis, T., Pratikakis, I.: A survey on facial expression recognition in 3D video sequences. Multimedia Tools Appl. 74(15), 5577–5615 (2015) CrossRef Danelakis, A., Theoharis, T., Pratikakis, I.: A survey on facial expression recognition in 3D video sequences. Multimedia Tools Appl. 74(15), 5577–5615 (2015) CrossRef
25.
go back to reference Aung, D.M., Aye, N.A.: Facial expression classification using histogram based method. In: International Conference on Signal Processing Systems (2012) Aung, D.M., Aye, N.A.: Facial expression classification using histogram based method. In: International Conference on Signal Processing Systems (2012)
26.
go back to reference Murthy, G.R.S., Jadon, R.S. Recognizing facial expressions using eigenspaces. In: IEEE International Conference on Computational Intelligence and Multimedia Applications. 3, pp. 201–207 (2007) Murthy, G.R.S., Jadon, R.S. Recognizing facial expressions using eigenspaces. In: IEEE International Conference on Computational Intelligence and Multimedia Applications. 3, pp. 201–207 (2007)
27.
go back to reference Thai, L.H., Nguyen, N.D.T., Hai, T.S.: A facial expression classification system integrating canny, principal component analysis and artificial neural network.(2011) arXiv preprint arXiv:​1111.​4052 Thai, L.H., Nguyen, N.D.T., Hai, T.S.: A facial expression classification system integrating canny, principal component analysis and artificial neural network.(2011) arXiv preprint arXiv:​1111.​4052
28.
go back to reference Perikos, I., Ziakopoulos, E., Hatzilygeroudis, I.: Recognizing emotions from facial expressions using neural network. In: IFIP International Conference on Artificial Intelligence Applications and Innovations, pp. 236–245. Springer, Heidelberg (2014) Perikos, I., Ziakopoulos, E., Hatzilygeroudis, I.: Recognizing emotions from facial expressions using neural network. In: IFIP International Conference on Artificial Intelligence Applications and Innovations, pp. 236–245. Springer, Heidelberg (2014)
29.
go back to reference Perikos, I., Ziakopoulos, E., & Hatzilygeroudis, I.: Recognize emotions from facial expressions using a SVM and neural network schema. In: Engineering Applications of Neural Networks, pp. 265–274. Springer International Publishing, (2015) Perikos, I., Ziakopoulos, E., & Hatzilygeroudis, I.: Recognize emotions from facial expressions using a SVM and neural network schema. In: Engineering Applications of Neural Networks, pp. 265–274. Springer International Publishing, (2015)
30.
go back to reference Anderson, K., McOwan, P.W.: A real-time automated system for the recognition of human facial expressions. IEEE Trans. Syst. Man Cybern. Part B (Cybern.), 36(1), 96–105 (2006) Anderson, K., McOwan, P.W.: A real-time automated system for the recognition of human facial expressions. IEEE Trans. Syst. Man Cybern. Part B (Cybern.), 36(1), 96–105 (2006)
31.
go back to reference Přinosil, J., Smékal, Z., Esposito, A.: Combining features for recognizing emotional facial expressions in static images. In: Esposito, A., Bourbakis, N.G., Avouris, N., Hatzilygeroudis, I. (eds.) Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction, pp. 56−69. Springer, Heidelberg (2008) Přinosil, J., Smékal, Z., Esposito, A.: Combining features for recognizing emotional facial expressions in static images. In: Esposito, A., Bourbakis, N.G., Avouris, N., Hatzilygeroudis, I. (eds.) Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction, pp. 56−69. Springer, Heidelberg (2008)
32.
go back to reference Shinohara, Y., Otsu, N.: Facial expression recognition using fisher weight maps. In: Proceedings Sixth IEEE International Conference on Automatic Face and Gesture Recognition, IEEE. pp. 499–504 (2004) Shinohara, Y., Otsu, N.: Facial expression recognition using fisher weight maps. In: Proceedings Sixth IEEE International Conference on Automatic Face and Gesture Recognition, IEEE. pp. 499–504 (2004)
33.
go back to reference Yang, J., Zhang, D., Frangi, A.F., Yang, J.Y.: Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 131–137 (2004) CrossRef Yang, J., Zhang, D., Frangi, A.F., Yang, J.Y.: Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 131–137 (2004) CrossRef
34.
go back to reference Oh, S.K., Yoo, S.H., Pedrycz, W.: Design of face recognition algorithm using PCA-LDA combined for hybrid data pre-processing and polynomial-based RBF neural networks: Design and its application. Expert Syst. Appl. 40(5), 1451–1466 (2013) CrossRef Oh, S.K., Yoo, S.H., Pedrycz, W.: Design of face recognition algorithm using PCA-LDA combined for hybrid data pre-processing and polynomial-based RBF neural networks: Design and its application. Expert Syst. Appl. 40(5), 1451–1466 (2013) CrossRef
35.
go back to reference Mohammadi, M.R., Fatemizadeh, E., Mahoor, M.H.: PCA-based dictionary building for accurate facial expression recognition via sparse representation. J. Vis. Commun. Image Represent. 25(5), 1082–1092 (2014) CrossRef Mohammadi, M.R., Fatemizadeh, E., Mahoor, M.H.: PCA-based dictionary building for accurate facial expression recognition via sparse representation. J. Vis. Commun. Image Represent. 25(5), 1082–1092 (2014) CrossRef
37.
go back to reference Ionescu, R.T., Popescu, M., Grozea, C.: Local learning to improve bag of visual words model for facial expression recognition. In: Workshop on Challenges in Representation Learning, ICML (2013) Ionescu, R.T., Popescu, M., Grozea, C.: Local learning to improve bag of visual words model for facial expression recognition. In: Workshop on Challenges in Representation Learning, ICML (2013)
38.
go back to reference Mollahosseini, A., Chan, D., & Mahoor, M.H. (2016, March). Going deeper in facial expression recognition using deep neural networks. In: IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE. pp. 1–10 (2016) Mollahosseini, A., Chan, D., & Mahoor, M.H. (2016, March). Going deeper in facial expression recognition using deep neural networks. In: IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE. pp. 1–10 (2016)
39.
go back to reference Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Zhou, Y.: Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing pp. 117–124. Springer, Heidelberg (2013) Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Zhou, Y.: Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing pp. 117–124. Springer, Heidelberg (2013)
40.
go back to reference Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. (2014) arXiv preprint arXiv:​1409.​4842 Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. (2014) arXiv preprint arXiv:​1409.​4842
41.
go back to reference Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ng, A.Y.: Large scale distributed deep networks. In: Advances in Neural Information Processing Systems, pp. 1223−1231 (2012) Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ng, A.Y.: Large scale distributed deep networks. In: Advances in Neural Information Processing Systems, pp. 1223−1231 (2012)
42.
go back to reference Krizhevsky, A., Sutskever, I., Hinton. G.E.: ImageNet classification with deep convolutional neural networks. Part of: Adv. Neural Inf. Process. Syst. NIPS, 25 (2012) Krizhevsky, A., Sutskever, I., Hinton. G.E.: ImageNet classification with deep convolutional neural networks. Part of: Adv. Neural Inf. Process. Syst. NIPS, 25 (2012)
43.
go back to reference Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675−678. ACM (2014) Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675−678. ACM (2014)
45.
go back to reference I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, Y. Zhou, C. Ramaiah, F. Feng, R. Li, X. Wang, D. Athanasakis, J. Shawe-Taylor, M. Milakov, J. Park, R. Ionescu, M. Popescu, C. Grozea, J. Bergstra, J. Xie, L. Romaszko, B. Xu, Z. Chuang, and Y. Bengio, Challenges in representation learning: A report on three machine learning contests, Neural Networks, vol. 64, pp. 59–63 (2015) I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, Y. Zhou, C. Ramaiah, F. Feng, R. Li, X. Wang, D. Athanasakis, J. Shawe-Taylor, M. Milakov, J. Park, R. Ionescu, M. Popescu, C. Grozea, J. Bergstra, J. Xie, L. Romaszko, B. Xu, Z. Chuang, and Y. Bengio, Challenges in representation learning: A report on three machine learning contests, Neural Networks, vol. 64, pp. 59–63 (2015)
Metadata
Title
Deep Learning Approaches for Facial Emotion Recognition: A Case Study on FER-2013
Authors
Panagiotis Giannopoulos
Isidoros Perikos
Ioannis Hatzilygeroudis
Copyright Year
2018
DOI
https://doi.org/10.1007/978-3-319-66790-4_1

Premium Partner