Skip to main content
Top

2024 | OriginalPaper | Chapter

A Review of Survey and Assessment of Facial Emotion Recognition (FER) by Convolutional Neural Networks

Authors : Sanyam Agarwal, Veer Daksh Agarwal, Ishaan Agarwal, Vipin Mittal, Lakshay Singla, Ahmed Hussein Alkhayyat

Published in: Micro-Electronics and Telecommunication Engineering

Publisher: Springer Nature Singapore

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Computer vision and the area of artificial intelligence (AI) both heavily rely on the detection of facial expressions. This article concentrates on operations based on face images. It demonstrates how visual articulations are most important data facilitates, despite the limitless possibilities of how FER can be analyzed by using various instruments. This essay provides a succinct analysis of recent FER research. However, theoretical FER structure designs and their initial evaluations are displayed close by conventional FER approaches. The presentation of numerous FER views using the “start to finish” learning permission through critical associating authorization follows. As a result, this study will help in connecting a convolutional neural network (CNN) for some LSTM components (long transient memory). This paper concludes with a short poll, evaluation assessment, findings, and standards that serve as a standard for measurable connections between all of these FER studies and experiments. For students in FER, this audit can serve as a succinct manual that provides pertinent details and evaluation for recent tests. Additionally, knowledgeable examiners are searching for promising paths for future work.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Walecki R, Rudovic O (2017) Deep structured learning for facial expression intensity estimation. Image Vis Comput Walecki R, Rudovic O (2017) Deep structured learning for facial expression intensity estimation. Image Vis Comput
2.
go back to reference Kaulard K, Cunningham DW, Bülthoff HH, Wallraven C (2012) The MPI facial expression database—a validated database of emotional and conversational facial expressions. PLoS ONE Kaulard K, Cunningham DW, Bülthoff HH, Wallraven C (2012) The MPI facial expression database—a validated database of emotional and conversational facial expressions. PLoS ONE
3.
go back to reference Chu WS, Torre FD, Cohn JF (2017) Learning spatial and temporal cues for multi-label facial action unit detection. In: Proceedings of the 12th IEEE international conference on automatic face and gesture recognition, Washington, DC, USA, 30 May–3 June 2017 Chu WS, Torre FD, Cohn JF (2017) Learning spatial and temporal cues for multi-label facial action unit detection. In: Proceedings of the 12th IEEE international conference on automatic face and gesture recognition, Washington, DC, USA, 30 May–3 June 2017
4.
go back to reference Gunawan AAS (2015) Face expression detection on Kinect using active appearance model and fuzzy logic. Procedia Comput Sci Gunawan AAS (2015) Face expression detection on Kinect using active appearance model and fuzzy logic. Procedia Comput Sci
5.
go back to reference Hickson S, Dufour N, Sud A, Kwatra V, Essa IA (2017) Eyemotion: classifying facial expressions in VR using eye-tracking cameras. arXiv Hickson S, Dufour N, Sud A, Kwatra V, Essa IA (2017) Eyemotion: classifying facial expressions in VR using eye-tracking cameras. arXiv
6.
go back to reference Assari MA, Rahmati M (2011) Driver drowsiness detection using face expression recognition. In: Proceedings of the IEEE international conference on signal and image processing applications, Kuala Lumpur, Malaysia, 16–18 Nov 2011 Assari MA, Rahmati M (2011) Driver drowsiness detection using face expression recognition. In: Proceedings of the IEEE international conference on signal and image processing applications, Kuala Lumpur, Malaysia, 16–18 Nov 2011
7.
go back to reference Chen CH, Lee IJ, Lin LY (2015) Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders. Res Dev Disabil Chen CH, Lee IJ, Lin LY (2015) Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders. Res Dev Disabil
8.
go back to reference Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I, The extended Cohn-Kanade Dataset (CK+): a complete dataset for action unit and emotion- specified expression. In: Proceedings of the IEEE conference on computer vision and Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I, The extended Cohn-Kanade Dataset (CK+): a complete dataset for action unit and emotion- specified expression. In: Proceedings of the IEEE conference on computer vision and
9.
go back to reference Kahou SE, Michalski, V, Konda K (2015) Recurrent neural networks for emotion recognition in video. In: Proceedings of the ACM on international conference on multimodal interaction, Seattle, WA, USA, 9–13 Nov 2015 Kahou SE, Michalski, V, Konda K (2015) Recurrent neural networks for emotion recognition in video. In: Proceedings of the ACM on international conference on multimodal interaction, Seattle, WA, USA, 9–13 Nov 2015
10.
go back to reference Kim DH, Baddar W, Jang J, Ro YM (2017) Multi-objective based Spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans Affect Comput Kim DH, Baddar W, Jang J, Ro YM (2017) Multi-objective based Spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans Affect Comput
11.
go back to reference Ekman P, Friesen WV (1978) Facial action coding system: investigator’s guide, 1st edn. Consulting Psychologists Press, Palo Alto, CA, USA Ekman P, Friesen WV (1978) Facial action coding system: investigator’s guide, 1st edn. Consulting Psychologists Press, Palo Alto, CA, USA
12.
go back to reference Hamm J, Kohler CG, Gur RC, Verma R (2011) Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders. J Neurosci Methods Hamm J, Kohler CG, Gur RC, Verma R (2011) Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders. J Neurosci Methods
13.
go back to reference Jeong M, Kwak SY, Ko BC, Nam JY (2017) Driver facial landmark detection in real driving situation. IEEE Trans Circ Syst Video Technol Jeong M, Kwak SY, Ko BC, Nam JY (2017) Driver facial landmark detection in real driving situation. IEEE Trans Circ Syst Video Technol
14.
go back to reference Tao SY, Martinez AM (2014) Compound facial expressions of emotion. Natl Acad Sci Tao SY, Martinez AM (2014) Compound facial expressions of emotion. Natl Acad Sci
15.
go back to reference Benitez-Quiroz CF, Srinivasan R, Martinez AM (2016) EmotioNet: an accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 26 June–1 July 2016 Benitez-Quiroz CF, Srinivasan R, Martinez AM (2016) EmotioNet: an accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 26 June–1 July 2016
16.
go back to reference Kolakowaska A (2013) A review of emotion recognition methods based on keystroke dynamics and mouse movements. In: Proceedings of the 6th international conference on human system interaction, Gdansk, Poland, 6–8 June 2013 Kolakowaska A (2013) A review of emotion recognition methods based on keystroke dynamics and mouse movements. In: Proceedings of the 6th international conference on human system interaction, Gdansk, Poland, 6–8 June 2013
17.
go back to reference Kumar S (2015) Facial expression recognition: a review. In: Proceedings of the national conference on cloud computing and big data, Shanghai, China, 4–6 Nov 2015 Kumar S (2015) Facial expression recognition: a review. In: Proceedings of the national conference on cloud computing and big data, Shanghai, China, 4–6 Nov 2015
18.
go back to reference Ghayoumi MA (2017) Quick review of deep learning in facial expression. J Commun Comput Ghayoumi MA (2017) Quick review of deep learning in facial expression. J Commun Comput
19.
go back to reference Suk M, Prabhakaran B (2014) Real-time mobile facial expression recognition system—a case study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, Columbus, OH, USA, 24–27 June 2014 Suk M, Prabhakaran B (2014) Real-time mobile facial expression recognition system—a case study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, Columbus, OH, USA, 24–27 June 2014
20.
go back to reference Ghimire D, Lee J (2013) Geometric feature- based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors Ghimire D, Lee J (2013) Geometric feature- based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors
21.
go back to reference Happy SL, George A, Routray A (2012) A real time facial expression classification system using local binary patterns. In: Proceedings of the 4th international conference on intelligent human computer interaction, Kharagpur, India, 27–29 Dec 2012 Happy SL, George A, Routray A (2012) A real time facial expression classification system using local binary patterns. In: Proceedings of the 4th international conference on intelligent human computer interaction, Kharagpur, India, 27–29 Dec 2012
22.
go back to reference Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl
23.
go back to reference Siddiqi MH, Ali R, Khan AM, Park YT, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEETrans Image Proc Siddiqi MH, Ali R, Khan AM, Park YT, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEETrans Image Proc
24.
go back to reference Khan RA, Meyer A, Konik H, Bouakaz S (2013) Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recognit Lett Khan RA, Meyer A, Konik H, Bouakaz S (2013) Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recognit Lett
25.
go back to reference Torre FD, Chu WS, Xiong X, Vicente F, Ding X, Cohn J (2015) IntraFace. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, Ljubljana, Slovenia, 4–8 May 2015 Torre FD, Chu WS, Xiong X, Vicente F, Ding X, Cohn J (2015) IntraFace. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, Ljubljana, Slovenia, 4–8 May 2015
26.
go back to reference Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: Proceedings of the 3rd international conference on crime detection and prevention, London, UK, 3 Dec 2009 Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: Proceedings of the 3rd international conference on crime detection and prevention, London, UK, 3 Dec 2009
27.
go back to reference Mavadati SM, Mahoor MH, Bartlett K, Trinh P, Cohn J (2013) DISFA: a spontaneous facial action intensity database. IEEE Trans Affect Comput Mavadati SM, Mahoor MH, Bartlett K, Trinh P, Cohn J (2013) DISFA: a spontaneous facial action intensity database. IEEE Trans Affect Comput
28.
go back to reference Szwoch M, Pienia˛z˙ek P (2015) Facial emotion recognition using depth data. In: Proceedings of the 8th international conference on human system interactions, Warsaw, Poland, 25–27 June 2015 Szwoch M, Pienia˛z˙ek P (2015) Facial emotion recognition using depth data. In: Proceedings of the 8th international conference on human system interactions, Warsaw, Poland, 25–27 June 2015
29.
go back to reference Maalej A, Amor BB, Daoudi M, Srivastava A, Berretti S (2011) Shape analysis of local facial patches for 3D facial expression recognition. Pattern Recognit Maalej A, Amor BB, Daoudi M, Srivastava A, Berretti S (2011) Shape analysis of local facial patches for 3D facial expression recognition. Pattern Recognit
30.
go back to reference Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In: Proceedings of the international conference on automatic face and gesture recognition, Southampton, UK, 10–12 April 2006 Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In: Proceedings of the international conference on automatic face and gesture recognition, Southampton, UK, 10–12 April 2006
31.
go back to reference Zhao G, Huang X, Taini M, Li SZ, Pietikäinen M (2011) Facial expression recognition from near-infrared videos. Image Vis Comput Zhao G, Huang X, Taini M, Li SZ, Pietikäinen M (2011) Facial expression recognition from near-infrared videos. Image Vis Comput
32.
go back to reference Lyons MJ, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with Gabor wave. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, Nara, Japan, 14– 16 Apr 1998 Lyons MJ, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with Gabor wave. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, Nara, Japan, 14– 16 Apr 1998
35.
go back to reference Walecki R, Rudovic O, Pavlovic V, Schuller B, Pantic M (2017) Deep structured learning for facial action unit intensity estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA, 21–26 July 2017 Walecki R, Rudovic O, Pavlovic V, Schuller B, Pantic M (2017) Deep structured learning for facial action unit intensity estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA, 21–26 July 2017
36.
go back to reference LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput
37.
go back to reference Ko BC, Lee EJ, Nam JY (2016) Genetic algorithm based filter bank design for light convolutional neural network. Adv Sci Lett Ko BC, Lee EJ, Nam JY (2016) Genetic algorithm based filter bank design for light convolutional neural network. Adv Sci Lett
38.
go back to reference Breuer R, Kimmel R (2017) A deep learning perspective on the origin of facial expressions Breuer R, Kimmel R (2017) A deep learning perspective on the origin of facial expressions
39.
go back to reference Jung H, Lee S, Yim J, Park S, Kim J (2015) Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE international conference on computer vision, Santiago, Chile, 7–12 Dec 2015 Jung H, Lee S, Yim J, Park S, Kim J (2015) Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE international conference on computer vision, Santiago, Chile, 7–12 Dec 2015
40.
go back to reference Zhao K, Chu WS, Zhang H (2016) Deep region and multi-label learning for facial action unit detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 26 June–1 July 2016 Zhao K, Chu WS, Zhang H (2016) Deep region and multi-label learning for facial action unit detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 26 June–1 July 2016
41.
go back to reference Shen P, Wang S, Liu Z (2013) Facial expression recognition from infrared thermal videos. Intell Auton Syst Shen P, Wang S, Liu Z (2013) Facial expression recognition from infrared thermal videos. Intell Auton Syst
42.
go back to reference Wei W, Jia Q, Chen G (2016) Real-time facial expression recognition for affective computing based on Kinect. In: Proceedings of the IEEE 11th conference on industrial electronics and applications, Hefei, China, 5–7 June 2016 Wei W, Jia Q, Chen G (2016) Real-time facial expression recognition for affective computing based on Kinect. In: Proceedings of the IEEE 11th conference on industrial electronics and applications, Hefei, China, 5–7 June 2016
43.
go back to reference Tian Y, Luo P, Luo X, Wang X, Tang X (2015) Pedestrian detection aided by deep learning semantic tasks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA, 8–10 June 2015 Tian Y, Luo P, Luo X, Wang X, Tang X (2015) Pedestrian detection aided by deep learning semantic tasks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA, 8–10 June 2015
44.
go back to reference Deshmukh S, Patwardhan M, Mahajan A (2016) Survey on real-time facial expression recognition techniques. IET Biom Deshmukh S, Patwardhan M, Mahajan A (2016) Survey on real-time facial expression recognition techniques. IET Biom
45.
go back to reference Dornaika F, Raducanu B (2007) Efficient facial expression recognition for human robot interaction. In: Proceedings of the 9th international work-conference on artificial neural networks on computational and ambient intelligence, San Sebastián, Spain, 20–22 June Dornaika F, Raducanu B (2007) Efficient facial expression recognition for human robot interaction. In: Proceedings of the 9th international work-conference on artificial neural networks on computational and ambient intelligence, San Sebastián, Spain, 20–22 June
46.
go back to reference Bartneck C, Lyons MJ (2007) HCI and the face: towards an art of the soluble. In: Proceedings of the international conference on human- computer interaction: interaction design and usability, Beijing, China, 22–27 July 2007 Bartneck C, Lyons MJ (2007) HCI and the face: towards an art of the soluble. In: Proceedings of the international conference on human- computer interaction: interaction design and usability, Beijing, China, 22–27 July 2007
47.
go back to reference Zhan C, Li W, Ogunbona P, Safaei F (2008) A real-time facial expression recognition system for online games. Int J Comput Games Technol Zhan C, Li W, Ogunbona P, Safaei F (2008) A real-time facial expression recognition system for online games. Int J Comput Games Technol
48.
go back to reference Mourão A, Magalhães J (2013) Competitive affective gaming: winning with a smile. In: Proceedings of the ACM international conference on multimedia, Barcelona, Spain, 21–25 Oct 2013 Mourão A, Magalhães J (2013) Competitive affective gaming: winning with a smile. In: Proceedings of the ACM international conference on multimedia, Barcelona, Spain, 21–25 Oct 2013
49.
go back to reference Sandbach G, Zafeiriou S, Pantic M, Yin L (2012) Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis Comput Sandbach G, Zafeiriou S, Pantic M, Yin L (2012) Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis Comput
Metadata
Title
A Review of Survey and Assessment of Facial Emotion Recognition (FER) by Convolutional Neural Networks
Authors
Sanyam Agarwal
Veer Daksh Agarwal
Ishaan Agarwal
Vipin Mittal
Lakshay Singla
Ahmed Hussein Alkhayyat
Copyright Year
2024
Publisher
Springer Nature Singapore
DOI
https://doi.org/10.1007/978-981-99-9562-2_63