nach oben

Erschienen in:

28.04.2023 | Data analytics and machine learning

Automated facial expression recognition using exemplar hybrid deep feature generation technique

verfasst von: Mehmet Baygin, Ilknur Tuncer, Sengul Dogan, Prabal Datta Barua, Turker Tuncer, Kang Hao Cheong, U. Rajendra Acharya

Erschienen in: Soft Computing | Ausgabe 13/2023

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

The perception and recognition of emotional expressions provide essential information about individuals’ social behavior. Therefore, decoding emotional expressions is very important. Facial expression recognition (FER) is one of the most frequently studied topics. An accurate FER model has four prime phases. (i) Facial areas are segmented from the face images. (ii) An exemplar deep feature-based model is proposed. Two pretrained deep models (AlexNet and MobileNetV2) are utilized as feature generators. By merging both pretrained networks, a feature generation function is presented. (iii) The most valuable 1000 features are selected by neighborhood component analysis (NCA). (iv) These 1000 features are selected on a support vector machine (SVM). We have developed our model using five FER corpora: TFEID, JAFFE, KDEF, CK+, and Oulu-CASIA. Our developed model is able to yield an accuracy of 97.01, 98.59, 96.54, 100, and 100%, using TFEID, JAFFE, KDEF, CK+, and Oulu-CASIA, respectively. The results obtained in this study showed that the proposed exemplar deep feature extraction approach has obtained high success rates in the automatic FER method using various databases.

Vorheriger Artikel The CNN-GRU model with frequency analysis module for sea surface temperature prediction

Nächster Artikel Artificial intelligence inspired framework for preventing sexual violence at public toilets of educational institutions with the improvisation of gender recognition from gait sequences

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Ahmed N, Al Aghbari Z, Girija S (2023) A systematic survey on multimodal emotion recognition using learning algorithms. Intell Syst Appl 17:200171

Akhand MAH, Roy S, Siddique N, Kamal MAS, Shimamura T (2021) Facial emotion recognition using transfer learning in the deep CNN. Electronics 10:1036CrossRef

Arul Vinayakam Rajasimman M, Manoharan RK, Subramani N, Aridoss M, Galety MG (2023) Robust facial expression recognition using an evolutionary algorithm with a deep learning model. Appl Sci 13:468CrossRef

Canal FZ, Müller TR, Matias JC, Scotton GG, de Sa Junior AR, Pozzebon E et al (2022) A survey on facial emotion recognition techniques: a state-of-the-art literature review. Inf Sci 582:593–617CrossRef

Celniak W, Augustyniak P (2022) Eye-tracking as a component of multimodal emotion recognition systems. In: International conference on information technologies in biomedicine. Springer, pp 66–75

Cha H-S, Im C-H (2022) Performance enhancement of facial electromyogram-based facial-expression recognition for social virtual reality applications using linear discriminant analysis adaptation. Virtual Real 26:385–398CrossRef

Chen L-F, Yen Y-S (2007) Taiwanese facial expression image database. Brain Mapp Lab Inst Brain Sci Natl Yang-Ming Univ Taipei, Taiwan

Chowdary MK, Nguyen TN, Hemanth DJ (2021) Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Comput Appl. https://doi.org/10.1007/S00521-021-06012-8CrossRef

Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255. IEEE

Ding C, Peng H (2005) Minimum redundancy feature selection from microarray gene expression data. J Bioinform Comput Biol 3:185–205CrossRef

Dzedzickis A, Kaklauskas A, Bucinskas V (2020) Human emotion recognition: review of sensors and methods. Sensors 20:592CrossRef

Ekman P (1973) Cross-cultural studies of facial expression. Darwin and facial expression: a century of research in review. Academic Press, New York, pp 169–222

Eng S, Ali H, Cheah A, Chong Y (2091) Facial expression recognition in JAFFE and KDEF Datasets using histogram of oriented gradients and support vector machine. In: IOP conference series: materials science and engineering. IOP Publishing, p 012031

Farajzadeh N, Hashemzadeh M (2018) Exemplar-based facial expression recognition. Inf Sci 460:318–330CrossRef

Febrian R, Halim BM, Christina M, Ramdhan D, Chowanda A (2023) Facial expression recognition using bidirectional LSTM-CNN. Procedia Comput Sci 216:39–47CrossRef

Foggia P, Greco A, Saggese A, Vento M (2023) Multi-task learning on the edge for effective gender, age, ethnicity and emotion recognition. Eng Appl Artif Intell 118:105651CrossRef

Gao H, Wu M, Chen Z, Li Y, Wang X, An S et al (2023) SSA-ICL: multi-domain adaptive attention with intra-dataset continual learning for Facial expression recognition. Neural Netw 158:228–238CrossRef

Geiger M, Wilhelm O (2023) Computerized facial emotion expression recognition. Digital phenotyping and mobile sensing: new developments in psychoinformatics. Springer, Cham, pp 43–56CrossRef

Ghosh S, Priyankar A, Ekbal A, Bhattacharyya P (2023) Multitasking of sentiment detection and emotion recognition in code-mixed Hinglish data. Knowl Based Syst 260:110182CrossRef

Gil S, Le Bigot L (2023) Emotional face recognition when a colored mask is worn: a cross-sectional study. Sci Rep 13:1–15CrossRef

Goldberger J, Hinton GE, Roweis S, Salakhutdinov RR (2004) Neighbourhood components analysis. Adv Neural Inf Process Syst 17:513–520

Goodfellow IJ, Erhan D, Luc Carrier P et al (2015) Challenges in representation learning: a report on three machine learning contests. Neural Networks 64:59–63. https://doi.org/10.1016/j.neunet.2014.09.005

Jupalli TK, Reddy MST, Kondaveeti HK (2023) Artificial intelligence in higher education. Mobile and sensor-based technologies in higher education. IGI Global, pp 1–30

Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proc - 4th IEEE Int Conf Autom Face Gesture Recognition, FG 2000, pp 46–53. https://doi.org/10.1109/AFGR.2000.840611

Kas M, Ruichek Y, Messoussi R (2021) New framework for person-independent facial expression recognition combining textural and shape analysis through new feature extraction approach. Inf Sci 549:200–220MathSciNetCrossRef

Kavitha M, RajivKannan A (2023) Hybrid convolutional neural network and long short-term memory approach for facial expression recognition. Intell Autom Soft Comput 35:689–704CrossRef

Khattak A, Asghar MZ, Ali M, Batool U (2022) An efficient deep learning technique for facial emotion recognition. Multimed Tools Appl 81:1649–1683CrossRef

Kononenko I (1994) Estimating attributes: analysis and extensions of RELIEF. In: European conference on machine learning, Springer, pp 171–182

Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

Kumari N, Bhatia R (2022) Efficient facial emotion recognition model using deep convolutional neural network and modified joint trilateral filter. Soft Comput. https://doi.org/10.21203/rs.3.rs-866042/v1CrossRef

Li S, Deng W, Du J (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2852–2861

Liu H, Setiono R (1995) Chi2: Feature selection and discretization of numeric attributes. In: Proceedings of 7th IEEE international conference on tools with artificial intelligence, IEEE, pp 388–391

Liu W-L, Gong Y-J, Chen W-N, Liu Z, Wang H, Zhang J (2019) Coordinated charging scheduling of electric vehicles: a mixed-variable differential evolution approach. IEEE Trans Intell Transp Syst 21:5094–5109CrossRef

Liu Y, Zeng J, Shan S, Zheng Z (2018) Multi-channel pose-aware convolution neural networks for multi-view facial expression recognition. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp 458–465. IEEE

Liu S, Gao P, Li Y, Fu W, Ding W (2023) Multi-modal fusion network with complementarity and importance for emotion recognition. Inf Sci 619:679–694CrossRef

Lucey P, Cohn JF, Kanade T et al (2010) The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Comput Soc Conf Comput Vis Pattern Recognit - Work CVPRW 2010, pp 94–101. https://doi.org/10.1109/CVPRW.2010.5543262

Lundqvist D, Flykt A, Ohman A (1998) The Karolinska directed emotional faces (KDEF). CD ROM from Dep Clin Neurosci Psychol Sect Karolinska Institutet 2–2

Lyons MJ (2021) Excavating AI Re-excavated: Debunking a Fallacious Account of the JAFFE Dataset. arXiv preprint http://arxiv.org/abs/2107.13998

Lyons MJ, Kamachi M, Gyoba J (2020) Coding facial expressions with Gabor wavelets (IVC special issue). arXiv preprint http://arxiv.org/abs/arXiv:2009.05938

Nikolaus M, Fourtassi A (2023) Communicative feedback in language acquisition. New Ideas Psychol 68:100985CrossRef

Othman E, Werner P, Saxen F, Al-Hamadi A, Gruss S, Walter S (2023) Classification networks for continuous automatic pain intensity monitoring in video using facial expression on the X-ITE Pain Database. J vis Commun Image Represent 91:103743CrossRef

Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. IEEE Int Conf Multimed Expo, ICME 2005:317–321. https://doi.org/10.1109/ICME.2005.1521424

Peng H, Long F, Ding C (2005) Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Machine Intell 27:1226–1238

Porcu S, Floris A, Atzori L (2020) Evaluation of data augmentation techniques for facial expression recognition systems. Electronics 9:1892CrossRef

Robnik-Šikonja M, Kononenko I (2003) Theoretical and empirical analysis of ReliefF and RReliefF. Mach Learn 53:23–69CrossRef

Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L-C (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–20

Shen J, Yang H, Li J, Cheng Z (2022) Assessing learning engagement based on facial expression recognition in MOOC’s scenario. Multimed Syst 28:469–478CrossRef

Sun Z, Chiong R, Hu Z-P (2020) Self-adaptive feature learning based on a priori knowledge for facial expression recognition. Knowl Based Syst 204:106124CrossRef

Tang Y, Zhang X, Hu X, Wang S, Wang H (2020) Facial expression recognition using frequency neural network. IEEE Trans Image Process 30:444–457CrossRef

Vapnik V (1998) The support vector method of function estimation. Nonlinear Modeling: advanced black-box techniques. Springer, pp 55–85CrossRef

Vedantham R, Reddy ES (2020) A robust feature extraction with optimized DBN-SMO for facial expression recognition. Multimed Tools Appl 79:21487–21512CrossRef

Wang Y, Song W, Tao W, Liotta A, Yang D, Li X et al (2022) A systematic review on affective computing: emotion models, databases, and recent advances. Inf Fus. https://doi.org/10.48550/arXiv.2203.06935CrossRef

Wani AH, Hashmy R (2023) A supervised multinomial classification framework for emotion recognition in textual social data. Int J Adv Intell Paradig 24:173–189

Yang W, Wang K, Zuo W (2012) Neighborhood component feature selection for high-dimensional data. J Comput 7:161–168CrossRef

Yin L, Wei X, Sun Y et al (2006) A 3D facial expression database for facial behavior research. FGR 2006 Proc 7th Int Conf Autom Face Gesture Recognit 2006:211–216. https://doi.org/10.1109/FGR.2006.6

Zhang Z, Luo P, Loy CC, Tang X (2018) From facial expression recognition to interpersonal relation prediction. Int J Comput Vis 126:550–569. https://doi.org/10.1007/s11263-017-1055-1

Zhao G, Huang X, Taini M et al (2011) Facial expression recognition from near-infrared videos. Image Vis Comput 29:607–619. https://doi.org/10.1016/j.imavis.2011.07.002

Zhao F, Di S, Wang L (2022a) A hyperheuristic with q-learning for the multiobjective energy-efficient distributed blocking flow shop scheduling problem. IEEE Trans Cybern

Zhao F, Hu X, Wang L, Zhao J, Tang J (2022b) A reinforcement learning brain storm optimization algorithm (BSO) with learning mechanism. Knowl Based Syst 235:107645CrossRef

Zhen R, Song W, He Q, Cao J, Shi L, Luo J (2023) Human-computer interaction system: a survey of talking-head generation. Electronics 12:218CrossRef

Zhou S, Xing L, Zheng X, Du N, Wang L, Zhang Q (2019) A self-adaptive differential evolution algorithm for scheduling a single batch-processing machine with arbitrary job sizes and release times. IEEE Trans Cybern 51:1430–1442CrossRef

Titel: Automated facial expression recognition using exemplar hybrid deep feature generation technique
verfasst von: Mehmet Baygin
Ilknur Tuncer
Sengul Dogan
Prabal Datta Barua
Turker Tuncer
Kang Hao Cheong
U. Rajendra Acharya
Publikationsdatum: 28.04.2023
Verlag: Springer Berlin Heidelberg
Erschienen in: Soft Computing / Ausgabe 13/2023
Print ISSN: 1432-7643
Elektronische ISSN: 1433-7479
DOI: https://doi.org/10.1007/s00500-023-08230-9

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Weitere Artikel der Ausgabe 13/2023

Time-dependent reliability calculation method of RC bridges based on the dual neural network

App interaction design in the pop music singing teaching system based on differential evolution algorithm

Applications of fuzzy conformable Laplace transforms for solving fuzzy conformable differential equations

A shared libraries aware and bank partitioning-based mechanism for multicore architecture

Oscillation criteria for fractional differential equations with a distributed delay

A comprehensive comparison of accuracy-based fitness functions of metaheuristics for feature selection

Premium Partner