Top

Cognitive Neurodynamics

22-02-2023 | Research Article

Machine to brain: facial expression recognition using brain machine generative adversarial networks

Authors: Dongjun Liu, Jin Cui, Zeyu Pan, Hangkui Zhang, Jianting Cao, Wanzeng Kong

Published in: Cognitive Neurodynamics

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The human brain can effectively perform Facial Expression Recognition (FER) with a few samples by utilizing its cognitive ability. However, unlike the human brain, even the well-trained deep neural network is data-dependent and lacks cognitive ability. To tackle this challenge, this paper proposes a novel framework, Brain Machine Generative Adversarial Networks (BM-GAN), which utilizes the concept of brain’s cognitive ability to guide a Convolutional Neural Network to generate LIKE-electroencephalograph (EEG) features. More specifically, we firstly obtain EEG signals triggered from facial emotion images, then we adopt BM-GAN to carry out the mutual generation of image visual features and EEG cognitive features. BM-GAN intends to use the cognitive knowledge learnt from EEG signals to instruct the model to perceive LIKE-EEG features. Thereby, BM-GAN has a superior performance for FER like the human brain. The proposed model consists of VisualNet, EEGNet, and BM-GAN. More specifically, VisualNet can obtain image visual features from facial emotion images and EEGNet can obtain EEG cognitive features from EEG signals. Subsequently, the BM-GAN completes the mutual generation of image visual features and EEG cognitive features. Finally, the predicted LIKE-EEG features of test images are used for FER. After learning, without the participation of the EEG signals, an average classification accuracy of 96.6 % is obtained on Chinese Facial Affective Picture System dataset using LIKE-EEG features for FER. Experiments demonstrate that the proposed method can produce an excellent performance for FER.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Aouayeb M, Hamidouche W, Soladié C, Kpalma K, Séguier R (2021)“Learning vision transformer with squeeze and excitation for facial expression recognition,” CoRR, vol. abs/2107.03107

Assari MA, Rahmati M (2011) “Driver drowsiness detection using face expression recognition,” in 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), pp. 337–341, IEEE

Barkau RL (1996)“Unet: One-dimensional unsteady flow through a full network of open channels. user’s manual,” tech. rep., Hydrologic Engineering Center Davis CA

Bartneck C, Lyons MJ, (2007)“Hci and the face: Towards an art of the soluble,” in International Conference on Human-computer Interaction, pp. 20–29, Springer

Belgiu M, Drăguţ L (2016) Random forest in remote sensing: A review of applications and future directions. ISPRS journal of photogrammetry and remote sensing 114:24–31CrossRef

Chatrian GE, Lettich E, Nelson PL (1985) Ten percent electrode system for topographic studies of spontaneous and evoked eeg activities. American Journal of EEG technology 25(2):83–92CrossRef

Chen Z, Liu G (2010) Application of eeg signal in emotion recognition. Computer Engineering 36(9):168–170

Cogswell M, Ahmed F, Girshick R, Zitnick L, Batra D (2015) “Reducing overfitting in deep networks by decorrelating representations,” arXiv preprint arXiv:1511.06068

Darwin C (2015) The expression of the emotions in man and animals. University of Chicago press

Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee

Dhavalikar AS, Kulkarni R (2014) “Face detection and facial expression recognition system,” in 2014 International Conference on Electronics and Communication Systems (ICECS), pp. 1–7, IEEE

Dornaika F, Raducanu B, (2007)“Efficient facial expression recognition for human robot interaction,” in International Work-Conference on Artificial Neural Networks, pp. 700–708, Springer

Fix E, Hodges JL (1989) Discriminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Review/Revue Internationale de Statistique 57(3):238–247

Fong RC, Scheirer WJ, Cox DD (2018) Using human brain activity to guide machine learning. Scientific reports 8(1):1–10CrossRef

Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) “Generative adversarial nets,” Advances in neural information processing systems, vol. 27,

Happy S, Routray A (2014) Automatic facial expression recognition using features of salient facial patches. IEEE transactions on Affective Computing 6(1):1–12CrossRef

Harper ER, Rodden T, Rogers Y, Sellen A, Human B, et al. (2008) “Human-computer interaction in the year 2020,”

Hartmann KG, Schirrmeister RT, Ball T (2018) “Eeg-gan: Generative adversarial networks for electroencephalograhic (eeg) brain signals,”

He K, Zhang X, Ren S, Sun J (2016) “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778

Horlings R, Datcu D, Rothkrantz LJ (2008)“Emotion recognition using brain activity,” in Proceedings of the 9th international conference on computer systems and technologies and workshop for PhD students in computing, pp. II–1

Howard A, Sandler M, Chu G, Chen LC, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V, et al. (2019) “Searching for mobilenetv3,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324

Hwang S, Hong K, Son G, Byun H (2019) “Ezsl-gan: Eeg-based zero-shot learning approach using a generative adversarial network,” in 2019 7th International Winter Conference on Brain-Computer Interface (BCI)

Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016)“Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size,” arXiv preprint arXiv:1602.07360

Iandola F, Moskewicz M, Karayev S, Girshick R, Darrell T, Keutzer K (2014) “Densenet: Implementing efficient convnet descriptor pyramids,” arXiv preprint arXiv:1404.1869

Jebelli H, Hwang S, Lee S (2018) Eeg signal-processing framework to obtain high-quality brain waves from an off-the-shelf wearable eeg device. Journal of Computing in Civil Engineering 32(1):04017070CrossRef

Khosla A, Khandnor P, Chand T (2020) A comparative analysis of signal processing and classification methods for different applications based on eeg signals. Biocybernetics and Biomedical Engineering 40(2):649–690CrossRef

Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25:1097–1105

Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ (2018) Eegnet: a compact convolutional neural network for eeg-based brain-computer interfaces. Journal of neural engineering 15(5):056013CrossRefPubMed

Lebedev MA, Nicolelis MA (2006) Brain-machine interfaces: past, present and future. TRENDS in Neurosciences 29(9):536–546CrossRefPubMed

LeCun Y, Bengio Y, Hinton G (2015) “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444

Li S, Deng W, (2020)“Deep facial expression recognition: A survey,” IEEE transactions on affective computing

Ling W, Kong W, Long Y, Yang C, Jin X, (2019)“Facial emotion recognition based on brain and machine collaborative intelligence,” in 2019 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA), pp. 1–5, IEEE

Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) “Coding facial expressions with gabor wavelets,” in Proceedings Third IEEE international conference on automatic face and gesture recognition, pp. 200–205, IEEE

Ma N, Zhang, Zheng HT, Sun J (2018) “Shufflenet v2: Practical guidelines for efficient cnn architecture design,” in Proceedings of the European conference on computer vision (ECCV), pp. 116–131

Minaee S, Minaei M, Abdolrashidi A (2021) “Deep-emotion: Facial expression recognition using attentional convolutional network,” Sensors, vol. 21, no. 9

Money AG, Agius H (2010) “Elvis: entertainment-led video summaries,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 6, no. 3, pp. 1–30

Moridian P, Shoeibi A, Khodatars M, Jafari M, Pachori RB, Khadem A, Alizadehsani R, Ling SH (2022) Automatic diagnosis of sleep apnea from biomedical signals using artificial intelligence techniques: Methods, challenges, and future works. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 12(6):e1478

Mourão A, Magalhães J, (2013)“Competitive affective gaming: winning with a smile,” in Proceedings of the 21st ACM international conference on Multimedia, pp. 83–92

Nishida S, Nakano Y, Blanc A, Maeda N, Kado M, Nishimoto S (2020) Brain-mediated transfer learning of convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence 34:5281–5288CrossRef

Palazzo S, Rundo F, Battiato S, Giordano D, Spampinato C (2020) “Visual saliency detection guided by neural signals,” in 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp. 525–531, IEEE

Pramerdorfer C, Kampel M, (2016) “Facial expression recognition using convolutional neural networks: State of the art,” CoRR, vol. abs/1612.02903

Rawat W, Wang Z (2017) Deep convolutional neural networks for image classification: A comprehensive review. Neural Computation 29(9):2352–2449CrossRefPubMed

Sajjad M, Zahir S, Ullah A, Akhtar Z, Muhammad K (2020) Human behavior understanding in big multimedia data using cnn based facial expression recognition. Mobile networks and applications 25(4):1611–1621CrossRef

Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510–4520

Schulz M-A, Yeo BT, Vogelstein JT, Mourao-Miranada J, Kather JN, Kording K, Richards B, Bzdok D (2020) Different scaling of linear models and deep learning in ukbiobank brain images versus machine-learning datasets. Nature communications 11(1):1–15CrossRef

Shoeibi A, Ghassemi N, Khodatars M, Moridian P, Khosravi A, Zare A, Gorriz JM, Chale-Chale AH, Khadem A, Rajendra Acharya U (2022) “Automatic diagnosis of schizophrenia and attention deficit hyperactivity disorder in rs-fmri modality using convolutional autoencoder model and interval type-2 fuzzy regression,” Cognitive Neurodynamics, pp. 1–23

Shoeibi A, Rezaei M, Ghassemi N, Namadchian Z, Zare A, Gorriz JM (2022)“Automatic diagnosis of schizophrenia in eeg signals using functional connectivity features and cnn-lstm model,” in International Work-Conference on the Interplay Between Natural and Artificial Computation, pp. 63–73, Springer

Simonyan K, Zisserman A (2014) “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556

Soleymani M, Asghari-Esfeden S, Fu Y, Pantic M (2016) Analysis of eeg signals and facial expressions for continuous emotion detection. IEEE Transactions on Affective Computing 7(1):17–28CrossRef

Soleymani M, Kierkels JJ, Chanel G, Pun T (2009) “A bayesian framework for video affective representation,” in 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, pp. 1–7, IEEE

Soleymani M, Pantic M, (2013) “Multimedia implicit tagging using eeg signals,” in 2013 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6, IEEE

Spampinato C, Palazzo S, Kavasidis I, Giordano D, Souly N, Shah M (2017) “Deep learning human mind for automated visual classification,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6809–6817

Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9

Tan M, Chen B, Pang R, Vasudevan V, Sandler M, Howard A, Le QV (2019) “Mnasnet: Platform-aware neural architecture search for mobile,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2820–2828

Tian Y-I, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Transactions on pattern analysis and machine intelligence 23(2):97–115CrossRefPubMedPubMedCentral

van der Maaten L, Hinton G (2008) Visualizing data using t-sne. Journal of Machine Learning Research 9(86):2579–2605

Xie S, Girshick R, Dollár P, Tu Z, He K (2017) “Aggregated residual transformations for deep neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492–1500

Xinying L, Xuebing L et al (2006) Effects of emotion on cognitive processing: Series of event-related potentials study. Advances in Psychological Science 14(04):505

Yang H, Ciftci U, Yin L, (2018) “Facial expression recognition by de-expression residue learning,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2168–2177

Yolcu G, Oztel I, Kazan S, Oz C, Palaniappan K, Lever TE, Bunyak F (2017) “Deep learning-based facial expression recognition for monitoring neurological disorders,” in 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1652–1657, IEEE

Zagoruyko S, Komodakis N (2016) “Wide residual networks,” arXiv preprint arXiv:1605.07146

Zhan C, Li W, Ogunbona P, Safaei F (2008) “A real-time facial expression recognition system for online games,” International Journal of Computer Games Technology, vol. 2008

Zhu JY, Park T, Isola P, Efros AA, (2017)“Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, pp. 2223–2232

Title: Machine to brain: facial expression recognition using brain machine generative adversarial networks
Authors: Dongjun Liu
Jin Cui
Zeyu Pan
Hangkui Zhang
Jianting Cao
Wanzeng Kong
Publication date: 22-02-2023
Publisher: Springer Netherlands
Published in: Cognitive Neurodynamics
Print ISSN: 1871-4080
Electronic ISSN: 1871-4099
DOI: https://doi.org/10.1007/s11571-023-09946-y

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"