Skip to main content
Top
Published in:

08-12-2022

Accurate Emotion Recognition Utilizing Extracted EEG Sources as Graph Neural Network Nodes

Authors: Shiva Asadzadeh, Tohid Yousefi Rezaii, Soosan Beheshti, Saeed Meshgini

Published in: Cognitive Computation | Issue 1/2023

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Automated analysis and recognition of human emotion play an important role in the development of a human–computer interface. High temporal resolution of EEG signals enables us to noninvasively study the emotional brain activities. However, one major obstacle in this procedure is extracting the essential information in presence of the low spatial resolution of EEG recordings. The pattern of each emotion is clearly defined by mapping from scalp sensors to brain sources using the standardized low-resolution electromagnetic tomography (sLORETA) method. A graph neural network (GNN) is then used for EEG-based emotion recognition in which sLORETA sources are considered as the nodes of the underlying graph. In the proposed method, the inter-source relations in EEG source signals are encoded in the adjacency matrix of GNN. Finally, the labels of the unseen emotions are recognized using a GNN classifier. The experiments on the recorded EEG dataset by inducing excitement through music (recorded in brain-computer interface research lab, University of Tabriz) indicate that the brain source activity modeling by ESB-G3N significantly improves the accuracy of emotion recognition. Experimental results show classification accuracy of 98.35% for two-class classification of positive and negative emotions. In this paper, we concentrate on extracting active emotional cortical sources using EEG source imaging (ESI) techniques. Auditory stimuli are used to rapidly and efficiently induce emotions in participants (visual stimuli in terms of video/image are either slow or inefficient in inducing emotions). We propose the use of active EEG sources as graph nodes by EEG source-based GNN node (ESB-G3N) algorithm. The results show an absolute improvement of 1–2% over subject-dependent and subject-independent scenarions compared to the existing approaches.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Johnson WR. A study of the emotions of college athletes: Boston University; 1950.132 pages. Johnson WR. A study of the emotions of college athletes: Boston University; 1950.132 pages.
2.
go back to reference Ngai WK, Xie H, Zou D, Chou KL. Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources. Inf Fusion. 2022;77:107–17.CrossRef Ngai WK, Xie H, Zou D, Chou KL. Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources. Inf Fusion. 2022;77:107–17.CrossRef
3.
go back to reference Kreibig SD. Autonomic nervous system activity in emotion: a review. Biol Psychol. 2010;84(3):394–421.CrossRef Kreibig SD. Autonomic nervous system activity in emotion: a review. Biol Psychol. 2010;84(3):394–421.CrossRef
4.
go back to reference R. Williams D, Williams-Morris R. Racism and mental health: The African American experience. Ethn Health. 2000;5(3–4):243–68. R. Williams D, Williams-Morris R. Racism and mental health: The African American experience. Ethn Health. 2000;5(3–4):243–68.
5.
go back to reference Rotton J, Frey J. Air pollution, weather, and violent crimes: concomitant time-series analysis of archival data. J Pers Soc Psychol. 1985;49(5):1207.CrossRef Rotton J, Frey J. Air pollution, weather, and violent crimes: concomitant time-series analysis of archival data. J Pers Soc Psychol. 1985;49(5):1207.CrossRef
6.
go back to reference Sanei S. Adaptive processing of brain signals: John Wiley & Sons; 2013. Sanei S. Adaptive processing of brain signals: John Wiley & Sons; 2013.
7.
go back to reference Baillet S, Riera J, Marin G, Mangin J, Aubert J, Garnero L. Evaluation of inverse methods and head models for EEG source localization using a human skull phantom. Phys Med Biol. 2001;46(1):77–96.CrossRef Baillet S, Riera J, Marin G, Mangin J, Aubert J, Garnero L. Evaluation of inverse methods and head models for EEG source localization using a human skull phantom. Phys Med Biol. 2001;46(1):77–96.CrossRef
8.
go back to reference Hämäläinen M, Hari R, Ilmoniemi RJ, Knuutila J, Lounasmaa OV. Magnetoencephalography—theory, instrumentation, and applications to noninvasive studies of the working human brain. Rev Mod Phys. 1993;65(2):413.CrossRef Hämäläinen M, Hari R, Ilmoniemi RJ, Knuutila J, Lounasmaa OV. Magnetoencephalography—theory, instrumentation, and applications to noninvasive studies of the working human brain. Rev Mod Phys. 1993;65(2):413.CrossRef
9.
go back to reference Michel CM, Murray MM. Towards the utilization of EEG as a brain imaging tool. Neuroimage. 2012;61(2):371–85.CrossRef Michel CM, Murray MM. Towards the utilization of EEG as a brain imaging tool. Neuroimage. 2012;61(2):371–85.CrossRef
10.
go back to reference da Silva FL. EEG and MEG: relevance to neuroscience. Neuron. 2013;80(5):1112–28.CrossRef da Silva FL. EEG and MEG: relevance to neuroscience. Neuron. 2013;80(5):1112–28.CrossRef
11.
go back to reference Hu J, Tian J, Pan X, Liu J, editors. A comparison between EEG source localization and fMRI during the processing of emotional visual stimuli. Medical Imaging 2007: Physiology, Function, and Structure from Medical Images. Int Soc Opt Photonics. 2007. Hu J, Tian J, Pan X, Liu J, editors. A comparison between EEG source localization and fMRI during the processing of emotional visual stimuli. Medical Imaging 2007: Physiology, Function, and Structure from Medical Images. Int Soc Opt Photonics. 2007.
12.
go back to reference Batabyal T, Muthukrishnan S, Sharma R, Tayade P, Kaur S. Neural substrates of emotional interference: A quantitative EEG study. Neurosci Lett. 2018;685:1–6.CrossRef Batabyal T, Muthukrishnan S, Sharma R, Tayade P, Kaur S. Neural substrates of emotional interference: A quantitative EEG study. Neurosci Lett. 2018;685:1–6.CrossRef
13.
go back to reference Ekman P. Are there basic emotions? Psychol Rev. 1992; 99(3), 550–553. Ekman P. Are there basic emotions? Psychol Rev. 1992; 99(3), 550–553.
14.
go back to reference Tsolaki AC, Kosmidou VE, Kompatsiaris IY, Papadaniil C, Hadjileontiadis L, Tsolaki M. Age-induced differences in brain neural activation elicited by visual emotional stimuli: A high-density EEG study. Neuroscience. 2017;340:268–78.CrossRef Tsolaki AC, Kosmidou VE, Kompatsiaris IY, Papadaniil C, Hadjileontiadis L, Tsolaki M. Age-induced differences in brain neural activation elicited by visual emotional stimuli: A high-density EEG study. Neuroscience. 2017;340:268–78.CrossRef
16.
go back to reference Goshvarpour A, Goshvarpour A. EEG spectral powers and source localization in depressing, sad, and fun music videos focusing on gender differences. Cogn Neurodyn. 2019;13(2):161–73.CrossRef Goshvarpour A, Goshvarpour A. EEG spectral powers and source localization in depressing, sad, and fun music videos focusing on gender differences. Cogn Neurodyn. 2019;13(2):161–73.CrossRef
17.
go back to reference Isotani T, Lehmann D, Pascual-Marqui RD, Fukushima M, Saito N, Yagyu T, et al. editors. Source localization of brain electric activity during positive, neutral and negative emotional states. International Congress Series. Elsevier. 2002. Isotani T, Lehmann D, Pascual-Marqui RD, Fukushima M, Saito N, Yagyu T, et al. editors. Source localization of brain electric activity during positive, neutral and negative emotional states. International Congress Series. Elsevier. 2002.
18.
go back to reference Pizzagalli D, Koenig T, Regard M, Lehmann D. Faces and emotions: brain electric field sources during covert emotional processing. Neuropsychologia. 1998;36(4):323–32.CrossRef Pizzagalli D, Koenig T, Regard M, Lehmann D. Faces and emotions: brain electric field sources during covert emotional processing. Neuropsychologia. 1998;36(4):323–32.CrossRef
19.
go back to reference Vogel H, Szondi L. Lehrbuch der experimentellen Triebdiagnostik. Textband, 2. völlig umgearbeitete Auflage. Bern und Stuttgart Hans Hu. Psyche. 1960;14(8):860–1. Vogel H, Szondi L. Lehrbuch der experimentellen Triebdiagnostik. Textband, 2. völlig umgearbeitete Auflage. Bern und Stuttgart Hans Hu. Psyche. 1960;14(8):860–1.
20.
go back to reference Pourtois G, Delplanque S, Michel C, Vuilleumier P. Beyond conventional event-related brain potential (ERP): exploring the time-course of visual emotion processing using topographic and principal component analyses. Brain Topogr. 2008;20(4):265–77.CrossRef Pourtois G, Delplanque S, Michel C, Vuilleumier P. Beyond conventional event-related brain potential (ERP): exploring the time-course of visual emotion processing using topographic and principal component analyses. Brain Topogr. 2008;20(4):265–77.CrossRef
21.
go back to reference Jäncke L, Alahmadi N. Detection of independent functional networks during music listening using electroencephalogram and sLORETA-ICA. NeuroReport. 2016;27(6):455–61.CrossRef Jäncke L, Alahmadi N. Detection of independent functional networks during music listening using electroencephalogram and sLORETA-ICA. NeuroReport. 2016;27(6):455–61.CrossRef
22.
go back to reference Padilla-Buritica JI, Martinez-Vargas JD, Castellanos-Dominguez G. Emotion discrimination using spatially compact regions of interest extracted from imaging EEG activity. Front Comput Neurosci. 2016;10:55.CrossRef Padilla-Buritica JI, Martinez-Vargas JD, Castellanos-Dominguez G. Emotion discrimination using spatially compact regions of interest extracted from imaging EEG activity. Front Comput Neurosci. 2016;10:55.CrossRef
23.
go back to reference Chen G, Zhang X, Sun Y, Zhang J. Emotion feature analysis and recognition based on reconstructed eeg sources. IEEE Access. 2020;8:11907–16.CrossRef Chen G, Zhang X, Sun Y, Zhang J. Emotion feature analysis and recognition based on reconstructed eeg sources. IEEE Access. 2020;8:11907–16.CrossRef
24.
go back to reference Wang F, Wu S, Zhang W, Xu Z, Zhang Y, Wu C, et al. Emotion recognition with convolutional neural network and EEG-based EFDMs. Neuropsychologia. 2020;146:107506.CrossRef Wang F, Wu S, Zhang W, Xu Z, Zhang Y, Wu C, et al. Emotion recognition with convolutional neural network and EEG-based EFDMs. Neuropsychologia. 2020;146:107506.CrossRef
27.
go back to reference Khare SK, Bajaj V. Time-frequency representation and convolutional neural network-based emotion recognition. IEEE Transactions on Neural Networks and Learning Systems. 2020. Khare SK, Bajaj V. Time-frequency representation and convolutional neural network-based emotion recognition. IEEE Transactions on Neural Networks and Learning Systems. 2020.
28.
go back to reference Wadhera T, Kakkar D, Rani R. Behavioral modeling using deep neural network framework for ASD diagnosis and prognosis. Emerging Technologies for Healthcare: Internet of Things and Deep Learning Models. 2021:279–98. Wadhera T, Kakkar D, Rani R. Behavioral modeling using deep neural network framework for ASD diagnosis and prognosis. Emerging Technologies for Healthcare: Internet of Things and Deep Learning Models. 2021:279–98.
29.
go back to reference Song T, Liu S, Zheng W, Zong Y, Cui Z, editors. Instance-adaptive graph for EEG emotion recognition. Proceedings of the AAAI Conference on Artificial Intelligence; 2020;34(03):2701-2708. Song T, Liu S, Zheng W, Zong Y, Cui Z, editors. Instance-adaptive graph for EEG emotion recognition. Proceedings of the AAAI Conference on Artificial Intelligence; 2020;34(03):2701-2708.
30.
go back to reference Song T, Zheng W, Song P, Cui Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput. 2018;11(3):532–41.CrossRef Song T, Zheng W, Song P, Cui Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput. 2018;11(3):532–41.CrossRef
32.
go back to reference Zhong P, Wang D, Miao C. EEG-based emotion recognition using regularized graph neural networks. IEEE Transactions on Affective Computing. 2020. Zhong P, Wang D, Miao C. EEG-based emotion recognition using regularized graph neural networks. IEEE Transactions on Affective Computing. 2020.
33.
go back to reference Jin L, Kim EY. Interpretable cross-subject EEG-based emotion recognition using channel-wise features. Sensors. 2020;20(23):6719.CrossRef Jin L, Kim EY. Interpretable cross-subject EEG-based emotion recognition using channel-wise features. Sensors. 2020;20(23):6719.CrossRef
34.
go back to reference Plummer C, Harvey AS, Cook M. EEG source localization in focal epilepsy: where are we now? Epilepsia. 2008;49(2):201–18.CrossRef Plummer C, Harvey AS, Cook M. EEG source localization in focal epilepsy: where are we now? Epilepsia. 2008;49(2):201–18.CrossRef
35.
go back to reference Bauer H, Pllana A. EEG-based local brain activity feedback training—tomographic neurofeedback. Front Hum Neurosci. 2014;8:1005.CrossRef Bauer H, Pllana A. EEG-based local brain activity feedback training—tomographic neurofeedback. Front Hum Neurosci. 2014;8:1005.CrossRef
36.
go back to reference Edelman BJ, Baxter B, He B. EEG source imaging enhances the decoding of complex right-hand motor imagery tasks. IEEE Trans Biomed Eng. 2015;63(1):4–14.CrossRef Edelman BJ, Baxter B, He B. EEG source imaging enhances the decoding of complex right-hand motor imagery tasks. IEEE Trans Biomed Eng. 2015;63(1):4–14.CrossRef
37.
go back to reference Haufe S, Treder MS, Gugler MF, Sagebaum M, Curio G, Blankertz B. EEG potentials predict upcoming emergency brakings during simulated driving. J Neural Eng. 2011;8(5):056001.CrossRef Haufe S, Treder MS, Gugler MF, Sagebaum M, Curio G, Blankertz B. EEG potentials predict upcoming emergency brakings during simulated driving. J Neural Eng. 2011;8(5):056001.CrossRef
38.
go back to reference Noirhomme Q, Kitney RI, Macq B. Single-trial EEG source reconstruction for brain–computer interface. IEEE Trans Biomed Eng. 2008;55(5):1592–601.CrossRef Noirhomme Q, Kitney RI, Macq B. Single-trial EEG source reconstruction for brain–computer interface. IEEE Trans Biomed Eng. 2008;55(5):1592–601.CrossRef
39.
go back to reference Grech R, Cassar T, Muscat J, Camilleri KP, Fabri SG, Zervakis M, et al. Review on solving the inverse problem in EEG source analysis. J Neuroeng Rehabil. 2008;5(1):1–33.CrossRef Grech R, Cassar T, Muscat J, Camilleri KP, Fabri SG, Zervakis M, et al. Review on solving the inverse problem in EEG source analysis. J Neuroeng Rehabil. 2008;5(1):1–33.CrossRef
40.
go back to reference Hallez H, Vanrumste B, Grech R, Muscat J, De Clercq W, Vergult A, et al. Review on solving the forward problem in EEG source analysis. J Neuroeng Rehabil. 2007;4(1):1–29.CrossRef Hallez H, Vanrumste B, Grech R, Muscat J, De Clercq W, Vergult A, et al. Review on solving the forward problem in EEG source analysis. J Neuroeng Rehabil. 2007;4(1):1–29.CrossRef
41.
go back to reference Hamalainen M. Interpreting measured magnetic fields of the brain: estimates of current distributions. Univ Helsinki, Finland Tech Rep TKK-F-A559. 1984. Hamalainen M. Interpreting measured magnetic fields of the brain: estimates of current distributions. Univ Helsinki, Finland Tech Rep TKK-F-A559. 1984.
42.
go back to reference Pascual-Marqui RD, Michel CM, Lehmann D. Low resolution electromagnetic tomography: a new method for localizing electrical activity in the brain. Int J Psychophysiol. 1994;18(1):49–65.CrossRef Pascual-Marqui RD, Michel CM, Lehmann D. Low resolution electromagnetic tomography: a new method for localizing electrical activity in the brain. Int J Psychophysiol. 1994;18(1):49–65.CrossRef
43.
go back to reference Pascual-Marqui RD. Standardized low-resolution brain electromagnetic tomography (sLORETA): technical details. Methods Find Exp Clin Pharmacol. 2002;24(Suppl D):5–12. Pascual-Marqui RD. Standardized low-resolution brain electromagnetic tomography (sLORETA): technical details. Methods Find Exp Clin Pharmacol. 2002;24(Suppl D):5–12.
44.
go back to reference Kiebel SJ, Daunizeau J, Phillips C, Friston KJ. Variational Bayesian inversion of the equivalent current dipole model in EEG/MEG. Neuroimage. 2008;39(2):728–41.CrossRef Kiebel SJ, Daunizeau J, Phillips C, Friston KJ. Variational Bayesian inversion of the equivalent current dipole model in EEG/MEG. Neuroimage. 2008;39(2):728–41.CrossRef
45.
go back to reference Mosher JC, Leahy RM, Lewis PS. EEG and MEG: forward solutions for inverse methods. IEEE Trans Biomed Eng. 1999;46(3):245–59.CrossRef Mosher JC, Leahy RM, Lewis PS. EEG and MEG: forward solutions for inverse methods. IEEE Trans Biomed Eng. 1999;46(3):245–59.CrossRef
46.
go back to reference Scarselli F, Gori M, Tsoi AC, Hagenbuchner M, Monfardini G. The graph neural network model. IEEE Trans Neural Networks. 2008;20(1):61–80.CrossRef Scarselli F, Gori M, Tsoi AC, Hagenbuchner M, Monfardini G. The graph neural network model. IEEE Trans Neural Networks. 2008;20(1):61–80.CrossRef
47.
go back to reference Meng Z, Adluru N, Kim HJ, Fung G, Singh V, editors. Efficient relative attribute learning using graph neural networks. Proceedings of the European conference on computer vision (ECCV). 2018. Meng Z, Adluru N, Kim HJ, Fung G, Singh V, editors. Efficient relative attribute learning using graph neural networks. Proceedings of the European conference on computer vision (ECCV). 2018.
48.
go back to reference Goodfellow I, Bengio Y, Courville A. Deep learning, ser. The adaptive computation and machine learning series. Cambridge, MA: The MIT Press. 2016. Goodfellow I, Bengio Y, Courville A. Deep learning, ser. The adaptive computation and machine learning series. Cambridge, MA: The MIT Press. 2016.
50.
go back to reference Sheykhivand S, Mousavi Z, Rezaii TY, Farzamnia A. Recognizing emotions evoked by music using CNN-LSTM networks on EEG signals. IEEE Access. 2020;8:139332–45.CrossRef Sheykhivand S, Mousavi Z, Rezaii TY, Farzamnia A. Recognizing emotions evoked by music using CNN-LSTM networks on EEG signals. IEEE Access. 2020;8:139332–45.CrossRef
51.
go back to reference Bradley MM, Lang PJ. Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry. 1994;25(1):49–59.CrossRef Bradley MM, Lang PJ. Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry. 1994;25(1):49–59.CrossRef
52.
go back to reference Beck AT, Steer RA, Brown G. Beck depression inventory–II. Psychol Assess. APA Psyc Tests. 1996. Beck AT, Steer RA, Brown G. Beck depression inventory–II. Psychol Assess. APA Psyc Tests. 1996.
54.
go back to reference Tournoux TJ. Co-planar stereotaxic atlas of the human brain: 3-dimensional proportional system—an approach to cerebral imaging. New York: Thieme Medical Publishers; 1988. Tournoux TJ. Co-planar stereotaxic atlas of the human brain: 3-dimensional proportional system—an approach to cerebral imaging. New York: Thieme Medical Publishers; 1988.
55.
go back to reference Brodmann K. Vergleichende Lokalisationslehre der Grosshirnrinde in ihren Prinzipien dargestellt auf Grund des Zellenbaues: Barth; 1909.324 pages. Brodmann K. Vergleichende Lokalisationslehre der Grosshirnrinde in ihren Prinzipien dargestellt auf Grund des Zellenbaues: Barth; 1909.324 pages.
56.
go back to reference Kingma D, Ba L. Adam: A Method for stochastic optimization. The 3rd International Conference for Learning Representations, San Diego, 2015. Kingma D, Ba L. Adam: A Method for stochastic optimization. The 3rd International Conference for Learning Representations, San Diego, 2015.
57.
go back to reference Kayalvizhi M. EEG signal extraction analysis techniques. Signal and Image Processing Techniques for the Development of Intelligent Healthcare Systems: Springer. 2021:223–39. Kayalvizhi M. EEG signal extraction analysis techniques. Signal and Image Processing Techniques for the Development of Intelligent Healthcare Systems: Springer. 2021:223–39.
58.
go back to reference Romanowicz K, Kozłowska K, Wichniak A. Psychomotor retardation in recurrent depression and the related factors. Adv Psychiat Neurol/Postępy Psychiatrii i Neurologii. 28(3):208–19. Romanowicz K, Kozłowska K, Wichniak A. Psychomotor retardation in recurrent depression and the related factors. Adv Psychiat Neurol/Postępy Psychiatrii i Neurologii. 28(3):208–19.
Metadata
Title
Accurate Emotion Recognition Utilizing Extracted EEG Sources as Graph Neural Network Nodes
Authors
Shiva Asadzadeh
Tohid Yousefi Rezaii
Soosan Beheshti
Saeed Meshgini
Publication date
08-12-2022
Publisher
Springer US
Published in
Cognitive Computation / Issue 1/2023
Print ISSN: 1866-9956
Electronic ISSN: 1866-9964
DOI
https://doi.org/10.1007/s12559-022-10077-5

Other articles of this Issue 1/2023

Cognitive Computation 1/2023 Go to the issue

Premium Partner