Skip to main content
Erschienen in: Neural Computing and Applications 2/2023

26.09.2022 | Original Article

Emotion recognition in EEG signals using the continuous wavelet transform and CNNs

verfasst von: Oscar Almanza-Conejo, Dora Luz Almanza-Ojeda, Jose Luis Contreras-Hernandez, Mario Alberto Ibarra-Manzano

Erschienen in: Neural Computing and Applications | Ausgabe 2/2023

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Emotions are mental states associated with changes that influence people’s behavior, thinking, and health. Emotional changes can also appear in the organs and tissues of the human body as electrical potential differences gathered as biosignals in datasets. This work proposes the classification of emotions in electroencephalographical signals, transforming these discrete signals into a time-scale representation by spectral analysis. Our approach uses the wavelet transform to obtain scalogram images of electroencephalographic signals, treating these images as the scaled distribution of energy associated with a sign. Feature extraction from the scalograms is performed using convolutional neural networks (CNNs), leading to the proposal of two classification models. The threshold values in primitive emotions define one model of four emotions and the second of eight. The data augmentation technique increases the dataset size to compensate for the extra classes added in the second CNN model. The classification results were evaluated using different performance metrics and compared with related works in the literature.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Ekman P, Friesen W (2003) Unmasking the face: a guide to recognizing emotions from facial clues. Malor Books, Katihar Ekman P, Friesen W (2003) Unmasking the face: a guide to recognizing emotions from facial clues. Malor Books, Katihar
2.
Zurück zum Zitat Verma G, Tiwary U (2017) Affect representation and recognition in 3D continuous valence-arousal-dominance space. Multimed Tools Appl 76:2159–2183CrossRef Verma G, Tiwary U (2017) Affect representation and recognition in 3D continuous valence-arousal-dominance space. Multimed Tools Appl 76:2159–2183CrossRef
3.
Zurück zum Zitat Kantz H, Kurths J, Mayer-Kress G (2012) Nonlinear analysis of physiological data. Springer Science & Business Media, BerlinMATH Kantz H, Kurths J, Mayer-Kress G (2012) Nonlinear analysis of physiological data. Springer Science & Business Media, BerlinMATH
6.
Zurück zum Zitat Madupu R, Kothapalli C, Yarra V, Harika S & Basha C (2020) Automatic human emotion recognition system using facial expressions with convolution neural network. 2020 4th international conference on electronics, communication and aerospace technology (ICECA), pp 1179–1183 Madupu R, Kothapalli C, Yarra V, Harika S & Basha C (2020) Automatic human emotion recognition system using facial expressions with convolution neural network. 2020 4th international conference on electronics, communication and aerospace technology (ICECA), pp 1179–1183
8.
Zurück zum Zitat Uddin M, Nilsson E (2020) Emotion recognition using speech and neural structured learning to facilitate edge intelligence. Eng Appl Arti Intell 94:103775CrossRef Uddin M, Nilsson E (2020) Emotion recognition using speech and neural structured learning to facilitate edge intelligence. Eng Appl Arti Intell 94:103775CrossRef
9.
Zurück zum Zitat Wang K, Su G, Liu L, Wang S (2020) Wavelet packet analysis for speaker-independent emotion recognition. Neurocomputing 398:257–264CrossRef Wang K, Su G, Liu L, Wang S (2020) Wavelet packet analysis for speaker-independent emotion recognition. Neurocomputing 398:257–264CrossRef
10.
Zurück zum Zitat Zhang X, Yao L, Wang X, Monaghan J, McAlpine D, Zhang Y (2021) A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers. J Neural Eng 18:031002CrossRef Zhang X, Yao L, Wang X, Monaghan J, McAlpine D, Zhang Y (2021) A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers. J Neural Eng 18:031002CrossRef
11.
Zurück zum Zitat Cai S, Li H, Wu Q, Liu J, Zhang Y (2022) Motor imagery decoding in the presence of distraction using graph sequence neural networks. IEEE Trans Neural Syst Rehabil Eng 30:1716–1726CrossRef Cai S, Li H, Wu Q, Liu J, Zhang Y (2022) Motor imagery decoding in the presence of distraction using graph sequence neural networks. IEEE Trans Neural Syst Rehabil Eng 30:1716–1726CrossRef
12.
Zurück zum Zitat Zhang Y, Zhou T, Wu W, Xie H, Zhu H, Zhou G, Cichocki A (2022) Improving EEG decoding via clustering-based multitask feature learning. IEEE Trans Neural Netw Learn Syst 33:3587–3597CrossRef Zhang Y, Zhou T, Wu W, Xie H, Zhu H, Zhou G, Cichocki A (2022) Improving EEG decoding via clustering-based multitask feature learning. IEEE Trans Neural Netw Learn Syst 33:3587–3597CrossRef
13.
Zurück zum Zitat Campion J, Javed A, Sartorius N, Marmot M (2020) Addressing the public mental health challenge of COVID-19. Lancet Psychiatry 7:657–659CrossRef Campion J, Javed A, Sartorius N, Marmot M (2020) Addressing the public mental health challenge of COVID-19. Lancet Psychiatry 7:657–659CrossRef
14.
Zurück zum Zitat Winkler P, Formanek T, Mlada K, Kagstrom A, Mohrova Z, Mohr P, Csemy L (2019) Increase in prevalence of current mental disorders in the context of COVID-19: analysis of repeated nationwide cross-sectional surveys. Epidemiol Psychiatric Sci 29:1–8 Winkler P, Formanek T, Mlada K, Kagstrom A, Mohrova Z, Mohr P, Csemy L (2019) Increase in prevalence of current mental disorders in the context of COVID-19: analysis of repeated nationwide cross-sectional surveys. Epidemiol Psychiatric Sci 29:1–8
15.
Zurück zum Zitat Andalibi, N. & Buss, J (2020) The human in emotion recognition on social media: attitudes, outcomes, risks. InProceedings Of The 2020 CHI conference On human factors in computing systems Andalibi, N. & Buss, J (2020) The human in emotion recognition on social media: attitudes, outcomes, risks. InProceedings Of The 2020 CHI conference On human factors in computing systems
16.
Zurück zum Zitat Zhang Y, Zhang S, Ji X (2018) EEG-based classification of emotions using empirical mode decomposition and autoregressive model. Multimed Tools Appl 77:26697–26710CrossRef Zhang Y, Zhang S, Ji X (2018) EEG-based classification of emotions using empirical mode decomposition and autoregressive model. Multimed Tools Appl 77:26697–26710CrossRef
17.
Zurück zum Zitat Zhuang N, Zeng Y, Tong L, Zhang C, Zhang H, Yan B (2017) Emotion recognition from EEG signals using multidimensional information in EMD domain. BioMed Res Int 2017:9CrossRef Zhuang N, Zeng Y, Tong L, Zhang C, Zhang H, Yan B (2017) Emotion recognition from EEG signals using multidimensional information in EMD domain. BioMed Res Int 2017:9CrossRef
18.
Zurück zum Zitat Lin Y, Wang C, Jung T, Wu T, Jeng S, Duann J, Chen J (2010) EEG-based emotion Recognition in music listening. IEEE Trans Biomed Eng 57:1798–1806CrossRef Lin Y, Wang C, Jung T, Wu T, Jeng S, Duann J, Chen J (2010) EEG-based emotion Recognition in music listening. IEEE Trans Biomed Eng 57:1798–1806CrossRef
19.
Zurück zum Zitat Liu, Y. & Sourina, O (2014) EEG-based subject-dependent emotion recognition algorithm using fractal dimension. In2014 IEEE international conference on systems, man, and cybernetics (SMC) Liu, Y. & Sourina, O (2014) EEG-based subject-dependent emotion recognition algorithm using fractal dimension. In2014 IEEE international conference on systems, man, and cybernetics (SMC)
20.
Zurück zum Zitat Taran S, Bajaj V (2019) Emotion recognition from single-channel EEG signals using a two-stage correlation and instantaneous frequency-based filtering method. Comput Methods Progr Biomed 173:157–165CrossRef Taran S, Bajaj V (2019) Emotion recognition from single-channel EEG signals using a two-stage correlation and instantaneous frequency-based filtering method. Comput Methods Progr Biomed 173:157–165CrossRef
21.
Zurück zum Zitat Koelstra S, Mühl C, Soleymani M, Lee J, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) DEAP: a database for emotion analysis; Using physiological signals. IEEE Trans Affect Comput 3:18–31CrossRef Koelstra S, Mühl C, Soleymani M, Lee J, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) DEAP: a database for emotion analysis; Using physiological signals. IEEE Trans Affect Comput 3:18–31CrossRef
22.
Zurück zum Zitat Yang K, Tong L, Shu J, Zhuang N, Yan B, Zeng Y (2020) High gamma band EEG closely related to emotion: evidence from functional network. Front Human Neurosci 14:89CrossRef Yang K, Tong L, Shu J, Zhuang N, Yan B, Zeng Y (2020) High gamma band EEG closely related to emotion: evidence from functional network. Front Human Neurosci 14:89CrossRef
23.
Zurück zum Zitat Norwood M, Lakhani A, Maujean A, Zeeman H, Creux O, Kendall E (2019) Brain activity, underlying mood and the environment: a systematic review. J Environ Psychol 65:101321CrossRef Norwood M, Lakhani A, Maujean A, Zeeman H, Creux O, Kendall E (2019) Brain activity, underlying mood and the environment: a systematic review. J Environ Psychol 65:101321CrossRef
24.
Zurück zum Zitat Khare S, Bajaj V (2020) Time–frequency representation and convolutional neural network-based emotion recognition. IEEE Trans Neural Netw Learn Syst 32:2901–2909CrossRef Khare S, Bajaj V (2020) Time–frequency representation and convolutional neural network-based emotion recognition. IEEE Trans Neural Netw Learn Syst 32:2901–2909CrossRef
25.
Zurück zum Zitat Gannouni S, Aledaily A, Belwafi K, Aboalsamh H (2020) Adaptive emotion detection using the valence-arousal-dominance model and EEG brain rhythmic activity changes in relevant brain lobes. IEEE Access 8:67444–67455CrossRef Gannouni S, Aledaily A, Belwafi K, Aboalsamh H (2020) Adaptive emotion detection using the valence-arousal-dominance model and EEG brain rhythmic activity changes in relevant brain lobes. IEEE Access 8:67444–67455CrossRef
26.
Zurück zum Zitat Garg, A., Kapoor, A., Bedi, A. & Sunkaria, R (2019). Merged LSTM Model for emotion classification using EEG signals. In: international conference on data science and engineering, ICDSE 2019. pp 139–143 Garg, A., Kapoor, A., Bedi, A. & Sunkaria, R (2019). Merged LSTM Model for emotion classification using EEG signals. In: international conference on data science and engineering, ICDSE 2019. pp 139–143
27.
Zurück zum Zitat Cimtay Y, Ekmekcioglu E (2020) Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition. Sensors (Switz) 20:1–20 Cimtay Y, Ekmekcioglu E (2020) Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition. Sensors (Switz) 20:1–20
28.
Zurück zum Zitat Garg D, Verma G (2020) Emotion recognition in valence-arousal space from multi-channel EEG data and wavelet based deep learning framework. Procedia Comput Sci 171:857–867CrossRef Garg D, Verma G (2020) Emotion recognition in valence-arousal space from multi-channel EEG data and wavelet based deep learning framework. Procedia Comput Sci 171:857–867CrossRef
29.
Zurück zum Zitat Salama, E., El-Khoribi, R., Shoman, M. & Shalaby, M(2021) A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition. Egypt Inform J 22:167–176 Salama, E., El-Khoribi, R., Shoman, M. & Shalaby, M(2021) A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition. Egypt Inform J 22:167–176
30.
Zurück zum Zitat Zheng W, Liu W, Lu Y, Lu B, Cichocki A. EmotionMeter (2018) A Multimodal framework for recognizing human emotions. IEEE Trans Cybern 49:1110–1122CrossRef Zheng W, Liu W, Lu Y, Lu B, Cichocki A. EmotionMeter (2018) A Multimodal framework for recognizing human emotions. IEEE Trans Cybern 49:1110–1122CrossRef
32.
Zurück zum Zitat Wang X, Nie D, Lu B (2011) EEG-based emotion recognition using frequency domain features and support vector machines. Lect Notes Comput Sci (Incl Subser Lect Notes Artif Intell Lect Notes Bioinform) 7062:734–743 Wang X, Nie D, Lu B (2011) EEG-based emotion recognition using frequency domain features and support vector machines. Lect Notes Comput Sci (Incl Subser Lect Notes Artif Intell Lect Notes Bioinform) 7062:734–743
33.
Zurück zum Zitat Alazrai R, Homoud R, Alwanni H, Daoud M (2018) EEG-based emotion recognition using quadratic time-frequency distribution. Sensors (Switz) 18:1–32 Alazrai R, Homoud R, Alwanni H, Daoud M (2018) EEG-based emotion recognition using quadratic time-frequency distribution. Sensors (Switz) 18:1–32
34.
Zurück zum Zitat Murugappan M, Ramachandran N, Sazali Y (2010) Classification of human emotion from EEG using discrete wavelet transform. J Biomed Sci Eng 3:390–396CrossRef Murugappan M, Ramachandran N, Sazali Y (2010) Classification of human emotion from EEG using discrete wavelet transform. J Biomed Sci Eng 3:390–396CrossRef
36.
Zurück zum Zitat Akin M (2002) Comparison of wavelet transform and FFT methods in the analysis of EEG signals. J Med Syst 26:241–247CrossRef Akin M (2002) Comparison of wavelet transform and FFT methods in the analysis of EEG signals. J Med Syst 26:241–247CrossRef
37.
Zurück zum Zitat Guo T, Wu C, Qu D (2004) Wavelet transform theory and its application progress: a review. Inf Control 33:67–71 Guo T, Wu C, Qu D (2004) Wavelet transform theory and its application progress: a review. Inf Control 33:67–71
38.
Zurück zum Zitat Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V & Rabinovich A (2015) Going deeper with convolutions. InProceedings of the IEEE computer society conference on computer vision and pattern recognition Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V & Rabinovich A (2015) Going deeper with convolutions. InProceedings of the IEEE computer society conference on computer vision and pattern recognition
39.
Zurück zum Zitat Türk Ö, Özerdem M (2019) Epilepsy detection by using scalogram based convolutional neural network from EEG signals. Brain Sci 9:115CrossRef Türk Ö, Özerdem M (2019) Epilepsy detection by using scalogram based convolutional neural network from EEG signals. Brain Sci 9:115CrossRef
41.
Zurück zum Zitat Sakalle A, Tomar P, Bhardwaj H, Acharya D, Bhardwaj A (2021) A LSTM based deep learning network for recognizing emotions using wireless brainwave driven system. Expert Syst Appl 173:114516CrossRef Sakalle A, Tomar P, Bhardwaj H, Acharya D, Bhardwaj A (2021) A LSTM based deep learning network for recognizing emotions using wireless brainwave driven system. Expert Syst Appl 173:114516CrossRef
42.
Zurück zum Zitat Fourati R, Ammar B, Sanchez-Medina J, Alimi A (2022) Unsupervised learning in reservoir computing for EEG-based emotion recognition. IEEE Trans Affect Comput 13:972–984CrossRef Fourati R, Ammar B, Sanchez-Medina J, Alimi A (2022) Unsupervised learning in reservoir computing for EEG-based emotion recognition. IEEE Trans Affect Comput 13:972–984CrossRef
43.
Zurück zum Zitat Nakisa B, Rastgoo M, Rakotonirainy A, Maire F, Chandran V (2018) Long short term memory hyperparameter optimization for a neural network based emotion recognition framework. IEEE Access 6:49325–49338CrossRef Nakisa B, Rastgoo M, Rakotonirainy A, Maire F, Chandran V (2018) Long short term memory hyperparameter optimization for a neural network based emotion recognition framework. IEEE Access 6:49325–49338CrossRef
44.
Zurück zum Zitat Zheng W, Zhu J, Lu B (2019) Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans Affect Comput 10:417–429CrossRef Zheng W, Zhu J, Lu B (2019) Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans Affect Comput 10:417–429CrossRef
45.
Zurück zum Zitat Xin L, Xiao-Qi S, Xiao-Ying Q & Xiao-Feng S (2016) Relevance vector machine based EEG emotion recognition. In2016 sixth international conference on instrumentation & measurement, computer, communication and control (IMCCC) Xin L, Xiao-Qi S, Xiao-Ying Q & Xiao-Feng S (2016) Relevance vector machine based EEG emotion recognition. In2016 sixth international conference on instrumentation & measurement, computer, communication and control (IMCCC)
46.
Zurück zum Zitat Ali M, Mosa A, Machot F & Kyamakya, K (2016) EEG-based emotion recognition approach for e-healthcare applications. In2016 eighth international conference on Ubiquitous and future networks (ICUFN) Ali M, Mosa A, Machot F & Kyamakya, K (2016) EEG-based emotion recognition approach for e-healthcare applications. In2016 eighth international conference on Ubiquitous and future networks (ICUFN)
Metadaten
Titel
Emotion recognition in EEG signals using the continuous wavelet transform and CNNs
verfasst von
Oscar Almanza-Conejo
Dora Luz Almanza-Ojeda
Jose Luis Contreras-Hernandez
Mario Alberto Ibarra-Manzano
Publikationsdatum
26.09.2022
Verlag
Springer London
Erschienen in
Neural Computing and Applications / Ausgabe 2/2023
Print ISSN: 0941-0643
Elektronische ISSN: 1433-3058
DOI
https://doi.org/10.1007/s00521-022-07843-9

Weitere Artikel der Ausgabe 2/2023

Neural Computing and Applications 2/2023 Zur Ausgabe

S.I. : 2019 India Intl. Congress on Computational Intelligence

Investigation of PCA as a compression pre-processing tool for X-ray image classification

Premium Partner