Abstract
The emotional state of people plays a key role in physiological and behavioral human interaction. Emotional state analysis entails many fields such as neuroscience, cognitive sciences, and biomedical engineering because the parameters of interest contain the complex neuronal activities of the brain. Electroencephalogram (EEG) signals are processed to communicate brain signals with external systems and make predictions over emotional states. This paper proposes a novel method for emotion recognition based on deep convolutional neural networks (CNNs) that are used to classify Valence, Arousal, Dominance, and Liking emotional states. Hence, a novel approach is proposed for emotion recognition with time series of multi-channel EEG signals from a Database for Emotion Analysis and Using Physiological Signals (DEAP). We propose a new approach to emotional state estimation utilizing CNN-based classification of multi-spectral topology images obtained from EEG signals. In contrast to most of the EEG-based approaches that eliminate spatial information of EEG signals, converting EEG signals into a sequence of multi-spectral topology images, temporal, spectral, and spatial information of EEG signals are preserved. The deep recurrent convolutional network is trained to learn important representations from a sequence of three-channel topographical images. We have achieved test accuracy of 90.62% for negative and positive Valence, 86.13% for high and low Arousal, 88.48% for high and low Dominance, and finally 86.23% for like–unlike. The evaluations of this method on emotion recognition problem revealed significant improvements in the classification accuracy when compared with other studies using deep neural networks (DNNs) and one-dimensional CNNs.
Funding source: Izmir Katip Celebi University
Award Identifier / Grant number: 2019-ONAP–MUMF-0001
Acknowledgments
This study was supported by Izmir Katip Celebi University Scientific Research Projects Coordination Unit: Project numbers: 2019-ONAP–MUMF-0001.
Research funding: This work was funded by Izmir Katip Celebi University Scientific Research Projects Coordination Unit (project number: 2019-ONAP–MUMF-0001). All funding is for equipment for the research project. There is no available funding for publication.
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
Conflict of interest: Authors state no conflict of interest.
Informed consent: Informed consent is not applicable.
Ethical approval: The research related to human use complies with all the relevant national regulations and institutional policies. Human EEG data were obtained from the DEAP dataset which compiles all relevant ethical processes.
References
1. Bălan, O, Moise, G, Moldoveanu, A, Leordeanu, M, Moldoveanu, F. Fear level classification based on emotional dimensions and machine learning techniques. Sensors 2019;19:1738.10.3390/s19071738Search in Google Scholar PubMed PubMed Central
2. Selvaraj, J, Murugappan, M, Wan, K, Yaacob, S. Frequency study of facial electromyography signals with respect to emotion recognition. Biomed Eng Biomed Tech 2014;59:241–9. https://doi.org/10.1515/bmt-2013-0118.10.1515/bmt-2013-0118Search in Google Scholar PubMed
3. Li, P, Liu, H, Si, Y, Li, C, Li, F, Zhu, X, et al. EEG based emotion recognition by combining functional connectivity network and local activations. IEEE Trans Biomed Eng 2019;66:2869–81. https://doi.org/10.1109/tbme.2019.2897651.10.1109/TBME.2019.2897651Search in Google Scholar PubMed
4. Pantic, M, Pentland, A, Nijholt, A, Huang, TS. Human computing and machine understanding of human behavior: a survey. In: Huang, T, Nijholt, A, Pantic, M, Pentland, A, editors. Artificial intelligence for human computing. Berlin: Springer Berlin Heidelberg; 2007. pp. 47–71.10.1145/1180995.1181044Search in Google Scholar
5. Ekman, P, Friesen, WV, O’Sullivan, M, Chan, A, Diacoyanni-Tarlatzis, I, Heider, K, et al. Universals and cultural differences in the judgments of facial expressions of emotion. J Pers Soc Psychol 1987;53:712. https://doi.org/10.1037/0022-3514.53.4.712.10.1037/0022-3514.53.4.712Search in Google Scholar
6. Ozdemir, MA, Degirmenci, M, Guren, O, Akan, A 2019, EEG based emotional state estimation using 2-D deep learning technique. In: 2019 Medical Technologies Congress (TIPTEKNO), 2019 IEEE: IEEE. pp. 1–4. https://doi.org/10.1109/tiptekno.2019.8895158.10.1109/TIPTEKNO.2019.8895158Search in Google Scholar
7. Othman, M, Wahab, A, Karim, I, Dzulkifli, MA, Alshaikli, IFT. EEG emotion recognition based on the dimensional models of emotions. Procedia Soc Behav Sci 2013;97:30–7. https://doi.org/10.1016/j.sbspro.2013.10.201.10.1016/j.sbspro.2013.10.201Search in Google Scholar
8. Yoon, S-a, Son, G, Kwon, S. Fear emotion classification in speech by acoustic and behavioral cues. Multimed Tools Appl 2019;78:2345–66. https://doi.org/10.1007/s11042-018-6329-2.10.1007/s11042-018-6329-2Search in Google Scholar
9. Mert, A, Akan, A. Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal Appl 2018;21:81–9. https://doi.org/10.1007/s10044-016-0567-6.10.1007/s10044-016-0567-6Search in Google Scholar
10. Lan, Z, Sourina, O, Wang, L, Liu, Y. Real-time EEG-based emotion monitoring using stable features. Visual Comput 2016;32:347–58. https://doi.org/10.1007/s00371-015-1183-y.10.1007/s00371-015-1183-ySearch in Google Scholar
11. Khosrowabadi, R, Quek, HC, Wahab, A, Ang, KK 2010, EEG-based emotion recognition using self-organizing map for boundary detection. In: 2010 20th International Conference on Pattern Recognition, 2010 IEEE: IEEE. pp. 4242–5.10.1109/ICPR.2010.1031Search in Google Scholar
12. Verma, GK, Tiwary, US. Affect representation and recognition in 3d continuous valence–arousal–dominance space. Multimed Tools Appl 2017;76:2159–83. https://doi.org/10.1007/s11042-015-3119-y.10.1007/s11042-015-3119-ySearch in Google Scholar
13. Koelstra, S, Muhl, C, Soleymani, M, Lee, J-S, Yazdani, A, Ebrahimi, T, et al. Deap: a database for emotion analysis; using physiological signals. IEEE Trans Affective Comput 2011;3:18–31.10.1109/T-AFFC.2011.15Search in Google Scholar
14. Li, J, Qiu, S, Shen, Y-Y, Liu, C-L, He, H. Multisource transfer learning for cross-subject EEG emotion recognition. IEEE Trans Cybern 2019:1–13. https://doi.org/10.1109/tcyb.2019.2904052.10.1109/TCYB.2019.2904052Search in Google Scholar PubMed
15. Shimizu, S, Ito, T, Yin, Y, Arakawa, S, Sawada, O, Aoyagi, I. Driver emotion estimation via convolutional neural network with ECG. Trans Jpn Soc Aeronaut Space Sci 2019;50:505–10.Search in Google Scholar
16. Kehri, V, Ingle, R, Patil, S, Awale, R. Analysis of facial EMG signal for emotion recognition using wavelet packet transform and SVM. In. Tanveer, M, Pachori, R, editors. Machine intelligence and signal analysis. Singapore: Springer Singapore; 2019. pp. 247–57.10.1007/978-981-13-0923-6_21Search in Google Scholar
17. García-Faura, Á, Hernández-García, A, Fernández-Martínez, F, Díaz-de-María, F, San-Segundo, R 2019, Emotion and attention: audiovisual models for group-level skin response recognition in short movies. In: Web intelligence, 2019 IOS Press: IOS Press. pp. 29–40.10.3233/WEB-190398Search in Google Scholar
18. Bazgir, O, Mohammadi, Z, Habibi, SAH 2019, Emotion recognition with machine learning using EEG signals. In: 2018 25th National and 3rd International Iranian Conference on Biomedical Engineering (ICBME), 2019 IEEE: IEEE. pp. 1–5.10.1109/ICBME.2018.8703559Search in Google Scholar
19. Soroush, MZ, Maghooli, K, Setarehdan, SK, Nasrabadi, AM. A review on EEG signals based emotion recognition. ICNSJ 2017;4:118.Search in Google Scholar
20. Li, X, Song, D, Zhang, P, Yu, G, Hou, Y, Hu, B 2016, Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2016 IEEE: IEEE. pp. 352–9.10.1109/BIBM.2016.7822545Search in Google Scholar
21. Gupta, V, Chopda, MD, Pachori, RB. Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals. IEEE Sens J 2019;19:2266–74. https://doi.org/10.1109/jsen.2018.2883497.10.1109/JSEN.2018.2883497Search in Google Scholar
22. Daimi, SN, Saha, G. Classification of emotions induced by music videos and correlation with participants’ rating. Expert Syst Appl 2014;41:6057–65. https://doi.org/10.1016/j.eswa.2014.03.050.10.1016/j.eswa.2014.03.050Search in Google Scholar
23. Mohammadi, Z, Frounchi, J, Amiri, M. Wavelet-based emotion recognition system using EEG signal. Neural Comput Appl 2017;28:1985–90. https://doi.org/10.1007/s00521-015-2149-8.10.1007/s00521-015-2149-8Search in Google Scholar
24. Zhang, Y, Ji, X, Zhang, S. An approach to EEG-based emotion recognition using combined feature extraction method. Neurosci Lett 2016;633:152–7. https://doi.org/10.1016/j.neulet.2016.09.037.10.1016/j.neulet.2016.09.037Search in Google Scholar PubMed
25. Zhuang, N, Zeng, Y, Tong, L, Zhang, C, Zhang, H, Yan, B. Emotion recognition from EEG signals using multidimensional information in EMD domain. BioMed Res Int 2017;2017:8317357. https://doi.org/10.1155/2017/8317357.10.1155/2017/8317357Search in Google Scholar PubMed PubMed Central
26. Zhang, Q, Chen, X, Zhan, Q, Yang, T, Xia, S. Respiration-based emotion recognition with deep learning. Comput Ind 2017;92:84–90. https://doi.org/10.1016/j.compind.2017.04.005.10.1016/j.compind.2017.04.005Search in Google Scholar
27. Liu, W, Zheng, W-L, Lu, B-L 2016, Emotion recognition using multimodal deep learning. In: International conference on neural information processing, 2016 Springer: Springer. pp. 521–9.10.1007/978-3-319-46672-9_58Search in Google Scholar
28. Pandey, P, Seeja, KR 2019, Subject-independent emotion detection from EEG signals using deep neural network. In: International Conference on Innovative Computing and Communications, 2019 Springer: Springer. pp. 41–6.10.1007/978-981-13-2354-6_5Search in Google Scholar
29. Kuanar, S, Athitsos, V, Pradhan, N, Mishra, A, Rao, KR 2018, Cognitive analysis of working memory load from EEG, by a deep recurrent neural network. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018 IEEE: IEEE. pp. 2576–80.10.1109/ICASSP.2018.8462243Search in Google Scholar
30. Jirayucharoensak, S, Pan-Ngum, S, Israsena, P. EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. Sci World J 2014;2014:1–10. https://doi.org/10.1155/2014/627892.10.1155/2014/627892Search in Google Scholar PubMed PubMed Central
31. Zhou, J, Wei, X, Cheng, C, Yang, Q, Li, Q. Multimodal emotion recognition method based on convolutional auto-encoder. Int J Comput Int Sys 2018;12:351–8.10.2991/ijcis.2019.125905651Search in Google Scholar
32. Tripathi, S, Acharya, S, Sharma, RD, Mittal, S, Bhattacharya, S. Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. In: AAAI’17: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017 AAAI Press: AAAI Press; 2017. pp. 4746–52.10.1609/aaai.v31i2.19105Search in Google Scholar
33. Snyder, JP. Map projections: a working manual, 1st ed. Washington: US Government Printing Office; 1987.10.3133/pp1395Search in Google Scholar
34. Morris, JD. Observations: SAM: the self-assessment manikin; an efficient cross-cultural measurement of emotional response. J Advert 1995;35:63–8.Search in Google Scholar
35. Pereira, ET, Gomes, HM 2016, The role of data balancing for emotion classification using EEG signals. In: 2016 IEEE International Conference on Digital Signal Processing (DSP), 2016 IEEE: IEEE. pp. 555–9.10.1109/ICDSP.2016.7868619Search in Google Scholar
36. Wichakam, I, Vateekul, P 2014, An evaluation of feature extraction in EEG-based emotion prediction with support vector machines. In: 2014 11th International Joint Conference on Computer Science and Software Engineering (JCSSE), 2014 IEEE: IEEE. pp. 106–10.10.1109/JCSSE.2014.6841851Search in Google Scholar
37. Li, J, Zhang, Z, He, H. Hierarchical convolutional neural networks for EEG-based emotion recognition. Cogn Comput 2018:1–13.10.1007/s12559-017-9533-xSearch in Google Scholar
38. Bashivan, P, Rish, I, Yeasin, M, Codella, N. Learning representations from EEG with deep recurrent-convolutional neural networks. CoRR 2015: 06448. abs/1511.Search in Google Scholar
39. Panigrahi, N, Mishra, CS. A generic method for azimuthal map projection. Def Sci J 2015;65:390–4. https://doi.org/10.14429/dsj.65.8598.10.14429/dsj.65.8598Search in Google Scholar
40. Alfeld, P. A trivariate clough—tocher scheme for tetrahedral data. Comput Aided Geom Des 1984;1:169–81. https://doi.org/10.1016/0167-8396(84)90029-3.10.1016/0167-8396(84)90029-3Search in Google Scholar
41. Savareh, BA, Emami, H, Hajiabadi, M, Azimi, SM, Ghafoori, M. Wavelet-enhanced convolutional neural network: a new idea in a deep learning paradigm. Biomed Eng Biomed Tech 2019;64:195–205. https://doi.org/10.1515/bmt-2017-0178.10.1515/bmt-2017-0178Search in Google Scholar PubMed
42. Bengio, Y. Practical recommendations for gradient-based training of deep architectures. In: Montavon, G, Orr, G, Müller, K, editors. Neural networks: tricks of the trade. Berlin: Springer; 2012. pp. 437–78.10.1007/978-3-642-35289-8_26Search in Google Scholar
43. Kiranyaz, S, Ince, T, Gabbouj, M. Real-time patient-specific ECG classification by 1-D convolutional neural networks. IEEE Trans Biomed Eng 2015;63:664–75.10.1109/TBME.2015.2468589Search in Google Scholar PubMed
44. Kim, B-K, Roh, J, Dong, S-Y, Lee, S-Y. Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J Multimodal User In 2016;10:173–89. https://doi.org/10.1007/s12193-015-0209-0.10.1007/s12193-015-0209-0Search in Google Scholar
45. Gargeya, R, Leng, T. Automated identification of diabetic retinopathy using deep learning. Ophthalmology 2017;124:962–9. https://doi.org/10.1016/j.ophtha.2017.02.008.10.1016/j.ophtha.2017.02.008Search in Google Scholar PubMed
46. Li, C, Bao, Z, Li, L, Zhao, Z. Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition. Inf Process Manag 2020;57:102185. https://doi.org/10.1016/j.ipm.2019.102185.10.1016/j.ipm.2019.102185Search in Google Scholar
47. Al-Nafjan, A, Hosny, M, Al-Wabil, A, Al-Ohali, Y. Classification of human emotions from electroencephalogram (EEG) signal using deep neural network. Int J Adv Comput Sci Appl 2017;8:419–25. https://doi.org/10.14569/ijacsa.2017.080955.10.14569/IJACSA.2017.080955Search in Google Scholar
48. Yin, Z, Zhao, M, Wang, Y, Yang, J, Zhang, J. Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Comput Methods Programs Biomed 2017;140:93–110. https://doi.org/10.1016/j.cmpb.2016.12.005.10.1016/j.cmpb.2016.12.005Search in Google Scholar PubMed
© 2020 Walter de Gruyter GmbH, Berlin/Boston
