Emotion development system by interacting with human EEG and natural scene understanding
Introduction
Research on computational and robotic models of human emotion has been very active with the rapid development of robotic technology (Picard, 2000). This has also witnessed the proliferation of commercial ‘affective’ toys and robots. The ultimate goal in this field is to make a truly human-like intelligent system that has artificial emotions, including the capability to recognize, understand, and express emotions. However, since emotion is a special dynamic form of cognition that is extremely complicated. Traditional programming or the task-specific machine learning is insufficient for constructing such an intelligent machine. Instead, a mental development scheme is needed (Weng et al., 2001). Humans autonomously develop their capabilities through lifelong mental development and interaction with the environment, facilitating independent thought and the invention of new abilities. Therefore, the ability of development is also crucial to model artificial emotions for robots.
Empirical research on emotion is characterized by a wide variety of methodologies. Recently, more and more researchers have been paying attention to the visual influences of images on human emotion, as in (Assfalg et al., 2002, Colombo et al., 1999, Iwadate et al., 2000, Yu and Xu, 2004), because visual information plays an important role in affecting a subject’s emotional status. It can be positive or negative depending on whether a concern is advanced or impeded, respectively (Oatley & Jenkins, 1996). However, most of conventional image processing ignores emotional factors, which can help us to describe and simulate the human feedback of the natural scene images. Accordingly, a lot of research was done with the presentation of faces with different facial expressions. It is clear that emotions cannot be exhaustively apprehended using only facial expressions (Gntekin & Basar, 2007), but also can be affected by other types of stimuli, which induce different emotional and physiological reactions in humans (Partala et al., 2000, Takahashi, 2005). Accumulating evidence suggests that natural stimuli are more reliable and stronger stimuli to activate the human emotional responses compared with well-control but often very unnatural stimuli (Hasson, Malach, & Heeger, 2010). Understanding the mechanism of emotion in natural scenes must be developed for a machine to understand the human emotions in various environments. Accordingly, this study builds a system that can analyze, understand emotions related to natural scene images and develop the obtained emotional knowledge under supervision and interaction with humans to understand more complicated human emotions by a developmental way.
The recognizability of different emotions depends on how well the emotion relevant features can be mapped onto chosen emotion representation and how successfully this mapping develops according to the increment of emotion. The emotion representation used in our study is the two-dimensional mapping with valence and arousal axes, which describe the extent of pleasure/sadness and calmness/excitation, respectively (Russell, 1980). We firstly consider categorizing human emotion as positive and negative in valence, and then sub-cluster the emotional features in positive and negative categories so that the system is able to develop its mental capability to understand more complicated emotions.
The features we need here should reflect the emotional factors of natural scenes. Some of them can be extracted from a scene image while others come from the subject’s responses. Human emotion can be induced by some stimulative factors of an image (Picard, 1997). Therefore, some features related to the emotion reflected by natural scenes should be considered as one part of emotional feature space. Here we take color and orientation information into consideration. Considering the smooth changing in term of human perception, L*C*H* space is selected for the color information. In term of the subject’s response analysis, a large body of literature describes neural correlates of human emotion in which different emotions are associated with the activation of specific neural networks (Damasio et al., 2000, Lane et al., 1997b, Phillips et al., 1997, Reiman et al., 1997). We could envision a mind (or brain) as composed of many different “resources” (Minsky, 2006). Each one is responsible for certain specialized jobs. This can help us to understand how a mind could make changes in its state, such as emotional states. For example, the state labeled “angry” could be what happens when people activate some resources that help them to react with more speed and strength-while also suppressing some other resources that make them act prudently. Based on this concept, we use EEG to explore how the “resource” are activated for a given emotional stimulus. When our brain perceives the emotional stimulus, a population of neurons in specific areas fire, then voltage changes occur. The best known correlates of emotionality found with EEG involve prefrontal asymmetry (Douglas-Cowie et al., 2004). That is, more active left frontal region indicates a positive reaction, and more active right anterior lobe indicates negative affection (Lewis & Haviland-Jones, 2004). This shows great potential for positive and negative emotion classification, but it can only indicate the state of valence. In order to make the proposed system understand more different types of emotion, arousal, as another element of emotion should be taken into consideration when the system develops its emotional knowledge. The EEG consists of the activity of an ensemble of generators producing rhythmic activity in several frequency ranges. By application of sensory stimulation these generators are coupled, and act together in a coherent way. This synchronization and enhancement of EEG activity give rise to ‘evoked’ or ‘induced rhythms’. Evoked potentials representing ensembles of neural population response were considered as a result of transition from disordered to an ordered state (Basar, Schumann, Demiralp, Basar-Eroglu, & Ademoglu, 2001). The observed frequencies have been divided into specific groups. The two that are most important for arousal state of brain are the alpha (8–12 Hz) and beta (12–30 Hz) frequencies. Alpha waves are typically for an alert/relaxed mental state, while beta activity is most prominent in the frontal cortex over other areas during intense focused mental activity (Kandel, Schwartz, & Jessell, 2000). Therefore, the beta/alpha ratio could be an indication of the arousal state of the brain. By taking arousal axis into consideration, we can sub-categorize positive and negative emotions into pleasure/joy and sadness/anger, respectively. When a human subject is stimulated by a natural scene, his or her EEG signals will give a pattern to indicate the emotion status and it will be used as one part to construct the emotional feature space.
Although emotion recognition is permeated with uncertainty, previous research on modeling human emotion has reduced this uncertainty (Gene Ball, 1999, Hudlicka, 2002, Zhang and Lee, 2009a), using probability theory to estimate the human emotional state by checking the presence or absence of a certain emotion. Yet, the tools of probability theory are still insufficient to handle all the facets of uncertainty (Dubois & Prade, 2001). Thus, fuzzy set theory provides a systematic approach to process linguistically uncertain information, just as humans are able to interpret imprecise and incomplete information. In order to incorporate human expertise, we use fuzzy C-means clustering (FCM) to assign one natural image into several different groups to a degree specified by a membership grade (Jang, Sun, & Mizutani, 1997). The FCM is used to cluster each component to get different emotional descriptors, and these descriptors are combined together to formalize the fuzzy-GIST to construct the emotional feature space for human emotion recognition (Zhang & Lee, 2009b). The fuzzy-GIST of an image is then used as the input of a neuro-fuzzy inference system, and the proposed system learns to understand the human affective states and develop its mental ability to understand complex emotions when interacting with the subject.
To model artificial emotion for a robot in the developmental context, the system needs to be built with the capability of interacting with humans, including the ability to analyze the visual features of natural scenes and human EEG characteristics. Emotional relevant features are firstly clustered into two categories with degrees of belongingness to each cluster to initialize the membership functions of neuro-fuzzy system (Jang et al., 1997). The IF–THEN rules of a neuro-fuzzy system to understand the positive and negative human emotions will be constructed by interacting with human. Then the system considers arousal indicator to extend the number of understandable emotion. Through the time, the system sub-clusters the emotional features so that the number of membership function of the neuro-fuzzy network will increase to incorporate more complicated human expertise considering more human emotions. Using such a developmental process, the proposed system can develop a mental ability to understand more complex human emotions by mining the characteristics of emotional features and interacting with its environment.
This paper analyzes the emotional space and proposes a new developmental emotion recognition system, which uses both emotional EEG information and visual information to extract the emotional factors of natural scenes at a semantic level and construct the feature space to conduct the human emotion recognition and its mental development. Based on the human subject feedback feelings evoked by natural scenes, the neuro-fuzzy inference system is adopted to learn the fuzzy-GIST as emotional feature at semantic level, and development its capability to understand the four different emotions including joy, pleasure, anger, and sadness.
The remainder of this paper is organized as follows. The proposed emotion development system will be presented and the methods used for this study will be introduced in the next section. In Section 3, we will give the experiment results and evaluate the performance of the proposed system. Some final conclusions and discussions are given in the last section.
Section snippets
Emotion development system
The configuration of the proposed system is shown in Fig. 1. The proposed emotion development system includes feature extraction, unsupervised clustering, and an adaptive neuro-fuzzy inference system (ANFIS) (Jang et al., 1997). Various images were selected from the International Affective Pictures System (IAPS) (Lang, Bradley, & Cuthbert, 2005). In the experiment, the system extracts visual information together with EEG features. In the early stage, clustering is performed to form primitive
Database
Color pictures were selected from the International Affective Picture System (IAPS) (Lang et al., 2005) as visual stimuli in the simulation. We chose 110 images from IAPS database. It is not allowed to show IAPS data in the publication, so we demonstrate some images similar in Fig. 8. Since the human emotion is subjective, we used each subject’s feedback including valence and arousal score to classify these 110 images into four groups, joy, pleasure, anger, and sadness. And for each emotional
Conclusion and discussion
A novel developmental scheme for analyzing the subject specific emotion reflected by a natural scene was proposed, in which the emotional feature space is built based on the fuzzy-GIST. According to the relationship between emotional factors and the characteristics of image, L*C*H* color and orientation information are adopted to analyze the visual features at semantic level, by incorporating the fuzzy concept to extract features with semantic meanings. Furthermore, after various pre-processing
Acknowledgements
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2010-0016927) (50%). It was also supported by the Converging Research Center Program funded by the Ministry of Education, Science and Technology (2010K001130) (30%), and the Original Technology Research Program for Brain Science through the National Research Foundation of Korea (NRF) funded by the Ministry of
References (51)
- et al.
Event-related oscillations are ‘real brain responses’ – Wavelet analysis and new strategies
International Journal of Psychophysiology
(2001) - et al.
Now comes the time to defuzzify neuro-fuzzy models
Fuzzy Sets and Systems
(1997) - et al.
Possibility theory is not fully compositional!: A comment on a short note by H.J. Greenberg
Fuzzy Sets and Systems
(1998) - et al.
Reliability of cortical activity during natural stimulation
Trends in Cognitive Sciences
(2010) - et al.
Processing of affective pictures modulates right-hemispheric gamma band eeg activity
Clinical Neurophysiology
(1999) - et al.
Building the gist of a scene: The role of global image features in recognition
Progress in Brain Research
(2006) - et al.
Structure identification of fuzzy model
Fuzzy Sets and Systems
(1988) - et al.
Analysis of positive and negative emotions in natural scene using brain activity and gist
Neurocomputing
(2009) - et al.
Semantic annotation of sports videos
IEEE Multimedia
(2002) - Bezdek, J. (1973). Fuzzy mathematics in pattern classification. Ph.D. thesis. Cornell University...
Introduction to wavelets and wavelets transforms
A study of colour emotion and colour preference. Part I: Colour emotions for single colours
Color Research and Application
Semantics in visual information retrieval
IEEE Multimedia
Subcortical and cortical brain activity during the feeling of self-generated emotions
Nature Neuroscience
Possibility theory, probability theory and multiple-valued logics: A clarification
Annals of Mathematics and Artificial Intelligence
Emotional face expressions are differentiated with brain oscillations
International Journal of Psychophysiology
Movement-produced stimulation in the development of visually guided behavior
Journal of Comparative and Physiological Psychology
Art of color
Neuro-fuzzy and soft computing: A computational approach to learning and machine intelligence
Principles of neural science
Cited by (33)
Effect of brightness of visual stimuli on EEG signals
2020, Behavioural Brain ResearchCitation Excerpt :There have been many studies in the literature using IAPS and similar image. These studies were mostly in the fields of visual processing, emotion classification, emotion evaluation and neuro-marketing [4,5,9,40,61–64]. Most of these studies did not consider the brightness of visual stimuli.
Emotion recognition based on 3D fuzzy visual and EEG features in movie clips
2014, NeurocomputingCitation Excerpt :FCM clustering, which employs fuzzy partitioning such that a given data point can be classified into several different groups to a degree specified by a membership grade, is used to obtain the fuzzy GIST of an image. An autonomous emotion recognition system was developed based on a fuzzy GIST, which can analyze and understand emotions related to natural scene images and develop itself by interactions with humans to understand more complicated human emotions [22–24]. However, video stimuli comprise successive image frames containing various types of affective information depending on the time sequence.
Electroencephalograph Emotion Classification Using a Novel Adaptive Ensemble Classifier Considering Personality Traits
2023, Basic and Clinical NeuroscienceDetermining the effect of emotional images brightness on EEG signals by classification algorithms
2022, Multidimensional Systems and Signal ProcessingEmotional Landscape Image Generation Using Generative Adversarial Networks
2021, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)