Affect Sensing Using Linguistic, Semantic and Cognitive Cues in Multi-threaded Improvisational Dialogue

In this paper, a context-based affect detection component embedded in an improvisational virtual platform is implemented. The software allows up to five human characters and one intelligent agent to be engaged in one session to conduct creative improvisation within loose scenarios. The transcripts produced showed several conversations being conducted in parallel. Some of these conversations reveal personal subjective opinions or feelings about situations, while others are caused by social interactions and show opinions and emotional responses to other participant characters. These two types of conversations serve to inform the descriptions of the personal and the social contexts, respectively. In order to detect affect from such contexts, first of all a naïve Bayes classifier is used to categorize these two types of conversations based on linguistic cues. A semantic-based analysis is also used to further derive the discussion themes and identify the target audiences for the social interaction inputs. Then, two statistical approaches have been developed to provide affect detection, respectively, in the social and personal emotion contexts. The emotional history of each individual character is used in interpreting affect relating to the personal contexts, while the social context affect detection takes account of interpersonal relationships, sentence types, emotions implied by the potential target audiences in their most recent interactions and discussion themes. The new development of context-based affect detection is integrated with the intelligent agent. The work addresses one challenging cognitive topic in the affective computing field, the detection and revealing of the relevant “context” to inform affect detection. The work addresses the journal’s themes on human emotion behavior analysis and understanding.