Affect Recognition for Multimodal Natural Language Processing

Excerpt

Language is inherently multimodal. It has many forms of appearance, like speech, gestures, facial expressions, and head-nods. In an ideal human-machine conversational system, machines should understand this multimodal language. Understanding human language also largely depends on the machines’ ability to interpret emotions. Emotional sensitivity can prevent desultory answers provided by these machines, thus making conversations more natural and engaging. For us humans, emotions aid our learning, communication, and decision-making. Hence, over the past two decades, there has been a significant effort to incorporate cognitive capabilities into machines so that they can interpret, comprehend and express emotions. Computational analysis of human multimodal language is an emerging research area in natural language processing (NLP). It expands the horizons of NLP to study language used in face to face communication and in online multimedia. This form of language contains modalities of language (in terms of spoken text), visual (in terms of gestures and facial expressions) and acoustic (in terms of changes in the voice tone). At its core, this research area is focused on modeling the three modalities and their complex interactions. This special issue on Affect Recognition in Human Multimodal Language aims to facilitate the growth of this new research direction in the community. The challenges of modeling human multimodal language can be split into two major categories: (1) studying each modality individually and modeling each in a manner that can be linked to other modalities (also known as intramodal dynamics, (2) linking the modalities by modeling the interactions between them (also known as intermodal dynamics). Common forms of these interactions include complementary or correlated information across modes. Intrinsic to each modality, modeling human multimodal language is complex due to factors such as idiosyncrasy in communicative styles, non-trivial alignment between modalities and unreliable or contradictory information across modalities. Therefore, computational analysis of multimodal language becomes a challenging research area. This special issue aimed at bringing together contributions from both academics and practitioners in the context of affect recognition in multimodal language with a primary focus on: …