Multimodal Expressive Embodied Conversational Agent Design

Embodied Conversational Agent (ECA) is a term encompassing virtual agents designed to converse with a human user, with a physical representation in its virtual environment. Other types of conversational agents like chatbots feature other functionalities like an image for text-based communication. However, nothing has been proposed to exploit a media such as an image, in the case of Embodied Conversational Agents, which would emphasize said media in the dialogue with an embodied character that can react to its content and display emotions. We propose a design for an Embodied Conversational Agent with multi-modal perception, able to express emotions, with which the conversation revolves around a media available to both the agent and the user. Using a BERT-based model, emotion classification of the user’s words and of his/her facial expressions captured through a webcam are combined to select an expression and an answer on a turn-based basis. The agent features both real-time lip-syncing and expression animation. The application case for the study is a discussion revolving around images of paintings, where the user wants to know more details about the artwork.