Skip to main content

Table of Contents


Rendering and Analysis

Simulation Level of Detail for Virtual Humans

Graphical level of detail (LOD) is a set of techniques for coping with the issue of limited computational resources by reducing the graphical detail of the scene far from the observer. Simulation LOD reduces quality of the simulation at the places unseen. Contrary to graphical LOD, simulation LOD has been almost unstudied. As a part of our on-going effort on a large virtual-storytelling game populated by tens of complex virtual humans, we have developed and implemented a set of simulation LOD algorithms for simplifying virtual space and behaviour of virtual humans. The main feature of our technique is that it allows for several degrees of detail, i.e. for


varying of simulation quality. In this paper, we summarise the main lessons learned, introduce the prototype implementation called IVE and discuss the possibility of scaling our technique to other applications featuring virtual humans.

Cyril Brom, Ondřej Šerý, Tomáš Poch

Towards Natural Gesture Synthesis: Evaluating Gesture Units in a Data-Driven Approach to Gesture Synthesis

Virtual humans still lack naturalness in their nonverbal behaviour. We present a data-driven solution that moves towards a more natural synthesis of hand and arm gestures by recreating gestural behaviour in the style of a human performer. Our algorithm exploits the concept of gesture units to make the produced gestures a continuous flow of movement. We empirically validated the use of gesture units in the generation and show that it causes the virtual human to be perceived as more natural.

Michael Kipp, Michael Neff, Kerstin H. Kipp, Irene Albrecht

3D Audiovisual Rendering and Real-Time Interactive Control of Expressivity in a Talking Head

The integration of virtual agents in real-time interactive virtual applications raises several challenges. The rendering of the movements of the virtual character in the virtual scene (locomotion of the character or rotation of its head) and the binaural rendering in 3D of the synthetic speech during these movements need to be spatially coordinated. Furthermore, the system must enable real-time adaptation of the agent’s expressive audiovisual signals to user’s on-going actions. In this paper, we describe a platform that we have designed to address these challenges as follows: (1) the modules enabling real time synthesis and spatial rendering of the synthetic speech, (2) the modules enabling 3D real time rendering of facial expressions using a GPU-based 3D graphic engine, and (3) the integration of these modules within an experimental platform using gesture as an input modality. A new model of phoneme-dependent human speech directivity patterns is included in the speech synthesis system, so that the agent can move in the virtual scene with realistic 3D visual and audio rendering. Future applications of this platform include perceptual studies about multimodal perception and interaction, expressive real time question and answer system and interactive arts.

Jean-Claude Martin, Christophe d’Alessandro, Christian Jacquemin, Brian Katz, Aurélien Max, Laurent Pointal, Albert Rilliard

Semantic Segmentation of Motion Capture Using Laban Movement Analysis

Many applications that utilize motion capture data require small, discrete, semantic segments of data, but most motion capture collection processes produce long sequences of data. The smaller segments are often created from the longer sequences manually. This segmentation process is very laborious and time consuming. This paper presents an automatic motion capture segmentation method based on movement qualities derived from Laban Movement Analysis (LMA). LMA provides a good compromise between high-level semantic features, which are difficult to extract for general motions, and low-level kinematic features which, often yield unsophisticated segmentations. The LMA features are computed using a collection of neural networks trained with temporal variance in order to create a classifier that is more robust with regard to input boundaries. The actual segmentation points are derived through simple time series analysis of the LMA features.

Durell Bouchard, Norman Badler

Culture and Identity

A Computational Model of Culture-Specific Conversational Behavior

This paper presents a model for simulating cultural differences in the conversational behavior of virtual agents. The model provides parameters for differences in proxemics, gaze and overlap in turn taking. We present a review of literature on these factors and show results of a study where native speakers of North American English, Mexican Spanish and Arabic were asked to rate the realism of the simulations generated based on different cultural parameters with respect to their culture.

Dušan Jan, David Herrera, Bilyana Martinovski, David Novick, David Traum

Ethnic Identity and Engagement in Embodied Conversational Agents

In this paper we present the design, development and initial evaluation of a virtual peer that models ethnicity through culturally authentic verbal and non-verbal behaviors. The behaviors chosen for the implementation come from an ethnographic study with African-American and Caucasian children and the evaluation of the virtual peer consists of a study in which children interacted with an African American or a Caucasian virtual peer and then assessed its ethnicity. Results suggest that it may be possible to tip the ethnicity of a embodied conversational agent by changing verbal and non-verbal behaviors instead of surface attributes, and that children engage with those virtual peers in ways that have promise for educational applications.

Francisco Iacobelli, Justine Cassell

Neuroticism – A Competitive Advantage (Also) for IVAs?

Real-Time Strategy (RTS) games are a challenging genre for the design of Intelligent Virtual Agents. We were interested whether incorporating a simple emotional model to an existing bot-script improves playing strength. We implemented a bot for Microsoft’s RTS game “Age of Mythology”. The emotional model is based on the “Big-Five” and the Emotion-Connectionist Model. Four variants of the bot were evaluated, each using different personality parameters; one of the variants was designed to show “neurotic” behaviour. The emotion-driven bots were evaluated in a bot-versus-bot setup, playing matches against the game’s default script. Evaluation results indicate a significant increase in playing strength, the “neurotic” bot being the strongest one.

Christoph Hermann, Helmuth Melcher, Stefan Rank, Robert Trappl

Behavior Models

Emotionally Expressive Head and Body Movement During Gaze Shifts

The current state of the art virtual characters fall far short of characters produced by skilled animators. One reason for this is that the physical behaviors of virtual characters do not express the emotions and attitudes of the character adequately. A key deficiency possessed by virtual characters is that their gaze behavior is not emotionally expressive. This paper describes work on expressing emotion through head movement and body posture during gaze shifts, with intent to integrate a model of emotionally expressive eye movement into this work in the future. The paper further describes an evaluation showing that users can recognize the emotional states generated by the model.

Brent Lance, Stacy C. Marsella

Fuzzy Similarity of Facial Expressions of Embodied Agents

In this paper we propose an algorithm based on fuzzy similarity which models the concept of resemblance between facial expressions of an Embodied Conversational Agent (ECA). The algorithm measures the degree of visual resemblance between any two facial expressions. We also present an evaluation study in which we compared the users’ perception of similarity of facial expressions. Finally we describe an application of this algorithm to generate complex facial expressions of an ECA.

Radosław Niewiadomski, Catherine Pelachaud

The Behavior Markup Language: Recent Developments and Challenges

Since the beginning of the SAIBA effort to unify key interfaces in the multi-modal behavior generation process, the Behavior Markup Language (BML) has both gained ground as an important component in many projects worldwide, and continues to undergo further refinement. This paper reports on the progress made in the last year in further developing BML. It discusses some of the key challenges identified that the effort is facing, and reviews a number of projects that already are making use of BML or support its use.

Hannes Vilhjálmsson, Nathan Cantelmo, Justine Cassell, Nicolas E. Chafai, Michael Kipp, Stefan Kopp, Maurizio Mancini, Stacy Marsella, Andrew N. Marshall, Catherine Pelachaud, Zsofi Ruttkay, Kristinn R. Thórisson, Herwin van Welbergen, Rick J. van der Werf

Dynamic Behavior Qualifiers for Conversational Agents

We aim at defining conversational agents that exhibit qualitatively distinctive behaviors. To this aim we provide a small set of parameters to allow one to define behavior profiles and then leave to the system the task of animating the agent. Our approach is to manipulate the behavior tendency of the agent depending on its communicative intention and emotional state.

In this paper we will define the concepts of




. The Baseline of an agent is defined as a set of fixed parameters that represent the personalized agent behavior, while the Dynamicline, is a set of parameters values that derive both from the Baseline and the current communicative goals and emotional state.

Maurizio Mancini, Catherine Pelachaud

Feedback Models

Creating Rapport with Virtual Agents

Recent research has established the potential for virtual characters to establish rapport with humans through simple contingent nonverbal behaviors. We hypothesized that the contingency, not just the frequency of positive feedback is crucial when it comes to creating rapport. The primary goal in this study was evaluative: can an agent generate behavior that engenders feelings of rapport in human speakers and how does this compare to human generated feedback? A secondary goal was to answer the question: Is contingency (as opposed to frequency) of agent feedback crucial when it comes to creating feelings of rapport? Results suggest that contingency matters when it comes to creating rapport and that agent generated behavior was as good as human listeners in creating rapport. A “virtual human listener” condition performed worse than other conditions.

Jonathan Gratch, Ning Wang, Jillian Gerten, Edward Fast, Robin Duffy

Incremental Multimodal Feedback for Conversational Agents

Just like humans, conversational computer systems should not listen silently to their input and then respond. Instead, they should enforce the speaker-listener link by attending actively and giving feedback on an utterance while perceiving it. Most existing systems produce direct feedback responses to decisive (e.g. prosodic) cues. We present a framework that conceives of feedback as a more complex system, resulting from the interplay of conventionalized responses to eliciting speaker events and the multimodal behavior that signals how internal states of the listener evolve. A model for producing such incremental feedback, based on multi-layered processes for perceiving, understanding, and evaluating input, is described.

Stefan Kopp, Thorsten Stocksmeier, Dafydd Gibbon

Searching for Prototypical Facial Feedback Signals

Embodied conversational agents should be able to provide feedback on what a human interlocutor is saying. We are compiling a list of facial feedback expressions that signal attention and interest, grounding and attitude. As expressions need to serve many functions at the same time and most of the component signals are ambiguous, it is important to get a better idea of the many to many mappings between displays and functions. We asked people to label several dynamic expressions as a probe into this semantic space. We compare simple signals and combined signals in order to find out whether a combination of signals can have a meaning on its own or not, i. e. the meaning of single signals is different from the meaning attached to the combination of these signals. Results show that in some cases a combination of signals alters the perceived meaning of the backchannel.

Dirk Heylen, Elisabetta Bevacqua, Marion Tellier, Catherine Pelachaud

Fluid Semantic Back-Channel Feedback in Dialogue: Challenges and Progress

Participation in natural, real-time dialogue calls for behaviors supported by perception-action cycles from around 100 msec and up. Generating certain kinds of such behaviors, namely envelope feedback, has been possible since the early 90s. Real-time backchannel feedback related to the content of a dialogue has been more difficult to achieve. In this paper we describe our progress in allowing virtual humans to give rapid within-utterance content-specific feedback in real-time dialogue. We present results from human-subject studies of content feedback, where results show that content feedback to a particular phrase or word in human-human dialogue comes 560-2500 msec from the phrase’s onset, 1 second on average. We also describe a system that produces such feedback with an autonomous agent in limited topic domains, present performance data of this agent in human-agent interactions experiments and discuss technical challenges in light of the observed human-subject data.

Gudny Ragna Jonsdottir, Jonathan Gratch, Edward Fast, Kristinn R. Thórisson


T2D: Generating Dialogues Between Virtual Agents Automatically from Text

The Text2Dialogue (T2D) system that we are developing allows digital content creators to generate attractive multi-modal dialogues presented by two virtual agents—by simply providing textual information as input. We use Rhetorical Structure Theory (RST) to decompose text into segments and to identify rhetorical discourse relations between them. These are then “acted out” by two 3D agents using synthetic speech and appropriate conversational gestures. In this paper, we present version 1.0 of the T2D system and focus on the novel technique that it uses for mapping rhetorical relations to question–answer pairs, thus transforming (monological) text into a form that supports dialogues between virtual agents.

Paul Piwek, Hugo Hernault, Helmut Prendinger, Mitsuru Ishizuka

So Let’s See: Taking and Keeping the Initiative in Collaborative Dialogues

In order to create and maintain social relationships with human users in mixed-initiative dialogues, IVAs have to give off coherent signals of claiming or relinquishing leadership in discourse. Quantitaive and qualitative analyses of human-human collaborative task-solving dialogues from the Ohio State University Quake Corpus reveal that discursive dominance is a shared achievement of speakers and given, taken or kept in a consensual way, up to the point where they incur “costs” in terms of efficiency in solving the task. Some verbal signals can be identified as relevant to this process.

Sabine Payr


Health Document Explanation by Virtual Agents

We describe the design and evaluation of a virtual agent that explains health documents to patients. The prevalence and impact of low health literacy is presented as a motivation for such agents, given that face-to-face interaction with health providers is cited as one of the most effective means of communicating with these patients. We analyze the form and distribution of pointing gestures used by experts in explaining health documents, and use this data to develop a computational model of agent-based document explanation. This model is evaluated in a randomized controlled trial. Preliminary results indicate that patients with low health literacy are more satisfied with health document explanation by a virtual agent compared to a human.

Timothy W. Bickmore, Laura M. Pfeifer, Michael K. Paasche-Orlow

Virtual Patients for Clinical Therapist Skills Training

Virtual humans offer an exciting and powerful potential for rich interactive experiences. Fully embodied virtual humans are growing in capability, ease, and utility. As a result, they present an opportunity for expanding research into burgeoning virtual patient medical applications. In this paper we consider the ways in which one may go about building and applying virtual human technology to the virtual patient domain. Specifically we aim to show that virtual human technology may be used to help develop the interviewing and diagnostics skills of developing clinicians. Herein we proffer a description of our iterative design process and preliminary results to show that virtual patients may be a useful adjunct to psychotherapy education.

Patrick Kenny, Thomas D. Parsons, Jonathan Gratch, Anton Leuski, Albert A. Rizzo

Integrating a Virtual Agent into the Real World: The Virtual Anatomy Assistant Ritchie

Augmented realities, which are partially real and partially virtual, open up new ways for humans to interact with Embodied Conversational Agents (ECAs) since they allow users to meet ECAs in the physical space. Nevertheless, attempts to integrate ECAs as digital overlays in a physical space have been rare. Obvious reasons are the high demands such an integration puts to the animation of ECAs as virtual augmentations of the physical space, their capabilities to perceive not only the virtual, but also the physical world as well as reactive behavior control. In this paper, we describe our technical contributions towards solving these challenges. To illustrate our ideas, we present the virtual anatomy assistant Ritchie that monitors the user’s actions in a physical space and dynamically responds to them.

Volker Wiendl, Klaus Dorfmüller-Ulhaas, Nicolas Schulz, Elisabeth André

Proactive Authoring for Interactive Drama: An Author’s Assistant

Interactive drama allows people to participate actively in a dynamically unfolding story, by playing a character or by exerting directorial control. One of the central challenges faced in the design of interactive dramas is how to ensure that the author’s goals for the user’s narrative experience are achieved in the face of the user’s actions in the story. This challenge is especially significant when a variety of users are expected. To address this challenge, we present an extension to Thespian, an authoring and simulating framework for interactive dramas. Each virtual character is controlled by a decision-theoretic goal driven agent. In our previous work on Thespian, we provided a semi-automated authoring approach that allows authors to configure virtual characters’ goals through specifying story paths. In this work, we extend Thespian into a more proactive authoring framework to further reduce authoring effort. The approach works by simulating potential users’ behaviors, generating corresponding story paths, filtering the generated paths to identify those that seem problematic and prompting the author to verify virtual characters’ behaviors in them. The author can correct virtual characters’ behaviors by modifying story paths. As new story paths are designed by the author, the system incrementally adjusts virtual characters’ configurations to reflect the author’s design ideas. Overall, this enables interactive testing and refinement of an interactive drama. The details of this approach will be presented in this paper, followed by preliminary results of applying it in authoring an interactive drama.

Mei Si, Stacy C. Marsella, David V. Pynadath


The Effects of an Embodied Conversational Agent’s Nonverbal Behavior on User’s Evaluation and Behavioral Mimicry

Against the background that recent studies on embodied conversational agents demonstrate the importance of their behavior, an experimental study is presented that assessed the effects of different nonverbal behaviors of an embodied conversational agent on the users´ experiences and evaluations as well as on their behavior. 50 participants conducted a conversation with different versions of the virtual agent Max, whose nonverbal communication was manipulated with regard to eyebrow movements and self-touching gestures. In a 2x2 between subjects design each behavior was varied in two levels: occurrence of the behavior compared to the absence of the behavior. Results show that self-touching gestures compared to no self-touching gestures have positive effects on the experiences and evaluations of the user, whereas eyebrow raising evoked less positive experiences and evaluations in contrast to no eyebrow raising. The nonverbal behavior of the participants was not affected by the agent’s nonverbal behavior.

Nicole C. Krämer, Nina Simons, Stefan Kopp

Spatial Social Behavior in Second Life

We have developed software bots that inhabit the popular online social environment SecondLife (SL). Our bots can wander around, collect data, engage in simple interactions, and carry out simple automated experiments. In this paper we use our bots to study spatial social behavior. We found an indication that SL users display distinct spatial behavior when interacting with other users. In addition, in an automated experiment carried out by our bot, we found that users, when their avatars were approached by our bot, tended to respond by moving their avatar, further indicating the significance of proxemics in SL.

Doron Friedman, Anthony Steed, Mel Slater

Generating Embodied Descriptions Tailored to User Preferences

We describe two user studies designed to measure the impact of using the characteristic displays of a speaker expressing different user-preference evaluations to select the head and eye behaviour of an animated talking head. In the first study, human judges were reliably able to identify positive and negative evaluations based only on the motions of the talking head. In the second study, subjects generally preferred positive displays to accompany positive sentences and negative displays to accompany negative ones, and showed a particular dislike for negative facial displays accompanying positive sentences.

Mary Ellen Foster

Gaze Models

Scrutinizing Natural Scenes: Controlling the Gaze of an Embodied Conversational Agent

We present here a system for controlling the eye gaze of a virtual embodied conversational agent able to perceive the physical environment in which it interacts. This system is inspired by known components of human visual attention system and reproduces its limitations in terms of visual acuity, sensitivity to movement, limitations of short-memory and object pursuit. The aim of this coupling between animation and visual scene analysis is to provide sense of presence and mutual attention to human interlocutors. After a brief introduction to this research project and a focused state of the art, we detail the components of our system and confront simulation results to eye gaze data collected from viewers observing the same natural scenes.

Antoine Picot, Gérard Bailly, Frédéric Elisei, Stephan Raidt

Attentive Presentation Agents

The paper describes an infotainment application where life-like characters present two MP3 players in a virtual showroom. The key feature of the system is that the presenter agents analyze the user’s gaze-behavior in real-time and may thus adapt the presentation flow accordingly. In particular, a user’s (non-)interest in interface objects and also preference in decision situations is estimated automatically by just using eye gaze as input modality. A formal study was conducted that compared two versions of the application. Results indicate that attentive presentation agents support successful grounding of deictic agent gestures and natural gaze behavior.

Tobias Eichner, Helmut Prendinger, Elisabeth André, Mitsuru Ishizuka

The Rickel Gaze Model: A Window on the Mind of a Virtual Human

Gaze plays a large number of cognitive, communicative and affective roles in face-to-face human interaction. To build a believable virtual human, it is imperative to construct a gaze model that generates realistic gaze behaviors. However, it is not enough to merely imitate a person’s eye movements. The gaze behaviors should reflect the internal states of the virtual human and users should be able to derive them by observing the behaviors. In this paper, we present a gaze model driven by the cognitive operations; the model processes the virtual human’s reasoning, dialog management, and goals to generate behaviors that reflect the agent’s inner thoughts. It has been implemented in our virtual human system and operates in real-time. The gaze model introduced in this paper was originally designed and developed by Jeff Rickel but has since been extended by the authors.

Jina Lee, Stacy Marsella, David Traum, Jonathan Gratch, Brent Lance


Embodied Creative Agents: A Preliminary Social-Cognitive Framework

The goal of this paper is to open discussion about industrial creativity as a potential application field for Embodied Conversational Agents. We introduce the domain of creativity and especially focus on a collective creativity tool, the brainstorming: we present the related research in Psychology which has identified several key cognitive and social mechanisms that influence brainstorming process and outcome. However, some dimensions remain unexplored, such as the influence of the partners’ personality or the facilitator’s personality on idea generation. We propose to explore these issues, among others, using Embodied Conversational Agents. The idea seems original given that Embodied Agents were never included into brainstorming computer tools. We draw some hypotheses and a research program, and conclude on the potential benefits for the knowledge on creativity process on the one hand, and for the field of Embodied Conversational Agents on the other hand.

Stéphanie Buisine, Améziane Aoussat, Jean-Claude Martin

Feel the Difference: A Guide with Attitude!

This paper describes a mobile context-aware ‘intelligent affective guide with attitude’ that guides visitors touring an outdoor attraction. Its behaviour is regulated by a biologically inspired architecture of emotion, allowing it to adapt to the user’s needs and feelings. In addition to giving an illusion of life, the guide emulates a real guide’s behaviour by presenting stories based on the user’s interests, its own interests, its belief and its current memory activation. A brief description of the system focusing on the core element - the guide’s emotional architecture - is given followed by findings from an evaluation with real users.

Mei Yii Lim, Ruth Aylett

It’s All in the Anticipation

Since the beginnings of character animation,


has been an effective part of the repertoire of tricks used to create believable animated characters. However, anticipation has had but a secondary role in the creation of synthetic virtual life forms. In this paper, we describe how a simple anticipatory mechanism that generates an affective signal resulting from the mismatch between sensed and predicted values — the


— can help in the creation of consistent believable behaviour for intelligent virtual characters.

Carlos Martinho, Ana Paiva

Incorporating Emotion Regulation into Virtual Stories

This paper presents an approach to incorporate emotion regulation as addressed within psychology literature into virtual characters. To this end, first Gross’ informal theory of emotion regulation has been formalised using a dynamical system style modelling approach. Next, a virtual environment has been created, involving a number of virtual agents, which have been equipped with the formalised model for emotion regulation. This environment has been used to successfully generate a number of emergent virtual stories, in which characters regulate their emotions by applying regulation strategies such as situation selection and attentional deployment. The behaviours shown in the stories were found consistent with descriptions of human regulation processes.

Tibor Bosse, Matthijs Pontier, Ghazanfar F. Siddiqui, Jan Treur

Poster Session

Endowing Emotional Agents with Coping Strategies: From Emotions to Emotional Behaviour

Emotion takes an increasingly important place in the design of intelligent virtual agents. Designers of emotional agents build on theories from cognitive psychology, that describe the cognitive functioning of emotions with two indivisible processes [1,2]: the


process triggers emotions, in particular intense negative emotions to point out threatening stimuli, and the


process modifies the behaviour to manage these stimuli. Nevertheless, among the existing emotional agents, a lot express emotions triggered by an appraisal process [3] but few have a coping process allowing their emotions to impact their behaviour [4,5,6]. In previous work [7] we provided a formalization of Ortony et al.’s appraisal process [8] in a BDI logic,

et al.’s

a logic of mental attitudes. The next step is to formalize the coping process in the same framework. Our aim here is to provide the theoretical basis of an agent architecture rather than an implementation. We only give here an overview of our framework (


. [9,10] for more details).

Carole Adam, Dominique Longin

From IVAs to Comics Generating Comic Strips from Emergent Stories with Autonomous Characters

Emergent narrative systems create stories which are always different from each other. Creating summaries of these stories is a challenge especially if we want to capture the richness of the characters. Our goal is to automatically generate summaries from emergent narrative using comics as the visual medium for the summary. We identify the most important situations in the story log looking at the emotional state of the characters, transform the resulting summary into a comics description and create the comic. We believe that a good summarization of a story that maintains the emotions of the characters together with an expressive visual representation is essential for the user to remember the story.

Tiago Alves, Ana Simões, Marco Vala, Ana Paiva, Adrian McMichael, Ruth Aylett

Emotional Agents with Team Roles to Support Human Group Training

In the teamwork research area there is an increasing interest about the principles behind team effectiveness and effective team training; for Intelligent Virtual Agents (IVAs) Team Training is an excellent application area; nevertheless, the few reported works about IVAs in team training, illustrate both the use for the individualized teaching (Pedagogical Agents) of procedural tasks and the substitution of missing team members (Teammate Agents) to promote the practice of team tasks in relation to functional roles (Taskwork) [1].

Our interest on Intelligent Virtual Environments for Training (IVETs) has led us to propose a Team Training Strategy (TTS) whose purpose is to promote social skills as well as knowledge and skills related to tasks of socio technical nature. The alternatives that we are evaluated to improve the performance of human groups and to promote effective teams deal with: the use of scaffolding as the best tutoring approach, the promotion of social skills before technical skills, and especially, the selection of the best nonfunctional roles (team roles) balance according to the task.

In addition, our aim is to incorporate into an IVA called Pancho (Pedagogical AgeNt to support Colaborative Human grOups) the particular behaviors of Team Roles defined by Belbin [2]; Pancho, with a selected team role —according to a team model— will join the human group with the intention of improving the performance of the team (Teamwork) and providing scaffolding to the trainees (Taskwork). The Belbin’s categorization is the earliest and still the most popular. He states that the team role can be defined as a tendency to behave, contribute and interrelate with each others at work in a certain distinctive ways; he also states that in teamwork, a good mix of team roles in the group is necessary for groups to use their technical skills optimally. The team roles defined by Belbin have very particular behaviors; we have selected a generic cognitive architecture for agents with emotionally influenced behaviors —called COGNITIVA— to realize those roles [3]. The constructs provided by this architecture (Personal traits, Concerns, Moods, Attitudes and Physical states) are being properly instantiated to generate the desired behaviors.

Raúl A. Aguilar, Angélica de Antonio, Ricardo Imbert

Context Awareness in Mobile Relational Agents

The development of virtual agents designed to draw users into personal and professional relationships with them represents a growing area of research [1]. Mobility and context awareness represent important directions of research for these relational agents, since they offer unique affordances for relationship development. A mobile/wearable agent has the potential to be with a user for a significant period of time, and frequency of contact alone has been shown to be associated with increased solidarity between people. The ability to sense some aspects of the user’s environment (context awareness) may also provide mobile agents with unique relational affordances. Automatically recognizing and commenting on situations in the user’s life can amplify many relational perceptions, including familiarity, common ground, solidarity and intimacy. In addition, an agent’s ability to proactively interrupt and help a user in a situation that is automatically sensed by the agent may lead to increased perceptions of trust and caring by the user.

Timothy W. Bickmore, Daniel Mauer, Thomas Brown

Implicit Training of Virtual Agents

This paper provides a brief overview of an implicit training method used for teaching autonomous agents to represent humans in 3D Virtual Worlds without any explicit training efforts being required.

Anton Bogdanovych, Marc Esteva, Simeon Simoff, Carles Sierra

Human Actors and Virtual Agents Playing Together to Transform Stage Direction Practices

In this article, we show how approaches based on interactive data-mining may inspire new conceptions of theatre staging. They may be applied thanks to virtual agent systems interacting with comedians. We then give an example of such a theatre production,

La Traversée de la nuit


Alain Bonardi, Christine Zeppenfeld

Towards Characters with a Full Episodic Memory

A typical present-day virtual actor is able to store episodes in an

ad hoc

manner, which does not allow for reconstructing the actor’s personal stories. We have prototyped a virtual RPG actor with a


episodic memory, which allows for this reconstruction. The paper overviews the work done and sketches the work in progress.

Cyril Brom, Klára Pešková, Jiří Lukavský

Towards Fast Prototyping of IVAs Behavior: Pogamut 2

We present the platform for IVAs development in the human like environment of the first-person shooter game Unreal Tournament 2004. This environment is extendible and supported by vast community of users. Based on our previous experience the problem of fast verification of models of artificial intelligence or IVAs is in implementation issues. The developer spends most of his time solving technical environment dependent issues and malfunctions, which drives him away from his goals. Therefore our modular platform provides a tool, which helps solving those problems and the developer can spend saved time by solving another AI based issues and model verification. The platform is aimed for research and educational purposes.

Ondřej Burkert, Rudolf Kadlec, Jakub Gemrot, Michal Bída, Jan Havlíček, Martin Dörfler, Cyril Brom

Towards a Multicultural ECA Tour Guide System

In this article we present an ongoing project in our research group, an ECA based multicultural tour guide system. Tour guide ECA agent provides information about the city of Dubrovnik and dynamically changes its behaviors among Japanese, Croatian and general western cultures speaking in English.

Aleksandra Cerekovic, Hung-Hsuan Huang, Igor S. Pandzic, Yukiko Nakano, Toyoaki Nishida

Towards the Specification of an ECA with Variants of Gestures

The animation of an ECA, for most of animation systems, implies that its behaviour is encoded in a representation language, giving a form for each modality of a behavior. Interested in gestures, several representation languages exist already that that usually give a physical description of the necessary information for the production and reproduction of gestures, and for its synchronization with the other modalities. In our work, we aim at enriching the SAIBA description of gestures with semantic considerations, enabling an ECA to use a gesture for a wide range of applications.

Nicolas Ech Chafai, Catherine Pelachaud, Danielle Pelé

AI-RPG Toolkit: Towards A Deep Model Implementation for Improvisational Virtual Drama

The form of improvisational drama allows participants to have their own choices to influence the ongoing story, and each play results in a different ending. However, authoring such story contents requires ad hoc scripting, and static story structures lose ingenuity once users hacked through them. Our purpose is to develop a toolkit for: (1) fast authoring the story content, and (2) allow it for repeated plays yet retaining fresh interactive experience. While most similar applications have explicit, sophisticated story structures to ensure the number of possible interactions and endings in specific situations, we argue that characters should have enough background knowledge to make any improvisational choices. The more knowledge they have, the more sophisticated course of actions they may express. As a result, we take a deep-model approach to implement virtual agents, allowing them to deliberate and act with established knowledge in unexpected situations.

Chung-Cheng Chiu, Edward Chao-Chun Kao, Paul Hsueh-Min Chang, Von-Wun Soo

Attention Based, Naive Strategies, for Guiding Intelligent Virtual Agents

The AtGentive project [1,2] focuses on the support of attention in learning environments. To achieve this objective the system analyses the learners’ computer activities and physical states and, on the basis of this analysis, it generates interventions. Such interventions either supply learners with information useful to support their current attentional focus, or are aimed at attracting the user’s attention to new foci.

Damien Clauzel, Claudia Roda, Laurent Ach, Benoît Morel

Behavioural Reactive Agents for Video Game Opponents with Personalities

Nowadays, the video gaming experience is shifting from merely realistic to believable. The behaviour of the computer driven player and non-playing characters is often poor when compared to their visual appearance. In this sense, there has been a recent interest in improving the video gaming experience with novel Artificial Intelligence (AI) techniques. This paper presents a robotics inspired behavioural AI to simulate characters’ personalities in a commercial video game.

Carlos Delgado-Mata, Jesús Ibáñez-Martínez

Adapting Hierarchical Social Organisation by Introducing Fear into an Agent Architecture

This paper considers possible affective roles in an agent-based social simulation, and in particular the effect of adding a simple model of fear into a replication of an agent-based social simulation.

Pablo Lucas dos Anjos, Ruth Aylett, Alison Cawsey

Roles of a Talking Head in a Cooperative Human-Robot Dialogue System

The JAST human-robot dialogue system [1] is designed as a platform to integrate empirical findings on cognition and cooperative dialogue with research on autonomous robots by supporting multimodal human-robot collaboration on a construction task. The robot consists of a pair of mechanical arms with grippers, mounted in a position to resemble human arms, and a Philips iCat animatronic talking head [2]. The user and the robot work together to assemble construction toys on a common work area, coordinating their actions through speech, gestures, and facial displays.

Mary Ellen Foster

Modeling Imitational Behavior Via Social Comparison Theory

(Extended Abstract)

Modeling crowd behaviors is an important challenge for intelligent virtual agents. We propose a general cognitive model of simulating crowd behaviors, based on Festinger’s Social Comparison Theory (SCT), a prominent social psychology theory. We present the use of the SCT model (using the Soar cognitive architecture) in the generation of imitational behavior in loosely-coupled groups and show that SCT generates behavior more in-tune with human crowd behavior.

Natalie Fridman, Gal A. Kaminka

Social Animation in Complex Environments

This work presents a market-based social model to produce good quality behavioral animations for groups of intelligent virtual agents. The social model coordinates the activities of groups of virtual characters and also includes social actions in the agent decision-making. We follow the Multi-Agent Resource Allocation approach presented in [2], where agents express their preferences using utility functions. The dynamics of social interactions is inspired by the theory of Piaget [3] over which we have implemented reciprocal task exchanges.

Francisco Grimaldo, Miguel Lozano, Fernando Barber

A Script Driven Multimodal Embodied Conversational Agent Based on a Generic Framework

Embodied Conversational Agents (ECAs) are life-like CG characters that interact with human users in face-to-face conversations. To achieve natural conversations, they need to understand the inputs from human users, deliberate the responding behaviors and realize those behaviors in multiple modalities. They are sophisticated, require numbers of building assemblies and are thus difficult for individual research groups to develop. To facilitate result sharing and rapid prototyping of ECA researches, a Generic ECA Framework that is meant to integrate ECA assemblies seamlessly is being developed by our group. This framework is composed of a low-level communication platform (GECA Platform), a set of communication API libraries (GECA Plugs) and a high-level protocol (GECA Protocol, GECAP).

Hung-Hsuan Huang, Aleksandra Cerekovic, Igor S. Pandzic, Yukiko Nakano, Toyoaki Nishida

A Quiz Game Console Based on a Generic Embodied Conversational Agent Framework

This article describes an attempt to build a quiz game kiosk for show-room use based on the Generic Embodied Conversational Agent (GECA) Framework [1] that provides a general purpose architecture for connecting modularized ECA functional components for multimodal human-agent interactions.

Hung-Hsuan Huang, Taku Inoue, Aleksandra Cerekovic, Igor S. Pandzic, Yukiko Nakano, Toyoaki Nishida

AVSML: An XML-Based Markup Language for Web Information Integration in 3D Virtual Space

3D virtual space can visually represent the spatial structure to users and it has been applied to many fields such as city planning, navigation, education, entertainment and so on. In the 3D virtual space, an agent can navigate a user in an interactive manner [5]. Various platforms to build a 3D virtual space and languages to control the agents have been proposed [1,2,3,4]. For example, VKSC (Virtual Kobe Sanda Campus) is a 3D virtual space of Kobe Sanda Campus, Kwansei Gakuin University [1]. In VKSC, an agent called Suzie guides a user in the campus upon his/her request.

Yasuhiko Kitamura, Yatsuho Shibata, Keisuke Tokuda, Kazuki Kobayashi, Noriko Nagata

Simulation Environment for Anticipatory Behaving Agents from the Artificial Life Domain

Our research is focused on simulation of agents - animates. The architecture of these agents is mainly inspired by nature therefore they are sometimes called artificial creatures. The main contribution of this paper is the description of designed simulation environment architecture for the Artificial Life (ALife) domain. It was named the World of Artificial Life (WAL). Our platform incorporates results of research in domain of hybrid agent architectures.

Karel Kohout, Pavel Nahodil

Towards an Architecture for Aligned Speech and Gesture Production

The automatic production of speech and gesture is one of the most challenging issues in building embodied conversational agents, due to the intricacy of how the two modalities seem to “align” with each other. Based on experiences from computational approaches and inspired by current theoretical modeling attempts, we propose an architecture to simulate how on-the-spot speech and gesture production might run and bring about more natural multimodal behavior.

Stefan Kopp, Kirsten Bergmann

Avatar Puppetry Using Real-Time Audio and Video Analysis

We present a system which consists of a lifelike agent animated in real-time using video and audio analysis from the user. This kind of system could be used for Instant Messaging where an avatar controlled like a puppet is displayed instead of the webcam flow. The overall system is made of video analysis based on Active Appearance Models and audio analysis based on Hidden Markov Model. The parameters from these two modules are sent to a control system driving the animation engine. The video analysis extracts the head orientation and the audio analysis provides the phonetic string used to move the lips.

Sylvain Le Gallou, Gaspard Breton, Renaud Séguier, Christophe Garcia

Double Appraisal for Synthetic Characters

The paper describes a double appraisal-based emotion system for synthetic characters. This approach gives intelligent agents the ability to make decisions with respect to the emotional states of others, thus implementing aspects of the theory of mind concept and laying the basis for cognitive empathy.

Sandy Louchart, Ruth Aylett, Joao Dias

An Expressive Avatar for Instant Messaging Endowed with Emotional Intelligence

In this paper, we propose to endow a graphical representation of a user in Instant Messaging – an avatar – with the ability to recognize and to express emotions and to play social nonverbal behaviour, on the basis of textual affect sensing and interpretation of communicative functions conveyed by online conversations. The developed Affect Analysis Model integrated with Instant Messaging (IM) media supports the recognition of affect from text to ensure avatar animation in an appropriate and expressive manner.

Alena Neviarouskaya, Helmut Prendinger, Mitsuru Ishizuka


An Architecture for Intelligent Affective Agents

One of the most important social ability for effective social interaction with people is the capacity to understand, feel and ultimately express emotions. In this paper we present an architecture, based on the BDI paradigm, employing a three layered approach and coupling an emotion engine which simulates the generation of affective states based on Scherer’s component process theory and influences decision making.

Marco Paleari, Brian Duffy, Benoit Huet

Towards a Unified Model of Social and Environment-Directed Agent Gaze Behaviour

When considering the gaze behaviours of embodied agents, and necessarily the underlying processes of visual attention that help to guide them, most work to date has been focused either on models aimed at controlling gaze in social situations, for example when one or more ECA’s and humans are interacting, or for controlling gaze directed at the environment in a more general sense, typically for helping AVA’s accomplish tasks such as navigation. We are designing a model of visual attention that attempts to amalgamate these concepts into a unified design in order to produce intelligent virtual agents that can behave in a social manner with social stimuli while also being attentive to events when mobile in the more general environment.

Christopher Peters

Automatic Generation of Expressive Gaze in Virtual Animated Characters: From Artists Craft to a Behavioral Animation Model

We present a model for automatic generation of expressive gaze in virtual agents. Our main focus is the eye behavior associated to expressiveness. Our approach is to collect data from animated Computer Graphics films, and codify such observations into an animation framework. The main contribution is the modeling aspects of an animation system, calibrated with empirical observations in order to generate realistic eyes motion. Results show that this approach generates convincing animations that improve the empathy of virtual agents.

Rossana B. Queiroz, Leandro M. Barros, Soraia R. Musse

Analyzing Gaze During Face-to-Face Interaction

We present here the analysis of multimodal data gathered during realistic face-to-face interaction of a target speaker with a number of interlocutors. Videos and gaze have been monitored with an experimental setup using coupled cameras and screens with integrated eye trackers. With the aim to understand the functions of gaze in social interaction and to develop a coherent gaze control model for our talking heads we investigate the influence of cognitive state and social role on the observed gaze behavior.

Stephan Raidt, Gérard Bailly, Frédéric Elisei

Affect and Metaphor in an ICA: Further Developments

We describe a computational treatment of certain sorts of affect-conveying metaphorical utterances. This is part of an affect detection system used by intelligent conversational agents (ICAs) operating in an edrama system.

C. J. Smith, T. H. Rumbell, J. A. Barnden, M. G. Lee, S. R. Glasbey, A. M. Wallington

A Case-Based Approach to Intelligent Virtual Agent’s Interaction Experience Representation

In this paper we describe a case-based representation of intelligent virtual agent’s interaction experience. This allows us to develop an approach to creation of IVAs by using case-based reasoning. We called this agent CBRIVA. We can define a CBRIVA as an entity that selects the next step based on previous interaction experience. A CBRIVA’s interaction experience is represented in the form of the three types of cases:

plan, contextual, and action cases.

Haris Supic

Modeling Spatiotemporal Uncertainty in Dynamic Virtual Environments

Current virtual agent [1] control architectures involve representations of the environment that must be adequate for effective deliberative behaviour rather than simple encoding of the environment’s state. Hence, they must take into account the element of uncertainty that is inherent to perceptual processes, in the form of predicting the future state of perceivable parts of the environment or the current state of non-perceivable parts. A number of approaches have been proposed to deal with uncertainty. A large proportion introducing uncertainty as a means towards increased perceptual believability [2]. A different group attempts to deal with inherent uncertainty in a variety of ways [3,4]. Although substantial benefits are gained in both cases, prediction over time is rarely addressed, while potential knowledge on targets’ intentions remains largely unexploited.

S. Vosinakis, G. Anastassakis, T. Panayiotopoulos

Industrial Demos

Avatars Contributions to Commercial Applications with Living ActorTM Technology

The technology and commercial experience of Cantoche put a new light on applications using avatars.

Laurent Ach, Benoît Morel

The CereVoice Characterful Speech Synthesiser SDK

CereProc® Ltd. have recently released a beta version of a commercial unit selection synthesiser featuring XML control of speech style. The system is freely available for academic use and allows fine control of the rendered speech as well as full timings to interface with avatars and other animation.

Matthew P. Aylett, Christopher J. Pidcock

VirtuOz Interactive Agents

VirtuOz is a software company that develops and sells an interactive agent technology capable of efficiently collaborating with clients’ customer service departments in order to provide its users with an innovative, immediate, assistance service based on artificial intelligence.

Aurélie Cousseau

Finger Tracking for Virtual Agents

Tracking is an essential tool for virtual agents. It provides (a) the pose of the interacting person for a correct 3d view of the virtual agent, (b) Motion Capture data of real persons that can be transferred to an avatar to obtain realistic movements or for interacting, (c) details of such movements like hands and fingers. This can be used for interactions like gesture recognition, too. With the example of tracking hands and fingers, the related problems and solutions are discussed: wireless active markers, addressing for identification, marker positions at finger tips, finger calibration etc.

Gerrit Hillebrand, Konrad Zuerl

Techniques of Dialogue Simulation

We claim that it is more effective to simulate intelligence than it is to recreate it. To this end several of the classic social psychological theories suggest strategies to transform the dialogue into an encounter with a consistent and cohesive personality. The secret is to use the mind-set of the user to the advantage of the conversation, and to provoke the user into showing typical behavior. This is presented in an online system:

Fred Roberts, Björn Gülsdorff


Additional information