Intelligent Virtual Agents

Graphical level of detail (LOD) is a set of techniques for coping with the issue of limited computational resources by reducing the graphical detail of the scene far from the observer. Simulation LOD reduces quality of the simulation at the places unseen. Contrary to graphical LOD, simulation LOD has been almost unstudied. As a part of our on-going effort on a large virtual-storytelling game populated by tens of complex virtual humans, we have developed and implemented a set of simulation LOD algorithms for simplifying virtual space and behaviour of virtual humans. The main feature of our technique is that it allows for several degrees of detail, i.e. for

gradual

varying of simulation quality. In this paper, we summarise the main lessons learned, introduce the prototype implementation called IVE and discuss the possibility of scaling our technique to other applications featuring virtual humans.

Virtual humans still lack naturalness in their nonverbal behaviour. We present a data-driven solution that moves towards a more natural synthesis of hand and arm gestures by recreating gestural behaviour in the style of a human performer. Our algorithm exploits the concept of gesture units to make the produced gestures a continuous flow of movement. We empirically validated the use of gesture units in the generation and show that it causes the virtual human to be perceived as more natural.

The integration of virtual agents in real-time interactive virtual applications raises several challenges. The rendering of the movements of the virtual character in the virtual scene (locomotion of the character or rotation of its head) and the binaural rendering in 3D of the synthetic speech during these movements need to be spatially coordinated. Furthermore, the system must enable real-time adaptation of the agent’s expressive audiovisual signals to user’s on-going actions. In this paper, we describe a platform that we have designed to address these challenges as follows: (1) the modules enabling real time synthesis and spatial rendering of the synthetic speech, (2) the modules enabling 3D real time rendering of facial expressions using a GPU-based 3D graphic engine, and (3) the integration of these modules within an experimental platform using gesture as an input modality. A new model of phoneme-dependent human speech directivity patterns is included in the speech synthesis system, so that the agent can move in the virtual scene with realistic 3D visual and audio rendering. Future applications of this platform include perceptual studies about multimodal perception and interaction, expressive real time question and answer system and interactive arts.

Many applications that utilize motion capture data require small, discrete, semantic segments of data, but most motion capture collection processes produce long sequences of data. The smaller segments are often created from the longer sequences manually. This segmentation process is very laborious and time consuming. This paper presents an automatic motion capture segmentation method based on movement qualities derived from Laban Movement Analysis (LMA). LMA provides a good compromise between high-level semantic features, which are difficult to extract for general motions, and low-level kinematic features which, often yield unsophisticated segmentations. The LMA features are computed using a collection of neural networks trained with temporal variance in order to create a classifier that is more robust with regard to input boundaries. The actual segmentation points are derived through simple time series analysis of the LMA features.

This paper presents a model for simulating cultural differences in the conversational behavior of virtual agents. The model provides parameters for differences in proxemics, gaze and overlap in turn taking. We present a review of literature on these factors and show results of a study where native speakers of North American English, Mexican Spanish and Arabic were asked to rate the realism of the simulations generated based on different cultural parameters with respect to their culture.

In this paper we present the design, development and initial evaluation of a virtual peer that models ethnicity through culturally authentic verbal and non-verbal behaviors. The behaviors chosen for the implementation come from an ethnographic study with African-American and Caucasian children and the evaluation of the virtual peer consists of a study in which children interacted with an African American or a Caucasian virtual peer and then assessed its ethnicity. Results suggest that it may be possible to tip the ethnicity of a embodied conversational agent by changing verbal and non-verbal behaviors instead of surface attributes, and that children engage with those virtual peers in ways that have promise for educational applications.

Real-Time Strategy (RTS) games are a challenging genre for the design of Intelligent Virtual Agents. We were interested whether incorporating a simple emotional model to an existing bot-script improves playing strength. We implemented a bot for Microsoft’s RTS game “Age of Mythology”. The emotional model is based on the “Big-Five” and the Emotion-Connectionist Model. Four variants of the bot were evaluated, each using different personality parameters; one of the variants was designed to show “neurotic” behaviour. The emotion-driven bots were evaluated in a bot-versus-bot setup, playing matches against the game’s default script. Evaluation results indicate a significant increase in playing strength, the “neurotic” bot being the strongest one.

The current state of the art virtual characters fall far short of characters produced by skilled animators. One reason for this is that the physical behaviors of virtual characters do not express the emotions and attitudes of the character adequately. A key deficiency possessed by virtual characters is that their gaze behavior is not emotionally expressive. This paper describes work on expressing emotion through head movement and body posture during gaze shifts, with intent to integrate a model of emotionally expressive eye movement into this work in the future. The paper further describes an evaluation showing that users can recognize the emotional states generated by the model.

In this paper we propose an algorithm based on fuzzy similarity which models the concept of resemblance between facial expressions of an Embodied Conversational Agent (ECA). The algorithm measures the degree of visual resemblance between any two facial expressions. We also present an evaluation study in which we compared the users’ perception of similarity of facial expressions. Finally we describe an application of this algorithm to generate complex facial expressions of an ECA.

Since the beginning of the SAIBA effort to unify key interfaces in the multi-modal behavior generation process, the Behavior Markup Language (BML) has both gained ground as an important component in many projects worldwide, and continues to undergo further refinement. This paper reports on the progress made in the last year in further developing BML. It discusses some of the key challenges identified that the effort is facing, and reviews a number of projects that already are making use of BML or support its use.

We aim at defining conversational agents that exhibit qualitatively distinctive behaviors. To this aim we provide a small set of parameters to allow one to define behavior profiles and then leave to the system the task of animating the agent. Our approach is to manipulate the behavior tendency of the agent depending on its communicative intention and emotional state.

In this paper we will define the concepts of

Baseline

and

Dynamicline

. The Baseline of an agent is defined as a set of fixed parameters that represent the personalized agent behavior, while the Dynamicline, is a set of parameters values that derive both from the Baseline and the current communicative goals and emotional state.

Recent research has established the potential for virtual characters to establish rapport with humans through simple contingent nonverbal behaviors. We hypothesized that the contingency, not just the frequency of positive feedback is crucial when it comes to creating rapport. The primary goal in this study was evaluative: can an agent generate behavior that engenders feelings of rapport in human speakers and how does this compare to human generated feedback? A secondary goal was to answer the question: Is contingency (as opposed to frequency) of agent feedback crucial when it comes to creating feelings of rapport? Results suggest that contingency matters when it comes to creating rapport and that agent generated behavior was as good as human listeners in creating rapport. A “virtual human listener” condition performed worse than other conditions.

Just like humans, conversational computer systems should not listen silently to their input and then respond. Instead, they should enforce the speaker-listener link by attending actively and giving feedback on an utterance while perceiving it. Most existing systems produce direct feedback responses to decisive (e.g. prosodic) cues. We present a framework that conceives of feedback as a more complex system, resulting from the interplay of conventionalized responses to eliciting speaker events and the multimodal behavior that signals how internal states of the listener evolve. A model for producing such incremental feedback, based on multi-layered processes for perceiving, understanding, and evaluating input, is described.

Embodied conversational agents should be able to provide feedback on what a human interlocutor is saying. We are compiling a list of facial feedback expressions that signal attention and interest, grounding and attitude. As expressions need to serve many functions at the same time and most of the component signals are ambiguous, it is important to get a better idea of the many to many mappings between displays and functions. We asked people to label several dynamic expressions as a probe into this semantic space. We compare simple signals and combined signals in order to find out whether a combination of signals can have a meaning on its own or not, i. e. the meaning of single signals is different from the meaning attached to the combination of these signals. Results show that in some cases a combination of signals alters the perceived meaning of the backchannel.

Participation in natural, real-time dialogue calls for behaviors supported by perception-action cycles from around 100 msec and up. Generating certain kinds of such behaviors, namely envelope feedback, has been possible since the early 90s. Real-time backchannel feedback related to the content of a dialogue has been more difficult to achieve. In this paper we describe our progress in allowing virtual humans to give rapid within-utterance content-specific feedback in real-time dialogue. We present results from human-subject studies of content feedback, where results show that content feedback to a particular phrase or word in human-human dialogue comes 560-2500 msec from the phrase’s onset, 1 second on average. We also describe a system that produces such feedback with an autonomous agent in limited topic domains, present performance data of this agent in human-agent interactions experiments and discuss technical challenges in light of the observed human-subject data.

The Text2Dialogue (T2D) system that we are developing allows digital content creators to generate attractive multi-modal dialogues presented by two virtual agents—by simply providing textual information as input. We use Rhetorical Structure Theory (RST) to decompose text into segments and to identify rhetorical discourse relations between them. These are then “acted out” by two 3D agents using synthetic speech and appropriate conversational gestures. In this paper, we present version 1.0 of the T2D system and focus on the novel technique that it uses for mapping rhetorical relations to question–answer pairs, thus transforming (monological) text into a form that supports dialogues between virtual agents.

In order to create and maintain social relationships with human users in mixed-initiative dialogues, IVAs have to give off coherent signals of claiming or relinquishing leadership in discourse. Quantitaive and qualitative analyses of human-human collaborative task-solving dialogues from the Ohio State University Quake Corpus reveal that discursive dominance is a shared achievement of speakers and given, taken or kept in a consensual way, up to the point where they incur “costs” in terms of efficiency in solving the task. Some verbal signals can be identified as relevant to this process.

We describe the design and evaluation of a virtual agent that explains health documents to patients. The prevalence and impact of low health literacy is presented as a motivation for such agents, given that face-to-face interaction with health providers is cited as one of the most effective means of communicating with these patients. We analyze the form and distribution of pointing gestures used by experts in explaining health documents, and use this data to develop a computational model of agent-based document explanation. This model is evaluated in a randomized controlled trial. Preliminary results indicate that patients with low health literacy are more satisfied with health document explanation by a virtual agent compared to a human.

Virtual humans offer an exciting and powerful potential for rich interactive experiences. Fully embodied virtual humans are growing in capability, ease, and utility. As a result, they present an opportunity for expanding research into burgeoning virtual patient medical applications. In this paper we consider the ways in which one may go about building and applying virtual human technology to the virtual patient domain. Specifically we aim to show that virtual human technology may be used to help develop the interviewing and diagnostics skills of developing clinicians. Herein we proffer a description of our iterative design process and preliminary results to show that virtual patients may be a useful adjunct to psychotherapy education.

Augmented realities, which are partially real and partially virtual, open up new ways for humans to interact with Embodied Conversational Agents (ECAs) since they allow users to meet ECAs in the physical space. Nevertheless, attempts to integrate ECAs as digital overlays in a physical space have been rare. Obvious reasons are the high demands such an integration puts to the animation of ECAs as virtual augmentations of the physical space, their capabilities to perceive not only the virtual, but also the physical world as well as reactive behavior control. In this paper, we describe our technical contributions towards solving these challenges. To illustrate our ideas, we present the virtual anatomy assistant Ritchie that monitors the user’s actions in a physical space and dynamically responds to them.

Interactive drama allows people to participate actively in a dynamically unfolding story, by playing a character or by exerting directorial control. One of the central challenges faced in the design of interactive dramas is how to ensure that the author’s goals for the user’s narrative experience are achieved in the face of the user’s actions in the story. This challenge is especially significant when a variety of users are expected. To address this challenge, we present an extension to Thespian, an authoring and simulating framework for interactive dramas. Each virtual character is controlled by a decision-theoretic goal driven agent. In our previous work on Thespian, we provided a semi-automated authoring approach that allows authors to configure virtual characters’ goals through specifying story paths. In this work, we extend Thespian into a more proactive authoring framework to further reduce authoring effort. The approach works by simulating potential users’ behaviors, generating corresponding story paths, filtering the generated paths to identify those that seem problematic and prompting the author to verify virtual characters’ behaviors in them. The author can correct virtual characters’ behaviors by modifying story paths. As new story paths are designed by the author, the system incrementally adjusts virtual characters’ configurations to reflect the author’s design ideas. Overall, this enables interactive testing and refinement of an interactive drama. The details of this approach will be presented in this paper, followed by preliminary results of applying it in authoring an interactive drama.

Against the background that recent studies on embodied conversational agents demonstrate the importance of their behavior, an experimental study is presented that assessed the effects of different nonverbal behaviors of an embodied conversational agent on the users´ experiences and evaluations as well as on their behavior. 50 participants conducted a conversation with different versions of the virtual agent Max, whose nonverbal communication was manipulated with regard to eyebrow movements and self-touching gestures. In a 2x2 between subjects design each behavior was varied in two levels: occurrence of the behavior compared to the absence of the behavior. Results show that self-touching gestures compared to no self-touching gestures have positive effects on the experiences and evaluations of the user, whereas eyebrow raising evoked less positive experiences and evaluations in contrast to no eyebrow raising. The nonverbal behavior of the participants was not affected by the agent’s nonverbal behavior.

We have developed software bots that inhabit the popular online social environment SecondLife (SL). Our bots can wander around, collect data, engage in simple interactions, and carry out simple automated experiments. In this paper we use our bots to study spatial social behavior. We found an indication that SL users display distinct spatial behavior when interacting with other users. In addition, in an automated experiment carried out by our bot, we found that users, when their avatars were approached by our bot, tended to respond by moving their avatar, further indicating the significance of proxemics in SL.

We describe two user studies designed to measure the impact of using the characteristic displays of a speaker expressing different user-preference evaluations to select the head and eye behaviour of an animated talking head. In the first study, human judges were reliably able to identify positive and negative evaluations based only on the motions of the talking head. In the second study, subjects generally preferred positive displays to accompany positive sentences and negative displays to accompany negative ones, and showed a particular dislike for negative facial displays accompanying positive sentences.

We present here a system for controlling the eye gaze of a virtual embodied conversational agent able to perceive the physical environment in which it interacts. This system is inspired by known components of human visual attention system and reproduces its limitations in terms of visual acuity, sensitivity to movement, limitations of short-memory and object pursuit. The aim of this coupling between animation and visual scene analysis is to provide sense of presence and mutual attention to human interlocutors. After a brief introduction to this research project and a focused state of the art, we detail the components of our system and confront simulation results to eye gaze data collected from viewers observing the same natural scenes.

The paper describes an infotainment application where life-like characters present two MP3 players in a virtual showroom. The key feature of the system is that the presenter agents analyze the user’s gaze-behavior in real-time and may thus adapt the presentation flow accordingly. In particular, a user’s (non-)interest in interface objects and also preference in decision situations is estimated automatically by just using eye gaze as input modality. A formal study was conducted that compared two versions of the application. Results indicate that attentive presentation agents support successful grounding of deictic agent gestures and natural gaze behavior.

Gaze plays a large number of cognitive, communicative and affective roles in face-to-face human interaction. To build a believable virtual human, it is imperative to construct a gaze model that generates realistic gaze behaviors. However, it is not enough to merely imitate a person’s eye movements. The gaze behaviors should reflect the internal states of the virtual human and users should be able to derive them by observing the behaviors. In this paper, we present a gaze model driven by the cognitive operations; the model processes the virtual human’s reasoning, dialog management, and goals to generate behaviors that reflect the agent’s inner thoughts. It has been implemented in our virtual human system and operates in real-time. The gaze model introduced in this paper was originally designed and developed by Jeff Rickel but has since been extended by the authors.

The goal of this paper is to open discussion about industrial creativity as a potential application field for Embodied Conversational Agents. We introduce the domain of creativity and especially focus on a collective creativity tool, the brainstorming: we present the related research in Psychology which has identified several key cognitive and social mechanisms that influence brainstorming process and outcome. However, some dimensions remain unexplored, such as the influence of the partners’ personality or the facilitator’s personality on idea generation. We propose to explore these issues, among others, using Embodied Conversational Agents. The idea seems original given that Embodied Agents were never included into brainstorming computer tools. We draw some hypotheses and a research program, and conclude on the potential benefits for the knowledge on creativity process on the one hand, and for the field of Embodied Conversational Agents on the other hand.

This paper describes a mobile context-aware ‘intelligent affective guide with attitude’ that guides visitors touring an outdoor attraction. Its behaviour is regulated by a biologically inspired architecture of emotion, allowing it to adapt to the user’s needs and feelings. In addition to giving an illusion of life, the guide emulates a real guide’s behaviour by presenting stories based on the user’s interests, its own interests, its belief and its current memory activation. A brief description of the system focusing on the core element - the guide’s emotional architecture - is given followed by findings from an evaluation with real users.

Since the beginnings of character animation,

anticipation

has been an effective part of the repertoire of tricks used to create believable animated characters. However, anticipation has had but a secondary role in the creation of synthetic virtual life forms. In this paper, we describe how a simple anticipatory mechanism that generates an affective signal resulting from the mismatch between sensed and predicted values — the

emotivector

— can help in the creation of consistent believable behaviour for intelligent virtual characters.

This paper presents an approach to incorporate emotion regulation as addressed within psychology literature into virtual characters. To this end, first Gross’ informal theory of emotion regulation has been formalised using a dynamical system style modelling approach. Next, a virtual environment has been created, involving a number of virtual agents, which have been equipped with the formalised model for emotion regulation. This environment has been used to successfully generate a number of emergent virtual stories, in which characters regulate their emotions by applying regulation strategies such as situation selection and attentional deployment. The behaviours shown in the stories were found consistent with descriptions of human regulation processes.

Emotion takes an increasingly important place in the design of intelligent virtual agents. Designers of emotional agents build on theories from cognitive psychology, that describe the cognitive functioning of emotions with two indivisible processes [1,2]: the

appraisal

process triggers emotions, in particular intense negative emotions to point out threatening stimuli, and the

coping

process modifies the behaviour to manage these stimuli. Nevertheless, among the existing emotional agents, a lot express emotions triggered by an appraisal process [3] but few have a coping process allowing their emotions to impact their behaviour [4,5,6]. In previous work [7] we provided a formalization of Ortony et al.’s appraisal process [8] in a BDI logic,

et al.’s

a logic of mental attitudes. The next step is to formalize the coping process in the same framework. Our aim here is to provide the theoretical basis of an agent architecture rather than an implementation. We only give here an overview of our framework (

cf

. [9,10] for more details).

Emergent narrative systems create stories which are always different from each other. Creating summaries of these stories is a challenge especially if we want to capture the richness of the characters. Our goal is to automatically generate summaries from emergent narrative using comics as the visual medium for the summary. We identify the most important situations in the story log looking at the emotional state of the characters, transform the resulting summary into a comics description and create the comic. We believe that a good summarization of a story that maintains the emotions of the characters together with an expressive visual representation is essential for the user to remember the story.

In the teamwork research area there is an increasing interest about the principles behind team effectiveness and effective team training; for Intelligent Virtual Agents (IVAs) Team Training is an excellent application area; nevertheless, the few reported works about IVAs in team training, illustrate both the use for the individualized teaching (Pedagogical Agents) of procedural tasks and the substitution of missing team members (Teammate Agents) to promote the practice of team tasks in relation to functional roles (Taskwork) [1].

Our interest on Intelligent Virtual Environments for Training (IVETs) has led us to propose a Team Training Strategy (TTS) whose purpose is to promote social skills as well as knowledge and skills related to tasks of socio technical nature. The alternatives that we are evaluated to improve the performance of human groups and to promote effective teams deal with: the use of scaffolding as the best tutoring approach, the promotion of social skills before technical skills, and especially, the selection of the best nonfunctional roles (team roles) balance according to the task.

In addition, our aim is to incorporate into an IVA called Pancho (Pedagogical AgeNt to support Colaborative Human grOups) the particular behaviors of Team Roles defined by Belbin [2]; Pancho, with a selected team role —according to a team model— will join the human group with the intention of improving the performance of the team (Teamwork) and providing scaffolding to the trainees (Taskwork). The Belbin’s categorization is the earliest and still the most popular. He states that the team role can be defined as a tendency to behave, contribute and interrelate with each others at work in a certain distinctive ways; he also states that in teamwork, a good mix of team roles in the group is necessary for groups to use their technical skills optimally. The team roles defined by Belbin have very particular behaviors; we have selected a generic cognitive architecture for agents with emotionally influenced behaviors —called COGNITIVA— to realize those roles [3]. The constructs provided by this architecture (Personal traits, Concerns, Moods, Attitudes and Physical states) are being properly instantiated to generate the desired behaviors.

The development of virtual agents designed to draw users into personal and professional relationships with them represents a growing area of research [1]. Mobility and context awareness represent important directions of research for these relational agents, since they offer unique affordances for relationship development. A mobile/wearable agent has the potential to be with a user for a significant period of time, and frequency of contact alone has been shown to be associated with increased solidarity between people. The ability to sense some aspects of the user’s environment (context awareness) may also provide mobile agents with unique relational affordances. Automatically recognizing and commenting on situations in the user’s life can amplify many relational perceptions, including familiarity, common ground, solidarity and intimacy. In addition, an agent’s ability to proactively interrupt and help a user in a situation that is automatically sensed by the agent may lead to increased perceptions of trust and caring by the user.

This paper provides a brief overview of an implicit training method used for teaching autonomous agents to represent humans in 3D Virtual Worlds without any explicit training efforts being required.

In this article, we show how approaches based on interactive data-mining may inspire new conceptions of theatre staging. They may be applied thanks to virtual agent systems interacting with comedians. We then give an example of such a theatre production,

La Traversée de la nuit

.

A typical present-day virtual actor is able to store episodes in an

ad hoc

manner, which does not allow for reconstructing the actor’s personal stories. We have prototyped a virtual RPG actor with a

full

episodic memory, which allows for this reconstruction. The paper overviews the work done and sketches the work in progress.

We present the platform for IVAs development in the human like environment of the first-person shooter game Unreal Tournament 2004. This environment is extendible and supported by vast community of users. Based on our previous experience the problem of fast verification of models of artificial intelligence or IVAs is in implementation issues. The developer spends most of his time solving technical environment dependent issues and malfunctions, which drives him away from his goals. Therefore our modular platform provides a tool, which helps solving those problems and the developer can spend saved time by solving another AI based issues and model verification. The platform is aimed for research and educational purposes.

In this article we present an ongoing project in our research group, an ECA based multicultural tour guide system. Tour guide ECA agent provides information about the city of Dubrovnik and dynamically changes its behaviors among Japanese, Croatian and general western cultures speaking in English.

The animation of an ECA, for most of animation systems, implies that its behaviour is encoded in a representation language, giving a form for each modality of a behavior. Interested in gestures, several representation languages exist already that that usually give a physical description of the necessary information for the production and reproduction of gestures, and for its synchronization with the other modalities. In our work, we aim at enriching the SAIBA description of gestures with semantic considerations, enabling an ECA to use a gesture for a wide range of applications.

The form of improvisational drama allows participants to have their own choices to influence the ongoing story, and each play results in a different ending. However, authoring such story contents requires ad hoc scripting, and static story structures lose ingenuity once users hacked through them. Our purpose is to develop a toolkit for: (1) fast authoring the story content, and (2) allow it for repeated plays yet retaining fresh interactive experience. While most similar applications have explicit, sophisticated story structures to ensure the number of possible interactions and endings in specific situations, we argue that characters should have enough background knowledge to make any improvisational choices. The more knowledge they have, the more sophisticated course of actions they may express. As a result, we take a deep-model approach to implement virtual agents, allowing them to deliberate and act with established knowledge in unexpected situations.

The AtGentive project [1,2] focuses on the support of attention in learning environments. To achieve this objective the system analyses the learners’ computer activities and physical states and, on the basis of this analysis, it generates interventions. Such interventions either supply learners with information useful to support their current attentional focus, or are aimed at attracting the user’s attention to new foci.

Nowadays, the video gaming experience is shifting from merely realistic to believable. The behaviour of the computer driven player and non-playing characters is often poor when compared to their visual appearance. In this sense, there has been a recent interest in improving the video gaming experience with novel Artificial Intelligence (AI) techniques. This paper presents a robotics inspired behavioural AI to simulate characters’ personalities in a commercial video game.

This paper considers possible affective roles in an agent-based social simulation, and in particular the effect of adding a simple model of fear into a replication of an agent-based social simulation.

The JAST human-robot dialogue system [1] is designed as a platform to integrate empirical findings on cognition and cooperative dialogue with research on autonomous robots by supporting multimodal human-robot collaboration on a construction task. The robot consists of a pair of mechanical arms with grippers, mounted in a position to resemble human arms, and a Philips iCat animatronic talking head [2]. The user and the robot work together to assemble construction toys on a common work area, coordinating their actions through speech, gestures, and facial displays.

Modeling crowd behaviors is an important challenge for intelligent virtual agents. We propose a general cognitive model of simulating crowd behaviors, based on Festinger’s Social Comparison Theory (SCT), a prominent social psychology theory. We present the use of the SCT model (using the Soar cognitive architecture) in the generation of imitational behavior in loosely-coupled groups and show that SCT generates behavior more in-tune with human crowd behavior.

This work presents a market-based social model to produce good quality behavioral animations for groups of intelligent virtual agents. The social model coordinates the activities of groups of virtual characters and also includes social actions in the agent decision-making. We follow the Multi-Agent Resource Allocation approach presented in [2], where agents express their preferences using utility functions. The dynamics of social interactions is inspired by the theory of Piaget [3] over which we have implemented reciprocal task exchanges.

Embodied Conversational Agents (ECAs) are life-like CG characters that interact with human users in face-to-face conversations. To achieve natural conversations, they need to understand the inputs from human users, deliberate the responding behaviors and realize those behaviors in multiple modalities. They are sophisticated, require numbers of building assemblies and are thus difficult for individual research groups to develop. To facilitate result sharing and rapid prototyping of ECA researches, a Generic ECA Framework that is meant to integrate ECA assemblies seamlessly is being developed by our group. This framework is composed of a low-level communication platform (GECA Platform), a set of communication API libraries (GECA Plugs) and a high-level protocol (GECA Protocol, GECAP).

This article describes an attempt to build a quiz game kiosk for show-room use based on the Generic Embodied Conversational Agent (GECA) Framework [1] that provides a general purpose architecture for connecting modularized ECA functional components for multimodal human-agent interactions.

3D virtual space can visually represent the spatial structure to users and it has been applied to many fields such as city planning, navigation, education, entertainment and so on. In the 3D virtual space, an agent can navigate a user in an interactive manner [5]. Various platforms to build a 3D virtual space and languages to control the agents have been proposed [1,2,3,4]. For example, VKSC (Virtual Kobe Sanda Campus) is a 3D virtual space of Kobe Sanda Campus, Kwansei Gakuin University [1]. In VKSC, an agent called Suzie guides a user in the campus upon his/her request.

Our research is focused on simulation of agents - animates. The architecture of these agents is mainly inspired by nature therefore they are sometimes called artificial creatures. The main contribution of this paper is the description of designed simulation environment architecture for the Artificial Life (ALife) domain. It was named the World of Artificial Life (WAL). Our platform incorporates results of research in domain of hybrid agent architectures.

The automatic production of speech and gesture is one of the most challenging issues in building embodied conversational agents, due to the intricacy of how the two modalities seem to “align” with each other. Based on experiences from computational approaches and inspired by current theoretical modeling attempts, we propose an architecture to simulate how on-the-spot speech and gesture production might run and bring about more natural multimodal behavior.

We present a system which consists of a lifelike agent animated in real-time using video and audio analysis from the user. This kind of system could be used for Instant Messaging where an avatar controlled like a puppet is displayed instead of the webcam flow. The overall system is made of video analysis based on Active Appearance Models and audio analysis based on Hidden Markov Model. The parameters from these two modules are sent to a control system driving the animation engine. The video analysis extracts the head orientation and the audio analysis provides the phonetic string used to move the lips.

The paper describes a double appraisal-based emotion system for synthetic characters. This approach gives intelligent agents the ability to make decisions with respect to the emotional states of others, thus implementing aspects of the theory of mind concept and laying the basis for cognitive empathy.

In this paper, we propose to endow a graphical representation of a user in Instant Messaging – an avatar – with the ability to recognize and to express emotions and to play social nonverbal behaviour, on the basis of textual affect sensing and interpretation of communicative functions conveyed by online conversations. The developed Affect Analysis Model integrated with Instant Messaging (IM) media supports the recognition of affect from text to ensure avatar animation in an appropriate and expressive manner.

One of the most important social ability for effective social interaction with people is the capacity to understand, feel and ultimately express emotions. In this paper we present an architecture, based on the BDI paradigm, employing a three layered approach and coupling an emotion engine which simulates the generation of affective states based on Scherer’s component process theory and influences decision making.

When considering the gaze behaviours of embodied agents, and necessarily the underlying processes of visual attention that help to guide them, most work to date has been focused either on models aimed at controlling gaze in social situations, for example when one or more ECA’s and humans are interacting, or for controlling gaze directed at the environment in a more general sense, typically for helping AVA’s accomplish tasks such as navigation. We are designing a model of visual attention that attempts to amalgamate these concepts into a unified design in order to produce intelligent virtual agents that can behave in a social manner with social stimuli while also being attentive to events when mobile in the more general environment.

We present a model for automatic generation of expressive gaze in virtual agents. Our main focus is the eye behavior associated to expressiveness. Our approach is to collect data from animated Computer Graphics films, and codify such observations into an animation framework. The main contribution is the modeling aspects of an animation system, calibrated with empirical observations in order to generate realistic eyes motion. Results show that this approach generates convincing animations that improve the empathy of virtual agents.

We present here the analysis of multimodal data gathered during realistic face-to-face interaction of a target speaker with a number of interlocutors. Videos and gaze have been monitored with an experimental setup using coupled cameras and screens with integrated eye trackers. With the aim to understand the functions of gaze in social interaction and to develop a coherent gaze control model for our talking heads we investigate the influence of cognitive state and social role on the observed gaze behavior.

We describe a computational treatment of certain sorts of affect-conveying metaphorical utterances. This is part of an affect detection system used by intelligent conversational agents (ICAs) operating in an edrama system.

In this paper we describe a case-based representation of intelligent virtual agent’s interaction experience. This allows us to develop an approach to creation of IVAs by using case-based reasoning. We called this agent CBRIVA. We can define a CBRIVA as an entity that selects the next step based on previous interaction experience. A CBRIVA’s interaction experience is represented in the form of the three types of cases:

plan, contextual, and action cases.

Current virtual agent [1] control architectures involve representations of the environment that must be adequate for effective deliberative behaviour rather than simple encoding of the environment’s state. Hence, they must take into account the element of uncertainty that is inherent to perceptual processes, in the form of predicting the future state of perceivable parts of the environment or the current state of non-perceivable parts. A number of approaches have been proposed to deal with uncertainty. A large proportion introducing uncertainty as a means towards increased perceptual believability [2]. A different group attempts to deal with inherent uncertainty in a variety of ways [3,4]. Although substantial benefits are gained in both cases, prediction over time is rarely addressed, while potential knowledge on targets’ intentions remains largely unexploited.

The technology and commercial experience of Cantoche put a new light on applications using avatars.

CereProc® Ltd. have recently released a beta version of a commercial unit selection synthesiser featuring XML control of speech style. The system is freely available for academic use and allows fine control of the rendered speech as well as full timings to interface with avatars and other animation.

VirtuOz is a software company that develops and sells an interactive agent technology capable of efficiently collaborating with clients’ customer service departments in order to provide its users with an innovative, immediate, assistance service based on artificial intelligence.

Tracking is an essential tool for virtual agents. It provides (a) the pose of the interacting person for a correct 3d view of the virtual agent, (b) Motion Capture data of real persons that can be transferred to an avatar to obtain realistic movements or for interacting, (c) details of such movements like hands and fingers. This can be used for interactions like gesture recognition, too. With the example of tracking hands and fingers, the related problems and solutions are discussed: wireless active markers, addressing for identification, marker positions at finger tips, finger calibration etc.

We claim that it is more effective to simulate intelligence than it is to recreate it. To this end several of the classic social psychological theories suggest strategies to transform the dialogue into an encounter with a consistent and cohesive personality. The secret is to use the mind-set of the user to the advantage of the conversation, and to provoke the user into showing typical behavior. This is presented in an online system: Elbot.com.