Intelligent Virtual Agents

When humans interact with one another, socially intelligent conversational behaviors arise from the interaction of a number of different factors: the conversants’ personality, cultural knowledge, the ability to observe and reason about social relationships, and the ability to project and detect affective cues. For virtual agents to be socially intelligent, they must have an expressive conversational repertoire. Moreover, scientific progress in this area requires that these expressive capabilities be easily parameterized, to support experimentation, and that at least some of the factors mentioned above be used to control the parameters. In this talk, I describe our research on expressive spoken language generation, and discuss how our work aims for both psychological plausibility and realistic usability. To achieve psychological plausibility we build on theories and detailed studies of human language use, such as the Big Five theory of personality, and Brown and Levinsons theory of politeness [1,2,3]. To achieve realistic usability, we have developed both rule-based and trainable generation methods that can dynamically, and in real time, change an agents linguistic style by modifying the values of these theoretically motivated parameters. This will allow us to experiment with dynamically modifying an agent’s linguistic style based on theories of audience design, entrainment and alignment.

By using parameterization techniques which model artistic, living or cognitive systems it is becoming possible to create new types of behavior and expression character systems. These techniques are allowing virtual agent creators to incorporate models of expression, emotion, behavior and even human creativity into their work. Additionally, rather than simply using realism as a goal, is it becoming possible to computationally model knowledge from other expression-based sources including artists, musicians and designers, to go beyond communication to creative expression.

Evolving beyond their origins as a novel pastime, videogames have developed into a medium with tremendous power to entertain and engage players through emotionally powerful interaction. These emotional connections are often powered by the quality of the digital actors that inhabit game worlds and bring them to life. But as technologies for creating lifelike characters escalate, so do the challenges of the creation process. This discussion examines methods used by cutting-edge games to create deeply compelling digital actors, and explores future challenges and solutions that will help videogames unlock the full potential of emotionally engaging human interaction.

We discuss the ethical and practical issues involved in developing virtual humans that relate personal, fictitious, human autobiographical stories (“back stories”) to their users. We describe a virtual human exercise counselor that interacts with users daily to promote exercise, and the integration of a dynamic social storytelling engine used to maintain user engagement with the agent and retention in the intervention. A longitudinal randomized controlled experiment tested user attitudes towards the agent when it presented the stories in first person (as its own history) compared to third person (as happening to humans that it knew). Participants in the first person condition reported enjoying their interactions with the agent significantly more and completed more conversations with the agent, compared to participants in the third person condition, while ratings of agent dishonesty were not significantly different between the groups.

Memory is a vital capability for intelligent social Companions. In this paper, we introduce a simple memory model that allows a Companion to maintain a long-term relationship with the user by remembering past experiences in order to personalise interaction. Additionally, we implemented a situational forgetting mechanism that gives the Companion the ability to protect the user’s privacy by not disclosing sensitive data. Two test scenarios are used to demonstrate these abilities in our Companions.

How do we construct credible personalities? The current SAL (Sensitive Artificial Listeners) characters were constructed intuitively and can be unconvincing. In addressing these issues, this paper considers a theory of personality and associated emotional traits, and discusses how behaviours associated with personality types in people may be adapted to develop characteristics of virtual agents. Our objective is to ensure that behavioural perceptions of a virtual agent credibly reflect the agent’s ‘actual’ personality as prescribed.

Users expect characters in role-playing games to be proactive and social, but these characters fail to deliver in this respect due to limitations of traditional game AI programming approaches. BDI-based approaches are suited for development of proactive systems, such as NPCs in games. This paper argues that a BDI-based approach is also highly suited for developing social NPCs in a principled way.

Believable spatial behaviour is important for intelligent virtual agents acting in human-like environments, such as buildings or cities. Existing models of spatial cognition and memory for these agents are predominantly aimed at issues of navigation and learning of topology of the environment. The issue of representing information about possible objects’ locations in a familiar environment, information that can evolve over long periods, has not been sufficiently studied. Here, we present a novel representation for “what-where” information: memory for locations of objects. We investigate how this representation is formed and how it evolves using a simplified model of a virtual character. The behaviour of the model is also compared with behaviour of real humans conducting an analogical task.

In this article we show how socio-psychological status theories can be formalized and used to control the expressive and communicative behavior of intelligent virtual agents. We describe

Exstasis

, a model that computes the status and action tendencies for all actors in a situation based on their status characteristics and behavior patterns. As a configurable software component

Exstasis

can be integrated in existing systems to extend the social capabilities of the virtual characters.

Authoring the AI for non-player characters (NPCs) in modern video games is an increasingly complex task. Designers and programmers must collaborate to resolve a tension between believable agents with emergent behaviours and scripted story lines. Behaviour trees (BTs) have been proposed as an expressive mechanism that let designers create complex behaviours along the lines of the story they want to tell. However, BTs are still too complex for non-programmers. In this paper, we propose the use of plan traces to assist designers when building BTs. In order to make this approach feasible within state-of-the-art video game technology, we generate the planning domain through an extension of the component-based approach, a widely used technique for representing entities in commercial video games.

We propose a computational model for building a tactile body schema for a virtual human. The learned body structure of the agent can enable it to acquire a perception of the space surrounding its body, namely its peripersonal space. The model uses tactile and proprioceptive informations and relies on an algorithm which was originally applied with visual and proprioceptive sensor data. In order to feed the model, we present work on obtaining the nessessary sensory data only from touch sensors and the motor system. Based on this, we explain the learning process for a tactile body schema. As there is not only a technical motivation for devising such a model but also an application of peripersonal action space, an interaction example with a conversational agent is described.

Expressing spatial information with iconic gestures is abundant in human communication and requires to transform a referent representation into resembling gestural form. This task is challenging as the mapping is determined by the visuo-spatial features of the referent, the overall discourse context as well as concomitant speech, and its outcome varies considerably across different speakers. We present a framework, GNetIc, that combines data-driven with model-based techniques to model the generation of iconic gestures with Bayesian decision networks. Drawing on extensive empirical data, we discuss how this method allows for simulating speaker-specific vs. speaker-independent gesture production. Modeling results from a prototype implementation are presented and evaluated.

Basic communication and coordination mechanisms of human social interaction are assumed to be mediated by perception-action links. These links ground the observation and understanding of others in one’s own action generation system, as evidenced by immediate motor resonances to perceived behavior. We present a model to endow virtual embodied agents with similar properties of embodied perception. With a focus of hand-arm gesture, the model comprises hierarchical levels of motor representation (commands, programs, schemas) that are employed and start to resonate probabilistically to visual stimuli of a demonstrated movement. The model is described and evaluation results are provided.

This paper presents an architecture for an intelligent virtual agent that imitates human drumming behaviour. Through imitation, the agent models the user-specific variations that constitute the “groove” of the drummer. The architecture comprises a motor system that imitates arm movements of a human drummer, and a sound system that produces the sound of the human playing style. The presence of a sound system alleviates the need to use physical models that will create sound when a drum is struck, instead focusing on creating an imitative agent that booth looks and sounds similar to its teacher. Such a virtual agent can be used in a musical setting, where its visualization and sound system would allow it to be regarded as an artificial musician. The architecture is implemented using Echo State Networks, and relies on self-organization and a bottom-up approach when learning human drum patterns.

In this paper, a new low-dimensional motion model that can parameterize human motion style is presented. Based on this model, a human motion synthesis approach by using constrainted optimization in a low-dimensional space is proposed. We define a new inverse kinematics solver in this low-dimensional space to generate the required motions meeting user-defined space constraints at key-frames. Our approach can also allow users to edit motion style explicitly by specifying the style parameter. The experimental results demonstrate the effectiveness of this approach which can be used for interactive motion editing.

Virtual humans are employed in many interactive applications, including (serious) games. Their motion should be natural and allow interaction with its surroundings and other (virtual) humans in real time. Physical controllers offer physical realism and (physical) interaction with the environment. Because they typically act on a selected set of joints, it is hard to evaluate their naturalness in isolation. We propose to augment the motion steered by such a controller with motion capture, using a mixed paradigm animation that creates coherent full body motion. A user evaluation of this resulting motion assesses the naturalness of the controller. Methods from Signal Detection Theory provide us with evaluation metrics that can be compared among different test setups, observers and motions. We demonstrate our approach by evaluating the naturalness of a balance controller. We compare different test paradigms, assessing their efficiency and sensitivity.

Virtual training systems provide an effective means to train people for complex, dynamic tasks such as crisis management or firefighting. Intelligent agents are often used to play the characters with whom a trainee interacts. To increase the trainee’s understanding of played scenarios, several accounts of agents that can explain the reasons for their actions have been proposed. This paper describes an empirical study of what instructors consider useful agent explanations for trainees. It was found that different explanations types were preferred for different actions, e.g. conditions enabling action execution, goals underlying an action, or goals that become achievable after action execution. When an action has important consequences for other agents, instructors suggest that the others’ perspectives should be part of the explanation.

In this paper, we present a study to evaluate the impact of adaptive feedback on the effectiveness of a pedagogical agent for an educational computer game. We compare a version of the game with no agent, and two versions with agents that differ only in the accuracy of the student model used to guide the agent’s interventions. We found no difference in student learning across the three conditions, and we report an analysis to understand the reasons of these results.

In human-computer interaction social behavior towards computers like flattery, reciprocity, and politeness have been observed [1]. In order to determine whether the results can be replicated when interacting with embodied conversational agents (ECA), we conducted an experimental study. 63 participants evaluated the ECA Max after a 10-minute conversation. The interview situation was manipulated in three conditions: Being questioned by Max himself, being questioned by paper-and-pencil questionnaire in the same room facing Max, and being questioned by means of a paper-and-pencil questionnaire in another room. Results show that participants were more polite to the ECA in terms of a better evaluation when they were questioned by Max himself compared to when they were questioned more indirectly by paper-and-pencil questionnaire in the same room. In contrast to previous studies [2] it was ruled out that some participants thought of the programmer when they were asked to evaluate the ECA. Additionally, user variables (e.g. gender, computer literacy) show an impact on the on the evaluation of the ECA.

This paper presents two attempts in integrating attentiveness into a virtual quiz agent in the situation when multiple game participants present. One of them features an utterance strategy to determine when and whom to talk to among the participants. The other one features a SVM (support vector machine) triggered transition state model of the agent’s attitude toward the participants in expressing observable behaviors. Both of them are driven by timings determined on video and audio information of the participants’ activity while they are trying to solve the quizzes. To evaluate these two prototype systems, we applied GNAT (Go/No-go Task) method in addition to questionnaires. From the joint results of the subject experiments, the direction in finding appropriate action timings of the agent is proved to be able to improve user impressions.

This paper presents the results of a field study of the COHIBIT museum exhibit. Its purpose is to convey knowledge about car-technology and virtual characters in an entertaining way. Two life-sized virtual characters interact with the visitors and support them in constructing car models with specific tangible car modules. The evaluation should reveal what makes the exhibit successful in terms of an overall user impression and how users rate the employed virtual characters. The focus of the evaluation is to measure the user experience that manifests in aspects like attraction, rejection and entertainment. Based on an analysis of relevant systems and evaluation models, an on-site evaluation for the COHIBIT exhibit has been designed. The results show that the positive user experience of the exhibit is mainly based on the non-task-specific aspects like virtual characters, joy of use and entertainment.

This paper presents a quest for the most suitable setting and method to assess the naturalness of the output of an existing algorithm for the generation of multimodal referring expressions. For the evaluation of this algorithm a setting in Second Life was built. This paper reports on a pilot study that aimed to assess (1) the suitability of the setting and (2) the design of our evaluation method. Results show that subjects are able to discriminate between different types of referring expressions the algorithm produces. Lessons learnt in designing questionnaires are also reported.

Wrinkles, blushing, sweating and tears are physiological manifestations of emotions in humans. Therefore, the simulation of these phenomena is important for the goal of building believable virtual humans which interact naturally and effectively with humans. This paper describes a real-time model for the simulation of wrinkles, blushing, sweating and tears. A study is also conducted to assess the influence of the model on the perception of surprise, sadness, anger, shame, pride and fear. The study follows a repeated-measures design where subjects compare how well is each emotion expressed by virtual humans with or without these phenomena. The results reveal a significant positive effect on the perception of surprise, sadness, anger, shame and fear. The relevance of these results is discussed for the fields of virtual humans and expression of emotions.

Facial animation has reached a high level of photorealism. Skin is rendered with grain and translucency, wrinkles are accurate and dynamic. These recent visual improvements are not fully tested for their contribution to the perceived expressiveness of virtual characters. This paper presents a perceptual study assessing the impact of different rendering modes of expressive wrinkles on users’ perception of facial expressions of basic and complex emotions. Our results suggest that realistic wrinkles increase agent’s expressivity and user’s preference, but not the recognition of emotion categories. This study was conducted using our real time facial animation platform that is designed for perceptive evaluations of affective interaction.

Displaying facial motions such as crying or laughing is difficult to achieve in real-time simulations and games. Not only because of the complicated simulation of the physical characteristics such as muscle motions or fluid simulations, but also because one needs to know how to control these motions on a higher level. In this paper, we propose a method that uses the MPEG-4 Facial Animation standard to control a realistic crying face in real-time. The tear simulation is based on the Smoothed Particle Hydrodynamics technique, which we optimized for real-time tear generation and control. Through simple parameters, a wide range of expressions and tears can be generated on the fly. Additionally, our method works independently of the graphics and physics engines that are used.

In human-human conversation, the first impression decides whether two people feel attracted by each other and whether contact between them will be continued or not. Starting from psychological work on flirting, we implemented an eye-gaze based model of interaction to investigate whether flirting tactics help improve first encounters between a human and an agent. Unlike earlier work, we concentrate on a very early phase of human-agent conversation (the initiation of contact) and investigate which non-verbal signals an agent should convey in order to create a favourable atmosphere for subsequent interactions and increase the user’s willingness to engage in an interaction with the agent. To validate our approach, we created a scenario with a realistic 3D agent called Alfred that seeks contact with a human user. Depending on whether the user signals interest in the agent by means of his or her gaze, the agent will finally engage in a conversation or not.

We introduce a physically-based facial animation system for natural-looking emotional expressions. Our system enables creation of simple facial expressions using Facial Action Coding System (FACS) and complex facial expressions by blending existing expressions. We propose a method for blending facial actions, based on a parameterization that combines elements of FACS and MPEG-4 standard. We explore the complex expression space by examining the blends of basic emotions produced with our model.

In realistic looking game environments it is important that virtual characters behave naturally. Our goal is to produce naturally looking gaze behavior for animated agents and avatars that are simply idling. We studied people standing and waiting, as well as people walking down a shopping street. From our observations we built a demo with the CADIA Populus multi-agent social simulation platform.

Many researchers associate a culture with some form of knowledge; other scholars stress the importance of the environment inhabited by the knowledge carriers; while archaeologists learn about cultures through the objects produced in the environment as a result of utilizing this knowledge. In our work we propose a model of virtual culture that preserves the environment, objects and knowledge associated with a certain culture in a 3D Virtual World. We highlight the significance of virtual agents in our model as, on the one hand, being the knowledge carriers and, on the other hand, being an important element establishing the connection between the environment, objects and knowledge. For testing the resulting model we have developed a research prototype simulating the culture of the ancient City of Uruk 3000 B. C. (one of the first human-built cities on Earth) within a Virtual World of Second Life.

With the increase in the development of autonomous agents, there is a bigger demand on their capability of interacting with other agents and users in ways that are natural and inspired by how humans interact. However, cultural aspects have been largely neglected so far, even though they are a crucial aspect of human societies. Our goal is to create an architecture able to model cultural groups of agents with perceivable distinct behaviour. In particular, this paper focus on how to use two cultural dimensions proposed by Hofstede in order to influence the agent’s goal selection and appraisal processes. Using our cultural architecture, we created two cultural groups of agents and asked users to visualise them performing a short emergent story. We then asked them to describe the two groups visualised. Results confirmed that users did perceived differences in the groups, and those differences were congruent with the cultural parametrisation used.

Psychology suggests highly synchronized expressions of emotion across different modalities. Few experiments jointly studied the relative contribution of facial expression and body posture to the overall perception of emotion. Computational models for expressive virtual characters have to consider how such combinations will be perceived by users. This paper reports on two studies exploring how subjects perceived a virtual agent. The first study evaluates the contribution of the facial and postural expressions to the overall perception of basic emotion categories, as well as the valence and activation dimensions. The second study explores the impact of incongruent expressions on the perception of superposed emotions which are known to be frequent in everyday life. Our results suggest that the congruence of facial and bodily expression facilitates the recognition of emotion categories. Yet, judgments were mainly based on the emotion expressed in the face but were nevertheless affected by postures for the perception of the activation dimension.

Moral emotions have been argued to play a central role in the emergence of cooperation in human-human interactions. This work describes an experiment which tests whether this insight carries to virtual human-human interactions. In particular, the paper describes a repeated-measures experiment where subjects play the iterated prisoner’s dilemma with two versions of the virtual human: (a) neutral, which is the control condition; (b) moral, which is identical to the control condition except that the virtual human expresses gratitude, distress, remorse, reproach and anger through the face according to the action history of the game. Our results indicate that subjects cooperate more with the virtual human in the moral condition and that they perceive it to be more human-like. We discuss the relevance these results have for building agents which are successful in cooperating with humans.

How to create effective body animations for virtual agents with emotions remains the state of the art for human animators and a great challenge for computer scientists. In this paper, we propose to use a model of hierarchal parameters to represent body animations:

emotional

,

style

,

motion,

and

procedural

parameters. Based on this model, we have created motions for a virtual character with generic animation procedures and mapped these procedural parameters into style parameters as proposed in the literature. The expressiveness of the generated animations was verified through experiments in our previous work. In this paper, we further report the results of two experiments attempting to verify how the style parameters are mapped into various emotions. The results reveal that the participants can successfully distinguish emotions based on the manipulation of style parameters for neutral motions such as walking. When these style parameters were used for emotive motions, including pounding, shivering, flourishing and crestfallen, the generated animations were even more effective for intended contexts.

In this paper we present a system which allows a virtual character to display multimodal sequential expressions i.e. expressions that are composed of different signals partially ordered in time and belonging to different nonverbal communicative channels. It is composed of a language for the description of such expressions from real data and of an algorithm that uses this description to automatically generate emotional displays. We explain in detail the process of creating multimodal sequential expressions, from the annotation to the synthesis of the behavior.

Computational modeling of emotion, physiology and personality is a major challenge in order to design believable virtual humans. These factors have an impact on both the individual behavior and the collective one. This requires to take into account the empathy phenomenon. Furthermore, in a crisis simulation context where the virtual humans can be contaminated by radiological or chemical substances, empathy may lead to placebo or nocebo effects.

Stemming from works in the multiagent systems domain, our virtual human decision process is designed as an autonomous agent. It has been shown that the environment can encapsulate the responsibility of spreading part of the agent state. The agent has two parts, its mind and its body. The mind contains the decision process and is autonomous. The body is influenced by the mind, but controlled by the environment which manages the empathy process. Combined with biased reasoning, favorable personality traits and situational factors, empathy can lead some agents to believe they are contaminated although they are not. We describe these mechanisms and show the results of several experiments.

Enabling virtual agents to quickly and accurately infer users’ psychological characteristics such as their personality could support a broad range of applications in education, training, and entertainment. With a focus on narrative-centered learning environments, this paper presents an inductive framework for inferring users’ psychological characteristics from observations of their interactions with virtual agents. Trained on traces of users’ interactions with virtual agents in the environment, psychological user models are induced from the interactions to accurately infer different aspects of a user’s personality. Further, analyses of timing data suggest that these induced models are also able to converge on correct predictions after a relatively small number of interactions with virtual agents.

This paper describes the challenges of getting ground truth affective labels for spontaneous video, and presents implications for systems such as virtual agents that have automated facial analysis capabilities. We first present a dataset from an intelligent tutoring application and describe the most prevalent approach to labeling such data. We then present an alternative labeling approach, which closely models how the majority of automated facial analysis systems are designed. We show that while participants, peers and trained judges report high inter-rater agreement on expressions of delight, confusion, flow, frustration, boredom, surprise, and neutral when shown the entire 30 minutes of video for each participant, inter-rater agreement drops below chance when human coders are asked to watch and label short 8 second clips for the same set of labels. We also perform discriminative analysis for facial action units for each affective state represented in the clips. The results emphasize that human coders heavily rely on factors such as familiarity of the person and context of the interaction to correctly infer a person’s affective state; without this information, the reliability of humans as well as machines attributing affective labels to spontaneous facial-head movements drops significantly.

The challenge of making a virtual world believable includes a requirement for AI entities which autonomously react to a dynamic environment. After the breakthroughs in believability introduced by modern lightning and physics techniques, the focus is shifting to better AI behavior sophistication. Avatars and agents in a realistic virtual environment must exhibit a certain degree of presence and awareness of the surroundings, reacting consistently to unexpected contingencies and social situations. Unconscious reactions serve as evidence of life, and can also signal social availability and spatial awareness to others. These behaviors get lost when avatar motion requires explicit user control. This paper presents a new approach for generating believable social behavior in avatars. The focus is on human territorial behaviors during social interactions, such as during conversations and gatherings. Driven by theories on human territoriality, we define a reactive framework which allows avatars group dynamics during social interaction. This approach gives us enough flexibility to model the territorial dynamics of social interactions as a set of social norms which constrain the avatar’s reactivity by running a set of behaviors which blend together. The resulting social group behavior appears relatively robust, but perhaps more importantly, it starts to bring a new sense of relevance and continuity to virtual bodies that often get separated from the simulated social situation.

This paper describes a model for generating steering behaviors of groups of characters based on the biologically-motivated space colonization algorithm. This algorithm has been used in the past for generating leaf venation patterns and tree structures, simulating the competition for space between growing veins or branches. Adapted to character animation, this model is responsible for the motion control of characters providing robust and realistic group behaviors by adjusting just a few parameters. The main contributions are related with the robustness, flexibility and simplicity to control groups of characters.

In this paper we present an implementation of a embodied conversational agent that serves as a virtual tour guide in Second Life. We show how we combined the abilities of a conversational agent with navigation in the world and present some preliminary evaluation results.

This paper describes the design and implementation of a virtual agent that is capable of providing customers in a 3D online shop with advice. Based on a product knowledge base, a conversation model and a model of the shop, the agent communicates with the customer through text based dialogues and leads the customer through the virtual world using gestures.

Actor agents in a virtual reality and interactive drama must deliver a believable performance. This is challenging and many issues need to be resolved. One prominent problem is that the goals of a character in an emergent interactive drama may conflict with demands for coherency in the drama’s progress. We propose an approach to resolve this that uses Object Oriented Bayesian networks and intelligent agents with dual roles: an actor role focused on the dramatic goals of the story, and a character role focused on believable actions in line with the character’s personality and goals.

Embodied agents are a powerful paradigm for current and future multimodal interfaces, yet require high effort and expertise for their creation, assembly and animation control. Therefore, open animation engines and high-level control languages are required to make embodied agents accessible to researchers and developers. In this paper, we present EMBR, a new realtime character animation engine that offers a high degree of animation control via the EMBRScript language. We argue that a new layer of control, the

animation layer

, is necessary to keep the higher-level control layers (behavioral/functional) consistent and slim, while allowing a unified and abstract access to the animation engine, e.g. for the procedural animation of nonverbal behavior. We describe the EMBRScript animation layer, the architecture of the EMBR engine, its integration into larger project contexts, and conclude with a concrete application.

Effective speakers engage their whole body when they gesture. It is difficult, however, to create such full body motion in animated agents while still supporting a large and flexible gesture set. This paper presents a hybrid system that combines motion capture data with a procedural animation system for arm gestures. Procedural approaches are well suited to supporting a large and easily modified set of gestures, but are less adept at producing subtle, full body movement. Our system aligns small motion capture samples of lower body movement, and procedurally generated spine rotation, with gesture strokes to create convincing full-body movement. A combined prediction model based on a Markov model and association rules is used to select these clips. Given basic information on the stroke, the system is fully automatic. A user study compares three cases: the model turned off, and two variants of our algorithm. Both versions of the model were shown to be preferable to no model and guidance is given on which variant is preferable.

Agents cannot be decoupled from their environment. An agent perceives and acts in a world and the model of the world influences how the agent makes decisions. Most systems with virtual embodied agents simulate the environment within a specific realization engine such as the graphics engine. As a consequence, these agents are bound to a particular kind of environment which compromises their reusability across different applications. We propose the ION Framework, a framework for simulating virtual environments which separates the simulation environment from the realization engine. In doing so, it facilitates the integration and reuse of the several components of the system. The ION Framework was used to create several 3D virtual worlds populated with autonomous embodied agents that were tested with hundreds of users.

Two tools for developing embodied conversational agents and deploying them over the world-wide web to standard web browsers are presented. DTask is a hierarchical task decomposition-based dialogue planner, based on the CEA-2018 task description language standard. LiteBody is an extensible, web-based BML renderer that runs in most contemporary web browsers with no additional software and provides a conversational virtual agent with a range of conversational nonverbal behavior adequate for many user-agent interaction applications. Together, these tools provide a complete platform for deploying web-based conversational agents, and are actively being used on two health counseling applications.

Over the past decade, motion capture data has become a popular research tool, and motion databases have grown exponentially. Indexing, querying, and retrieving data has thus become more difficult, and has necessitated innovative approaches to using these databases. Our aim is to make this approach feasible for virtual agents signing in French Sign Language (LSF), taking into account the semantic information implicitly contained in language data. We thus structure a database in two autonomous units, taking advantage of differing indexing methods within each. This allows us to effectively retrieve captured motions to produce LSF animations. We describe our methods for querying motion in the semantic database, computing transitory segments between concatenated signs, and producing realistic animations of a virtual LSF signer.

This paper presents the process of creation of virtual agents used in a virtual reality performance. The performance aimed to investigate how drama and performance could inform the creation of virtual agents and also how virtual reality could raise questions for drama and performance. The virtual agents were based on the performance of 2 actors. This paper describes the process of preparing the actors, capturing their performances and transferring them to the virtual agents. A second set of agents was created using non-professional ‘naïve performers’ rather than actors.

Several challenges remain in the effort to build software capable of conducting realtime dialogue with people. Part of the problem has been a lack of realtime flexibility, especially with regards to turntaking. We have built a system that can adapt its turntaking behavior in natural dialogue, learning to minimize unwanted interruptions and “awkward silences”. The system learns this dynamically during the interaction in less than 30 turns, without special training sessions. Here we describe the system and its performance when interacting with people in the role of an interviewer. A prior evaluation of the system included 10 interactions with a single artificial agent (a non-learning version of itself); the new data consists of 10 interaction sessions with 10 different humans. Results show performance to be close to a human’s in natural, polite dialogue, with 20% of the turn transitions taking place in under 300 msecs and 60% under 500 msecs. The system works in real-world settings, achieving robust learning in spite of noisy data. The modularity of the architecture gives it significant potential for extensions beyond the interview scenario described here.

We describe the design and evaluation of an agent that uses the fillers

um

and

uh

in its speech. We describe an empirical study of human-human dialogue, analyzing gaze behavior during the production of fillers and use this data to develop a model of agent-based gaze behavior. We find that speakers are significantly more likely to gaze away from their dialogue partner while uttering fillers, especially if the filler occurs at the beginning of a speaking turn. This model is evaluated in a preliminary experiment. Results indicate mixed attitudes towards an agent that uses conversational fillers in its speech.

We look at how some basic choices in the management of turns influence the impression that people get from an agent. We look at scales concerning personality, emotion and interpersonal stance. We do this by a person perception study, or rather an agent perception study, using simulated conversations that systematically vary basic turn-taking strategies. We show how we can create different impressions of friendliness, rudeness, arousal and several other dimensions by varying the timing of the start of a turn with respect to the ending of the interlocutor’s turn and by varying the strategy of ending or not ending a turn when overlap is detected.

Currently, applications that focus on providing conversations with virtual humans require extensive work to create robust conversational models. We present a new approach called Human-centered Distributed Conversational Modeling. Using this approach, users create conversational models in a distributed manner. To do this, end-users interact with virtual humans to provide new stimuli (questions and statements), and domain-specific experts (e.g. medical/psychology educators) provide new virtual human responses. Using this process, users become the primary developers of conversational models. We tested our approach by creating an example application, Virtual People Factory. Using Virtual People Factory, a pharmacy instructor and 186 pharmacy students were able to create a robust conversational model in 15 hours. This is approximately 10% of the time typical in current approaches and results in more comprehensive coverage of the conversational space. In addition, surveys demonstrate the acceptability of this approach by both educators and students.

In this paper, we present our current study on the design of a virtual agent able to express sentences in Sign Language (SL), so-called ‘Virtual Signer’, in the context of interactive web-based applications.

Up now, little work has been carried out on web-based virtual signers. As for current research on virtual signers in standalone applications, it relies on two main approaches: Pre-synthesized animations based on motion capture [1] or rotoscoping [2], and generated animations [3], with very little interactivity. Our aim is to provide the virtual signer with richer interacting capabilities with the data contained in the web page.

Episodic memory

(EM) is an umbrella term for memory systems that operate with representations of personal history of an entity. The content of EM is related to particular places and moments, and connected to subjective feelings and current goals.

Recently, it has been argued that EM is one of the key components contributing to believability of intelligent virtual agents (IVAs), at least when agents interact with humans for more than a couple of minutes, because it allows the user to understand better the agent’s history, personality, and internal state: both actual state and past state [e.g. 2, 5]. Technically, the EM is merely a data structure for loss compression of the flow of external events. The EM cannot be implemented as a pure log/video, because these are bad data structures (including human level). Why are they bad? First, they produce too large data. Second, they are not well organised with respect to future possible queries: neither a log nor a video have appropriate indexes in terms of database systems. A better approach is needed.

In this paper we present RealActor, a character behavior realization system for embodied conversational agents based on the Behavior Markup Language (BML). Developed several years ago as part of the SAIBA framework, BML is an XML dialect for describing physical realizations of multimodal human behaviors. It allows modeling of complex communicative utterances which include both verbal and non-verbal behavior. BML elements represent various primitive actions (e.g. speech, facial and body gestures) and multimodal behavior can be modeled by specifying temporal relationships between these elements.

This paper provides a method to make locomotion animation using riding motion, which is synthesized by computer simulation. Our locomotion system uses riding motion to avoid collision between objects instead of traditional obstacle avoidance. The two-dimensional locomotion using obstacle avoidance has a problem in that positions that the agent move is limited if the number of agents is large. In game or film, user or artist should throw up natural movement to locate agents in the specific area to avoid obstacle. The major strengths of the algorithm is fast calculation and natural locomotion motion, the weakness is that the algorithm is not applied to various creatures

Emotive virtual humans (VHs) are important for affective interactions with embodied conversation agents [1]. However, emotive VHs require significant resources and time. As an example, the VHs in movies and video games require teams of animators and months of work. VHs can also be imbued with emotion using appraisal theory methods that use psychology based models to generate emotions by using the VH’s goals and beliefs to evaluate external events. These external events require manual tagging or natural language understanding [2]. As an alternative approach, we propose tagging VH responses with emotions using textual affect sensing methods. The method developed by Neviarouskaya et al. [3] uses syntactic parses and a database of words and associated emotion intensities.We use this database, and because these emotions are associated with specific words, we can combine the emotions with audio timing information to generate lip-synched facial expressions. Our approach, AutoEmotion, allows us to automatically add basic emotions to VHs without the need for manual animation or tagging or natural language understanding.

This paper aims to present e the work developed at Cnotinfor within the LIREC project. We will present Little Mozart mind architecture and facial expressions. Our goal is to create a virtual e-learning agent capable of establishing meaningful interactions with children on how to compose and improve their knowledge of melodic composition and basics of musical language. In order to make our agent believable and engaging we resorted to facial and body 3D modeling techniques as a strategy for expressing emotion through multimodal communication.

A great challenge that is to be faced in the design of virtual agents is the issue of credibility, not only in the agent’s aspect but also in its behavior [1]. To be believable, the agent has to decide what to do next according to the internal and external variables of the agent. Besides others, we have to deal with the problem of action selection which can be resumed to choose the most appropriate action among all possible (conflicting) ones [2]. In our case, actions are backchannels. This work is part of the STREP EU SEMAINE project in which a real-time Embodied Conversational Agent (ECA) will be a Sensitive Artificial Listener (SAL) [3]. This project aims to build an autonomous talking agent able to exhibit autonomously appropriate verbal and non verbal behaviors in real-time when it plays the role of the listener in a conversation with a user.

This paper presents the work to enhance virtual agents with the abilities of interacting with the environment and planning based on goals and internal states. We describe how we designed intelligent virtual agents that are able to perform tasks on a high-risk plant and that respond to expected as well as unexpected events. We proposed an informed virtual environment where agents are able to interact with other agents or avatars. By enhancing their reasoning abilities with cognitive rules and the environment with knowledge the virtual agents are able to exhibit deviated behaviors.

Games and Virtual Environments populated with virtual agents are usually tested and evaluated assessing the player’s experience and flow outside the game experience. However, with player enjoyment related to their sense of immersion, there is clear potential for immersing evaluation into game play. We have developed an immersive in-role evaluation approach and applied it to ORIENT (

Overcoming

Refugee Integration with Empathic Novel Technology)

, a role-playing game designed for intercultural training for 13-14 year olds.

Gunslinger isan interactive-entertainment application of virtual humans that transforms an iconic Wild West movie scene into a vivid semblance of reality. The project combines virtual humans technology with Hollywood storytelling and set building into an engaging, mixed-reality, story-driven experience, where a single participant can interact verbally and non-verbally with multiple virtual characters that are imbedded in a real, physical saloon. The Gunslinger project pushes the frontier of virtual humans research by combining question-answering dialogue techniques with explicit story representation. It incorporates speech recognition techniques and visual sensing to recognize multimodal user input. It further extends existing behavior generation methods such as SmartBody to drive tightly coupled dialogue amongst characters. These capabilities strive to seek a balance between open ended dialogue interaction and carefully crafted narrative.

Social relationships are fundamentally important to humans. Children are born into a family social structure and gradually they become part of a wider social structure including peers, in which each individual’s relationships are affected by other relationships. To most people expressing appropriate non-verbal social behaviours when interacting with others in a virtual environment (VE) is natural and effortless as they do so daily in the physical world. There is a minority group, such as people diagnosed with autistic spectrum disorder, who encounter difficulties when carrying out this task because of impairments of specific skills regarding social communication and communication [1]. In this paper, we study the design of a system using advanced VE and IVA technologies with the goal to help children with autism to explore and better understand social dynamics occurring in social networks involving groups of people.

For simulating communicative behavior, realistic appearance and plausible behavior is important. Postures and mimics also reflect emotional behavior. There exist various models of emotion, like the psycho-evolutionary theory developed by Plutchik [1], or Ekman’s FACS. But most models are only suited for muscular expressions. A more unattended field however, also in graphics, is the change in color. Also, other physiological symptoms like crying are left unconsidered.

We present recent extensions for the intelligent virtual agent development platform Pogamut 3. These extensions are: the agent coordination language StorySpeak designed for the purposes of storytelling applications, support for gestures based on BML, the visual editor of reactive plans, connection of the cognitive architecture ACT-R, an educational application aimed at orientation in urban areas and finally an emotional model.

Recent studies have shown that virtual humans can facilitate social interactions among people who have difficulty in forming social bonds or help develop their social skills by interaction with virtual practice. Virtual human research tends to focus exclusively on appearance or behavior in assessing agent effectiveness, whereas other studies of human social interaction emphasize aspects of the social context, such as anticipated future interaction (AFI) which has been implicated as a key moderator of people’s behavior in virtual interactions. It has been reported that anonymity [2] or interactants’ AFI with their interaction partners [4,5] have a critical role in entailing greater self-disclosure. However, no studies of virtual humans have investigated the impact of the combination of interactants’ visual appearance and AFI on their social responses, specifically revealing their intimate information. In this study, we examined the impact of different visual realism of virtual humans and interactants’ AFI on interactants’ self-disclosure, when their anonymity is secured and future interaction may be anticipated.

Interactive Virtual Standardized Patients (VP) can provide meaningful training for clinicians. These VP’s portray interactive embodied conversational characters with realistic representations of a mental or physical problem to be diagnosed or discussed. This research is a continuation of evaluating of our VP “Justina” [2] which suffers from Posttraumatic Stress Disorder (PTSD) from a sexual attack and presents the results of comparing novices, test subjects without medical training, and experts interacting with ‘Justina’ to find out if they could elicit the proper responses to make a diagnosis and to investigate the topics and questions the novices asked for coverage of the categories and criteria of PTSD as defined in the DSM-IV [1]. It is assumed that novices will perform better than experts, however the main investigation is to gather empirical data and understand why this is true and how this can be used to improve the system. There have not been, to the authors’ knowledge, any studies in evaluating experts and non-experts with virtual human characters in the psychological domain.

Video game technology now enables a new set of experiences for players in addition to conventional video games. Although the intensity of the technological research in this field is shifting from graphics to AI design and characters with embodied agents [1], there have been few studies involving the assessment of real-time sequential, emotional changes of video-game players. Research of this issue and the implications may assist in preventing game addiction and decreasing one’s zest for learning. This paper proposes a novel interface to provide voice feedback to video-game players using real-time sequential emotion estimation from facial expressions.

Coming to open domain dialog it is still unrealistic to implement needed knowledge resources and dialog skills linguistically. Since the Non Player Characters (NPC) in our NPC Engine should be capable of open conversation we decided to use an Artificial Intelligence Markup Language (AIML) [1] chatbot as a first workaround. However AIML chatbots are not linguistically motivated, they use surface structures in tradition of Weizenbaums ELIZA [2], wherefore they do have too many shortcomings to use them in real dialog applications. One of the major problems is the handling of surface variation. To be able to process one sentence with different structures, they need as much patterns as there are syntactic alternatives, which leads to an exploding number of pattern template pairs. To reduce the costs of the manual development, AIML authors use simple regular expression operators, what in turn leads to the problem of being overly permissive. A possibility to abstract from the surface level and still be able to control the structure is to use information from syntactic and semantic analysis. In the presented system two scenarios were implemented: 1. Using part of speech information and 2. Using the results from a semantic analysis.

Avatars are frequently used in virtual worlds and online games to convey emotions across cultures. However, our previous study suggested there are cultural differences in recognizing avatar’s facial expressions [1]. Yuki et al.’s study using emoticons and photorealistic human face images suggests Americans tend to interpret emotions based on the mouth, while Japanese tend to focus on the eyes [2]. Inspired by Yuki’s study, this study uses cartoonish avatar faces to find cultural differences in using facial parts as cues to recognize avatar emotions. This paper reports the preliminary result of an experiment conducted between Japanese and European subjects.

We present the Adaptive Mind Agent, an intelligent virtual agent that is able to actively participate in a real-time, dynamic environment. The agent is equipped with a collection of processing tools that form the basis of its perception from and action on the environment consisting of web documents, URLs, RRS feeds, domain-specific knowledgebases, other accessible virtual agents and the user. How these predispositions are finally shaped into unique agent behaviour depends on the agent’s abilities to learn through actual interactions, in particular the abilities: (i) to memorize and evaluate episodes comprising the actions the agent had performed on its environment in the past depending on its perceptions of the user requests and its interpretation of the user’s feedback reinforcing or inhibiting a certain action; (ii) to dynamically develop user-driven interest and preference profiles through memorizing and evaluating the user clicks on selected web pages.

Embodied conversational agents (ECAs) provide one possible way of incorporating nonverbal portion of speech into voice based user interfaces. Part of agent’s visual behavior and its appearance is often solved “statically.” There is another possibility to change appearance -

dynamically

(for example head or eye movements, mouth opening, etc.). Our hypothesis is that some “dynamic” parameters are more important for the user than the other ones. In this paper we present the pilot user study and statistical evaluation of four parameters. Eye blinking is one of the parameter we investigate. The importance of agent’s gaze (blinking as part of it) was discussed several times for example by Raidt [1].

The importance of sound and music in movies (Film Scoring) is undeniable. Film Scoring is used as a complement to the visual content helping to better understand it. The use of these artistic elements can be extended to interactive storytelling (IS) environments where stories are also told in a dynamic and interactive way. Because of IS particularities, such as the unpredictability of user actions and story development, this extension is not straightforward, and presents a problem that needs to be addressed. Further, the presence of autonomous virtual agents makes the creation of these stories more emergent, making difficult the process of creating sounds and music that is adequate. This paper presents a framework that proposes a possible solution to score a story created in these environments by intelligent virtual agents. Additionally, we studied the influence of some particular elements of sound and music (such as tempo, use of different instruments, etc) on the viewers and their perception of the actions of the characters and thus, on the consequent understanding of the story.

Intelligent Virtual Agent (IVA) systems are notoriously difficult to evaluate, particularly due to the subjectivity involved. From the various efforts to develop standard evaluation schemes for IVA systems the scheme proposed by Isbister & Doyle, which evaluates systems across five categories, seems particularly appropriate. To examine how these categories are being used, the evaluations presented in the proceedings of IVA ’07 and IVA ’08 are summarised and the extent to which the five categories in the Isbister & Doyle scheme are used is highlighted.

Embodied Conversational Agents (ECAs) are commonly accepted as a new metaphor of Human Computer Interaction in several application domains. They are also used with the roles of influence and behavior change. We present an empirical evaluation study to compare the motivational impact of a persuasive message when conveyed by Valentina (a female young character [1]) and when presented through a text. The believability of our agent has been successfully evaluated in our previous research.

Du

arte

Digital is an agent that engages in inquiry-oriented conversations about an art artifact. Since it was build for a Museum, interactions are supposed to be directed to different types of audience: an interaction with an art expert should be carried out in a different way than an interaction with a child; likewise, interactions with users interested in learning should be distinct from interactions with users having only entertainment goals. Being so, an agent needs to undergo two tasks: it must understand the user’s knowledge about the topic, and his/her learning goals; it should adapt its vocabulary and dialogue strategy to cope with the user’s characteristics and expectations.

This paper presents a simple and straighforward model of interaction that allows a virtual agent to understand its interlocutors based on their vocabulary and to adapt to their expertise and needs.

Thanks to the progress of computer vision technologies and human sensing technologies, human behaviors, such as gaze and head poses, can be accurately measured in real time. Previous studies in multimodal user interfaces and intelligent virtual agents presented many interesting applications by exploiting such sensing technologies [1, 2]. However, little has been studied how to extract communication signals from a huge amount of data, and how to use such data in dialogue management in conversational agents.

Researchers of systems for Digital Entertainment have resorted to Artificial Intelligence to create characters that are more adaptable to new situations, less predictable, with fast learning capabilities, memory of past situations and a variety of convincing and consistent behaviors.

Recent studies in cognitive psychology and neuroscience analyze the fundamental role of personality and emotions in human cognition, based on the notions of perception, attention, planning, reasoning, learning, memory and decision making. These notions can be characterized as modules in a formal model to describe the personality and emotions of autonomous agents, whose manifestations can be directly dependent upon personality and emotional states. Future research in affective computing must explore how emotions interact with other modules in agent architectures (such as memory and decision making), as well as how emotions influence the interactions with other agents [1].

Although there is progress in modeling and implementing affective phenomena – computation of emotion – the new direction proposed here is to model and represent irrational and subconscious processes, the hidden aspects of the human psyche as analysed by Freud and his successors, and by other disciplines in psychotherapy.

The outcome will be a ‘psychodynamics engine’ that does not simply respond emotionally to context and interaction, but is driven by its own convoluted patterns, triggers and processes. This can potentially be used in an affective architecture for games, companioniable robotics, online virtual entities etc., interacting with humans.

In multi-user interaction, Embodied Conversational Agents (ECAs) have to detect mental states of each user to interact smoothly and naturally. In addition, mental states of a person may be affected by extrinsic factors, such as states of people around. Physiological indices was useful to understand mental states of a person. However, it was impractical that ECAs measure physiological indices of users. The purpose of this study was whether we could detect intrinsic “involvement, enjoyment, and/or excitement” (“IEE”) of a person and extrinsic “IEE” of an atmosphere, by using visual information. In other words, the purpose is to develop a method to detect an atmosphere of “IEE” in multi-user interaction.

This paper describes the ORACLE, an embodied intelligent virtual agent designed and built for helping players of a serious game named ORIENT [1]. The game challenges players at the socio-cultural level, because it requires them to learn how to interact smoothly with alien characters in order to save their planet. The ORACLE is a virtual agent on a mobile phone, whose goal is to enhance the users’ learning in the game. The agent can both react to the players’ questions, and proactively intervene for suggesting things to users or for commenting on their performance.

This poster reports the results of two experiments to test a personality framework for virtual characters. We use the Tactical Questioning dialogue system architecture (TACQ) [1] as a testbed for this effort. Characters built using the TACQ architecture can be used by trainees to practice their questioning skills by engaging in a role-play with a virtual human. The architecture supports advanced behavior in a questioning setting, including deceptive behavior, simple negotiations about whether to answer, tracking subdialogues for offers/threats, grounding behavior, and maintenance of the affective state of the virtual human. Trainees can use different questioning tactics in their sessions. In order for the questioning training to be effective, trainees should have experience of interacting with virtual humans with different personalities, who react in different ways to the same questioning tactics.

This paper describes the design and training of an agent that helps users navigate in Second Life (SL), a massively multi-player online game environment.SL allows users to create a virtual avatar and explore areas constructed by other users.With the increasing number of places to visit it is difficult for a user to explore all the places.The built-in keyword search mechanism is fairly limited and users typically find new places through personal exploration or tips from their friends.This motivates the need for a recommendation system that can suggest places to visit, personalized with the user’s destination preferences.

Game agents(NPCs) should have the ability to react in cooperate, and have the ability to learn from mistakes and build up their own experience. In this paper, we describe a general approach for transfer learning and adaptive mechanism for game agents’ real-time planning and learning system in which agents modify their behavior in response to changes of the PCs.

Business strategy educators increasingly rely on simulation games to provide students with experience in strategic decision making. Business games however often limit interaction to operational decision making so discovering the value of alternative strategies in different situations may take a lot of time. Furthermore, to ensure students can grasp the system complexity at this operational level, game scenarios remain relatively simple. We hypothesize that business game effectiveness for strategy education can increase if players instead create teams of delegate agents to appropriately handle operations in more complex settings. To test this we are working on an intelligent learning environment that will help players create successful teams of agents to handle operations.

This study investigated whether emotional expressions of an ECA influence the participants´ nonverbal and verbal behavior. 70 participants took part in a small talk (10 min.) situation with the ECA MAX who was presented with two different types of feedback: emotional feedback (EMO), which provided a feedback about the emotional state of MAX (including smiles and compliments) and envelope feedback (ENV), which provided a feedback about the comprehension of the participants´ contributions. In general we found that participants showed frequent behavior known from human-human-communication, such as proactive greetings or waving goodbye. Additionally, with regard to some behaviors, the agent´s behavior had an impact: Participants in EMO significantly gave more compliments and exhibited more phrases of politeness (“Thank you!”) than in ENV.

Web-based tests are an established method for the evaluation (e.g. [1]). However, the validity of web-experiments has not yet been evaluated for the field of embodied conversation agents (ECAs). In this paper, evaluation results obtained in a non-interactive scenario in our lab [2] are compared to a similar web-based experiment. 3 different talking head components and 2 freely available speech synthesis systems (TTS) are combined. The head components used are the thinking head (TH), Massy (MS), and a speaker cloning system (CL). The TTS include Mary (‘hmm-bits3’) and Mbrola (’de2’). We recorded videos of the talking heads speaking sentences related to the smart home domain like:

The following devices can be turned on or off: the TV, the lamps and the fan.’

Our aim is to implement a motion capture database for the synthesis of virtual agents signing in French Sign Language (LSF), taking into account the semantic information implicitly contained in language data. The Gérard system is a first iteration along the path toward efficient and convincing animations of an LSF signer.

The virtual world of Second Life

TM

does not offer support for complex facial animations, such as those needed for an intelligent virtual agent to lip sync to audio clips. However, it is possible to access a limited range of default facial animations through the native scripting language, LSL. Our solution to produce lip sync in this environment is to rapidly trigger and stop these default animations in custom sequences to produce the illusion that the intelligent virtual agent is speaking the phrases being heard.

The tennis game that is described in the xml is a double, thus there are four tennis players on the court. There is a handful of spectators that are really excited about the game. The spectators have different preferences about the teams that they favor or are neutral. To vary the mood-states, each spectator has an euphoria factor that determines how much a change in the mood-state will affect it. The mood-state of a spectator is shown with expressive facial expressions and some typical animations. Additionally to the animations and expressions the mood-state is expressed by speech as well.

Multimodal user interfaces are becoming more and more important in human–machine communication. Essential representatives of such interfaces are virtual agents that aim to act like humans in the way they employ gestures, facial expression, posture and prosody to convey their emotions in face-to-face communication. Furthermore, if we employ a presentation team [1] to convey facts, the performance becomes more entertaining for the audience. Distinct characters should have different roles and personality profiles. Moreover if we integrate interaction capabilities for the user, the system becomes more personalized and enjoyable.

We present ongoing work on

CREACTOR,

a research oriented authoring system for virtual actors. CREACTOR will provide a framework for experimenting with different authoring processes and AI-technologies. The main goal is the creation of virtual actors that can be employed for the development of 3D movies, following the analogy of a real director who can issue a variety of commands to real actors. Here, we present a concept called AI-tweening: employing AI to create in-between behaviors.

The multi-modal rock-paper-scissors game is an interactive computer game where the opponent of the human player is a virtual agent appearing on the computer screen. The game is similar to the game between humans, the communication takes place by the tools of image and sound processing.

In conjunction with Anvil and suitable annotation schemes,

GAnTooL

(A Gesture Annotation And Modeling Tool for Anvil) is a tool to annotate human nonverbal behavior like gestures and poses efficiently with the help of a skeleton. Using intuitive controls the user can quickly mirror the observed speaker’s poses. The results can be used to build gesture descriptions and whole lexicons that transfer human behavior to interactive embodied agents. These agents can then be animated realtime with a character animation engine like

EMBR

.

GAnTooL

can also be used to create rough animations for games or movies: an export option in the

Collada

standard allows further editing in standard 3D modeling tools.